Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-27
18:00:08 mnaser melwitt: wanna give me something to copy pasta into the topic?
18:00:29 melwitt yeah, lemme see. I can't remember what order it was in
18:01:46 mriedem i've talked with dtroyer about the osc live migration support, i think he basically wants to just re-write the command on a major version
18:02:04 mriedem mnaser: if you do specify a host with osc's live migration command, make sure you are using the microversion that doesn't bypass the scheduler
18:02:35 mriedem https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id27
18:02:45 mnaser mriedem: i switched to using python-novaclient because i explicitly want the scheduler to decide for me
18:03:02 mriedem tbc, the live migration API explicitly requires that the host param is sent, it's not optional, but the value in the REST API can be None (it's dumb)
18:03:17 mriedem but ^ isn't possible in osc b/c you can't specify None on the command line
18:03:34 mriedem you would need to make --host optional
18:04:05 mriedem er --live <hostname>
18:04:06 mriedem https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-migrate
18:04:12 mriedem like, --live ''
18:04:18 mnaser mriedem: hence https://review.openstack.org/#/c/589012/
18:04:28 mriedem yes, there is another one as well
18:04:42 mnaser but yeah i dunno
18:04:49 mnaser microversions for osc api?
18:04:53 mnaser heck yeah
18:05:10 mriedem https://review.openstack.org/#/c/460059/
18:05:15 mnaser (i'm kidding please don't end me)
18:05:18 mriedem you can pass microversions through for osc
18:05:34 mriedem openstack --os-compute-api-version 2.30 migrate --live
18:05:39 mriedem or set an env var
18:05:48 mnaser yeah no but i meant like
18:05:51 mriedem osc doesn't do version negotiation to default to the latest like nova cli
18:05:54 mnaser the actaul openstack client shell api or whatever
18:06:13 mnaser but i guess we call those just *releases* of the client
18:06:48 mriedem https://review.openstack.org/#/c/460059/7//COMMIT_MSG@15
18:06:51 mriedem has my suggestion in it
18:08:07 mriedem you also can't bfv with osc
18:08:13 mriedem unless you use an existing volume
18:08:22 mriedem those are two pretty big gaps in functionality
18:08:33 mnaser now on another note i think live migrations are scheduled when they are received, not when they start, right?
18:08:41 mriedem correct,
18:08:44 mriedem we have to pick the dest host
18:09:03 mriedem so we can setup things like port bindings and volume attachments
18:09:05 mnaser makes our hypervisor evacuations a bit more annoying in that we can kinda do one at a time
18:09:27 mriedem can only do?
18:09:54 mnaser well if we do host-evacuate-live on 3 nodes at once
18:09:58 mnaser its possible that they schedule to each other
18:10:14 mnaser (sometimes we actually do host-evacuate-live only for the purpose of having instances go in the right place after we make scheduler changes)
18:10:22 mriedem if you are evacuating the hosts, you could disable the compute service
18:10:33 mriedem not evacuate like the evacuate API, i mean "get the vms off this host"
18:10:48 mnaser yeah when we're shutting things down that's what we go for, but when shuffling things around
18:11:29 mnaser you kinda just want instances to move into where they are supposed to go
18:11:36 mnaser anyways
18:11:40 mnaser very minor thing
18:11:45 mriedem "you kinda just want instances to move into where they are supposed to go"
18:11:46 mriedem ha
18:11:50 mriedem of course!
18:11:55 mriedem silly scheduler
18:12:17 mriedem well as a tc big wig,
18:12:23 mriedem you can influence the goal setting for T
18:12:37 mnaser usually after we make scheduling tweaks
18:12:41 mnaser we just do a rolling live migration
18:12:52 mriedem so watcher but without watcher
18:13:00 mnaser just a one time watcher
18:13:57 mnaser i think watcher is super interesting but i think it depends on too many things
18:14:57 mriedem well as a tc big wig,
18:15:13 mriedem you can influence adoption of a new top-level project: watcher-lite
18:15:39 mriedem watcher zero
18:15:46 mriedem all of the flavor, none of the guilt
18:17:15 mnaser lols
18:21:25 melwitt mnaser: Current runways: use-nested-allocation-candidates -- This channel is for Nova development. For support of Nova deployments, please use #openstack.
18:22:51 mnaser melwitt: voila i've made myself useful for today
18:23:14 melwitt \o/
18:23:55 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: nova-manage - fix online_data_migrations counts https://review.openstack.org/605828
18:26:10 mriedem imacdonn: +W on https://review.openstack.org/#/c/605329/ and i found an example of a migration that has total > 0 with completed == 0
18:26:39 imacdonn mriedem: ack. I was just about to ask you if you intentionally didn't +W with your +2 :)
18:26:48 mriedem it was intentional,
18:27:01 mriedem because i was going to backport to stable and get a grenade run where i knew we actually had things to migrate
18:27:05 mriedem but i found one in stein too
18:27:10 imacdonn got it
18:27:13 mriedem http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536
18:27:17 mriedem 2 rows matched query migrate_instances_add_request_spec, 0 migrated
18:27:30 mriedem ^ is with your change
18:27:41 mriedem | populate_queued_for_delete | 2 | 2 |
18:27:46 mriedem http://logs.openstack.org/88/605488/1/check/neutron-grenade/d64e316/logs/grenade.sh.txt.gz#_2018-09-27_01_15_47_182 is without
18:27:52 mriedem | populate_queued_for_delete | 0 | 0 |
18:27:56 imacdonn \o/
18:28:16 mriedem https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/objects/request_spec.py#L707 is the example where total can be >0 but we don't actually migrate anything
18:29:04 mriedem so that with the explanation here https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/cmd/manage.py#L374 is confusing
18:29:20 imacdonn that may be a bug
18:29:31 mriedem it does say, "If found is nonzero and done is zero, some records are # not migratable, but all migrations that can complete have # finished."
18:29:48 mriedem "not migrateable" should really be, "don't require migration"
18:29:52 imacdonn per Dan's description, count_all should never be greater than max_count
18:30:15 mriedem a lot of the migrations return found==done because the query to find the $found number is filtering on things that need to be migrated
18:30:25 mriedem e.g. select * bdms where uuid is None;
18:30:33 mriedem found == done ^
18:30:41 mriedem but that's not the same with the request spec migratoin
18:30:44 mriedem *migration
18:30:48 mriedem since we have to hit 2 different dbs
18:31:44 imacdonn personally I think the batch mechanism is a bit broken, at least as it's described in the comments
18:31:50 mriedem http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536
18:31:50 mriedem as i mentioned on your change, the 'Total Needed' column is misleading
18:31:52 imacdonn but, as you said, we need Dan for that conversation
18:41:07 mnaser so has anyone ever thought what happens when we hit instance-ffffffff
18:43:10 imacdonn fsshhh ... that'll never happen
18:44:08 melwitt in case anyone is wondering about the failing ceph job, I'm trying out a fix here https://review.openstack.org/605833
18:48:19 mnaser mordred: it's a bit of a difficult position but ideally figuring out what the best way to deal with cold/live migration and reworking it.. (openstack server migrate)
18:48:37 mordred yah. as you know, the sdk code for that is ... fun :)

Earlier   Later