| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-27 | |||
| 18:04:06 | mriedem | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-migrate | |
| 18:04:12 | mriedem | like, --live '' | |
| 18:04:18 | mnaser | mriedem: hence https://review.openstack.org/#/c/589012/ | |
| 18:04:28 | mriedem | yes, there is another one as well | |
| 18:04:42 | mnaser | but yeah i dunno | |
| 18:04:49 | mnaser | microversions for osc api? | |
| 18:04:53 | mnaser | heck yeah | |
| 18:05:10 | mriedem | https://review.openstack.org/#/c/460059/ | |
| 18:05:15 | mnaser | (i'm kidding please don't end me) | |
| 18:05:18 | mriedem | you can pass microversions through for osc | |
| 18:05:34 | mriedem | openstack --os-compute-api-version 2.30 migrate --live | |
| 18:05:39 | mriedem | or set an env var | |
| 18:05:48 | mnaser | yeah no but i meant like | |
| 18:05:51 | mriedem | osc doesn't do version negotiation to default to the latest like nova cli | |
| 18:05:54 | mnaser | the actaul openstack client shell api or whatever | |
| 18:06:13 | mnaser | but i guess we call those just *releases* of the client | |
| 18:06:48 | mriedem | https://review.openstack.org/#/c/460059/7//COMMIT_MSG@15 | |
| 18:06:51 | mriedem | has my suggestion in it | |
| 18:08:07 | mriedem | you also can't bfv with osc | |
| 18:08:13 | mriedem | unless you use an existing volume | |
| 18:08:22 | mriedem | those are two pretty big gaps in functionality | |
| 18:08:33 | mnaser | now on another note i think live migrations are scheduled when they are received, not when they start, right? | |
| 18:08:41 | mriedem | correct, | |
| 18:08:44 | mriedem | we have to pick the dest host | |
| 18:09:03 | mriedem | so we can setup things like port bindings and volume attachments | |
| 18:09:05 | mnaser | makes our hypervisor evacuations a bit more annoying in that we can kinda do one at a time | |
| 18:09:27 | mriedem | can only do? | |
| 18:09:54 | mnaser | well if we do host-evacuate-live on 3 nodes at once | |
| 18:09:58 | mnaser | its possible that they schedule to each other | |
| 18:10:14 | mnaser | (sometimes we actually do host-evacuate-live only for the purpose of having instances go in the right place after we make scheduler changes) | |
| 18:10:22 | mriedem | if you are evacuating the hosts, you could disable the compute service | |
| 18:10:33 | mriedem | not evacuate like the evacuate API, i mean "get the vms off this host" | |
| 18:10:48 | mnaser | yeah when we're shutting things down that's what we go for, but when shuffling things around | |
| 18:11:29 | mnaser | you kinda just want instances to move into where they are supposed to go | |
| 18:11:36 | mnaser | anyways | |
| 18:11:40 | mnaser | very minor thing | |
| 18:11:45 | mriedem | "you kinda just want instances to move into where they are supposed to go" | |
| 18:11:46 | mriedem | ha | |
| 18:11:50 | mriedem | of course! | |
| 18:11:55 | mriedem | silly scheduler | |
| 18:12:17 | mriedem | well as a tc big wig, | |
| 18:12:23 | mriedem | you can influence the goal setting for T | |
| 18:12:37 | mnaser | usually after we make scheduling tweaks | |
| 18:12:41 | mnaser | we just do a rolling live migration | |
| 18:12:52 | mriedem | so watcher but without watcher | |
| 18:13:00 | mnaser | just a one time watcher | |
| 18:13:57 | mnaser | i think watcher is super interesting but i think it depends on too many things | |
| 18:14:57 | mriedem | well as a tc big wig, | |
| 18:15:13 | mriedem | you can influence adoption of a new top-level project: watcher-lite | |
| 18:15:39 | mriedem | watcher zero | |
| 18:15:46 | mriedem | all of the flavor, none of the guilt | |
| 18:17:15 | mnaser | lols | |
| 18:21:25 | melwitt | mnaser: Current runways: use-nested-allocation-candidates -- This channel is for Nova development. For support of Nova deployments, please use #openstack. | |
| 18:22:51 | mnaser | melwitt: voila i've made myself useful for today | |
| 18:23:14 | melwitt | \o/ | |
| 18:23:55 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: nova-manage - fix online_data_migrations counts https://review.openstack.org/605828 | |
| 18:26:10 | mriedem | imacdonn: +W on https://review.openstack.org/#/c/605329/ and i found an example of a migration that has total > 0 with completed == 0 | |
| 18:26:39 | imacdonn | mriedem: ack. I was just about to ask you if you intentionally didn't +W with your +2 :) | |
| 18:26:48 | mriedem | it was intentional, | |
| 18:27:01 | mriedem | because i was going to backport to stable and get a grenade run where i knew we actually had things to migrate | |
| 18:27:05 | mriedem | but i found one in stein too | |
| 18:27:10 | imacdonn | got it | |
| 18:27:13 | mriedem | http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536 | |
| 18:27:17 | mriedem | 2 rows matched query migrate_instances_add_request_spec, 0 migrated | |
| 18:27:30 | mriedem | ^ is with your change | |
| 18:27:41 | mriedem | | populate_queued_for_delete | 2 | 2 | | |
| 18:27:46 | mriedem | http://logs.openstack.org/88/605488/1/check/neutron-grenade/d64e316/logs/grenade.sh.txt.gz#_2018-09-27_01_15_47_182 is without | |
| 18:27:52 | mriedem | | populate_queued_for_delete | 0 | 0 | | |
| 18:27:56 | imacdonn | \o/ | |
| 18:28:16 | mriedem | https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/objects/request_spec.py#L707 is the example where total can be >0 but we don't actually migrate anything | |
| 18:29:04 | mriedem | so that with the explanation here https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/cmd/manage.py#L374 is confusing | |
| 18:29:20 | imacdonn | that may be a bug | |
| 18:29:31 | mriedem | it does say, "If found is nonzero and done is zero, some records are # not migratable, but all migrations that can complete have # finished." | |
| 18:29:48 | mriedem | "not migrateable" should really be, "don't require migration" | |
| 18:29:52 | imacdonn | per Dan's description, count_all should never be greater than max_count | |
| 18:30:15 | mriedem | a lot of the migrations return found==done because the query to find the $found number is filtering on things that need to be migrated | |
| 18:30:25 | mriedem | e.g. select * bdms where uuid is None; | |
| 18:30:33 | mriedem | found == done ^ | |
| 18:30:41 | mriedem | but that's not the same with the request spec migratoin | |
| 18:30:44 | mriedem | *migration | |
| 18:30:48 | mriedem | since we have to hit 2 different dbs | |
| 18:31:44 | imacdonn | personally I think the batch mechanism is a bit broken, at least as it's described in the comments | |
| 18:31:50 | mriedem | as i mentioned on your change, the 'Total Needed' column is misleading | |
| 18:31:50 | mriedem | http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536 | |
| 18:31:52 | imacdonn | but, as you said, we need Dan for that conversation | |
| 18:41:07 | mnaser | so has anyone ever thought what happens when we hit instance-ffffffff | |
| 18:43:10 | imacdonn | fsshhh ... that'll never happen | |
| 18:44:08 | melwitt | in case anyone is wondering about the failing ceph job, I'm trying out a fix here https://review.openstack.org/605833 | |
| 18:48:19 | mnaser | mordred: it's a bit of a difficult position but ideally figuring out what the best way to deal with cold/live migration and reworking it.. (openstack server migrate) | |
| 18:48:37 | mordred | yah. as you know, the sdk code for that is ... fun :) | |
| 18:48:47 | mnaser | mainly my issue was osc forces you to specify a host when its optional | |
| 18:48:47 | mordred | fwiw - mriedem is right - osc doesn't currently do version negotiation. once we start migrating it to sdk though, it'll pick up that ability | |
| 18:49:24 | mriedem | mnaser: easy: make --live just an option with no value, add --host (optional, takes a value), and add --cold | |
| 18:49:58 | mnaser | mriedem: but the not breaking scripts part i guess | |
| 18:50:02 | mriedem | or, let --live take a value for compat but proxy it to --host if --host isn't used | |
| 18:50:39 | mriedem | i'm not sure how you could specify --live w/o a host though if --live can take a host | |
| 18:50:41 | mriedem | gd CLIs | |
| 18:50:52 | mriedem | empty string? | |
| 18:50:58 | mriedem | pretty janky | |
| 18:51:25 | mriedem | openstack server migrate --live-but-with-no-host-seriously my_server | |