| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-27 | |||
| 18:09:05 | mnaser | makes our hypervisor evacuations a bit more annoying in that we can kinda do one at a time | |
| 18:09:27 | mriedem | can only do? | |
| 18:09:54 | mnaser | well if we do host-evacuate-live on 3 nodes at once | |
| 18:09:58 | mnaser | its possible that they schedule to each other | |
| 18:10:14 | mnaser | (sometimes we actually do host-evacuate-live only for the purpose of having instances go in the right place after we make scheduler changes) | |
| 18:10:22 | mriedem | if you are evacuating the hosts, you could disable the compute service | |
| 18:10:33 | mriedem | not evacuate like the evacuate API, i mean "get the vms off this host" | |
| 18:10:48 | mnaser | yeah when we're shutting things down that's what we go for, but when shuffling things around | |
| 18:11:29 | mnaser | you kinda just want instances to move into where they are supposed to go | |
| 18:11:36 | mnaser | anyways | |
| 18:11:40 | mnaser | very minor thing | |
| 18:11:45 | mriedem | "you kinda just want instances to move into where they are supposed to go" | |
| 18:11:46 | mriedem | ha | |
| 18:11:50 | mriedem | of course! | |
| 18:11:55 | mriedem | silly scheduler | |
| 18:12:17 | mriedem | well as a tc big wig, | |
| 18:12:23 | mriedem | you can influence the goal setting for T | |
| 18:12:37 | mnaser | usually after we make scheduling tweaks | |
| 18:12:41 | mnaser | we just do a rolling live migration | |
| 18:12:52 | mriedem | so watcher but without watcher | |
| 18:13:00 | mnaser | just a one time watcher | |
| 18:13:57 | mnaser | i think watcher is super interesting but i think it depends on too many things | |
| 18:14:57 | mriedem | well as a tc big wig, | |
| 18:15:13 | mriedem | you can influence adoption of a new top-level project: watcher-lite | |
| 18:15:39 | mriedem | watcher zero | |
| 18:15:46 | mriedem | all of the flavor, none of the guilt | |
| 18:17:15 | mnaser | lols | |
| 18:21:25 | melwitt | mnaser: Current runways: use-nested-allocation-candidates -- This channel is for Nova development. For support of Nova deployments, please use #openstack. | |
| 18:22:51 | mnaser | melwitt: voila i've made myself useful for today | |
| 18:23:14 | melwitt | \o/ | |
| 18:23:55 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: nova-manage - fix online_data_migrations counts https://review.openstack.org/605828 | |
| 18:26:10 | mriedem | imacdonn: +W on https://review.openstack.org/#/c/605329/ and i found an example of a migration that has total > 0 with completed == 0 | |
| 18:26:39 | imacdonn | mriedem: ack. I was just about to ask you if you intentionally didn't +W with your +2 :) | |
| 18:26:48 | mriedem | it was intentional, | |
| 18:27:01 | mriedem | because i was going to backport to stable and get a grenade run where i knew we actually had things to migrate | |
| 18:27:05 | mriedem | but i found one in stein too | |
| 18:27:10 | imacdonn | got it | |
| 18:27:13 | mriedem | http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536 | |
| 18:27:17 | mriedem | 2 rows matched query migrate_instances_add_request_spec, 0 migrated | |
| 18:27:30 | mriedem | ^ is with your change | |
| 18:27:41 | mriedem | | populate_queued_for_delete | 2 | 2 | | |
| 18:27:46 | mriedem | http://logs.openstack.org/88/605488/1/check/neutron-grenade/d64e316/logs/grenade.sh.txt.gz#_2018-09-27_01_15_47_182 is without | |
| 18:27:52 | mriedem | | populate_queued_for_delete | 0 | 0 | | |
| 18:27:56 | imacdonn | \o/ | |
| 18:28:16 | mriedem | https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/objects/request_spec.py#L707 is the example where total can be >0 but we don't actually migrate anything | |
| 18:29:04 | mriedem | so that with the explanation here https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/cmd/manage.py#L374 is confusing | |
| 18:29:20 | imacdonn | that may be a bug | |
| 18:29:31 | mriedem | it does say, "If found is nonzero and done is zero, some records are # not migratable, but all migrations that can complete have # finished." | |
| 18:29:48 | mriedem | "not migrateable" should really be, "don't require migration" | |
| 18:29:52 | imacdonn | per Dan's description, count_all should never be greater than max_count | |
| 18:30:15 | mriedem | a lot of the migrations return found==done because the query to find the $found number is filtering on things that need to be migrated | |
| 18:30:25 | mriedem | e.g. select * bdms where uuid is None; | |
| 18:30:33 | mriedem | found == done ^ | |
| 18:30:41 | mriedem | but that's not the same with the request spec migratoin | |
| 18:30:44 | mriedem | *migration | |
| 18:30:48 | mriedem | since we have to hit 2 different dbs | |
| 18:31:44 | imacdonn | personally I think the batch mechanism is a bit broken, at least as it's described in the comments | |
| 18:31:50 | mriedem | as i mentioned on your change, the 'Total Needed' column is misleading | |
| 18:31:50 | mriedem | http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536 | |
| 18:31:52 | imacdonn | but, as you said, we need Dan for that conversation | |
| 18:41:07 | mnaser | so has anyone ever thought what happens when we hit instance-ffffffff | |
| 18:43:10 | imacdonn | fsshhh ... that'll never happen | |
| 18:44:08 | melwitt | in case anyone is wondering about the failing ceph job, I'm trying out a fix here https://review.openstack.org/605833 | |
| 18:48:19 | mnaser | mordred: it's a bit of a difficult position but ideally figuring out what the best way to deal with cold/live migration and reworking it.. (openstack server migrate) | |
| 18:48:37 | mordred | yah. as you know, the sdk code for that is ... fun :) | |
| 18:48:47 | mnaser | mainly my issue was osc forces you to specify a host when its optional | |
| 18:48:47 | mordred | fwiw - mriedem is right - osc doesn't currently do version negotiation. once we start migrating it to sdk though, it'll pick up that ability | |
| 18:49:24 | mriedem | mnaser: easy: make --live just an option with no value, add --host (optional, takes a value), and add --cold | |
| 18:49:58 | mnaser | mriedem: but the not breaking scripts part i guess | |
| 18:50:02 | mriedem | or, let --live take a value for compat but proxy it to --host if --host isn't used | |
| 18:50:39 | mriedem | i'm not sure how you could specify --live w/o a host though if --live can take a host | |
| 18:50:41 | mriedem | gd CLIs | |
| 18:50:52 | mriedem | empty string? | |
| 18:50:58 | mriedem | pretty janky | |
| 18:51:25 | mriedem | openstack server migrate --live-but-with-no-host-seriously my_server | |
| 18:51:32 | mordred | mriedem: ++ | |
| 18:51:35 | mordred | that's totally the right answer | |
| 18:51:50 | mriedem | do i win something? | |
| 18:52:00 | mordred | you win this bucket of parts I just found | |
| 18:54:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836 | |
| 18:54:42 | mriedem | imacdonn: efried: ^ does this make life better? | |
| 18:55:54 | imacdonn | only slightly, IMO | |
| 18:56:27 | mriedem | :( | |
| 18:57:12 | imacdonn | if those two rows don't need migration, then they couldn't be included in something named "Total Needed"......... ? | |
| 18:57:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: nova-manage - fix online_data_migrations counts https://review.openstack.org/605839 | |
| 18:57:45 | mriedem | like i said, total needed is a bad title, | |
| 18:57:55 | imacdonn | if "Total Needed" means "Total Rows that exist that may or may not need it", we should see a lot less zeroes | |
| 18:57:56 | mriedem | i'm not sure if renaming that to Total Found breaks any kind of compat, | |
| 18:58:07 | mriedem | Total Candidates | |
| 18:58:09 | mriedem | something like that | |
| 18:58:28 | mriedem | no one should be parsing the output of this command for column headers and such anyway | |
| 18:58:46 | efried | hah | |
| 18:58:48 | imacdonn | I suspect that the existing migration methods may already interpret it inconsistently, but I haven't analysed it to confirm | |
| 18:59:42 | efried | yeah, to me, seeing needed=2/completed=0 feels like it should be an error | |
| 19:00:01 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: nova-manage - fix online_data_migrations counts https://review.openstack.org/605840 | |
| 19:01:17 | efried | I mean, this is better than it was before, because there's at least *some* attempt to explain wtf is going on. | |
| 19:01:41 | efried | Is this dansmith's bailiwick btw? Something he would want to review? | |
| 19:05:45 | mriedem | i assume he would yes | |
| 19:06:22 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: nova-manage - fix online_data_migrations counts https://review.openstack.org/605842 | |
| 19:11:13 | mordred | mriedem, cfriesen: I just left a suggestion on the osc live migration patch about a way to make --live take an optional argument | |