| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-09 | |||
| 20:44:52 | mriedem | the only backend i know that won't work, | |
| 20:44:54 | mriedem | is ODL | |
| 20:45:10 | dansmith | in fact | |
| 20:45:10 | mriedem | because that doesn't send events on vif plug/unplug, only port host binding changes | |
| 20:45:13 | sean-k-mooney | mriedem: lb polls for new interfaces and can miss the addtion and removal of interfaces in some cases | |
| 20:45:25 | dansmith | mriedem: you're not only waiting for ovs interfaces there right? | |
| 20:45:29 | sean-k-mooney | so we can rely on lb to emit the event | |
| 20:45:31 | mriedem | dansmith: correct | |
| 20:45:46 | sean-k-mooney | or rathar the lb l2 agent | |
| 20:45:49 | dansmith | mriedem: so you should be eating them up there, and then his wait is definitely never going to get them right? | |
| 20:46:10 | dansmith | so in pike, I expect it races, | |
| 20:46:16 | dansmith | and in rocky it never ever works at all | |
| 20:47:03 | mriedem | yeah maybe | |
| 20:47:10 | mriedem | i could dig up my 2 node lb ci patch, | |
| 20:47:12 | dansmith | oh sweet baby jesus thank you for this day | |
| 20:47:17 | mriedem | and enable this waiter in nova on master, | |
| 20:47:40 | mriedem | and we'd have to probably turn the vif plugging timeout way down to actually see if we hit a timeout | |
| 20:47:53 | mriedem | otherwise i'd expect in the gate, live migration with a tiny cirros guest not doing anything transfers pretty fast | |
| 20:48:03 | dansmith | yeah | |
| 20:48:17 | mriedem | heh, and i was just going to start mowing and packing | |
| 20:48:37 | dansmith | melwitt: so, honestly, reverting sahid's thing for rocky needs to be high prio I think | |
| 20:48:47 | dansmith | melwitt: live migration with LB is completely broken I expect | |
| 20:48:55 | mriedem | i'll update that ci patch | |
| 20:49:02 | melwitt | ok, so is this a RC1 thing or a RC2 thing? | |
| 20:49:06 | dansmith | melwitt: and it probably needs to be reverted out of the older releases too | |
| 20:49:13 | dansmith | melwitt: your call but RCsomething, IMHO | |
| 20:49:36 | prometheanfire | back | |
| 20:49:41 | prometheanfire | but leaving soonish | |
| 20:49:42 | dansmith | I would think we could do something like what mriedem did in rocky for those releases | |
| 20:50:05 | melwitt | ok, definitely RC2. trying to do it by the end of today will be hard unless it gets approved, stat | |
| 20:50:20 | melwitt | *definitely RC2, at least | |
| 20:50:54 | dansmith | well, maybe not since he needed a signal from the remote machine that it was going to do the wait ... | |
| 20:51:02 | prometheanfire | mriedem: I'll try to get a bug reported if you think that's the next step | |
| 20:51:04 | dansmith | although the event should still trigger | |
| 20:51:07 | mriedem | prometheanfire: yes please | |
| 20:51:09 | dansmith | prometheanfire: yes | |
| 20:51:10 | mriedem | we'll track that for rc2 | |
| 20:51:27 | prometheanfire | rc2? | |
| 20:51:29 | mriedem | was going to try and see how long a guest transfer takes in the gate | |
| 20:51:37 | mriedem | prometheanfire: today is release candidate 1 day | |
| 20:51:43 | prometheanfire | I thought this didn't hit rocky | |
| 20:51:50 | prometheanfire | but I'll leave that to you | |
| 20:51:53 | mriedem | prometheanfire: new wrinkle | |
| 20:51:57 | prometheanfire | oh, nice | |
| 20:52:00 | dansmith | worse wrinkle | |
| 20:52:02 | mriedem | (1) probably race fail on pike | |
| 20:52:05 | prometheanfire | happy to help :P | |
| 20:52:06 | mriedem | (2) totes broken on master | |
| 20:52:13 | prometheanfire | even better | |
| 20:53:40 | dansmith | prometheanfire: hook me up with a bug number and I'll propose the revert | |
| 20:53:41 | dansmith | and I can comment on the bug with all the deets | |
| 20:53:50 | dansmith | since mriedem will be busy with the ci patch and packing for da nang | |
| 20:53:59 | mriedem | isn't da nang vietnam? | |
| 20:54:02 | dansmith | yes | |
| 20:54:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 20:54:33 | prometheanfire | I'm hoping to get the user to report, but I will if he went home | |
| 20:55:16 | dansmith | the revert is complete conflict | |
| 20:55:19 | dansmith | wonderful. | |
| 20:56:33 | dansmith | the other benefit of doing this in the manager is that we don't need the silly artificial speed limit | |
| 20:56:48 | dansmith | although we probably need sahid and libvirt people to confirm that there's not something we're missing here | |
| 20:57:32 | dansmith | because I asked him specifically about doing this early on in his patch and he said it wasn't possible, but I believed that qemu/libvirt on the dest machine were responsible for the plugging at the time | |
| 20:57:34 | dansmith | and maybe he did too | |
| 21:00:46 | mriedem | meanwhile, our granite seller is being an ass and i have to get back to our plumber | |
| 21:01:37 | dansmith | #firstworldrichpersonproblems | |
| 21:06:47 | mriedem | ok so looking at a job, we register waiting for events starting here http://logs.openstack.org/98/587498/1/check/nova-live-migration/5ff805a/logs/screen-n-cpu.txt.gz#_Jul_31_17_18_19_562999 | |
| 21:06:52 | mriedem | Jul 31 17:18:19.562999 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: DEBUG nova.compute.manager [None req-942f438c-3cbb-4ce7-8afb-ecd250c98f75 tempest-LiveAutoBlockMigrationV225Test-1515676049 tempest-LiveAutoBlockMigrationV225Test-1515676049] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] Preparing to wait for external event network-vif-plugged-6b030652-5fe6-471a-b7ec-0b70e95159a4 {{(pid=2320) prepare_for_instanc | |
| 21:06:52 | mriedem | ent /opt/stack/new/nova/nova/compute/manager.py:328}} | |
| 21:07:51 | mriedem | pre_live_migration takes about 7 seconds | |
| 21:07:51 | mriedem | Jul 31 17:18:26.178239 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: INFO nova.compute.manager [None req-994d9893-545f-47b9-b93e-1e21cb439db7 tempest-LiveMigrationTest-233418614 tempest-LiveMigrationTest-233418614] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] Took 6.61 seconds for pre_live_migration on destination host ubuntu-xenial-inap-mtl01-0001077053. | |
| 21:09:09 | mriedem | Jul 31 17:18:26.237174 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: DEBUG nova.virt.libvirt.driver [None req-994d9893-545f-47b9-b93e-1e21cb439db7 tempest-LiveMigrationTest-233418614 tempest-LiveMigrationTest-233418614] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] Starting monitoring of live migration {{(pid=2320) _live_migration /opt/stack/new/nova/nova/virt/libvirt/driver.py:7555}} | |
| 21:09:15 | mriedem | start monitoring the live migration there ^ | |
| 21:09:59 | mriedem | live migration complete: | |
| 21:09:59 | mriedem | Jul 31 17:18:27.486275 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: INFO nova.compute.manager [None req-14ccce2e-8610-47b7-aba2-77f6fc468b61 tempest-LiveMigrationRemoteConsolesV26Test-255629851 tempest-LiveMigrationRemoteConsolesV26Test-255629851] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] VM Migration completed (Lifecycle Event) | |
| 21:10:12 | mriedem | heh 1 second? | |
| 21:10:52 | dansmith | makes sense.. I doubt a cirros guest has more than a hundred meg of dirty ram | |
| 21:11:02 | dansmith | which is 1 second at gigE | |
| 21:11:13 | mriedem | i never see "VIF events received, continuing migration" | |
| 21:11:32 | dansmith | it's not LB | |
| 21:11:35 | dansmith | right? | |
| 21:11:39 | mriedem | oh right duh | |
| 21:11:41 | prometheanfire | you'll probably want to retitle the bug https://bugs.launchpad.net/nova/+bug/1786346 | |
| 21:11:41 | openstack | Launchpad bug 1786346 in OpenStack Compute (nova) "live migrations slow" [Undecided,New] | |
| 21:11:44 | prometheanfire | mriedem: dansmith ^ | |
| 21:11:49 | dansmith | prometheanfire: thanks | |
| 21:11:59 | prometheanfire | if you can let me know when you update the bug with details I'd appreciate it | |
| 21:12:20 | dansmith | I'm trying to get the revert to even pass tests and then I will | |
| 21:14:03 | prometheanfire | thanks | |
| 21:18:25 | mriedem | dansmith: ok so https://review.openstack.org/553608 should do the wait in compute now | |
| 21:19:09 | dansmith | mriedem: cool, updating the bug now and working on the revert in parallel, so we can make that depend on the revert to be sure we don't get the timeout message at least right? | |
| 21:19:39 | mriedem | well, that's why i was looking at timings, | |
| 21:20:02 | mriedem | because this means we'll consume the network-vif-plugged event from pre_live_migration before we call driver.live_migration which does the bw stuff, | |
| 21:20:10 | mriedem | the event won't come for that waiter, | |
| 21:20:27 | mriedem | but the guest transfer is so fast, won't we just finish the operatoin before we ever had a chance to timeout? | |
| 21:20:41 | mriedem | like, do i need a patch that puts a fake sleep in the driver's live migratoin metohd? | |
| 21:20:42 | mriedem | *method | |
| 21:21:15 | dansmith | even at 1MB/s? | |
| 21:21:19 | mriedem | could set the vif_plugging_timeout to like 1 minute, and add a 30 second sleep in the driver | |
| 21:21:28 | mriedem | well, | |
| 21:21:39 | dansmith | should go slower there, but I guess it won't take long enough | |
| 21:21:41 | mriedem | maybe not, but the test will timeout before the 5 minute vif_plugging_timeout i think | |