| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-09 | |||
| 21:01:37 | dansmith | #firstworldrichpersonproblems | |
| 21:06:47 | mriedem | ok so looking at a job, we register waiting for events starting here http://logs.openstack.org/98/587498/1/check/nova-live-migration/5ff805a/logs/screen-n-cpu.txt.gz#_Jul_31_17_18_19_562999 | |
| 21:06:52 | mriedem | Jul 31 17:18:19.562999 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: DEBUG nova.compute.manager [None req-942f438c-3cbb-4ce7-8afb-ecd250c98f75 tempest-LiveAutoBlockMigrationV225Test-1515676049 tempest-LiveAutoBlockMigrationV225Test-1515676049] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] Preparing to wait for external event network-vif-plugged-6b030652-5fe6-471a-b7ec-0b70e95159a4 {{(pid=2320) prepare_for_instanc | |
| 21:06:52 | mriedem | ent /opt/stack/new/nova/nova/compute/manager.py:328}} | |
| 21:07:51 | mriedem | pre_live_migration takes about 7 seconds | |
| 21:07:51 | mriedem | Jul 31 17:18:26.178239 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: INFO nova.compute.manager [None req-994d9893-545f-47b9-b93e-1e21cb439db7 tempest-LiveMigrationTest-233418614 tempest-LiveMigrationTest-233418614] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] Took 6.61 seconds for pre_live_migration on destination host ubuntu-xenial-inap-mtl01-0001077053. | |
| 21:09:09 | mriedem | Jul 31 17:18:26.237174 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: DEBUG nova.virt.libvirt.driver [None req-994d9893-545f-47b9-b93e-1e21cb439db7 tempest-LiveMigrationTest-233418614 tempest-LiveMigrationTest-233418614] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] Starting monitoring of live migration {{(pid=2320) _live_migration /opt/stack/new/nova/nova/virt/libvirt/driver.py:7555}} | |
| 21:09:15 | mriedem | start monitoring the live migration there ^ | |
| 21:09:59 | mriedem | live migration complete: | |
| 21:09:59 | mriedem | Jul 31 17:18:27.486275 ubuntu-xenial-inap-mtl01-0001077052 nova-compute[2320]: INFO nova.compute.manager [None req-14ccce2e-8610-47b7-aba2-77f6fc468b61 tempest-LiveMigrationRemoteConsolesV26Test-255629851 tempest-LiveMigrationRemoteConsolesV26Test-255629851] [instance: 7f68c430-f565-433a-8f87-27b9a00d29a0] VM Migration completed (Lifecycle Event) | |
| 21:10:12 | mriedem | heh 1 second? | |
| 21:10:52 | dansmith | makes sense.. I doubt a cirros guest has more than a hundred meg of dirty ram | |
| 21:11:02 | dansmith | which is 1 second at gigE | |
| 21:11:13 | mriedem | i never see "VIF events received, continuing migration" | |
| 21:11:32 | dansmith | it's not LB | |
| 21:11:35 | dansmith | right? | |
| 21:11:39 | mriedem | oh right duh | |
| 21:11:41 | prometheanfire | you'll probably want to retitle the bug https://bugs.launchpad.net/nova/+bug/1786346 | |
| 21:11:41 | openstack | Launchpad bug 1786346 in OpenStack Compute (nova) "live migrations slow" [Undecided,New] | |
| 21:11:44 | prometheanfire | mriedem: dansmith ^ | |
| 21:11:49 | dansmith | prometheanfire: thanks | |
| 21:11:59 | prometheanfire | if you can let me know when you update the bug with details I'd appreciate it | |
| 21:12:20 | dansmith | I'm trying to get the revert to even pass tests and then I will | |
| 21:14:03 | prometheanfire | thanks | |
| 21:18:25 | mriedem | dansmith: ok so https://review.openstack.org/553608 should do the wait in compute now | |
| 21:19:09 | dansmith | mriedem: cool, updating the bug now and working on the revert in parallel, so we can make that depend on the revert to be sure we don't get the timeout message at least right? | |
| 21:19:39 | mriedem | well, that's why i was looking at timings, | |
| 21:20:02 | mriedem | because this means we'll consume the network-vif-plugged event from pre_live_migration before we call driver.live_migration which does the bw stuff, | |
| 21:20:10 | mriedem | the event won't come for that waiter, | |
| 21:20:27 | mriedem | but the guest transfer is so fast, won't we just finish the operatoin before we ever had a chance to timeout? | |
| 21:20:41 | mriedem | like, do i need a patch that puts a fake sleep in the driver's live migratoin metohd? | |
| 21:20:42 | mriedem | *method | |
| 21:21:15 | dansmith | even at 1MB/s? | |
| 21:21:19 | mriedem | could set the vif_plugging_timeout to like 1 minute, and add a 30 second sleep in the driver | |
| 21:21:28 | mriedem | well, | |
| 21:21:39 | dansmith | should go slower there, but I guess it won't take long enough | |
| 21:21:41 | mriedem | maybe not, but the test will timeout before the 5 minute vif_plugging_timeout i think | |
| 21:21:56 | dansmith | yeah okay so we'll have to force it down I guess | |
| 21:22:00 | mriedem | just wondering if i should set the vif_plugging_timeout to like 1 minute | |
| 21:23:15 | dansmith | 30sec but yeah | |
| 21:23:44 | mriedem | ok updated; hopefully my zuul fu is strong enough | |
| 21:23:44 | jaypipes | melwitt, dansmith: regarding https://review.openstack.org/#/c/540258, even if we fix the scheduler/top-level issues around server group affinity and multiple cells, that's still not going to fix the eleventh-hour on-the-compute-node checks that currently run just for affinity groups, though, right? I mean, the computes can't talk cross-cell anyway so there would be no way for those on-compute-node checks to run... | |
| 21:24:58 | mriedem | jaypipes: yes https://review.openstack.org/#/c/540258/8/nova/scheduler/utils.py@738 | |
| 21:25:11 | mriedem | "Also note that we could be racing if we have multiple server create requests for the same affinity group and the scheduler decides to put them each in different cells - the late affinity check in the compute won't resolve that because the upcall check is targeted to the cell the compute is in, and won't see any other hosts for other members in other cells. Separate bug though..." | |
| 21:26:42 | jaypipes | mriedem: ack, ok, just wanted to verify I wasn't crazypants. | |
| 21:26:58 | mriedem | good to have more than just me thinking that | |
| 21:27:03 | mriedem | *that it's an issue.. | |
| 21:27:13 | mriedem | not that you're (not?) crazy | |
| 21:27:50 | dansmith | prometheanfire: commented on the bug | |
| 21:28:18 | melwitt | yeah, it makes sense. I was re-thinking about the upcalls described in https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls and it's true that they can only work if you're single cell | |
| 21:28:49 | melwitt | once you're multi-cell/split-MQ I think none of them can work | |
| 21:29:09 | smcginnis | melwitt: Howdy. How are things coming along for the RC? | |
| 21:30:11 | mriedem | ha | |
| 21:30:31 | dansmith | smcginnis: awesome | |
| 21:30:34 | dansmith | they're going awesome | |
| 21:30:35 | dansmith | thanks for asking | |
| 21:30:40 | melwitt | haha .... | |
| 21:31:00 | smcginnis | And I know no one here would ever be sarcastic so... that's great! | |
| 21:31:00 | smcginnis | :P | |
| 21:31:42 | melwitt | smcginnis: we stumbled upon something that we'll need to fix for rc2. but as for rc1, I'm waiting on the RPC version alias patch to land, then will propose the release for rc1 | |
| 21:32:23 | smcginnis | melwitt: Cool, sounds good. I know a few others already know they will need to get an RC2, so that's no big deal. Thanks. | |
| 21:33:20 | mriedem | dansmith: so one comment on the bug, | |
| 21:33:28 | mriedem | by default the compute manager won't wait for the event | |
| 21:33:35 | mriedem | the config is false for backward compat | |
| 21:33:39 | melwitt | mriedem: I guess, since we're having a rc2, should I just leave the rpc version alias for then? or get it for rc1? | |
| 21:33:55 | melwitt | smcginnis: we're not alone... :) | |
| 21:34:02 | dansmith | mriedem: oh? I didn't see a config valve | |
| 21:34:10 | mriedem | yes, live_migration_wait_for_vif_plug | |
| 21:34:24 | mriedem | b/c not all network backends send the event for just vif plugging (ODL) | |
| 21:34:31 | smcginnis | melwitt: We can wait a bit. Unless you think it will be several hours yet. | |
| 21:35:02 | mriedem | melwitt: probably fine either way | |
| 21:35:31 | dansmith | mriedem: ah, so we handled that in sahid's patch by looking for the vif_Type | |
| 21:35:46 | melwitt | mriedem: thx. probably will just go ahead now and let that be in rc2 | |
| 21:35:46 | dansmith | making people know to opt into proper behavior kidna sucks | |
| 21:36:02 | melwitt | the version alias was recheck at 13:30 so it's got awhile before it will have a chance to merge | |
| 21:36:03 | mriedem | true, | |
| 21:36:09 | mriedem | but ODL shows up as ovs vif_type | |
| 21:36:13 | dansmith | oh | |
| 21:36:16 | dansmith | well, that sucks | |
| 21:36:17 | mriedem | which means we have no idea how to not wait for ODL | |
| 21:36:24 | mriedem | remember the hard reboot wait fiasco? | |
| 21:36:28 | dansmith | well, I'd say let them opt out | |
| 21:36:31 | dansmith | personallt | |
| 21:36:34 | dansmith | but whatever | |
| 21:36:42 | mriedem | that's why i plan on making the option True by default in stein | |
| 21:37:06 | melwitt | smcginnis: the version alias patch was rechecked an hour ago, so it'll be awhile to merge, if the gate doesn't fail on us again. but since we're having a rc2, that patch will be fine to go into rc2, so I can just cut rc1 now | |
| 21:37:09 | dansmith | oh I see, I just read your comment | |
| 21:37:11 | dansmith | okay | |
| 21:37:12 | mriedem | i might have also defaulted to False when i originally thought we'd backport this | |
| 21:37:40 | smcginnis | melwitt: Up to you. I'm fine waiting awhile. | |
| 21:37:51 | mriedem | smcginnis doesn't have to go to china tomorrow | |
| 21:37:56 | melwitt | heh | |
| 21:38:23 | smcginnis | :) | |
| 21:40:02 | openstackgerrit | Dan Smith proposed openstack/nova master: Revert "libvirt: slow live-migration to ensure network is ready" https://review.openstack.org/590538 | |
| 21:40:09 | dansmith | that was a super nasty revert, fyi | |
| 21:40:15 | dansmith | so look at it with critical eyes | |
| 21:40:33 | melwitt | mriedem, get out soul crusher #3 | |
| 21:45:01 | melwitt | smcginnis: am I to include cycle-highlights in the patch? I see that was done for queens | |
| 21:45:08 | melwitt | *in the release patch | |
| 21:46:06 | prometheanfire | dansmith: thanks | |