| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-09 | |||
| 20:08:19 | mriedem | but that should mean we wait until we do "opthread.link(thread_finished, finish_event)" | |
| 20:08:53 | mriedem | even if we get the timeout, it seems this is pretty dangerous if we've already started the live migration in the hypervisor | |
| 20:09:02 | mriedem | i +2ed this code too... | |
| 20:09:12 | prometheanfire | so we can blame you :P | |
| 20:09:15 | mriedem | kinda need dansmith here | |
| 20:09:33 | prometheanfire | ya | |
| 20:09:36 | mriedem | i still don't know why you wouldn't see the timeout message | |
| 20:09:49 | prometheanfire | same | |
| 20:09:59 | mriedem | 2018-08-09 21:27:20.197 12111 DEBUG nova.virt.libvirt.driver [req-84ca4a17-0d3d-4597-91d6-f5721989dd41 143ee57edd4d4e3b9a165d375d0e7e1a a727713d2c0a4ed69b730d9cb2116af6 - default default] [instance: c37d7489-a67b-47ea-a4f7-9323804cc552] Migration operation thread notification thread_finished /openstack/venvs/nova-r16.2.2/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6952 | |
| 20:09:59 | mriedem | from your log | |
| 20:11:28 | mriedem | sahid will be around in the morning if you can catch him | |
| 20:11:33 | dansmith | catch me up? | |
| 20:11:47 | mriedem | dansmith: prometheanfire is running a linuxbridge live migration in pike, | |
| 20:11:54 | mriedem | with that bw slow down patch of sahid's | |
| 20:12:05 | mriedem | the live migration takes longer than our default vif plugging timeout | |
| 20:12:10 | dansmith | that got backported I assume? | |
| 20:12:25 | mriedem | yeah (from us). and looks like we actually get the event before sahid's code registers to wait, | |
| 20:12:35 | mriedem | but the weird thing is we don't get the timeout event after 5 minutes | |
| 20:12:45 | mriedem | https://gist.githubusercontent.com/mheler/475d21b741aa58f320a456c3ac0d0f45/raw/ff76c45c3b6968b0d0514a9a7dcf478f054f6b70/gistfile1.txt | |
| 20:12:55 | mriedem | instance is c37d7489-a67b-47ea-a4f7-9323804cc552 | |
| 20:12:56 | dansmith | if it comes before we register it gets dropped, | |
| 20:12:59 | prometheanfire | that's the sender | |
| 20:13:09 | dansmith | but obviously the point is it's supposed to come after the register as you said | |
| 20:13:24 | mriedem | gets dropped but won't we wait for something that doesn't come and timeout? | |
| 20:13:39 | dansmith | should timeout yeah | |
| 20:13:50 | mriedem | the network-vif-plugged is triggered via plug_vifs during pre_live_migration on the dest, | |
| 20:13:57 | mriedem | which happens before his code runs to register the waiter | |
| 20:14:00 | mriedem | so it's a total race window | |
| 20:14:14 | dansmith | ugh | |
| 20:14:21 | mriedem | which is why we added https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L675 | |
| 20:14:24 | mriedem | but not backportable | |
| 20:14:38 | dansmith | I thought it gets triggered by the actual guest starting on the other side, which came from the actual live migration op | |
| 20:14:52 | mriedem | i'd need sahid to confirm that | |
| 20:15:04 | mriedem | but network-vif-plugged, as far as i know, comes from plug_vifs on the dest during pre_live_migratoin | |
| 20:15:08 | mriedem | which is before his code runs | |
| 20:15:11 | mriedem | on the source | |
| 20:15:18 | dansmith | so, | |
| 20:15:30 | dansmith | the even comes from the tap being created actually | |
| 20:15:35 | dansmith | *event | |
| 20:16:16 | dansmith | so maybe plug is creating a tap before libvirt does but I'm not sure how we'd give it to it | |
| 20:16:44 | mriedem | prometheanfire: i'm assuming you have this https://review.openstack.org/#/c/586965/ | |
| 20:16:48 | mriedem | ^ fix for the pike backport | |
| 20:17:00 | prometheanfire | ya | |
| 20:17:07 | dansmith | I learned this after we were working on that patch though | |
| 20:17:10 | prometheanfire | that's within the sha I posted earlier | |
| 20:17:21 | prometheanfire | otherwise it wouldn't succeed at all :P | |
| 20:17:23 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: use simple code paths when possible https://review.openstack.org/590388 | |
| 20:17:24 | openstackgerrit | Jay Pipes proposed openstack/nova master: split gigantor SQL placement query into multiple https://review.openstack.org/590041 | |
| 20:17:26 | mriedem | prometheanfire: yeah | |
| 20:17:29 | mriedem | very obvious explosion | |
| 20:17:35 | openstackgerrit | Jay Pipes proposed openstack/nova master: Adds a test for _get_provider_ids_matching() https://review.openstack.org/590150 | |
| 20:18:10 | mriedem | prometheanfire: you have this? https://review.openstack.org/#/c/510013/ | |
| 20:18:11 | sean-k-mooney | mriedem: netwrokg-vif-plugged comre from neutron when it finishes wiering up the port | |
| 20:18:29 | mriedem | prometheanfire: this one was fun in that it depended on neutron backports as well | |
| 20:18:39 | sean-k-mooney | also i just as pluging stuff so need to scoll back to get context | |
| 20:18:59 | prometheanfire | that one I'm not sure, but probably | |
| 20:19:09 | mriedem | might want to check | |
| 20:19:23 | prometheanfire | checking | |
| 20:20:17 | prometheanfire | merged dec 3 into stable pike https://github.com/openstack/neutron/commits/stable/pike?after=ad8f00236cc57ce9a8f077dd2d32c6fada00e817+139 | |
| 20:22:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/587498 | |
| 20:22:48 | dansmith | if the event comes from stuff we're doing in pre_dest, it seems unlikely we'd ever win the race in gate | |
| 20:23:16 | prometheanfire | using at least this version of neutron https://github.com/openstack/openstack-ansible/blob/5c341a7bada78edab5f3d132d55adb00eaf2413f/playbooks/defaults/repo_packages/openstack_services.yml#L125 | |
| 20:23:26 | prometheanfire | which is from 2018 in may | |
| 20:23:55 | mriedem | idk this is where i thought we'd generate the event https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7775 | |
| 20:24:24 | prometheanfire | ok, I have to go for a bit, but will be back | |
| 20:24:34 | dansmith | well, | |
| 20:24:45 | dansmith | I did some debugging on plug stuff with the godaddy people a month or so ago, | |
| 20:24:51 | dansmith | and read through all the neutron code related to this | |
| 20:24:59 | dansmith | and I was surprised to learn that actually what happens is, | |
| 20:25:06 | dansmith | something creates an interface with the right name, | |
| 20:25:17 | dansmith | a periodic in the neutron agent notices, hooks it up and sends the event | |
| 20:25:30 | dansmith | so it's a little less connected to us than I would have thought | |
| 20:26:04 | mriedem | i knew the linuxbridge-agent does polling only b/c sean mooney explained that when we had the issue with waiting for hard reboot events for linuxbridge in the gate | |
| 20:26:20 | mriedem | ovs agent listens for an actual event from ovs itself | |
| 20:26:46 | dansmith | maybe that's why we can win it in the gate, since there was a polling loop | |
| 20:27:04 | mriedem | we use ovs in the gate | |
| 20:27:07 | mriedem | for most everything | |
| 20:27:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/587498 | |
| 20:27:13 | dansmith | I thought we had a LB job? | |
| 20:27:14 | mriedem | neutron has a couple of lb jobs i think | |
| 20:27:32 | dansmith | regardless, you did a LB migration job and it passed a few times at least | |
| 20:27:34 | mriedem | i had rigged up a patch that ran linuxbridge multinode to test sahid's patch | |
| 20:27:41 | mriedem | yeah, could have won the race though right? | |
| 20:27:43 | Sundar | efried: Please ping me when you have a moment | |
| 20:28:07 | efried | Hi Sundar, what's up? | |
| 20:28:16 | dansmith | I guess because of the polling loop we have a decent chance | |
| 20:28:32 | mriedem | and all of our controller services on a single node slowing things down maybe, idk | |
| 20:28:38 | mriedem | don't know what prometheanfire's env setup is like | |
| 20:28:38 | dansmith | because pre-dest is pretty long before we get to the event wait, which seems ...crazy to ever win it | |
| 20:28:59 | dansmith | if so, presumably this patch broke live migration for anyone with a fast system | |
| 20:29:17 | mriedem | god i hope so | |
| 20:29:19 | mriedem | that would be magical | |
| 20:29:32 | dansmith | so, sahid has been arguing to make the timeout non-fatal, reportedly because someone was using a custom driver or something | |
| 20:29:39 | dansmith | but I wonder if it's actually because this is actually totes broken | |
| 20:29:54 | Sundar | We were discussing the relative roles of plugins and drivers. It seems to me that that the distinction is not a hard one. We just need some extension agent with clear APIs for both os-acc and Cyborg. It could be the same module providing two sets of APIs. Does that work for you? | |
| 20:30:20 | efried | Sundar: Absolutely. | |
| 20:30:33 | dansmith | mriedem: melwitt: tbh, if we really think this is that broken (and sounds like it is) then we should probably revert it from the release immediately | |
| 20:31:06 | mriedem | well the other thing i was saying above was, | |
| 20:31:16 | Sundar | Great. I was trying to delineate the two and it was getting rather ambiguous. Good to have this sorted out. Will send out the os-acc spec with this update | |
| 20:31:16 | melwitt | oof | |