| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-09 | |||
| 20:13:24 | mriedem | gets dropped but won't we wait for something that doesn't come and timeout? | |
| 20:13:39 | dansmith | should timeout yeah | |
| 20:13:50 | mriedem | the network-vif-plugged is triggered via plug_vifs during pre_live_migration on the dest, | |
| 20:13:57 | mriedem | which happens before his code runs to register the waiter | |
| 20:14:00 | mriedem | so it's a total race window | |
| 20:14:14 | dansmith | ugh | |
| 20:14:21 | mriedem | which is why we added https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L675 | |
| 20:14:24 | mriedem | but not backportable | |
| 20:14:38 | dansmith | I thought it gets triggered by the actual guest starting on the other side, which came from the actual live migration op | |
| 20:14:52 | mriedem | i'd need sahid to confirm that | |
| 20:15:04 | mriedem | but network-vif-plugged, as far as i know, comes from plug_vifs on the dest during pre_live_migratoin | |
| 20:15:08 | mriedem | which is before his code runs | |
| 20:15:11 | mriedem | on the source | |
| 20:15:18 | dansmith | so, | |
| 20:15:30 | dansmith | the even comes from the tap being created actually | |
| 20:15:35 | dansmith | *event | |
| 20:16:16 | dansmith | so maybe plug is creating a tap before libvirt does but I'm not sure how we'd give it to it | |
| 20:16:44 | mriedem | prometheanfire: i'm assuming you have this https://review.openstack.org/#/c/586965/ | |
| 20:16:48 | mriedem | ^ fix for the pike backport | |
| 20:17:00 | prometheanfire | ya | |
| 20:17:07 | dansmith | I learned this after we were working on that patch though | |
| 20:17:10 | prometheanfire | that's within the sha I posted earlier | |
| 20:17:21 | prometheanfire | otherwise it wouldn't succeed at all :P | |
| 20:17:23 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: use simple code paths when possible https://review.openstack.org/590388 | |
| 20:17:24 | openstackgerrit | Jay Pipes proposed openstack/nova master: split gigantor SQL placement query into multiple https://review.openstack.org/590041 | |
| 20:17:26 | mriedem | prometheanfire: yeah | |
| 20:17:29 | mriedem | very obvious explosion | |
| 20:17:35 | openstackgerrit | Jay Pipes proposed openstack/nova master: Adds a test for _get_provider_ids_matching() https://review.openstack.org/590150 | |
| 20:18:10 | mriedem | prometheanfire: you have this? https://review.openstack.org/#/c/510013/ | |
| 20:18:11 | sean-k-mooney | mriedem: netwrokg-vif-plugged comre from neutron when it finishes wiering up the port | |
| 20:18:29 | mriedem | prometheanfire: this one was fun in that it depended on neutron backports as well | |
| 20:18:39 | sean-k-mooney | also i just as pluging stuff so need to scoll back to get context | |
| 20:18:59 | prometheanfire | that one I'm not sure, but probably | |
| 20:19:09 | mriedem | might want to check | |
| 20:19:23 | prometheanfire | checking | |
| 20:20:17 | prometheanfire | merged dec 3 into stable pike https://github.com/openstack/neutron/commits/stable/pike?after=ad8f00236cc57ce9a8f077dd2d32c6fada00e817+139 | |
| 20:22:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/587498 | |
| 20:22:48 | dansmith | if the event comes from stuff we're doing in pre_dest, it seems unlikely we'd ever win the race in gate | |
| 20:23:16 | prometheanfire | using at least this version of neutron https://github.com/openstack/openstack-ansible/blob/5c341a7bada78edab5f3d132d55adb00eaf2413f/playbooks/defaults/repo_packages/openstack_services.yml#L125 | |
| 20:23:26 | prometheanfire | which is from 2018 in may | |
| 20:23:55 | mriedem | idk this is where i thought we'd generate the event https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7775 | |
| 20:24:24 | prometheanfire | ok, I have to go for a bit, but will be back | |
| 20:24:34 | dansmith | well, | |
| 20:24:45 | dansmith | I did some debugging on plug stuff with the godaddy people a month or so ago, | |
| 20:24:51 | dansmith | and read through all the neutron code related to this | |
| 20:24:59 | dansmith | and I was surprised to learn that actually what happens is, | |
| 20:25:06 | dansmith | something creates an interface with the right name, | |
| 20:25:17 | dansmith | a periodic in the neutron agent notices, hooks it up and sends the event | |
| 20:25:30 | dansmith | so it's a little less connected to us than I would have thought | |
| 20:26:04 | mriedem | i knew the linuxbridge-agent does polling only b/c sean mooney explained that when we had the issue with waiting for hard reboot events for linuxbridge in the gate | |
| 20:26:20 | mriedem | ovs agent listens for an actual event from ovs itself | |
| 20:26:46 | dansmith | maybe that's why we can win it in the gate, since there was a polling loop | |
| 20:27:04 | mriedem | we use ovs in the gate | |
| 20:27:07 | mriedem | for most everything | |
| 20:27:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/587498 | |
| 20:27:13 | dansmith | I thought we had a LB job? | |
| 20:27:14 | mriedem | neutron has a couple of lb jobs i think | |
| 20:27:32 | dansmith | regardless, you did a LB migration job and it passed a few times at least | |
| 20:27:34 | mriedem | i had rigged up a patch that ran linuxbridge multinode to test sahid's patch | |
| 20:27:41 | mriedem | yeah, could have won the race though right? | |
| 20:27:43 | Sundar | efried: Please ping me when you have a moment | |
| 20:28:07 | efried | Hi Sundar, what's up? | |
| 20:28:16 | dansmith | I guess because of the polling loop we have a decent chance | |
| 20:28:32 | mriedem | and all of our controller services on a single node slowing things down maybe, idk | |
| 20:28:38 | dansmith | because pre-dest is pretty long before we get to the event wait, which seems ...crazy to ever win it | |
| 20:28:38 | mriedem | don't know what prometheanfire's env setup is like | |
| 20:28:59 | dansmith | if so, presumably this patch broke live migration for anyone with a fast system | |
| 20:29:17 | mriedem | god i hope so | |
| 20:29:19 | mriedem | that would be magical | |
| 20:29:32 | dansmith | so, sahid has been arguing to make the timeout non-fatal, reportedly because someone was using a custom driver or something | |
| 20:29:39 | dansmith | but I wonder if it's actually because this is actually totes broken | |
| 20:29:54 | Sundar | We were discussing the relative roles of plugins and drivers. It seems to me that that the distinction is not a hard one. We just need some extension agent with clear APIs for both os-acc and Cyborg. It could be the same module providing two sets of APIs. Does that work for you? | |
| 20:30:20 | efried | Sundar: Absolutely. | |
| 20:30:33 | dansmith | mriedem: melwitt: tbh, if we really think this is that broken (and sounds like it is) then we should probably revert it from the release immediately | |
| 20:31:06 | mriedem | well the other thing i was saying above was, | |
| 20:31:16 | melwitt | oof | |
| 20:31:16 | Sundar | Great. I was trying to delineate the two and it was getting rather ambiguous. Good to have this sorted out. Will send out the os-acc spec with this update | |
| 20:31:20 | mriedem | even if we get the TimeoutError, we raise MigrationError or whatever, | |
| 20:31:32 | mriedem | but i *think* by that point we've already started the guest transfer | |
| 20:31:43 | mriedem | b/c we call _live_migration_operation | |
| 20:31:43 | efried | Sundar: Well, we should still delineate the two, if they're going to be *able* to be separate modules. | |
| 20:31:44 | dansmith | correct | |
| 20:31:44 | mriedem | the nwait | |
| 20:32:07 | mriedem | so starting the guest transfer and then timing out on the watier b/c we registered late kills the nova live migration option but not what's in the hypervisor | |
| 20:32:09 | mriedem | right? | |
| 20:32:18 | mriedem | s/option/operation/ | |
| 20:32:29 | dansmith | mriedem: um, what? | |
| 20:32:52 | mriedem | if we get the timeout and raise MigrationError | |
| 20:32:56 | mriedem | but have already started the guest transfer | |
| 20:33:07 | mriedem | the only thing that happens in nova is we call the _rollback_live_migration code, | |
| 20:33:16 | mriedem | we don't attempt to kill any live migration job that's running in libvirt | |
| 20:33:17 | mriedem | right? | |
| 20:33:17 | Sundar | efried: There will certainly be two sets of APIs: one for instance half of the attach and one for the device half. But both can do device-specific, platform-specific and vendor-specific actions. | |
| 20:33:31 | efried | Sundar: I dig it. | |
| 20:33:49 | efried | Sundar: And I like the idea of being able to supply that code in one module or two. | |
| 20:33:54 | dansmith | lemme look | |
| 20:34:00 | mriedem | in other words, | |
| 20:34:10 | mriedem | nova will say "migratoin failed" but the guest might actually get transferred | |
| 20:34:17 | mriedem | just really f'ing slowly | |
| 20:34:52 | dansmith | well, yeah, I mean, the point of this code was to not raise the speed limit until it came | |