Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-09
20:14:14 dansmith ugh
20:14:21 mriedem which is why we added https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L675
20:14:24 mriedem but not backportable
20:14:38 dansmith I thought it gets triggered by the actual guest starting on the other side, which came from the actual live migration op
20:14:52 mriedem i'd need sahid to confirm that
20:15:04 mriedem but network-vif-plugged, as far as i know, comes from plug_vifs on the dest during pre_live_migratoin
20:15:08 mriedem which is before his code runs
20:15:11 mriedem on the source
20:15:18 dansmith so,
20:15:30 dansmith the even comes from the tap being created actually
20:15:35 dansmith *event
20:16:16 dansmith so maybe plug is creating a tap before libvirt does but I'm not sure how we'd give it to it
20:16:44 mriedem prometheanfire: i'm assuming you have this https://review.openstack.org/#/c/586965/
20:16:48 mriedem ^ fix for the pike backport
20:17:00 prometheanfire ya
20:17:07 dansmith I learned this after we were working on that patch though
20:17:10 prometheanfire that's within the sha I posted earlier
20:17:21 prometheanfire otherwise it wouldn't succeed at all :P
20:17:23 openstackgerrit Jay Pipes proposed openstack/nova master: placement: use simple code paths when possible https://review.openstack.org/590388
20:17:24 openstackgerrit Jay Pipes proposed openstack/nova master: split gigantor SQL placement query into multiple https://review.openstack.org/590041
20:17:26 mriedem prometheanfire: yeah
20:17:29 mriedem very obvious explosion
20:17:35 openstackgerrit Jay Pipes proposed openstack/nova master: Adds a test for _get_provider_ids_matching() https://review.openstack.org/590150
20:18:10 mriedem prometheanfire: you have this? https://review.openstack.org/#/c/510013/
20:18:11 sean-k-mooney mriedem: netwrokg-vif-plugged comre from neutron when it finishes wiering up the port
20:18:29 mriedem prometheanfire: this one was fun in that it depended on neutron backports as well
20:18:39 sean-k-mooney also i just as pluging stuff so need to scoll back to get context
20:18:59 prometheanfire that one I'm not sure, but probably
20:19:09 mriedem might want to check
20:19:23 prometheanfire checking
20:20:17 prometheanfire merged dec 3 into stable pike https://github.com/openstack/neutron/commits/stable/pike?after=ad8f00236cc57ce9a8f077dd2d32c6fada00e817+139
20:22:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/587498
20:22:48 dansmith if the event comes from stuff we're doing in pre_dest, it seems unlikely we'd ever win the race in gate
20:23:16 prometheanfire using at least this version of neutron https://github.com/openstack/openstack-ansible/blob/5c341a7bada78edab5f3d132d55adb00eaf2413f/playbooks/defaults/repo_packages/openstack_services.yml#L125
20:23:26 prometheanfire which is from 2018 in may
20:23:55 mriedem idk this is where i thought we'd generate the event https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7775
20:24:24 prometheanfire ok, I have to go for a bit, but will be back
20:24:34 dansmith well,
20:24:45 dansmith I did some debugging on plug stuff with the godaddy people a month or so ago,
20:24:51 dansmith and read through all the neutron code related to this
20:24:59 dansmith and I was surprised to learn that actually what happens is,
20:25:06 dansmith something creates an interface with the right name,
20:25:17 dansmith a periodic in the neutron agent notices, hooks it up and sends the event
20:25:30 dansmith so it's a little less connected to us than I would have thought
20:26:04 mriedem i knew the linuxbridge-agent does polling only b/c sean mooney explained that when we had the issue with waiting for hard reboot events for linuxbridge in the gate
20:26:20 mriedem ovs agent listens for an actual event from ovs itself
20:26:46 dansmith maybe that's why we can win it in the gate, since there was a polling loop
20:27:04 mriedem we use ovs in the gate
20:27:07 mriedem for most everything
20:27:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/587498
20:27:13 dansmith I thought we had a LB job?
20:27:14 mriedem neutron has a couple of lb jobs i think
20:27:32 dansmith regardless, you did a LB migration job and it passed a few times at least
20:27:34 mriedem i had rigged up a patch that ran linuxbridge multinode to test sahid's patch
20:27:41 mriedem yeah, could have won the race though right?
20:27:43 Sundar efried: Please ping me when you have a moment
20:28:07 efried Hi Sundar, what's up?
20:28:16 dansmith I guess because of the polling loop we have a decent chance
20:28:32 mriedem and all of our controller services on a single node slowing things down maybe, idk
20:28:38 mriedem don't know what prometheanfire's env setup is like
20:28:38 dansmith because pre-dest is pretty long before we get to the event wait, which seems ...crazy to ever win it
20:28:59 dansmith if so, presumably this patch broke live migration for anyone with a fast system
20:29:17 mriedem god i hope so
20:29:19 mriedem that would be magical
20:29:32 dansmith so, sahid has been arguing to make the timeout non-fatal, reportedly because someone was using a custom driver or something
20:29:39 dansmith but I wonder if it's actually because this is actually totes broken
20:29:54 Sundar We were discussing the relative roles of plugins and drivers. It seems to me that that the distinction is not a hard one. We just need some extension agent with clear APIs for both os-acc and Cyborg. It could be the same module providing two sets of APIs. Does that work for you?
20:30:20 efried Sundar: Absolutely.
20:30:33 dansmith mriedem: melwitt: tbh, if we really think this is that broken (and sounds like it is) then we should probably revert it from the release immediately
20:31:06 mriedem well the other thing i was saying above was,
20:31:16 Sundar Great. I was trying to delineate the two and it was getting rather ambiguous. Good to have this sorted out. Will send out the os-acc spec with this update
20:31:16 melwitt oof
20:31:20 mriedem even if we get the TimeoutError, we raise MigrationError or whatever,
20:31:32 mriedem but i *think* by that point we've already started the guest transfer
20:31:43 efried Sundar: Well, we should still delineate the two, if they're going to be *able* to be separate modules.
20:31:43 mriedem b/c we call _live_migration_operation
20:31:44 mriedem the nwait
20:31:44 dansmith correct
20:32:07 mriedem so starting the guest transfer and then timing out on the watier b/c we registered late kills the nova live migration option but not what's in the hypervisor
20:32:09 mriedem right?
20:32:18 mriedem s/option/operation/
20:32:29 dansmith mriedem: um, what?
20:32:52 mriedem if we get the timeout and raise MigrationError
20:32:56 mriedem but have already started the guest transfer
20:33:07 mriedem the only thing that happens in nova is we call the _rollback_live_migration code,
20:33:16 mriedem we don't attempt to kill any live migration job that's running in libvirt
20:33:17 Sundar efried: There will certainly be two sets of APIs: one for instance half of the attach and one for the device half. But both can do device-specific, platform-specific and vendor-specific actions.
20:33:17 mriedem right?
20:33:31 efried Sundar: I dig it.
20:33:49 efried Sundar: And I like the idea of being able to supply that code in one module or two.
20:33:54 dansmith lemme look
20:34:00 mriedem in other words,
20:34:10 mriedem nova will say "migratoin failed" but the guest might actually get transferred
20:34:17 mriedem just really f'ing slowly
20:34:52 dansmith well, yeah, I mean, the point of this code was to not raise the speed limit until it came
20:35:24 mriedem so did sahid want the timeout to just log and we'd have a finally that always set the bw back up?
20:35:54 dansmith yes
20:36:01 mriedem given what seems to be a pretty easy race to fail, that seems like it would have been better
20:36:04 dansmith which means you let it go to the other side but without networking
20:36:20 Sundar efried: So, it may be superfluous to have two separate modules, which are separately loaded by Stevedore. os-acc would have to load both, and the distinction in terms of what each module does seems to come down to APIs, rather than anything else. So, we might as well define two sets of APIs, and have one module do both. Internally, of course, the module may have separate packages/sub-modules for different functionalities.

Earlier   Later