| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-05 | |||
| 20:00:17 | dansmith | there's one other place I remember, ahgn on | |
| 20:00:21 | artom | Unless it's implicit in the 'ACTIVE' check | |
| 20:01:46 | dansmith | hmm, I thought there was also a vif type filter in mriedem's more generic check for live migration, but I don't see it there either | |
| 20:01:48 | dansmith | so maybe I'm just thinking about that patch from sahid that we reverted | |
| 20:01:48 | artom | And actually in libvirt _create_domain_and_network is the only time we call wait_for_instance_event | |
| 20:01:59 | dansmith | ...in the libvirt driver, yes | |
| 20:03:04 | artom | Ah, there's _get_neutron_events_for_live_migration in the compute manager | |
| 20:03:17 | artom | No vif type filter though | |
| 20:03:51 | dansmith | yeah, I really thought that one had a filter on it, maybe mriedem can comment when he sees this | |
| 20:03:58 | dansmith | lemme dig up the sahid patch to prove there was one at least | |
| 20:04:24 | mriedem | vif type == linuxbridge? | |
| 20:04:34 | mriedem | vif type goes out the window because we can't rely on it | |
| 20:04:44 | mriedem | ala https://review.openstack.org/#/c/553035/ | |
| 20:05:01 | mriedem | opendaylight is ovs vif type but doesn't send network-vif-plugged on vif plugging, only port binding | |
| 20:05:16 | mriedem | aspiers: no i don't need a graph of the code patches, | |
| 20:05:17 | dansmith | artom: https://review.openstack.org/#/c/497457/30/nova/virt/libvirt/driver.py@5536 | |
| 20:05:50 | mriedem | aspiers: the reason i was asking is because there doesn't seem to be a reason, to me, to land the sev discovery / exposing the capability from the host change (which jaypipes is +2 on) until we actually have code that can plumb the guest | |
| 20:05:56 | dansmith | mriedem: ah, did we have a check for that before we had to remove because of ODL? | |
| 20:06:00 | artom | dansmith, hrmm, why'd that get reverted out? | |
| 20:06:07 | dansmith | artom: for so many reasons | |
| 20:06:12 | dansmith | artom: amazingly broken :) | |
| 20:06:28 | mriedem | artom: for one thing it was setting up the waiter *after* we'd plugged vifs on the dest | |
| 20:06:32 | mriedem | and the waiter code was waiting on the source | |
| 20:06:34 | mriedem | so a race and all that | |
| 20:06:53 | artom | I was thinking more narrowly in terms of the VIF type logic | |
| 20:07:08 | dansmith | artom: https://review.openstack.org/#/q/Ibf2b5eeafd962e93ae4ab6290015d58c33024132 | |
| 20:07:17 | artom | IOW, is "Currently we only use the mechanism of waiting for neutron events during live-migration for linux-bridge" false? | |
| 20:08:01 | dansmith | artom: it turned out that neutron was sending those events, this just excluded waiting for them because of other bugs.. papering over the real problem | |
| 20:08:13 | dansmith | it's been a while, but there was a LOT of analysis and discussion | |
| 20:08:30 | mriedem | artom: true that is false :) | |
| 20:08:35 | aspiers | mriedem: well one reason is that we (SUSE) will need to carry our own downstream backport to Rocky, so the more which lands in Stein the easier it will get for us ... but if that is in conflict with upstream goals then of course I would entirely understand | |
| 20:08:48 | mriedem | artom: https://docs.openstack.org/nova/latest/configuration/config.html#compute.live_migration_wait_for_vif_plug | |
| 20:08:59 | mriedem | ^ we wait for networking events during live migration based on config | |
| 20:09:01 | mriedem | since rocky | |
| 20:09:07 | mriedem | and part of that was backported | |
| 20:09:14 | mriedem | by mr smith | |
| 20:09:48 | mriedem | aspiers: heh, sure - i just don't think we want the sev capability showing up automatically in stein on the compute node resource provider when there is really nothing you can do with it | |
| 20:10:33 | dansmith | ah right, artom, the Note:: in that doc explains why the vif_type thing isn't usable and why you have to config your way out of this for certain backends | |
| 20:10:34 | dansmith | which sucsk | |
| 20:10:48 | aspiers | mriedem: yeah, totally understood ... we were really hoping to get all the patches ready in time for stein but it looks like we just weren't quite quick enough | |
| 20:11:33 | mriedem | aspiers: no problem, you got that capabilities patch landed which i'd abandoned essentially, which is pretty good for your first big contribution to nova | |
| 20:11:52 | artom | dansmith, I get all of it except for "without an accompanying port binding:host_id" | |
| 20:11:54 | mriedem | plus you had that acid trip of a venn diagram | |
| 20:12:06 | mriedem | artom: that's what i was saying earlier, | |
| 20:12:15 | dansmith | artom: ever get the feeling you stepped into a hornet's nest? | |
| 20:12:17 | mriedem | ODL will only send network-vif-plugged for port binding changes to the host | |
| 20:12:34 | mriedem | if the host doesn't change on the port binding, but we plug/unplug vifs or whatever, ODL doesn't send the event | |
| 20:12:36 | aspiers | mriedem: thanks :) feels good to be giving to upstream not just "taking" (i.e. asking for our own pet feature to be reviewed and merged) | |
| 20:12:52 | mriedem | artom: hence why it had to be reverted from that hard reboot change | |
| 20:13:00 | mriedem | this: https://review.openstack.org/#/c/553035/ | |
| 20:13:01 | artom | dansmith, yes | |
| 20:13:25 | aspiers | mriedem: haha, well my first version of that venn diagram was quite different, you can blame^Wthank efried for a lot of that second version | |
| 20:13:37 | mriedem | linuxbridge also had some problems with that in the neutron lb job because something about polling for changes and it would miss them and not send the event | |
| 20:13:42 | melwitt | I concur, thanks aspiers++ | |
| 20:13:48 | artom | mriedem, ah, so the note is just worded poorly to say "ODL will only send us vif-plugged if the host has changed" | |
| 20:13:51 | dansmith | mriedem: ah, yeah that's the thing I was expecting to find in there, forgot that had to be reverted too | |
| 20:13:53 | aspiers | thanks melwitt :) | |
| 20:14:04 | artom | ... which obviously it doesn't on hard reboot | |
| 20:14:05 | mriedem | artom: it is worded perfectly | |
| 20:14:38 | artom | Nuh uh, communication is only perfect if the thickest of idiots understand it. I didn't understand this. QED. | |
| 20:14:49 | mriedem | of course the port binding host *does* change during live migration, | |
| 20:15:03 | mriedem | but that happens after / during the guest transfer, not as a result of plug_vifs during pre_live_migration | |
| 20:15:06 | mriedem | which is what that config is used for | |
| 20:15:53 | mriedem | also, let me say, that i'm afraid sean-k-mooney might be dead or something because this entire conversation is the kind of thing he stays up until 3am for | |
| 20:16:01 | artom | Seriously. | |
| 20:16:23 | aspiers | LOL | |
| 20:16:25 | artom | RIP sean I guess. We'll do a minute of silence i Denver. | |
| 20:18:21 | artom | mriedem, wait, so why is that note even there? It's talking about not sending events when the host binding doesn't change. | |
| 20:18:30 | artom | That doesn't apply to live migration... | |
| 20:18:52 | jaypipes | mriedem: was I not supposed to +2 something? | |
| 20:20:47 | aspiers | jaypipes: hopefully +2 is OK, W+1 might not have been though :) | |
| 20:21:16 | artom | So, I think I should ignore the whole Neutron backend thing for now. It's out of scope, and I shouldn't introduce new bugs by ignoring it because it's currently ignored anyways. We can continue to wait for events in the same situations as currently, we just need to change the timing of the waits. | |
| 20:23:20 | melwitt | poor wording, that reminds me, I was supposed to update the consoleauth reno (or was it nova-status?) yet again to make more sense. I need to find IRC log from that convo with imacdonn | |
| 20:29:22 | imacdonn | melwitt: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-02-20.log.html#t2019-02-20T18:46:52 | |
| 20:29:58 | edmondsw | mriedem yeah, I agree with you and efried. Should be fine, except that we do need to go update the OOT driver as well, so the heads up is definitely appreciated. | |
| 20:30:16 | melwitt | imacdonn: thanks, you're awesome | |
| 20:30:22 | imacdonn | :) | |
| 20:32:05 | mriedem | artom: ... | |
| 20:32:18 | mriedem | artom: it's saying where this config is used, might not make sense for all networking backends (ODL), | |
| 20:32:27 | mriedem | because the config is only for vif plugging, not port binding host changes, | |
| 20:32:34 | mriedem | ODL doesn't send events for the former, but does for the latter, | |
| 20:32:38 | mriedem | but the latter is not where that config is used | |
| 20:32:49 | mriedem | port binding happens during/after the guest transfer, | |
| 20:32:57 | mriedem | we're waiting for the event before starting the guest transfer | |
| 20:33:09 | mriedem | because if the dest plug fails or something, we don't want to start transferring the guest | |
| 20:33:57 | mriedem | dansmith: check my comments on takashin's fix here https://review.openstack.org/#/c/636271/ - should i just not care about a functional test for live migration or stack it up on top myself if i care? | |
| 20:34:06 | mriedem | i'm probably reaching | |
| 20:34:27 | mriedem | jaypipes: see my comment on https://review.openstack.org/#/c/638680/ | |
| 20:35:05 | dansmith | mriedem: I dunno, the real change is in that object loader, which seems pretty well covered by two operations, so .. seemed like enough to me | |
| 20:35:21 | artom | mriedem, ah, got it, I think. Sorry, am slow. | |
| 20:35:28 | dansmith | I pulled it down and poked at it too, both fail in the expected way... | |
| 20:35:33 | mriedem | dansmith: yeah ok, i just don't like being told it's not an issue for live migration :) | |
| 20:36:04 | dansmith | well, I don't think stacking one on top is a problem if it makes you feel better | |
| 20:36:33 | mriedem | it won't fill the hole inside | |
| 20:36:44 | dansmith | hah | |
| 20:36:59 | artom | mriedem, oh, ODL *doesn't* send events for binding host changes | |
| 20:37:27 | mriedem | WRONG | |
| 20:37:35 | artom | DAMMIT | |
| 20:37:41 | mriedem | it *only* sends events in that case | |
| 20:37:42 | dansmith | ARGH | |