| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-16 | |||
| 21:15:19 | artom | melwitt, at first pass I'd say no - I have to reload context, but it was a *really* specific scenario with revert resize | |
| 21:15:37 | melwitt | mriedem: lyarwood figured it out. it's failing nearly 100% on the pike branch | |
| 21:16:02 | mriedem | yeah i gave up on https://review.opendev.org/#/c/700072/ and thought it was due to some other pike thing that was fixed by QA awhile back | |
| 21:16:03 | melwitt | artom: ah, k. yeah even that vague info helps. I know nothing about it till now | |
| 21:16:04 | mriedem | but obviously not | |
| 21:16:33 | mriedem | artom: at the time we had talked about the same issue in evacuate | |
| 21:16:34 | artom | melwitt, like, the source host had to have the NIC already wired, and it had to be OVS | |
| 21:16:39 | mriedem | there is an old gate bug for that race | |
| 21:16:48 | artom | mriedem, yeah, but it can't have been the same root cause | |
| 21:16:52 | mriedem | http://status.openstack.org/elastic-recheck/#1813789 | |
| 21:17:14 | mriedem | no it's not due to using OVN or whatever :) | |
| 21:17:51 | artom | mriedem, right, which is why I ended up filing https://bugs.launchpad.net/nova/+bug/1832028 and using that in my patch | |
| 21:17:51 | openstack | Launchpad bug 1832028 in OpenStack Compute (nova) stein "revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug" [Medium,Fix committed] - Assigned to Artom Lifshitz (notartom) | |
| 21:18:06 | artom | Because turns out my thing from downstream was different than the intermittent upstream evacuate failures | |
| 21:18:24 | melwitt | dang | |
| 21:18:48 | mriedem | unless i was wrong on https://bugs.launchpad.net/nova/+bug/1813789 i had left comments about the order of events that showed the race | |
| 21:18:49 | openstack | Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,In progress] - Assigned to Artom Lifshitz (notartom) | |
| 21:19:03 | artom | melwitt, brutal honesty: stay away :P It's not a can of worms you want to open | |
| 21:19:06 | mriedem | i think by "the same" i meant the fix for evacuate is similar, we need to register the callback before plugging vifs | |
| 21:19:36 | mriedem | because right now for evacuate we bind ports to the new host and then spawn the guest and it's the low level spawn in the driver that registers the callback | |
| 21:19:37 | artom | mriedem, yeah, but did we ever work out *why* that was necessary? | |
| 21:19:41 | mriedem | and we could have already gotten the response from the port bind | |
| 21:19:49 | melwitt | artom: yeah. I already got my ass kicked looking at http://status.openstack.org/elastic-recheck/#1844929 spent days digging in and no dice so far | |
| 21:20:11 | artom | mriedem, wouldn't that depend on the Neutron backend though? | |
| 21:20:17 | mriedem | artom: i think i just said why :) and it's in the bug | |
| 21:20:34 | artom | mriedem, I'm pretty sure at least some of them would wait until libvirt plugs the VIF before sending out the event | |
| 21:20:39 | melwitt | artom: but anecdotally I see http://status.openstack.org/elastic-recheck/#1813789 fail really often in my gerrit notifications so argh ... just want to fix some of these | |
| 21:21:01 | mriedem | we use ovs in the gate and for that backend neutron sends the event when the port binding host changes | |
| 21:21:09 | sean-k-mooney | artom: ovs waits to send the event yes | |
| 21:21:11 | mriedem | which is why we *don't* get the event for things like hard reboot | |
| 21:21:32 | artom | mriedem, sean-k-mooney, get your stuff in line and stop contradicting yourselves ;) | |
| 21:22:16 | artom | melwitt, I get you - it's annoying and you want to fix it | |
| 21:22:22 | artom | It's just such a mess | |
| 21:22:26 | sean-k-mooney | mriedem: do you know if we create a second port binding and activate it our just update the host | |
| 21:22:35 | sean-k-mooney | mriedem: that would cahnge when teh event is sent | |
| 21:22:47 | mriedem | evacuate doesn't use multiple port bindings like live migration | |
| 21:23:07 | mriedem | when i left the only things that used multiple port bindings were live migration and cross-cell resize | |
| 21:23:12 | sean-k-mooney | ok we the condition to sent the event is the port must be in the active state and be bound to a host | |
| 21:23:40 | sean-k-mooney | so since its already in an active state when we bind it in the evacuate it might send the event imideatly | |
| 21:23:41 | artom | sean-k-mooney, so what mriedem was saying then | |
| 21:23:58 | sean-k-mooney | yep i havent checkt it but i would guess that he is correct | |
| 21:24:18 | artom | melwitt, well, if you want to take a whack at it, you could probably use the bind-time stuff I added to the model to change when evacuate starts listening for the event | |
| 21:24:23 | melwitt | ok so the main idea is find a way to register the callback earlier on | |
| 21:24:25 | artom | But there be dragons | |
| 21:24:47 | sean-k-mooney | artom: if we careted a second port binding it would not send it until we activate it | |
| 21:25:49 | melwitt | artom: thanks | |
| 21:26:17 | artom | melwitt, well, yes and no - looking at my own code, depending on whether the port has what I called "bind_time_events", you wait in the compute manager when you send the Neutron request | |
| 21:26:45 | artom | And if they're "plug_time_events", you wait in the virt driver when you plug the VIFs | |
| 21:26:53 | melwitt | ahhh ok | |
| 21:26:54 | sean-k-mooney | mriedem: yes those are still the only things that use the multiple prot bindings | |
| 21:27:04 | artom | melwitt, also, ask sean-k-mooney ;) | |
| 21:27:08 | artom | (Bus, meet Sean :D ) | |
| 21:27:13 | melwitt | lol | |
| 21:27:32 | sean-k-mooney | artom: then you just delete the code and start again | |
| 21:27:46 | artom | Can we do that for all of Nova? ;) | |
| 21:28:16 | melwitt | I think I vaguely get it. I can read though your patch, just knowing a generic idea of what's going on helps a lot. saves a lot of time | |
| 21:29:04 | mriedem | "if we careted a second port binding it would not send it until we activate it" is a bigger non-backportable change most likely because of the behavior changes between compute and conductor | |
| 21:29:06 | artom | melwitt, ping me if you have questions / need review / whatever | |
| 21:29:34 | melwitt | thanks ++ | |
| 21:29:38 | artom | (/me needs pressure to "re-join" upstream) | |
| 21:29:45 | artom | I've been neglecting y'all | |
| 21:30:08 | mriedem | don't forget to loop dansmith into this when you want to talk about it, i'm sure he'd love to | |
| 21:30:38 | artom | You're such a good friend | |
| 21:30:38 | melwitt | artom: heh. I might end up running away from this screaming after I try to work on it, so if that doesn't happen maybe I'll ping you | |
| 21:31:01 | artom | melwitt, screaming would be a good sign, actually | |
| 21:31:02 | sean-k-mooney | ya we cant backport adopting multiple port bindings for evacuate | |
| 21:31:11 | artom | Means you're sane (inasmuch as that's still possible) | |
| 21:31:14 | sean-k-mooney | im not sure if we want to do that or not in general | |
| 21:31:18 | melwitt | lol | |
| 21:31:36 | sean-k-mooney | it might be useful but its alot of work to untangel things and make sure it works | |
| 21:32:14 | artom | Yep. We'd also need to run it with a couple of other Neutron backends | |
| 21:32:28 | artom | IIRC we created a DNM job to run against... OVS? OVN? | |
| 21:32:38 | artom | OVB? OG? | |
| 21:32:39 | sean-k-mooney | ovs ovn and lb | |
| 21:32:47 | artom | RunDMC? | |
| 21:33:20 | melwitt | 😂 | |
| 21:33:20 | melwitt | 😂 | |
| 21:33:30 | melwitt | had to use an emoji for that one | |
| 21:34:00 | sean-k-mooney | sure you "had too" :P | |
| 21:34:24 | melwitt | yeah, it wouldn't let me type anything else until I posted the emoji | |
| 21:35:21 | sean-k-mooney | did https://bugs.launchpad.net/nova/+bug/1813789 come up recently downstream or in relation to the nova-livemigation job | |
| 21:35:21 | openstack | Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,In progress] - Assigned to Artom Lifshitz (notartom) | |
| 21:35:33 | sean-k-mooney | i rembere talking about it a few days ago | |
| 21:35:39 | sean-k-mooney | i just dont recall the context | |
| 21:35:53 | melwitt | lyarwood mentioned it in the nova meeting | |
| 21:36:15 | sean-k-mooney | ah ya it was in context of the zuul v3 migration | |
| 21:36:18 | melwitt | just saying he hit that bug and http://status.openstack.org/elastic-recheck/#1844929 a bunch of times while trying to get some work done | |
| 21:36:35 | sean-k-mooney | yep | |
| 21:37:08 | melwitt | so I started looking at http://status.openstack.org/elastic-recheck/#1844929 and got nowhere. and now I know http://status.openstack.org/elastic-recheck/#1813789 is also horrid | |
| 21:38:44 | sean-k-mooney | so given we dont use the multiple port bindings flow i think we can assume that as long as the port is still active it will recive a bind time even rather then plug time | |
| 21:38:50 | mriedem | those probably aren't even in the same ballpark of terrible | |
| 21:39:15 | sean-k-mooney | but i would need to think that true more carefully to make sure that is correct | |
| 21:39:46 | mriedem | in the bug i linked in logs where the things were happening so it's not really a question of where the race is | |
| 21:39:57 | mriedem | though those log links are going to be dead by now | |
| 21:40:11 | mriedem | unshelve has the same issue | |
| 21:40:39 | mriedem | 1. bind to new host triggers async network-vif-plugged event, 2. driver.spawn plugs vifs which sets up the callback handler | |
| 21:40:47 | mriedem | if you get the event before 2 you're stuck | |
| 21:40:55 | melwitt | right, ok | |
| 21:41:42 | sean-k-mooney | mriedem: in the unsevle case you could argue that when we go to shevle offloaded teh status of the port shoudl be down | |
| 21:42:08 | sean-k-mooney | which woudl prevent the event being sent in the ovs case atleast until its plugged in the driver | |