| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-16 | |||
| 19:01:41 | sean-k-mooney | the end to end functional test you asked for found the issue | |
| 19:01:58 | sean-k-mooney | im just wonder what the best way to adress that is | |
| 19:03:16 | sean-k-mooney | gibi: this is what im doing which works fine for inventories as we start from scratch each time https://review.opendev.org/#/c/676522/44/nova/compute/resource_tracker.py@1751 | |
| 19:03:42 | sean-k-mooney | but for taits we start with the traits from placment | |
| 19:03:55 | sean-k-mooney | i guess i need to think about it again | |
| 19:04:12 | gibi | sean-k-mooney: let me sleep on it | |
| 19:04:30 | sean-k-mooney | ya kno worries | |
| 19:04:53 | gibi | sean-k-mooney: can it be that we say CUSTOM traits are always overwritten by the provider config as we don't expect that a virt driver reports CUSTOM traits anyhiw | |
| 19:04:57 | sean-k-mooney | it might be as simple as if we have prover.yaml remove all custome triats | |
| 19:05:13 | sean-k-mooney | gibi: ya so i think that is what erric wanted | |
| 19:05:28 | sean-k-mooney | either you manage if form the api and dont use the provider.yaml | |
| 19:05:47 | sean-k-mooney | or you use the provider.yaml in which case we can reset the traits and build them up again | |
| 19:07:04 | gibi | sean-k-mooney: yeah, this make senese | |
| 19:07:07 | gibi | sense | |
| 19:07:11 | sean-k-mooney | ill give that a try if you think that is valid | |
| 19:07:18 | gibi | we just need to document it carefully | |
| 19:07:39 | sean-k-mooney | ya ok ill see if i can make that work and get back to you. | |
| 19:08:09 | sean-k-mooney | the pardes may be canceled but its still st patricks day tomorrow so ill be off until wednesday | |
| 19:09:40 | gibi | sean-k-mooney: sure. happy st patricks day! | |
| 19:11:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ensures that COMPUTE_RESOURCE_SEMAPHORE usage is fair https://review.opendev.org/712674 | |
| 19:12:09 | gibi | melwitt: fixed it up real quick | |
| 19:12:35 | gibi | and now I'm gone for today | |
| 19:13:06 | melwitt | gibi: awesome thanks | |
| 19:50:16 | melwitt | dansmith: test coverage for fair locking is ready https://review.opendev.org/712674 | |
| 19:51:32 | dansmith | ah, I was like "this is dumb you're just testing your fixture" but I see now | |
| 19:51:57 | dansmith | like I said, I'm not really sure it's that important, but as long as it doesn't get in the way too much.. | |
| 19:57:34 | melwitt | yeah, I thought it's a nice way to cover this and catch any future uses without fair=True for the compute semaphore | |
| 20:04:44 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460 | |
| 20:36:45 | openstackgerrit | Merged openstack/nova stable/train: Functional test for UnexpectedDeletingTaskStateError https://review.opendev.org/711210 | |
| 21:11:39 | melwitt | artom: I just hit https://bugs.launchpad.net/nova/+bug/1813789 intermittent gate failure on one of my patches and saw you have patches/comments in the lp bug. do you have any idea where this is at right now? I saw you landed https://review.opendev.org/644881 9 months ago but it wasn't for this bug. just wondering you happen to know anything about the current bug we have in the gate | |
| 21:11:39 | openstack | Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,In progress] - Assigned to Artom Lifshitz (notartom) | |
| 21:13:19 | melwitt | mriedem: thought your ghost might find this interesting https://review.opendev.org/713035 | |
| 21:13:51 | artom | melwitt, IIRC my patch only addresses the revert-resize case | |
| 21:14:11 | artom | melwitt, so any other race was outsize of its scope | |
| 21:14:38 | mriedem | spooky | |
| 21:14:41 | melwitt | artom: oh, I see. thanks, helps to know that. I wonder if the same pattern could be applied to the evacuate case | |
| 21:14:48 | mriedem | so that's why the changes for that job on pike always failed? | |
| 21:15:01 | melwitt | mriedem: yeah | |
| 21:15:19 | artom | melwitt, at first pass I'd say no - I have to reload context, but it was a *really* specific scenario with revert resize | |
| 21:15:37 | melwitt | mriedem: lyarwood figured it out. it's failing nearly 100% on the pike branch | |
| 21:16:02 | mriedem | yeah i gave up on https://review.opendev.org/#/c/700072/ and thought it was due to some other pike thing that was fixed by QA awhile back | |
| 21:16:03 | melwitt | artom: ah, k. yeah even that vague info helps. I know nothing about it till now | |
| 21:16:04 | mriedem | but obviously not | |
| 21:16:33 | mriedem | artom: at the time we had talked about the same issue in evacuate | |
| 21:16:34 | artom | melwitt, like, the source host had to have the NIC already wired, and it had to be OVS | |
| 21:16:39 | mriedem | there is an old gate bug for that race | |
| 21:16:48 | artom | mriedem, yeah, but it can't have been the same root cause | |
| 21:16:52 | mriedem | http://status.openstack.org/elastic-recheck/#1813789 | |
| 21:17:14 | mriedem | no it's not due to using OVN or whatever :) | |
| 21:17:51 | artom | mriedem, right, which is why I ended up filing https://bugs.launchpad.net/nova/+bug/1832028 and using that in my patch | |
| 21:17:51 | openstack | Launchpad bug 1832028 in OpenStack Compute (nova) stein "revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug" [Medium,Fix committed] - Assigned to Artom Lifshitz (notartom) | |
| 21:18:06 | artom | Because turns out my thing from downstream was different than the intermittent upstream evacuate failures | |
| 21:18:24 | melwitt | dang | |
| 21:18:48 | mriedem | unless i was wrong on https://bugs.launchpad.net/nova/+bug/1813789 i had left comments about the order of events that showed the race | |
| 21:18:49 | openstack | Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,In progress] - Assigned to Artom Lifshitz (notartom) | |
| 21:19:03 | artom | melwitt, brutal honesty: stay away :P It's not a can of worms you want to open | |
| 21:19:06 | mriedem | i think by "the same" i meant the fix for evacuate is similar, we need to register the callback before plugging vifs | |
| 21:19:36 | mriedem | because right now for evacuate we bind ports to the new host and then spawn the guest and it's the low level spawn in the driver that registers the callback | |
| 21:19:37 | artom | mriedem, yeah, but did we ever work out *why* that was necessary? | |
| 21:19:41 | mriedem | and we could have already gotten the response from the port bind | |
| 21:19:49 | melwitt | artom: yeah. I already got my ass kicked looking at http://status.openstack.org/elastic-recheck/#1844929 spent days digging in and no dice so far | |
| 21:20:11 | artom | mriedem, wouldn't that depend on the Neutron backend though? | |
| 21:20:17 | mriedem | artom: i think i just said why :) and it's in the bug | |
| 21:20:34 | artom | mriedem, I'm pretty sure at least some of them would wait until libvirt plugs the VIF before sending out the event | |
| 21:20:39 | melwitt | artom: but anecdotally I see http://status.openstack.org/elastic-recheck/#1813789 fail really often in my gerrit notifications so argh ... just want to fix some of these | |
| 21:21:01 | mriedem | we use ovs in the gate and for that backend neutron sends the event when the port binding host changes | |
| 21:21:09 | sean-k-mooney | artom: ovs waits to send the event yes | |
| 21:21:11 | mriedem | which is why we *don't* get the event for things like hard reboot | |
| 21:21:32 | artom | mriedem, sean-k-mooney, get your stuff in line and stop contradicting yourselves ;) | |
| 21:22:16 | artom | melwitt, I get you - it's annoying and you want to fix it | |
| 21:22:22 | artom | It's just such a mess | |
| 21:22:26 | sean-k-mooney | mriedem: do you know if we create a second port binding and activate it our just update the host | |
| 21:22:35 | sean-k-mooney | mriedem: that would cahnge when teh event is sent | |
| 21:22:47 | mriedem | evacuate doesn't use multiple port bindings like live migration | |
| 21:23:07 | mriedem | when i left the only things that used multiple port bindings were live migration and cross-cell resize | |
| 21:23:12 | sean-k-mooney | ok we the condition to sent the event is the port must be in the active state and be bound to a host | |
| 21:23:40 | sean-k-mooney | so since its already in an active state when we bind it in the evacuate it might send the event imideatly | |
| 21:23:41 | artom | sean-k-mooney, so what mriedem was saying then | |
| 21:23:58 | sean-k-mooney | yep i havent checkt it but i would guess that he is correct | |
| 21:24:18 | artom | melwitt, well, if you want to take a whack at it, you could probably use the bind-time stuff I added to the model to change when evacuate starts listening for the event | |
| 21:24:23 | melwitt | ok so the main idea is find a way to register the callback earlier on | |
| 21:24:25 | artom | But there be dragons | |
| 21:24:47 | sean-k-mooney | artom: if we careted a second port binding it would not send it until we activate it | |
| 21:25:49 | melwitt | artom: thanks | |
| 21:26:17 | artom | melwitt, well, yes and no - looking at my own code, depending on whether the port has what I called "bind_time_events", you wait in the compute manager when you send the Neutron request | |
| 21:26:45 | artom | And if they're "plug_time_events", you wait in the virt driver when you plug the VIFs | |
| 21:26:53 | melwitt | ahhh ok | |
| 21:26:54 | sean-k-mooney | mriedem: yes those are still the only things that use the multiple prot bindings | |
| 21:27:04 | artom | melwitt, also, ask sean-k-mooney ;) | |
| 21:27:08 | artom | (Bus, meet Sean :D ) | |
| 21:27:13 | melwitt | lol | |
| 21:27:32 | sean-k-mooney | artom: then you just delete the code and start again | |
| 21:27:46 | artom | Can we do that for all of Nova? ;) | |
| 21:28:16 | melwitt | I think I vaguely get it. I can read though your patch, just knowing a generic idea of what's going on helps a lot. saves a lot of time | |
| 21:29:04 | mriedem | "if we careted a second port binding it would not send it until we activate it" is a bigger non-backportable change most likely because of the behavior changes between compute and conductor | |
| 21:29:06 | artom | melwitt, ping me if you have questions / need review / whatever | |
| 21:29:34 | melwitt | thanks ++ | |
| 21:29:38 | artom | (/me needs pressure to "re-join" upstream) | |
| 21:29:45 | artom | I've been neglecting y'all | |
| 21:30:08 | mriedem | don't forget to loop dansmith into this when you want to talk about it, i'm sure he'd love to | |