Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-16
16:22:01 artom melwitt, oh? So what happens when the host isn't reachable? We report the last recorded status from the DB?
16:22:29 melwitt artom: correct. only host status will say UNKNOWN
16:22:40 dansmith host_status on the instance gives you a sanitized "don't expect this instance to be actionable because the host is not healthy"
16:23:02 dansmith normally only admins can see info about hosts, so that field is the indicator to the user that "things are not as they appear" without exposing too much
16:23:28 melwitt artom: and host status is normally admin-only, so I added a new policy rule host_status:unknown-only that defaults to admin-only intended for operators who want to let non-admin users see UNKNOWN host status
16:24:23 melwitt host_status policy rule includes showing UP, DOWN, MAINTENANCE, UNKNOWN and host_status:unknown-only shows only UNKNOWN
16:24:46 artom melwitt, aha, thanks :) Can you find a link to that patch/spec/whatever?
16:24:59 melwitt artom: https://review.opendev.org/679181
16:34:06 sean-k-mooney dansmith: no worries. gibi and erric were the main reviews so im just thinking of who can take over form erric. when its ready ill add it to the runway list and see how is interested
16:43:42 gibi sean-k-mooney: the provider config series are also on my radar
16:44:05 evrardjp https://review.opendev.org/711950 has merged. Congratulations!
16:44:43 sean-k-mooney gibi: yep im adressing the feedback on the last patch currently
16:44:52 gibi sean-k-mooney: cool. thanks
16:45:11 sean-k-mooney ill take a look at the unit tests once i have that done
16:58:08 gibi cores, there is a trivial, functional test only change: https://review.opendev.org/#/c/713243/
17:21:50 openstackgerrit melanie witt proposed openstack/nova master: Follow-ups for host_status:unknown-only policy rule https://review.opendev.org/713295
17:23:30 openstackgerrit Merged openstack/nova master: Fix intermittently failing regression case https://review.opendev.org/713243
17:28:39 melwitt dansmith: pedantic correction to what I said earlier, the spec was abandoned bc it was decided to no longer need a spec when it became policy rule only. and then blueprint was approved after it became policy rule only
17:29:32 melwitt (in case you ever go looking at the spec and can't find it, like I just did)
17:29:44 melwitt *for
17:49:42 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Tell oslo.limit how to count nova resources https://review.opendev.org/713301
17:49:42 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Enforce resource limits using oslo.limit https://review.opendev.org/615180
17:54:46 melwitt gibi: I dunno if you saw my comment in https://review.opendev.org/712674 test just needs a tweak to handle kwarg vs positional arg
17:55:22 dansmith melwitt: heh okay..
17:57:16 melwitt just in case you opened it and went, hWHAT! abandoned! ?!?!
18:09:24 openstackgerrit Dan Smith proposed openstack/nova master: Remove non-optional kwarg for virt block_device_info https://review.opendev.org/713310
18:36:38 gibi melwitt: ack, now see it. thanks. I will get back to that tomorrow
18:38:18 melwitt gibi: k. I didn't want to update it, so I will be able to +2
18:39:46 gibi melwitt: sure
18:42:42 openstackgerrit Balazs Gibizer proposed openstack/nova master: [Community goal] Update contributor documentation https://review.opendev.org/712420
18:43:44 gibi stephenfin, brinzhang_: fixed the comments ^^
18:58:34 sean-k-mooney gibi are you done for the day or are you still around
18:59:42 gibi sean-k-mooney: I'm here for quick questions, not for longer things
19:00:17 sean-k-mooney gibi: its related to the provider.conf
19:00:48 sean-k-mooney basically the way i was asserting that the triats dont conflict with the virt driver traits does not quite work
19:01:16 sean-k-mooney what hapens is it works on the first iteration then fails on the second as the trait is already there
19:01:41 sean-k-mooney the end to end functional test you asked for found the issue
19:01:58 sean-k-mooney im just wonder what the best way to adress that is
19:03:16 sean-k-mooney gibi: this is what im doing which works fine for inventories as we start from scratch each time https://review.opendev.org/#/c/676522/44/nova/compute/resource_tracker.py@1751
19:03:42 sean-k-mooney but for taits we start with the traits from placment
19:03:55 sean-k-mooney i guess i need to think about it again
19:04:12 gibi sean-k-mooney: let me sleep on it
19:04:30 sean-k-mooney ya kno worries
19:04:53 gibi sean-k-mooney: can it be that we say CUSTOM traits are always overwritten by the provider config as we don't expect that a virt driver reports CUSTOM traits anyhiw
19:04:57 sean-k-mooney it might be as simple as if we have prover.yaml remove all custome triats
19:05:13 sean-k-mooney gibi: ya so i think that is what erric wanted
19:05:28 sean-k-mooney either you manage if form the api and dont use the provider.yaml
19:05:47 sean-k-mooney or you use the provider.yaml in which case we can reset the traits and build them up again
19:07:04 gibi sean-k-mooney: yeah, this make senese
19:07:07 gibi sense
19:07:11 sean-k-mooney ill give that a try if you think that is valid
19:07:18 gibi we just need to document it carefully
19:07:39 sean-k-mooney ya ok ill see if i can make that work and get back to you.
19:08:09 sean-k-mooney the pardes may be canceled but its still st patricks day tomorrow so ill be off until wednesday
19:09:40 gibi sean-k-mooney: sure. happy st patricks day!
19:11:59 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ensures that COMPUTE_RESOURCE_SEMAPHORE usage is fair https://review.opendev.org/712674
19:12:09 gibi melwitt: fixed it up real quick
19:12:35 gibi and now I'm gone for today
19:13:06 melwitt gibi: awesome thanks
19:50:16 melwitt dansmith: test coverage for fair locking is ready https://review.opendev.org/712674
19:51:32 dansmith ah, I was like "this is dumb you're just testing your fixture" but I see now
19:51:57 dansmith like I said, I'm not really sure it's that important, but as long as it doesn't get in the way too much..
19:57:34 melwitt yeah, I thought it's a nice way to cover this and catch any future uses without fair=True for the compute semaphore
20:04:44 openstackgerrit sean mooney proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460
20:36:45 openstackgerrit Merged openstack/nova stable/train: Functional test for UnexpectedDeletingTaskStateError https://review.opendev.org/711210
21:11:39 openstack Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,In progress] - Assigned to Artom Lifshitz (notartom)
21:11:39 melwitt artom: I just hit https://bugs.launchpad.net/nova/+bug/1813789 intermittent gate failure on one of my patches and saw you have patches/comments in the lp bug. do you have any idea where this is at right now? I saw you landed https://review.opendev.org/644881 9 months ago but it wasn't for this bug. just wondering you happen to know anything about the current bug we have in the gate
21:13:19 melwitt mriedem: thought your ghost might find this interesting https://review.opendev.org/713035
21:13:51 artom melwitt, IIRC my patch only addresses the revert-resize case
21:14:11 artom melwitt, so any other race was outsize of its scope
21:14:38 mriedem spooky
21:14:41 melwitt artom: oh, I see. thanks, helps to know that. I wonder if the same pattern could be applied to the evacuate case
21:14:48 mriedem so that's why the changes for that job on pike always failed?
21:15:01 melwitt mriedem: yeah
21:15:19 artom melwitt, at first pass I'd say no - I have to reload context, but it was a *really* specific scenario with revert resize
21:15:37 melwitt mriedem: lyarwood figured it out. it's failing nearly 100% on the pike branch
21:16:02 mriedem yeah i gave up on https://review.opendev.org/#/c/700072/ and thought it was due to some other pike thing that was fixed by QA awhile back
21:16:03 melwitt artom: ah, k. yeah even that vague info helps. I know nothing about it till now
21:16:04 mriedem but obviously not
21:16:33 mriedem artom: at the time we had talked about the same issue in evacuate
21:16:34 artom melwitt, like, the source host had to have the NIC already wired, and it had to be OVS
21:16:39 mriedem there is an old gate bug for that race
21:16:48 artom mriedem, yeah, but it can't have been the same root cause
21:16:52 mriedem http://status.openstack.org/elastic-recheck/#1813789
21:17:14 mriedem no it's not due to using OVN or whatever :)
21:17:51 openstack Launchpad bug 1832028 in OpenStack Compute (nova) stein "revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug" [Medium,Fix committed] - Assigned to Artom Lifshitz (notartom)
21:17:51 artom mriedem, right, which is why I ended up filing https://bugs.launchpad.net/nova/+bug/1832028 and using that in my patch
21:18:06 artom Because turns out my thing from downstream was different than the intermittent upstream evacuate failures
21:18:24 melwitt dang
21:18:48 mriedem unless i was wrong on https://bugs.launchpad.net/nova/+bug/1813789 i had left comments about the order of events that showed the race
21:18:49 openstack Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,In progress] - Assigned to Artom Lifshitz (notartom)
21:19:03 artom melwitt, brutal honesty: stay away :P It's not a can of worms you want to open
21:19:06 mriedem i think by "the same" i meant the fix for evacuate is similar, we need to register the callback before plugging vifs
21:19:36 mriedem because right now for evacuate we bind ports to the new host and then spawn the guest and it's the low level spawn in the driver that registers the callback
21:19:37 artom mriedem, yeah, but did we ever work out *why* that was necessary?
21:19:41 mriedem and we could have already gotten the response from the port bind
21:19:49 melwitt artom: yeah. I already got my ass kicked looking at http://status.openstack.org/elastic-recheck/#1844929 spent days digging in and no dice so far
21:20:11 artom mriedem, wouldn't that depend on the Neutron backend though?
21:20:17 mriedem artom: i think i just said why :) and it's in the bug
21:20:34 artom mriedem, I'm pretty sure at least some of them would wait until libvirt plugs the VIF before sending out the event

Earlier   Later