Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-09
19:41:00 sean-k-mooney mriedem: are you flying out to berlin today/tomorow?
19:41:04 mriedem the problem the huawei ops team ran into was policy changed on the neutron side which started returning an empty list of ports, which was then saved into the info cache in the nova db,
19:41:18 mriedem and the heal periodic relies on the info cache rather than the source of truth to fix the cache
19:41:22 mriedem tonight
19:41:59 mriedem there are other ways to simply rebuild the cache if that's what is needed, e.g. https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-refresh-network
19:42:06 sean-k-mooney right. i think we discussed this in the past at some point too it feels familar but i have not reviewd this before
19:42:18 mriedem it came up at the ptg i think
19:42:39 mriedem and the public cloud sig has brought it up before (OVH obviously)
19:42:51 sean-k-mooney mriedem: perhaps. oh so we can force a rebuild via the nova client today?
19:43:01 mriedem it doesn't rebuild from neutron
19:43:14 mriedem i don't think anyway
19:43:21 mriedem it just sends a network-changed event to the compute
19:43:41 sean-k-mooney oh it rebuilds form the vif table in the nova db?
19:43:53 mriedem yes
19:43:57 mriedem well,
19:43:59 mriedem from the info cache
19:44:18 mriedem iow,
19:44:26 sean-k-mooney ok but if the info cache got currpted when the host rebooted your still stuck
19:44:32 mriedem the network-changed event and _heal_instance_info_cache periodic do the same thing today
19:44:36 mriedem correct
19:44:48 mriedem hence the reason for making the periodic actually "heal" from the source of truth, which is neutron
19:44:53 mriedem and not our potentially corrupted cache
19:45:13 sean-k-mooney so i suggested they detach the missing interfaces and reattach them as a workaround for now to try and force nova and neutron to resync
19:45:30 sean-k-mooney i think that would work in this case but its not ideal
19:47:04 sean-k-mooney mriedem: anyway thanks. i was sraching my head for the last day or so trying to parse what was going on from incomplete logs but im 98% sure this is it.
19:47:49 sean-k-mooney mriedem: have a safe trip.
20:48:18 mriedem thanks
20:58:12 dansmith mriedem: so you found a test that confirmed the behavior of that thing?
20:58:19 dansmith mriedem: that deletes the consumer?
21:02:45 mriedem DeleteConsumerIfNoAllocsTestCase is the functional test that covers that case,
21:02:50 mriedem and it looks like a correct test to me,
21:03:02 mriedem creates 2 consumers each with 2 allocations on different resource classes,
21:03:09 mriedem clears the allocations for one of them and asserts the consumer is gone
21:03:29 mriedem i think we're just hitting a race with the shelve offloaded status change before we cleanup the allocations
21:03:40 mriedem but i've posted a couple of patches to add debug logs to help determine if that's the case
21:03:51 mriedem https://review.openstack.org/617016
21:04:55 dansmith okay I'm not sure how we could race and see no allocations but a consumer and get that generation conflict
21:05:13 dansmith it'd be one thing if we thought the consumer was there and then disappeared out from under us
21:17:11 mriedem during unshelve the scheduler does see allocations
21:17:31 mriedem and it thinks we're doing a move
21:17:56 dansmith okay I thought you pasted a line showing that there was only one allocation going back to placement
21:18:07 mriedem there are 3 PUTs for allocations
21:18:11 mriedem 1. create the server - initial
21:18:23 mriedem 2. shelve offload - wipe the allocations to {} - which should delete the consumer
21:18:33 mriedem 3. unshelve - scheduler claims resources with the wrong consumer generation
21:18:45 mriedem and when 3 happens, the scheduler gets allocations for hte consumer and they are there,
21:18:47 dansmith ...right
21:18:55 mriedem so it uses the consumer generation (1) from those allocations
21:19:03 mriedem then i think what happens is,
21:19:05 dansmith oh, so it passes generation=1 instead of generation=0, meaning new consumer?
21:19:12 mriedem placement recreates the consumer which will have generation null
21:19:15 mriedem yes
21:19:22 dansmith okay I see
21:19:46 dansmith I thought you were seeing consumer generation was null or zero or whatever in the third put, but still getting a conflict
21:19:49 dansmith but that makes sense now
21:20:03 mriedem Nov 06 19:48:37.013780 ubuntu-xenial-inap-mtl01-0000379614 nova-scheduler[12154]: WARNING nova.scheduler.client.report [None req-f266a0ff-2840-413d-9877-4500e61512f5 tempest-ServersNegativeTestJSON-477704048 tempest-ServersNegativeTestJSON-477704048] Failed to save allocation for 6665f00a-dcf1-4286-b075-d7dcd7c37487. Got HTTP 409: {"errors": [{"status": 409, "request_id": "req-c9ba6cbd-3b6e-4e5d-b550-9588be8a49d2", "code": "p
21:20:03 mriedem ment.concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "title": "Conflict"}]}
21:20:12 mriedem consumer generation conflict - expected null but got 1
21:20:20 mriedem yup - so new consumer but we're passing a generation of 1
21:20:25 mriedem from the old, now deleted consumer
21:21:07 dansmith cool
21:21:26 mriedem so,
21:21:42 dansmith I wish there was something better to communicate that, but any time we get "expected null" in that case, we should be able to re-try but as a non-move sort of thing
21:21:43 mriedem we can paper over this by deleting the allocations before marking the instance as shelved offloaded, but that's whack-a-moley
21:21:49 dansmith yeah
21:22:00 mriedem right we need to retry from claim_resources but i'm not sure what's the best way to do that
21:22:03 dansmith and like I said, I think it's not really any better, it just changes the problem
21:22:09 dansmith yeah
21:22:22 mriedem if we do retry that method, the next get for allocations will see there are none and we should be good
21:22:31 dansmith right
21:22:40 mriedem b/c we'll pass consumer_generation=None
21:22:59 mriedem i think i know what we can do
21:23:07 mriedem if we hit
21:23:07 mriedem if 'consumer generation conflict' in err['detail']:
21:23:13 mriedem we get the allocs again, and if empty,
21:23:15 mriedem we retry
21:23:17 mriedem easy peasy
21:23:22 mriedem it's a double get but meh?
21:23:29 dansmith yeah, it's just that it takes us an extra op,
21:23:34 dansmith when "expected null" should be enough
21:23:45 dansmith yeah
21:23:45 mriedem i can parse that out of the message if we want..
21:23:57 dansmith I know we can, it's just icky and unfortunate
21:23:59 mriedem with a TODO for more granular error codes later
21:24:02 dansmith like all the other cases in there
21:24:04 mriedem right
21:24:15 mriedem i haven't even started packing yet
21:24:24 mriedem laura is starting to check in on me every 30 minutes
21:24:31 mriedem "this is what i'm wearing all week! god!"
21:24:52 dansmith hah
21:25:35 mriedem plus my mother in law is here,
21:25:42 mriedem so lots of teenage angst memories coming back right now
21:25:51 mriedem the coffee and metallica doesn't hel[p
21:25:58 dansmith isn't that a good reason to pack and get out?
21:26:09 mriedem i've just holed up in my office
21:26:26 mriedem i'll crank out a patch for this bug and be off
21:45:39 openstackgerrit Jack Ding proposed openstack/nova master: Use virt.images.convert_image for qemu-img convert https://review.openstack.org/616692
21:52:35 fried_rice mriedem: Sanity check, please. The compute manager has a report client via the scheduler client, that's *not* the same as the report client the resource tracker has.
21:53:19 fried_rice which means my current SIGHUP doesn't do shit to the RT's cache
21:53:26 fried_rice I need to make the report client a singleton.

Earlier   Later