Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-18
15:46:53 mriedem if we had consumer types i could determine from move_allocations if the instance still exists or not...
15:47:13 mriedem or just pass the types into move_allocations but that kind of sucks
15:48:12 efried mriedem: don't we still have a race in that situation?
15:48:47 mriedem as in when i refresh the instance is still exists but is gone by the time move_allocations does the POST?
15:49:03 efried yeah
15:49:18 efried Small window -- but the window you're closing is pretty durn small too.
15:49:34 mriedem i guess that's possible, but i i'm not sure how to detect that when consumers in placement are ephemeral
15:50:39 mriedem meaning when we get the consumer allocs we pass the generation if we have one otherwise we just pass None if it's a new consumer
15:50:47 efried iow you need to know *before* you decide what to do in placement?
15:51:15 efried Instead of refreshing the instance, you could GET its allocations from placement.
15:51:39 efried then the window is n/a because when you push back if the generation doesn't match you'll blow up.
15:51:44 mriedem which is what move_allocations does
15:52:48 efried mriedem: let me stare at it a bit after my meeting here.
15:52:59 efried (by which time you'll have probably figured it out anyway)
15:53:11 mriedem dealing with a head cold so my brain isn't really functioning
16:16:53 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141
16:16:54 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query https://review.opendev.org/689537
17:09:21 efried mriedem: when an instance is in CONFIRM_RESIZE, is the migration allocation still on the dest?
17:15:07 eandersson Would enabling randomize_allocation_candidates help with race conditions (RescheduleException)?
17:15:50 eandersson We heavily favor stacking, which tends to cause race conditions when deploying a lot of small flavors.
17:16:16 eandersson Especially bad when deploying 1k+ VMs
17:18:53 efried eandersson: are you deploying them with a single command (max_instances=N or whatever that thing is) or multiple concurrent/quick operations?
17:20:09 eandersson multiple concurrent using terraform or ansible
17:20:16 eandersson if I do a single command it works fine
17:20:49 dansmith efried: the migration one is yeah
17:20:52 eandersson we have a lot of custom weighers and filters that probably make things.. slower (more likely to end in a race condition)
17:20:56 dansmith efried: the instance one is on the dest
17:21:10 dansmith efried: sorry, the migration allocation is on the _source_, the instance is on the _dest_
17:21:19 dansmith assume that's what you meant
17:21:45 eandersson We used host_subset_size and max_attempts to counter it
17:34:12 efried mriedem: Correct me if I'm wrong, move_allocations needs to deal with these scenarios:
17:34:12 efried - (live/cold) migrate execute: source is instance, target is migration.
17:34:12 efried - revert, including from a failed migration: source is migration, target is instance.
17:34:12 efried - In either case, if instance disappears, we want the operation fail, but that's something the caller needs to be aware of, not move_allocations itself.
17:34:12 efried - If an oob operation updates the instance's allocs (which only applies to the execute case; I assume you can't resize an instance that's in CONFIRM_RESIZE or ERROR, right?) we also want the migration to fail.
17:34:12 efried However, just like the delete thing that started you down this rathole, we can't close that window effectively while doing the GET from within move_allocations
17:34:13 efried So here's what I think needs to happen:
17:34:13 efried - move_allocations should always use 'allocations': {} to delete, never DELETE.
17:34:13 efried - move_allocations should *not retry on a consumer 409*; it should raise AllocationMoveFailed. It *should* retry on a *provider* 409 though. (This is a little complicated, and we have seen this issue elsewhere -- gibi L@@K.)
17:34:14 efried - Eventually (probably not now) the calling flows should be responsible for pulling the instance consumer (at least the generation, if not the whole allocation record) *early*, before entering the "critical path", and funneling that information into move_allocations. This is the only way we're going to effectively detect the race.
17:34:15 efried TL;DR the only thing that should change right now is: don't retry on consumer 409.
17:35:38 efried eandersson: randomize sounds like it would help there, yes.
17:36:41 efried eandersson: when you said you favor "stacking", did you mean "packing" (as opposed to "spread")?
17:37:47 efried If you're doing that ^ via your weighers/filters, then randomize probably won't do you a lot of good.
17:38:16 efried because your weigher will trawl down the list, random or not, and pick the fullest host anyway.
17:38:42 melwitt eandersson: what version are you running again? as of claims in placement, the rescheduling due to racing parallel requests shouldn't be a thing
17:39:08 efried melwitt: he's using separate parallel processes
17:39:15 melwitt I know
17:39:29 efried so each is doing a GET /a_c at the same time, getting the same answers, and all thinking they can hit the same host at the same time.
17:39:49 efried even with three retries, that'll race easily.
17:40:18 melwitt the whole point of claims in placement was to stop that though, that was the "multiple schedulers" problem of old. am I missing something?
17:40:35 efried more likely I am
17:41:36 eandersson efried, correct, packing
17:41:49 efried eandersson: If you did randomize and then fiddled around with a small ?limit= you might be able to reduce a little bit. But none of this sounds like a really good approach.
17:42:00 eandersson we have a lot of game servers that, and our goal is to fit as many game servers as possible
17:42:10 eandersson game servers take up one numa
17:42:19 eandersson but usually there are 2-4 vcpus left
17:42:51 eandersson so our filter/weighers tries to put small vms on top of that
17:43:03 eandersson without taking up enough to prevent a "game server" from fitting
17:43:13 eandersson e.g. if small vms take up 6 cores of a numa, no large server could fit
17:43:25 efried are the races happening on "small vm" or "game server"?
17:43:30 eandersson small vms only
17:43:42 efried well, you said "numa", which could be the crux of the problem.
17:43:48 efried placement doesn't know about numa yet.
17:43:51 eandersson because I think what is happening is that we have 2-3 computes with the "perfect fit" so they all try to land at those
17:43:55 melwitt placement claims were added in pike https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/placement-claims.html
17:44:01 eandersson this is rocky btw
17:44:22 melwitt oh, numa isn't checked until after it lands on the compute, so that's why you're hitting this
17:44:23 melwitt yeah
17:44:27 efried So if you have 1vcpu in each of two numas and you ask for 2VCPU, placement will give you what it thinks is a viable candidate
17:44:31 melwitt (as efried said earlier)
17:44:32 efried but the numa topo filter will punt it.
17:45:05 efried eandersson: does your "small vm" have a numa topo specified?
17:45:11 eandersson yea
17:45:16 efried does it need to? :P
17:45:34 efried if you allow it to float across numas, I suspect your race will go away.
17:45:38 eandersson I would assume so, but not 100% sure
17:46:28 eandersson If just ranzomizing the results a bit fixes it I probably prefer that route
17:46:39 eandersson but if that isn't enough I could look into doing that
17:47:48 melwitt fwiw I think you're already doing all you can to mitigate, by increasing the subset size and increasing retries. there's another option that could help, https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts
17:48:07 melwitt (assuming you can't change anything about the numa)
17:48:35 eandersson interesting, didn't know about that setting
17:48:50 melwitt yeah, that's the only other one involved in all this that you haven't looked at already
17:49:19 efried eandersson: if the problem is numa as described, randomizing won't help at all.
17:49:35 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query https://review.opendev.org/689537
17:49:36 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141
17:49:37 eandersson I see - I think what melwitt just linked might be the issue
17:49:46 melwitt eandersson: the randomize_allocation_candidates won't help you I don't think because it returns things in a deterministic order (so as not to necessarily break scheduling stacking)
17:50:05 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query https://review.opendev.org/689537
17:50:06 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141
17:51:41 efried melwitt: "it returns things" <== "it" being the filter?
17:51:49 melwitt no, placement
17:52:09 efried with randomize_allocation_candidates placement does *not* return things in deterministic order.
17:52:36 melwitt oh, lol, sorry I read the help backwards
17:53:20 efried yeah, with it *off* the order is deterministic (but inscrutable, based on natural db order of some nonobvious kind)
17:53:45 melwitt right
17:54:49 melwitt eandersson: ^ ignore what I said earlier about randomize_allocation_candidates
17:54:57 eandersson randomize_allocation_candidates = good?
17:56:35 efried eandersson: if the problem is the numa thing, randomize_allocation_candidates = irrelevant.
17:57:20 melwitt eandersson: what efried said. it won't help you because your problem is the numa fitment. I was just correcting what I said about it being a deterministic order, it would be the opposite of that, a random order

Earlier   Later