| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-18 | |||
| 17:20:56 | dansmith | efried: the instance one is on the dest | |
| 17:21:10 | dansmith | efried: sorry, the migration allocation is on the _source_, the instance is on the _dest_ | |
| 17:21:19 | dansmith | assume that's what you meant | |
| 17:21:45 | eandersson | We used host_subset_size and max_attempts to counter it | |
| 17:34:12 | efried | mriedem: Correct me if I'm wrong, move_allocations needs to deal with these scenarios: | |
| 17:34:12 | efried | - (live/cold) migrate execute: source is instance, target is migration. | |
| 17:34:12 | efried | - revert, including from a failed migration: source is migration, target is instance. | |
| 17:34:12 | efried | - In either case, if instance disappears, we want the operation fail, but that's something the caller needs to be aware of, not move_allocations itself. | |
| 17:34:12 | efried | - If an oob operation updates the instance's allocs (which only applies to the execute case; I assume you can't resize an instance that's in CONFIRM_RESIZE or ERROR, right?) we also want the migration to fail. | |
| 17:34:12 | efried | However, just like the delete thing that started you down this rathole, we can't close that window effectively while doing the GET from within move_allocations | |
| 17:34:13 | efried | So here's what I think needs to happen: | |
| 17:34:13 | efried | - move_allocations should always use 'allocations': {} to delete, never DELETE. | |
| 17:34:13 | efried | - move_allocations should *not retry on a consumer 409*; it should raise AllocationMoveFailed. It *should* retry on a *provider* 409 though. (This is a little complicated, and we have seen this issue elsewhere -- gibi L@@K.) | |
| 17:34:14 | efried | - Eventually (probably not now) the calling flows should be responsible for pulling the instance consumer (at least the generation, if not the whole allocation record) *early*, before entering the "critical path", and funneling that information into move_allocations. This is the only way we're going to effectively detect the race. | |
| 17:34:15 | efried | TL;DR the only thing that should change right now is: don't retry on consumer 409. | |
| 17:35:38 | efried | eandersson: randomize sounds like it would help there, yes. | |
| 17:36:41 | efried | eandersson: when you said you favor "stacking", did you mean "packing" (as opposed to "spread")? | |
| 17:37:47 | efried | If you're doing that ^ via your weighers/filters, then randomize probably won't do you a lot of good. | |
| 17:38:16 | efried | because your weigher will trawl down the list, random or not, and pick the fullest host anyway. | |
| 17:38:42 | melwitt | eandersson: what version are you running again? as of claims in placement, the rescheduling due to racing parallel requests shouldn't be a thing | |
| 17:39:08 | efried | melwitt: he's using separate parallel processes | |
| 17:39:15 | melwitt | I know | |
| 17:39:29 | efried | so each is doing a GET /a_c at the same time, getting the same answers, and all thinking they can hit the same host at the same time. | |
| 17:39:49 | efried | even with three retries, that'll race easily. | |
| 17:40:18 | melwitt | the whole point of claims in placement was to stop that though, that was the "multiple schedulers" problem of old. am I missing something? | |
| 17:40:35 | efried | more likely I am | |
| 17:41:36 | eandersson | efried, correct, packing | |
| 17:41:49 | efried | eandersson: If you did randomize and then fiddled around with a small ?limit= you might be able to reduce a little bit. But none of this sounds like a really good approach. | |
| 17:42:00 | eandersson | we have a lot of game servers that, and our goal is to fit as many game servers as possible | |
| 17:42:10 | eandersson | game servers take up one numa | |
| 17:42:19 | eandersson | but usually there are 2-4 vcpus left | |
| 17:42:51 | eandersson | so our filter/weighers tries to put small vms on top of that | |
| 17:43:03 | eandersson | without taking up enough to prevent a "game server" from fitting | |
| 17:43:13 | eandersson | e.g. if small vms take up 6 cores of a numa, no large server could fit | |
| 17:43:25 | efried | are the races happening on "small vm" or "game server"? | |
| 17:43:30 | eandersson | small vms only | |
| 17:43:42 | efried | well, you said "numa", which could be the crux of the problem. | |
| 17:43:48 | efried | placement doesn't know about numa yet. | |
| 17:43:51 | eandersson | because I think what is happening is that we have 2-3 computes with the "perfect fit" so they all try to land at those | |
| 17:43:55 | melwitt | placement claims were added in pike https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/placement-claims.html | |
| 17:44:01 | eandersson | this is rocky btw | |
| 17:44:22 | melwitt | oh, numa isn't checked until after it lands on the compute, so that's why you're hitting this | |
| 17:44:23 | melwitt | yeah | |
| 17:44:27 | efried | So if you have 1vcpu in each of two numas and you ask for 2VCPU, placement will give you what it thinks is a viable candidate | |
| 17:44:31 | melwitt | (as efried said earlier) | |
| 17:44:32 | efried | but the numa topo filter will punt it. | |
| 17:45:05 | efried | eandersson: does your "small vm" have a numa topo specified? | |
| 17:45:11 | eandersson | yea | |
| 17:45:16 | efried | does it need to? :P | |
| 17:45:34 | efried | if you allow it to float across numas, I suspect your race will go away. | |
| 17:45:38 | eandersson | I would assume so, but not 100% sure | |
| 17:46:28 | eandersson | If just ranzomizing the results a bit fixes it I probably prefer that route | |
| 17:46:39 | eandersson | but if that isn't enough I could look into doing that | |
| 17:47:48 | melwitt | fwiw I think you're already doing all you can to mitigate, by increasing the subset size and increasing retries. there's another option that could help, https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts | |
| 17:48:07 | melwitt | (assuming you can't change anything about the numa) | |
| 17:48:35 | eandersson | interesting, didn't know about that setting | |
| 17:48:50 | melwitt | yeah, that's the only other one involved in all this that you haven't looked at already | |
| 17:49:19 | efried | eandersson: if the problem is numa as described, randomizing won't help at all. | |
| 17:49:35 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query https://review.opendev.org/689537 | |
| 17:49:36 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141 | |
| 17:49:37 | eandersson | I see - I think what melwitt just linked might be the issue | |
| 17:49:46 | melwitt | eandersson: the randomize_allocation_candidates won't help you I don't think because it returns things in a deterministic order (so as not to necessarily break scheduling stacking) | |
| 17:50:05 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query https://review.opendev.org/689537 | |
| 17:50:06 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141 | |
| 17:51:41 | efried | melwitt: "it returns things" <== "it" being the filter? | |
| 17:51:49 | melwitt | no, placement | |
| 17:52:09 | efried | with randomize_allocation_candidates placement does *not* return things in deterministic order. | |
| 17:52:36 | melwitt | oh, lol, sorry I read the help backwards | |
| 17:53:20 | efried | yeah, with it *off* the order is deterministic (but inscrutable, based on natural db order of some nonobvious kind) | |
| 17:53:45 | melwitt | right | |
| 17:54:49 | melwitt | eandersson: ^ ignore what I said earlier about randomize_allocation_candidates | |
| 17:54:57 | eandersson | randomize_allocation_candidates = good? | |
| 17:56:35 | efried | eandersson: if the problem is the numa thing, randomize_allocation_candidates = irrelevant. | |
| 17:57:20 | melwitt | eandersson: what efried said. it won't help you because your problem is the numa fitment. I was just correcting what I said about it being a deterministic order, it would be the opposite of that, a random order | |
| 17:57:43 | eandersson | I see - I assumed so, but wanted to make sure | |
| 17:57:47 | melwitt | I read the help backwards and... I don't have much of an excuse | |
| 17:58:14 | efried | The only way it would be at all likely to help you is if $number_of_concurrent_deploys < $number_of_possible_hosts *and* you use a ?limit < $number_of_possible_hosts | |
| 17:58:55 | efried | but like I said, that's a pretty hacky thing to try to figure out | |
| 17:59:19 | melwitt | eandersson: the shuffle_best_same_weighed_hosts might help you, I think. numa stuff might be why it was added in the first place | |
| 17:59:35 | eandersson | Yea - sounds super promising. | |
| 18:02:02 | melwitt | +1 efried. eandersson: the randomize_allocation_candidates use case is for very large deployments where the default limit for returned candidates is < the total number of compute hosts in the deployment. think CERN with multiple cells | |
| 18:02:19 | eandersson | Just fyi we have ~1k nodes in each deployment | |
| 18:03:42 | melwitt | ack | |
| 18:04:54 | eandersson | per cell | |
| 18:06:07 | melwitt | oh, ok that is a lot | |
| 18:06:08 | eandersson | We use regions instead of cells at the moment, but might want to change that in the future | |
| 18:08:15 | melwitt | here's the background on the option https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/allocation-candidates-limit.html | |
| 18:08:57 | eandersson | We limit ourself to 1k nodes today because we don't know if neutron can handle it | |
| 18:09:20 | eandersson | (used to be nova + neutron, but doubt nova will have a problem today) | |
| 18:09:49 | eandersson | when we upgraded to rocky neutron used up like 500GB memory lol | |
| 18:10:02 | eandersson | each neutron process peaked at 8.2GB memory used | |
| 18:10:31 | melwitt | gotcha. belmiro from CERN would be the one to chat with about neutron scaling | |
| 18:10:39 | eandersson | We work with them a lot | |
| 18:10:47 | melwitt | I don't remember if they are doing something special re: neutron | |
| 18:10:49 | eandersson | They use one big Layer 2 network | |
| 18:10:50 | melwitt | ok cool | |
| 18:11:19 | melwitt | and that's... a lot of memory O.o | |
| 18:11:32 | eandersson | Yea it's nuts. I tend to ping the neutron channel but no one ever replies | |
| 18:11:38 | eandersson | Nova channel is way better :p | |
| 18:12:07 | melwitt | ah yeah, I think they might be mostly EU timezone | |