Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-03-19
17:56:42 sean-k-mooney related to our converation about non numa instances
17:56:49 dansmith sean-k-mooney: I really don't know currently, but I'm pretty sure that was one of the early limitations.. we could ask for inclusion but not exclusion
17:57:27 sean-k-mooney ack i think its still that way. i belive we coudl have a new filter that does the following check
17:58:26 sean-k-mooney for a given instance look at the guest falvor/image to see if it is requesting numa aware memory i.e. hw:mem_page_size
17:58:54 sean-k-mooney if it is not check the allcoation candates for the host to see if it has either the file backed memory trait or an inventory of PCPU
17:59:02 sean-k-mooney if it has either reject the host
18:00:05 sean-k-mooney that would allow operators to opt in/out by not using the filter
18:00:39 sean-k-mooney we can add it to the default filters to enforce the correct behavior.
18:00:54 dansmith well, that feels more formal of a feature opt-out than a workaround named I_like_to_oom_my_instances=True
18:01:12 dansmith but it would mean an abort in scheduling instead of compute, which would be better
18:02:07 sean-k-mooney well we cloud perhaps not make it a filter but a config option that woudl work like a filter
18:02:19 sean-k-mooney i.e. have a workaround to remove this mandatory filter
18:02:21 dansmith sean-k-mooney: btw, I am still interested in the answer to my question here: https://review.opendev.org/c/openstack/nova/+/877773/9/nova/virt/libvirt/migration.py
18:02:44 dansmith because it seems like it should always fail and I don't know why it's not
18:03:41 sean-k-mooney reading that now ill need to check but if i rememebr correctly we pop the value form the dict before it ever get there
18:04:00 dansmith that's a bad solution IMHO if so because it would break computes that don't have this code yet,
18:04:05 dansmith but I also couldn't find any such pop
18:04:14 sean-k-mooney your asertion is calling int(a string containing a list of ints)
18:04:35 dansmith sean-k-mooney: no it's calling int() on the fake key
18:04:36 sean-k-mooney should always explode so either we are not hitting that path in the test or there is somethign else going on here
18:04:58 dansmith that dict is supposed to be keys of '1', '2', etc and so it int()'s the keys,
18:05:08 dansmith but now there's a key of "dst_cpu_whatever" in the dict
18:05:20 dansmith and I don't know how it's not failing to int() that when it iterates the items
18:05:35 dansmith and I'm wondering if it's because we don't have test coverage that was hitting that code path with the three conditions being true
18:07:58 sean-k-mooney oh right "guest_id, host_ids in info.cpu_pins.items()" so guest_id would be the key
18:08:04 dansmith right
18:08:09 dansmith so I don't know how that's not breaking
19:23:13 sean-k-mooney artom: https://review.opendev.org/c/openstack/nova/+/631053 and https://bugs.launchpad.net/nova/+bug/1811886 are related to our converstaion about the issue relating to not requesting a page size.
19:23:52 sean-k-mooney its not the example i was giving when we talks but its a similar equally wrong case.
19:42:02 opendevreview Merged openstack/nova master: Remove nova.wsgi module https://review.opendev.org/c/openstack/nova/+/902686
19:42:13 opendevreview Merged openstack/nova master: Add new nova.wsgi module https://review.opendev.org/c/openstack/nova/+/902687
19:48:25 sean-k-mooney dansmith: i looked brifly but i dont know why that is not brekaing things but my best guess is we are somehow mocking this and we dont have cpu_shared_set defiend in any of the jobs
19:49:05 sean-k-mooney there is no reason why we cant have cpu share set in one of our live migration jobs so i might try and enable that
19:50:27 sean-k-mooney i.e. 0-5 on one node and 2-7 on the other
19:51:00 sean-k-mooney that pretty trivial to do so ill add a patch on top or renes serise to do that
19:52:13 sean-k-mooney in theory rene's code should be runnign even if that is not devied but we should be able to test this in the gate anyway
19:59:42 opendevreview Stephen Finucane proposed openstack/nova master: Make overcommit check for pinned instance pagesize aware https://review.opendev.org/c/openstack/nova/+/631053
20:01:56 dansmith sean-k-mooney: yeah, I'm unsure either, but I'm positive that means we're just not running the code we think we're running
20:02:02 opendevreview ribaudr proposed openstack/nova master: Fix: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/903706
20:02:02 opendevreview ribaudr proposed openstack/nova master: Fix [backport]: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/877773
20:07:51 sean-k-mooney dansmith: my guess right now is we are never populating the numa toplogy info in the migrate_data
20:08:18 dansmith sean-k-mooney: you mean not running a test that causes it to be populated right?
20:08:46 dansmith I mean...not running a test _case_ where it would be populated and that code path exercised
20:10:40 sean-k-mooney no i ment not seting the object that had the new data at all but we appear to be
20:10:54 artom sean-k-mooney, dansmith, hrmm, I wonder if we're fine because... let my type this out
20:11:08 artom The NUMA live migration code kicks in only if the instance has a NUMA topology
20:11:39 artom Uggla's patch _always_ adds that hidden item in the dst_numa_info
20:12:19 sean-k-mooney so it alwasy uses the numa migration code?
20:12:42 artom Hold on
20:13:50 artom No, I thought I had it...
20:16:11 sean-k-mooney the fact we have to have this converstaion and its not obvious how its working after a few of us lokking at it is a code smell to me in either case
20:16:31 sean-k-mooney the code might be correct but its to complex to debug
20:18:31 artom So to trigger it, all that should be required in live migration a NUMA VM to a dest with cpu_shared_set, right?
20:19:10 artom That would cause Uggla's patch to stash the cpu_shared_set in the "hidden" key in dst_numa_info, and *should* cause the update XML code to trip over and explode?
20:19:26 artom We've _definitely_ done that ^^ in whitebox, and nothing has exploded
20:19:42 sean-k-mooney artom: no
20:19:56 sean-k-mooney to trigger the bug yuou need to migate a non numa vm
20:20:06 sean-k-mooney oh
20:20:17 sean-k-mooney you mena the the driver issue
20:20:37 artom Yeah, the thing that we're trying to understand how it even works
20:20:39 sean-k-mooney yes if the loop over the keys was broken then you woudl have to migrate a pinned instance
20:22:06 sean-k-mooney right but since this is alwasy set https://review.opendev.org/c/openstack/nova/+/877773/10/nova/virt/libvirt/driver.py#10028
20:22:32 sean-k-mooney any numa instance should triger this code path
20:22:38 artom Exactly
20:23:05 artom I'm expecting migrate_data.dst_numa_info.cpu_pins["dst_cpu_shared_set_info"] to always be there
20:25:05 artom Ohhhhh
20:25:57 artom We just clobber the entire object for NUMA live migration: https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L10106
20:26:22 sean-k-mooney but again numa instance
20:26:24 artom And that ^^ happens later in the flow
20:26:32 sean-k-mooney sorry that was in my buffer
20:26:48 sean-k-mooney ya that would make sense
20:27:08 sean-k-mooney as that is where it woudl normally be created
20:27:25 artom That's after we make the claim on the dest, and Uggla's code was adding its thing earlier in check_can_live_migrate_destination
20:27:58 artom So it just gets overwritten
20:28:33 sean-k-mooney is it in check_can_live_migrate_destination i tought it was in pre_live_migration (at dest)
20:28:44 dansmith but as sean-k-mooney said now and I said earlier, this is hiding demons in the details on so many levels, especially since this transforms to a real object property in the next patch
20:29:14 dansmith artom: just a thought - are you running the tests you expect to pass now on the lower patch or only the stack? if the latter, then this code is "gone" by then right?
20:29:36 artom sean-k-mooney, https://review.opendev.org/c/openstack/nova/+/877773/10/nova/virt/libvirt/driver.py#9958
20:29:45 sean-k-mooney oh its in check_can_live_migrate_destination in the driver i was thinking of the conductor method with a similer name
20:29:58 artom dansmith, definitely the backportable one
20:30:06 dansmith okay just checking
20:30:12 artom dansmith, but I think we've figured it out :)
20:31:39 dansmith yeah, but I haven't figured out yet if you're thinking that we're not covering the case where we would trip over that code, or if they're mutually exclusive somehow (not that it makes it okay)
20:32:01 sean-k-mooney well at a minium this show we are missign a funtional test to demonstrate that the xml is updated properly between the srouce and destinatnio hosts
20:33:26 artom dansmith, they're not mutually exclusive, they definitely both run, and the NUMA live migration code runs _after_ the hack, so it clobbers the entire dst_numa_info object that was used as the hiding place for the "secret" cpu shared set info
20:34:18 dansmith I don't understand how that can be
20:34:26 artom dansmith lulz another call ^_^ ?
20:34:31 sean-k-mooney its because of the 3 conditions in the if
20:34:34 dansmith we have to put the dst info into the blob so it can be used to generate the xml right?
20:34:36 sean-k-mooney if ('cpu_pins' in info and
20:34:38 sean-k-mooney 'cell_pins' in info and
20:34:40 sean-k-mooney 'emulator_pins' in info):
20:34:52 sean-k-mooney in the non numa case only one of those 3 in checks is true
20:35:00 sean-k-mooney so we never enter the loop
20:35:08 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/877773/9/nova/virt/libvirt/migration.py#186
20:35:28 sean-k-mooney in the numa case the numa live migration code clobers the object that rene modifed
20:36:05 sean-k-mooney so it just happens to work becasue we get lucky and the case where its not clobpered we dont enter the loop
20:36:29 dansmith and to be clear, that's always the case or because we're missing a test scenario where we could get unlucky?
20:36:31 sean-k-mooney cell_pins and emulator_pins are not defiend in the non numa case
20:36:36 artom The case where it's not clobbered is when a VM doesn't have a NUMA topoogy

Earlier   Later