Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-12
13:58:14 gibi (except the PCI one outside of the Claim obj)
13:59:05 artom I agree with you that it's strange
13:59:21 artom But it's been so long (Train!) that I've lost most context for the decisions that were made
13:59:49 gibi yeah I understand you don't have a context. I will try to charachterize the my feelings more
14:00:12 artom I talk about it here: https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/numa-aware-live-migration.html#resource-claims
14:00:15 gibi best would be to find a way how this could break our resource tracking and the move from there
14:00:32 gibi I need to jump on a call, I will be back in 30 mins (hopefully)
14:01:14 artom OK
14:02:28 gibi that section reads for me like we wanted to do the claim regardless of NUMA awareness
14:17:30 artom Yeah, I dunno anymore :(
14:17:53 artom Maybe let's come at it from a different angle - what led you to notice this, and what are you trying to do?
14:20:46 dansmith ah, the old "let me answer a different question" deflection :D
14:21:52 artom Hah
14:22:00 artom I'd _like_ to answer the original question
14:22:03 artom I just don't remember
14:24:23 dansmith I recognize the strategy well
14:52:12 dansmith gibi: so, there's a race happening, which maybe/probably affects other stuff.. any chance you have a few minutes to gmeet for me to describe and see if you're aware already?
14:53:18 dansmith specifically a provider_tree update race
14:55:46 sean-k-mooney i dont think we update the provier tree resereved value in nova for anything else, although i know neutron does for l3 routed networks
14:56:14 sean-k-mooney so they might have seen whatever the race is with that
14:58:00 dansmith it's not a problem specific to reserved, that's just how I'm seeing it
14:58:31 dansmith although perhaps it doesn't get exposed because of the nature of the external update in this case, although that might mean things like neutron/cinder do apply
14:59:08 sean-k-mooney are you getting a generation conflcit?
15:00:15 dansmith no
15:00:30 dansmith sean-k-mooney: there was no discussion of this before, I just discovered this while testing locally
15:01:28 dansmith gibi: there's nothing to read back except up to my ask for a gmeet consult :D
15:01:37 gibi artom: so this come up when testing the vgpu live migration series
15:01:56 gibi until now all our testing was done with non NUMA aware VMs
15:02:16 gibi and then Uggla did the first NUMA aware testing and hit by the check in the move claim that was not hit before
15:02:26 gibi this check https://review.opendev.org/c/openstack/nova/+/944105/1/nova/compute/claims.py
15:02:41 gibi so we started to check why we did not hit this code before in our testing
15:03:02 gibi and it turns out because nova does not do a move claim during live migration if the VM is not NUMA aware
15:03:08 gibi which sounds very strange to me
15:04:49 gibi so I try to figure out if this is an old bug, or it is intentional
15:05:47 gibi the code read likes the condition around the move claim wanted to handle rolling upgrade but somehow but actually it does more than that in reality as it disables the move claim for non NUMA aware VMs
15:07:45 gibi dansmith: sure we can have a gmeet. I'm free until the top of the hour
15:07:57 dansmith meet.google.com/jad-dzbb-nkb
15:08:22 dansmith sean-k-mooney if you're free and interested ^
15:19:19 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing instance-action finish_time https://review.opendev.org/c/openstack/nova/+/928933
15:24:32 Uggla @gibi @dansmith interested by the chat, but I have noticed it too late. However I'd like to know your findings.
16:06:42 opendevreview ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106
16:06:43 opendevreview ribaudr proposed openstack/nova master: FUP Add a warning to make non-explicit live migration request debugging easier https://review.opendev.org/c/openstack/nova/+/944133
17:10:21 dansmith gibi: first quick stab at invalidating the cached RP shows that we immediately refresh it when we go to do the placement sync
17:10:34 dansmith so I think that's the ticket out of the problem
17:14:27 bauzas folks, I'm just moving to thelounger IRC bouncer and client after 12 years of ZNC, but I continue to run ZNC with the bauzas_ nick
17:14:47 bauzas just in case you don't get a reply from me, ping both of us :)
17:16:44 gibi Uggla: we discovered that there is a provider tree cache in the resouce tracker that causes a race condition if an OTU device is allocated, reserverd, deallocated, externally unreserved, and allocated again. It is a cache invalidation problem between placement and nova, and Dan is now adding a way to invalidate the caches explicitly when the OTU device is reseved by nova
17:18:38 gibi Uggla: we also touched on the non NUMA aware live migration move claim question as well. We think the move claim was added to that code path when NUMA aware live migration was added, and before that only the periodic update_available_resources healed the allocation in the resouce tracker
17:18:52 gibi so probably that is what happens today for non NUMA aware VMs when live migrated
17:20:43 gibi this gives me enough certanty that I can comment on you patch removing the PCI check from the live migration claim. That check happens in the conductor in any case so I think that was just a dupicated check. I will comment on the patch tomorrow
17:20:51 gibi cc artom ^^
17:25:24 Uggla @gibi, may I rephrase to ensure I got it: non numa vm will not claimed resources explicitly they are spawned and then a preriodic adjusts the allocations in the RT ?
17:25:49 artom > move claim was added to that code path when NUMA aware live migration was added, and before that only the periodic update_available_resources healed the allocation in the resouce tracker
17:25:51 artom Correct
17:26:57 Uggla so there is a short amount of time where the RT allocations are not the correct one ?
17:34:01 gibi yeah that is my current understanding
17:34:49 gibi we are probably not hit by this in real life as most of the non NUMA aware VM resources are tracked in placement and the allocation there is correct and independent form the move claim
17:35:38 gibi also the PCI claim happens outside of the move claim for live migration therefore that always happen correctly even for non NUMA aware VMs and for non PCI in Placement enabled
17:36:24 gibi this is a bit of a mess but a well behaving mess :)
17:42:06 Uggla in French we have an expression for this: "C'est tombé en marche". :)
17:50:35 sean-k-mooney gibi: i belive the check in the move claim for pci devicve is for cold migration
17:52:23 sean-k-mooney oh maybe im wrong
17:53:46 sean-k-mooney ok no it was checking but it was inherited form the base claim
17:54:35 sean-k-mooney and it was modifed for the live migration case
17:55:21 sean-k-mooney Uggla: gibi so ya we can likely drop https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L209-L233 now
17:55:42 sean-k-mooney well
17:56:26 sean-k-mooney if we rely on the conducotr to validate that all teh request from the flavor are live migrtable
17:56:33 Uggla sean-k-mooney, it is what I have done. But we were a bit afraid to not hit that code before.
17:57:11 sean-k-mooney you have ad a patch up for that for a long time no?
17:57:22 Uggla sean-k-mooney, fyi https://review.opendev.org/c/openstack/nova/+/944105
17:57:22 sean-k-mooney i tought i looked at a patch for this like 2 weeks ago
17:58:15 Uggla sean-k-mooney, no created it only today.
17:58:31 sean-k-mooney i guess im thinking fo the conducotr change
17:59:15 sean-k-mooney this https://review.opendev.org/c/openstack/nova/+/942146/7/nova/conductor/tasks/live_migrate.py#244
17:59:42 Uggla oh yes sorry.
17:59:52 sean-k-mooney how were you teting this without that change in the move claims?
18:01:26 sean-k-mooney oh
18:01:27 Uggla This was tested with a unit tests and the fn tests were ok because all the live migration tests were done without numa VM.
18:01:45 sean-k-mooney no but you tested this manually
18:01:49 sean-k-mooney with real hardware
18:02:05 sean-k-mooney oh....
18:02:14 Uggla not real hardware was fine too, because I use a non numa vm.
18:02:20 sean-k-mooney that why gibi was askign about numa
18:02:39 sean-k-mooney that move claim code was orgianl ment to work for non numa instance too...
18:02:52 sean-k-mooney i guess that was just alway broken then
18:02:59 Uggla yep, I discovered that this morning, then "pass" the ball to gibi.
18:03:15 sean-k-mooney got it
18:03:38 sean-k-mooney for the record i still hate everyting about claims and still want to burn them with fire
18:03:39 Uggla because I could not understand why it was working fine without numa and failed with numa.
18:05:16 sean-k-mooney you should see this fail in the funtional test if you enabled hugepages or cpu pinning
18:05:22 sean-k-mooney we can add test coverage for that
18:05:27 Uggla @sean-k-mooney, ha ha, openstack claims game of thrones.
18:05:57 Uggla I saw the issue with the new fn tests this morning, because I enabled cpu pinning.
18:06:08 sean-k-mooney i strongly objected to extending claims when we added numa live migration which is why i didnt use them when adding sriov live migration in the same cycle
18:06:22 sean-k-mooney Uggla: oh ok cool
18:09:34 Uggla sean-k-mooney, in fact I leverage "your" test_live_migrate_server_with_neutron to do a mix test with port request and flavor based request.
18:10:26 bauzas cores, I need your help for paperwork changes RC1 : https://review.opendev.org/c/openstack/nova/+/943948
18:10:36 bauzas ditto for https://review.opendev.org/c/openstack/nova/+/943952
18:10:49 sean-k-mooney Uggla: shoudl we be keepign this https://review.opendev.org/c/openstack/nova/+/944105/1/nova/tests/unit/compute/test_claims.py#b375 but updating it to show it nolonger raise
18:11:06 sean-k-mooney bauzas: im going to finish soon but ill take a look now

Earlier   Later