Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-12
15:02:26 gibi this check https://review.opendev.org/c/openstack/nova/+/944105/1/nova/compute/claims.py
15:02:41 gibi so we started to check why we did not hit this code before in our testing
15:03:02 gibi and it turns out because nova does not do a move claim during live migration if the VM is not NUMA aware
15:03:08 gibi which sounds very strange to me
15:04:49 gibi so I try to figure out if this is an old bug, or it is intentional
15:05:47 gibi the code read likes the condition around the move claim wanted to handle rolling upgrade but somehow but actually it does more than that in reality as it disables the move claim for non NUMA aware VMs
15:07:45 gibi dansmith: sure we can have a gmeet. I'm free until the top of the hour
15:07:57 dansmith meet.google.com/jad-dzbb-nkb
15:08:22 dansmith sean-k-mooney if you're free and interested ^
15:19:19 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing instance-action finish_time https://review.opendev.org/c/openstack/nova/+/928933
15:24:32 Uggla @gibi @dansmith interested by the chat, but I have noticed it too late. However I'd like to know your findings.
16:06:42 opendevreview ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106
16:06:43 opendevreview ribaudr proposed openstack/nova master: FUP Add a warning to make non-explicit live migration request debugging easier https://review.opendev.org/c/openstack/nova/+/944133
17:10:21 dansmith gibi: first quick stab at invalidating the cached RP shows that we immediately refresh it when we go to do the placement sync
17:10:34 dansmith so I think that's the ticket out of the problem
17:14:27 bauzas folks, I'm just moving to thelounger IRC bouncer and client after 12 years of ZNC, but I continue to run ZNC with the bauzas_ nick
17:14:47 bauzas just in case you don't get a reply from me, ping both of us :)
17:16:44 gibi Uggla: we discovered that there is a provider tree cache in the resouce tracker that causes a race condition if an OTU device is allocated, reserverd, deallocated, externally unreserved, and allocated again. It is a cache invalidation problem between placement and nova, and Dan is now adding a way to invalidate the caches explicitly when the OTU device is reseved by nova
17:18:38 gibi Uggla: we also touched on the non NUMA aware live migration move claim question as well. We think the move claim was added to that code path when NUMA aware live migration was added, and before that only the periodic update_available_resources healed the allocation in the resouce tracker
17:18:52 gibi so probably that is what happens today for non NUMA aware VMs when live migrated
17:20:43 gibi this gives me enough certanty that I can comment on you patch removing the PCI check from the live migration claim. That check happens in the conductor in any case so I think that was just a dupicated check. I will comment on the patch tomorrow
17:20:51 gibi cc artom ^^
17:25:24 Uggla @gibi, may I rephrase to ensure I got it: non numa vm will not claimed resources explicitly they are spawned and then a preriodic adjusts the allocations in the RT ?
17:25:49 artom > move claim was added to that code path when NUMA aware live migration was added, and before that only the periodic update_available_resources healed the allocation in the resouce tracker
17:25:51 artom Correct
17:26:57 Uggla so there is a short amount of time where the RT allocations are not the correct one ?
17:34:01 gibi yeah that is my current understanding
17:34:49 gibi we are probably not hit by this in real life as most of the non NUMA aware VM resources are tracked in placement and the allocation there is correct and independent form the move claim
17:35:38 gibi also the PCI claim happens outside of the move claim for live migration therefore that always happen correctly even for non NUMA aware VMs and for non PCI in Placement enabled
17:36:24 gibi this is a bit of a mess but a well behaving mess :)
17:42:06 Uggla in French we have an expression for this: "C'est tombé en marche". :)
17:50:35 sean-k-mooney gibi: i belive the check in the move claim for pci devicve is for cold migration
17:52:23 sean-k-mooney oh maybe im wrong
17:53:46 sean-k-mooney ok no it was checking but it was inherited form the base claim
17:54:35 sean-k-mooney and it was modifed for the live migration case
17:55:21 sean-k-mooney Uggla: gibi so ya we can likely drop https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L209-L233 now
17:55:42 sean-k-mooney well
17:56:26 sean-k-mooney if we rely on the conducotr to validate that all teh request from the flavor are live migrtable
17:56:33 Uggla sean-k-mooney, it is what I have done. But we were a bit afraid to not hit that code before.
17:57:11 sean-k-mooney you have ad a patch up for that for a long time no?
17:57:22 sean-k-mooney i tought i looked at a patch for this like 2 weeks ago
17:57:22 Uggla sean-k-mooney, fyi https://review.opendev.org/c/openstack/nova/+/944105
17:58:15 Uggla sean-k-mooney, no created it only today.
17:58:31 sean-k-mooney i guess im thinking fo the conducotr change
17:59:15 sean-k-mooney this https://review.opendev.org/c/openstack/nova/+/942146/7/nova/conductor/tasks/live_migrate.py#244
17:59:42 Uggla oh yes sorry.
17:59:52 sean-k-mooney how were you teting this without that change in the move claims?
18:01:26 sean-k-mooney oh
18:01:27 Uggla This was tested with a unit tests and the fn tests were ok because all the live migration tests were done without numa VM.
18:01:45 sean-k-mooney no but you tested this manually
18:01:49 sean-k-mooney with real hardware
18:02:05 sean-k-mooney oh....
18:02:14 Uggla not real hardware was fine too, because I use a non numa vm.
18:02:20 sean-k-mooney that why gibi was askign about numa
18:02:39 sean-k-mooney that move claim code was orgianl ment to work for non numa instance too...
18:02:52 sean-k-mooney i guess that was just alway broken then
18:02:59 Uggla yep, I discovered that this morning, then "pass" the ball to gibi.
18:03:15 sean-k-mooney got it
18:03:38 sean-k-mooney for the record i still hate everyting about claims and still want to burn them with fire
18:03:39 Uggla because I could not understand why it was working fine without numa and failed with numa.
18:05:16 sean-k-mooney you should see this fail in the funtional test if you enabled hugepages or cpu pinning
18:05:22 sean-k-mooney we can add test coverage for that
18:05:27 Uggla @sean-k-mooney, ha ha, openstack claims game of thrones.
18:05:57 Uggla I saw the issue with the new fn tests this morning, because I enabled cpu pinning.
18:06:08 sean-k-mooney i strongly objected to extending claims when we added numa live migration which is why i didnt use them when adding sriov live migration in the same cycle
18:06:22 sean-k-mooney Uggla: oh ok cool
18:09:34 Uggla sean-k-mooney, in fact I leverage "your" test_live_migrate_server_with_neutron to do a mix test with port request and flavor based request.
18:10:26 bauzas cores, I need your help for paperwork changes RC1 : https://review.opendev.org/c/openstack/nova/+/943948
18:10:36 bauzas ditto for https://review.opendev.org/c/openstack/nova/+/943952
18:10:49 sean-k-mooney Uggla: shoudl we be keepign this https://review.opendev.org/c/openstack/nova/+/944105/1/nova/tests/unit/compute/test_claims.py#b375 but updating it to show it nolonger raise
18:11:06 sean-k-mooney bauzas: im going to finish soon but ill take a look now
18:11:29 bauzas also, I'd like cores to review our cycle highlights https://review.opendev.org/c/openstack/releases/+/943801
18:11:52 bauzas and nothing urgent but I'd appreciate https://review.opendev.org/c/openstack/nova-specs/+/944019
18:13:20 bauzas and we need a second core for https://review.opendev.org/c/openstack/nova/+/942871 (libvirt/qemu bump)
18:13:22 Uggla bauzas, reminder about 942871: Bump MIN_{LIBVIRT,QEMU} for "Epoxy" | https://review.opendev.org/c/openstack/nova/+/942871 to not forget it.
18:13:40 bauzas Uggla: jinxed !
18:13:55 sean-k-mooney bauzas: you havnt updated https://review.opendev.org/c/openstack/releases/+/943801/2/deliverables/epoxy/nova.yaml based on what already there
18:14:19 bauzas sean-k-mooney: that's litterally my next action item
18:14:26 sean-k-mooney ah
18:14:27 bauzas even at 7:15pm
18:15:34 sean-k-mooney bauzas: i have reviewd everything except the libvirt bump ill take a look at that now
18:15:48 bauzas <3
18:16:05 bauzas I'll write the prelude patch tomorrow morning
18:17:53 Uggla bauzas, keep a little bandwith tomorrow morning to review the fup for Vf live migration.
18:18:02 sean-k-mooney so the bump patch is split in two right, the first does the bump and then https://review.opendev.org/c/openstack/nova/+/942872/2 is a follow up to drop the supprot for the older versions
18:18:21 sean-k-mooney so we can merge the bump and come back to the WIP patch next cycle yes
18:19:01 Uggla sean-k-mooney, yes.
18:19:36 sean-k-mooney ok i have appoved that as well and rechecked it for the timeouts
18:21:05 Uggla sean-k-mooney, I was running out of time for doing the cleanup this cycle.
18:21:34 sean-k-mooney thats ok. once we have signaled the new min version and updated the constants
18:21:43 sean-k-mooney the agent will check for the new min verison and report it
18:21:55 sean-k-mooney so we shoudl be fine
18:22:24 Uggla I hope too
18:26:35 opendevreview Merged openstack/nova-specs master: Move Epoxy implemented specs https://review.opendev.org/c/openstack/nova-specs/+/944019
18:47:11 bauzas updated the cycle highlights : https://review.opendev.org/c/openstack/releases/+/943801
18:49:22 opendevreview Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816
18:49:22 opendevreview Dan Smith proposed openstack/nova master: Extend invalidate_rp to only invalidate cache https://review.opendev.org/c/openstack/nova/+/944148
18:49:23 opendevreview Dan Smith proposed openstack/nova master: Invalidate PCI-in-placement cached RPs during claim https://review.opendev.org/c/openstack/nova/+/944149
20:42:11 opendevreview ribaudr proposed openstack/nova master: FUP Update pci-passthrough and virtual-gpu documentation https://review.opendev.org/c/openstack/nova/+/944153
21:02:09 opendevreview Steve Baker proposed openstack/nova master: Add VNC console support for the Ironic driver https://review.opendev.org/c/openstack/nova/+/942528

Earlier   Later