Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-12
08:26:48 gibi the unplug/plug live migration for neutron requested PCI probably works as that is not real live migration of the device, but the variant driver based will lead to very sad guest kernels
08:27:07 sean-k-mooney[m] yep
08:28:25 sean-k-mooney[m] so i know why they orginally did this which is they were trying to make differnt configurations of intels QAT device look the same\
08:28:39 sean-k-mooney[m] and allow you to use ask for QAT
08:28:47 sean-k-mooney[m] without careing which one you got
08:29:29 sean-k-mooney[m] that only works if you never have to deal with live migration
08:29:59 sean-k-mooney[m] so i think we need to disable this when live_migratable is set to true or you use pci_in_placment
08:30:15 sean-k-mooney[m] either should result in a config error on startup
08:31:20 sean-k-mooney[m] we can allow it for non llive migratbale device or when not using PCI in placement for backward compatibality
08:58:23 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing instance-action finish_time https://review.opendev.org/c/openstack/nova/+/928933
09:12:16 sean-k-mooney it is day 2 or alt world where my fundemental core belife that "their should only be 1 7:30 in a day and it should happen after noon" is being chllanged by the concept of 7 am...
09:13:27 sean-k-mooney its currntly 09:13 which is still an hour before im useually awake
11:43:09 opendevreview ribaudr proposed openstack/nova master: FUP improve comment accuracy and variable naming for tag removal https://review.opendev.org/c/openstack/nova/+/943124
11:43:10 opendevreview ribaudr proposed openstack/nova master: FUP Remove unnecessary PCI check https://review.opendev.org/c/openstack/nova/+/944105
11:43:10 opendevreview ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106
11:52:58 andrewbonney sean-k-mooney: in case you have time today for another look, https://review.opendev.org/c/openstack/nova/+/919961 seems fine in CI
12:13:32 sean-k-mooney andrewbonney: ill review it propelry shortly. if you need to respin it might be nice to add a fixes release note
12:13:40 sean-k-mooney but thats not strictly required
12:13:56 andrewbonney thanks
12:13:58 sean-k-mooney so ya ill take a look shortly
12:23:17 sean-k-mooney andrewbonney: +2 form me
12:24:02 sean-k-mooney andrewbonney: this proably wont merge for RC1 but it can merge and be backported as normal after
12:24:27 andrewbonney sure, just trying to get it ticked off before I head off for extended leave
12:24:50 sean-k-mooney ack
12:26:07 opendevreview Merged openstack/nova master: Fix serial console for ironic https://review.opendev.org/c/openstack/nova/+/942575
13:44:08 gibi folks, Why do we only use the move claim during live migration if the VM is NUMA aware? context https://github.com/openstack/nova/blob/f71a0a6204744b7774caf85c16f27964a4f851db/nova/compute/manager.py#L9057-L9066
13:45:06 gibi based on my local testing that src_supports_numa_live_migration check there is only true if the VM is NUMA aware
13:45:43 gibi so it seems like we doesn't do any claim for the instance during live migration if the instance is not NUMA aware (except claiming things outside of the Claim object like PCI)
13:48:43 Uggla @artom, can you have a look at gibi's question above.
13:50:01 artom gibi, I don't remember :(
13:50:11 artom But I'm pretty sure it's not _just_ NUMA-aware VMs
13:50:17 artom Other code added later started using that too
13:50:26 artom VPMEM I think?
13:51:04 artom And src_supports_numa_live_migration was there to handle the rolling upgrade case at the time of implementation
13:51:18 gibi if you look at the code linked above there is a check that looks for src_supports_numa_live_migration in the migrate data, that condition is false for me locally for a non NUMA aware VM
13:51:22 artom We can _probably_ remove it now, with the usual caveats around RPC
13:51:30 Uggla @artom, look at https://review.opendev.org/c/openstack/nova/+/944105 you left a comment that may remember you things.
13:52:48 artom Oh yeah, there was the SRIOV live migration thing as well
13:52:53 artom It's handled by a separate code path
13:53:06 Uggla yep below in the manager
13:53:12 gibi yeah the PCI stuff claimed differently, but what about the rest of the resources?
13:53:31 artom You mean non-Placement resources?
13:55:14 Uggla @gibi this is claimed at scheduler step before no ?
13:55:58 gibi artom: yea
13:56:19 gibi all the normal things we do during an instance claim or during a move claim in other ops
13:56:59 artom I _think_ as live migration stuff got added they all stated using the MovaClaim?
13:57:00 gibi obviously what is not done today does not casue any highly visible issue so maybe those things does not need to be claimed during live migration, but it is very strange
13:57:50 gibi yeah using the MoveClaim would be logical during live migration, but my observation is that for a non NUMA aware VM the code does not use any claim
13:58:14 gibi (except the PCI one outside of the Claim obj)
13:59:05 artom I agree with you that it's strange
13:59:21 artom But it's been so long (Train!) that I've lost most context for the decisions that were made
13:59:49 gibi yeah I understand you don't have a context. I will try to charachterize the my feelings more
14:00:12 artom I talk about it here: https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/numa-aware-live-migration.html#resource-claims
14:00:15 gibi best would be to find a way how this could break our resource tracking and the move from there
14:00:32 gibi I need to jump on a call, I will be back in 30 mins (hopefully)
14:01:14 artom OK
14:02:28 gibi that section reads for me like we wanted to do the claim regardless of NUMA awareness
14:17:30 artom Yeah, I dunno anymore :(
14:17:53 artom Maybe let's come at it from a different angle - what led you to notice this, and what are you trying to do?
14:20:46 dansmith ah, the old "let me answer a different question" deflection :D
14:21:52 artom Hah
14:22:00 artom I'd _like_ to answer the original question
14:22:03 artom I just don't remember
14:24:23 dansmith I recognize the strategy well
14:52:12 dansmith gibi: so, there's a race happening, which maybe/probably affects other stuff.. any chance you have a few minutes to gmeet for me to describe and see if you're aware already?
14:53:18 dansmith specifically a provider_tree update race
14:55:46 sean-k-mooney i dont think we update the provier tree resereved value in nova for anything else, although i know neutron does for l3 routed networks
14:56:14 sean-k-mooney so they might have seen whatever the race is with that
14:58:00 dansmith it's not a problem specific to reserved, that's just how I'm seeing it
14:58:31 dansmith although perhaps it doesn't get exposed because of the nature of the external update in this case, although that might mean things like neutron/cinder do apply
14:59:08 sean-k-mooney are you getting a generation conflcit?
15:00:15 dansmith no
15:00:30 dansmith sean-k-mooney: there was no discussion of this before, I just discovered this while testing locally
15:01:28 dansmith gibi: there's nothing to read back except up to my ask for a gmeet consult :D
15:01:37 gibi artom: so this come up when testing the vgpu live migration series
15:01:56 gibi until now all our testing was done with non NUMA aware VMs
15:02:16 gibi and then Uggla did the first NUMA aware testing and hit by the check in the move claim that was not hit before
15:02:26 gibi this check https://review.opendev.org/c/openstack/nova/+/944105/1/nova/compute/claims.py
15:02:41 gibi so we started to check why we did not hit this code before in our testing
15:03:02 gibi and it turns out because nova does not do a move claim during live migration if the VM is not NUMA aware
15:03:08 gibi which sounds very strange to me
15:04:49 gibi so I try to figure out if this is an old bug, or it is intentional
15:05:47 gibi the code read likes the condition around the move claim wanted to handle rolling upgrade but somehow but actually it does more than that in reality as it disables the move claim for non NUMA aware VMs
15:07:45 gibi dansmith: sure we can have a gmeet. I'm free until the top of the hour
15:07:57 dansmith meet.google.com/jad-dzbb-nkb
15:08:22 dansmith sean-k-mooney if you're free and interested ^
15:19:19 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing instance-action finish_time https://review.opendev.org/c/openstack/nova/+/928933
15:24:32 Uggla @gibi @dansmith interested by the chat, but I have noticed it too late. However I'd like to know your findings.
16:06:42 opendevreview ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106
16:06:43 opendevreview ribaudr proposed openstack/nova master: FUP Add a warning to make non-explicit live migration request debugging easier https://review.opendev.org/c/openstack/nova/+/944133
17:10:21 dansmith gibi: first quick stab at invalidating the cached RP shows that we immediately refresh it when we go to do the placement sync
17:10:34 dansmith so I think that's the ticket out of the problem
17:14:27 bauzas folks, I'm just moving to thelounger IRC bouncer and client after 12 years of ZNC, but I continue to run ZNC with the bauzas_ nick
17:14:47 bauzas just in case you don't get a reply from me, ping both of us :)
17:16:44 gibi Uggla: we discovered that there is a provider tree cache in the resouce tracker that causes a race condition if an OTU device is allocated, reserverd, deallocated, externally unreserved, and allocated again. It is a cache invalidation problem between placement and nova, and Dan is now adding a way to invalidate the caches explicitly when the OTU device is reseved by nova
17:18:38 gibi Uggla: we also touched on the non NUMA aware live migration move claim question as well. We think the move claim was added to that code path when NUMA aware live migration was added, and before that only the periodic update_available_resources healed the allocation in the resouce tracker
17:18:52 gibi so probably that is what happens today for non NUMA aware VMs when live migrated
17:20:43 gibi this gives me enough certanty that I can comment on you patch removing the PCI check from the live migration claim. That check happens in the conductor in any case so I think that was just a dupicated check. I will comment on the patch tomorrow
17:20:51 gibi cc artom ^^
17:25:24 Uggla @gibi, may I rephrase to ensure I got it: non numa vm will not claimed resources explicitly they are spawned and then a preriodic adjusts the allocations in the RT ?

Earlier   Later