| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-03-11 | |||
| 20:46:07 | opendevreview | mitya-eremeev-2 proposed openstack/nova master: Add more thorough handling of evacuated instances https://review.opendev.org/c/openstack/nova/+/944058 | |
| #openstack-nova - 2025-03-12 | |||
| 06:27:40 | opendevreview | Amit Uniyal proposed openstack/nova master: DNM: enable NFS backend job https://review.opendev.org/c/openstack/nova/+/944008 | |
| 06:51:50 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing instance-action finish_time https://review.opendev.org/c/openstack/nova/+/928933 | |
| 08:05:26 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra spec for sound device. https://review.opendev.org/c/openstack/nova/+/926126 | |
| 08:05:27 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra specs for USB redirection. https://review.opendev.org/c/openstack/nova/+/927354 | |
| 08:05:27 | opendevreview | Michael Still proposed openstack/nova master: Protect older compute managers from sound model requests. https://review.opendev.org/c/openstack/nova/+/940770 | |
| 08:22:14 | gibi | sean-k-mooney: I have a feeling that the pool matching logic in stats.py also intentionally handles the multi spec case. See https://github.com/openstack/nova/blob/0d484ce37d86e989c8abdf57aec5e334f68206ef/nova/pci/utils.py#L75 it basically implements an OR between the multiple specs | |
| 08:22:49 | sean-k-mooney[m] | it did in the really early days | |
| 08:23:02 | sean-k-mooney[m] | but i tough twe removed this like 10+ years ago | |
| 08:23:21 | gibi | appreanlty we did not | |
| 08:23:51 | gibi | PCI in Placement never supported it though I just broke the error reporting leading to http 500 | |
| 08:23:55 | sean-k-mooney[m] | so the vfio-variant dirver live migration depend on 1:1 alias mappings | |
| 08:24:12 | sean-k-mooney[m] | at least when not using pci in placement | |
| 08:24:56 | gibi | sean-k-mooney: do you mean that the live migration will only work if the same exact device is on both sides and therefore having specs in OR relationship with different product id would not work? | |
| 08:25:15 | sean-k-mooney[m] | that is my understanding | |
| 08:25:20 | gibi | yeah I agree | |
| 08:25:44 | sean-k-mooney[m] | i dont se how it could work any other way when the deviece is directly exposed to the guest kernel as a vf | |
| 08:26:21 | sean-k-mooney[m] | i think we need to at least restict when oring can happen | |
| 08:26:48 | gibi | the unplug/plug live migration for neutron requested PCI probably works as that is not real live migration of the device, but the variant driver based will lead to very sad guest kernels | |
| 08:27:07 | sean-k-mooney[m] | yep | |
| 08:28:25 | sean-k-mooney[m] | so i know why they orginally did this which is they were trying to make differnt configurations of intels QAT device look the same\ | |
| 08:28:39 | sean-k-mooney[m] | and allow you to use ask for QAT | |
| 08:28:47 | sean-k-mooney[m] | without careing which one you got | |
| 08:29:29 | sean-k-mooney[m] | that only works if you never have to deal with live migration | |
| 08:29:59 | sean-k-mooney[m] | so i think we need to disable this when live_migratable is set to true or you use pci_in_placment | |
| 08:30:15 | sean-k-mooney[m] | either should result in a config error on startup | |
| 08:31:20 | sean-k-mooney[m] | we can allow it for non llive migratbale device or when not using PCI in placement for backward compatibality | |
| 08:58:23 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing instance-action finish_time https://review.opendev.org/c/openstack/nova/+/928933 | |
| 09:12:16 | sean-k-mooney | it is day 2 or alt world where my fundemental core belife that "their should only be 1 7:30 in a day and it should happen after noon" is being chllanged by the concept of 7 am... | |
| 09:13:27 | sean-k-mooney | its currntly 09:13 which is still an hour before im useually awake | |
| 11:43:09 | opendevreview | ribaudr proposed openstack/nova master: FUP improve comment accuracy and variable naming for tag removal https://review.opendev.org/c/openstack/nova/+/943124 | |
| 11:43:10 | opendevreview | ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106 | |
| 11:43:10 | opendevreview | ribaudr proposed openstack/nova master: FUP Remove unnecessary PCI check https://review.opendev.org/c/openstack/nova/+/944105 | |
| 11:52:58 | andrewbonney | sean-k-mooney: in case you have time today for another look, https://review.opendev.org/c/openstack/nova/+/919961 seems fine in CI | |
| 12:13:32 | sean-k-mooney | andrewbonney: ill review it propelry shortly. if you need to respin it might be nice to add a fixes release note | |
| 12:13:40 | sean-k-mooney | but thats not strictly required | |
| 12:13:56 | andrewbonney | thanks | |
| 12:13:58 | sean-k-mooney | so ya ill take a look shortly | |
| 12:23:17 | sean-k-mooney | andrewbonney: +2 form me | |
| 12:24:02 | sean-k-mooney | andrewbonney: this proably wont merge for RC1 but it can merge and be backported as normal after | |
| 12:24:27 | andrewbonney | sure, just trying to get it ticked off before I head off for extended leave | |
| 12:24:50 | sean-k-mooney | ack | |
| 12:26:07 | opendevreview | Merged openstack/nova master: Fix serial console for ironic https://review.opendev.org/c/openstack/nova/+/942575 | |
| 13:44:08 | gibi | folks, Why do we only use the move claim during live migration if the VM is NUMA aware? context https://github.com/openstack/nova/blob/f71a0a6204744b7774caf85c16f27964a4f851db/nova/compute/manager.py#L9057-L9066 | |
| 13:45:06 | gibi | based on my local testing that src_supports_numa_live_migration check there is only true if the VM is NUMA aware | |
| 13:45:43 | gibi | so it seems like we doesn't do any claim for the instance during live migration if the instance is not NUMA aware (except claiming things outside of the Claim object like PCI) | |
| 13:48:43 | Uggla | @artom, can you have a look at gibi's question above. | |
| 13:50:01 | artom | gibi, I don't remember :( | |
| 13:50:11 | artom | But I'm pretty sure it's not _just_ NUMA-aware VMs | |
| 13:50:17 | artom | Other code added later started using that too | |
| 13:50:26 | artom | VPMEM I think? | |
| 13:51:04 | artom | And src_supports_numa_live_migration was there to handle the rolling upgrade case at the time of implementation | |
| 13:51:18 | gibi | if you look at the code linked above there is a check that looks for src_supports_numa_live_migration in the migrate data, that condition is false for me locally for a non NUMA aware VM | |
| 13:51:22 | artom | We can _probably_ remove it now, with the usual caveats around RPC | |
| 13:51:30 | Uggla | @artom, look at https://review.opendev.org/c/openstack/nova/+/944105 you left a comment that may remember you things. | |
| 13:52:48 | artom | Oh yeah, there was the SRIOV live migration thing as well | |
| 13:52:53 | artom | It's handled by a separate code path | |
| 13:53:06 | Uggla | yep below in the manager | |
| 13:53:12 | gibi | yeah the PCI stuff claimed differently, but what about the rest of the resources? | |
| 13:53:31 | artom | You mean non-Placement resources? | |
| 13:55:14 | Uggla | @gibi this is claimed at scheduler step before no ? | |
| 13:55:58 | gibi | artom: yea | |
| 13:56:19 | gibi | all the normal things we do during an instance claim or during a move claim in other ops | |
| 13:56:59 | artom | I _think_ as live migration stuff got added they all stated using the MovaClaim? | |
| 13:57:00 | gibi | obviously what is not done today does not casue any highly visible issue so maybe those things does not need to be claimed during live migration, but it is very strange | |
| 13:57:50 | gibi | yeah using the MoveClaim would be logical during live migration, but my observation is that for a non NUMA aware VM the code does not use any claim | |
| 13:58:14 | gibi | (except the PCI one outside of the Claim obj) | |
| 13:59:05 | artom | I agree with you that it's strange | |
| 13:59:21 | artom | But it's been so long (Train!) that I've lost most context for the decisions that were made | |
| 13:59:49 | gibi | yeah I understand you don't have a context. I will try to charachterize the my feelings more | |
| 14:00:12 | artom | I talk about it here: https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/numa-aware-live-migration.html#resource-claims | |
| 14:00:15 | gibi | best would be to find a way how this could break our resource tracking and the move from there | |
| 14:00:32 | gibi | I need to jump on a call, I will be back in 30 mins (hopefully) | |
| 14:01:14 | artom | OK | |
| 14:02:28 | gibi | that section reads for me like we wanted to do the claim regardless of NUMA awareness | |
| 14:17:30 | artom | Yeah, I dunno anymore :( | |
| 14:17:53 | artom | Maybe let's come at it from a different angle - what led you to notice this, and what are you trying to do? | |
| 14:20:46 | dansmith | ah, the old "let me answer a different question" deflection :D | |
| 14:21:52 | artom | Hah | |
| 14:22:00 | artom | I'd _like_ to answer the original question | |
| 14:22:03 | artom | I just don't remember | |
| 14:24:23 | dansmith | I recognize the strategy well | |
| 14:52:12 | dansmith | gibi: so, there's a race happening, which maybe/probably affects other stuff.. any chance you have a few minutes to gmeet for me to describe and see if you're aware already? | |
| 14:53:18 | dansmith | specifically a provider_tree update race | |
| 14:55:46 | sean-k-mooney | i dont think we update the provier tree resereved value in nova for anything else, although i know neutron does for l3 routed networks | |
| 14:56:14 | sean-k-mooney | so they might have seen whatever the race is with that | |
| 14:58:00 | dansmith | it's not a problem specific to reserved, that's just how I'm seeing it | |
| 14:58:31 | dansmith | although perhaps it doesn't get exposed because of the nature of the external update in this case, although that might mean things like neutron/cinder do apply | |
| 14:59:08 | sean-k-mooney | are you getting a generation conflcit? | |
| 15:00:15 | dansmith | no | |
| 15:00:30 | dansmith | sean-k-mooney: there was no discussion of this before, I just discovered this while testing locally | |
| 15:01:28 | dansmith | gibi: there's nothing to read back except up to my ask for a gmeet consult :D | |
| 15:01:37 | gibi | artom: so this come up when testing the vgpu live migration series | |
| 15:01:56 | gibi | until now all our testing was done with non NUMA aware VMs | |
| 15:02:16 | gibi | and then Uggla did the first NUMA aware testing and hit by the check in the move claim that was not hit before | |
| 15:02:26 | gibi | this check https://review.opendev.org/c/openstack/nova/+/944105/1/nova/compute/claims.py | |
| 15:02:41 | gibi | so we started to check why we did not hit this code before in our testing | |
| 15:03:02 | gibi | and it turns out because nova does not do a move claim during live migration if the VM is not NUMA aware | |
| 15:03:08 | gibi | which sounds very strange to me | |
| 15:04:49 | gibi | so I try to figure out if this is an old bug, or it is intentional | |