| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-02 | |||
| 12:35:43 | sean-k-mooney | grami[m]: changign this is nto going to be easy in nova. there is an upgrade impact to this change and it will likely require a spec. | |
| 12:36:33 | sean-k-mooney | it may be possibel to jsut alwasy use a pcie-switch-upstream-port instead of pcie-root-port but i susspect that will increase teh qemu memory usage | |
| 12:36:46 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix duplicate words https://review.opendev.org/c/openstack/nova/+/959136 | |
| 12:36:50 | sean-k-mooney | so that may not be accpatabel to do in general | |
| 12:37:47 | grami[m] | sean-k-mooney: thanks for the info will do some digging on the spec and look to do one. | |
| 12:38:43 | sean-k-mooney | before gettign that far have you done any basic performace turning like ensuring numa affintiy between the vm and pci device | |
| 12:38:56 | sean-k-mooney | by defualt nova only does that if the guest has a numa toplgoy | |
| 12:39:25 | sean-k-mooney | i.e. addh hw:mem_page_size=small (exiplcit small 4k numa local pages) | |
| 12:39:46 | sean-k-mooney | will significnatly boost your vm performance if it was previosul going across numa | |
| 12:39:51 | grami[m] | sean-k-mooney: The memory issue is already a problem with 256gb as the BAR doesn't have enough so you have to increase the this just to get the GPU to boot. | |
| 12:40:25 | grami[m] | Already have numa dedicated CPU and huge page | |
| 12:41:02 | sean-k-mooney | that not what i was referint too. so each pci_root_port add a couple of megabyte of overhead even if un used | |
| 12:41:51 | sean-k-mooney | so qhen you enable the q35 machine type with the max number of pcie-root-ports we supprot today it has doble the qemu overhad as the pc machien type | |
| 12:42:21 | sean-k-mooney | usign a pcie-swithc might actully help with that but that one of the thing we woudl need to consier | |
| 12:42:40 | sean-k-mooney | a long time ago i wanted to allcoate 1 pci switch per guest numa node by default | |
| 12:42:59 | sean-k-mooney | and then attach the passhtough device to the relevnet virtual numa node pcie switch | |
| 12:43:14 | sean-k-mooney | so that the guess coudl make proper schdulign decisions | |
| 12:43:30 | sean-k-mooney | today all device in teh guest have nuam node -1 i.e. not numa info aviable | |
| 12:43:45 | sean-k-mooney | so the gest kernel assume its on virtual numa ndoe 0 effectvily | |
| 12:44:11 | sean-k-mooney | which can degreed performance so if you have not already tried expliclty having only 1 numa node in the guest that is wroth testing | |
| 12:44:33 | grami[m] | Ah yeah right now I'm going to see if I can get the instance to work by editing the xml then will look to see what needs to change. | |
| 12:45:10 | grami[m] | Yeah made sure it's only one numa zone for testing and will go from that. | |
| 12:46:53 | sean-k-mooney | right now i dont know of a way to change the bar space allocated to the viruatlised device. i assuem that need qemu supprot and im not sure if livbirt and qemu are hooked up such that it can be adjusted | |
| 12:47:48 | sean-k-mooney | there is memReserve | |
| 12:47:56 | sean-k-mooney | ```memReserve | |
| 12:47:58 | sean-k-mooney | Some PCI devices have non-prefetchable memory bar larger than 2MiB. Use this attribute to override value computed by firmware and thus make controller reserve more memory (in KiB) so that such PCI device can be hot plugged. For cold plugged PCI devices, the firmware will automatically reserve the correct amount of memory``` | |
| 12:48:25 | sean-k-mooney | but i dont think that is correect | |
| 12:59:59 | sean-k-mooney | grami[m]: you not trying to use resizable bar ar you? | |
| 13:04:16 | sean-k-mooney | just doing a little quick resarch qemu/libvirt wont do the resize for you you have to do that manually today before bindign the device to vfio-pci. second there is a limisation in ovmf by default where the adress apce resverd sames to default to 32GB for mmio mappings https://edk2.groups.io/g/discuss/topic/59340711 | |
| 13:05:37 | sean-k-mooney | we do not allow any use of <qemu:commandline> in nova so you woudl need that to be supproted in libvirt nativly before turnign the mmio apature via "fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536" coudl be supproted | |
| 13:05:56 | gibi | sean-k-mooney: | |
| 13:06:07 | gibi | thanks for the review in https://review.opendev.org/c/openstack/nova/+/954149 | |
| 13:06:42 | gibi | I responded inline. bottom line your reserved idea is probably correct, but I'm afraid of the complexity of the patch already | |
| 13:07:08 | sean-k-mooney | gibi: that not a blocker | |
| 13:07:16 | sean-k-mooney | we can consider it in a followup | |
| 13:07:24 | sean-k-mooney | i was jsut trying to think is there a way to close that gap | |
| 13:09:21 | grami[m] | sean-k-mooney: Yeah had to add that Nvidia says it's a requirement for the larger gpus | |
| 13:09:50 | sean-k-mooney | grami[m]: based on https://bugzilla.redhat.com/show_bug.cgi?id=1578278 it looks lime https://specs.openstack.org/openstack/nova-specs/specs/2023.1/approved/libvirt-maxphysaddr-support.html may also impact that | |
| 13:09:57 | sean-k-mooney | but i dont think the two ar the same | |
| 13:10:12 | grami[m] | Already have a patch for this but it's medieval in its method but works for now. | |
| 13:10:16 | sean-k-mooney | we cant accapt any change that uses raw <qemu:commandline> elements in nova | |
| 13:10:52 | sean-k-mooney | so if we need to tweak the ovmf parmater seperatly form teh cpu adress space | |
| 13:10:58 | sean-k-mooney | we will need to modify libvirt first | |
| 13:11:52 | grami[m] | Ah thanks for that | |
| 13:37:43 | opendevreview | ribaudr proposed openstack/nova master: doc: mark the maximum microversion for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/959180 | |
| 13:40:42 | opendevreview | ribaudr proposed openstack/nova master: Update compute rpc alias for epoxy https://review.opendev.org/c/openstack/nova/+/959181 | |
| 13:41:01 | gibi | sean-k-mooney: it is the freqency and the impact of the gap vs. my fears of the complexity to close and test that gap is what makes me reluctant | |
| 13:45:44 | gibi | bauzas: re | |
| 13:45:45 | gibi | 10:32 < bauzas> honestly, I don't have any number, if the job works up to tomorrow, then if so, let's leave it to be voting :) | |
| 13:45:55 | gibi | It works https://zuul.opendev.org/t/openstack/builds?job_name=nova-tox-py312-threading&project=openstack/nova | |
| 13:48:16 | bauzas | perfect, then let me hit the button | |
| 13:49:13 | gibi | thanks! | |
| 13:52:34 | opendevreview | ribaudr proposed openstack/nova master: Add service version for Falmingo https://review.opendev.org/c/openstack/nova/+/959184 | |
| 14:15:58 | opendevreview | ribaudr proposed openstack/nova master: Add Flamingo prelude section https://review.opendev.org/c/openstack/nova/+/959188 | |
| 15:00:36 | Uggla | Nova meeting in ~1h | |
| 15:22:20 | opendevreview | ribaudr proposed openstack/nova-specs master: Move Flamingo implemented specs https://review.opendev.org/c/openstack/nova-specs/+/959202 | |
| 15:34:57 | opendevreview | Merged openstack/nova stable/2025.1: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957739 | |
| 15:47:40 | opendevreview | Merged openstack/nova-specs master: Move Flamingo implemented specs https://review.opendev.org/c/openstack/nova-specs/+/959202 | |
| 15:49:03 | opendevreview | Merged openstack/nova master: doc: mark the maximum microversion for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/959180 | |
| 16:00:21 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:21 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:21 | opendevmeet | Meeting started Tue Sep 2 16:00:21 2025 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:21 | Uggla | #startmeeting nova | |
| 16:00:30 | Uggla | Hello everyone | |
| 16:01:18 | gibi | o/ | |
| 16:01:27 | jayaanand | o/ | |
| 16:01:41 | gmaan | o/ | |
| 16:01:44 | dansmith | o. | |
| 16:02:42 | bauzas | o/ | |
| 16:02:44 | Uggla | Let's start. | |
| 16:02:48 | elodilles | o/ | |
| 16:02:57 | Uggla | #topic Bugs (stuck/critical) | |
| 16:03:04 | Uggla | #info No Critical bug | |
| 16:03:18 | Uggla | #topic Gate status | |
| 16:03:25 | Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:03:30 | Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:03:37 | Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | |
| 16:03:43 | Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:03:50 | Uggla | #info Please try to provide a meaningful comment when you recheck | |
| 16:04:25 | Uggla | Anythink special with the gates ? | |
| 16:04:33 | Uggla | Anything special with the gates ? | |
| 16:04:34 | gibi | just a heads up | |
| 16:05:15 | gibi | that I see multiple cases when the ceph multistore job failed due to volume build issues and mostly in the encrypted volume tempest tests | |
| 16:05:41 | gibi | I did not dig deeper but I feel it is a pattern | |
| 16:07:06 | Uggla | an idea of the success/failure rate ? | |
| 16:07:30 | gibi | https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack/nova | |
| 16:07:33 | gibi | not terrible | |
| 16:08:49 | gibi | I will try to dig into the failure tomorrow | |
| 16:08:59 | Uggla | ok thanks gibi | |
| 16:09:16 | Uggla | although that looks not easy. | |
| 16:09:42 | Uggla | moving on to next topic | |
| 16:09:49 | Uggla | #topic tempest-with-latest-microversion job status | |
| 16:09:57 | Uggla | #link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-with-latest-microversion&skip=0 | |
| 16:10:11 | Uggla | gmaan something you would like to share on this topic ? | |
| 16:10:19 | gmaan | no progress, I am going to spend time on it this week. | |
| 16:10:42 | Uggla | 👍 | |
| 16:10:58 | Uggla | #topic Release Planning | |
| 16:11:06 | Uggla | #link https://releases.openstack.org/flamingo/schedule.html | |
| 16:11:13 | Uggla | #info Nova deadlines are set in the above schedule | |
| 16:11:21 | Uggla | #info RC1 target is next week. | |