| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-02 | |||
| 12:00:44 | elodilles | i see | |
| 12:01:45 | sean-k-mooney | i want to eventully explore using playwrite (instead of creating selenium tests) to see if we can get some end to end test coverage but we also coudl use django unit test framework more then we do | |
| 12:02:07 | sean-k-mooney | horizon has historically never relaly facilated testing plugins well | |
| 12:02:23 | sean-k-mooney | so as a resutl most plugins were nto proeprly tested | |
| 12:02:53 | sean-k-mooney | fixing that is on our todo list just not as high as it proably shoudl be | |
| 12:06:13 | elodilles | maybe next cycle ;) | |
| 12:11:27 | grami[m] | Has anyone run into the problem with Nvidia GPU and requiring pci-expander-bus and vswitch-pcis? | |
| 12:19:56 | sean-k-mooney | grami[m]: it should not require them however they recommend them | |
| 12:20:01 | sean-k-mooney | for performance reasons | |
| 12:20:23 | sean-k-mooney | vswitch-pcis is not something im famialr with | |
| 12:21:20 | sean-k-mooney | nvidia recomemtn in one of there kvm docs that you attache thre gpus and nics to a virutal pcie switch in the vm | |
| 12:21:48 | sean-k-mooney | nova cant do that but its too hlep with device to device comuication within the vm | |
| 12:26:49 | sean-k-mooney | instead of conencting the gpu to a pcie-root-port you add a pcie-switch-upstream-port contoler instead and you can then group multipel gpus or other device as downstream ports on that switch. | |
| 12:27:18 | sean-k-mooney | there is very little upstream libvirt docs on how this all works https://libvirt.org/formatdomain.html#controllers | |
| 12:27:38 | sean-k-mooney | and hte nvidia docs on this topic are relivitly new. | |
| 12:27:54 | sean-k-mooney | but equally hard to actuly find | |
| 12:31:24 | grami[m] | sean-k-mooney: thanks yeah I'm in the process of passing h200 with the normal pci-root-port performance is worse then the CPU. I guess at some point I will have to right a patch but the nova code is confusing to me and need a starting point. Do you know what line in nova or the file to start in? | |
| 12:35:15 | sean-k-mooney | honestly this sould liek a nivida hardware bug as this really shoudl not have any impact if tis only for 1 h200 device | |
| 12:35:43 | sean-k-mooney | grami[m]: changign this is nto going to be easy in nova. there is an upgrade impact to this change and it will likely require a spec. | |
| 12:36:33 | sean-k-mooney | it may be possibel to jsut alwasy use a pcie-switch-upstream-port instead of pcie-root-port but i susspect that will increase teh qemu memory usage | |
| 12:36:46 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix duplicate words https://review.opendev.org/c/openstack/nova/+/959136 | |
| 12:36:50 | sean-k-mooney | so that may not be accpatabel to do in general | |
| 12:37:47 | grami[m] | sean-k-mooney: thanks for the info will do some digging on the spec and look to do one. | |
| 12:38:43 | sean-k-mooney | before gettign that far have you done any basic performace turning like ensuring numa affintiy between the vm and pci device | |
| 12:38:56 | sean-k-mooney | by defualt nova only does that if the guest has a numa toplgoy | |
| 12:39:25 | sean-k-mooney | i.e. addh hw:mem_page_size=small (exiplcit small 4k numa local pages) | |
| 12:39:46 | sean-k-mooney | will significnatly boost your vm performance if it was previosul going across numa | |
| 12:39:51 | grami[m] | sean-k-mooney: The memory issue is already a problem with 256gb as the BAR doesn't have enough so you have to increase the this just to get the GPU to boot. | |
| 12:40:25 | grami[m] | Already have numa dedicated CPU and huge page | |
| 12:41:02 | sean-k-mooney | that not what i was referint too. so each pci_root_port add a couple of megabyte of overhead even if un used | |
| 12:41:51 | sean-k-mooney | so qhen you enable the q35 machine type with the max number of pcie-root-ports we supprot today it has doble the qemu overhad as the pc machien type | |
| 12:42:21 | sean-k-mooney | usign a pcie-swithc might actully help with that but that one of the thing we woudl need to consier | |
| 12:42:40 | sean-k-mooney | a long time ago i wanted to allcoate 1 pci switch per guest numa node by default | |
| 12:42:59 | sean-k-mooney | and then attach the passhtough device to the relevnet virtual numa node pcie switch | |
| 12:43:14 | sean-k-mooney | so that the guess coudl make proper schdulign decisions | |
| 12:43:30 | sean-k-mooney | today all device in teh guest have nuam node -1 i.e. not numa info aviable | |
| 12:43:45 | sean-k-mooney | so the gest kernel assume its on virtual numa ndoe 0 effectvily | |
| 12:44:11 | sean-k-mooney | which can degreed performance so if you have not already tried expliclty having only 1 numa node in the guest that is wroth testing | |
| 12:44:33 | grami[m] | Ah yeah right now I'm going to see if I can get the instance to work by editing the xml then will look to see what needs to change. | |
| 12:45:10 | grami[m] | Yeah made sure it's only one numa zone for testing and will go from that. | |
| 12:46:53 | sean-k-mooney | right now i dont know of a way to change the bar space allocated to the viruatlised device. i assuem that need qemu supprot and im not sure if livbirt and qemu are hooked up such that it can be adjusted | |
| 12:47:48 | sean-k-mooney | there is memReserve | |
| 12:47:56 | sean-k-mooney | ```memReserve | |
| 12:47:58 | sean-k-mooney | Some PCI devices have non-prefetchable memory bar larger than 2MiB. Use this attribute to override value computed by firmware and thus make controller reserve more memory (in KiB) so that such PCI device can be hot plugged. For cold plugged PCI devices, the firmware will automatically reserve the correct amount of memory``` | |
| 12:48:25 | sean-k-mooney | but i dont think that is correect | |
| 12:59:59 | sean-k-mooney | grami[m]: you not trying to use resizable bar ar you? | |
| 13:04:16 | sean-k-mooney | just doing a little quick resarch qemu/libvirt wont do the resize for you you have to do that manually today before bindign the device to vfio-pci. second there is a limisation in ovmf by default where the adress apce resverd sames to default to 32GB for mmio mappings https://edk2.groups.io/g/discuss/topic/59340711 | |
| 13:05:37 | sean-k-mooney | we do not allow any use of <qemu:commandline> in nova so you woudl need that to be supproted in libvirt nativly before turnign the mmio apature via "fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536" coudl be supproted | |
| 13:05:56 | gibi | sean-k-mooney: | |
| 13:06:07 | gibi | thanks for the review in https://review.opendev.org/c/openstack/nova/+/954149 | |
| 13:06:42 | gibi | I responded inline. bottom line your reserved idea is probably correct, but I'm afraid of the complexity of the patch already | |
| 13:07:08 | sean-k-mooney | gibi: that not a blocker | |
| 13:07:16 | sean-k-mooney | we can consider it in a followup | |
| 13:07:24 | sean-k-mooney | i was jsut trying to think is there a way to close that gap | |
| 13:09:21 | grami[m] | sean-k-mooney: Yeah had to add that Nvidia says it's a requirement for the larger gpus | |
| 13:09:50 | sean-k-mooney | grami[m]: based on https://bugzilla.redhat.com/show_bug.cgi?id=1578278 it looks lime https://specs.openstack.org/openstack/nova-specs/specs/2023.1/approved/libvirt-maxphysaddr-support.html may also impact that | |
| 13:09:57 | sean-k-mooney | but i dont think the two ar the same | |
| 13:10:12 | grami[m] | Already have a patch for this but it's medieval in its method but works for now. | |
| 13:10:16 | sean-k-mooney | we cant accapt any change that uses raw <qemu:commandline> elements in nova | |
| 13:10:52 | sean-k-mooney | so if we need to tweak the ovmf parmater seperatly form teh cpu adress space | |
| 13:10:58 | sean-k-mooney | we will need to modify libvirt first | |
| 13:11:52 | grami[m] | Ah thanks for that | |
| 13:37:43 | opendevreview | ribaudr proposed openstack/nova master: doc: mark the maximum microversion for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/959180 | |
| 13:40:42 | opendevreview | ribaudr proposed openstack/nova master: Update compute rpc alias for epoxy https://review.opendev.org/c/openstack/nova/+/959181 | |
| 13:41:01 | gibi | sean-k-mooney: it is the freqency and the impact of the gap vs. my fears of the complexity to close and test that gap is what makes me reluctant | |
| 13:45:44 | gibi | bauzas: re | |
| 13:45:45 | gibi | 10:32 < bauzas> honestly, I don't have any number, if the job works up to tomorrow, then if so, let's leave it to be voting :) | |
| 13:45:55 | gibi | It works https://zuul.opendev.org/t/openstack/builds?job_name=nova-tox-py312-threading&project=openstack/nova | |
| 13:48:16 | bauzas | perfect, then let me hit the button | |
| 13:49:13 | gibi | thanks! | |
| 13:52:34 | opendevreview | ribaudr proposed openstack/nova master: Add service version for Falmingo https://review.opendev.org/c/openstack/nova/+/959184 | |
| 14:15:58 | opendevreview | ribaudr proposed openstack/nova master: Add Flamingo prelude section https://review.opendev.org/c/openstack/nova/+/959188 | |
| 15:00:36 | Uggla | Nova meeting in ~1h | |
| 15:22:20 | opendevreview | ribaudr proposed openstack/nova-specs master: Move Flamingo implemented specs https://review.opendev.org/c/openstack/nova-specs/+/959202 | |
| 15:34:57 | opendevreview | Merged openstack/nova stable/2025.1: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957739 | |
| 15:47:40 | opendevreview | Merged openstack/nova-specs master: Move Flamingo implemented specs https://review.opendev.org/c/openstack/nova-specs/+/959202 | |
| 15:49:03 | opendevreview | Merged openstack/nova master: doc: mark the maximum microversion for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/959180 | |
| 16:00:21 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:21 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:21 | opendevmeet | Meeting started Tue Sep 2 16:00:21 2025 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:21 | Uggla | #startmeeting nova | |
| 16:00:30 | Uggla | Hello everyone | |
| 16:01:18 | gibi | o/ | |
| 16:01:27 | jayaanand | o/ | |
| 16:01:41 | gmaan | o/ | |
| 16:01:44 | dansmith | o. | |
| 16:02:42 | bauzas | o/ | |
| 16:02:44 | Uggla | Let's start. | |
| 16:02:48 | elodilles | o/ | |
| 16:02:57 | Uggla | #topic Bugs (stuck/critical) | |
| 16:03:04 | Uggla | #info No Critical bug | |
| 16:03:18 | Uggla | #topic Gate status | |
| 16:03:25 | Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:03:30 | Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:03:37 | Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | |
| 16:03:43 | Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:03:50 | Uggla | #info Please try to provide a meaningful comment when you recheck | |
| 16:04:25 | Uggla | Anythink special with the gates ? | |
| 16:04:33 | Uggla | Anything special with the gates ? | |
| 16:04:34 | gibi | just a heads up | |