| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-17 | |||
| 10:43:06 | opendevreview | Stephen Finucane proposed openstack/nova master: Migrate setup configuration to pyproject.toml https://review.opendev.org/c/openstack/nova/+/953705 | |
| 10:43:06 | opendevreview | Stephen Finucane proposed openstack/nova master: setup: Remove pbr's wsgi_scripts https://review.opendev.org/c/openstack/nova/+/902688 | |
| 10:44:59 | sean-k-mooney | stephenfin: sure | |
| 11:19:44 | gibi | ralonsoh: no worries :) | |
| 14:12:45 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754 | |
| 15:21:36 | gmaan | stephenfin: ack, not sure if I can do today and I am on PTO tomorrow. But I will do early next week. not just this but the remaining series | |
| 15:22:19 | stephenfin | nw. doesn't look like they will merge anytime soon. The gate is super unstable :( | |
| 15:23:10 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.1: Add ServersViewBuilderTestV296 unit test class https://review.opendev.org/c/openstack/nova/+/955304 | |
| 15:23:11 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.1: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955305 | |
| 16:17:59 | opendevreview | Merged openstack/nova master: Move ConductorManager to use spawn_on https://review.opendev.org/c/openstack/nova/+/948187 | |
| 16:29:25 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.2: Add ServersViewBuilderTestV296 unit test class https://review.opendev.org/c/openstack/nova/+/955309 | |
| 16:29:26 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.2: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955310 | |
| #openstack-nova - 2025-07-18 | |||
| 04:24:07 | aravindh5murugesan | Hello folks. I'm running into a problem when I set CPU models to custom and the specifying the models. The default config that we receive from our vendor sets the cpu-model to broadwell-notsx-IBS something. But our CPU model is saphirerapids and the oldest we need to support is Icelake. But When I set it to Icelake or SaphireRapids, I used the same name as listed in virsh cpu-models, but nova complains that CPU | |
| 04:27:37 | aravindh5murugesan | can assist me here. 2025-07-17 20:44:36.072 ERROR oslo_service.service [req-fa1ce5c0-d687-4d09-ac10-7ca05eb98461 None None] Error starting thread.: nova.exception.InvalidCPUInfo: Configured CPU model: SapphireRapids-noTSX and CPU Flags [] ar not compatible with host CPU. Please correct your config and try again. Unacceptable CPU info: CPU doesn't have compatibility | |
| 04:28:08 | aravindh5murugesan | CPU model in hypervisor - Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Silver 4416+ | |
| 09:09:51 | sean-k-mooney | aravindh5murugesan the problem you are having is that icelake has feature flags like tsk tha tintel disabled in microcode and does tno supprot on saphire rapids | |
| 09:11:11 | sean-k-mooney | *like the flags for tsx that intel ... | |
| 10:53:08 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop unused parameter type validation method https://review.opendev.org/c/openstack/nova/+/909189 | |
| 12:03:33 | aravindh0murugesan | sean-k-mooney: In that case, would it possible to disable those unsupported flags and somehow make this work? | |
| 12:20:19 | sean-k-mooney | yes there are 2 ways, libvirt provide some custom models for older cpu i.e. the *-notsx-IBS variants | |
| 12:20:36 | sean-k-mooney | but nova also has a config option that allows you to add/remove cpu flags | |
| 12:37:05 | aravindh0murugesan | I compared the icelake.xml and my lscpu flags and found these flags are missing in my host cpu - avx512vnni,mpx,pclmuldq,spec-ctrl,sse4.1,sse4.2,tsc-deadline, | |
| 12:37:18 | aravindh0murugesan | so I just do a - on these flags in nova configs? | |
| 12:38:45 | aravindh0murugesan | cpu_models = Cascadelake-Server-noTSX cpu-model-extra-flags = -avx512vnni,-mpx,-pclmuldq,-spec-ctrl,-sse4.1,-sse4.2,-tsc-deadline this does not work either | |
| 12:44:45 | aravindh0murugesan | sean-k-mooney: Editing the XML to say removed='yes' kinda worked. But I'm not sure if this is the suggested way to do things. | |
| 12:58:38 | aravindh0murugesan | My bad, changing this cpu-model-extra-flags to snake case fixed it. Thanks. | |
| 13:23:40 | sean-k-mooney | ack so you were able to get this wokring | |
| 13:25:27 | sean-k-mooney | and yes https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_model_extra_flags is the cofnig opiton | |
| 13:26:13 | sean-k-mooney | by the wya i have alwasy know cpu_model_extra_flags as SLUG case and cpu-model-extra-flags as snake case | |
| 13:28:24 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.2: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955310 | |
| 13:28:39 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.1: Add ServersViewBuilderTestV296 unit test class https://review.opendev.org/c/openstack/nova/+/955304 | |
| 13:28:40 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.1: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955305 | |
| 14:31:42 | dansmith | gibi: so, with my patch to actually fix s-g, I'm hitting the db poison fixture on NoDBTestCase tests.. I was thinking that I must be working around one of the mocks for that, but I haven't found it yet | |
| 14:32:26 | dansmith | I'm wondering if maybe before we were scheduling things to threads but not actually running them and these tests are actually missing db mocks that are now getting tickled | |
| 14:32:32 | dansmith | have you run into that anywhere else? | |
| 15:08:18 | gibi | I havent seen this type of issues yet with tasks using the futurists executors especially not in legacy mode spawning GreenThreads. | |
| 15:09:31 | gibi | I can take a look at the actual failure on Monday | |
| 15:10:02 | dansmith | gibi: wait, I think I'm just stupid | |
| 15:10:24 | dansmith | I was raising the exception the task raised in my result() but I think our contract with the sg caller is to return the Exception if it raised | |
| 15:11:29 | gibi | yes I think scatter gather expects the exception to be returned | |
| 15:11:55 | gibi | https://github.com/openstack/nova/blob/770a6f1eabc6cfb77d3ac5afd91e6e8ce9f8b9d9/nova/context.py#L425 | |
| 15:12:16 | gibi | https://github.com/openstack/nova/blob/770a6f1eabc6cfb77d3ac5afd91e6e8ce9f8b9d9/nova/context.py#L463 | |
| 15:12:34 | gibi | so yes the exception caught on the task thread and returned az the result value of the task | |
| 15:14:26 | dansmith | yup | |
| 15:23:32 | opendevreview | Merged openstack/nova master: Make nova.utils.pass_context private https://review.opendev.org/c/openstack/nova/+/948188 | |
| 15:29:47 | opendevreview | Dan Smith proposed openstack/nova master: Remove eventlet timer from multi_cell_list https://review.opendev.org/c/openstack/nova/+/954990 | |
| 15:29:47 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Parallelize s-g generators https://review.opendev.org/c/openstack/nova/+/955091 | |
| 15:30:16 | dansmith | that ^ should fix the unit/fixture failure on the eventlet timer patch and get the WIP much closer to passing.. more work to be done there of course, but at least it's not 100% fail | |
| #openstack-nova - 2025-07-19 | |||
| 03:11:10 | aravindh1murugesan | sean-k-mooney[m]: We are doing a POC with lightbits nvme/TCP. I get about 1.5 million IOPS/s on hypervisors directly, but on the VM running in the same hypervisor, its hardly touching 400k. | |
| 03:12:54 | aravindh1murugesan | Lightbits suggestes adding iothreads to libvirt domain XML and dedicating some to the disk. But seems like a hack to edit domain XML, I would to know if there is a way to set this from Nova, or any plans to add scalable IO threads to the VM? | |
| 10:20:11 | sean-k-mooney[m] | aravindh1murugesan: there are plans to add iothread support, the desigin missed the spec freeze for this release so we will hopefully have them next release 2026.1 | |
| 10:21:31 | sean-k-mooney[m] | for now the best you can do is use a dedicated emulator thread and test with hw_disk_bus=scisi with hw_scsi_model=virtio | |
| 10:23:50 | sean-k-mooney[m] | aravindh1murugesan: im not sure how familar you are with nova but the libvirt xml is and internal implmenation detail and not something you are allowed to modify. nova will regenerate the xml when you reboot or move the vm so it not a viable option. lightbit should be aware of that so im not sure why they would recommend that | |
| #openstack-nova - 2025-07-21 | |||
| 07:11:48 | gibi | FYI Finally I got my ceph bug tracker account approved so I filed https://tracker.ceph.com/issues/72203 about the stack allocation error issue we saw in nova-ceph-multistore last week with v19.2.1 | |
| 11:09:13 | opendevreview | Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149 | |
| 12:01:14 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754 | |
| 14:21:13 | opendevreview | Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149 | |
| 14:28:09 | gibi | if somebody has an inclination to review bugfixes in our PCI tracker / PCI in Placement codepath then there are two bugfix series ready for review | |
| 14:28:28 | gibi | * https://review.opendev.org/q/topic:%22bug/2115729%22 - this fixes an ugly but simple device removal inefficency | |
| 14:28:59 | gibi | * https://review.opendev.org/q/topic:%22bug/2115905%22 - this is a more complicated fix about allocated but removed devices | |
| 17:16:09 | aravindh4murugesan | sean-k-mooney: I created a GPU alias for our L40s Passthrough. The flavors are created with proper metadata and the VM is assigned to the correct GPU host. However, the VM boot logs in horizon shows nothing and console says "Guest has not initalized display yet". I can not ssh to this VM either. Our goal is to run headless GPU workloads in the VM, kinda confused where this is stuck? | |
| 17:16:34 | aravindh4murugesan | For this usecase, I'm using host-passthrough as the CPU model | |
| 17:17:39 | sean-k-mooney | aravindh4murugesan: gpu passthohg and vnc are seperate things. in general we recommend setting hw_video_model=none when using gpu passhtough | |
| 17:18:08 | aravindh4murugesan | sean-k-mooney: But if I set to none, will I be able to control the VM over console? | |
| 17:18:16 | sean-k-mooney | having both an emulated gpu and a pasthough gpu can somethime prevent the guest form booting | |
| 17:18:46 | sean-k-mooney | in genral no, while that is technially possibel to do at the libvirt level its not supported in openstack | |
| 17:19:03 | sean-k-mooney | you can via the serial console but not via vnc | |
| 17:19:33 | sean-k-mooney | you likely dont have the serial console deployed and configured | |
| 17:19:48 | aravindh4murugesan | Oh I see, so I will have to set it to hw_video_model=none on image, (is this a valid meta on flavor?) and ssh from other vms those should be fine right? | |
| 17:20:20 | sean-k-mooney | i dont think so the *_model options are normaly only valid on the image | |
| 17:20:58 | sean-k-mooney | we may have added supprot for the video model but i would have to check | |
| 17:22:02 | sean-k-mooney | no we supprot hw_video:ram_max_mb in the falvor but not hw:video_model | |
| 17:22:25 | aravindh4murugesan | sean-k-mooney: Allow me a minute, I'm adding this to my image and see if that works. | |
| 17:22:40 | sean-k-mooney | you will need to create a new vm | |
| 17:22:51 | sean-k-mooney | for the upstad image properies to take effect | |
| 17:23:01 | aravindh4murugesan | Okay if its not available on flavor, I will maintain a separate images for GPU hosts. | |
| 17:23:07 | aravindh4murugesan | Yes new vm in progress | |
| 17:25:01 | aravindh4murugesan | sean-k-mooney: I created the VM now. It shows VM has no display configured as expected. I'm waiting for a while and checking if the Ping works from the nearby vm on the same subnet | |
| 17:25:47 | sean-k-mooney | you can use "openstack server console log show <uuid>" to view the console log of the boot process | |
| 17:26:11 | sean-k-mooney | assumign the image's kernel is configured to log kernel logs to the serial port | |
| 17:26:22 | sean-k-mooney | thats aviable in horizon too | |
| 17:28:02 | aravindh4murugesan | That command is showing empty logs. the console subcommand was not found in server, but openstack console log show worked, but returned zero logs | |
| 17:28:11 | aravindh4murugesan | I can get logs for CPU only VM this way fine | |
| 17:28:39 | aravindh4murugesan | Same image was used. We use Ubuntu 24 Cloud img from canonical to test our changes | |
| 17:30:13 | aravindh4murugesan | sean-k-mooney: I apologize for bugging you often, but this GPU stuff are pretty new to me, so trying to make sense of it. | |
| 17:30:29 | sean-k-mooney | hum its hard to say without logs from the vm about what is happeing. in theory it shoudl work but it sound like there is something happingn with the boot process. | |
| 17:31:33 | aravindh4murugesan | sean-k-mooney: is there anything from virsh that can help understand whats going on? | |
| 17:32:11 | sean-k-mooney | the reason i mentioned the boot issue is because on windows 11 and the ubuntu installer i have to have at most 1 gpus for it to work properly in my libvirt vms. once installed i have got ubuntu server to boot with both a passthough gpu and virtio gpu but i have seen it be quite picky | |
| 17:33:56 | sean-k-mooney | what you kind fo need to do is change the kernel boot args. you could try using a cirror rescuse image. that wont have supprot fo the gpu but it might be able to boot enouch for you to ssh in and update grub config on the ubunut boot partion to enable early cosole kernel looging | |
| 17:34:12 | sean-k-mooney | you could also enabel the qemu boot menu instead | |
| 17:34:51 | sean-k-mooney | i.e. if you dont disable the video model and can see can bios output you coudl edit the grub commandlien that way | |
| 17:35:25 | sean-k-mooney | if you have virsh adcess then yes you could use virsh console and also virsh reset? to reboot the vm and try and edit grub directly | |
| 17:35:30 | aravindh4murugesan | I'm booting this from a volume, not sure if I can rescue it with cirros image. But let me give it a shot | |
| 17:35:55 | aravindh4murugesan | I have virsh access | |
| 17:36:34 | sean-k-mooney | you can but you might need to set some image properties. BFV needs the stable rescue feature to support rescue | |
| 17:37:26 | aravindh4murugesan | Is there anything I can do with virsh now? | |
| 17:37:43 | aravindh4murugesan | When I do virsh console, its says connected, but I cannot type anything to it | |
| 17:39:41 | aravindh4murugesan | I will also test your suggestion on rescue image and see if I can get something out of it | |
| 19:05:39 | opendevreview | Merged openstack/nova stable/2025.1: Fix neutron client dict grabbing https://review.opendev.org/c/openstack/nova/+/953737 | |
| 20:26:33 | opendevreview | Wesley Hershberger proposed openstack/nova stable/2024.2: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955310 | |