Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-07-18
14:32:32 dansmith have you run into that anywhere else?
15:08:18 gibi I havent seen this type of issues yet with tasks using the futurists executors especially not in legacy mode spawning GreenThreads.
15:09:31 gibi I can take a look at the actual failure on Monday
15:10:02 dansmith gibi: wait, I think I'm just stupid
15:10:24 dansmith I was raising the exception the task raised in my result() but I think our contract with the sg caller is to return the Exception if it raised
15:11:29 gibi yes I think scatter gather expects the exception to be returned
15:11:55 gibi https://github.com/openstack/nova/blob/770a6f1eabc6cfb77d3ac5afd91e6e8ce9f8b9d9/nova/context.py#L425
15:12:16 gibi https://github.com/openstack/nova/blob/770a6f1eabc6cfb77d3ac5afd91e6e8ce9f8b9d9/nova/context.py#L463
15:12:34 gibi so yes the exception caught on the task thread and returned az the result value of the task
15:14:26 dansmith yup
15:23:32 opendevreview Merged openstack/nova master: Make nova.utils.pass_context private https://review.opendev.org/c/openstack/nova/+/948188
15:29:47 opendevreview Dan Smith proposed openstack/nova master: Remove eventlet timer from multi_cell_list https://review.opendev.org/c/openstack/nova/+/954990
15:29:47 opendevreview Dan Smith proposed openstack/nova master: WIP: Parallelize s-g generators https://review.opendev.org/c/openstack/nova/+/955091
15:30:16 dansmith that ^ should fix the unit/fixture failure on the eventlet timer patch and get the WIP much closer to passing.. more work to be done there of course, but at least it's not 100% fail
#openstack-nova - 2025-07-19
03:11:10 aravindh1murugesan sean-k-mooney[m]: We are doing a POC with lightbits nvme/TCP. I get about 1.5 million IOPS/s on hypervisors directly, but on the VM running in the same hypervisor, its hardly touching 400k.
03:12:54 aravindh1murugesan Lightbits suggestes adding iothreads to libvirt domain XML and dedicating some to the disk. But seems like a hack to edit domain XML, I would to know if there is a way to set this from Nova, or any plans to add scalable IO threads to the VM?
10:20:11 sean-k-mooney[m] aravindh1murugesan: there are plans to add iothread support, the desigin missed the spec freeze for this release so we will hopefully have them next release 2026.1
10:21:31 sean-k-mooney[m] for now the best you can do is use a dedicated emulator thread and test with hw_disk_bus=scisi with hw_scsi_model=virtio
10:23:50 sean-k-mooney[m] aravindh1murugesan: im not sure how familar you are with nova but the libvirt xml is and internal implmenation detail and not something you are allowed to modify. nova will regenerate the xml when you reboot or move the vm so it not a viable option. lightbit should be aware of that so im not sure why they would recommend that
#openstack-nova - 2025-07-21
07:11:48 gibi FYI Finally I got my ceph bug tracker account approved so I filed https://tracker.ceph.com/issues/72203 about the stack allocation error issue we saw in nova-ceph-multistore last week with v19.2.1
11:09:13 opendevreview Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149
12:01:14 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754
14:21:13 opendevreview Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149
14:28:09 gibi if somebody has an inclination to review bugfixes in our PCI tracker / PCI in Placement codepath then there are two bugfix series ready for review
14:28:28 gibi * https://review.opendev.org/q/topic:%22bug/2115729%22 - this fixes an ugly but simple device removal inefficency
14:28:59 gibi * https://review.opendev.org/q/topic:%22bug/2115905%22 - this is a more complicated fix about allocated but removed devices
17:16:09 aravindh4murugesan sean-k-mooney: I created a GPU alias for our L40s Passthrough. The flavors are created with proper metadata and the VM is assigned to the correct GPU host. However, the VM boot logs in horizon shows nothing and console says "Guest has not initalized display yet". I can not ssh to this VM either. Our goal is to run headless GPU workloads in the VM, kinda confused where this is stuck?
17:16:34 aravindh4murugesan For this usecase, I'm using host-passthrough as the CPU model
17:17:39 sean-k-mooney aravindh4murugesan: gpu passthohg and vnc are seperate things. in general we recommend setting hw_video_model=none when using gpu passhtough
17:18:08 aravindh4murugesan sean-k-mooney: But if I set to none, will I be able to control the VM over console?
17:18:16 sean-k-mooney having both an emulated gpu and a pasthough gpu can somethime prevent the guest form booting
17:18:46 sean-k-mooney in genral no, while that is technially possibel to do at the libvirt level its not supported in openstack
17:19:03 sean-k-mooney you can via the serial console but not via vnc
17:19:33 sean-k-mooney you likely dont have the serial console deployed and configured
17:19:48 aravindh4murugesan Oh I see, so I will have to set it to hw_video_model=none on image, (is this a valid meta on flavor?) and ssh from other vms those should be fine right?
17:20:20 sean-k-mooney i dont think so the *_model options are normaly only valid on the image
17:20:58 sean-k-mooney we may have added supprot for the video model but i would have to check
17:22:02 sean-k-mooney no we supprot hw_video:ram_max_mb in the falvor but not hw:video_model
17:22:25 aravindh4murugesan sean-k-mooney: Allow me a minute, I'm adding this to my image and see if that works.
17:22:40 sean-k-mooney you will need to create a new vm
17:22:51 sean-k-mooney for the upstad image properies to take effect
17:23:01 aravindh4murugesan Okay if its not available on flavor, I will maintain a separate images for GPU hosts.
17:23:07 aravindh4murugesan Yes new vm in progress
17:25:01 aravindh4murugesan sean-k-mooney: I created the VM now. It shows VM has no display configured as expected. I'm waiting for a while and checking if the Ping works from the nearby vm on the same subnet
17:25:47 sean-k-mooney you can use "openstack server console log show <uuid>" to view the console log of the boot process
17:26:11 sean-k-mooney assumign the image's kernel is configured to log kernel logs to the serial port
17:26:22 sean-k-mooney thats aviable in horizon too
17:28:02 aravindh4murugesan That command is showing empty logs. the console subcommand was not found in server, but openstack console log show worked, but returned zero logs
17:28:11 aravindh4murugesan I can get logs for CPU only VM this way fine
17:28:39 aravindh4murugesan Same image was used. We use Ubuntu 24 Cloud img from canonical to test our changes
17:30:13 aravindh4murugesan sean-k-mooney: I apologize for bugging you often, but this GPU stuff are pretty new to me, so trying to make sense of it.
17:30:29 sean-k-mooney hum its hard to say without logs from the vm about what is happeing. in theory it shoudl work but it sound like there is something happingn with the boot process.
17:31:33 aravindh4murugesan sean-k-mooney: is there anything from virsh that can help understand whats going on?
17:32:11 sean-k-mooney the reason i mentioned the boot issue is because on windows 11 and the ubuntu installer i have to have at most 1 gpus for it to work properly in my libvirt vms. once installed i have got ubuntu server to boot with both a passthough gpu and virtio gpu but i have seen it be quite picky
17:33:56 sean-k-mooney what you kind fo need to do is change the kernel boot args. you could try using a cirror rescuse image. that wont have supprot fo the gpu but it might be able to boot enouch for you to ssh in and update grub config on the ubunut boot partion to enable early cosole kernel looging
17:34:12 sean-k-mooney you could also enabel the qemu boot menu instead
17:34:51 sean-k-mooney i.e. if you dont disable the video model and can see can bios output you coudl edit the grub commandlien that way
17:35:25 sean-k-mooney if you have virsh adcess then yes you could use virsh console and also virsh reset? to reboot the vm and try and edit grub directly
17:35:30 aravindh4murugesan I'm booting this from a volume, not sure if I can rescue it with cirros image. But let me give it a shot
17:35:55 aravindh4murugesan I have virsh access
17:36:34 sean-k-mooney you can but you might need to set some image properties. BFV needs the stable rescue feature to support rescue
17:37:26 aravindh4murugesan Is there anything I can do with virsh now?
17:37:43 aravindh4murugesan When I do virsh console, its says connected, but I cannot type anything to it
17:39:41 aravindh4murugesan I will also test your suggestion on rescue image and see if I can get something out of it
19:05:39 opendevreview Merged openstack/nova stable/2025.1: Fix neutron client dict grabbing https://review.opendev.org/c/openstack/nova/+/953737
20:26:33 opendevreview Wesley Hershberger proposed openstack/nova stable/2024.2: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955310
20:26:48 opendevreview Wesley Hershberger proposed openstack/nova stable/2024.1: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/955305
21:43:13 opendevreview Merged openstack/nova master: api: Add response body schemas for server external events API https://review.opendev.org/c/openstack/nova/+/940508
#openstack-nova - 2025-07-22
06:38:11 aravindh7murugesan sean-k-mooney: The issue I was talking about yesterday about the VM not booting is only if I set the VM to UEFI boot mode with image meta. Its working fine with BIOS mode. Is there anything specific for GPU passthrough and UEFI I should take care of?
07:58:26 opendevreview Pierre Riteau proposed openstack/nova master: doc: Fix typo in nova-manage command https://review.opendev.org/c/openstack/nova/+/955566
10:01:20 opendevreview Biser Milanov proposed openstack/nova master: StorPool: Pass the instance UUID and device_name to os-brick https://review.opendev.org/c/openstack/nova/+/930297
10:02:41 opendevreview Biser Milanov proposed openstack/nova stable/2024.1: StorPool: Pass the instance UUID and device_name to os-brick https://review.opendev.org/c/openstack/nova/+/954115
10:09:48 sean-k-mooney aravindh7murugesan not that i know of specificly but apprently yes
11:20:14 opendevreview Florian proposed openstack/nova master: Add check for PCIe devices attach limit for volume and ports https://review.opendev.org/c/openstack/nova/+/955584
12:58:37 opendevreview Florian proposed openstack/nova master: Add check for PCIe devices attach limit for volume and ports https://review.opendev.org/c/openstack/nova/+/955584
13:48:36 opendevreview Merged openstack/nova master: api: Add response body schemas for server password APIs https://review.opendev.org/c/openstack/nova/+/945736
14:27:33 opendevreview Dan Smith proposed openstack/nova master: Remove eventlet timer from multi_cell_list https://review.opendev.org/c/openstack/nova/+/954990
15:01:57 noonedeadpunk fwiw, tunnelled migrations are around 20-30% faster then native ones with TLS enabled...
15:24:32 bauzas folks, I need to skip today's upstream nova meeting
15:24:41 bauzas can someone run it ?
15:24:43 bauzas gibi ?
15:29:22 gibi bauzas: I can run it but I will run it with a quick "does anybody has anything to raise to the team" style
15:29:37 gibi and if silence then I will not iterate the agenda
15:33:04 jssfr I have two questions, which may be answerable outside the team meeting: (a) the AMD SEV-ES patches are at the top of the review list, is there anything we as a contributing company (not to that patchset specifically, but in general) can do to help?
15:33:37 jssfr (b) I pointed out the loss of vTPM state upon start/stop of an instance the other day and I have something which resembles a patch. Do I need to file an issue first, or is it enough if I submit a patchset to gerrit?
15:34:00 bauzas thanks gibi
15:34:12 dansmith jssfr: B will probably warrant a spec
15:34:35 gibi jssfr: a) honestly I don't have a good answer for that. Uggla noted that it is ready for review and even pinged us downstream about it. I'm think most of our cores are busy with other review promises. Keep pinging us (I know it is tiring)
15:34:38 jssfr dansmith, uh-uh, how so?
15:35:08 jssfr gibi, oh I am fine with pinging, I just don't want to come off as annoying. :-)
15:35:13 dansmith jssfr: IIRC there are some gotchas around both vTPM and NVRAM state that cause us to dump them at certain times, and there's an overlap with the planned/ongoing vTPM live migration work
15:35:40 jssfr okay, that's good to know. fwiw, the patch is basically just passing VIR_DOMAIN_UNDEFINE_KEEP_TPM if destroy_secrets == False, which seems logical enough to me.
15:36:06 jssfr maybe it makes sense to submit the patch first and then you can still tell me it needs a spec :-).
15:36:22 jssfr (always easier to talk about an actual diff than about a hypothetical one, in my experience)
15:36:27 gibi jssfr: you need to find the balance of being seen but not beeing annoying :) Anyhow I can tell you we are aware of your patches waiting for review
15:36:39 jssfr (not my patches, but thanks :))
15:36:41 dansmith jssfr: there's no reason not to push the code. ever.
15:37:16 gibi (don't push a code if you feel it is a security bug :)
15:37:27 dansmith gibi: dammit gibi :)
15:37:59 jssfr fair point :D

Earlier   Later