| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-02-03 | |||
| 01:36:43 | opendevreview | melanie witt proposed openstack/nova master: TPM: opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 01:36:43 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 03:35:09 | melwitt | I have noticed nova-next has been failing a lot on the master branch today https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master&skip=0 | |
| 03:37:34 | melwitt | error like this "Failed to start libvirt guest: libvirt.libvirtError: internal error: process exited while connecting to monitor: 2026-01-30T20:30:26.821021Z qemu-system-x86_64: -device {"driver":"scsi-block","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"share-rw":true,"drive":"libvirt-1-format","id":"ua-409f3368-f41a-4915-aa22-29dff53bf25b","bootindex":1,"logical_block_size":512,"physical_block_size":512}: Property | |
| 03:37:34 | melwitt | 'scsi-block.physical_block_size' not found" | |
| 08:51:36 | gibi | dansmith: sorry I was away | |
| 09:05:27 | opendevreview | Merged openstack/nova stable/2024.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975166 | |
| 10:46:55 | gibi | sean-k-mooney: left two questions in https://review.opendev.org/c/openstack/nova/+/973149 | |
| 10:56:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:56:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove spawn_after https://review.opendev.org/c/openstack/nova/+/975396 | |
| 10:56:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:56:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | |
| 10:56:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:56:21 | gibi | gmaan: ^^ fixed up the wrapper based on your comments | |
| 11:14:30 | sean-k-mooney | gibi: thanks ill take a look shortly | |
| 11:31:42 | sean-k-mooney | gibi: replied inline | |
| 12:01:38 | opendevreview | sean mooney proposed openstack/nova master: enable tap creation in nova-live-migration https://review.opendev.org/c/openstack/nova/+/975500 | |
| 12:09:33 | sean-k-mooney | melwitt: so nova-next is using debian 13 to work around some qemu img issues with encypted volumes so it and the nova-alt-config and ceph jobs actully using a newer qemu then say tempest-integrated-comptue | |
| 12:09:59 | sean-k-mooney | the ‘Property scsi-block.physical_block_size not found’ might be related to that | |
| 12:10:27 | sean-k-mooney | i.e. some chagne in qemu/libvirt or perhaps we are testing a combination there that we dont have enabel elsehwere | |
| 12:10:37 | sean-k-mooney | im not sure what that might be in this case | |
| 12:19:02 | opendevreview | Merged openstack/nova stable/2024.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975167 | |
| 13:01:41 | sean-k-mooney | melwitt: here is the relevent data https://paste.opendev.org/show/b6sWRx6VMAdktImClzCD/ ill ask about it on the virt channel and see if it rigns a bell. it looks like there is a incompatiablty between the libvirt xml we gereate and the qemu command line args | |
| 13:01:59 | sean-k-mooney | the code on teh nova side for this looks like it has not chaged in 2+ years so this looks liek a qemu regression to me | |
| 13:31:05 | gibi | sean-k-mooney: OK, all looks good to me | |
| 14:25:22 | sean-k-mooney | melwitt: so this is a latent nova bug exposed by a regression in libvirt. https://github.com/libvirt/libvirt/commit/718721f0f9d33aa8abafd71bfb9ac1cfef95dc3e assuems physical_block_size is supproted for disk type = lun but it is not | |
| 14:25:24 | sean-k-mooney | https://issues.redhat.com/browse/RHEL-145937 | |
| 14:25:47 | sean-k-mooney | tracks fixing the libvirt validation to reject it. | |
| 14:26:12 | sean-k-mooney | but what that means for nova is we need to update the code to not generate <blockio logical_block_size='512' physical_block_size='512'/> for dev type = lun | |
| 14:27:10 | sean-k-mooney | when we use lun mode we are directly passhtihg though the underlying lun form the stroage server and we cant change thing like the physical_block_size the same way we cant set the serial of the exported lun. | |
| 14:27:41 | sean-k-mooney | so we will need to make this conditoinal just as we did for serial in https://review.opendev.org/c/openstack/nova/+/918089 | |
| 14:28:00 | sean-k-mooney | melwitt: ill write this up in a bug report adn add the gate blocker flag | |
| 14:42:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Troubleshoot deadlocks in threading mode https://review.opendev.org/c/openstack/nova/+/975515 | |
| 14:50:10 | sean-k-mooney | melwitt: it looks like the xml we generate has also changed. | |
| 14:51:08 | sean-k-mooney | the last succesfull nova-next run the the xml looked like thsi https://paste.opendev.org/show/bSZXkYi0lSv491NEjhW0/ | |
| 14:51:52 | sean-k-mooney | so this could be a bug in a recent nova or os-brick change | |
| 14:54:21 | sean-k-mooney | os-brick was bumped 3 weesk ago https://github.com/openstack/requirements/commit/d2639f76ce38c8a14ff7bd00d913601963646b1d | |
| 14:54:32 | sean-k-mooney | the lass succesfuly nova-next was 2026-01-30 21:02:10 | |
| 14:54:44 | sean-k-mooney | so that implies somethign changed in nova | |
| 15:07:00 | sean-k-mooney | fun i foudn the code in nova that sets this and that has not chagne since it was intoduced 13 years ago :) the joys of latent bugs | |
| 15:07:16 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Use firmware auto-selection by libvirt https://review.opendev.org/c/openstack/nova/+/969132 | |
| 15:07:46 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/volume.py#L55-L62 | |
| 15:09:01 | sean-k-mooney | ok foudn the root cause | |
| 15:09:03 | sean-k-mooney | https://github.com/openstack/cinder/commit/47107b4c84b060be834d6f430d32c3ebcac7d35a | |
| 15:10:35 | yuval_ | gibi: the iothreads merge https://review.opendev.org/c/openstack/nova/+/939254 - is it happening for any vms I create? | |
| 15:11:09 | sean-k-mooney | yuval_: that on by default yes | |
| 15:11:16 | yuval_ | I see | |
| 15:11:16 | sean-k-mooney | for all new vms | |
| 15:11:39 | sean-k-mooney | its technially on unconditionally for all vms now | |
| 15:11:42 | yuval_ | so, I want to control how many cpu's are allocated for iothreading | |
| 15:11:51 | sean-k-mooney | we are not suprpoting that this cycle | |
| 15:11:55 | yuval_ | that is not possible | |
| 15:12:02 | yuval_ | I see | |
| 15:12:05 | sean-k-mooney | but that was going to be a disucssion topic of next steps | |
| 15:12:16 | yuval_ | I see | |
| 15:12:22 | sean-k-mooney | we will likely add a flavor extra spec of similar next cycle | |
| 15:12:38 | sean-k-mooney | there isnt currently an agreement on how that shoudl look | |
| 15:12:59 | sean-k-mooney | yuval_: do you have a prefered ux/workflow for this? | |
| 15:13:51 | yuval_ | so, currently what I done to make it work, I would edit the xml and choose which cpu allocated for it. but its nightmare. | |
| 15:14:07 | yuval_ | unmanageable | |
| 15:14:24 | sean-k-mooney | and also unsupproted upsteam | |
| 15:14:29 | yuval_ | yes yes | |
| 15:14:36 | sean-k-mooney | the 3 proposal are as follows | |
| 15:14:51 | sean-k-mooney | 1 add a simple hw:io_threads=n extra specs | |
| 15:15:09 | sean-k-mooney | 2 add the ablityto add an addtionl n io thread per cidner voume | |
| 15:15:35 | sean-k-mooney | 3 use libvirt native iothread pooling feature and add hw:io_thread_max extra spec instead | |
| 15:15:43 | sean-k-mooney | 4 somethimg more complicated | |
| 15:16:19 | sean-k-mooney | in all cases we would have nova select the cores for the tread to run on automaticly for you using the cpu_shared_set cores | |
| 15:16:33 | yuval_ | so 1 - we choose which cpu's will be allocated for iothreading. 3 - there is a libvirt feature that does that? | |
| 15:16:55 | sean-k-mooney | no 1 and 3 the value is just an integer so you chosoe the count | |
| 15:16:58 | sean-k-mooney | not the cores | |
| 15:17:14 | sean-k-mooney | https://libvirt.org/formatdomain.html#iothreads-allocation | |
| 15:17:34 | sean-k-mooney | and yes libvit has a feature fo iothread pooling | |
| 15:17:36 | sean-k-mooney | <iothread id="8" thread_pool_min="2" thread_pool_max="32"> | |
| 15:17:57 | sean-k-mooney | so it can be static or dynimc and you can hotplug/unplug iothread at runtime | |
| 15:18:46 | sean-k-mooney | so we could just add 1 per cidner voluem, even on attach/detach, or it could be a static value in teh flavor or a max value in the falvor with a min of 1 | |
| 15:19:20 | sean-k-mooney | for this cycle we just hardcode 1 and we wanted operator feedback before doing anything more complex | |
| 15:23:08 | fwiesel | gibi: I changed the vmwareapi CI to set the OS_NOVA_DISABLE_EVENTLET_PATCHING environment variable when there is a change under the eventlet-removal topic. | |
| 15:23:29 | fwiesel | You can see it here for the latest run: http://openstack-ci-logs.global.cloud.sap/openstack-nova-973468-bj769/index.html#file=devstack@n-cpu-1.service&highlight=11 | |
| 15:23:32 | yuval_ | sean-k-mooney: first of all that amazing | |
| 15:25:07 | yuval_ | but I am a bit confused | |
| 15:25:15 | yuval_ | what I know about is iothreads per disk | |
| 15:25:18 | yuval_ | <disk …> | |
| 15:25:18 | yuval_ | <driver name='qemu' cache=’none’ io=’native’ …> | |
| 15:25:18 | yuval_ | <iothreads> | |
| 15:25:18 | yuval_ | <iothread id='1'></iothread> | |
| 15:25:18 | yuval_ | <iothread id='2'></iothread> | |
| 15:25:19 | yuval_ | <iothread id='3'></iothread> | |
| 15:25:20 | yuval_ | <iothread id='4'></iothread> | |
| 15:25:24 | yuval_ | </iothreads> | |
| 15:30:19 | sean-k-mooney | so we are not currently proposting doign that maping in nova | |
| 15:30:34 | sean-k-mooney | we could but libvirt wil round robbin the disk across the thread iteslf | |
| 15:31:25 | gibi | fwiesel: thanks. I will let you know when I have a WIP patch for the deadlock issue | |
| 15:32:30 | yuval_ | so like just adding: <iothreads>4</iothreads> | |
| 15:33:02 | gibi | fwiesel: env var looks good | |
| 15:37:39 | sean-k-mooney | yuval_: yes | |
| 15:38:28 | yuval_ | do we know for sure the vm will use the iothreads without mapping them to a disk? | |
| 15:40:12 | sean-k-mooney | i belive it will but i have nto tested it | |
| 15:40:24 | sean-k-mooney | base on the docs i expect it shoul | |
| 15:47:04 | dansmith | melwitt: I +2d that prep-service-bump patch above the object but now that I'm thinking about it, I need to remind myself what the plan is for the dance we need to do when we enable that | |