| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-02-03 | |||
| 10:56:21 | gibi | gmaan: ^^ fixed up the wrapper based on your comments | |
| 11:14:30 | sean-k-mooney | gibi: thanks ill take a look shortly | |
| 11:31:42 | sean-k-mooney | gibi: replied inline | |
| 12:01:38 | opendevreview | sean mooney proposed openstack/nova master: enable tap creation in nova-live-migration https://review.opendev.org/c/openstack/nova/+/975500 | |
| 12:09:33 | sean-k-mooney | melwitt: so nova-next is using debian 13 to work around some qemu img issues with encypted volumes so it and the nova-alt-config and ceph jobs actully using a newer qemu then say tempest-integrated-comptue | |
| 12:09:59 | sean-k-mooney | the ‘Property scsi-block.physical_block_size not found’ might be related to that | |
| 12:10:27 | sean-k-mooney | i.e. some chagne in qemu/libvirt or perhaps we are testing a combination there that we dont have enabel elsehwere | |
| 12:10:37 | sean-k-mooney | im not sure what that might be in this case | |
| 12:19:02 | opendevreview | Merged openstack/nova stable/2024.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975167 | |
| 13:01:41 | sean-k-mooney | melwitt: here is the relevent data https://paste.opendev.org/show/b6sWRx6VMAdktImClzCD/ ill ask about it on the virt channel and see if it rigns a bell. it looks like there is a incompatiablty between the libvirt xml we gereate and the qemu command line args | |
| 13:01:59 | sean-k-mooney | the code on teh nova side for this looks like it has not chaged in 2+ years so this looks liek a qemu regression to me | |
| 13:31:05 | gibi | sean-k-mooney: OK, all looks good to me | |
| 14:25:22 | sean-k-mooney | melwitt: so this is a latent nova bug exposed by a regression in libvirt. https://github.com/libvirt/libvirt/commit/718721f0f9d33aa8abafd71bfb9ac1cfef95dc3e assuems physical_block_size is supproted for disk type = lun but it is not | |
| 14:25:24 | sean-k-mooney | https://issues.redhat.com/browse/RHEL-145937 | |
| 14:25:47 | sean-k-mooney | tracks fixing the libvirt validation to reject it. | |
| 14:26:12 | sean-k-mooney | but what that means for nova is we need to update the code to not generate <blockio logical_block_size='512' physical_block_size='512'/> for dev type = lun | |
| 14:27:10 | sean-k-mooney | when we use lun mode we are directly passhtihg though the underlying lun form the stroage server and we cant change thing like the physical_block_size the same way we cant set the serial of the exported lun. | |
| 14:27:41 | sean-k-mooney | so we will need to make this conditoinal just as we did for serial in https://review.opendev.org/c/openstack/nova/+/918089 | |
| 14:28:00 | sean-k-mooney | melwitt: ill write this up in a bug report adn add the gate blocker flag | |
| 14:42:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Troubleshoot deadlocks in threading mode https://review.opendev.org/c/openstack/nova/+/975515 | |
| 14:50:10 | sean-k-mooney | melwitt: it looks like the xml we generate has also changed. | |
| 14:51:08 | sean-k-mooney | the last succesfull nova-next run the the xml looked like thsi https://paste.opendev.org/show/bSZXkYi0lSv491NEjhW0/ | |
| 14:51:52 | sean-k-mooney | so this could be a bug in a recent nova or os-brick change | |
| 14:54:21 | sean-k-mooney | os-brick was bumped 3 weesk ago https://github.com/openstack/requirements/commit/d2639f76ce38c8a14ff7bd00d913601963646b1d | |
| 14:54:32 | sean-k-mooney | the lass succesfuly nova-next was 2026-01-30 21:02:10 | |
| 14:54:44 | sean-k-mooney | so that implies somethign changed in nova | |
| 15:07:00 | sean-k-mooney | fun i foudn the code in nova that sets this and that has not chagne since it was intoduced 13 years ago :) the joys of latent bugs | |
| 15:07:16 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Use firmware auto-selection by libvirt https://review.opendev.org/c/openstack/nova/+/969132 | |
| 15:07:46 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/volume.py#L55-L62 | |
| 15:09:01 | sean-k-mooney | ok foudn the root cause | |
| 15:09:03 | sean-k-mooney | https://github.com/openstack/cinder/commit/47107b4c84b060be834d6f430d32c3ebcac7d35a | |
| 15:10:35 | yuval_ | gibi: the iothreads merge https://review.opendev.org/c/openstack/nova/+/939254 - is it happening for any vms I create? | |
| 15:11:09 | sean-k-mooney | yuval_: that on by default yes | |
| 15:11:16 | yuval_ | I see | |
| 15:11:16 | sean-k-mooney | for all new vms | |
| 15:11:39 | sean-k-mooney | its technially on unconditionally for all vms now | |
| 15:11:42 | yuval_ | so, I want to control how many cpu's are allocated for iothreading | |
| 15:11:51 | sean-k-mooney | we are not suprpoting that this cycle | |
| 15:11:55 | yuval_ | that is not possible | |
| 15:12:02 | yuval_ | I see | |
| 15:12:05 | sean-k-mooney | but that was going to be a disucssion topic of next steps | |
| 15:12:16 | yuval_ | I see | |
| 15:12:22 | sean-k-mooney | we will likely add a flavor extra spec of similar next cycle | |
| 15:12:38 | sean-k-mooney | there isnt currently an agreement on how that shoudl look | |
| 15:12:59 | sean-k-mooney | yuval_: do you have a prefered ux/workflow for this? | |
| 15:13:51 | yuval_ | so, currently what I done to make it work, I would edit the xml and choose which cpu allocated for it. but its nightmare. | |
| 15:14:07 | yuval_ | unmanageable | |
| 15:14:24 | sean-k-mooney | and also unsupproted upsteam | |
| 15:14:29 | yuval_ | yes yes | |
| 15:14:36 | sean-k-mooney | the 3 proposal are as follows | |
| 15:14:51 | sean-k-mooney | 1 add a simple hw:io_threads=n extra specs | |
| 15:15:09 | sean-k-mooney | 2 add the ablityto add an addtionl n io thread per cidner voume | |
| 15:15:35 | sean-k-mooney | 3 use libvirt native iothread pooling feature and add hw:io_thread_max extra spec instead | |
| 15:15:43 | sean-k-mooney | 4 somethimg more complicated | |
| 15:16:19 | sean-k-mooney | in all cases we would have nova select the cores for the tread to run on automaticly for you using the cpu_shared_set cores | |
| 15:16:33 | yuval_ | so 1 - we choose which cpu's will be allocated for iothreading. 3 - there is a libvirt feature that does that? | |
| 15:16:55 | sean-k-mooney | no 1 and 3 the value is just an integer so you chosoe the count | |
| 15:16:58 | sean-k-mooney | not the cores | |
| 15:17:14 | sean-k-mooney | https://libvirt.org/formatdomain.html#iothreads-allocation | |
| 15:17:34 | sean-k-mooney | and yes libvit has a feature fo iothread pooling | |
| 15:17:36 | sean-k-mooney | <iothread id="8" thread_pool_min="2" thread_pool_max="32"> | |
| 15:17:57 | sean-k-mooney | so it can be static or dynimc and you can hotplug/unplug iothread at runtime | |
| 15:18:46 | sean-k-mooney | so we could just add 1 per cidner voluem, even on attach/detach, or it could be a static value in teh flavor or a max value in the falvor with a min of 1 | |
| 15:19:20 | sean-k-mooney | for this cycle we just hardcode 1 and we wanted operator feedback before doing anything more complex | |
| 15:23:08 | fwiesel | gibi: I changed the vmwareapi CI to set the OS_NOVA_DISABLE_EVENTLET_PATCHING environment variable when there is a change under the eventlet-removal topic. | |
| 15:23:29 | fwiesel | You can see it here for the latest run: http://openstack-ci-logs.global.cloud.sap/openstack-nova-973468-bj769/index.html#file=devstack@n-cpu-1.service&highlight=11 | |
| 15:23:32 | yuval_ | sean-k-mooney: first of all that amazing | |
| 15:25:07 | yuval_ | but I am a bit confused | |
| 15:25:15 | yuval_ | what I know about is iothreads per disk | |
| 15:25:18 | yuval_ | <disk …> | |
| 15:25:18 | yuval_ | <driver name='qemu' cache=’none’ io=’native’ …> | |
| 15:25:18 | yuval_ | <iothreads> | |
| 15:25:18 | yuval_ | <iothread id='1'></iothread> | |
| 15:25:18 | yuval_ | <iothread id='2'></iothread> | |
| 15:25:19 | yuval_ | <iothread id='3'></iothread> | |
| 15:25:20 | yuval_ | <iothread id='4'></iothread> | |
| 15:25:24 | yuval_ | </iothreads> | |
| 15:30:19 | sean-k-mooney | so we are not currently proposting doign that maping in nova | |
| 15:30:34 | sean-k-mooney | we could but libvirt wil round robbin the disk across the thread iteslf | |
| 15:31:25 | gibi | fwiesel: thanks. I will let you know when I have a WIP patch for the deadlock issue | |
| 15:32:30 | yuval_ | so like just adding: <iothreads>4</iothreads> | |
| 15:33:02 | gibi | fwiesel: env var looks good | |
| 15:37:39 | sean-k-mooney | yuval_: yes | |
| 15:38:28 | yuval_ | do we know for sure the vm will use the iothreads without mapping them to a disk? | |
| 15:40:12 | sean-k-mooney | i belive it will but i have nto tested it | |
| 15:40:24 | sean-k-mooney | base on the docs i expect it shoul | |
| 15:47:04 | dansmith | melwitt: I +2d that prep-service-bump patch above the object but now that I'm thinking about it, I need to remind myself what the plan is for the dance we need to do when we enable that | |
| 15:47:18 | dansmith | we have a call coming up in a bit, so maybe we discuss after, but... | |
| 15:47:44 | dansmith | we're going to add the object field, but not have it used, we're going to add the decorator for the service bump, but no service bump, and then... | |
| 15:47:57 | dansmith | in the next patch we add host support but not deployment and bump the service version | |
| 15:48:55 | dansmith | that will open the flood gates, but only host will work | |
| 15:49:28 | dansmith | I see in the next patch that you remove deployment from the legacy decorator.. is that the way you're preventing it from working? | |
| 15:49:54 | dansmith | the problem with that is that API could be upgraded to the patch that supports both, but it has no way to check that compute has been upgraded to that point, | |
| 15:49:59 | dansmith | which I know is a small window, but.. | |
| 15:50:53 | gmaan | gibi: ack | |
| 15:51:00 | dansmith | I think we can either (a) move the service bump to deployment before we merge host, or we could bump twice.. the latter would sort of regress host support once the api is upgraded, but at least it's safe | |
| 15:51:08 | dansmith | hmm, missed a (b) in there | |
| 16:04:28 | sean-k-mooney | melwitt: so https://bugs.launchpad.net/cinder/+bug/2127196 was the tracker for enableing the geometry reporting in cidner that broke nova-next | |
| 16:06:18 | melwitt | dansmith: ok. yeah, that's how it's preventing deployment mode and that's a good point ... I feel like bumping once at deployment seems better but either way works for me | |
| 16:06:46 | melwitt | sean-k-mooney: ahh thanks! I was curious what caused it | |