| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-02-03 | |||
| 14:42:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Troubleshoot deadlocks in threading mode https://review.opendev.org/c/openstack/nova/+/975515 | |
| 14:50:10 | sean-k-mooney | melwitt: it looks like the xml we generate has also changed. | |
| 14:51:08 | sean-k-mooney | the last succesfull nova-next run the the xml looked like thsi https://paste.opendev.org/show/bSZXkYi0lSv491NEjhW0/ | |
| 14:51:52 | sean-k-mooney | so this could be a bug in a recent nova or os-brick change | |
| 14:54:21 | sean-k-mooney | os-brick was bumped 3 weesk ago https://github.com/openstack/requirements/commit/d2639f76ce38c8a14ff7bd00d913601963646b1d | |
| 14:54:32 | sean-k-mooney | the lass succesfuly nova-next was 2026-01-30 21:02:10 | |
| 14:54:44 | sean-k-mooney | so that implies somethign changed in nova | |
| 15:07:00 | sean-k-mooney | fun i foudn the code in nova that sets this and that has not chagne since it was intoduced 13 years ago :) the joys of latent bugs | |
| 15:07:16 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Use firmware auto-selection by libvirt https://review.opendev.org/c/openstack/nova/+/969132 | |
| 15:07:46 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/volume.py#L55-L62 | |
| 15:09:01 | sean-k-mooney | ok foudn the root cause | |
| 15:09:03 | sean-k-mooney | https://github.com/openstack/cinder/commit/47107b4c84b060be834d6f430d32c3ebcac7d35a | |
| 15:10:35 | yuval_ | gibi: the iothreads merge https://review.opendev.org/c/openstack/nova/+/939254 - is it happening for any vms I create? | |
| 15:11:09 | sean-k-mooney | yuval_: that on by default yes | |
| 15:11:16 | yuval_ | I see | |
| 15:11:16 | sean-k-mooney | for all new vms | |
| 15:11:39 | sean-k-mooney | its technially on unconditionally for all vms now | |
| 15:11:42 | yuval_ | so, I want to control how many cpu's are allocated for iothreading | |
| 15:11:51 | sean-k-mooney | we are not suprpoting that this cycle | |
| 15:11:55 | yuval_ | that is not possible | |
| 15:12:02 | yuval_ | I see | |
| 15:12:05 | sean-k-mooney | but that was going to be a disucssion topic of next steps | |
| 15:12:16 | yuval_ | I see | |
| 15:12:22 | sean-k-mooney | we will likely add a flavor extra spec of similar next cycle | |
| 15:12:38 | sean-k-mooney | there isnt currently an agreement on how that shoudl look | |
| 15:12:59 | sean-k-mooney | yuval_: do you have a prefered ux/workflow for this? | |
| 15:13:51 | yuval_ | so, currently what I done to make it work, I would edit the xml and choose which cpu allocated for it. but its nightmare. | |
| 15:14:07 | yuval_ | unmanageable | |
| 15:14:24 | sean-k-mooney | and also unsupproted upsteam | |
| 15:14:29 | yuval_ | yes yes | |
| 15:14:36 | sean-k-mooney | the 3 proposal are as follows | |
| 15:14:51 | sean-k-mooney | 1 add a simple hw:io_threads=n extra specs | |
| 15:15:09 | sean-k-mooney | 2 add the ablityto add an addtionl n io thread per cidner voume | |
| 15:15:35 | sean-k-mooney | 3 use libvirt native iothread pooling feature and add hw:io_thread_max extra spec instead | |
| 15:15:43 | sean-k-mooney | 4 somethimg more complicated | |
| 15:16:19 | sean-k-mooney | in all cases we would have nova select the cores for the tread to run on automaticly for you using the cpu_shared_set cores | |
| 15:16:33 | yuval_ | so 1 - we choose which cpu's will be allocated for iothreading. 3 - there is a libvirt feature that does that? | |
| 15:16:55 | sean-k-mooney | no 1 and 3 the value is just an integer so you chosoe the count | |
| 15:16:58 | sean-k-mooney | not the cores | |
| 15:17:14 | sean-k-mooney | https://libvirt.org/formatdomain.html#iothreads-allocation | |
| 15:17:34 | sean-k-mooney | and yes libvit has a feature fo iothread pooling | |
| 15:17:36 | sean-k-mooney | <iothread id="8" thread_pool_min="2" thread_pool_max="32"> | |
| 15:17:57 | sean-k-mooney | so it can be static or dynimc and you can hotplug/unplug iothread at runtime | |
| 15:18:46 | sean-k-mooney | so we could just add 1 per cidner voluem, even on attach/detach, or it could be a static value in teh flavor or a max value in the falvor with a min of 1 | |
| 15:19:20 | sean-k-mooney | for this cycle we just hardcode 1 and we wanted operator feedback before doing anything more complex | |
| 15:23:08 | fwiesel | gibi: I changed the vmwareapi CI to set the OS_NOVA_DISABLE_EVENTLET_PATCHING environment variable when there is a change under the eventlet-removal topic. | |
| 15:23:29 | fwiesel | You can see it here for the latest run: http://openstack-ci-logs.global.cloud.sap/openstack-nova-973468-bj769/index.html#file=devstack@n-cpu-1.service&highlight=11 | |
| 15:23:32 | yuval_ | sean-k-mooney: first of all that amazing | |
| 15:25:07 | yuval_ | but I am a bit confused | |
| 15:25:15 | yuval_ | what I know about is iothreads per disk | |
| 15:25:18 | yuval_ | <disk …> | |
| 15:25:18 | yuval_ | <driver name='qemu' cache=’none’ io=’native’ …> | |
| 15:25:18 | yuval_ | <iothreads> | |
| 15:25:18 | yuval_ | <iothread id='1'></iothread> | |
| 15:25:18 | yuval_ | <iothread id='2'></iothread> | |
| 15:25:19 | yuval_ | <iothread id='3'></iothread> | |
| 15:25:20 | yuval_ | <iothread id='4'></iothread> | |
| 15:25:24 | yuval_ | </iothreads> | |
| 15:30:19 | sean-k-mooney | so we are not currently proposting doign that maping in nova | |
| 15:30:34 | sean-k-mooney | we could but libvirt wil round robbin the disk across the thread iteslf | |
| 15:31:25 | gibi | fwiesel: thanks. I will let you know when I have a WIP patch for the deadlock issue | |
| 15:32:30 | yuval_ | so like just adding: <iothreads>4</iothreads> | |
| 15:33:02 | gibi | fwiesel: env var looks good | |
| 15:37:39 | sean-k-mooney | yuval_: yes | |
| 15:38:28 | yuval_ | do we know for sure the vm will use the iothreads without mapping them to a disk? | |
| 15:40:12 | sean-k-mooney | i belive it will but i have nto tested it | |
| 15:40:24 | sean-k-mooney | base on the docs i expect it shoul | |
| 15:47:04 | dansmith | melwitt: I +2d that prep-service-bump patch above the object but now that I'm thinking about it, I need to remind myself what the plan is for the dance we need to do when we enable that | |
| 15:47:18 | dansmith | we have a call coming up in a bit, so maybe we discuss after, but... | |
| 15:47:44 | dansmith | we're going to add the object field, but not have it used, we're going to add the decorator for the service bump, but no service bump, and then... | |
| 15:47:57 | dansmith | in the next patch we add host support but not deployment and bump the service version | |
| 15:48:55 | dansmith | that will open the flood gates, but only host will work | |
| 15:49:28 | dansmith | I see in the next patch that you remove deployment from the legacy decorator.. is that the way you're preventing it from working? | |
| 15:49:54 | dansmith | the problem with that is that API could be upgraded to the patch that supports both, but it has no way to check that compute has been upgraded to that point, | |
| 15:49:59 | dansmith | which I know is a small window, but.. | |
| 15:50:53 | gmaan | gibi: ack | |
| 15:51:00 | dansmith | I think we can either (a) move the service bump to deployment before we merge host, or we could bump twice.. the latter would sort of regress host support once the api is upgraded, but at least it's safe | |
| 15:51:08 | dansmith | hmm, missed a (b) in there | |
| 16:04:28 | sean-k-mooney | melwitt: so https://bugs.launchpad.net/cinder/+bug/2127196 was the tracker for enableing the geometry reporting in cidner that broke nova-next | |
| 16:06:18 | melwitt | dansmith: ok. yeah, that's how it's preventing deployment mode and that's a good point ... I feel like bumping once at deployment seems better but either way works for me | |
| 16:06:46 | melwitt | sean-k-mooney: ahh thanks! I was curious what caused it | |
| 16:07:17 | dansmith | melwitt: I think that's the cleanest thing, yeah. we could also go back and bump for host if we run out of time before deployment is ready and we want to make the release or something | |
| 16:07:42 | dansmith | I haven't looked at either in detail, and not at all for months so I don't even know, just saying | |
| 16:07:51 | melwitt | dansmith: that was my main concern heh. yeah that makes sense | |
| 16:13:15 | dansmith | melwitt: yeah, because I think host is the most useful of these modes so I definitely don't want to hold it hostage | |
| 16:13:56 | melwitt | ++ | |
| 16:38:26 | raineszm | sean-k-mooney: melwitt: sorry about that. I know you're looking at the issue on the Nova side, but I put up a patch to set the default behavior of disk_geometry to unset https://review.opendev.org/c/openstack/cinder/+/975540 | |
| 16:54:22 | sean-k-mooney | raineszm: i have had ai crete a fix on the nova side while i was on calls ill push it soon and iteragte on it | |
| 16:55:09 | raineszm | sean-k-mooney: ack. ping me if I can help with anything on my end. | |
| 16:55:42 | opendevreview | sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542 | |
| 16:55:59 | sean-k-mooney | raineszm: its failing 100% on the nova-next job so if ^ passes i think we are good | |
| 16:56:50 | sean-k-mooney | ill see if i can pull that into a devstack env and test it too seperately | |
| 17:07:58 | sean-k-mooney | that actully pretty good form a 1 shot promt https://paste.opendev.org/show/bJ8ytZ1Kb6LMmv9CCLe0/ ia sed for 2 commit but it was over egere and did it in 1 but ignoreing that it created a resonable plan and created a potially viable fix | |
| 17:13:04 | melwitt | thanks raineszm and sean-k-mooney. I'm sending a heads up email to openstack-discuss@ | |
| 17:13:48 | sean-k-mooney | oh ya thats a good idea to avoid rechecks | |
| 19:34:55 | sean-k-mooney | melwitt: https://review.opendev.org/c/openstack/nova/+/975542 passed nova-next if you have time to review | |
| 19:36:43 | melwitt | sean-k-mooney: will look, thanks | |
| 19:37:44 | sean-k-mooney | im going to finish for today shortly so if you or others want to update it go ahead otherwise ill take a look at any comments tomorrow | |
| 20:15:21 | opendevreview | Merged openstack/nova stable/2025.1: libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/958674 | |
| 22:07:34 | opendevreview | sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542 | |