| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-12-04 | |||
| 19:18:52 | melwitt | in general I wasn't thinking config like this would be different than two computes having various config not matching but maybe none of the others affect xml | |
| 19:19:10 | sean-k-mooney | so if you enable it on one or both host it will give us interesting data | |
| 19:19:30 | melwitt | ok, I'll try it | |
| 19:20:06 | sean-k-mooney | well libvirt might not care im currnetly tying to see if this is confire a backend option or not | |
| 19:22:01 | sean-k-mooney | some thing it will happly allow change in a live migrtion and others it wont an i have never played with seting io="threads" vs io="native" ectra difently on a migration | |
| 19:23:20 | sean-k-mooney | melwitt: related question do we just let libvit choose for local disks or is that also hardcoded to native today | |
| 19:24:58 | melwitt | sean-k-mooney: yes local disks are left to default. this io=native hard-coded thing is only for iSCSI, NFS, and FC Cinder volumes | |
| 19:25:29 | sean-k-mooney | ack | |
| 19:25:59 | melwitt | it comes from this spec from mitaka https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/libvirt-aio-mode.html | |
| 19:27:11 | sean-k-mooney | ya i was asking ai about libvirt io modes and that actullythe firt link it found | |
| 19:27:53 | sean-k-mooney | so it seam like we did adopt the best practices that were recommend at the tiem and optied into native but we pessimsed some backend so the change makes sens | |
| 19:28:38 | melwitt | oh heh | |
| 19:29:24 | sean-k-mooney | number 2 was https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-blockio-io_mode which is a littel more useful but im actully surpised this is not coverd in the libvirt domain xml docs | |
| 19:29:41 | sean-k-mooney | they are useable pretty good at docuemtning the difent values | |
| 19:31:52 | melwitt | yeah I guess it only says "The optional io attribute controls specific policies on I/O; qemu guests support "threads" and "native" Since 0.8.8, io_uring Since 6.3.0 (QEMU 5.0)." in https://libvirt.org/formatdomain.html#hard-drives-floppy-disks-cdroms | |
| 19:32:57 | sean-k-mooney | ah yep just roudn it deep in the driver section | |
| 19:33:20 | sean-k-mooney | unfortuetly you cant deep link that far into the doc | |
| 19:33:44 | sean-k-mooney | io_uring support could be very interesting in teh future | |
| 19:34:14 | melwitt | yeah I wish their doc pages had more fine grain links | |
| 19:39:08 | sean-k-mooney | https://kb.blockbridge.com/technote/proxmox-aio-vs-iouring/ is interesting but i have not had time to read i tyet | |
| 19:39:46 | sean-k-mooney | finding 1 "IOThreads significantly improve performance for most workloads" well i guess its a good thing we are finally going to turn them on | |
| 19:42:02 | melwitt | haha yeah. I saw similar things when i first started reading up on io=native vs io=threads | |
| 19:42:39 | opendevreview | melanie witt proposed openstack/nova master: DNM test io=threads => io=native live migration https://review.opendev.org/c/openstack/nova/+/969844 | |
| 19:47:00 | gmaan | ratailor if you see it when online. I commented in tempest change. maybe you can run the same tests which is executing same action on isntance in your devstack end and see why finsih_time is None - https://review.opendev.org/c/openstack/tempest/+/953331 | |
| 19:47:23 | opendevreview | melanie witt proposed openstack/nova master: DNM test io=threads => io=native live migration https://review.opendev.org/c/openstack/nova/+/969844 | |
| 19:48:10 | gmaan | there should be difference between API, action you call via tempest test or osc but somewhere it is missing the recording of finish time. | |
| 19:52:12 | sean-k-mooney | gmaan: os that comment thread on the swap voluem one only show up on v2 for me | |
| 19:52:29 | sean-k-mooney | i didnt see that in the latest revsion at all when i went to revie it | |
| 19:53:51 | gmaan | sean-k-mooney: yeah, there are multiple way to do lile task state as melwitt suggested early or checking migration status or based on request (your latest comment). but we should not make RPC call whenever we have GET volume attachment call | |
| 19:54:27 | sean-k-mooney | ya i agree | |
| 19:54:43 | gmaan | and that is something we need to agree as design in spec and should not delay to implementation. | |
| 19:54:50 | sean-k-mooney | i just didnt see that rad on the spec yestready because the commens are not propagating the way i expect | |
| 19:54:55 | melwitt | gmaan: yeah, thanks for commenting on that spec. it seemed crazy to me to do the RPC call about swap progress every time when 99% of the time no swap would be in progress | |
| 19:55:06 | gmaan | true | |
| 19:56:09 | gmaan | sean-k-mooney: comment propagation in gerrit is really weird. i hate that many times :) | |
| 19:58:08 | gmaan | last spec I am going through today is rajesh finish_time one, and previous comments are all lost there either due to Gerrit or they are marked resolved before agreement was reached. that is adding more time to review specs | |
| 20:02:20 | sean-k-mooney | ya i normlly try not to mark thing resovled unlless i made the change that was requested | |
| 20:02:59 | sean-k-mooney | even then i generaly prefer for non trivial thing if the orginal review acks it intesad of me | |
| 20:04:15 | gmaan | yeah | |
| 21:26:32 | opendevreview | melanie witt proposed openstack/nova master: DNM test io=threads => io=native live migration https://review.opendev.org/c/openstack/nova/+/969844 | |
| 21:27:06 | opendevreview | melanie witt proposed openstack/nova master: DNM test io=threads => io=native live migration https://review.opendev.org/c/openstack/nova/+/969844 | |
| 21:31:06 | opendevreview | melanie witt proposed openstack/nova master: DNM test io=threads => io=native live migration https://review.opendev.org/c/openstack/nova/+/969844 | |
| 22:10:57 | opendevreview | melanie witt proposed openstack/nova master: Make QEMU_IMG_LIMITS process limits configurable https://review.opendev.org/c/openstack/nova/+/969538 | |
| 22:30:45 | melwitt | sean-k-mooney: fyi the DNM patch test run finished -- nova-live-migration passes if [libvirt]use_default_aio_mode_for_volumes = true on the subnode only https://review.opendev.org/c/openstack/nova/+/969844 | |
| 22:33:29 | melwitt | looks like live_migrate_back_and_forth was not enabled though | |
| 22:35:31 | opendevreview | melanie witt proposed openstack/nova master: DNM test io=threads => io=native live migration https://review.opendev.org/c/openstack/nova/+/969844 | |
| 22:35:50 | melwitt | running again with live_migrate_back_and_forth = true ^ | |
| 22:37:35 | opendevreview | melanie witt proposed openstack/nova master: Call volume detach rollback API if detach fails https://review.opendev.org/c/openstack/nova/+/880399 | |
| 22:37:35 | opendevreview | melanie witt proposed openstack/nova master: Reproducer for bug 2016173 https://review.opendev.org/c/openstack/nova/+/946222 | |
| 23:31:32 | opendevreview | melanie witt proposed openstack/nova master: DNM test io=threads => io=native live migration https://review.opendev.org/c/openstack/nova/+/969844 | |
| #openstack-nova - 2025-12-05 | |||
| 01:43:31 | melwitt | did anyone else stop receiving gerrit email notifications after about 4 hours ago? | |
| 05:20:31 | opendevreview | Merged openstack/nova master: libvirt: add configuration option for volume AIO mode https://review.opendev.org/c/openstack/nova/+/964848 | |
| 09:10:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Collect result of _live_migration_operation https://review.opendev.org/c/openstack/nova/+/969501 | |
| 09:10:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix missing guest.migrate(parallel=) mock arg https://review.opendev.org/c/openstack/nova/+/969503 | |
| 09:45:08 | gibi | Uggla: I'm not sure anything is left tirageing in https://etherpad.opendev.org/p/nova-bug-triage-roster#L24 Does that duplicted bug shows up in your query as untriaged? It should not. As it is connected to another bug as duplicate and that bug is fixed | |
| 09:53:33 | Uggla | gibi, if you set it as duplicate that's fine. It should not appear anymore. | |
| 10:10:50 | noonedeadpunk | hey folks! do you know if passing of any of these flags are supproted by nova? https://www.qemu.org/docs/master/system/i386/hyperv.html | |
| 10:11:07 | noonedeadpunk | I guess not, unless I'm missing a way to pass something completely arbitrary... | |
| 10:12:52 | gibi | Uggla: it is set as duplicate since 2025-07-29 hence my question | |
| 10:14:08 | Uggla | gibi, oh ok that's probably because I get the bugs from a previous doc I did, not fresh at all. Sorry about that. | |
| 10:14:48 | gibi | no worries | |
| 10:14:55 | gibi | thanks for organizing the triage | |
| 10:15:22 | gibi | noonedeadpunk: appreantly yes, based on the fields here we have set of enlightments https://github.com/openstack/nova/blob/1712ae48e3111e0a2c178b9d601572d624a56374/nova/virt/libvirt/config.py#L2964 | |
| 10:17:31 | gibi | noonedeadpunk: https://github.com/openstack/nova/blob/1712ae48e3111e0a2c178b9d601572d624a56374/nova/virt/libvirt/driver.py#L6678 you need set os_type | |
| 10:17:44 | noonedeadpunk | huh | |
| 10:17:52 | noonedeadpunk | Ok, so it's done on it's own, nice! | |
| 10:17:55 | noonedeadpunk | thanks gibi! | |
| 10:17:58 | gibi | https://docs.openstack.org/glance/latest/admin/useful-image-properties.html | |
| 10:18:51 | noonedeadpunk | yeah, I knew about os_type, but I didn't know about hidden logic behind it:) | |
| 10:19:35 | noonedeadpunk | will propose documentation update probably :) | |
| 10:19:58 | gibi | noonedeadpunk: thanks | |
| 10:32:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: Upgrade note for concurrency mode default change https://review.opendev.org/c/openstack/nova/+/969888 | |
| 10:33:49 | gibi | gmaan: sean-k-mooney: dansmith: ^^ this is what I come up with based on the discussion about the upgrade impact of eventlet removal and our default mode change in G | |
| 10:54:20 | opendevreview | Lajos Katona proposed openstack/nova master: WIP: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 11:00:46 | sean-k-mooney | gibi: im not sure i agree with the recommendation to decouple the cahnge in mode in gerneal but its a more reasonable recomendtion for slurp upgrades | |
| 11:01:49 | sean-k-mooney | i dont really think we shoudl be recommendign that operators every use the env var out side of optin in to threadign to test early | |
| 11:02:15 | sean-k-mooney | it does exist as an escape hatch if we messed up | |
| 11:03:10 | sean-k-mooney | so i dont object to it existing just that for me makeing the swtich trasnparent to the end user is very imporant but i get the performance tuning concern | |
| 11:03:54 | sean-k-mooney | gibi: do we know if any of the larger operators have started testign treading mode since the 2025.2 release? | |
| 11:15:53 | opendevreview | Will Szumski proposed openstack/nova master: Fix race betweem instance claim and anti affinity check https://review.opendev.org/c/openstack/nova/+/969894 | |
| 11:15:53 | opendevreview | Will Szumski proposed openstack/nova master: WIP: Adds regression test for bug LP#2132984 https://review.opendev.org/c/openstack/nova/+/969380 | |
| 12:14:49 | gibi | sean-k-mooney: I got no indication if any large operator testing or tested threading mode in nova | |
| 12:16:03 | gibi | sean-k-mooney: I think one of the main reason to have a way to switch between modes is to avoid big bang integration of the threading mode during upgrade. Especially due to the performance tuning needs. | |
| 12:16:46 | gibi | sean-k-mooney: if it helps then I can try to give an oslo config flag instead of a env variable, but I don't see a good ROI of that at the moment | |
| 12:17:32 | sean-k-mooney | so that was explcit not a goal for me because i saw it as a failure on our part if operatos needed to think about htis to that degree | |
| 12:17:54 | sean-k-mooney | ie.. the uptrade shoudl be tansparent becuase we shoudl get the default right before chanigng them | |
| 12:18:11 | sean-k-mooney | but ya i dont think we need to make it an oslo config | |
| 12:18:51 | sean-k-mooney | im just really hoping that most operator will upgrade form 2026.1 to 2027.1 and not notice that we changed the threadning mode | |
| 12:19:23 | gibi | the problem for me is that I cannot define good default that will work for differntly sized and shaped deployments | |
| 12:19:42 | gibi | in the past we used large numbers as pool size so that covered a wide range of deployments | |
| 12:19:53 | gibi | with threading we cannot use those large numbers any more without consequences | |
| 12:26:42 | sean-k-mooney | yep i understand. given we only supprot say 10 boot operation on a server or a limite number of migraton ectra i think the small numbers for the comptue are actully likely to be reasonble | |
| 12:27:09 | sean-k-mooney | for the api the scaling mode has alwasy been mainly via proceses, | |
| 12:27:25 | sean-k-mooney | the schduler and conductor are probaly the ones i woudl be most concerned with | |
| 12:28:14 | sean-k-mooney | both are horizontally scable and virtucally scalabel with workers but ya they may need different tuneing | |
| 12:28:57 | sean-k-mooney | gibi: i am quitely hopeful that we will see a similar uplift in performance that ironic saw although we may have been using less problematic patterns | |
| 12:58:15 | gibi | scatter gather across multi cell clouds will not scale by scaling the nova-api process | |
| 12:58:30 | gibi | it will only scale if you scale the number of threads close to the number of cells in the threadpool | |
| 14:16:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable mypy on nova/utils.py https://review.opendev.org/c/openstack/nova/+/969936 | |
| 15:01:07 | dansmith | bauzas: can you +W this trivial one-liner mock fix? https://review.opendev.org/c/openstack/nova/+/969503 | |