| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-02 | |||
| 20:39:09 | clarkb | naively I'm not sure what the issue is. From the end user standpoint config drive is simple and bullet proof | |
| 20:40:26 | clarkb | the ability to mount iso9660 or fat32 devices is near univeral, the data is small and static. Using a complicated api service that requires hosts know how to magically negotiate a protocal more complicated than `mount` is always going to be more error prone. | |
| 20:40:53 | clarkb | iirc there are layers of proxies involved ont he backend too (which I want to say was the assumed source of the flakyness when we shifted all our ci nodes to config drive) | |
| 21:00:54 | mikal | I am not sure there is an issue, I am just surprised given config drive was meant to be a temporary fix which seems to have "rusted on". If it's genuinely around forever it could probably do with some love. | |
| 21:07:30 | clarkb | I think the primary downside to config drive is that it consumes one of your disk slots. I know this is a problem for Xen. Not sure about KVM | |
| 21:08:11 | clarkb | with Xen you only get something like 16 disks. One of which is implicitly owned by the root disk then config drive consumes a second. This limits you to 14 devices which may limit the total storage attachable to the instance if you also have volume size limits | |
| #openstack-nova - 2025-07-03 | |||
| 01:45:58 | sean-k-mooney[m] | i dont think we should supprot the dmi info at all | |
| 01:46:28 | sean-k-mooney[m] | its a libvirt internal implation detail so im really not a fan of building on that for config drive or any other metadata mechanium | |
| 01:47:12 | sean-k-mooney[m] | i certainly dont want to rush into adding supprot for this as a bug fix without propper discussion as this si an undocumjented internal detail | |
| 01:51:04 | sean-k-mooney[m] | if your using virtio-scsi you can have 100s of disks if needed but iconfig drive does consume a pci/pcie sloct by default but iits not generally a problem | |
| 06:43:33 | mikal | sean-k-mooney[m]: I finally have all the USB and sound changes passing CI, so that's nice. | |
| 09:00:03 | Uggla | sean-k-mooney[m], fyi I did a first review pass on 953940: support iothread for nova vms | https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 09:25:49 | bauzas | sean-k-mooney[m]: honestly, I'd prefer to leave the current owner to provide a new revision from him | |
| 09:54:34 | mikal | For what its worth, https://review.opendev.org/q/topic:%22libvirt-vdi%22 is a series of four patches all with passing CI and a +2 on them if someone has a spare minute or two. | |
| 10:01:26 | sean-k-mooney | bauzas: well i wanted to hevilaly revise there proposal, also i have propsoed thie in some form twice before so i wrote it form scratch based on my prior exploration of this in past cycles | |
| 10:38:14 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Don't create instance_extra entry for deleted instance https://review.opendev.org/c/openstack/nova/+/412771 | |
| 11:49:18 | opendevreview | Rajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show https://review.opendev.org/c/openstack/nova-specs/+/929780 | |
| 12:10:16 | opendevreview | sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 12:43:01 | opendevreview | Biser Milanov proposed openstack/nova master: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/918669 | |
| 12:43:31 | opendevreview | Biser Milanov proposed openstack/nova master: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/918669 | |
| 12:54:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode https://review.opendev.org/c/openstack/nova/+/951957 | |
| 12:54:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666 | |
| 12:54:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436 | |
| 12:54:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475 | |
| 12:54:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815 | |
| 12:59:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Do not yield in threading mode https://review.opendev.org/c/openstack/nova/+/950994 | |
| 12:59:20 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode https://review.opendev.org/c/openstack/nova/+/951957 | |
| 12:59:20 | opendevreview | Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666 | |
| 12:59:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436 | |
| 12:59:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475 | |
| 12:59:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815 | |
| 13:04:32 | sean-k-mooney | bauzas: can you provide input on specifcly this comment https://review.opendev.org/c/openstack/nova-specs/+/953940/comment/c0db3aad_9199e6e7/ do you want me to move hw:iothreads_per_disk to the alternitives or only update the iothread on reboot for this cycle. | |
| 13:04:53 | sean-k-mooney | ill simplfy the spec based on that input. | |
| 13:06:14 | sean-k-mooney | long term we defintly shoudl supprot hw:iothreads_per_disk btu we dont need to supprot it initally my expection is we will enhance the iothrad funciotnaltiy incremnetally over a few rleases as we need too. | |
| 13:08:31 | sean-k-mooney | the ohter thing i was undeceded on was shoudl it be hw:iothreads_per_disk or hw:iothreads_per_volume i.e. the number of addtional io treads to add per cinder volume or per any disk assocated with the guest i.e. swap or ephemreal | |
| 13:09:21 | sean-k-mooney | i think for simplictly to have a baseline this cycle we shoudl just simplfy and move all the per disk part to a diffent spec for next cycle like you were suggesting | |
| 13:15:46 | bauzas | sean-k-mooney: yeah my concern was about to say "given all the discussions we need to have for iothreads per disk, maybe we should just do that by another spec" | |
| 13:16:30 | bauzas | and my main concern with that is the fact that we need to restart an instance for using other io threads if we modify the number when we add a new volume | |
| 13:18:50 | sean-k-mooney | so we dont actuuly need the restart as far as i can tell but we can just update the xml we need to call a sperat api to add or remove the iothread | |
| 13:19:31 | sean-k-mooney | so ok ill move alll fo that to alterniivs and simplfy the spec to only static iothread set via the hw:iothreads extra spec | |
| 13:20:19 | bauzas | oh, the libvirt API accepts a specific call to add a new iothread ? | |
| 13:20:44 | bauzas | if so, tell that please :) | |
| 13:21:13 | sean-k-mooney | so you can do it on the cli to a live domain. i commented on that in my reply | |
| 13:21:29 | sean-k-mooney | so there is an api but i need to go find what virsh is calling | |
| 13:21:47 | sean-k-mooney | but we do need to track/assign the iothread id to use it | |
| 13:22:02 | sean-k-mooney | so we can use it but its more complexity then i would ike to do this cycle | |
| 13:29:10 | sean-k-mooney | im going to simplfy the cpu affintiy too. | |
| 13:51:30 | opendevreview | sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 14:05:50 | opendevreview | Rajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show https://review.opendev.org/c/openstack/nova-specs/+/929780 | |
| 14:30:35 | opendevreview | sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 14:34:33 | opendevreview | sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 14:43:56 | sean-k-mooney | Uggla: bauzas ^ the only open quesiton left in that version is triat or compute service bump. a triat is simpler in my view and will be a better ux but i am not going to update the spec until folks provide feed back. i need to go test somethign else for a bit | |
| 14:51:03 | Uggla | sean-k-mooney, I'm more in favor of trait, but I'm not sure I have rational behind that. I need to think about it. | |
| 14:51:54 | sean-k-mooney | i do it work for mixed clouds with diffent virt drivers | |
| 14:52:13 | sean-k-mooney | either differnt version or vmware + libvirt or libvirt + ironic ectra | |
| 14:52:27 | sean-k-mooney | we can do it th eother way but to me its harder to implement correctly | |
| 16:50:55 | opendevreview | Masahito Muroi proposed openstack/nova-specs master: Add spec for virtio-blk multiqueue extra_spec https://review.opendev.org/c/openstack/nova-specs/+/951636 | |
| 16:55:07 | masahito | Thanks for the quick reviews. I narrow down the spec scope only supporting multiqueue to enable the incremental supports and to simplify the discussion. | |
| 16:58:40 | sean-k-mooney | masahito: im sorry that we didnt really have tiem to work on this mroe closely earlier in the cycle | |
| 16:59:59 | sean-k-mooney | masahito: im not entrily sure what the right thing to do is. technial the approval deadline is end of buisse today | |
| 17:00:46 | sean-k-mooney | ill try and take a look at your spec again shortly | |
| 17:00:55 | sean-k-mooney | btu im not sure if other will have time today | |
| 17:09:56 | masahito | it's ok. I understand the iothreads part needs extra discussion now because of the entire Nova consistency. | |
| 17:11:13 | opendevreview | Merged openstack/nova master: Use futurist for _get_default_green_pool() https://review.opendev.org/c/openstack/nova/+/948072 | |
| 17:11:32 | dansmith | sean-k-mooney: my request is that instead of you proposing a competing spec, you propose a revision on top of masahito's for whatever you think needs to be different | |
| 17:11:41 | masahito | and if we keep talking the iothreads part we could merge it early timing of G development cycle even though the spec is slipped from F release. | |
| 17:11:51 | sean-k-mooney | dansmith: they are not descibign very diffent things | |
| 17:11:53 | dansmith | it will make it much easier to understand any differences and avoid the appearance that we're going to choose one or the other | |
| 17:11:59 | dansmith | sean-k-mooney: cool, then should be easy | |
| 17:12:22 | sean-k-mooney | dansmith: masahito spec if for block multi queue only and mine is only for iothreads | |
| 17:12:28 | sean-k-mooney | and the two compose together | |
| 17:13:04 | sean-k-mooney | dansmith: masahito's orginally did both in a more adanced way the either not propose | |
| 17:13:36 | masahito | ^ iothreads part means mapping disk to iothread. | |
| 17:14:13 | dansmith | sean-k-mooney: okay I only skimmed them but I thought yours was proposing both | |
| 17:15:12 | sean-k-mooney | no i was going to rely on the fact multi is enable by default by qemu for now and focus only on the iothrad part | |
| 17:15:12 | dansmith | multi-queue has the same concerns for blk as eth I assume, where you have to scale them appropriately without using up all the .. what is it, pci root ports or something? | |
| 17:15:27 | sean-k-mooney | no | |
| 17:15:31 | masahito | yup. sean's spec gives me clear view of road to goal. | |
| 17:15:59 | sean-k-mooney | so multi queue for blk is on by default but you get i think 1 queue per vcpu? masahito is that correct | |
| 17:16:17 | sean-k-mooney | dansmith: for networkign the problem was that on the host side all the quese were used by ovs | |
| 17:16:30 | sean-k-mooney | dansmith: but you and to enable them manually in the guest driver | |
| 17:16:53 | sean-k-mooney | about 2-3 year ago the virtio net driver in the guest was updated to enabel all the queue if the nic has more then one | |
| 17:17:13 | sean-k-mooney | so you dont actully need to do anyting to have netowrking work propelry with a moderen kernel | |
| 17:18:35 | masahito | right. 1 queue per vcpu is default from 5.2 | |
| 17:19:12 | dansmith | okay so blk multiqueue on its own gets you more queues, but limited actual parallelism by the default of $vcpu threads to service them, | |
| 17:19:50 | dansmith | and iothreads lets you increase that past that number if you want, say 16 queues, 2 vcpus, and 16 io threads to serve those queues? | |
| 17:20:24 | sean-k-mooney | dansmith: no by default it create a virtual device with 1 queue pre vcpu but all the io is handel by the single emulator thread | |
| 17:20:41 | sean-k-mooney | dansmith: so the only thing it gives you is lock free reade/write in the guest kernel | |
| 17:20:52 | sean-k-mooney | if you mape one queue to each core or somethign like that | |
| 17:20:53 | dansmith | by default, I understand | |
| 17:21:18 | dansmith | er, well, okay, | |
| 17:21:40 | dansmith | by default one queue per vcpu, all funneling through a single thread shared with emulator right? | |
| 17:21:48 | sean-k-mooney | yep | |
| 17:22:00 | dansmith | so two devices, four vcpus means 8 queues for one thread shared with emulator to handle | |
| 17:22:09 | sean-k-mooney | yep | |
| 17:22:33 | dansmith | so even more iothreads with the default queues will improve performance | |
| 17:22:41 | sean-k-mooney | masahito can correct me if im wrong but what they ahve observied is if you have too many queue it does not actully help with perfoamce | |
| 17:23:20 | dansmith | I'm not sure why multiple queues per device would really help with performance if you don't increase threads, but maybe it's because block devices are scheduled and have barriers unlike nics? | |
| 17:23:27 | sean-k-mooney | so if you ahve a vm with 64vcpus and 4 qeues it can perfroae better the 64Qs in some cases | |
| 17:23:35 | masahito | right for disk performance, not network performance. | |