| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-03 | |||
| 12:59:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815 | |
| 13:04:32 | sean-k-mooney | bauzas: can you provide input on specifcly this comment https://review.opendev.org/c/openstack/nova-specs/+/953940/comment/c0db3aad_9199e6e7/ do you want me to move hw:iothreads_per_disk to the alternitives or only update the iothread on reboot for this cycle. | |
| 13:04:53 | sean-k-mooney | ill simplfy the spec based on that input. | |
| 13:06:14 | sean-k-mooney | long term we defintly shoudl supprot hw:iothreads_per_disk btu we dont need to supprot it initally my expection is we will enhance the iothrad funciotnaltiy incremnetally over a few rleases as we need too. | |
| 13:08:31 | sean-k-mooney | the ohter thing i was undeceded on was shoudl it be hw:iothreads_per_disk or hw:iothreads_per_volume i.e. the number of addtional io treads to add per cinder volume or per any disk assocated with the guest i.e. swap or ephemreal | |
| 13:09:21 | sean-k-mooney | i think for simplictly to have a baseline this cycle we shoudl just simplfy and move all the per disk part to a diffent spec for next cycle like you were suggesting | |
| 13:15:46 | bauzas | sean-k-mooney: yeah my concern was about to say "given all the discussions we need to have for iothreads per disk, maybe we should just do that by another spec" | |
| 13:16:30 | bauzas | and my main concern with that is the fact that we need to restart an instance for using other io threads if we modify the number when we add a new volume | |
| 13:18:50 | sean-k-mooney | so we dont actuuly need the restart as far as i can tell but we can just update the xml we need to call a sperat api to add or remove the iothread | |
| 13:19:31 | sean-k-mooney | so ok ill move alll fo that to alterniivs and simplfy the spec to only static iothread set via the hw:iothreads extra spec | |
| 13:20:19 | bauzas | oh, the libvirt API accepts a specific call to add a new iothread ? | |
| 13:20:44 | bauzas | if so, tell that please :) | |
| 13:21:13 | sean-k-mooney | so you can do it on the cli to a live domain. i commented on that in my reply | |
| 13:21:29 | sean-k-mooney | so there is an api but i need to go find what virsh is calling | |
| 13:21:47 | sean-k-mooney | but we do need to track/assign the iothread id to use it | |
| 13:22:02 | sean-k-mooney | so we can use it but its more complexity then i would ike to do this cycle | |
| 13:29:10 | sean-k-mooney | im going to simplfy the cpu affintiy too. | |
| 13:51:30 | opendevreview | sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 14:05:50 | opendevreview | Rajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show https://review.opendev.org/c/openstack/nova-specs/+/929780 | |
| 14:30:35 | opendevreview | sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 14:34:33 | opendevreview | sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940 | |
| 14:43:56 | sean-k-mooney | Uggla: bauzas ^ the only open quesiton left in that version is triat or compute service bump. a triat is simpler in my view and will be a better ux but i am not going to update the spec until folks provide feed back. i need to go test somethign else for a bit | |
| 14:51:03 | Uggla | sean-k-mooney, I'm more in favor of trait, but I'm not sure I have rational behind that. I need to think about it. | |
| 14:51:54 | sean-k-mooney | i do it work for mixed clouds with diffent virt drivers | |
| 14:52:13 | sean-k-mooney | either differnt version or vmware + libvirt or libvirt + ironic ectra | |
| 14:52:27 | sean-k-mooney | we can do it th eother way but to me its harder to implement correctly | |
| 16:50:55 | opendevreview | Masahito Muroi proposed openstack/nova-specs master: Add spec for virtio-blk multiqueue extra_spec https://review.opendev.org/c/openstack/nova-specs/+/951636 | |
| 16:55:07 | masahito | Thanks for the quick reviews. I narrow down the spec scope only supporting multiqueue to enable the incremental supports and to simplify the discussion. | |
| 16:58:40 | sean-k-mooney | masahito: im sorry that we didnt really have tiem to work on this mroe closely earlier in the cycle | |
| 16:59:59 | sean-k-mooney | masahito: im not entrily sure what the right thing to do is. technial the approval deadline is end of buisse today | |
| 17:00:46 | sean-k-mooney | ill try and take a look at your spec again shortly | |
| 17:00:55 | sean-k-mooney | btu im not sure if other will have time today | |
| 17:09:56 | masahito | it's ok. I understand the iothreads part needs extra discussion now because of the entire Nova consistency. | |
| 17:11:13 | opendevreview | Merged openstack/nova master: Use futurist for _get_default_green_pool() https://review.opendev.org/c/openstack/nova/+/948072 | |
| 17:11:32 | dansmith | sean-k-mooney: my request is that instead of you proposing a competing spec, you propose a revision on top of masahito's for whatever you think needs to be different | |
| 17:11:41 | masahito | and if we keep talking the iothreads part we could merge it early timing of G development cycle even though the spec is slipped from F release. | |
| 17:11:51 | sean-k-mooney | dansmith: they are not descibign very diffent things | |
| 17:11:53 | dansmith | it will make it much easier to understand any differences and avoid the appearance that we're going to choose one or the other | |
| 17:11:59 | dansmith | sean-k-mooney: cool, then should be easy | |
| 17:12:22 | sean-k-mooney | dansmith: masahito spec if for block multi queue only and mine is only for iothreads | |
| 17:12:28 | sean-k-mooney | and the two compose together | |
| 17:13:04 | sean-k-mooney | dansmith: masahito's orginally did both in a more adanced way the either not propose | |
| 17:13:36 | masahito | ^ iothreads part means mapping disk to iothread. | |
| 17:14:13 | dansmith | sean-k-mooney: okay I only skimmed them but I thought yours was proposing both | |
| 17:15:12 | dansmith | multi-queue has the same concerns for blk as eth I assume, where you have to scale them appropriately without using up all the .. what is it, pci root ports or something? | |
| 17:15:12 | sean-k-mooney | no i was going to rely on the fact multi is enable by default by qemu for now and focus only on the iothrad part | |
| 17:15:27 | sean-k-mooney | no | |
| 17:15:31 | masahito | yup. sean's spec gives me clear view of road to goal. | |
| 17:15:59 | sean-k-mooney | so multi queue for blk is on by default but you get i think 1 queue per vcpu? masahito is that correct | |
| 17:16:17 | sean-k-mooney | dansmith: for networkign the problem was that on the host side all the quese were used by ovs | |
| 17:16:30 | sean-k-mooney | dansmith: but you and to enable them manually in the guest driver | |
| 17:16:53 | sean-k-mooney | about 2-3 year ago the virtio net driver in the guest was updated to enabel all the queue if the nic has more then one | |
| 17:17:13 | sean-k-mooney | so you dont actully need to do anyting to have netowrking work propelry with a moderen kernel | |
| 17:18:35 | masahito | right. 1 queue per vcpu is default from 5.2 | |
| 17:19:12 | dansmith | okay so blk multiqueue on its own gets you more queues, but limited actual parallelism by the default of $vcpu threads to service them, | |
| 17:19:50 | dansmith | and iothreads lets you increase that past that number if you want, say 16 queues, 2 vcpus, and 16 io threads to serve those queues? | |
| 17:20:24 | sean-k-mooney | dansmith: no by default it create a virtual device with 1 queue pre vcpu but all the io is handel by the single emulator thread | |
| 17:20:41 | sean-k-mooney | dansmith: so the only thing it gives you is lock free reade/write in the guest kernel | |
| 17:20:52 | sean-k-mooney | if you mape one queue to each core or somethign like that | |
| 17:20:53 | dansmith | by default, I understand | |
| 17:21:18 | dansmith | er, well, okay, | |
| 17:21:40 | dansmith | by default one queue per vcpu, all funneling through a single thread shared with emulator right? | |
| 17:21:48 | sean-k-mooney | yep | |
| 17:22:00 | dansmith | so two devices, four vcpus means 8 queues for one thread shared with emulator to handle | |
| 17:22:09 | sean-k-mooney | yep | |
| 17:22:33 | dansmith | so even more iothreads with the default queues will improve performance | |
| 17:22:41 | sean-k-mooney | masahito can correct me if im wrong but what they ahve observied is if you have too many queue it does not actully help with perfoamce | |
| 17:23:20 | dansmith | I'm not sure why multiple queues per device would really help with performance if you don't increase threads, but maybe it's because block devices are scheduled and have barriers unlike nics? | |
| 17:23:27 | sean-k-mooney | so if you ahve a vm with 64vcpus and 4 qeues it can perfroae better the 64Qs in some cases | |
| 17:23:35 | masahito | right for disk performance, not network performance. | |
| 17:24:26 | sean-k-mooney | dansmith: masahito want to decouble the queu count form teh vcpu count manily to reduce the queu count on large vms and i gues increase it on really small ones | |
| 17:25:02 | dansmith | ah, okay I see that in the spec but I glossed over it before | |
| 17:25:04 | sean-k-mooney | but the reall perfromace uplift is expcted to come form the addtion of iothreads | |
| 17:25:15 | masahito | 2 or 4 queue with 1 emulator thread performance is better than 1 queue. But in case of large CPU flavor, 16 or more, the number of queues seems to be too much for single threads. | |
| 17:25:32 | dansmith | yeah okay, I had a different impression from reading these two things, but I see why we'd need both now | |
| 17:26:37 | masahito | The IO performance doesn't increase even though the number of queue is increased. 1 queue and 1 emulator thread is better performance than 16 queues and 1 emulator thread in case of 16 cores | |
| 17:28:29 | sean-k-mooney | so multi queu for disks results in one kernel vhost thread per queue if i recall, at least with ovs-dpdk it allwos more dpdk cores to process packtest as it did a rount robbin of the quce across teh aviabel dpdk cores. | |
| 17:28:44 | sean-k-mooney | but for block queue we dont get that automcati implict scaling | |
| 17:30:45 | sean-k-mooney | i think that partly becasue there is no vhost offload for block devices | |
| 17:31:31 | sean-k-mooney | the main benifit to havign a multi queu dis is it allwos the use of the multi quei disk io schduelr in the kernel to do io priotisitation | |
| 17:32:00 | sean-k-mooney | so you can use mq-deadline or bfq or kyber in the guest ectra | |
| 17:32:36 | dansmith | yeah, that's what I meant above in terms of benefit for the scheduled block requests | |
| 17:32:46 | sean-k-mooney | ack | |
| 17:33:30 | dansmith | sean-k-mooney: can we not combine these, at least at the trait layer, to allow advertising and requesting these blk queues and threads? | |
| 17:33:54 | dansmith | I know the threads are useful for more than just blk, but if we can assume that threads mean you can do the blk queue that might be nice | |
| 17:33:56 | sean-k-mooney | we can we can also combine the spec if we really want ot at this point | |
| 17:34:01 | dansmith | just hate the trait explosion we get | |
| 17:34:52 | sean-k-mooney | orgially masahito goal was much more advance then the combidnaiton of these two spec, we can still evlove to that goal but the intent was to reduce the scope to soemthign achivable this cycle | |
| 17:36:23 | sean-k-mooney | dansmith: if we can do both this cycle i dont se why we would need 2 traits | |
| 17:37:07 | sean-k-mooney | so if we put the multi queue enhacement on top of the iotreads one when we are writign the code for this i think we can resue it to mean both | |
| 17:37:25 | dansmith | yeah | |
| 17:37:26 | sean-k-mooney | the multi queu change will now be very small so im not worried about that missing | |
| 17:37:54 | dansmith | I don't really see how we can get these both merged this cycle at this point | |
| 17:38:07 | dansmith | I'm out for the next ten days starting tomorrow | |
| 17:38:11 | dansmith | I dunno how others feel | |
| 18:12:24 | dansmith | gibi: just noticed you rechecking for rabbit reasons again... any chance we're mucking with things that are causing us to stall a real thread or something? | |
| 18:12:44 | dansmith | I mean, I realize I should know as well as you, but.. | |
| 18:12:53 | dansmith | are we seeing those same failures elsewhere? | |
| 18:13:28 | clarkb | dansmith: have a link to an example failure? | |
| 18:13:35 | clarkb | (I want to rule one thing out or in) | |