Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-07-03
14:30:35 opendevreview sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940
14:34:33 opendevreview sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940
14:43:56 sean-k-mooney Uggla: bauzas ^ the only open quesiton left in that version is triat or compute service bump. a triat is simpler in my view and will be a better ux but i am not going to update the spec until folks provide feed back. i need to go test somethign else for a bit
14:51:03 Uggla sean-k-mooney, I'm more in favor of trait, but I'm not sure I have rational behind that. I need to think about it.
14:51:54 sean-k-mooney i do it work for mixed clouds with diffent virt drivers
14:52:13 sean-k-mooney either differnt version or vmware + libvirt or libvirt + ironic ectra
14:52:27 sean-k-mooney we can do it th eother way but to me its harder to implement correctly
16:50:55 opendevreview Masahito Muroi proposed openstack/nova-specs master: Add spec for virtio-blk multiqueue extra_spec https://review.opendev.org/c/openstack/nova-specs/+/951636
16:55:07 masahito Thanks for the quick reviews. I narrow down the spec scope only supporting multiqueue to enable the incremental supports and to simplify the discussion.
16:58:40 sean-k-mooney masahito: im sorry that we didnt really have tiem to work on this mroe closely earlier in the cycle
16:59:59 sean-k-mooney masahito: im not entrily sure what the right thing to do is. technial the approval deadline is end of buisse today
17:00:46 sean-k-mooney ill try and take a look at your spec again shortly
17:00:55 sean-k-mooney btu im not sure if other will have time today
17:09:56 masahito it's ok. I understand the iothreads part needs extra discussion now because of the entire Nova consistency.
17:11:13 opendevreview Merged openstack/nova master: Use futurist for _get_default_green_pool() https://review.opendev.org/c/openstack/nova/+/948072
17:11:32 dansmith sean-k-mooney: my request is that instead of you proposing a competing spec, you propose a revision on top of masahito's for whatever you think needs to be different
17:11:41 masahito and if we keep talking the iothreads part we could merge it early timing of G development cycle even though the spec is slipped from F release.
17:11:51 sean-k-mooney dansmith: they are not descibign very diffent things
17:11:53 dansmith it will make it much easier to understand any differences and avoid the appearance that we're going to choose one or the other
17:11:59 dansmith sean-k-mooney: cool, then should be easy
17:12:22 sean-k-mooney dansmith: masahito spec if for block multi queue only and mine is only for iothreads
17:12:28 sean-k-mooney and the two compose together
17:13:04 sean-k-mooney dansmith: masahito's orginally did both in a more adanced way the either not propose
17:13:36 masahito ^ iothreads part means mapping disk to iothread.
17:14:13 dansmith sean-k-mooney: okay I only skimmed them but I thought yours was proposing both
17:15:12 dansmith multi-queue has the same concerns for blk as eth I assume, where you have to scale them appropriately without using up all the .. what is it, pci root ports or something?
17:15:12 sean-k-mooney no i was going to rely on the fact multi is enable by default by qemu for now and focus only on the iothrad part
17:15:27 sean-k-mooney no
17:15:31 masahito yup. sean's spec gives me clear view of road to goal.
17:15:59 sean-k-mooney so multi queue for blk is on by default but you get i think 1 queue per vcpu? masahito is that correct
17:16:17 sean-k-mooney dansmith: for networkign the problem was that on the host side all the quese were used by ovs
17:16:30 sean-k-mooney dansmith: but you and to enable them manually in the guest driver
17:16:53 sean-k-mooney about 2-3 year ago the virtio net driver in the guest was updated to enabel all the queue if the nic has more then one
17:17:13 sean-k-mooney so you dont actully need to do anyting to have netowrking work propelry with a moderen kernel
17:18:35 masahito right. 1 queue per vcpu is default from 5.2
17:19:12 dansmith okay so blk multiqueue on its own gets you more queues, but limited actual parallelism by the default of $vcpu threads to service them,
17:19:50 dansmith and iothreads lets you increase that past that number if you want, say 16 queues, 2 vcpus, and 16 io threads to serve those queues?
17:20:24 sean-k-mooney dansmith: no by default it create a virtual device with 1 queue pre vcpu but all the io is handel by the single emulator thread
17:20:41 sean-k-mooney dansmith: so the only thing it gives you is lock free reade/write in the guest kernel
17:20:52 sean-k-mooney if you mape one queue to each core or somethign like that
17:20:53 dansmith by default, I understand
17:21:18 dansmith er, well, okay,
17:21:40 dansmith by default one queue per vcpu, all funneling through a single thread shared with emulator right?
17:21:48 sean-k-mooney yep
17:22:00 dansmith so two devices, four vcpus means 8 queues for one thread shared with emulator to handle
17:22:09 sean-k-mooney yep
17:22:33 dansmith so even more iothreads with the default queues will improve performance
17:22:41 sean-k-mooney masahito can correct me if im wrong but what they ahve observied is if you have too many queue it does not actully help with perfoamce
17:23:20 dansmith I'm not sure why multiple queues per device would really help with performance if you don't increase threads, but maybe it's because block devices are scheduled and have barriers unlike nics?
17:23:27 sean-k-mooney so if you ahve a vm with 64vcpus and 4 qeues it can perfroae better the 64Qs in some cases
17:23:35 masahito right for disk performance, not network performance.
17:24:26 sean-k-mooney dansmith: masahito want to decouble the queu count form teh vcpu count manily to reduce the queu count on large vms and i gues increase it on really small ones
17:25:02 dansmith ah, okay I see that in the spec but I glossed over it before
17:25:04 sean-k-mooney but the reall perfromace uplift is expcted to come form the addtion of iothreads
17:25:15 masahito 2 or 4 queue with 1 emulator thread performance is better than 1 queue. But in case of large CPU flavor, 16 or more, the number of queues seems to be too much for single threads.
17:25:32 dansmith yeah okay, I had a different impression from reading these two things, but I see why we'd need both now
17:26:37 masahito The IO performance doesn't increase even though the number of queue is increased. 1 queue and 1 emulator thread is better performance than 16 queues and 1 emulator thread in case of 16 cores
17:28:29 sean-k-mooney so multi queu for disks results in one kernel vhost thread per queue if i recall, at least with ovs-dpdk it allwos more dpdk cores to process packtest as it did a rount robbin of the quce across teh aviabel dpdk cores.
17:28:44 sean-k-mooney but for block queue we dont get that automcati implict scaling
17:30:45 sean-k-mooney i think that partly becasue there is no vhost offload for block devices
17:31:31 sean-k-mooney the main benifit to havign a multi queu dis is it allwos the use of the multi quei disk io schduelr in the kernel to do io priotisitation
17:32:00 sean-k-mooney so you can use mq-deadline or bfq or kyber in the guest ectra
17:32:36 dansmith yeah, that's what I meant above in terms of benefit for the scheduled block requests
17:32:46 sean-k-mooney ack
17:33:30 dansmith sean-k-mooney: can we not combine these, at least at the trait layer, to allow advertising and requesting these blk queues and threads?
17:33:54 dansmith I know the threads are useful for more than just blk, but if we can assume that threads mean you can do the blk queue that might be nice
17:33:56 sean-k-mooney we can we can also combine the spec if we really want ot at this point
17:34:01 dansmith just hate the trait explosion we get
17:34:52 sean-k-mooney orgially masahito goal was much more advance then the combidnaiton of these two spec, we can still evlove to that goal but the intent was to reduce the scope to soemthign achivable this cycle
17:36:23 sean-k-mooney dansmith: if we can do both this cycle i dont se why we would need 2 traits
17:37:07 sean-k-mooney so if we put the multi queue enhacement on top of the iotreads one when we are writign the code for this i think we can resue it to mean both
17:37:25 dansmith yeah
17:37:26 sean-k-mooney the multi queu change will now be very small so im not worried about that missing
17:37:54 dansmith I don't really see how we can get these both merged this cycle at this point
17:38:07 dansmith I'm out for the next ten days starting tomorrow
17:38:11 dansmith I dunno how others feel
18:12:24 dansmith gibi: just noticed you rechecking for rabbit reasons again... any chance we're mucking with things that are causing us to stall a real thread or something?
18:12:44 dansmith I mean, I realize I should know as well as you, but..
18:12:53 dansmith are we seeing those same failures elsewhere?
18:13:28 clarkb dansmith: have a link to an example failure?
18:13:35 clarkb (I want to rule one thing out or in)
18:13:53 dansmith clarkb: https://review.opendev.org/c/openstack/nova/+/948079?tab=change-view-tab-header-zuul-results-summary
18:14:03 dansmith not sure which of those fails is the one he rechecked for
18:15:08 dansmith (man we have /got/ to squelch the verbose tracebacking from os-brick)
18:16:19 clarkb each of the three failures were multinode jobs and zuul-launcher (the new nodepool replacement) gave you mixed cloud provider nodes. Its only really supposed to do this when there is no other choice, but there was at least one bug causing it to do that too often that should be fixed now
18:16:39 clarkb those jobs ran ~5 hours ago so the fix for that particular problem should've been in place.
18:16:40 dansmith ah
18:16:55 clarkb But anyway if your multinode job can't handle that (due to ip addressing or latency or whatever) that could be a reason
18:19:27 dansmith so, nova officially doesn't support cross-site MQ, even though lots of people do it
18:19:40 dansmith redhat has strict latency requirements for when we allow people to do it
18:19:54 dansmith getting nodes from multiple providers would seem to make that kindof difficult
18:20:12 clarkb I've brought up that the mixed nodesets are still happening in #opendev as we're trying to track zuul-launcher behaviors that are unexpected or cause problems there
18:20:19 clarkb ya its not the default
18:20:36 dansmith ack
18:21:00 clarkb and I think the vast majority of the time it isn't happening that way (opendev's own multinode jobs testing deployment of our software is also sensitive to it and I noticed immediately when it was doing it too often. Since I think I'e only seen one failure on our side btu we also run fewer jobs than nova and have fewer changes)
18:24:36 dansmith in general, we're a lot more tolerant of it than I would have thought, so it's probably not really a problem in a lot of cases (and maybe good to be throwing some of it in there)
18:24:49 dansmith I'm just saying, it's definitely not what we expect to happen
18:25:40 dansmith looks like this is the fail he was referring to: https://zuul.opendev.org/t/openstack/build/66557ea92bae4e809c09526eba619c07/log/compute1/logs/screen-n-cpu.txt
18:25:42 dansmith which is just fully unable to talk to rabbit
18:25:59 clarkb which may be an issue of ip routing

Earlier   Later