Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-07-02
19:59:49 fungi a fifo-like proc interface in the kernel or something would be amazing
20:00:29 fungi and/or pseudofiles like device drivers tend to expose
20:00:32 mikal So qemu-agent uses a unix domain socket on the outside and virtio-serial on the inside and speaks JSON blobs, which is a bit terrible. Serial means only one talker at a time.
20:01:39 fungi though at boot time, one talker in the guest is generally sufficient?
20:01:48 mikal ShakenFist does this with a unix domain socket on the outside and virtio-vsock on the inside, and speaks protobufs. That's actually really cool if I don't say so myself because the in-guest agent can just run a socket server like you're used to and serve N requests at the same time because virtio-vsock does all the multiplexing for you.
20:01:59 fungi or maybe not these days, with the advent of parallel boot processing
20:02:40 mikal Serialized requests forces you to come up with some sort of queueing / locking mechanism to handle the case of multiplem requests, even if its not used much.
20:03:14 fungi the protobuf design sounds nice, if only protobuf libraries weren't such shifting sand these days (ut maybe they're getting better)
20:03:53 mikal https://www.madebymikal.com/virtio-vsock-python-examples-of-running-the-server-in-the-guest/ is what I wrote up when I was playing with virtio-vsock.
20:06:23 fungi yeah, i guess low-level socket access like that is probably enough for these cases, you don't need google's own protobuf libs
20:09:34 fungi oh, i see what you're saying, shakenfist tunnels protobufs over an interface like that
20:09:52 fungi you'd still need separate libs for the serialization i guess
20:12:10 mikal Yes, virtio-vsock is the transport, and I talk protobuf down it. Qemu talks JSON in its similar but slightly more terrible case.
20:12:21 mikal Notably I do not talk gRPC, just the serialization protocol.
20:12:39 mikal But meh, its just serialization. It could be JSON for all it matters.
20:12:59 mikal The in-guest agent could present a REST API for example over virtio-vsock and that would not be insane.
20:15:57 fungi makes sense, and yes grpc is a much bigger mess
20:17:34 mikal On reflection, I am also a bit confused about the "tangled python deependencies" answer to why glean is needed compared to cloud-init. Isn't that just a case of using venvs for the various competing components? Especially when modern pythons are super opposed to you installing directly into the system pip?
20:29:41 fungi devstack is finally in a position to be able to isolate openstack services from the system python module search path, but that's an extremely recent change
20:35:21 fungi i recall clarkb hacked on versions of that improvement for years before it reached a state others in the community were comfortable with
20:36:41 clarkb from the CI/OpenDev perspective I think there are two primary reasons we liek config drive. The first is that it is incredibly reliable. The boot error rate with metadata service is >0 and in some cases quite high. The other is that it is simple. Cloud-init configures a lot of stuff and sometimes in desctructive ways (it once decided to do bad things to volume mounts). Glean
20:36:43 clarkb sets up your network and ssh keys and that is about it
20:37:17 clarkb today we can only boot ubuntu noble in rackspace classic with config drive. Metadata does not work. This is with the upstream ubuntu noble cloud image which uses cloud init (not glean)
20:37:55 clarkb (that is because rackspace doesn't do metadata in the way normal clouds do iirc and they don't have their custom image for noble yet (or ever) so the only way it works with standard tools is config drive)
20:39:09 clarkb naively I'm not sure what the issue is. From the end user standpoint config drive is simple and bullet proof
20:40:26 clarkb the ability to mount iso9660 or fat32 devices is near univeral, the data is small and static. Using a complicated api service that requires hosts know how to magically negotiate a protocal more complicated than `mount` is always going to be more error prone.
20:40:53 clarkb iirc there are layers of proxies involved ont he backend too (which I want to say was the assumed source of the flakyness when we shifted all our ci nodes to config drive)
21:00:54 mikal I am not sure there is an issue, I am just surprised given config drive was meant to be a temporary fix which seems to have "rusted on". If it's genuinely around forever it could probably do with some love.
21:07:30 clarkb I think the primary downside to config drive is that it consumes one of your disk slots. I know this is a problem for Xen. Not sure about KVM
21:08:11 clarkb with Xen you only get something like 16 disks. One of which is implicitly owned by the root disk then config drive consumes a second. This limits you to 14 devices which may limit the total storage attachable to the instance if you also have volume size limits
#openstack-nova - 2025-07-03
01:45:58 sean-k-mooney[m] i dont think we should supprot the dmi info at all
01:46:28 sean-k-mooney[m] its a libvirt internal implation detail so im really not a fan of building on that for config drive or any other metadata mechanium
01:47:12 sean-k-mooney[m] i certainly dont want to rush into adding supprot for this as a bug fix without propper discussion as this si an undocumjented internal detail
01:51:04 sean-k-mooney[m] if your using virtio-scsi you can have 100s of disks if needed but iconfig drive does consume a pci/pcie sloct by default but iits not generally a problem
06:43:33 mikal sean-k-mooney[m]: I finally have all the USB and sound changes passing CI, so that's nice.
09:00:03 Uggla sean-k-mooney[m], fyi I did a first review pass on 953940: support iothread for nova vms | https://review.opendev.org/c/openstack/nova-specs/+/953940
09:25:49 bauzas sean-k-mooney[m]: honestly, I'd prefer to leave the current owner to provide a new revision from him
09:54:34 mikal For what its worth, https://review.opendev.org/q/topic:%22libvirt-vdi%22 is a series of four patches all with passing CI and a +2 on them if someone has a spare minute or two.
10:01:26 sean-k-mooney bauzas: well i wanted to hevilaly revise there proposal, also i have propsoed thie in some form twice before so i wrote it form scratch based on my prior exploration of this in past cycles
10:38:14 opendevreview Alexey Stupnikov proposed openstack/nova master: Don't create instance_extra entry for deleted instance https://review.opendev.org/c/openstack/nova/+/412771
11:49:18 opendevreview Rajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show https://review.opendev.org/c/openstack/nova-specs/+/929780
12:10:16 opendevreview sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940
12:43:01 opendevreview Biser Milanov proposed openstack/nova master: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/918669
12:43:31 opendevreview Biser Milanov proposed openstack/nova master: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/918669
12:54:22 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode https://review.opendev.org/c/openstack/nova/+/951957
12:54:22 opendevreview Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666
12:54:23 opendevreview Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436
12:54:23 opendevreview Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475
12:54:24 opendevreview Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815
12:59:19 opendevreview Balazs Gibizer proposed openstack/nova master: Do not yield in threading mode https://review.opendev.org/c/openstack/nova/+/950994
12:59:20 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode https://review.opendev.org/c/openstack/nova/+/951957
12:59:20 opendevreview Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666
12:59:21 opendevreview Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436
12:59:21 opendevreview Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475
12:59:22 opendevreview Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815
13:04:32 sean-k-mooney bauzas: can you provide input on specifcly this comment https://review.opendev.org/c/openstack/nova-specs/+/953940/comment/c0db3aad_9199e6e7/ do you want me to move hw:iothreads_per_disk to the alternitives or only update the iothread on reboot for this cycle.
13:04:53 sean-k-mooney ill simplfy the spec based on that input.
13:06:14 sean-k-mooney long term we defintly shoudl supprot hw:iothreads_per_disk btu we dont need to supprot it initally my expection is we will enhance the iothrad funciotnaltiy incremnetally over a few rleases as we need too.
13:08:31 sean-k-mooney the ohter thing i was undeceded on was shoudl it be hw:iothreads_per_disk or hw:iothreads_per_volume i.e. the number of addtional io treads to add per cinder volume or per any disk assocated with the guest i.e. swap or ephemreal
13:09:21 sean-k-mooney i think for simplictly to have a baseline this cycle we shoudl just simplfy and move all the per disk part to a diffent spec for next cycle like you were suggesting
13:15:46 bauzas sean-k-mooney: yeah my concern was about to say "given all the discussions we need to have for iothreads per disk, maybe we should just do that by another spec"
13:16:30 bauzas and my main concern with that is the fact that we need to restart an instance for using other io threads if we modify the number when we add a new volume
13:18:50 sean-k-mooney so we dont actuuly need the restart as far as i can tell but we can just update the xml we need to call a sperat api to add or remove the iothread
13:19:31 sean-k-mooney so ok ill move alll fo that to alterniivs and simplfy the spec to only static iothread set via the hw:iothreads extra spec
13:20:19 bauzas oh, the libvirt API accepts a specific call to add a new iothread ?
13:20:44 bauzas if so, tell that please :)
13:21:13 sean-k-mooney so you can do it on the cli to a live domain. i commented on that in my reply
13:21:29 sean-k-mooney so there is an api but i need to go find what virsh is calling
13:21:47 sean-k-mooney but we do need to track/assign the iothread id to use it
13:22:02 sean-k-mooney so we can use it but its more complexity then i would ike to do this cycle
13:29:10 sean-k-mooney im going to simplfy the cpu affintiy too.
13:51:30 opendevreview sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940
14:05:50 opendevreview Rajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show https://review.opendev.org/c/openstack/nova-specs/+/929780
14:30:35 opendevreview sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940
14:34:33 opendevreview sean mooney proposed openstack/nova-specs master: support iothread for nova vms https://review.opendev.org/c/openstack/nova-specs/+/953940
14:43:56 sean-k-mooney Uggla: bauzas ^ the only open quesiton left in that version is triat or compute service bump. a triat is simpler in my view and will be a better ux but i am not going to update the spec until folks provide feed back. i need to go test somethign else for a bit
14:51:03 Uggla sean-k-mooney, I'm more in favor of trait, but I'm not sure I have rational behind that. I need to think about it.
14:51:54 sean-k-mooney i do it work for mixed clouds with diffent virt drivers
14:52:13 sean-k-mooney either differnt version or vmware + libvirt or libvirt + ironic ectra
14:52:27 sean-k-mooney we can do it th eother way but to me its harder to implement correctly
16:50:55 opendevreview Masahito Muroi proposed openstack/nova-specs master: Add spec for virtio-blk multiqueue extra_spec https://review.opendev.org/c/openstack/nova-specs/+/951636
16:55:07 masahito Thanks for the quick reviews. I narrow down the spec scope only supporting multiqueue to enable the incremental supports and to simplify the discussion.
16:58:40 sean-k-mooney masahito: im sorry that we didnt really have tiem to work on this mroe closely earlier in the cycle
16:59:59 sean-k-mooney masahito: im not entrily sure what the right thing to do is. technial the approval deadline is end of buisse today
17:00:46 sean-k-mooney ill try and take a look at your spec again shortly
17:00:55 sean-k-mooney btu im not sure if other will have time today
17:09:56 masahito it's ok. I understand the iothreads part needs extra discussion now because of the entire Nova consistency.
17:11:13 opendevreview Merged openstack/nova master: Use futurist for _get_default_green_pool() https://review.opendev.org/c/openstack/nova/+/948072
17:11:32 dansmith sean-k-mooney: my request is that instead of you proposing a competing spec, you propose a revision on top of masahito's for whatever you think needs to be different
17:11:41 masahito and if we keep talking the iothreads part we could merge it early timing of G development cycle even though the spec is slipped from F release.
17:11:51 sean-k-mooney dansmith: they are not descibign very diffent things
17:11:53 dansmith it will make it much easier to understand any differences and avoid the appearance that we're going to choose one or the other
17:11:59 dansmith sean-k-mooney: cool, then should be easy
17:12:22 sean-k-mooney dansmith: masahito spec if for block multi queue only and mine is only for iothreads
17:12:28 sean-k-mooney and the two compose together
17:13:04 sean-k-mooney dansmith: masahito's orginally did both in a more adanced way the either not propose
17:13:36 masahito ^ iothreads part means mapping disk to iothread.
17:14:13 dansmith sean-k-mooney: okay I only skimmed them but I thought yours was proposing both
17:15:12 sean-k-mooney no i was going to rely on the fact multi is enable by default by qemu for now and focus only on the iothrad part
17:15:12 dansmith multi-queue has the same concerns for blk as eth I assume, where you have to scale them appropriately without using up all the .. what is it, pci root ports or something?

Earlier   Later