Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-05-22
10:29:11 sean-k-mooney well it has
10:29:13 sean-k-mooney 2025-05-22 06:55:49.882+0000: initiating migration
10:29:15 sean-k-mooney 2025-05-22 07:18:40.025+0000: shutting down, reason=crashed
10:29:17 noonedeadpunk shutting down, reason=migrated and next thing it crashed
10:29:22 opendevreview Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for sound model. https://review.opendev.org/c/openstack/nova/+/926126
10:29:22 opendevreview Michael Still proposed openstack/nova master: Implement sound model extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/940770
10:29:23 opendevreview Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for USB controller model. https://review.opendev.org/c/openstack/nova/+/927354
10:29:23 opendevreview Michael Still proposed openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643
10:29:51 noonedeadpunk ah, ok, I think it was first attempt on migrating
10:29:52 noonedeadpunk yeah
10:30:10 noonedeadpunk but it does not point to any reason....
10:31:33 mikal sean-k-mooney: the fourth patch lacks the compute protection, I'll do that tomorrow. So don't go mad and merge that one too early!
10:33:44 noonedeadpunk *spice-direct
10:34:39 sean-k-mooney noonedeadpunk: this all look very stnadard. not nessiarly optimised for performacne but this is a very typeical looking vm so you not enabling anyting that raises red falgs to me
10:35:58 mikal noonedeadpunk: So its sort of half done -- there are now Nova APIs to create a console of type spice-direct, with openstacksdk and python-openstackclient support. Sound and USB are still missing (that's the patches above), so if that's a thing you care about you might need to hold your breath a bit. I have patches for kolla-ansible to deploy it,
10:35:58 mikal but I am sure it has bugs like no HA -- its still a work in progress. My pain focus right now is working through load testing the SPICE native proxy so I can try and quantify if it needs tweaking and what scaling to recommend with it.
10:36:47 noonedeadpunk mikal: not specifically care about spice-direct itself, but more wanted to implement that for OSA as well
10:36:56 sean-k-mooney mikal: ack
10:37:26 mikal noonedeadpunk: so OSA would need to know how to deploy the proxy and to twiddle some Nova config options. The existing Kolla-Ansible patches demonstrate how to do that.
10:37:57 noonedeadpunk mikal: do you have link top patches somewhere nearby?
10:38:32 noonedeadpunk there's support for spice-html5 but direct looks quite different
10:38:48 mikal noonedeadpunk: this is the list of patches I am carrying against Kolla-Ansible right now -- https://github.com/shakenfist/kerbside-patches/blob/develop/kolla-ansible/ORDER. There are also some minor patches against Kolla itself at https://github.com/shakenfist/kerbside-patches/blob/develop/kolla/ORDER. I'm sorry that's not the most helpful way to
10:38:48 mikal express them but its what I have right now.
10:39:17 sean-k-mooney noonedeadpunk: it is its using a out of tree proyx that mikal is developing called kerbside that actully allows raw spice conenction form a client to the vm
10:39:21 mikal noonedeadpunk: HTML5 and spice-direct can coexist I think, as long as you don't turn on the new require_secure option. The HTML5 client doesn't support TLS.
10:39:25 sean-k-mooney the proxy makes that safe
10:39:37 sean-k-mooney at least in thory
10:39:57 mikal noonedeadpunk: the proxy is at https://github.com/shakenfist/kerbside for what its worth. Like I said, the proxy itself is still a bit of a work in progress.
10:40:39 noonedeadpunk mikal: ok, so proxy needs to be started on control side I assume?
10:41:02 noonedeadpunk and then nova should be configured to point to it?
10:42:07 mikal noonedeadpunk: yes. The proxy sits running all the time and needs to be network accessible to your users. Nova is configured with the public URL to the proxy, and generates console access URLs using that URL template when a user creates a spice-direct console and then requests access to it. The proxy uses the console auth token to turn that back
10:42:07 mikal into hypervisor / port and then proxies the SPICE traffic between the client and the hypervisor.
10:42:52 mikal noonedeadpunk: https://github.com/shakenfist/kerbside-patches/blob/develop/tools/test-console#L88 is a worked example of that with a terrible fake client.
10:43:03 noonedeadpunk so in theory, L4 balancing can be added on top for public access?
10:43:55 mikal noonedeadpunk: yes. I expect the proxy would be clustered and it would be fine with traffic hitting multiple proxies. I suspect you'd want to try and keep a single client limited to a single backend for failure domain reasons, but apart from that it should just work.
10:44:01 noonedeadpunk ok, I should try making some patches, looks quite doable so far
10:44:08 mikal noonedeadpunk: state is in a central mysql database much like nova does.
10:44:31 mikal noonedeadpunk: by far my biggest concern is if the proxy is performant enough, but that's why that's the bit I am focused on right now.
10:44:47 mikal sean-k-mooney should still merge my patches though ;)
10:45:27 mikal (oVirt does something similar to all this, but they use squid as the proxy which means the client needs to know which hypervisor and port to connect to because its basically just doing SSL CONNECTs under the hood).
10:47:03 noonedeadpunk mikal: um... is the config respecting only ENV vars?
10:48:48 mikal noonedeadpunk: so kerbside itself isn't very "openstacky". Its written more like other personal projects of mine. It expects its configuration in environment variables, which can be in a systemd envvar file if that is less messy. Kolla-Ansible likes these start scripts, which is why it is done like that for those patches. It wouldn't be too hard to
10:48:49 mikal support a more traditional config file as well, its all just pydantic under the hood -- https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py
10:49:43 noonedeadpunk systemd envvar works as well, ofc. was just asking if I'm not missing more "conventional" config :)
10:50:13 noonedeadpunk And then servcice itself launches gunicorn? meaning https://github.com/shakenfist/kerbside/blob/053d068b0cc5b25a6263fe39a3548ca985b972ee/kerbside/api.py#L555-L563
10:50:30 mikal noonedeadpunk: you didn't miss it. Pydantic can do YAML etc as well apparently https://docs.pydantic.dev/latest/concepts/pydantic_settings/#other-settings-source
10:50:36 mikal noonedeadpunk: correct.
10:50:40 noonedeadpunk so, like, it does not make sense to try to launch it with uWSGI or smth like that?
10:51:07 noonedeadpunk as if I can run service as wsgi - I'd rather do that...
10:51:24 mikal noonedeadpunk: maybe? The gunicorn bit is only for the REST API and these Nova generated access URLs, so its not particularly high traffic or anything. I am not opposed to supporting other runtimes, it just hasn't been a priority for me yet.
10:52:04 mikal noonedeadpunk: like I said my current focus is working through some sort of performance testing, because I'd say there's a 50% change kerbside is going to get a rewrite in golang or C or something.
10:52:24 noonedeadpunk right.
10:52:40 noonedeadpunk not sure if golang will help a lot, but yeah
10:52:48 mikal So it works right now, but I am unclear if it is performant enough. The proxy is basically a proof of concept right now.
10:56:59 noonedeadpunk From my prespective separation of uwsgi API and worker itself would be really nice to have. Probably bercause too much got used to how openstack is written :)
10:57:55 mikal Ok, I'll make a note to circle back to that once I have a stronger view of if a rewrite is required or not.
10:58:02 noonedeadpunk As for instance we have quite established way of running wsgi apps, as well as wiring around TLS generation and deployment for it
10:58:36 mikal Basically the focus so far has been on getting a proxy to work at all, given the poor state of SPICE protocol documentation, and then on landing the supporting nova patches. I am only just now circling back to productionizing the proxy.
10:58:45 noonedeadpunk yeah, right
10:59:03 mikal I don't know much about uwsgi. If you have best practise recommendations I'd be quite interested in those.
10:59:07 noonedeadpunk it's not that I'm blaming or anything, just commenting :)
10:59:48 mikal Yeah its all good. I'm mildly excited someone is considering deploying this at all. Its sometimes lonely over here.
11:00:00 noonedeadpunk we have a uwsgi role we're actively using, which produced config file like this one: https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/templates/uwsgi.ini.j2
11:00:27 noonedeadpunk I totally see how useful this all might be in light of VMware migrations
11:01:00 mikal Yeah, I'm basically aiming at Citrix-like functionality here. Its just going to take a little while to get there.
11:01:11 sean-k-mooney noonedeadpunk: i dont even think you need to get partically fancy with the loadbalancing just have haproxxy or whatere use the souce ip/mac affiniteis the connetion to one of the active backends
11:01:16 noonedeadpunk but also this allows to secure communication with API side with TLS, as I think it's not covered now
11:01:29 sean-k-mooney noonedeadpunk: basiclly like you would for the novnc or html spice console
11:01:32 noonedeadpunk sean-k-mooney: exactly what I was thinking about, yes
11:01:47 noonedeadpunk jsut not L7 but L4 I guess
11:01:56 mikal noonedeadpunk: Kerbside API TLS is done with a frontend like apache2, not natively. SPICE protocol TLS is done natively within Kerbside.
11:02:14 sean-k-mooney ya so use the source ip and tcp port to balance
11:02:20 sean-k-mooney if it fails over they will have to reconenct
11:02:26 noonedeadpunk right, but how you communicate between apache2 and Kerbside API
11:02:34 sean-k-mooney but i dotn see a way around that without clustering in kerbside
11:02:35 noonedeadpunk as it;s listening on 0.0.0.0 explicitly
11:02:44 mikal If you squint at it, Kerbside is sort of a layer 7 load balancer. Its the thing which understands the protocol well enough to direct traffic to the right backend.
11:02:58 noonedeadpunk sean-k-mooney: I think it's totally fine tbh and more then reasonable
11:03:19 mikal noonedeadpunk: that's just the default in the config, you can override it.
11:03:27 noonedeadpunk mikal: um, can you? https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L56
11:03:45 noonedeadpunk and if that is a bare metal host and not docker container?
11:04:03 mikal noonedeadpunk: yeah, that whole template can be overridden by exporting KERBSIDE_API_COMMAND_LINE in the starting environment.
11:04:15 noonedeadpunk ah
11:04:17 noonedeadpunk ok
11:04:23 sean-k-mooney its just a wsgi app by the looks of it
11:04:33 noonedeadpunk so I can also set `echo ''` to it kinda?
11:04:37 noonedeadpunk it is
11:04:49 noonedeadpunk jsut started by the main service
11:04:53 mikal noonedeadpunk: I don't follow the echo comment?
11:05:28 sean-k-mooney mikal: are you using https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L54 to gengerate a script to run kerbside
11:05:29 noonedeadpunk so if I want to run wsgi separately, I shoudl be able to replace KERBSIDE_API_COMMAND_LINE with noop command quite successfully I assiume
11:05:33 sean-k-mooney as in a console script
11:05:45 sean-k-mooney or are you using that to spwan serpte isntance of kerbside per conection
11:05:47 mikal noonedeadpunk: secretly, this is just a cut down (and a bit older) version of https://github.com/shakenfist/shakenfist/blob/develop/shakenfist/config.py which is the project that code was lifted from.
11:06:01 mikal Oh interesting.
11:06:38 mikal Like I said, I don't know much about uwsgi. If flask can work in that mode then I think that would work?
11:07:02 sean-k-mooney noonedeadpunk: https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L566-L568
11:07:06 mikal The protocol native proxy is a separate socket server.
11:07:21 noonedeadpunk usually gunicorn, uwsgi and mod_wsgi are interchangable
11:07:36 sean-k-mooney mikal: so your api consol script currently pawans a wsgi server form api.py

Earlier   Later