Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-05-22
10:52:48 mikal So it works right now, but I am unclear if it is performant enough. The proxy is basically a proof of concept right now.
10:56:59 noonedeadpunk From my prespective separation of uwsgi API and worker itself would be really nice to have. Probably bercause too much got used to how openstack is written :)
10:57:55 mikal Ok, I'll make a note to circle back to that once I have a stronger view of if a rewrite is required or not.
10:58:02 noonedeadpunk As for instance we have quite established way of running wsgi apps, as well as wiring around TLS generation and deployment for it
10:58:36 mikal Basically the focus so far has been on getting a proxy to work at all, given the poor state of SPICE protocol documentation, and then on landing the supporting nova patches. I am only just now circling back to productionizing the proxy.
10:58:45 noonedeadpunk yeah, right
10:59:03 mikal I don't know much about uwsgi. If you have best practise recommendations I'd be quite interested in those.
10:59:07 noonedeadpunk it's not that I'm blaming or anything, just commenting :)
10:59:48 mikal Yeah its all good. I'm mildly excited someone is considering deploying this at all. Its sometimes lonely over here.
11:00:00 noonedeadpunk we have a uwsgi role we're actively using, which produced config file like this one: https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/templates/uwsgi.ini.j2
11:00:27 noonedeadpunk I totally see how useful this all might be in light of VMware migrations
11:01:00 mikal Yeah, I'm basically aiming at Citrix-like functionality here. Its just going to take a little while to get there.
11:01:11 sean-k-mooney noonedeadpunk: i dont even think you need to get partically fancy with the loadbalancing just have haproxxy or whatere use the souce ip/mac affiniteis the connetion to one of the active backends
11:01:16 noonedeadpunk but also this allows to secure communication with API side with TLS, as I think it's not covered now
11:01:29 sean-k-mooney noonedeadpunk: basiclly like you would for the novnc or html spice console
11:01:32 noonedeadpunk sean-k-mooney: exactly what I was thinking about, yes
11:01:47 noonedeadpunk jsut not L7 but L4 I guess
11:01:56 mikal noonedeadpunk: Kerbside API TLS is done with a frontend like apache2, not natively. SPICE protocol TLS is done natively within Kerbside.
11:02:14 sean-k-mooney ya so use the source ip and tcp port to balance
11:02:20 sean-k-mooney if it fails over they will have to reconenct
11:02:26 noonedeadpunk right, but how you communicate between apache2 and Kerbside API
11:02:34 sean-k-mooney but i dotn see a way around that without clustering in kerbside
11:02:35 noonedeadpunk as it;s listening on 0.0.0.0 explicitly
11:02:44 mikal If you squint at it, Kerbside is sort of a layer 7 load balancer. Its the thing which understands the protocol well enough to direct traffic to the right backend.
11:02:58 noonedeadpunk sean-k-mooney: I think it's totally fine tbh and more then reasonable
11:03:19 mikal noonedeadpunk: that's just the default in the config, you can override it.
11:03:27 noonedeadpunk mikal: um, can you? https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L56
11:03:45 noonedeadpunk and if that is a bare metal host and not docker container?
11:04:03 mikal noonedeadpunk: yeah, that whole template can be overridden by exporting KERBSIDE_API_COMMAND_LINE in the starting environment.
11:04:15 noonedeadpunk ah
11:04:17 noonedeadpunk ok
11:04:23 sean-k-mooney its just a wsgi app by the looks of it
11:04:33 noonedeadpunk so I can also set `echo ''` to it kinda?
11:04:37 noonedeadpunk it is
11:04:49 noonedeadpunk jsut started by the main service
11:04:53 mikal noonedeadpunk: I don't follow the echo comment?
11:05:28 sean-k-mooney mikal: are you using https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L54 to gengerate a script to run kerbside
11:05:29 noonedeadpunk so if I want to run wsgi separately, I shoudl be able to replace KERBSIDE_API_COMMAND_LINE with noop command quite successfully I assiume
11:05:33 sean-k-mooney as in a console script
11:05:45 sean-k-mooney or are you using that to spwan serpte isntance of kerbside per conection
11:05:47 mikal noonedeadpunk: secretly, this is just a cut down (and a bit older) version of https://github.com/shakenfist/shakenfist/blob/develop/shakenfist/config.py which is the project that code was lifted from.
11:06:01 mikal Oh interesting.
11:06:38 mikal Like I said, I don't know much about uwsgi. If flask can work in that mode then I think that would work?
11:07:02 sean-k-mooney noonedeadpunk: https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L566-L568
11:07:06 mikal The protocol native proxy is a separate socket server.
11:07:21 noonedeadpunk usually gunicorn, uwsgi and mod_wsgi are interchangable
11:07:36 sean-k-mooney mikal: so your api consol script currently pawans a wsgi server form api.py
11:08:17 sean-k-mooney but normlaly you woudl want to spin up the wsgi appolcation directly with a webserver and not spawn it with Popen like that if it can be shared between conenctions
11:08:30 noonedeadpunk and then if your intention to have apache2 in front for TLS, using mod_wsgi might be even more wise
11:08:34 sean-k-mooney if ithis is one process per api request however that is differnt
11:08:59 mikal The Shaken Fist Foundation [1] welcomes your feedback and potential contributions.
11:08:59 mikal 1: Said foundation does not exist.
11:09:58 sean-k-mooney mikal: without deep diving i think noonedeadpunk could ignore your main an api.py
11:10:11 mikal I think what I am getting from this conversation is that I should go read more about uwsgi, which I pinky swear to do.
11:10:24 sean-k-mooney and just spin up the wsig app directly and as they said use apache in front to handell tls
11:10:48 sean-k-mooney wsgi and uwsgi are also two diffent things
11:11:22 sean-k-mooney uwsgi, gurnicorn, and mod_wsgi are 3 examples wor wsgi applcation servers
11:11:37 sean-k-mooney as in they host a wsgi applcation
11:11:44 sean-k-mooney wsgi its self is a protocol
11:12:03 noonedeadpunk yeah, so usually you just don;t need to spawn gunicorn as part of the service, but do that separately
11:12:11 noonedeadpunk and leave user to choose wsgi server
11:12:41 mikal Ok, I will put reading about all this on my todo list for tomorrow.
11:12:42 noonedeadpunk as some might want to run granian or smth like that even
11:13:02 mikal Although, writing load testing tooling sounds like more fun ;)
11:13:30 sean-k-mooney so mikal what actully happenign when you creat taht subprcess is your stattign a wsgi sever that is importing https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L39-L42
11:14:25 sean-k-mooney mikal: then en that applcation host is bidnign to the port in the temlsate and handeign the request ot of band of the kerbside command you use to run it
11:15:24 sean-k-mooney mikal: by the way the way you wrote looks liek you wanted it to work how nova-api worked usign the eventlet webserver whild also not using eventlet :)
11:16:11 sean-k-mooney i.e. you wanted to provide a simpel way to run it form the cli via the kerbside command generated by this entry point https://github.com/shakenfist/kerbside/blob/develop/setup.cfg#L24
11:16:23 mikal sean-k-mooney: Oh I was definitely thinking about nova-api when I wrote it, but it was also over four years ago when I wrote that which means its all long forgotten now.
11:17:40 sean-k-mooney mikal: kolla recently moved form apache with mod_wsi to uwsgi
11:18:17 sean-k-mooney you shoudl be able to skip the commdn without any code changes and just run that flask ap directly by provieing the moduel path like you did to grunicorn
11:18:34 mikal sean-k-mooney: that's probably good because this is a topic I am entirely not passionate about. I just want to do something people think doesn't suck and then move onto more interesting things.
11:19:03 sean-k-mooney the main benifity fo that approch is you dont need to do the proces management in your cli
11:19:13 mikal Like I want it to be performant and not make people angry, but it is not my calling.
11:19:15 sean-k-mooney you can leave it up to docker/systemd to do that for you
11:20:56 mikal Ok well I will play with this tomorrow. I am being assured it is past my bed time now. Feel free to file a github issue if you're keen, otherwise I'll just work through a fix and smash it in.
11:22:23 sean-k-mooney well none of this is a blcoerk on the nova side. but fore scalablity and security reasons it woudl be good to do when doing a production install
11:22:36 sean-k-mooney so its somethign that noonedeadpunk and the kolla folks wil care about
11:23:24 sean-k-mooney but its not liek its need a big code change at first glance
11:25:24 noonedeadpunk I can try to submit a report later today
11:25:34 noonedeadpunk I need to deal with failing migrations right now :(
11:31:34 mikal Thanks guys.
11:43:06 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754
12:39:11 opendevreview Merged openstack/nova stable/2025.1: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950213
13:52:02 opendevreview Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659
13:52:10 dansmith gibi: Uggla, per yesterday, some refinement ^
13:53:18 opendevreview Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659
14:08:48 opendevreview yaguang proposed openstack/nova stable/2024.2: Handle iso+gpt detections https://review.opendev.org/c/openstack/nova/+/943940
14:14:44 Uggla dansmith, +1 for the new pci.alias example.
14:33:36 opendevreview Merged openstack/nova stable/2025.1: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950214
15:23:02 opendevreview melanie witt proposed openstack/nova-specs master: Re-propose vTPM live migration https://review.opendev.org/c/openstack/nova-specs/+/947542
15:23:42 melwitt gibi: fixed the typo you found ^
15:26:25 melwitt dansmith, sean-k-mooney: the re-proposal for vtpm live migration spec is ready for review if you may be interested. I have noted the changes from the last approval in the commit message
15:34:03 opendevreview Fabian Wiesel proposed openstack/nova master: libvirt: Expose number of migration threads https://review.opendev.org/c/openstack/nova/+/950667
15:55:49 gibi melwitt: thanks
15:55:56 gibi I added my +2 back
16:03:51 dansmith melwitt: ack, I'm several layers deep in the stack right now
16:54:31 opendevreview Merged openstack/nova stable/2024.2: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950215
16:58:30 opendevreview Merged openstack/nova stable/2024.2: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950216
18:09:09 melwitt dansmith: ack, it is not urgent but it is there when you are ready

Earlier   Later