Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-05-22
10:43:55 mikal noonedeadpunk: yes. I expect the proxy would be clustered and it would be fine with traffic hitting multiple proxies. I suspect you'd want to try and keep a single client limited to a single backend for failure domain reasons, but apart from that it should just work.
10:44:01 noonedeadpunk ok, I should try making some patches, looks quite doable so far
10:44:08 mikal noonedeadpunk: state is in a central mysql database much like nova does.
10:44:31 mikal noonedeadpunk: by far my biggest concern is if the proxy is performant enough, but that's why that's the bit I am focused on right now.
10:44:47 mikal sean-k-mooney should still merge my patches though ;)
10:45:27 mikal (oVirt does something similar to all this, but they use squid as the proxy which means the client needs to know which hypervisor and port to connect to because its basically just doing SSL CONNECTs under the hood).
10:47:03 noonedeadpunk mikal: um... is the config respecting only ENV vars?
10:48:48 mikal noonedeadpunk: so kerbside itself isn't very "openstacky". Its written more like other personal projects of mine. It expects its configuration in environment variables, which can be in a systemd envvar file if that is less messy. Kolla-Ansible likes these start scripts, which is why it is done like that for those patches. It wouldn't be too hard to
10:48:49 mikal support a more traditional config file as well, its all just pydantic under the hood -- https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py
10:49:43 noonedeadpunk systemd envvar works as well, ofc. was just asking if I'm not missing more "conventional" config :)
10:50:13 noonedeadpunk And then servcice itself launches gunicorn? meaning https://github.com/shakenfist/kerbside/blob/053d068b0cc5b25a6263fe39a3548ca985b972ee/kerbside/api.py#L555-L563
10:50:30 mikal noonedeadpunk: you didn't miss it. Pydantic can do YAML etc as well apparently https://docs.pydantic.dev/latest/concepts/pydantic_settings/#other-settings-source
10:50:36 mikal noonedeadpunk: correct.
10:50:40 noonedeadpunk so, like, it does not make sense to try to launch it with uWSGI or smth like that?
10:51:07 noonedeadpunk as if I can run service as wsgi - I'd rather do that...
10:51:24 mikal noonedeadpunk: maybe? The gunicorn bit is only for the REST API and these Nova generated access URLs, so its not particularly high traffic or anything. I am not opposed to supporting other runtimes, it just hasn't been a priority for me yet.
10:52:04 mikal noonedeadpunk: like I said my current focus is working through some sort of performance testing, because I'd say there's a 50% change kerbside is going to get a rewrite in golang or C or something.
10:52:24 noonedeadpunk right.
10:52:40 noonedeadpunk not sure if golang will help a lot, but yeah
10:52:48 mikal So it works right now, but I am unclear if it is performant enough. The proxy is basically a proof of concept right now.
10:56:59 noonedeadpunk From my prespective separation of uwsgi API and worker itself would be really nice to have. Probably bercause too much got used to how openstack is written :)
10:57:55 mikal Ok, I'll make a note to circle back to that once I have a stronger view of if a rewrite is required or not.
10:58:02 noonedeadpunk As for instance we have quite established way of running wsgi apps, as well as wiring around TLS generation and deployment for it
10:58:36 mikal Basically the focus so far has been on getting a proxy to work at all, given the poor state of SPICE protocol documentation, and then on landing the supporting nova patches. I am only just now circling back to productionizing the proxy.
10:58:45 noonedeadpunk yeah, right
10:59:03 mikal I don't know much about uwsgi. If you have best practise recommendations I'd be quite interested in those.
10:59:07 noonedeadpunk it's not that I'm blaming or anything, just commenting :)
10:59:48 mikal Yeah its all good. I'm mildly excited someone is considering deploying this at all. Its sometimes lonely over here.
11:00:00 noonedeadpunk we have a uwsgi role we're actively using, which produced config file like this one: https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/templates/uwsgi.ini.j2
11:00:27 noonedeadpunk I totally see how useful this all might be in light of VMware migrations
11:01:00 mikal Yeah, I'm basically aiming at Citrix-like functionality here. Its just going to take a little while to get there.
11:01:11 sean-k-mooney noonedeadpunk: i dont even think you need to get partically fancy with the loadbalancing just have haproxxy or whatere use the souce ip/mac affiniteis the connetion to one of the active backends
11:01:16 noonedeadpunk but also this allows to secure communication with API side with TLS, as I think it's not covered now
11:01:29 sean-k-mooney noonedeadpunk: basiclly like you would for the novnc or html spice console
11:01:32 noonedeadpunk sean-k-mooney: exactly what I was thinking about, yes
11:01:47 noonedeadpunk jsut not L7 but L4 I guess
11:01:56 mikal noonedeadpunk: Kerbside API TLS is done with a frontend like apache2, not natively. SPICE protocol TLS is done natively within Kerbside.
11:02:14 sean-k-mooney ya so use the source ip and tcp port to balance
11:02:20 sean-k-mooney if it fails over they will have to reconenct
11:02:26 noonedeadpunk right, but how you communicate between apache2 and Kerbside API
11:02:34 sean-k-mooney but i dotn see a way around that without clustering in kerbside
11:02:35 noonedeadpunk as it;s listening on 0.0.0.0 explicitly
11:02:44 mikal If you squint at it, Kerbside is sort of a layer 7 load balancer. Its the thing which understands the protocol well enough to direct traffic to the right backend.
11:02:58 noonedeadpunk sean-k-mooney: I think it's totally fine tbh and more then reasonable
11:03:19 mikal noonedeadpunk: that's just the default in the config, you can override it.
11:03:27 noonedeadpunk mikal: um, can you? https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L56
11:03:45 noonedeadpunk and if that is a bare metal host and not docker container?
11:04:03 mikal noonedeadpunk: yeah, that whole template can be overridden by exporting KERBSIDE_API_COMMAND_LINE in the starting environment.
11:04:15 noonedeadpunk ah
11:04:17 noonedeadpunk ok
11:04:23 sean-k-mooney its just a wsgi app by the looks of it
11:04:33 noonedeadpunk so I can also set `echo ''` to it kinda?
11:04:37 noonedeadpunk it is
11:04:49 noonedeadpunk jsut started by the main service
11:04:53 mikal noonedeadpunk: I don't follow the echo comment?
11:05:28 sean-k-mooney mikal: are you using https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L54 to gengerate a script to run kerbside
11:05:29 noonedeadpunk so if I want to run wsgi separately, I shoudl be able to replace KERBSIDE_API_COMMAND_LINE with noop command quite successfully I assiume
11:05:33 sean-k-mooney as in a console script
11:05:45 sean-k-mooney or are you using that to spwan serpte isntance of kerbside per conection
11:05:47 mikal noonedeadpunk: secretly, this is just a cut down (and a bit older) version of https://github.com/shakenfist/shakenfist/blob/develop/shakenfist/config.py which is the project that code was lifted from.
11:06:01 mikal Oh interesting.
11:06:38 mikal Like I said, I don't know much about uwsgi. If flask can work in that mode then I think that would work?
11:07:02 sean-k-mooney noonedeadpunk: https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L566-L568
11:07:06 mikal The protocol native proxy is a separate socket server.
11:07:21 noonedeadpunk usually gunicorn, uwsgi and mod_wsgi are interchangable
11:07:36 sean-k-mooney mikal: so your api consol script currently pawans a wsgi server form api.py
11:08:17 sean-k-mooney but normlaly you woudl want to spin up the wsgi appolcation directly with a webserver and not spawn it with Popen like that if it can be shared between conenctions
11:08:30 noonedeadpunk and then if your intention to have apache2 in front for TLS, using mod_wsgi might be even more wise
11:08:34 sean-k-mooney if ithis is one process per api request however that is differnt
11:08:59 mikal The Shaken Fist Foundation [1] welcomes your feedback and potential contributions.
11:08:59 mikal 1: Said foundation does not exist.
11:09:58 sean-k-mooney mikal: without deep diving i think noonedeadpunk could ignore your main an api.py
11:10:11 mikal I think what I am getting from this conversation is that I should go read more about uwsgi, which I pinky swear to do.
11:10:24 sean-k-mooney and just spin up the wsig app directly and as they said use apache in front to handell tls
11:10:48 sean-k-mooney wsgi and uwsgi are also two diffent things
11:11:22 sean-k-mooney uwsgi, gurnicorn, and mod_wsgi are 3 examples wor wsgi applcation servers
11:11:37 sean-k-mooney as in they host a wsgi applcation
11:11:44 sean-k-mooney wsgi its self is a protocol
11:12:03 noonedeadpunk yeah, so usually you just don;t need to spawn gunicorn as part of the service, but do that separately
11:12:11 noonedeadpunk and leave user to choose wsgi server
11:12:41 mikal Ok, I will put reading about all this on my todo list for tomorrow.
11:12:42 noonedeadpunk as some might want to run granian or smth like that even
11:13:02 mikal Although, writing load testing tooling sounds like more fun ;)
11:13:30 sean-k-mooney so mikal what actully happenign when you creat taht subprcess is your stattign a wsgi sever that is importing https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L39-L42
11:14:25 sean-k-mooney mikal: then en that applcation host is bidnign to the port in the temlsate and handeign the request ot of band of the kerbside command you use to run it
11:15:24 sean-k-mooney mikal: by the way the way you wrote looks liek you wanted it to work how nova-api worked usign the eventlet webserver whild also not using eventlet :)
11:16:11 sean-k-mooney i.e. you wanted to provide a simpel way to run it form the cli via the kerbside command generated by this entry point https://github.com/shakenfist/kerbside/blob/develop/setup.cfg#L24
11:16:23 mikal sean-k-mooney: Oh I was definitely thinking about nova-api when I wrote it, but it was also over four years ago when I wrote that which means its all long forgotten now.
11:17:40 sean-k-mooney mikal: kolla recently moved form apache with mod_wsi to uwsgi
11:18:17 sean-k-mooney you shoudl be able to skip the commdn without any code changes and just run that flask ap directly by provieing the moduel path like you did to grunicorn
11:18:34 mikal sean-k-mooney: that's probably good because this is a topic I am entirely not passionate about. I just want to do something people think doesn't suck and then move onto more interesting things.
11:19:03 sean-k-mooney the main benifity fo that approch is you dont need to do the proces management in your cli
11:19:13 mikal Like I want it to be performant and not make people angry, but it is not my calling.
11:19:15 sean-k-mooney you can leave it up to docker/systemd to do that for you
11:20:56 mikal Ok well I will play with this tomorrow. I am being assured it is past my bed time now. Feel free to file a github issue if you're keen, otherwise I'll just work through a fix and smash it in.
11:22:23 sean-k-mooney well none of this is a blcoerk on the nova side. but fore scalablity and security reasons it woudl be good to do when doing a production install
11:22:36 sean-k-mooney so its somethign that noonedeadpunk and the kolla folks wil care about
11:23:24 sean-k-mooney but its not liek its need a big code change at first glance
11:25:24 noonedeadpunk I can try to submit a report later today
11:25:34 noonedeadpunk I need to deal with failing migrations right now :(

Earlier   Later