Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-05-22
11:01:47 noonedeadpunk jsut not L7 but L4 I guess
11:01:56 mikal noonedeadpunk: Kerbside API TLS is done with a frontend like apache2, not natively. SPICE protocol TLS is done natively within Kerbside.
11:02:14 sean-k-mooney ya so use the source ip and tcp port to balance
11:02:20 sean-k-mooney if it fails over they will have to reconenct
11:02:26 noonedeadpunk right, but how you communicate between apache2 and Kerbside API
11:02:34 sean-k-mooney but i dotn see a way around that without clustering in kerbside
11:02:35 noonedeadpunk as it;s listening on 0.0.0.0 explicitly
11:02:44 mikal If you squint at it, Kerbside is sort of a layer 7 load balancer. Its the thing which understands the protocol well enough to direct traffic to the right backend.
11:02:58 noonedeadpunk sean-k-mooney: I think it's totally fine tbh and more then reasonable
11:03:19 mikal noonedeadpunk: that's just the default in the config, you can override it.
11:03:27 noonedeadpunk mikal: um, can you? https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L56
11:03:45 noonedeadpunk and if that is a bare metal host and not docker container?
11:04:03 mikal noonedeadpunk: yeah, that whole template can be overridden by exporting KERBSIDE_API_COMMAND_LINE in the starting environment.
11:04:15 noonedeadpunk ah
11:04:17 noonedeadpunk ok
11:04:23 sean-k-mooney its just a wsgi app by the looks of it
11:04:33 noonedeadpunk so I can also set `echo ''` to it kinda?
11:04:37 noonedeadpunk it is
11:04:49 noonedeadpunk jsut started by the main service
11:04:53 mikal noonedeadpunk: I don't follow the echo comment?
11:05:28 sean-k-mooney mikal: are you using https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L54 to gengerate a script to run kerbside
11:05:29 noonedeadpunk so if I want to run wsgi separately, I shoudl be able to replace KERBSIDE_API_COMMAND_LINE with noop command quite successfully I assiume
11:05:33 sean-k-mooney as in a console script
11:05:45 sean-k-mooney or are you using that to spwan serpte isntance of kerbside per conection
11:05:47 mikal noonedeadpunk: secretly, this is just a cut down (and a bit older) version of https://github.com/shakenfist/shakenfist/blob/develop/shakenfist/config.py which is the project that code was lifted from.
11:06:01 mikal Oh interesting.
11:06:38 mikal Like I said, I don't know much about uwsgi. If flask can work in that mode then I think that would work?
11:07:02 sean-k-mooney noonedeadpunk: https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L566-L568
11:07:06 mikal The protocol native proxy is a separate socket server.
11:07:21 noonedeadpunk usually gunicorn, uwsgi and mod_wsgi are interchangable
11:07:36 sean-k-mooney mikal: so your api consol script currently pawans a wsgi server form api.py
11:08:17 sean-k-mooney but normlaly you woudl want to spin up the wsgi appolcation directly with a webserver and not spawn it with Popen like that if it can be shared between conenctions
11:08:30 noonedeadpunk and then if your intention to have apache2 in front for TLS, using mod_wsgi might be even more wise
11:08:34 sean-k-mooney if ithis is one process per api request however that is differnt
11:08:59 mikal The Shaken Fist Foundation [1] welcomes your feedback and potential contributions.
11:08:59 mikal 1: Said foundation does not exist.
11:09:58 sean-k-mooney mikal: without deep diving i think noonedeadpunk could ignore your main an api.py
11:10:11 mikal I think what I am getting from this conversation is that I should go read more about uwsgi, which I pinky swear to do.
11:10:24 sean-k-mooney and just spin up the wsig app directly and as they said use apache in front to handell tls
11:10:48 sean-k-mooney wsgi and uwsgi are also two diffent things
11:11:22 sean-k-mooney uwsgi, gurnicorn, and mod_wsgi are 3 examples wor wsgi applcation servers
11:11:37 sean-k-mooney as in they host a wsgi applcation
11:11:44 sean-k-mooney wsgi its self is a protocol
11:12:03 noonedeadpunk yeah, so usually you just don;t need to spawn gunicorn as part of the service, but do that separately
11:12:11 noonedeadpunk and leave user to choose wsgi server
11:12:41 mikal Ok, I will put reading about all this on my todo list for tomorrow.
11:12:42 noonedeadpunk as some might want to run granian or smth like that even
11:13:02 mikal Although, writing load testing tooling sounds like more fun ;)
11:13:30 sean-k-mooney so mikal what actully happenign when you creat taht subprcess is your stattign a wsgi sever that is importing https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L39-L42
11:14:25 sean-k-mooney mikal: then en that applcation host is bidnign to the port in the temlsate and handeign the request ot of band of the kerbside command you use to run it
11:15:24 sean-k-mooney mikal: by the way the way you wrote looks liek you wanted it to work how nova-api worked usign the eventlet webserver whild also not using eventlet :)
11:16:11 sean-k-mooney i.e. you wanted to provide a simpel way to run it form the cli via the kerbside command generated by this entry point https://github.com/shakenfist/kerbside/blob/develop/setup.cfg#L24
11:16:23 mikal sean-k-mooney: Oh I was definitely thinking about nova-api when I wrote it, but it was also over four years ago when I wrote that which means its all long forgotten now.
11:17:40 sean-k-mooney mikal: kolla recently moved form apache with mod_wsi to uwsgi
11:18:17 sean-k-mooney you shoudl be able to skip the commdn without any code changes and just run that flask ap directly by provieing the moduel path like you did to grunicorn
11:18:34 mikal sean-k-mooney: that's probably good because this is a topic I am entirely not passionate about. I just want to do something people think doesn't suck and then move onto more interesting things.
11:19:03 sean-k-mooney the main benifity fo that approch is you dont need to do the proces management in your cli
11:19:13 mikal Like I want it to be performant and not make people angry, but it is not my calling.
11:19:15 sean-k-mooney you can leave it up to docker/systemd to do that for you
11:20:56 mikal Ok well I will play with this tomorrow. I am being assured it is past my bed time now. Feel free to file a github issue if you're keen, otherwise I'll just work through a fix and smash it in.
11:22:23 sean-k-mooney well none of this is a blcoerk on the nova side. but fore scalablity and security reasons it woudl be good to do when doing a production install
11:22:36 sean-k-mooney so its somethign that noonedeadpunk and the kolla folks wil care about
11:23:24 sean-k-mooney but its not liek its need a big code change at first glance
11:25:24 noonedeadpunk I can try to submit a report later today
11:25:34 noonedeadpunk I need to deal with failing migrations right now :(
11:31:34 mikal Thanks guys.
11:43:06 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754
12:39:11 opendevreview Merged openstack/nova stable/2025.1: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950213
13:52:02 opendevreview Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659
13:52:10 dansmith gibi: Uggla, per yesterday, some refinement ^
13:53:18 opendevreview Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659
14:08:48 opendevreview yaguang proposed openstack/nova stable/2024.2: Handle iso+gpt detections https://review.opendev.org/c/openstack/nova/+/943940
14:14:44 Uggla dansmith, +1 for the new pci.alias example.
14:33:36 opendevreview Merged openstack/nova stable/2025.1: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950214
15:23:02 opendevreview melanie witt proposed openstack/nova-specs master: Re-propose vTPM live migration https://review.opendev.org/c/openstack/nova-specs/+/947542
15:23:42 melwitt gibi: fixed the typo you found ^
15:26:25 melwitt dansmith, sean-k-mooney: the re-proposal for vtpm live migration spec is ready for review if you may be interested. I have noted the changes from the last approval in the commit message
15:34:03 opendevreview Fabian Wiesel proposed openstack/nova master: libvirt: Expose number of migration threads https://review.opendev.org/c/openstack/nova/+/950667
15:55:49 gibi melwitt: thanks
15:55:56 gibi I added my +2 back
16:03:51 dansmith melwitt: ack, I'm several layers deep in the stack right now
16:54:31 opendevreview Merged openstack/nova stable/2024.2: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950215
16:58:30 opendevreview Merged openstack/nova stable/2024.2: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950216
18:09:09 melwitt dansmith: ack, it is not urgent but it is there when you are ready
#openstack-nova - 2025-05-23
11:17:18 manumoh Hello all, I see the following error in my nova-api logs intermittently since last 24 hours . Nothing has been changed from service or config side. Any help in highly appreciated
11:18:51 manumoh https://www.irccloud.com/pastebin/0nfGEBS7/connection_timeout_error
11:20:23 sean-k-mooney that looks like your having network connectiviy issues betwhen nova and keystone
11:21:21 sean-k-mooney assuming your not seeign issues in the keyston logs this is likely infra issue in your datacenter networking
11:22:15 sean-k-mooney the call that is failling is to neutron to get the prots for the current tenant to check if there is enough quota
11:22:48 sean-k-mooney so its posisbel that the keystaauth succeded and then then call to neutron is actully whats timign out
11:24:09 sean-k-mooney i thinik thats actully more likely then it beign a keystone issue
11:29:07 manumoh sean-k-mooney: ohhkk .. if its a neutron issue , then why the trace is giving timeout at /usr/lib/python3.9/site-packages/keystoneauth1/session.py
11:29:37 sean-k-mooney the time out is nova calling neutron api via neutron client
11:31:13 sean-k-mooney this is nova callign neutron https://cloud.test.com:9696/v2.0/ports?tenant_id=ad0ddb4fd2424410ae923d7f01746a83&fields=id is listing all port for that teant and only return the uuids so that we can count how many are in use to check if we can create teh vm without exceeding the port quota
11:33:36 manumoh ok .. so if the neutron server fails to respond back for the port list api within 60 sec we will see the same error in logs
11:34:06 sean-k-mooney yes
11:34:40 sean-k-mooney this is part of the early validation logic in teh api so the user proably got a 500
11:35:15 sean-k-mooney if you have the request id you should be able to validate if neutron every recived the request and if it did why it took so long to reply
11:36:01 sean-k-mooney FYI myself and other redhatter have a company "recharge day" today so we are all on PTO so im not sure how active the neutron channle is today
11:36:31 manumoh yeah noticed it while creating an LB and the 500 errror in present in octavia logs

Earlier   Later