| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-22 | |||
| 11:04:37 | noonedeadpunk | it is | |
| 11:04:49 | noonedeadpunk | jsut started by the main service | |
| 11:04:53 | mikal | noonedeadpunk: I don't follow the echo comment? | |
| 11:05:28 | sean-k-mooney | mikal: are you using https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py#L54 to gengerate a script to run kerbside | |
| 11:05:29 | noonedeadpunk | so if I want to run wsgi separately, I shoudl be able to replace KERBSIDE_API_COMMAND_LINE with noop command quite successfully I assiume | |
| 11:05:33 | sean-k-mooney | as in a console script | |
| 11:05:45 | sean-k-mooney | or are you using that to spwan serpte isntance of kerbside per conection | |
| 11:05:47 | mikal | noonedeadpunk: secretly, this is just a cut down (and a bit older) version of https://github.com/shakenfist/shakenfist/blob/develop/shakenfist/config.py which is the project that code was lifted from. | |
| 11:06:01 | mikal | Oh interesting. | |
| 11:06:38 | mikal | Like I said, I don't know much about uwsgi. If flask can work in that mode then I think that would work? | |
| 11:07:02 | sean-k-mooney | noonedeadpunk: https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L566-L568 | |
| 11:07:06 | mikal | The protocol native proxy is a separate socket server. | |
| 11:07:21 | noonedeadpunk | usually gunicorn, uwsgi and mod_wsgi are interchangable | |
| 11:07:36 | sean-k-mooney | mikal: so your api consol script currently pawans a wsgi server form api.py | |
| 11:08:17 | sean-k-mooney | but normlaly you woudl want to spin up the wsgi appolcation directly with a webserver and not spawn it with Popen like that if it can be shared between conenctions | |
| 11:08:30 | noonedeadpunk | and then if your intention to have apache2 in front for TLS, using mod_wsgi might be even more wise | |
| 11:08:34 | sean-k-mooney | if ithis is one process per api request however that is differnt | |
| 11:08:59 | mikal | The Shaken Fist Foundation [1] welcomes your feedback and potential contributions. | |
| 11:08:59 | mikal | 1: Said foundation does not exist. | |
| 11:09:58 | sean-k-mooney | mikal: without deep diving i think noonedeadpunk could ignore your main an api.py | |
| 11:10:11 | mikal | I think what I am getting from this conversation is that I should go read more about uwsgi, which I pinky swear to do. | |
| 11:10:24 | sean-k-mooney | and just spin up the wsig app directly and as they said use apache in front to handell tls | |
| 11:10:48 | sean-k-mooney | wsgi and uwsgi are also two diffent things | |
| 11:11:22 | sean-k-mooney | uwsgi, gurnicorn, and mod_wsgi are 3 examples wor wsgi applcation servers | |
| 11:11:37 | sean-k-mooney | as in they host a wsgi applcation | |
| 11:11:44 | sean-k-mooney | wsgi its self is a protocol | |
| 11:12:03 | noonedeadpunk | yeah, so usually you just don;t need to spawn gunicorn as part of the service, but do that separately | |
| 11:12:11 | noonedeadpunk | and leave user to choose wsgi server | |
| 11:12:41 | mikal | Ok, I will put reading about all this on my todo list for tomorrow. | |
| 11:12:42 | noonedeadpunk | as some might want to run granian or smth like that even | |
| 11:13:02 | mikal | Although, writing load testing tooling sounds like more fun ;) | |
| 11:13:30 | sean-k-mooney | so mikal what actully happenign when you creat taht subprcess is your stattign a wsgi sever that is importing https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L39-L42 | |
| 11:14:25 | sean-k-mooney | mikal: then en that applcation host is bidnign to the port in the temlsate and handeign the request ot of band of the kerbside command you use to run it | |
| 11:15:24 | sean-k-mooney | mikal: by the way the way you wrote looks liek you wanted it to work how nova-api worked usign the eventlet webserver whild also not using eventlet :) | |
| 11:16:11 | sean-k-mooney | i.e. you wanted to provide a simpel way to run it form the cli via the kerbside command generated by this entry point https://github.com/shakenfist/kerbside/blob/develop/setup.cfg#L24 | |
| 11:16:23 | mikal | sean-k-mooney: Oh I was definitely thinking about nova-api when I wrote it, but it was also over four years ago when I wrote that which means its all long forgotten now. | |
| 11:17:40 | sean-k-mooney | mikal: kolla recently moved form apache with mod_wsi to uwsgi | |
| 11:18:17 | sean-k-mooney | you shoudl be able to skip the commdn without any code changes and just run that flask ap directly by provieing the moduel path like you did to grunicorn | |
| 11:18:34 | mikal | sean-k-mooney: that's probably good because this is a topic I am entirely not passionate about. I just want to do something people think doesn't suck and then move onto more interesting things. | |
| 11:19:03 | sean-k-mooney | the main benifity fo that approch is you dont need to do the proces management in your cli | |
| 11:19:13 | mikal | Like I want it to be performant and not make people angry, but it is not my calling. | |
| 11:19:15 | sean-k-mooney | you can leave it up to docker/systemd to do that for you | |
| 11:20:56 | mikal | Ok well I will play with this tomorrow. I am being assured it is past my bed time now. Feel free to file a github issue if you're keen, otherwise I'll just work through a fix and smash it in. | |
| 11:22:23 | sean-k-mooney | well none of this is a blcoerk on the nova side. but fore scalablity and security reasons it woudl be good to do when doing a production install | |
| 11:22:36 | sean-k-mooney | so its somethign that noonedeadpunk and the kolla folks wil care about | |
| 11:23:24 | sean-k-mooney | but its not liek its need a big code change at first glance | |
| 11:25:24 | noonedeadpunk | I can try to submit a report later today | |
| 11:25:34 | noonedeadpunk | I need to deal with failing migrations right now :( | |
| 11:31:34 | mikal | Thanks guys. | |
| 11:43:06 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754 | |
| 12:39:11 | opendevreview | Merged openstack/nova stable/2025.1: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950213 | |
| 13:52:02 | opendevreview | Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659 | |
| 13:52:10 | dansmith | gibi: Uggla, per yesterday, some refinement ^ | |
| 13:53:18 | opendevreview | Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659 | |
| 14:08:48 | opendevreview | yaguang proposed openstack/nova stable/2024.2: Handle iso+gpt detections https://review.opendev.org/c/openstack/nova/+/943940 | |
| 14:14:44 | Uggla | dansmith, +1 for the new pci.alias example. | |
| 14:33:36 | opendevreview | Merged openstack/nova stable/2025.1: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950214 | |
| 15:23:02 | opendevreview | melanie witt proposed openstack/nova-specs master: Re-propose vTPM live migration https://review.opendev.org/c/openstack/nova-specs/+/947542 | |
| 15:23:42 | melwitt | gibi: fixed the typo you found ^ | |
| 15:26:25 | melwitt | dansmith, sean-k-mooney: the re-proposal for vtpm live migration spec is ready for review if you may be interested. I have noted the changes from the last approval in the commit message | |
| 15:34:03 | opendevreview | Fabian Wiesel proposed openstack/nova master: libvirt: Expose number of migration threads https://review.opendev.org/c/openstack/nova/+/950667 | |
| 15:55:49 | gibi | melwitt: thanks | |
| 15:55:56 | gibi | I added my +2 back | |
| 16:03:51 | dansmith | melwitt: ack, I'm several layers deep in the stack right now | |
| 16:54:31 | opendevreview | Merged openstack/nova stable/2024.2: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950215 | |
| 16:58:30 | opendevreview | Merged openstack/nova stable/2024.2: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950216 | |
| 18:09:09 | melwitt | dansmith: ack, it is not urgent but it is there when you are ready | |
| #openstack-nova - 2025-05-23 | |||
| 11:17:18 | manumoh | Hello all, I see the following error in my nova-api logs intermittently since last 24 hours . Nothing has been changed from service or config side. Any help in highly appreciated | |
| 11:18:51 | manumoh | https://www.irccloud.com/pastebin/0nfGEBS7/connection_timeout_error | |
| 11:20:23 | sean-k-mooney | that looks like your having network connectiviy issues betwhen nova and keystone | |
| 11:21:21 | sean-k-mooney | assuming your not seeign issues in the keyston logs this is likely infra issue in your datacenter networking | |
| 11:22:15 | sean-k-mooney | the call that is failling is to neutron to get the prots for the current tenant to check if there is enough quota | |
| 11:22:48 | sean-k-mooney | so its posisbel that the keystaauth succeded and then then call to neutron is actully whats timign out | |
| 11:24:09 | sean-k-mooney | i thinik thats actully more likely then it beign a keystone issue | |
| 11:29:07 | manumoh | sean-k-mooney: ohhkk .. if its a neutron issue , then why the trace is giving timeout at /usr/lib/python3.9/site-packages/keystoneauth1/session.py | |
| 11:29:37 | sean-k-mooney | the time out is nova calling neutron api via neutron client | |
| 11:31:13 | sean-k-mooney | this is nova callign neutron https://cloud.test.com:9696/v2.0/ports?tenant_id=ad0ddb4fd2424410ae923d7f01746a83&fields=id is listing all port for that teant and only return the uuids so that we can count how many are in use to check if we can create teh vm without exceeding the port quota | |
| 11:33:36 | manumoh | ok .. so if the neutron server fails to respond back for the port list api within 60 sec we will see the same error in logs | |
| 11:34:06 | sean-k-mooney | yes | |
| 11:34:40 | sean-k-mooney | this is part of the early validation logic in teh api so the user proably got a 500 | |
| 11:35:15 | sean-k-mooney | if you have the request id you should be able to validate if neutron every recived the request and if it did why it took so long to reply | |
| 11:36:01 | sean-k-mooney | FYI myself and other redhatter have a company "recharge day" today so we are all on PTO so im not sure how active the neutron channle is today | |
| 11:36:31 | manumoh | yeah noticed it while creating an LB and the 500 errror in present in octavia logs | |
| 11:36:48 | manumoh | @sean-k-mooney thanks for the reply .. i will check further | |
| 11:42:28 | manumoh | I don't see the request ids in neutron logs | |
| 13:19:37 | opendevreview | Merged openstack/nova stable/2024.1: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950217 | |
| 14:41:12 | opendevreview | Merged openstack/nova stable/2024.1: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950218 | |
| #openstack-nova - 2025-05-24 | |||
| 00:49:47 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for sound model. https://review.opendev.org/c/openstack/nova/+/926126 | |
| 00:49:48 | opendevreview | Michael Still proposed openstack/nova master: Implement sound model extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/940770 | |
| 00:49:48 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for USB controller model. https://review.opendev.org/c/openstack/nova/+/927354 | |
| 00:49:49 | opendevreview | Michael Still proposed openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643 | |
| #openstack-nova - 2025-05-26 | |||
| 09:11:06 | mikal | noonedeadpunk / sean-k-mooney: I had some more thoughts about wsgi and wrote a reply to https://github.com/shakenfist/kerbside/issues/14, so I guess we should keep the conversation there at this point... | |
| 10:24:12 | sean-k-mooney | mikal: ack. normally you woudl run this using two systemd unit files and 2 contianer in the kolla world and ues the fact that you can have depencies between the units/containers to enforce startup order. ovs in kolla for exampel has 2 contaiern one for the ovsdb and one for the ovsvswitchd | |
| 10:31:23 | mikal | sean-k-mooney: that's true, and its what I think I should do here. Its more a case of that not having been how its already done, and me wanted to make sure I don't need to re-write the SPICE proxy before putting much more effort into the current implementation. | |
| 10:33:16 | sean-k-mooney | mikal: yep i understand. im kind fo surpised that kolla are ok with the current approch, it was very contoverial when i added bifrost supprot with is the only example fo a multi service contaienr in kolla outside of how openstack service spawn a privesep service | |
| 10:34:33 | mikal | sean-k-mooney: I don't think they know. Whilst the patches aren't secret I haven't gotten as far as proposing them in gerrit yet -- I needed the nova patches and client changes to merge first and that happened too late for 2025.1. I also want to sort out the plan for scaling first too. | |
| 10:34:50 | sean-k-mooney | mikal: form a secity and oerational side i think it would make sense to supprot splitign kerbside long term but short term what you have now will work. i said the same in the pr | |
| 10:35:43 | mikal | sean-k-mooney: I do think scaling the HTTP and SPICE aspects separately make sense, I just didn't occur to me a year or more ago when I wrote that code. | |
| 10:38:12 | sean-k-mooney | mikal: i have not looked at the kerbside code in any detail but does the http side need to actully interact with the SPICE side directly via IPC | |
| 10:38:58 | sean-k-mooney | if not or even if so it wprobaly woudl be easy to add flags to the helper to only run one or the other and use a unix socket for comunicatin | |