| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-22 | |||
| 11:11:22 | sean-k-mooney | uwsgi, gurnicorn, and mod_wsgi are 3 examples wor wsgi applcation servers | |
| 11:11:37 | sean-k-mooney | as in they host a wsgi applcation | |
| 11:11:44 | sean-k-mooney | wsgi its self is a protocol | |
| 11:12:03 | noonedeadpunk | yeah, so usually you just don;t need to spawn gunicorn as part of the service, but do that separately | |
| 11:12:11 | noonedeadpunk | and leave user to choose wsgi server | |
| 11:12:41 | mikal | Ok, I will put reading about all this on my todo list for tomorrow. | |
| 11:12:42 | noonedeadpunk | as some might want to run granian or smth like that even | |
| 11:13:02 | mikal | Although, writing load testing tooling sounds like more fun ;) | |
| 11:13:30 | sean-k-mooney | so mikal what actully happenign when you creat taht subprcess is your stattign a wsgi sever that is importing https://github.com/shakenfist/kerbside/blob/develop/kerbside/api.py#L39-L42 | |
| 11:14:25 | sean-k-mooney | mikal: then en that applcation host is bidnign to the port in the temlsate and handeign the request ot of band of the kerbside command you use to run it | |
| 11:15:24 | sean-k-mooney | mikal: by the way the way you wrote looks liek you wanted it to work how nova-api worked usign the eventlet webserver whild also not using eventlet :) | |
| 11:16:11 | sean-k-mooney | i.e. you wanted to provide a simpel way to run it form the cli via the kerbside command generated by this entry point https://github.com/shakenfist/kerbside/blob/develop/setup.cfg#L24 | |
| 11:16:23 | mikal | sean-k-mooney: Oh I was definitely thinking about nova-api when I wrote it, but it was also over four years ago when I wrote that which means its all long forgotten now. | |
| 11:17:40 | sean-k-mooney | mikal: kolla recently moved form apache with mod_wsi to uwsgi | |
| 11:18:17 | sean-k-mooney | you shoudl be able to skip the commdn without any code changes and just run that flask ap directly by provieing the moduel path like you did to grunicorn | |
| 11:18:34 | mikal | sean-k-mooney: that's probably good because this is a topic I am entirely not passionate about. I just want to do something people think doesn't suck and then move onto more interesting things. | |
| 11:19:03 | sean-k-mooney | the main benifity fo that approch is you dont need to do the proces management in your cli | |
| 11:19:13 | mikal | Like I want it to be performant and not make people angry, but it is not my calling. | |
| 11:19:15 | sean-k-mooney | you can leave it up to docker/systemd to do that for you | |
| 11:20:56 | mikal | Ok well I will play with this tomorrow. I am being assured it is past my bed time now. Feel free to file a github issue if you're keen, otherwise I'll just work through a fix and smash it in. | |
| 11:22:23 | sean-k-mooney | well none of this is a blcoerk on the nova side. but fore scalablity and security reasons it woudl be good to do when doing a production install | |
| 11:22:36 | sean-k-mooney | so its somethign that noonedeadpunk and the kolla folks wil care about | |
| 11:23:24 | sean-k-mooney | but its not liek its need a big code change at first glance | |
| 11:25:24 | noonedeadpunk | I can try to submit a report later today | |
| 11:25:34 | noonedeadpunk | I need to deal with failing migrations right now :( | |
| 11:31:34 | mikal | Thanks guys. | |
| 11:43:06 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754 | |
| 12:39:11 | opendevreview | Merged openstack/nova stable/2025.1: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950213 | |
| 13:52:02 | opendevreview | Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659 | |
| 13:52:10 | dansmith | gibi: Uggla, per yesterday, some refinement ^ | |
| 13:53:18 | opendevreview | Dan Smith proposed openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659 | |
| 14:08:48 | opendevreview | yaguang proposed openstack/nova stable/2024.2: Handle iso+gpt detections https://review.opendev.org/c/openstack/nova/+/943940 | |
| 14:14:44 | Uggla | dansmith, +1 for the new pci.alias example. | |
| 14:33:36 | opendevreview | Merged openstack/nova stable/2025.1: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950214 | |
| 15:23:02 | opendevreview | melanie witt proposed openstack/nova-specs master: Re-propose vTPM live migration https://review.opendev.org/c/openstack/nova-specs/+/947542 | |
| 15:23:42 | melwitt | gibi: fixed the typo you found ^ | |
| 15:26:25 | melwitt | dansmith, sean-k-mooney: the re-proposal for vtpm live migration spec is ready for review if you may be interested. I have noted the changes from the last approval in the commit message | |
| 15:34:03 | opendevreview | Fabian Wiesel proposed openstack/nova master: libvirt: Expose number of migration threads https://review.opendev.org/c/openstack/nova/+/950667 | |
| 15:55:49 | gibi | melwitt: thanks | |
| 15:55:56 | gibi | I added my +2 back | |
| 16:03:51 | dansmith | melwitt: ack, I'm several layers deep in the stack right now | |
| 16:54:31 | opendevreview | Merged openstack/nova stable/2024.2: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950215 | |
| 16:58:30 | opendevreview | Merged openstack/nova stable/2024.2: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950216 | |
| 18:09:09 | melwitt | dansmith: ack, it is not urgent but it is there when you are ready | |
| #openstack-nova - 2025-05-23 | |||
| 11:17:18 | manumoh | Hello all, I see the following error in my nova-api logs intermittently since last 24 hours . Nothing has been changed from service or config side. Any help in highly appreciated | |
| 11:18:51 | manumoh | https://www.irccloud.com/pastebin/0nfGEBS7/connection_timeout_error | |
| 11:20:23 | sean-k-mooney | that looks like your having network connectiviy issues betwhen nova and keystone | |
| 11:21:21 | sean-k-mooney | assuming your not seeign issues in the keyston logs this is likely infra issue in your datacenter networking | |
| 11:22:15 | sean-k-mooney | the call that is failling is to neutron to get the prots for the current tenant to check if there is enough quota | |
| 11:22:48 | sean-k-mooney | so its posisbel that the keystaauth succeded and then then call to neutron is actully whats timign out | |
| 11:24:09 | sean-k-mooney | i thinik thats actully more likely then it beign a keystone issue | |
| 11:29:07 | manumoh | sean-k-mooney: ohhkk .. if its a neutron issue , then why the trace is giving timeout at /usr/lib/python3.9/site-packages/keystoneauth1/session.py | |
| 11:29:37 | sean-k-mooney | the time out is nova calling neutron api via neutron client | |
| 11:31:13 | sean-k-mooney | this is nova callign neutron https://cloud.test.com:9696/v2.0/ports?tenant_id=ad0ddb4fd2424410ae923d7f01746a83&fields=id is listing all port for that teant and only return the uuids so that we can count how many are in use to check if we can create teh vm without exceeding the port quota | |
| 11:33:36 | manumoh | ok .. so if the neutron server fails to respond back for the port list api within 60 sec we will see the same error in logs | |
| 11:34:06 | sean-k-mooney | yes | |
| 11:34:40 | sean-k-mooney | this is part of the early validation logic in teh api so the user proably got a 500 | |
| 11:35:15 | sean-k-mooney | if you have the request id you should be able to validate if neutron every recived the request and if it did why it took so long to reply | |
| 11:36:01 | sean-k-mooney | FYI myself and other redhatter have a company "recharge day" today so we are all on PTO so im not sure how active the neutron channle is today | |
| 11:36:31 | manumoh | yeah noticed it while creating an LB and the 500 errror in present in octavia logs | |
| 11:36:48 | manumoh | @sean-k-mooney thanks for the reply .. i will check further | |
| 11:42:28 | manumoh | I don't see the request ids in neutron logs | |
| 13:19:37 | opendevreview | Merged openstack/nova stable/2024.1: Amend functional reproducer for bug 1899835 https://review.opendev.org/c/openstack/nova/+/950217 | |
| 14:41:12 | opendevreview | Merged openstack/nova stable/2024.1: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/950218 | |
| #openstack-nova - 2025-05-24 | |||
| 00:49:47 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for sound model. https://review.opendev.org/c/openstack/nova/+/926126 | |
| 00:49:48 | opendevreview | Michael Still proposed openstack/nova master: Implement sound model extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/940770 | |
| 00:49:48 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for USB controller model. https://review.opendev.org/c/openstack/nova/+/927354 | |
| 00:49:49 | opendevreview | Michael Still proposed openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643 | |
| #openstack-nova - 2025-05-26 | |||
| 09:11:06 | mikal | noonedeadpunk / sean-k-mooney: I had some more thoughts about wsgi and wrote a reply to https://github.com/shakenfist/kerbside/issues/14, so I guess we should keep the conversation there at this point... | |
| 10:24:12 | sean-k-mooney | mikal: ack. normally you woudl run this using two systemd unit files and 2 contianer in the kolla world and ues the fact that you can have depencies between the units/containers to enforce startup order. ovs in kolla for exampel has 2 contaiern one for the ovsdb and one for the ovsvswitchd | |
| 10:31:23 | mikal | sean-k-mooney: that's true, and its what I think I should do here. Its more a case of that not having been how its already done, and me wanted to make sure I don't need to re-write the SPICE proxy before putting much more effort into the current implementation. | |
| 10:33:16 | sean-k-mooney | mikal: yep i understand. im kind fo surpised that kolla are ok with the current approch, it was very contoverial when i added bifrost supprot with is the only example fo a multi service contaienr in kolla outside of how openstack service spawn a privesep service | |
| 10:34:33 | mikal | sean-k-mooney: I don't think they know. Whilst the patches aren't secret I haven't gotten as far as proposing them in gerrit yet -- I needed the nova patches and client changes to merge first and that happened too late for 2025.1. I also want to sort out the plan for scaling first too. | |
| 10:34:50 | sean-k-mooney | mikal: form a secity and oerational side i think it would make sense to supprot splitign kerbside long term but short term what you have now will work. i said the same in the pr | |
| 10:35:43 | mikal | sean-k-mooney: I do think scaling the HTTP and SPICE aspects separately make sense, I just didn't occur to me a year or more ago when I wrote that code. | |
| 10:38:12 | sean-k-mooney | mikal: i have not looked at the kerbside code in any detail but does the http side need to actully interact with the SPICE side directly via IPC | |
| 10:38:58 | sean-k-mooney | if not or even if so it wprobaly woudl be easy to add flags to the helper to only run one or the other and use a unix socket for comunicatin | |
| 10:39:17 | sean-k-mooney | that woudl allow you to supprot the all in one or split deplyment pretty simply | |
| 10:39:45 | sean-k-mooney | and if someone didnt want to use the helper for running the http part that already possibel today | |
| 10:40:13 | sean-k-mooney | they would just need to make sure to pass it the unix socket path to talk to the spice part assumign there is some IPC requireemnt | |
| 10:40:20 | sean-k-mooney | if there isnt the its even simpler | |
| 10:40:33 | sean-k-mooney | i assume there is since you spwaning it with a pipe today | |
| 10:41:27 | mikal | All the interaction is via the MySQL database fortunately. The REST API / Nova Auth URL thingie create state (authorized tokens basically) that the SPICE side consumes. In return the SPICE side adds data to MySQL for the REST API to use (mostly information about currently connected sessions). | |
| 10:41:47 | mikal | Its really a fairly simple app. The hard bit was the incompete state of the SPICE protocol documentation. | |
| 10:42:04 | mikal | Especially the proprietary compression algorithms. That kept me out of mischief for a while. | |
| 11:13:37 | sean-k-mooney | mikal: in that case it shoudl be trivial ot split the process and scale them indepently later. they just need to both have access to the db | |
| 12:11:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow services to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 12:11:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450 | |
| 13:03:23 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Address issues with hypervisors APIs https://review.opendev.org/c/openstack/nova/+/950867 | |
| 13:03:31 | stephenfin | sean-k-mooney: ^ | |
| 13:23:46 | sean-k-mooney | ack so you are injectign that to the side fo the main seriese. i guess as long as we merge the 4 hypervior api paches at the same time its probaly fine | |
| 15:22:10 | sean-k-mooney | gibi: i looked at a couple of your eventlet series and im mostly happy with the ones i looked at. i left a +1 with some comoens on 2? of them | |
| 15:22:24 | sean-k-mooney | ill take a look at them again later in teh week | |
| 15:22:42 | sean-k-mooney | but im going to switch to a diffent topic for a while | |
| 17:17:42 | gibi | sean-k-mooney: ack, I will check you comments tomorrow | |
| 17:18:33 | gibi | as the oslo.service patch is landed and released I bumped to that release in the series. Also I changed the env application order in the devstack patch as it was suggested on IRC | |
| 17:21:38 | sean-k-mooney | yep i saw that update in my email (devstack order) | |
| 17:22:41 | sean-k-mooney | i have not had time to test out yoru series locally but ill be deploying a multinode devstack to test some wather chagnes proably in the next week so once i have that up an running ill probaly check ot your schduelr branch | |
| 17:46:26 | gibi | cool | |
| 17:46:28 | gibi | thanks | |