| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-04 | |||
| 07:52:26 | sean-k-mooney | zigo: ya the stack track looks like someting is wrong in our packigng somehow | |
| 07:52:48 | sean-k-mooney | have you chekced the content of /usr/lib/python3/dist-packages/keystoneauth1/adapter.py | |
| 07:53:06 | sean-k-mooney | and confirmed that the base adapter in the file on disk has endpoint overried | |
| 07:53:17 | sean-k-mooney | and tries to import that in a python terminal and acess it? | |
| 07:53:46 | sean-k-mooney | or even one step back | |
| 07:53:59 | sean-k-mooney | if you use the neutron client cli does it have the same issue | |
| 07:54:27 | sean-k-mooney | i woudl expect a neutron port list to be broken in the same way on that host | |
| 07:57:46 | zigo | I'm not sure what / how you're asking me to check. | |
| 07:59:07 | zigo | :( | |
| 08:01:27 | sean-k-mooney | grep endpoint_override /usr/lib/python3/dist-packages/keystoneauth1/adapter.py | |
| 08:01:43 | sean-k-mooney | and also check if `neutron port-list` works | |
| 08:03:00 | zigo | openstack port list works, indeed. | |
| 08:03:03 | sean-k-mooney | the neutron cli supprot clouds.yaml so you can just use it like osc | |
| 08:03:11 | sean-k-mooney | no not openstack port list | |
| 08:03:26 | sean-k-mooney | exiplcited `neutron port-list` | |
| 08:03:38 | zigo | I don't think neutronclient is providing an /usr/bin/neutron anymore. | |
| 08:03:39 | sean-k-mooney | openstack port list uses the openstack sdk not the neutron clinet i think | |
| 08:03:52 | sean-k-mooney | they may have droped it in epoxy | |
| 08:05:03 | zigo | https://paste.opendev.org/show/bWJcYW3nYUAkjMn71RN6/ | |
| 08:07:08 | sean-k-mooney | ya they did. you code on disk all looks correct toteh 5.10.0 tag | |
| 08:07:27 | jkulik | https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413 could that be related? looks like having the same stacktrace | |
| 08:08:03 | sean-k-mooney | oh perhaps | |
| 08:08:23 | sean-k-mooney | if eventlet is nuking part of the obejct by garbage collecting too early | |
| 08:08:45 | zigo | Oh, thanks ! :) | |
| 08:08:47 | sean-k-mooney | that does look like the same o r a very similar trace | |
| 08:09:04 | zigo | I'm not surprised if eventlet is playing with me, as I'm on Python 3.13. | |
| 08:09:13 | sean-k-mooney | zigo: that is unfortually not very good news for yo since it means it because eventlet does not supprot 3.13 yet | |
| 08:09:24 | sean-k-mooney | which is why openstack does nto supprot 3.13 yet | |
| 08:09:34 | zigo | sean-k-mooney: I have no choice, 3.13 it is ... | |
| 08:09:54 | zigo | I was expecting things would go wrong. | |
| 08:10:04 | sean-k-mooney | so i belive there is an eventlet issue for this | |
| 08:10:15 | sean-k-mooney | we also know the thread id si broken | |
| 08:10:19 | zigo | Maybe I should try the latest eventlet release. | |
| 08:10:27 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/1032 | |
| 08:10:31 | zigo | Yeah, which is probably what's breaking nova-conductor too. | |
| 08:10:33 | sean-k-mooney | zigo: its not fix yet | |
| 08:10:57 | gboutry | That was exactly the error I got zigo, this is python 3.13 and eventlet not playing nice together | |
| 08:11:08 | zigo | SHIT ! :( | |
| 08:11:15 | gboutry | the error would manifest with attributes in the object that was initialized correctly | |
| 08:11:18 | zigo | Die eventlet die ... | |
| 08:16:12 | sean-k-mooney | i have commeted on both the 3.13 bugs to say we need to supprot 3.13 in eventlet for master this cycle per the project runtimes and that its required to complete the eventlet removal | |
| 08:16:26 | sean-k-mooney | the real quetion is will anyoen have time to actully work on that | |
| 08:16:54 | sean-k-mooney | this should be highlighted in the eventlet removal channel too i guess | |
| 08:17:15 | zigo | Well, for me, this means there wont be a working OpenStack for the Trixie release. That's really bad ... :( | |
| 08:17:42 | sean-k-mooney | cannonical are in the same situration for ubuntu 25.04 | |
| 08:17:56 | zigo | Except it's not an LTS, so they don't really care. | |
| 08:18:08 | sean-k-mooney | its the first tiem in openstack's history that im aware of that the latest point release of ubuntu has not supproted the latest openstack release | |
| 08:18:13 | zigo | They mostly provide OpenStack on top of LTS. | |
| 08:18:21 | sean-k-mooney | they ship it in both | |
| 08:18:25 | zigo | Right. | |
| 08:18:33 | sean-k-mooney | even if most of there custoemr are not on the point releases | |
| 08:18:44 | sean-k-mooney | it where they get ther early qa | |
| 08:20:03 | sean-k-mooney | hopefully herve and co will have time to look at the reported bugs | |
| 08:20:06 | zigo | Epoxy is not a skipable release, so it *must* be fixed. | |
| 08:20:56 | zigo | Saying, it's ok, Flamigo will have the fix, is not an option. | |
| 08:22:05 | sean-k-mooney | its not an openstack bug currently | |
| 08:22:23 | sean-k-mooney | its an eventlet one so ther isnt anythign nova can do to enable this | |
| 08:22:58 | sean-k-mooney | as an assid e apprenetly there is a propsoal to add "virtual thread" ala greentthread to core python https://discuss.python.org/t/add-virtual-threads-to-python/91403 | |
| 08:44:45 | zigo | I'll still have a try with eventlet 0.40 and see how it goes. | |
| 08:48:02 | zigo | Same stuff with eventlet 0.40 ... :( | |
| 08:49:50 | sean-k-mooney | there is thsi work in progress hack https://github.com/eventlet/eventlet/pull/1031/files | |
| 08:49:57 | sean-k-mooney | but that not complete | |
| 08:50:15 | sean-k-mooney | and its techinially for the thread issue rather then the one you currently hittign with the GC | |
| 08:50:26 | sean-k-mooney | that may get you slighly closer howerever | |
| 08:50:36 | zigo | Will try the patch ! :) | |
| 08:50:40 | zigo | I was in fact looking into it. | |
| 08:52:40 | sean-k-mooney | its just a guess but https://github.com/eventlet/eventlet/pull/1042 might be related to ge gc issue | |
| 08:53:06 | sean-k-mooney | although it does not directly refence the exiting issue so im just going off the cover letter | |
| 08:54:36 | zigo | Looks like I had a version of that patch already in my package. | |
| 08:54:42 | zigo | Not sure if it was the latest one, so trying again. | |
| 08:55:48 | sean-k-mooney | i just looked at the top few open prs so that just a guess on my part | |
| 08:56:10 | zigo | Yeah... | |
| 08:57:31 | zigo | Still broken ... :( | |
| 08:58:04 | sean-k-mooney | i would reach out to herve and see if they have any ideas on a path forward | |
| 08:58:15 | zigo | sean-k-mooney: What's the progress on eventlet removal in nova-compute? | |
| 08:58:22 | zigo | Are there patches available already? | |
| 08:58:42 | sean-k-mooney | zigo: we dont expect to complete that until 2026.2 | |
| 08:58:44 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Replace keystoneclient with keystoneauth1 https://review.opendev.org/c/openstack/nova/+/951744 | |
| 08:58:49 | sean-k-mooney | we might get the intiall verison in 2026.1 | |
| 08:58:59 | sean-k-mooney | but we are workign on the schduler first this cycle | |
| 08:59:15 | sean-k-mooney | then maybe api and or oneof the other contoler services | |
| 08:59:28 | sean-k-mooney | nova-compute is the hardest to move and will be the last service we move | |
| 09:00:08 | sean-k-mooney | our hope is that in 2026.1 we might be able to run all the nova againet in threaded mode but we dont know if we will get that far | |
| 09:00:54 | sean-k-mooney | makeing it the defautl and or remvoing eventlet supprot is a 2026.2+ activity once we have at least 1 slurp releease that suprpot threaded mode | |
| 09:01:41 | sean-k-mooney | that why we are aiming to have 2026.1 be the first release that can run without eventlet (maybe) | |
| 09:02:00 | sean-k-mooney | zigo: gibi has been makign some promissing progress but its a lot of work | |
| 09:03:44 | zigo | At the end of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413/comments/1 Guillaume Boutry says: | |
| 09:03:44 | zigo | using `gc.disable()` makes the issue disappear (yay! disable gc!) or actually holding a hardref to `admin_client.baseclient.httpclient` makes the method pass most of the time... | |
| 09:04:34 | zigo | Not sure where/how he's doing the garbage collector disabling. | |
| 09:04:45 | zigo | Is this a gc in eventlet ? | |
| 09:05:59 | sean-k-mooney | no thats the main python one | |
| 09:06:37 | sean-k-mooney | we would have to hack in explict calls to the garbage collector somewhere and have hard refences to stop it beign nuked | |
| 09:06:59 | zigo | Maybe, disabling it would be in eventlet/patcher.py ? | |
| 09:07:18 | gboutry | gc.disable() to disable the GC, but that's really NOT a good idea | |
| 09:07:41 | zigo | That would mean memory leak right? | |
| 09:07:57 | sean-k-mooney | zigo: that will prevent object created by nova forom being deallcoate dy python automaticly when we exit scope | |
| 09:08:08 | gboutry | yes, nothing would be freed anymore. | |
| 09:08:21 | sean-k-mooney | zigo: yes it would mean nova would never deallocate any python object | |
| 09:08:38 | zigo | At this point, I need to validate the release, so anything is better than completely broken. | |
| 09:08:46 | zigo | I'll try and see what happens. :) | |