| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-04 | |||
| 08:08:45 | zigo | Oh, thanks ! :) | |
| 08:08:47 | sean-k-mooney | that does look like the same o r a very similar trace | |
| 08:09:04 | zigo | I'm not surprised if eventlet is playing with me, as I'm on Python 3.13. | |
| 08:09:13 | sean-k-mooney | zigo: that is unfortually not very good news for yo since it means it because eventlet does not supprot 3.13 yet | |
| 08:09:24 | sean-k-mooney | which is why openstack does nto supprot 3.13 yet | |
| 08:09:34 | zigo | sean-k-mooney: I have no choice, 3.13 it is ... | |
| 08:09:54 | zigo | I was expecting things would go wrong. | |
| 08:10:04 | sean-k-mooney | so i belive there is an eventlet issue for this | |
| 08:10:15 | sean-k-mooney | we also know the thread id si broken | |
| 08:10:19 | zigo | Maybe I should try the latest eventlet release. | |
| 08:10:27 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/1032 | |
| 08:10:31 | zigo | Yeah, which is probably what's breaking nova-conductor too. | |
| 08:10:33 | sean-k-mooney | zigo: its not fix yet | |
| 08:10:57 | gboutry | That was exactly the error I got zigo, this is python 3.13 and eventlet not playing nice together | |
| 08:11:08 | zigo | SHIT ! :( | |
| 08:11:15 | gboutry | the error would manifest with attributes in the object that was initialized correctly | |
| 08:11:18 | zigo | Die eventlet die ... | |
| 08:16:12 | sean-k-mooney | i have commeted on both the 3.13 bugs to say we need to supprot 3.13 in eventlet for master this cycle per the project runtimes and that its required to complete the eventlet removal | |
| 08:16:26 | sean-k-mooney | the real quetion is will anyoen have time to actully work on that | |
| 08:16:54 | sean-k-mooney | this should be highlighted in the eventlet removal channel too i guess | |
| 08:17:15 | zigo | Well, for me, this means there wont be a working OpenStack for the Trixie release. That's really bad ... :( | |
| 08:17:42 | sean-k-mooney | cannonical are in the same situration for ubuntu 25.04 | |
| 08:17:56 | zigo | Except it's not an LTS, so they don't really care. | |
| 08:18:08 | sean-k-mooney | its the first tiem in openstack's history that im aware of that the latest point release of ubuntu has not supproted the latest openstack release | |
| 08:18:13 | zigo | They mostly provide OpenStack on top of LTS. | |
| 08:18:21 | sean-k-mooney | they ship it in both | |
| 08:18:25 | zigo | Right. | |
| 08:18:33 | sean-k-mooney | even if most of there custoemr are not on the point releases | |
| 08:18:44 | sean-k-mooney | it where they get ther early qa | |
| 08:20:03 | sean-k-mooney | hopefully herve and co will have time to look at the reported bugs | |
| 08:20:06 | zigo | Epoxy is not a skipable release, so it *must* be fixed. | |
| 08:20:56 | zigo | Saying, it's ok, Flamigo will have the fix, is not an option. | |
| 08:22:05 | sean-k-mooney | its not an openstack bug currently | |
| 08:22:23 | sean-k-mooney | its an eventlet one so ther isnt anythign nova can do to enable this | |
| 08:22:58 | sean-k-mooney | as an assid e apprenetly there is a propsoal to add "virtual thread" ala greentthread to core python https://discuss.python.org/t/add-virtual-threads-to-python/91403 | |
| 08:44:45 | zigo | I'll still have a try with eventlet 0.40 and see how it goes. | |
| 08:48:02 | zigo | Same stuff with eventlet 0.40 ... :( | |
| 08:49:50 | sean-k-mooney | there is thsi work in progress hack https://github.com/eventlet/eventlet/pull/1031/files | |
| 08:49:57 | sean-k-mooney | but that not complete | |
| 08:50:15 | sean-k-mooney | and its techinially for the thread issue rather then the one you currently hittign with the GC | |
| 08:50:26 | sean-k-mooney | that may get you slighly closer howerever | |
| 08:50:36 | zigo | Will try the patch ! :) | |
| 08:50:40 | zigo | I was in fact looking into it. | |
| 08:52:40 | sean-k-mooney | its just a guess but https://github.com/eventlet/eventlet/pull/1042 might be related to ge gc issue | |
| 08:53:06 | sean-k-mooney | although it does not directly refence the exiting issue so im just going off the cover letter | |
| 08:54:36 | zigo | Looks like I had a version of that patch already in my package. | |
| 08:54:42 | zigo | Not sure if it was the latest one, so trying again. | |
| 08:55:48 | sean-k-mooney | i just looked at the top few open prs so that just a guess on my part | |
| 08:56:10 | zigo | Yeah... | |
| 08:57:31 | zigo | Still broken ... :( | |
| 08:58:04 | sean-k-mooney | i would reach out to herve and see if they have any ideas on a path forward | |
| 08:58:15 | zigo | sean-k-mooney: What's the progress on eventlet removal in nova-compute? | |
| 08:58:22 | zigo | Are there patches available already? | |
| 08:58:42 | sean-k-mooney | zigo: we dont expect to complete that until 2026.2 | |
| 08:58:44 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Replace keystoneclient with keystoneauth1 https://review.opendev.org/c/openstack/nova/+/951744 | |
| 08:58:49 | sean-k-mooney | we might get the intiall verison in 2026.1 | |
| 08:58:59 | sean-k-mooney | but we are workign on the schduler first this cycle | |
| 08:59:15 | sean-k-mooney | then maybe api and or oneof the other contoler services | |
| 08:59:28 | sean-k-mooney | nova-compute is the hardest to move and will be the last service we move | |
| 09:00:08 | sean-k-mooney | our hope is that in 2026.1 we might be able to run all the nova againet in threaded mode but we dont know if we will get that far | |
| 09:00:54 | sean-k-mooney | makeing it the defautl and or remvoing eventlet supprot is a 2026.2+ activity once we have at least 1 slurp releease that suprpot threaded mode | |
| 09:01:41 | sean-k-mooney | that why we are aiming to have 2026.1 be the first release that can run without eventlet (maybe) | |
| 09:02:00 | sean-k-mooney | zigo: gibi has been makign some promissing progress but its a lot of work | |
| 09:03:44 | zigo | using `gc.disable()` makes the issue disappear (yay! disable gc!) or actually holding a hardref to `admin_client.baseclient.httpclient` makes the method pass most of the time... | |
| 09:03:44 | zigo | At the end of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413/comments/1 Guillaume Boutry says: | |
| 09:04:34 | zigo | Not sure where/how he's doing the garbage collector disabling. | |
| 09:04:45 | zigo | Is this a gc in eventlet ? | |
| 09:05:59 | sean-k-mooney | no thats the main python one | |
| 09:06:37 | sean-k-mooney | we would have to hack in explict calls to the garbage collector somewhere and have hard refences to stop it beign nuked | |
| 09:06:59 | zigo | Maybe, disabling it would be in eventlet/patcher.py ? | |
| 09:07:18 | gboutry | gc.disable() to disable the GC, but that's really NOT a good idea | |
| 09:07:41 | zigo | That would mean memory leak right? | |
| 09:07:57 | sean-k-mooney | zigo: that will prevent object created by nova forom being deallcoate dy python automaticly when we exit scope | |
| 09:08:08 | gboutry | yes, nothing would be freed anymore. | |
| 09:08:21 | sean-k-mooney | zigo: yes it would mean nova would never deallocate any python object | |
| 09:08:38 | zigo | At this point, I need to validate the release, so anything is better than completely broken. | |
| 09:08:46 | zigo | I'll try and see what happens. :) | |
| 09:08:54 | sean-k-mooney | well that will completely break nova | |
| 09:09:13 | sean-k-mooney | it will cause OOM kill events | |
| 09:11:20 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:12:53 | sean-k-mooney | zigo: looking at how they were tracing it https://pastebin.ubuntu.com/p/cj7tb3kmGV/ | |
| 09:13:24 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:13:38 | sean-k-mooney | i wonder if on line 37 if we did somethign like base_client_ref = admin_client.base_client.httpclient | |
| 09:13:58 | sean-k-mooney | zigo: would that keep it form being gc'd | |
| 09:16:29 | sean-k-mooney | so that is https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L1217 | |
| 09:17:16 | sean-k-mooney | we could maybve modify our get_client functionto return 3 refs one to the actully client one to the base client and one to the http clinet in the base. | |
| 09:17:28 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:17:49 | sean-k-mooney | that shoudl keep them in scope until that function body exits | |
| 09:19:45 | sean-k-mooney | although we already have a refence to the baseclinet in our wrapper | |
| 09:19:47 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L184-L185 | |
| 09:20:11 | gboutry | But that wouldn't prevent the code from breaking somewhere else? | |
| 09:20:42 | sean-k-mooney | gboutry: it woudl at best mask the issue for neturonclient | |
| 09:20:50 | sean-k-mooney | its not an actual fix. | |
| 09:21:13 | sean-k-mooney | im just tryign to thinkis there a way nova can keep the relevent object alive with a hardref fo some kind | |
| 09:21:35 | sean-k-mooney | im not seing anything obvious | |
| 09:21:59 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:24:53 | sean-k-mooney | gboutry: i need to go look at somethign else but ill quickly add a direct refernce to the http client to our client wrapper and see fi that changes anything | |
| 09:26:14 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:38:53 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:39:22 | sean-k-mooney | zigo: gboutry no idea if ^ will work but maybe it will provide more data :shrug: | |