| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-04 | |||
| 08:22:05 | sean-k-mooney | its not an openstack bug currently | |
| 08:22:23 | sean-k-mooney | its an eventlet one so ther isnt anythign nova can do to enable this | |
| 08:22:58 | sean-k-mooney | as an assid e apprenetly there is a propsoal to add "virtual thread" ala greentthread to core python https://discuss.python.org/t/add-virtual-threads-to-python/91403 | |
| 08:44:45 | zigo | I'll still have a try with eventlet 0.40 and see how it goes. | |
| 08:48:02 | zigo | Same stuff with eventlet 0.40 ... :( | |
| 08:49:50 | sean-k-mooney | there is thsi work in progress hack https://github.com/eventlet/eventlet/pull/1031/files | |
| 08:49:57 | sean-k-mooney | but that not complete | |
| 08:50:15 | sean-k-mooney | and its techinially for the thread issue rather then the one you currently hittign with the GC | |
| 08:50:26 | sean-k-mooney | that may get you slighly closer howerever | |
| 08:50:36 | zigo | Will try the patch ! :) | |
| 08:50:40 | zigo | I was in fact looking into it. | |
| 08:52:40 | sean-k-mooney | its just a guess but https://github.com/eventlet/eventlet/pull/1042 might be related to ge gc issue | |
| 08:53:06 | sean-k-mooney | although it does not directly refence the exiting issue so im just going off the cover letter | |
| 08:54:36 | zigo | Looks like I had a version of that patch already in my package. | |
| 08:54:42 | zigo | Not sure if it was the latest one, so trying again. | |
| 08:55:48 | sean-k-mooney | i just looked at the top few open prs so that just a guess on my part | |
| 08:56:10 | zigo | Yeah... | |
| 08:57:31 | zigo | Still broken ... :( | |
| 08:58:04 | sean-k-mooney | i would reach out to herve and see if they have any ideas on a path forward | |
| 08:58:15 | zigo | sean-k-mooney: What's the progress on eventlet removal in nova-compute? | |
| 08:58:22 | zigo | Are there patches available already? | |
| 08:58:42 | sean-k-mooney | zigo: we dont expect to complete that until 2026.2 | |
| 08:58:44 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Replace keystoneclient with keystoneauth1 https://review.opendev.org/c/openstack/nova/+/951744 | |
| 08:58:49 | sean-k-mooney | we might get the intiall verison in 2026.1 | |
| 08:58:59 | sean-k-mooney | but we are workign on the schduler first this cycle | |
| 08:59:15 | sean-k-mooney | then maybe api and or oneof the other contoler services | |
| 08:59:28 | sean-k-mooney | nova-compute is the hardest to move and will be the last service we move | |
| 09:00:08 | sean-k-mooney | our hope is that in 2026.1 we might be able to run all the nova againet in threaded mode but we dont know if we will get that far | |
| 09:00:54 | sean-k-mooney | makeing it the defautl and or remvoing eventlet supprot is a 2026.2+ activity once we have at least 1 slurp releease that suprpot threaded mode | |
| 09:01:41 | sean-k-mooney | that why we are aiming to have 2026.1 be the first release that can run without eventlet (maybe) | |
| 09:02:00 | sean-k-mooney | zigo: gibi has been makign some promissing progress but its a lot of work | |
| 09:03:44 | zigo | At the end of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413/comments/1 Guillaume Boutry says: | |
| 09:03:44 | zigo | using `gc.disable()` makes the issue disappear (yay! disable gc!) or actually holding a hardref to `admin_client.baseclient.httpclient` makes the method pass most of the time... | |
| 09:04:34 | zigo | Not sure where/how he's doing the garbage collector disabling. | |
| 09:04:45 | zigo | Is this a gc in eventlet ? | |
| 09:05:59 | sean-k-mooney | no thats the main python one | |
| 09:06:37 | sean-k-mooney | we would have to hack in explict calls to the garbage collector somewhere and have hard refences to stop it beign nuked | |
| 09:06:59 | zigo | Maybe, disabling it would be in eventlet/patcher.py ? | |
| 09:07:18 | gboutry | gc.disable() to disable the GC, but that's really NOT a good idea | |
| 09:07:41 | zigo | That would mean memory leak right? | |
| 09:07:57 | sean-k-mooney | zigo: that will prevent object created by nova forom being deallcoate dy python automaticly when we exit scope | |
| 09:08:08 | gboutry | yes, nothing would be freed anymore. | |
| 09:08:21 | sean-k-mooney | zigo: yes it would mean nova would never deallocate any python object | |
| 09:08:38 | zigo | At this point, I need to validate the release, so anything is better than completely broken. | |
| 09:08:46 | zigo | I'll try and see what happens. :) | |
| 09:08:54 | sean-k-mooney | well that will completely break nova | |
| 09:09:13 | sean-k-mooney | it will cause OOM kill events | |
| 09:11:20 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:12:53 | sean-k-mooney | zigo: looking at how they were tracing it https://pastebin.ubuntu.com/p/cj7tb3kmGV/ | |
| 09:13:24 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:13:38 | sean-k-mooney | i wonder if on line 37 if we did somethign like base_client_ref = admin_client.base_client.httpclient | |
| 09:13:58 | sean-k-mooney | zigo: would that keep it form being gc'd | |
| 09:16:29 | sean-k-mooney | so that is https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L1217 | |
| 09:17:16 | sean-k-mooney | we could maybve modify our get_client functionto return 3 refs one to the actully client one to the base client and one to the http clinet in the base. | |
| 09:17:28 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:17:49 | sean-k-mooney | that shoudl keep them in scope until that function body exits | |
| 09:19:45 | sean-k-mooney | although we already have a refence to the baseclinet in our wrapper | |
| 09:19:47 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L184-L185 | |
| 09:20:11 | gboutry | But that wouldn't prevent the code from breaking somewhere else? | |
| 09:20:42 | sean-k-mooney | gboutry: it woudl at best mask the issue for neturonclient | |
| 09:20:50 | sean-k-mooney | its not an actual fix. | |
| 09:21:13 | sean-k-mooney | im just tryign to thinkis there a way nova can keep the relevent object alive with a hardref fo some kind | |
| 09:21:35 | sean-k-mooney | im not seing anything obvious | |
| 09:21:59 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:24:53 | sean-k-mooney | gboutry: i need to go look at somethign else but ill quickly add a direct refernce to the http client to our client wrapper and see fi that changes anything | |
| 09:26:14 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:38:53 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:39:22 | sean-k-mooney | zigo: gboutry no idea if ^ will work but maybe it will provide more data :shrug: | |
| 09:40:19 | sean-k-mooney | well that an excelnet start... | |
| 09:42:59 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:44:27 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:44:42 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:45:41 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:45:52 | sean-k-mooney | it helps to actully save all your chagnes before commiting... | |
| 09:47:44 | sean-k-mooney | zigo: gboutry: that add a direct refence to the http clint but it also moves nova-next to try and ue 3.13 assuming it a think on ubuntu 24.04 and it also adds a 3.13 functional job to see just how broken nova really is | |
| 09:48:52 | sean-k-mooney | i expect the answer to be very but ci shoudl tell us soon assuming my hacks actully work. | |
| 10:38:24 | zigo | sean-k-mooney: I just tried, added gc.disable() at the end of eventlet/patcher.py's _green_existing_locks(), and I could spawn a VM ! | |
| 10:51:01 | sean-k-mooney | zigo: sure but you just added a out of memory issue. that not a solution you can include in your pakcaging of nova | |
| 10:51:25 | zigo | I know, I just wanted to try what gboutry wrote. | |
| 10:51:35 | zigo | I can even see the memory leak in real time. | |
| 10:51:36 | sean-k-mooney | ack | |
| 10:51:55 | zigo | nova-compute used to take 0.9% of my VM's RAM, now it's up to 1.0. | |
| 10:52:12 | zigo | I guess it's going to never stop leaking ... | |
| 10:52:12 | sean-k-mooney | if you have debug logging enabled it will leak faster | |
| 10:52:21 | zigo | I do ! :) | |
| 10:55:25 | sean-k-mooney | it looks like python3.13 is not aviable in ubuntu 24.04 is it aviabel in debian 12? | |
| 10:55:42 | sean-k-mooney | it proably in universe in noble | |
| 10:56:57 | gboutry | python 3.13 is only available through the deadsnakes ppa on noble | |
| 10:58:06 | sean-k-mooney | ack, im tryign to see if there is an easy way to hack in 3.13 in to one of our devstack jobs | |
| 10:58:24 | sean-k-mooney | i could enable that ppa in a pre playbook | |
| 10:58:37 | zigo | sean-k-mooney: No, only in Debian 13, though Trixie is in hard freeze, so it's a good moment to start using it. | |
| 10:59:24 | sean-k-mooney | zigo: well locally im using sid :) but ya its more work then i have time for today to get trixie in ci | |
| 11:01:25 | sean-k-mooney | zigo: have you spoken to the infra team about supportign it when it is released yet? | |
| 11:01:33 | stephenfin | Uggla: fyi, looks like there's a bug here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L13370 The attribute on the class is called accessmode, not access_mode | |
| 11:01:39 | zigo | Nop, no time for that yet. | |
| 11:08:44 | zigo | sean-k-mooney: It doesn't work. | |
| 11:09:11 | zigo | I mean, your patch with self._http_client = base_client.httpclient in network/neutron.py | |
| 11:09:14 | zigo | Just tried ... | |
| 11:10:28 | sean-k-mooney | ack | |
| 11:10:31 | sean-k-mooney | it was a long shot | |