| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-04 | |||
| 09:01:41 | sean-k-mooney | that why we are aiming to have 2026.1 be the first release that can run without eventlet (maybe) | |
| 09:02:00 | sean-k-mooney | zigo: gibi has been makign some promissing progress but its a lot of work | |
| 09:03:44 | zigo | using `gc.disable()` makes the issue disappear (yay! disable gc!) or actually holding a hardref to `admin_client.baseclient.httpclient` makes the method pass most of the time... | |
| 09:03:44 | zigo | At the end of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413/comments/1 Guillaume Boutry says: | |
| 09:04:34 | zigo | Not sure where/how he's doing the garbage collector disabling. | |
| 09:04:45 | zigo | Is this a gc in eventlet ? | |
| 09:05:59 | sean-k-mooney | no thats the main python one | |
| 09:06:37 | sean-k-mooney | we would have to hack in explict calls to the garbage collector somewhere and have hard refences to stop it beign nuked | |
| 09:06:59 | zigo | Maybe, disabling it would be in eventlet/patcher.py ? | |
| 09:07:18 | gboutry | gc.disable() to disable the GC, but that's really NOT a good idea | |
| 09:07:41 | zigo | That would mean memory leak right? | |
| 09:07:57 | sean-k-mooney | zigo: that will prevent object created by nova forom being deallcoate dy python automaticly when we exit scope | |
| 09:08:08 | gboutry | yes, nothing would be freed anymore. | |
| 09:08:21 | sean-k-mooney | zigo: yes it would mean nova would never deallocate any python object | |
| 09:08:38 | zigo | At this point, I need to validate the release, so anything is better than completely broken. | |
| 09:08:46 | zigo | I'll try and see what happens. :) | |
| 09:08:54 | sean-k-mooney | well that will completely break nova | |
| 09:09:13 | sean-k-mooney | it will cause OOM kill events | |
| 09:11:20 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:12:53 | sean-k-mooney | zigo: looking at how they were tracing it https://pastebin.ubuntu.com/p/cj7tb3kmGV/ | |
| 09:13:24 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:13:38 | sean-k-mooney | i wonder if on line 37 if we did somethign like base_client_ref = admin_client.base_client.httpclient | |
| 09:13:58 | sean-k-mooney | zigo: would that keep it form being gc'd | |
| 09:16:29 | sean-k-mooney | so that is https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L1217 | |
| 09:17:16 | sean-k-mooney | we could maybve modify our get_client functionto return 3 refs one to the actully client one to the base client and one to the http clinet in the base. | |
| 09:17:28 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:17:49 | sean-k-mooney | that shoudl keep them in scope until that function body exits | |
| 09:19:45 | sean-k-mooney | although we already have a refence to the baseclinet in our wrapper | |
| 09:19:47 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L184-L185 | |
| 09:20:11 | gboutry | But that wouldn't prevent the code from breaking somewhere else? | |
| 09:20:42 | sean-k-mooney | gboutry: it woudl at best mask the issue for neturonclient | |
| 09:20:50 | sean-k-mooney | its not an actual fix. | |
| 09:21:13 | sean-k-mooney | im just tryign to thinkis there a way nova can keep the relevent object alive with a hardref fo some kind | |
| 09:21:35 | sean-k-mooney | im not seing anything obvious | |
| 09:21:59 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:24:53 | sean-k-mooney | gboutry: i need to go look at somethign else but ill quickly add a direct refernce to the http client to our client wrapper and see fi that changes anything | |
| 09:26:14 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:38:53 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:39:22 | sean-k-mooney | zigo: gboutry no idea if ^ will work but maybe it will provide more data :shrug: | |
| 09:40:19 | sean-k-mooney | well that an excelnet start... | |
| 09:42:59 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:44:27 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:44:42 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:45:41 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:45:52 | sean-k-mooney | it helps to actully save all your chagnes before commiting... | |
| 09:47:44 | sean-k-mooney | zigo: gboutry: that add a direct refence to the http clint but it also moves nova-next to try and ue 3.13 assuming it a think on ubuntu 24.04 and it also adds a 3.13 functional job to see just how broken nova really is | |
| 09:48:52 | sean-k-mooney | i expect the answer to be very but ci shoudl tell us soon assuming my hacks actully work. | |
| 10:38:24 | zigo | sean-k-mooney: I just tried, added gc.disable() at the end of eventlet/patcher.py's _green_existing_locks(), and I could spawn a VM ! | |
| 10:51:01 | sean-k-mooney | zigo: sure but you just added a out of memory issue. that not a solution you can include in your pakcaging of nova | |
| 10:51:25 | zigo | I know, I just wanted to try what gboutry wrote. | |
| 10:51:35 | zigo | I can even see the memory leak in real time. | |
| 10:51:36 | sean-k-mooney | ack | |
| 10:51:55 | zigo | nova-compute used to take 0.9% of my VM's RAM, now it's up to 1.0. | |
| 10:52:12 | sean-k-mooney | if you have debug logging enabled it will leak faster | |
| 10:52:12 | zigo | I guess it's going to never stop leaking ... | |
| 10:52:21 | zigo | I do ! :) | |
| 10:55:25 | sean-k-mooney | it looks like python3.13 is not aviable in ubuntu 24.04 is it aviabel in debian 12? | |
| 10:55:42 | sean-k-mooney | it proably in universe in noble | |
| 10:56:57 | gboutry | python 3.13 is only available through the deadsnakes ppa on noble | |
| 10:58:06 | sean-k-mooney | ack, im tryign to see if there is an easy way to hack in 3.13 in to one of our devstack jobs | |
| 10:58:24 | sean-k-mooney | i could enable that ppa in a pre playbook | |
| 10:58:37 | zigo | sean-k-mooney: No, only in Debian 13, though Trixie is in hard freeze, so it's a good moment to start using it. | |
| 10:59:24 | sean-k-mooney | zigo: well locally im using sid :) but ya its more work then i have time for today to get trixie in ci | |
| 11:01:25 | sean-k-mooney | zigo: have you spoken to the infra team about supportign it when it is released yet? | |
| 11:01:33 | stephenfin | Uggla: fyi, looks like there's a bug here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L13370 The attribute on the class is called accessmode, not access_mode | |
| 11:01:39 | zigo | Nop, no time for that yet. | |
| 11:08:44 | zigo | sean-k-mooney: It doesn't work. | |
| 11:09:11 | zigo | I mean, your patch with self._http_client = base_client.httpclient in network/neutron.py | |
| 11:09:14 | zigo | Just tried ... | |
| 11:10:28 | sean-k-mooney | ack | |
| 11:10:31 | sean-k-mooney | it was a long shot | |
| 11:10:41 | zigo | Thanks for trying ! :) | |
| 11:11:00 | sean-k-mooney | im going to try one other thing quickly to provied 3.13 via pyenv the same way we do for functional tests | |
| 11:11:03 | zigo | I'm guessing there's going to be this kind of issue a bit everywhere anyways, so we need a better global eventlet fix. | |
| 11:15:45 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 11:17:48 | sean-k-mooney | that shoudl complie 3.13 from souce on all the nodes although it might fail if we dont have gcc ectra avaibale but if it does it will fail fast | |
| 11:18:28 | sean-k-mooney | actully i shoudl add bindep first https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/tox-docs/pre.yaml#L3 | |
| 11:20:51 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 11:54:46 | gibi | zigo: sean-k-mooney: I'm torn on prioritizzng chasing down python3.13 eventlet bugs over focusing on removing eventlet from nova. Eventlet is on life support as we speak investing there is problematic. I know that I there is a mismatch between what python version OpenStack supports and what the distros want to use. Still I firmly belive we are better of removing eventlet than patching. | |
| 11:55:13 | sean-k-mooney | fun installing python 3.13. form sorce worked but https://paste.opendev.org/show/bcQL03mSHktFaQf6diQX/ mysql failed to install properly | |
| 11:55:54 | sean-k-mooney | gibi: well in epoxy 3.13 is exmperimetnal but its in the mandatory testign runtim for 2025.2 | |
| 11:56:06 | sean-k-mooney | so there kind fo need to be two efforts here | |
| 11:56:27 | sean-k-mooney | we either need to change the testing runtims or make it work on 3.13 this cycle regardless fo the eventlet removal | |
| 11:56:40 | sean-k-mooney | gibi: i think you should focus on the eventlet removal | |
| 11:57:06 | sean-k-mooney | but we need to work with the eventlet maintianers to make sure they have time to fix 3.13 | |
| 11:57:16 | sean-k-mooney | i think that were zigo and other could help with that effort | |
| 11:57:46 | sean-k-mooney | if we manage to get master workign with 3.13 we can access if we need to backprot stuff to epoxy | |
| 11:57:54 | sean-k-mooney | we do get clean unit and functional tests | |
| 11:58:08 | sean-k-mooney | so i dont think things are massively borken in nova | |
| 11:58:34 | sean-k-mooney | i.e. if the eventlet bugs were fix openstack might "just work" without code changes on expoxy | |
| 12:00:48 | gibi | we are the eventlet maintainers | |
| 12:01:07 | sean-k-mooney | kind of | |
| 12:01:30 | gibi | you can talk to hberaud about it | |
| 12:02:16 | sean-k-mooney | to me 3.13 supprot is not really optional | |
| 12:02:18 | gibi | if zigo could help maintaining eventlet in py313 that is a win for sure | |
| 12:02:56 | sean-k-mooney | by ya its partly a prioty probelm | |
| 12:03:27 | sean-k-mooney | gibi: there is currently a python discsion happenig about addign virtual tread to core pyton started by the gevent folks | |
| 12:03:28 | gibi | I think the reality is that we have limited eventlet internal knowledge to maintain it even if we find the time | |
| 12:04:21 | zigo | Hervé is in PTO this week, so I can't talk to him until Monday. | |
| 12:04:43 | gibi | zigo: yepp I know | |