| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-04 | |||
| 09:19:47 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L184-L185 | |
| 09:20:11 | gboutry | But that wouldn't prevent the code from breaking somewhere else? | |
| 09:20:42 | sean-k-mooney | gboutry: it woudl at best mask the issue for neturonclient | |
| 09:20:50 | sean-k-mooney | its not an actual fix. | |
| 09:21:13 | sean-k-mooney | im just tryign to thinkis there a way nova can keep the relevent object alive with a hardref fo some kind | |
| 09:21:35 | sean-k-mooney | im not seing anything obvious | |
| 09:21:59 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:24:53 | sean-k-mooney | gboutry: i need to go look at somethign else but ill quickly add a direct refernce to the http client to our client wrapper and see fi that changes anything | |
| 09:26:14 | opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update installation documentation https://review.opendev.org/c/openstack/nova/+/938680 | |
| 09:38:53 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:39:22 | sean-k-mooney | zigo: gboutry no idea if ^ will work but maybe it will provide more data :shrug: | |
| 09:40:19 | sean-k-mooney | well that an excelnet start... | |
| 09:42:59 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:44:27 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:44:42 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:45:41 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 09:45:52 | sean-k-mooney | it helps to actully save all your chagnes before commiting... | |
| 09:47:44 | sean-k-mooney | zigo: gboutry: that add a direct refence to the http clint but it also moves nova-next to try and ue 3.13 assuming it a think on ubuntu 24.04 and it also adds a 3.13 functional job to see just how broken nova really is | |
| 09:48:52 | sean-k-mooney | i expect the answer to be very but ci shoudl tell us soon assuming my hacks actully work. | |
| 10:38:24 | zigo | sean-k-mooney: I just tried, added gc.disable() at the end of eventlet/patcher.py's _green_existing_locks(), and I could spawn a VM ! | |
| 10:51:01 | sean-k-mooney | zigo: sure but you just added a out of memory issue. that not a solution you can include in your pakcaging of nova | |
| 10:51:25 | zigo | I know, I just wanted to try what gboutry wrote. | |
| 10:51:35 | zigo | I can even see the memory leak in real time. | |
| 10:51:36 | sean-k-mooney | ack | |
| 10:51:55 | zigo | nova-compute used to take 0.9% of my VM's RAM, now it's up to 1.0. | |
| 10:52:12 | sean-k-mooney | if you have debug logging enabled it will leak faster | |
| 10:52:12 | zigo | I guess it's going to never stop leaking ... | |
| 10:52:21 | zigo | I do ! :) | |
| 10:55:25 | sean-k-mooney | it looks like python3.13 is not aviable in ubuntu 24.04 is it aviabel in debian 12? | |
| 10:55:42 | sean-k-mooney | it proably in universe in noble | |
| 10:56:57 | gboutry | python 3.13 is only available through the deadsnakes ppa on noble | |
| 10:58:06 | sean-k-mooney | ack, im tryign to see if there is an easy way to hack in 3.13 in to one of our devstack jobs | |
| 10:58:24 | sean-k-mooney | i could enable that ppa in a pre playbook | |
| 10:58:37 | zigo | sean-k-mooney: No, only in Debian 13, though Trixie is in hard freeze, so it's a good moment to start using it. | |
| 10:59:24 | sean-k-mooney | zigo: well locally im using sid :) but ya its more work then i have time for today to get trixie in ci | |
| 11:01:25 | sean-k-mooney | zigo: have you spoken to the infra team about supportign it when it is released yet? | |
| 11:01:33 | stephenfin | Uggla: fyi, looks like there's a bug here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L13370 The attribute on the class is called accessmode, not access_mode | |
| 11:01:39 | zigo | Nop, no time for that yet. | |
| 11:08:44 | zigo | sean-k-mooney: It doesn't work. | |
| 11:09:11 | zigo | I mean, your patch with self._http_client = base_client.httpclient in network/neutron.py | |
| 11:09:14 | zigo | Just tried ... | |
| 11:10:28 | sean-k-mooney | ack | |
| 11:10:31 | sean-k-mooney | it was a long shot | |
| 11:10:41 | zigo | Thanks for trying ! :) | |
| 11:11:00 | sean-k-mooney | im going to try one other thing quickly to provied 3.13 via pyenv the same way we do for functional tests | |
| 11:11:03 | zigo | I'm guessing there's going to be this kind of issue a bit everywhere anyways, so we need a better global eventlet fix. | |
| 11:15:45 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 11:17:48 | sean-k-mooney | that shoudl complie 3.13 from souce on all the nodes although it might fail if we dont have gcc ectra avaibale but if it does it will fail fast | |
| 11:18:28 | sean-k-mooney | actully i shoudl add bindep first https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/tox-docs/pre.yaml#L3 | |
| 11:20:51 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749 | |
| 11:54:46 | gibi | zigo: sean-k-mooney: I'm torn on prioritizzng chasing down python3.13 eventlet bugs over focusing on removing eventlet from nova. Eventlet is on life support as we speak investing there is problematic. I know that I there is a mismatch between what python version OpenStack supports and what the distros want to use. Still I firmly belive we are better of removing eventlet than patching. | |
| 11:55:13 | sean-k-mooney | fun installing python 3.13. form sorce worked but https://paste.opendev.org/show/bcQL03mSHktFaQf6diQX/ mysql failed to install properly | |
| 11:55:54 | sean-k-mooney | gibi: well in epoxy 3.13 is exmperimetnal but its in the mandatory testign runtim for 2025.2 | |
| 11:56:06 | sean-k-mooney | so there kind fo need to be two efforts here | |
| 11:56:27 | sean-k-mooney | we either need to change the testing runtims or make it work on 3.13 this cycle regardless fo the eventlet removal | |
| 11:56:40 | sean-k-mooney | gibi: i think you should focus on the eventlet removal | |
| 11:57:06 | sean-k-mooney | but we need to work with the eventlet maintianers to make sure they have time to fix 3.13 | |
| 11:57:16 | sean-k-mooney | i think that were zigo and other could help with that effort | |
| 11:57:46 | sean-k-mooney | if we manage to get master workign with 3.13 we can access if we need to backprot stuff to epoxy | |
| 11:57:54 | sean-k-mooney | we do get clean unit and functional tests | |
| 11:58:08 | sean-k-mooney | so i dont think things are massively borken in nova | |
| 11:58:34 | sean-k-mooney | i.e. if the eventlet bugs were fix openstack might "just work" without code changes on expoxy | |
| 12:00:48 | gibi | we are the eventlet maintainers | |
| 12:01:07 | sean-k-mooney | kind of | |
| 12:01:30 | gibi | you can talk to hberaud about it | |
| 12:02:16 | sean-k-mooney | to me 3.13 supprot is not really optional | |
| 12:02:18 | gibi | if zigo could help maintaining eventlet in py313 that is a win for sure | |
| 12:02:56 | sean-k-mooney | by ya its partly a prioty probelm | |
| 12:03:27 | sean-k-mooney | gibi: there is currently a python discsion happenig about addign virtual tread to core pyton started by the gevent folks | |
| 12:03:28 | gibi | I think the reality is that we have limited eventlet internal knowledge to maintain it even if we find the time | |
| 12:04:21 | zigo | Hervé is in PTO this week, so I can't talk to him until Monday. | |
| 12:04:43 | gibi | zigo: yepp I know | |
| 12:39:38 | Uggla | stephenfin, thanks finding this bug. I'll fix it asap. | |
| 14:23:07 | Uggla | gibi, fyi I have reviewed the first patch of your eventlet serie and left comments/questions. | |
| 14:27:01 | gibi | Uggla: thanks a lot. I will reply you probably tomorrow | |
| 14:29:43 | Uggla | gibi, sure no hurries. As I mentioned it took me a while to review this first patch, I just hope I have not asked too much silly questions ? | |
| 14:31:25 | gibi | all questions are usefull :) | |
| 14:31:28 | gibi | so no worries | |
| 17:47:19 | opendevreview | Dan Smith proposed openstack/nova master: Remove contrib/clean-on-delete.py https://review.opendev.org/c/openstack/nova/+/950592 | |
| #openstack-nova - 2025-06-05 | |||
| 01:07:39 | gmaan | stephenfin | |
| 01:08:15 | gmaan | stephenfin: when you have time, can you please check these specs https://review.opendev.org/c/openstack/nova-specs/+/937650 https://review.opendev.org/c/openstack/nova-specs/+/951218 | |
| 10:11:11 | stephenfin | gmaan: ack, will do | |
| 10:12:06 | stephenfin | gmaan: let me know if you want to discuss https://review.opendev.org/c/openstack/nova/+/936368 too. I'm *very* surprised there's been so much pushback on that, since it seems a no-brainer and is spec-compliant. I suspect you (and Uggla) might be missing something and that I haven't explained it properly | |
| 13:21:25 | opendevreview | Merged openstack/nova-specs master: Re-propose API policy service role spec https://review.opendev.org/c/openstack/nova-specs/+/951218 | |
| 13:54:11 | gibi | Uggla: thanks for the review on the eventlet-removal series. I replied. I will respin the series based on you comments as there are points that worth to mention in the commit messages / doc tests | |
| 13:55:49 | Uggla | gibi, you're welcome. I have not continue reviewing the serie today, but I will try tomorrow or next week. | |
| 13:56:37 | gibi | cool | |
| 14:06:25 | Uggla | Maybe I should recommend for reviewers to read your latest blog post, before reviewing the first patch. It clarifies the context and the "service worker" bug. | |
| 14:08:06 | gibi | yeah the blog is there to give more context that is hard to give via a single commit message as it is not related to a single commit but a series of commits / events | |
| 14:32:16 | stephenfin | Uggla: Got another one for you to revisit when you have a chance https://review.opendev.org/c/openstack/nova/+/937047 | |
| 15:10:42 | gmaan | stephenfin: thanks for review specs. for 936368, I will ping you top of hr, hope that is not late for u. | |
| 15:10:54 | stephenfin | ack | |
| 16:43:30 | gmaan | stephenfin: I read your reply, let me check some test for non-string as part of jsonschema as whole. I agree not to include those in format-checker tests | |
| 16:46:14 | opendevreview | Pierre Riteau proposed openstack/nova master: doc: Fix Caracal release number https://review.opendev.org/c/openstack/nova/+/951883 | |
| 17:22:19 | gmaan | stephenfin: I am good on 936368, found the testing in controller tests which is what I was looking for | |
| 17:23:31 | gmaan | sean-k-mooney: in case you forgot :), manager role spec is ready https://review.opendev.org/c/openstack/nova-specs/+/937650 | |
| 17:29:08 | sean-k-mooney | ill take a look later this evening | |
| 17:29:55 | sean-k-mooney | ah you renamed the file so i cant just see what chagned between v5 and v9 no worreis | |
| 17:46:37 | gmaan | for v5, yes it is renamed but you can see from ps7-ps9 (your last +1 ) https://review.opendev.org/c/openstack/nova-specs/+/937650/7..9 | |
| 18:02:19 | sean-k-mooney | ack im writing something up which will take me about anouther hour or so. ill try an take a look after that | |