| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-04 | |||
| 07:31:00 | zigo | In my setup, conductor nodes are not compute nodes. | |
| 07:32:08 | zigo | My host = directive isn't set. | |
| 07:32:16 | zigo | So probably it's trying to use the FQDN ? | |
| 07:32:30 | sean-k-mooney | it will use socket.hostname() | |
| 07:32:57 | sean-k-mooney | by default becaus by defualt openstack requires unique hostnames not just unique FQDNs | |
| 07:33:08 | zigo | Right. | |
| 07:33:19 | sean-k-mooney | if you have two instance of the conductor running on the same host by mistake | |
| 07:33:29 | zigo | I don't. | |
| 07:33:32 | sean-k-mooney | then it could cuase the conductor issue but i have not seen it other then that | |
| 07:33:39 | zigo | I'm not running in containers, so that would be impossible. | |
| 07:33:39 | sean-k-mooney | your compute issue looks diffent | |
| 07:34:34 | sean-k-mooney | without digin in too deeply it looks like perhaps you have an incompatible verion of libs | |
| 07:34:39 | zigo | To me, it seems like I could ignore the nova-conductor reporting issue. It's ugly, but the service is still reported as alive, so that's ok. | |
| 07:34:53 | zigo | Of what lib?!? | |
| 07:34:53 | sean-k-mooney | perhaps an old neutron client? | |
| 07:35:16 | zigo | Unlikely, it does work under bookworm + epoxy, and that's the same version of the libs. | |
| 07:35:42 | zigo | ie: neutronclient 11.4.0-2 | |
| 07:36:02 | sean-k-mooney | well the error is coming form keystonauth within neutronclient | |
| 07:36:24 | zigo | Right, though it's saying "no attribute 'endpoint_override'" which is weird. | |
| 07:36:35 | zigo | FYI, I did set endpoint_override in nova.conf for the URL of Neutron. | |
| 07:37:07 | sean-k-mooney | what is you keystonauth version | |
| 07:37:31 | zigo | 5.10.0 | |
| 07:38:00 | zigo | So both keystoneauth1 and neutronclient as the released version for Epoxy. | |
| 07:39:16 | sean-k-mooney | ya that shoudl be new enough | |
| 07:39:50 | sean-k-mooney | so in neutronclinet its failling here https://github.com/openstack/python-neutronclient/blob/11.4.0/neutronclient/client.py#L348 and that is just delegating to keystonatuh so i guess ill check there next | |
| 07:40:38 | sean-k-mooney | so its calling https://github.com/openstack/keystoneauth/blob/5.10.0/keystoneauth1/adapter.py#L313-L334 | |
| 07:42:23 | zigo | I'll have a try when not setting endpoint_override in my config file. | |
| 07:42:40 | zigo | What's weird is that I didn't get this under Bookworm. | |
| 07:43:00 | zigo | So, this smells like a non-openstack-maintained lib is at play here. | |
| 07:44:05 | sean-k-mooney | that should not result in an atibute error | |
| 07:44:13 | sean-k-mooney | that what is really odd | |
| 07:46:01 | sean-k-mooney | nova is still properly registring the relevent config options, if that was broken https://docs.openstack.org/nova/latest/configuration/config.html#neutron.endpoint_override woudl not render | |
| 07:46:20 | sean-k-mooney | so it should be vaild to set that direcly | |
| 07:47:04 | zigo | Removing endpoint_override form nova.conf doesn't fix anything. :/ | |
| 07:47:28 | zigo | I still get the same stack trace. | |
| 07:52:26 | sean-k-mooney | zigo: ya the stack track looks like someting is wrong in our packigng somehow | |
| 07:52:48 | sean-k-mooney | have you chekced the content of /usr/lib/python3/dist-packages/keystoneauth1/adapter.py | |
| 07:53:06 | sean-k-mooney | and confirmed that the base adapter in the file on disk has endpoint overried | |
| 07:53:17 | sean-k-mooney | and tries to import that in a python terminal and acess it? | |
| 07:53:46 | sean-k-mooney | or even one step back | |
| 07:53:59 | sean-k-mooney | if you use the neutron client cli does it have the same issue | |
| 07:54:27 | sean-k-mooney | i woudl expect a neutron port list to be broken in the same way on that host | |
| 07:57:46 | zigo | I'm not sure what / how you're asking me to check. | |
| 07:59:07 | zigo | :( | |
| 08:01:27 | sean-k-mooney | grep endpoint_override /usr/lib/python3/dist-packages/keystoneauth1/adapter.py | |
| 08:01:43 | sean-k-mooney | and also check if `neutron port-list` works | |
| 08:03:00 | zigo | openstack port list works, indeed. | |
| 08:03:03 | sean-k-mooney | the neutron cli supprot clouds.yaml so you can just use it like osc | |
| 08:03:11 | sean-k-mooney | no not openstack port list | |
| 08:03:26 | sean-k-mooney | exiplcited `neutron port-list` | |
| 08:03:38 | zigo | I don't think neutronclient is providing an /usr/bin/neutron anymore. | |
| 08:03:39 | sean-k-mooney | openstack port list uses the openstack sdk not the neutron clinet i think | |
| 08:03:52 | sean-k-mooney | they may have droped it in epoxy | |
| 08:05:03 | zigo | https://paste.opendev.org/show/bWJcYW3nYUAkjMn71RN6/ | |
| 08:07:08 | sean-k-mooney | ya they did. you code on disk all looks correct toteh 5.10.0 tag | |
| 08:07:27 | jkulik | https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413 could that be related? looks like having the same stacktrace | |
| 08:08:03 | sean-k-mooney | oh perhaps | |
| 08:08:23 | sean-k-mooney | if eventlet is nuking part of the obejct by garbage collecting too early | |
| 08:08:45 | zigo | Oh, thanks ! :) | |
| 08:08:47 | sean-k-mooney | that does look like the same o r a very similar trace | |
| 08:09:04 | zigo | I'm not surprised if eventlet is playing with me, as I'm on Python 3.13. | |
| 08:09:13 | sean-k-mooney | zigo: that is unfortually not very good news for yo since it means it because eventlet does not supprot 3.13 yet | |
| 08:09:24 | sean-k-mooney | which is why openstack does nto supprot 3.13 yet | |
| 08:09:34 | zigo | sean-k-mooney: I have no choice, 3.13 it is ... | |
| 08:09:54 | zigo | I was expecting things would go wrong. | |
| 08:10:04 | sean-k-mooney | so i belive there is an eventlet issue for this | |
| 08:10:15 | sean-k-mooney | we also know the thread id si broken | |
| 08:10:19 | zigo | Maybe I should try the latest eventlet release. | |
| 08:10:27 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/1032 | |
| 08:10:31 | zigo | Yeah, which is probably what's breaking nova-conductor too. | |
| 08:10:33 | sean-k-mooney | zigo: its not fix yet | |
| 08:10:57 | gboutry | That was exactly the error I got zigo, this is python 3.13 and eventlet not playing nice together | |
| 08:11:08 | zigo | SHIT ! :( | |
| 08:11:15 | gboutry | the error would manifest with attributes in the object that was initialized correctly | |
| 08:11:18 | zigo | Die eventlet die ... | |
| 08:16:12 | sean-k-mooney | i have commeted on both the 3.13 bugs to say we need to supprot 3.13 in eventlet for master this cycle per the project runtimes and that its required to complete the eventlet removal | |
| 08:16:26 | sean-k-mooney | the real quetion is will anyoen have time to actully work on that | |
| 08:16:54 | sean-k-mooney | this should be highlighted in the eventlet removal channel too i guess | |
| 08:17:15 | zigo | Well, for me, this means there wont be a working OpenStack for the Trixie release. That's really bad ... :( | |
| 08:17:42 | sean-k-mooney | cannonical are in the same situration for ubuntu 25.04 | |
| 08:17:56 | zigo | Except it's not an LTS, so they don't really care. | |
| 08:18:08 | sean-k-mooney | its the first tiem in openstack's history that im aware of that the latest point release of ubuntu has not supproted the latest openstack release | |
| 08:18:13 | zigo | They mostly provide OpenStack on top of LTS. | |
| 08:18:21 | sean-k-mooney | they ship it in both | |
| 08:18:25 | zigo | Right. | |
| 08:18:33 | sean-k-mooney | even if most of there custoemr are not on the point releases | |
| 08:18:44 | sean-k-mooney | it where they get ther early qa | |
| 08:20:03 | sean-k-mooney | hopefully herve and co will have time to look at the reported bugs | |
| 08:20:06 | zigo | Epoxy is not a skipable release, so it *must* be fixed. | |
| 08:20:56 | zigo | Saying, it's ok, Flamigo will have the fix, is not an option. | |
| 08:22:05 | sean-k-mooney | its not an openstack bug currently | |
| 08:22:23 | sean-k-mooney | its an eventlet one so ther isnt anythign nova can do to enable this | |
| 08:22:58 | sean-k-mooney | as an assid e apprenetly there is a propsoal to add "virtual thread" ala greentthread to core python https://discuss.python.org/t/add-virtual-threads-to-python/91403 | |
| 08:44:45 | zigo | I'll still have a try with eventlet 0.40 and see how it goes. | |
| 08:48:02 | zigo | Same stuff with eventlet 0.40 ... :( | |
| 08:49:50 | sean-k-mooney | there is thsi work in progress hack https://github.com/eventlet/eventlet/pull/1031/files | |
| 08:49:57 | sean-k-mooney | but that not complete | |
| 08:50:15 | sean-k-mooney | and its techinially for the thread issue rather then the one you currently hittign with the GC | |
| 08:50:26 | sean-k-mooney | that may get you slighly closer howerever | |
| 08:50:36 | zigo | Will try the patch ! :) | |