| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-03 | |||
| 20:49:13 | sean-k-mooney | gmaan: i have not reached out on the mailing list yet | |
| 20:49:16 | gmaan | Its up to you. I mean I can raise it in TC if you need help on this but wanted to check with you if any response you had other than gerrit? | |
| 20:49:56 | sean-k-mooney | what i can do is send a mail to the list with the cyborg and tc tags and ask for review on the gate blocker | |
| 20:50:14 | sean-k-mooney | if we dont get a responce to that in a week or so we can bring it up in the next relevent tc meeting | |
| 20:50:20 | gmaan | ++, if no response then I can add it in TC agenda to discuss | |
| 20:50:24 | gmaan | sounds good | |
| 20:50:48 | sean-k-mooney | im just finishing for today but ill do that shortly before i do | |
| 20:51:00 | gmaan | it is helpful because that helps TC to mark project status which is required by the release team before m-2 | |
| 20:51:08 | gmaan | sure, no hurry | |
| 20:51:21 | gmaan | it was open in one of my tab so thought of pinging you | |
| 21:18:11 | sean-k-mooney | i sent a mail with subject "[cyborg][nova][tc] cyborg support for pyproject.toml and failing ci jobs." | |
| 21:18:49 | sean-k-mooney | gmaan: i did not offer to help maintian cyborg by reviewing the bot pathces and ci fixes liek this in that mail | |
| 21:19:32 | sean-k-mooney | if we get no reponce i might be able to help with that this cycle but at taht poitn we would liekly need to have the tc review cyborg overall health | |
| 21:21:19 | gmaan | yeah, fair enough. | |
| 21:22:39 | sean-k-mooney | ok its 22:22 which means i start about 12 hours ago so now is a good time to warp up before i start somethign else | |
| 21:22:45 | sean-k-mooney | chat to you tommrow o/ | |
| 21:26:19 | gmaan | o/ | |
| #openstack-nova - 2025-06-04 | |||
| 07:20:02 | zigo | Hi there! | |
| 07:20:02 | zigo | Under trixie (aka: Debian 13) + epoxy, I get lots of stack dumps like this one in nova-conductor.log: | |
| 07:20:02 | zigo | https://paste.opendev.org/show/bECP9P2mCs2kelxQhbgK/ | |
| 07:20:02 | zigo | Has anyone seen something like this ? | |
| 07:22:51 | zigo | Looks like I'm also getting this when spawning a VM: | |
| 07:22:51 | zigo | https://paste.opendev.org/show/bOocYhL8mYOV0jf3m7ip/ | |
| 07:22:58 | zigo | Not sure if the 2 are related... | |
| 07:28:57 | sean-k-mooney | zigo: are you use galera or something like that? nova does not suppprot Active active galera nor those most of openstack for that matter | |
| 07:29:22 | zigo | Yes, but everything goes through *one* single node for writing. | |
| 07:29:31 | zigo | So it's more like active/passive... | |
| 07:29:45 | sean-k-mooney | ok that does work | |
| 07:29:50 | sean-k-mooney | its what we use downstream too | |
| 07:30:27 | sean-k-mooney | am the only other thing that comes to mind is maybe you have two compute service (condocutors in this case) with the same conf.HOST | |
| 07:30:59 | sean-k-mooney | i.e. you might have two conductor binaires trying to update teh same service record | |
| 07:31:00 | zigo | In my setup, conductor nodes are not compute nodes. | |
| 07:32:08 | zigo | My host = directive isn't set. | |
| 07:32:16 | zigo | So probably it's trying to use the FQDN ? | |
| 07:32:30 | sean-k-mooney | it will use socket.hostname() | |
| 07:32:57 | sean-k-mooney | by default becaus by defualt openstack requires unique hostnames not just unique FQDNs | |
| 07:33:08 | zigo | Right. | |
| 07:33:19 | sean-k-mooney | if you have two instance of the conductor running on the same host by mistake | |
| 07:33:29 | zigo | I don't. | |
| 07:33:32 | sean-k-mooney | then it could cuase the conductor issue but i have not seen it other then that | |
| 07:33:39 | zigo | I'm not running in containers, so that would be impossible. | |
| 07:33:39 | sean-k-mooney | your compute issue looks diffent | |
| 07:34:34 | sean-k-mooney | without digin in too deeply it looks like perhaps you have an incompatible verion of libs | |
| 07:34:39 | zigo | To me, it seems like I could ignore the nova-conductor reporting issue. It's ugly, but the service is still reported as alive, so that's ok. | |
| 07:34:53 | zigo | Of what lib?!? | |
| 07:34:53 | sean-k-mooney | perhaps an old neutron client? | |
| 07:35:16 | zigo | Unlikely, it does work under bookworm + epoxy, and that's the same version of the libs. | |
| 07:35:42 | zigo | ie: neutronclient 11.4.0-2 | |
| 07:36:02 | sean-k-mooney | well the error is coming form keystonauth within neutronclient | |
| 07:36:24 | zigo | Right, though it's saying "no attribute 'endpoint_override'" which is weird. | |
| 07:36:35 | zigo | FYI, I did set endpoint_override in nova.conf for the URL of Neutron. | |
| 07:37:07 | sean-k-mooney | what is you keystonauth version | |
| 07:37:31 | zigo | 5.10.0 | |
| 07:38:00 | zigo | So both keystoneauth1 and neutronclient as the released version for Epoxy. | |
| 07:39:16 | sean-k-mooney | ya that shoudl be new enough | |
| 07:39:50 | sean-k-mooney | so in neutronclinet its failling here https://github.com/openstack/python-neutronclient/blob/11.4.0/neutronclient/client.py#L348 and that is just delegating to keystonatuh so i guess ill check there next | |
| 07:40:38 | sean-k-mooney | so its calling https://github.com/openstack/keystoneauth/blob/5.10.0/keystoneauth1/adapter.py#L313-L334 | |
| 07:42:23 | zigo | I'll have a try when not setting endpoint_override in my config file. | |
| 07:42:40 | zigo | What's weird is that I didn't get this under Bookworm. | |
| 07:43:00 | zigo | So, this smells like a non-openstack-maintained lib is at play here. | |
| 07:44:05 | sean-k-mooney | that should not result in an atibute error | |
| 07:44:13 | sean-k-mooney | that what is really odd | |
| 07:46:01 | sean-k-mooney | nova is still properly registring the relevent config options, if that was broken https://docs.openstack.org/nova/latest/configuration/config.html#neutron.endpoint_override woudl not render | |
| 07:46:20 | sean-k-mooney | so it should be vaild to set that direcly | |
| 07:47:04 | zigo | Removing endpoint_override form nova.conf doesn't fix anything. :/ | |
| 07:47:28 | zigo | I still get the same stack trace. | |
| 07:52:26 | sean-k-mooney | zigo: ya the stack track looks like someting is wrong in our packigng somehow | |
| 07:52:48 | sean-k-mooney | have you chekced the content of /usr/lib/python3/dist-packages/keystoneauth1/adapter.py | |
| 07:53:06 | sean-k-mooney | and confirmed that the base adapter in the file on disk has endpoint overried | |
| 07:53:17 | sean-k-mooney | and tries to import that in a python terminal and acess it? | |
| 07:53:46 | sean-k-mooney | or even one step back | |
| 07:53:59 | sean-k-mooney | if you use the neutron client cli does it have the same issue | |
| 07:54:27 | sean-k-mooney | i woudl expect a neutron port list to be broken in the same way on that host | |
| 07:57:46 | zigo | I'm not sure what / how you're asking me to check. | |
| 07:59:07 | zigo | :( | |
| 08:01:27 | sean-k-mooney | grep endpoint_override /usr/lib/python3/dist-packages/keystoneauth1/adapter.py | |
| 08:01:43 | sean-k-mooney | and also check if `neutron port-list` works | |
| 08:03:00 | zigo | openstack port list works, indeed. | |
| 08:03:03 | sean-k-mooney | the neutron cli supprot clouds.yaml so you can just use it like osc | |
| 08:03:11 | sean-k-mooney | no not openstack port list | |
| 08:03:26 | sean-k-mooney | exiplcited `neutron port-list` | |
| 08:03:38 | zigo | I don't think neutronclient is providing an /usr/bin/neutron anymore. | |
| 08:03:39 | sean-k-mooney | openstack port list uses the openstack sdk not the neutron clinet i think | |
| 08:03:52 | sean-k-mooney | they may have droped it in epoxy | |
| 08:05:03 | zigo | https://paste.opendev.org/show/bWJcYW3nYUAkjMn71RN6/ | |
| 08:07:08 | sean-k-mooney | ya they did. you code on disk all looks correct toteh 5.10.0 tag | |
| 08:07:27 | jkulik | https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413 could that be related? looks like having the same stacktrace | |
| 08:08:03 | sean-k-mooney | oh perhaps | |
| 08:08:23 | sean-k-mooney | if eventlet is nuking part of the obejct by garbage collecting too early | |
| 08:08:45 | zigo | Oh, thanks ! :) | |
| 08:08:47 | sean-k-mooney | that does look like the same o r a very similar trace | |
| 08:09:04 | zigo | I'm not surprised if eventlet is playing with me, as I'm on Python 3.13. | |
| 08:09:13 | sean-k-mooney | zigo: that is unfortually not very good news for yo since it means it because eventlet does not supprot 3.13 yet | |
| 08:09:24 | sean-k-mooney | which is why openstack does nto supprot 3.13 yet | |
| 08:09:34 | zigo | sean-k-mooney: I have no choice, 3.13 it is ... | |
| 08:09:54 | zigo | I was expecting things would go wrong. | |
| 08:10:04 | sean-k-mooney | so i belive there is an eventlet issue for this | |
| 08:10:15 | sean-k-mooney | we also know the thread id si broken | |
| 08:10:19 | zigo | Maybe I should try the latest eventlet release. | |
| 08:10:27 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/1032 | |