| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-12 | |||
| 21:27:17 | sean-k-mooney | so maybe if it was integrated via that | |
| 21:27:45 | sean-k-mooney | Matko[m]: it woudl proably be better to also check with the ironic folks | |
| 21:28:09 | sean-k-mooney | this i beilve is what it sued form that lib https://opendev.org/openstack/tooz/src/branch/master/tooz/hashring.py#L30 | |
| 21:29:37 | sean-k-mooney | Matko[m]: thos ecould also be commign form the keystoneAuth optiosn we add | |
| 21:29:44 | Matko[m] | yes, I found the usage of tooz. However there is nothing feeding the config to tooz | |
| 21:30:06 | Matko[m] | I did look into keystoneauth too | |
| 21:30:09 | Matko[m] | There is no trace of memcache in keystoneauth project: https://github.com/search?q=repo%3Aopenstack%2Fkeystoneauth+memcache&type=code | |
| 21:30:09 | Matko[m] | No dependency on oslo.cache or dogpile that might consume the config: https://github.com/openstack/keystoneauth/blob/master/requirements.txt | |
| 21:30:37 | Matko[m] | I'll ask the Ironic team then. | |
| 21:30:54 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conf/ironic.py | |
| 21:31:10 | sean-k-mooney | that is where that config section is regeistered and its not an option we are addign explcitly | |
| 21:31:37 | sean-k-mooney | so that measn it coming from either keystoen auth or the sdk i think | |
| 21:31:46 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conf/ironic.py#L128-L131 | |
| 21:32:42 | sean-k-mooney | so if those have any effect i suspect it via the sdk/keystoneAuth common logic not anythin ironic specific | |
| 21:33:10 | Matko[m] | I looked into those as well and couldn't find anything. The openstack-helm code I linked to was added 9 years ago. And I wonder/suspect if it was a bad copy/paste from similar code in the chart for keystone_authtoken which has memcache configs. | |
| 21:33:38 | sean-k-mooney | did you check our current config ref | |
| 21:33:43 | Matko[m] | yes | |
| 21:33:47 | Matko[m] | I would also assume list_opts would list and include all the supported configs in the docs. | |
| 21:34:03 | sean-k-mooney | ya we generate the config ref form that | |
| 21:34:09 | Matko[m] | https://docs.openstack.org/nova/latest/configuration/config.html#ironic | |
| 21:34:37 | Matko[m] | if it doesn't ring a bell to anybody, then I'll conclude it was a bad copy/paste. | |
| 21:34:43 | sean-k-mooney | i dont see them there | |
| 21:34:49 | sean-k-mooney | so i think they are not used any more | |
| 21:35:13 | Matko[m] | alright, thanks for your help and time | |
| 21:35:14 | sean-k-mooney | i suspect they may have been used when we use the python-ironic client | |
| 21:35:22 | sean-k-mooney | we swapped to the sdk about 3-4 releases ago | |
| 22:33:50 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | |
| #openstack-nova - 2026-03-13 | |||
| 06:31:57 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | |
| 09:04:57 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 09:56:13 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 10:14:25 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2026.1: Update .gitreview for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980404 | |
| 10:14:29 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2026.1: Update TOX_CONSTRAINTS_FILE for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980405 | |
| 10:14:32 | opendevreview | OpenStack Release Bot proposed openstack/nova master: Update master for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980406 | |
| 10:51:41 | opendevreview | ribaudr proposed openstack/nova-specs master: Move Gazpacho implemented specs https://review.opendev.org/c/openstack/nova-specs/+/979283 | |
| 10:58:29 | zigo | Hi there! | |
| 10:58:29 | zigo | From Gazpacho Nova's release notes, I can read: | |
| 10:58:29 | zigo | The default value of [DEFAULT]/graceful_shutdown_timeout has been changed from 60 to 180 seconds for all Nova services. This means that when a Nova service receives SIGTERM, it will now wait up to 180 seconds for a graceful shutdown before being forcefully terminated. | |
| 10:58:29 | zigo | What service is impacted for this? I | |
| 10:58:39 | zigo | I'm guessing: nova-compute and nova-conductor, right? | |
| 10:58:44 | zigo | But scheduler as well? | |
| 10:59:24 | zigo | To me, it looks like, yes (reading the code). | |
| 11:02:13 | zigo | I think it'd be nicer for Nova to read what's in systemd, using the d-bus interface. Would this kind of patch be accepted (if made optional, since not everyone uses systemd) ? | |
| 11:16:07 | gibi | gmaan: ^^ | |
| 11:16:42 | zigo | Also, when attempting to run Nova unit tests, I get failure during test discovery: | |
| 11:16:42 | zigo | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading' | |
| 11:16:42 | zigo | Am I missing something? | |
| 11:17:13 | gibi | zigo: I think we agree that synchorizing the timeouts between systemd / k8s liveness probe / etc with the nova's shutdown timout is needed. But it is hard to automate it as it depends on how nova is deployed. It is not always systemd | |
| 11:17:31 | gibi | zigo: which python version are you using for the unit test? | |
| 11:18:02 | gibi | zigo: I think the graceful shutdown timeout change effects all nova sevices except the APIs. But gmaan can confirm | |
| 11:20:19 | gibi | zigo: we see similar unit test issue in py3.14 https://zuul.opendev.org/t/openstack/build/156fe0f67bff4e4da43a31bd0f02e667 | |
| 11:20:34 | gibi | I had not time to dig. But py3.13 is pretty stable | |
| 11:21:06 | gibi | (3.14 is still not voting for us so it is easy to ignore'l | |
| 11:21:08 | gibi | ) | |
| 11:26:56 | zigo | gibi: 3.13 and 3.14. | |
| 11:27:19 | zigo | Flamingo on 3.14 was no issue. | |
| 11:27:26 | zigo | (sorry, in a meeting, wont take too long) | |
| 11:28:36 | gibi | ack. If you see the issue in 3.13 that is strange. But anyho please file a bug I will try to get to it | |
| 11:36:06 | zigo | What I'm seeing: https://paste.opendev.org/show/bB5CJ9SHLPXOjKGyu0N8/ | |
| 11:36:40 | zigo | I'm seeing the issue in 3.14 only, because it's not running 3.13 tests if 3.14 fails. | |
| 11:38:15 | zigo | gibi: Well, I can dig it out, just, do I have a workaround? Maybe export a variable so it uses only threading? | |
| 11:40:15 | zigo | Found it: OS_NOVA_DISABLE_EVENTLET_PATCHING | |
| 11:42:44 | zigo | Ok, tests are running now... Hopefully, no breakage because of this! :) | |
| 11:49:32 | zigo | gibi: How safe is it to use OS_NOVA_DISABLE_EVENTLET_PATCHING in all services? | |
| 11:49:44 | zigo | Looks like one unit test is stuck... :( | |
| 12:04:03 | opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | |
| 12:04:04 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | |
| 12:08:56 | nicolairuckel | Could anyone take a look at this? https://zuul.opendev.org/t/openstack/build/59d2bc927a5d4e40936da1ccc05722e6 The job run successfully before with exactly the same code. | |
| 12:16:32 | zigo | gibi: Unit tests are passing, if I set OS_NOVA_DISABLE_EVENTLET_PATCHING=false in d/rules. | |
| 12:49:04 | gibi | zig | |
| 12:50:02 | gibi | zigo: so using native threading via OS_NOVA_DISABLE_EVENTLET_PATCHING=true is experimental. If you can test it the we are eager to hear feedback. All services except the console proxies can be run with native threading | |
| 12:50:33 | zigo | Well, for tests to run, I had to set it to false ! :/ | |
| 12:50:34 | gibi | https://docs.openstack.org/nova/latest/admin/concurrency.html | |
| 12:50:39 | gibi | this is the main doc | |
| 12:50:50 | zigo | I'll try whenever I'm at the tempest phase. | |
| 12:51:25 | gibi | for unit testing we have an exclude list https://github.com/openstack/nova/blob/d5c64e7aba09052601b59bf1297187d85da6b90b/tox.ini#L83 | |
| 12:51:49 | gibi | functional tests does not yet work with native threading | |
| 12:52:09 | gibi | tempest passes in nova-next job where we configure each service with native threading | |
| 12:52:24 | zigo | Oh... Thanks, that's really helpful. | |
| 14:29:38 | sean-k-mooney | zigo: we have optional systemd/debus integration | |
| 14:29:46 | sean-k-mooney | for sending ready | |
| 14:29:54 | sean-k-mooney | vai the notifcaiotn socket | |
| 14:30:06 | sean-k-mooney | but im not sure we shoudl add any mroe integration for the graceful shutdown timeout | |
| 14:30:21 | sean-k-mooney | at least not nativly in nova | |
| 14:30:41 | sean-k-mooney | puting it in oslo and having nova use it might be more reasonable | |
| 14:31:46 | sean-k-mooney | zigo: by the way we see the same issue in the 3.14 tox job upstream | |
| 14:31:51 | sean-k-mooney | we just have nto had time to fix it | |
| 14:32:10 | sean-k-mooney | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading' is why that job is currently failing | |
| 14:33:27 | sean-k-mooney | zigo: one issue is we have not added py314 here yet https://github.com/openstack/nova/blob/master/tox.ini#L59 | |
| 14:34:16 | zigo | sean-k-mooney: Exactly my thought, which is why I just wrote: https://review.opendev.org/c/openstack/oslo.service/+/980440 | |
| 14:34:20 | sean-k-mooney | so its going to fallback to https://github.com/openstack/nova/blob/master/tox.ini#L55 | |
| 14:34:31 | sean-k-mooney | which does not include the exclude list | |
| 14:34:45 | zigo | I haven't used dbus-python, that'd be an external dependency, so I had to use subprocess to get the TimeoutStopUSec. | |
| 15:10:45 | gmaan | zigo: RE on graceful shutdown: yes that timeout is in effect for the services using oslo.services which are 1. compute 2. sch 3. conductor. The other remaining services api, metadata api, console proxy shutdown path is different and it depends/handled by their underlying wsgi server/websokify lib | |
| 15:12:37 | gmaan | considering the different external system to deploy the services, its not good idea to automate the timeout as per the single deployment model like systemd. We recommend to override this timeout in conf as per the external system timeout | |
| 15:15:10 | sean-k-mooney | we woudl need ot have the config take precedence anyway | |
| 15:15:25 | sean-k-mooney | we coudl perhaps take a defualt form systemd if not set in the config | |
| 15:15:52 | sean-k-mooney | but we need the config when nova is deployed in a continer or on a system without systemd | |
| 15:17:15 | gmaan | yeah, I am not sure how much value it will add to just automate for systemd instead operator checking and setting it in config if current default is not good is better and explicit way | |
| 15:17:54 | gmaan | i think default in systemd is 60 sec which is anyways not enough for nova as we have manager wait time default to 160 | |
| 15:20:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix running test with py314 https://review.opendev.org/c/openstack/nova/+/980457 | |