| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-12 | |||
| 21:35:22 | sean-k-mooney | we swapped to the sdk about 3-4 releases ago | |
| 22:33:50 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | |
| #openstack-nova - 2026-03-13 | |||
| 06:31:57 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | |
| 09:04:57 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 09:56:13 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 10:14:25 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2026.1: Update .gitreview for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980404 | |
| 10:14:29 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2026.1: Update TOX_CONSTRAINTS_FILE for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980405 | |
| 10:14:32 | opendevreview | OpenStack Release Bot proposed openstack/nova master: Update master for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980406 | |
| 10:51:41 | opendevreview | ribaudr proposed openstack/nova-specs master: Move Gazpacho implemented specs https://review.opendev.org/c/openstack/nova-specs/+/979283 | |
| 10:58:29 | zigo | What service is impacted for this? I | |
| 10:58:29 | zigo | The default value of [DEFAULT]/graceful_shutdown_timeout has been changed from 60 to 180 seconds for all Nova services. This means that when a Nova service receives SIGTERM, it will now wait up to 180 seconds for a graceful shutdown before being forcefully terminated. | |
| 10:58:29 | zigo | From Gazpacho Nova's release notes, I can read: | |
| 10:58:29 | zigo | Hi there! | |
| 10:58:39 | zigo | I'm guessing: nova-compute and nova-conductor, right? | |
| 10:58:44 | zigo | But scheduler as well? | |
| 10:59:24 | zigo | To me, it looks like, yes (reading the code). | |
| 11:02:13 | zigo | I think it'd be nicer for Nova to read what's in systemd, using the d-bus interface. Would this kind of patch be accepted (if made optional, since not everyone uses systemd) ? | |
| 11:16:07 | gibi | gmaan: ^^ | |
| 11:16:42 | zigo | Am I missing something? | |
| 11:16:42 | zigo | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading' | |
| 11:16:42 | zigo | Also, when attempting to run Nova unit tests, I get failure during test discovery: | |
| 11:17:13 | gibi | zigo: I think we agree that synchorizing the timeouts between systemd / k8s liveness probe / etc with the nova's shutdown timout is needed. But it is hard to automate it as it depends on how nova is deployed. It is not always systemd | |
| 11:17:31 | gibi | zigo: which python version are you using for the unit test? | |
| 11:18:02 | gibi | zigo: I think the graceful shutdown timeout change effects all nova sevices except the APIs. But gmaan can confirm | |
| 11:20:19 | gibi | zigo: we see similar unit test issue in py3.14 https://zuul.opendev.org/t/openstack/build/156fe0f67bff4e4da43a31bd0f02e667 | |
| 11:20:34 | gibi | I had not time to dig. But py3.13 is pretty stable | |
| 11:21:06 | gibi | (3.14 is still not voting for us so it is easy to ignore'l | |
| 11:21:08 | gibi | ) | |
| 11:26:56 | zigo | gibi: 3.13 and 3.14. | |
| 11:27:19 | zigo | Flamingo on 3.14 was no issue. | |
| 11:27:26 | zigo | (sorry, in a meeting, wont take too long) | |
| 11:28:36 | gibi | ack. If you see the issue in 3.13 that is strange. But anyho please file a bug I will try to get to it | |
| 11:36:06 | zigo | What I'm seeing: https://paste.opendev.org/show/bB5CJ9SHLPXOjKGyu0N8/ | |
| 11:36:40 | zigo | I'm seeing the issue in 3.14 only, because it's not running 3.13 tests if 3.14 fails. | |
| 11:38:15 | zigo | gibi: Well, I can dig it out, just, do I have a workaround? Maybe export a variable so it uses only threading? | |
| 11:40:15 | zigo | Found it: OS_NOVA_DISABLE_EVENTLET_PATCHING | |
| 11:42:44 | zigo | Ok, tests are running now... Hopefully, no breakage because of this! :) | |
| 11:49:32 | zigo | gibi: How safe is it to use OS_NOVA_DISABLE_EVENTLET_PATCHING in all services? | |
| 11:49:44 | zigo | Looks like one unit test is stuck... :( | |
| 12:04:03 | opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | |
| 12:04:04 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | |
| 12:08:56 | nicolairuckel | Could anyone take a look at this? https://zuul.opendev.org/t/openstack/build/59d2bc927a5d4e40936da1ccc05722e6 The job run successfully before with exactly the same code. | |
| 12:16:32 | zigo | gibi: Unit tests are passing, if I set OS_NOVA_DISABLE_EVENTLET_PATCHING=false in d/rules. | |
| 12:49:04 | gibi | zig | |
| 12:50:02 | gibi | zigo: so using native threading via OS_NOVA_DISABLE_EVENTLET_PATCHING=true is experimental. If you can test it the we are eager to hear feedback. All services except the console proxies can be run with native threading | |
| 12:50:33 | zigo | Well, for tests to run, I had to set it to false ! :/ | |
| 12:50:34 | gibi | https://docs.openstack.org/nova/latest/admin/concurrency.html | |
| 12:50:39 | gibi | this is the main doc | |
| 12:50:50 | zigo | I'll try whenever I'm at the tempest phase. | |
| 12:51:25 | gibi | for unit testing we have an exclude list https://github.com/openstack/nova/blob/d5c64e7aba09052601b59bf1297187d85da6b90b/tox.ini#L83 | |
| 12:51:49 | gibi | functional tests does not yet work with native threading | |
| 12:52:09 | gibi | tempest passes in nova-next job where we configure each service with native threading | |
| 12:52:24 | zigo | Oh... Thanks, that's really helpful. | |
| 14:29:38 | sean-k-mooney | zigo: we have optional systemd/debus integration | |
| 14:29:46 | sean-k-mooney | for sending ready | |
| 14:29:54 | sean-k-mooney | vai the notifcaiotn socket | |
| 14:30:06 | sean-k-mooney | but im not sure we shoudl add any mroe integration for the graceful shutdown timeout | |
| 14:30:21 | sean-k-mooney | at least not nativly in nova | |
| 14:30:41 | sean-k-mooney | puting it in oslo and having nova use it might be more reasonable | |
| 14:31:46 | sean-k-mooney | zigo: by the way we see the same issue in the 3.14 tox job upstream | |
| 14:31:51 | sean-k-mooney | we just have nto had time to fix it | |
| 14:32:10 | sean-k-mooney | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading' is why that job is currently failing | |
| 14:33:27 | sean-k-mooney | zigo: one issue is we have not added py314 here yet https://github.com/openstack/nova/blob/master/tox.ini#L59 | |
| 14:34:16 | zigo | sean-k-mooney: Exactly my thought, which is why I just wrote: https://review.opendev.org/c/openstack/oslo.service/+/980440 | |
| 14:34:20 | sean-k-mooney | so its going to fallback to https://github.com/openstack/nova/blob/master/tox.ini#L55 | |
| 14:34:31 | sean-k-mooney | which does not include the exclude list | |
| 14:34:45 | zigo | I haven't used dbus-python, that'd be an external dependency, so I had to use subprocess to get the TimeoutStopUSec. | |
| 15:10:45 | gmaan | zigo: RE on graceful shutdown: yes that timeout is in effect for the services using oslo.services which are 1. compute 2. sch 3. conductor. The other remaining services api, metadata api, console proxy shutdown path is different and it depends/handled by their underlying wsgi server/websokify lib | |
| 15:12:37 | gmaan | considering the different external system to deploy the services, its not good idea to automate the timeout as per the single deployment model like systemd. We recommend to override this timeout in conf as per the external system timeout | |
| 15:15:10 | sean-k-mooney | we woudl need ot have the config take precedence anyway | |
| 15:15:25 | sean-k-mooney | we coudl perhaps take a defualt form systemd if not set in the config | |
| 15:15:52 | sean-k-mooney | but we need the config when nova is deployed in a continer or on a system without systemd | |
| 15:17:15 | gmaan | yeah, I am not sure how much value it will add to just automate for systemd instead operator checking and setting it in config if current default is not good is better and explicit way | |
| 15:17:54 | gmaan | i think default in systemd is 60 sec which is anyways not enough for nova as we have manager wait time default to 160 | |
| 15:20:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move py312-threading to py313 https://review.opendev.org/c/openstack/nova/+/980458 | |
| 15:20:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix running test with py314 https://review.opendev.org/c/openstack/nova/+/980457 | |
| 15:20:34 | gibi | sean-k-mooney: zigo: ^^ fix for the py314 unit test job | |
| 15:21:04 | sean-k-mooney | gibi: cool i just didnt have a time to look into that myself but ill review thoese before i leave today | |
| 15:36:27 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | |
| 15:36:27 | opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | |
| 15:53:01 | dansmith | nicolairuckel: around by chance? | |
| 15:55:56 | nicolairuckel | dansmith: I just sat down on my computer | |
| 15:55:58 | nicolairuckel | good timing | |
| 15:56:08 | nicolairuckel | at my computer | |
| 15:56:13 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 15:56:13 | nicolairuckel | on would be weird I guess | |
| 15:56:34 | dansmith | nicolairuckel: hey, so I was trying to verify your nvram fix (which maybe I should have done before) but.. I think I must be missing something... | |
| 15:57:00 | nicolairuckel | the backport for 2025.2? | |
| 15:57:00 | dansmith | nicolairuckel: to make sure it works, I was going to just tweak the efibootmgr config a bit and then power cycle the instance to make sure it kept the tweak | |
| 15:57:08 | dansmith | nicolairuckel: actually just the thing we already merged | |
| 15:57:35 | dansmith | so if I create a new fake boot entry and then reboot, it's gone when it comes back, but I see my label string in the VARS.fd for the instance | |
| 15:58:10 | dansmith | what I really wanted to do was just use efivar to poke something in there to look for, but was never able to get that undocumented monstrosity to do anything :P | |
| 15:58:37 | dansmith | anyway, am I missing something about the "template" behavior overwriting the boot config as expected behavior or something? | |
| 15:59:02 | nicolairuckel | that sounds odd to me. In my testing the variable and content were still there. | |
| 15:59:43 | dansmith | all I did was set the firmware type to uefi on the image.. does the platform need to be q35 or something maybe? I wasn't expecting that to matter but... | |
| 16:00:13 | nicolairuckel | did you use efivar to set the variable? | |
| 16:00:26 | dansmith | no, I used efibootmgr | |
| 16:00:34 | dansmith | if you can tell me how to get efivar to work I'd love to hear it :) | |
| 16:00:45 | nicolairuckel | one second | |
| 16:01:17 | nicolairuckel | what I did was the following | |