| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-16 | |||
| 11:12:22 | gibi | thanks that is a good data point | |
| 11:13:58 | zigo | I'm asking GLM-5 if he knows why... :P | |
| 11:14:19 | gibi | yepp I see that locally too (super conductor is not affected even though running in threading mode) | |
| 11:26:24 | gibi | zigo: it is due to cotyledon + nova-compute using no-fork=True | |
| 11:26:32 | zigo | Ok, the difference, according to my AI, is because nova-conductor calls server(server, workers=N) with default to no_fork=False, but nova-compute calls it with the added no_fork=True, added in https://review.opendev.org/c/openstack/nova/+/965466 (Do not fork compute workers in native threading mode) that actually, you're the patch owner. | |
| 11:26:39 | zigo | Yeah... :P | |
| 11:26:44 | gibi | our LLMs are agreeing :) | |
| 11:26:52 | gibi | I will draft a fix | |
| 11:27:26 | zigo | gibi: Do you think we should fix this is olso.service rather than in Nova ? | |
| 11:27:30 | zigo | (I do...) | |
| 11:29:10 | gibi | I do too | |
| 11:29:18 | gibi | as olso.service give the support for no-fork | |
| 11:29:33 | gibi | but this probably means no fix in Hibiscus GA | |
| 11:29:54 | gibi | or at least it become more complicated | |
| 11:29:59 | gibi | due to req freeze | |
| 11:30:19 | gibi | but it also means this bug only affects services that are not forking | |
| 11:30:59 | gibi | (this is not first bug due to nova-compute being special, so we should also think about working there a single worker and basically load the nova-compute service there) | |
| 11:31:12 | gibi | s/working/forking/ | |
| 11:36:36 | zigo | Well, I can carry the patch in my Debian package. It's really only 2 lines, as much as I can see. | |
| 11:39:08 | zigo | https://review.opendev.org/c/openstack/oslo.service/+/1005864 | |
| 11:39:10 | zigo | :) | |
| 11:39:57 | gibi | you type faster than me | |
| 11:39:58 | gibi | :) | |
| 11:40:10 | gibi | but I tested exactly the same thing now locally | |
| 11:40:15 | gibi | and it resolve the prolbme | |
| 11:41:14 | zigo | I'll try to patch my oslo.service Debian package and see if nova-compute is happy. :P | |
| 11:41:36 | gibi | it is happy for me :) | |
| 11:42:17 | gibi | please add a Closes-Bug: #2167448 line to the commit message otherwise it looks good to me | |
| 11:44:00 | opendevreview | Ashish Gupta proposed openstack/nova master: Fix init_host crash migration test under native threading https://review.opendev.org/c/openstack/nova/+/1001369 | |
| 11:45:01 | zigo | Right. | |
| 11:51:04 | zigo | Yes, works for me as well ! :) | |
| 11:51:45 | zigo | I can see "Creating RPC server for service: nova-compute on topic: compute start" in my logs, what's that for?!? | |
| 11:53:18 | zigo | Also, I'm seeing this in my log: | |
| 11:53:18 | zigo | https://paste.opendev.org/show/b6Z3DQTexWsKxupCuLit/ | |
| 11:53:18 | zigo | Should I worry? | |
| 11:54:02 | zigo | Missing sentence in my above paste: A dynamic interval looping call should supply either an interval or periodic_interval_max | |
| 11:54:59 | gibi | do you see that during shutdown? | |
| 11:55:04 | gibi | i.e. seeing Skipping periodic tasks during graceful shutdown. | |
| 11:55:14 | zigo | Ah, yes. | |
| 11:55:44 | zigo | Can't see anything now, because I'm restarting my CI from scratch, but yes, it was during shutdown. | |
| 11:56:38 | gibi | then I would not worry much especially if nova-compute exited | |
| 11:56:51 | gibi | which I assume did to be able to start up again | |
| 11:56:59 | gibi | I will look into it | |
| 11:57:25 | gibi | btw it seems only nova-compute and ironic vncproxy service are using no_fork=True https://codesearch.opendev.org/?q=no_fork%3DTrue&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 11:58:05 | gibi | yepp this is happening to me too during shutdown https://paste.openstack.org/show/bG4BK2BV3lcOqBKtC5ry/ | |
| 11:58:50 | gibi | https://paste.openstack.org/show/bEAmK5c6VltFQ1OmI9JD/ more complete paste | |
| 11:59:04 | gibi | so it is cosmetic, the shutdown happens | |
| 12:05:50 | opendevreview | Merged openstack/placement stable/2025.1: Reproduce bug 2160721 https://review.opendev.org/c/openstack/placement/+/998877 | |
| 12:32:00 | gibi | zigo: we should return a number here https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/manager.py#L131 to fulfill the expected proptocol of oslo and to avoid the stack trace, purely cosmetic | |
| 12:32:33 | zigo | Oh ok. | |
| 12:32:41 | zigo | Would be nice though ... :P | |
| 12:32:48 | gibi | yeah I will make a patch | |
| 12:33:08 | zigo | Any stack trace in our production triggers some alerts. :P | |
| 12:33:20 | gibi | understandable | |
| 13:16:26 | opendevreview | Merged openstack/placement stable/2025.1: Check overall capacity before generating candidates https://review.opendev.org/c/openstack/placement/+/998878 | |
| 13:36:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2167481 https://review.opendev.org/c/openstack/nova/+/1005879 | |
| 13:36:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stop periodic tasks during shutdown https://review.opendev.org/c/openstack/nova/+/1005880 | |
| 13:36:51 | gibi | zigo: ^^ | |
| 13:36:58 | gibi | this removes the stack trace | |
| 13:37:14 | gibi | cc gmaan as this is a shutdown improvement | |
| 13:39:25 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 13:46:06 | eduardomorais[m] | I'm a new contributor and I'm looking for simpler tasks to get familiar with the codebase. I found a very simple TODO and implemented what it asked for. I'd really appreciate it if you could review it! https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 13:46:55 | opendevreview | Stephen Finucane proposed openstack/placement master: Fix tests on Python 3.14 https://review.opendev.org/c/openstack/placement/+/1005883 | |
| 13:52:23 | zigo | Thanks, added to my Debian package. | |
| 13:52:55 | gibi | zigo: thanks for reporting the issues | |
| 13:54:49 | gibi | Uggla: I could not join today's upstream bug triage. Just to summarize the above. We have a regression in systemd service status reporting in nova-compute but as this is limited to that service (and maybe ironic vncproxy) zigo is OK to carry a patch in debian until we can land the fix and backport to H. So this is not an RC2 trigger in my eyes (also it is an oslo.service fix) | |
| 13:55:10 | gibi | Uggla: the secondary bug of having an extra stack trace at shutdown is just cosmetic, can wait | |
| 13:55:15 | gibi | https://bugs.launchpad.net/nova/+bug/2167481 | |
| 13:55:19 | gibi | https://bugs.launchpad.net/oslo.service/+bug/2167448 | |
| 13:55:24 | gibi | this is the two bugs | |
| 13:55:27 | opendevreview | Stephen Finucane proposed openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | |
| 13:55:33 | gibi | both have fix proposed | |
| 13:56:28 | opendevreview | Stephen Finucane proposed openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | |
| 14:10:37 | Uggla | gibi, super cool thanks. | |
| 14:12:02 | dansmith | gibi: gmaan remind me, are we aware of hanging functional tests right now? in my local series (which has some potentially impactful functional tests) I'm getting a hang | |
| 14:13:06 | gibi | dansmith: the tox cover job showed functional hangs and I have suspicion that it might reproducable outside of cover | |
| 14:14:05 | dansmith | oh right right that's what I was thinking of, and as you pointed out, the functional part is all its own command | |
| 14:14:05 | gibi | https://bugs.launchpad.net/nova/+bug/2167088 | |
| 14:14:10 | gibi | yepp | |
| 14:14:13 | gibi | this is the bug | |
| 14:14:17 | gibi | it has some hints where the hang is | |
| 14:14:30 | dansmith | I'm walking up my series a bit to see if it still happens there | |
| 14:14:32 | gibi | or at least one hang I can sometimes reproduce locally | |
| 14:14:47 | dansmith | ah, good to know.. do you have to run cover or just functional alone? | |
| 14:14:56 | gibi | so far I needed to run cover | |
| 14:15:01 | gibi | and severy limit my cpu | |
| 14:15:05 | dansmith | isn't that really bizarre? | |
| 14:15:20 | dansmith | unless it's just the regex filter that is different there or somethin | |
| 14:15:21 | gibi | systemd-run --user --scope -p CPUQuota=10% tox -e cover -- --concurrency 1 nova.tests.functional.libvirt.test_numa_live_migration.NUMALiveMigrationRollbackTests --until-failure | |
| 14:15:32 | gibi | this reproduces to me semi frequently | |
| 14:15:47 | gibi | I haven't run much of this without cover | |
| 14:15:50 | gibi | it is on my todo list | |
| 14:16:08 | dansmith | I'm seeing it on arm64 with 10 asymmetric cores, fwiw | |
| 14:16:21 | dansmith | I'll try master if it repros earlier in my set | |
| 14:16:44 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 14:18:09 | gibi | btw you are touching live migration and my repro points to a live migration test class it cannot be a coincidence | |
| 14:19:14 | dansmith | perhaps | |
| 14:19:51 | dansmith | how did you determine it was that test class? | |
| 14:24:04 | gibi | I looked at multiple failing runs to see what test cases did not run at all and which was the last test case executed successfully by the hanging worker | |
| 14:24:12 | gibi | s/failing/hanging/ | |