| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-16 | |||
| 11:53:18 | zigo | Should I worry? | |
| 11:53:18 | zigo | https://paste.opendev.org/show/b6Z3DQTexWsKxupCuLit/ | |
| 11:53:18 | zigo | Also, I'm seeing this in my log: | |
| 11:54:02 | zigo | Missing sentence in my above paste: A dynamic interval looping call should supply either an interval or periodic_interval_max | |
| 11:54:59 | gibi | do you see that during shutdown? | |
| 11:55:04 | gibi | i.e. seeing Skipping periodic tasks during graceful shutdown. | |
| 11:55:14 | zigo | Ah, yes. | |
| 11:55:44 | zigo | Can't see anything now, because I'm restarting my CI from scratch, but yes, it was during shutdown. | |
| 11:56:38 | gibi | then I would not worry much especially if nova-compute exited | |
| 11:56:51 | gibi | which I assume did to be able to start up again | |
| 11:56:59 | gibi | I will look into it | |
| 11:57:25 | gibi | btw it seems only nova-compute and ironic vncproxy service are using no_fork=True https://codesearch.opendev.org/?q=no_fork%3DTrue&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 11:58:05 | gibi | yepp this is happening to me too during shutdown https://paste.openstack.org/show/bG4BK2BV3lcOqBKtC5ry/ | |
| 11:58:50 | gibi | https://paste.openstack.org/show/bEAmK5c6VltFQ1OmI9JD/ more complete paste | |
| 11:59:04 | gibi | so it is cosmetic, the shutdown happens | |
| 12:05:50 | opendevreview | Merged openstack/placement stable/2025.1: Reproduce bug 2160721 https://review.opendev.org/c/openstack/placement/+/998877 | |
| 12:32:00 | gibi | zigo: we should return a number here https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/manager.py#L131 to fulfill the expected proptocol of oslo and to avoid the stack trace, purely cosmetic | |
| 12:32:33 | zigo | Oh ok. | |
| 12:32:41 | zigo | Would be nice though ... :P | |
| 12:32:48 | gibi | yeah I will make a patch | |
| 12:33:08 | zigo | Any stack trace in our production triggers some alerts. :P | |
| 12:33:20 | gibi | understandable | |
| 13:16:26 | opendevreview | Merged openstack/placement stable/2025.1: Check overall capacity before generating candidates https://review.opendev.org/c/openstack/placement/+/998878 | |
| 13:36:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stop periodic tasks during shutdown https://review.opendev.org/c/openstack/nova/+/1005880 | |
| 13:36:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2167481 https://review.opendev.org/c/openstack/nova/+/1005879 | |
| 13:36:51 | gibi | zigo: ^^ | |
| 13:36:58 | gibi | this removes the stack trace | |
| 13:37:14 | gibi | cc gmaan as this is a shutdown improvement | |
| 13:39:25 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 13:46:06 | eduardomorais[m] | I'm a new contributor and I'm looking for simpler tasks to get familiar with the codebase. I found a very simple TODO and implemented what it asked for. I'd really appreciate it if you could review it! https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 13:46:55 | opendevreview | Stephen Finucane proposed openstack/placement master: Fix tests on Python 3.14 https://review.opendev.org/c/openstack/placement/+/1005883 | |
| 13:52:23 | zigo | Thanks, added to my Debian package. | |
| 13:52:55 | gibi | zigo: thanks for reporting the issues | |
| 13:54:49 | gibi | Uggla: I could not join today's upstream bug triage. Just to summarize the above. We have a regression in systemd service status reporting in nova-compute but as this is limited to that service (and maybe ironic vncproxy) zigo is OK to carry a patch in debian until we can land the fix and backport to H. So this is not an RC2 trigger in my eyes (also it is an oslo.service fix) | |
| 13:55:10 | gibi | Uggla: the secondary bug of having an extra stack trace at shutdown is just cosmetic, can wait | |
| 13:55:15 | gibi | https://bugs.launchpad.net/nova/+bug/2167481 | |
| 13:55:19 | gibi | https://bugs.launchpad.net/oslo.service/+bug/2167448 | |
| 13:55:24 | gibi | this is the two bugs | |
| 13:55:27 | opendevreview | Stephen Finucane proposed openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | |
| 13:55:33 | gibi | both have fix proposed | |
| 13:56:28 | opendevreview | Stephen Finucane proposed openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | |
| 14:10:37 | Uggla | gibi, super cool thanks. | |
| 14:12:02 | dansmith | gibi: gmaan remind me, are we aware of hanging functional tests right now? in my local series (which has some potentially impactful functional tests) I'm getting a hang | |
| 14:13:06 | gibi | dansmith: the tox cover job showed functional hangs and I have suspicion that it might reproducable outside of cover | |
| 14:14:05 | gibi | https://bugs.launchpad.net/nova/+bug/2167088 | |
| 14:14:05 | dansmith | oh right right that's what I was thinking of, and as you pointed out, the functional part is all its own command | |
| 14:14:10 | gibi | yepp | |
| 14:14:13 | gibi | this is the bug | |
| 14:14:17 | gibi | it has some hints where the hang is | |
| 14:14:30 | dansmith | I'm walking up my series a bit to see if it still happens there | |
| 14:14:32 | gibi | or at least one hang I can sometimes reproduce locally | |
| 14:14:47 | dansmith | ah, good to know.. do you have to run cover or just functional alone? | |
| 14:14:56 | gibi | so far I needed to run cover | |
| 14:15:01 | gibi | and severy limit my cpu | |
| 14:15:05 | dansmith | isn't that really bizarre? | |
| 14:15:20 | dansmith | unless it's just the regex filter that is different there or somethin | |
| 14:15:21 | gibi | systemd-run --user --scope -p CPUQuota=10% tox -e cover -- --concurrency 1 nova.tests.functional.libvirt.test_numa_live_migration.NUMALiveMigrationRollbackTests --until-failure | |
| 14:15:32 | gibi | this reproduces to me semi frequently | |
| 14:15:47 | gibi | I haven't run much of this without cover | |
| 14:15:50 | gibi | it is on my todo list | |
| 14:16:08 | dansmith | I'm seeing it on arm64 with 10 asymmetric cores, fwiw | |
| 14:16:21 | dansmith | I'll try master if it repros earlier in my set | |
| 14:16:44 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 14:18:09 | gibi | btw you are touching live migration and my repro points to a live migration test class it cannot be a coincidence | |
| 14:19:14 | dansmith | perhaps | |
| 14:19:51 | dansmith | how did you determine it was that test class? | |
| 14:24:04 | gibi | I looked at multiple failing runs to see what test cases did not run at all and which was the last test case executed successfully by the hanging worker | |
| 14:24:12 | gibi | s/failing/hanging/ | |
| 14:24:29 | gibi | then made a guess | |
| 14:25:05 | gibi | if you reproduce locally you can see the hanging worker's assigned tests in a temp file | |
| 14:25:26 | gibi | if you look at the command line of the hanging stestr process | |
| 14:25:47 | gibi | in the CI we don't have that data saved | |
| 14:26:16 | gibi | so there I can only looked at the subunit partial result to see what was finished and the rest was assumed not started or hanging | |
| 14:27:35 | dansmith | ack | |
| 14:28:02 | dansmith | it didn't hang earlier in the series, so I suspect it's something I've done to the live migration path | |
| 14:28:16 | dansmith | either a total coincidence or maybe exacerbating whatever you're seeing | |
| 14:59:03 | Uggla | Reminder upstream triage in ~30mn. (https://meet.google.com/zjr-rxus-hzj) | |
| 15:01:02 | gibi | Uggla: see my note above for the triage | |
| 15:01:07 | gibi | ahh you sas | |
| 15:01:09 | gibi | saw | |
| 15:01:14 | gibi | I missed the ack | |
| 15:01:30 | Uggla | yep, nw | |
| 15:14:39 | elodilles | Uggla: i'll skip today's meeting as I'm still lagging behind since coming back from summit :/ | |
| 15:16:05 | Uggla | elodilles nw, I do not expect lots of progress due to the release. Anyway thx for letting me know. | |
| 15:16:21 | elodilles | +1 | |
| 15:48:20 | opendevreview | ribaudr proposed openstack/nova master: Fix evacuate race conditions with periodic task and task_state https://review.opendev.org/c/openstack/nova/+/999978 | |
| 15:48:20 | opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug 2162835 https://review.opendev.org/c/openstack/nova/+/999977 | |
| 17:19:08 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 17:33:54 | gmaan | gibi_: thanks for periodic_task fix for shutdown case, I commented in gerrit https://review.opendev.org/c/openstack/nova/+/1005880/comment/2facdbfa_bb30dd75/ | |
| 17:34:36 | gmaan | basically instead of raising LoopingCallDone(), I think we should return value which makes oslo.service to gracefully shutdown peropdic task thread | |
| 17:35:09 | gmaan | gibi_: another things, we should backport this fix to stable/2026.2 right? | |
| 17:36:19 | gmaan | btw, other fix for systemd notify in olso.service lgtm. | |
| 17:40:10 | opendevreview | Merged openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | |
| 21:06:04 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 21:13:36 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 21:46:13 | ashigupt | gmaan: gibi_ Does https://review.opendev.org/c/openstack/nova/+/1001369 looks good to you now ? | |
| #openstack-nova - 2026-09-17 | |||
| 07:03:10 | gibi_ | ashigupt: I will try to get to it but at the moment it is somewhere in the middle of my todo lit | |
| 07:03:13 | gibi_ | list | |
| 07:03:13 | gibi_ | lis | |
| 08:09:16 | bauzas | hey folks, back | |