| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-16 | |||
| 14:12:02 | dansmith | gibi: gmaan remind me, are we aware of hanging functional tests right now? in my local series (which has some potentially impactful functional tests) I'm getting a hang | |
| 14:13:06 | gibi | dansmith: the tox cover job showed functional hangs and I have suspicion that it might reproducable outside of cover | |
| 14:14:05 | dansmith | oh right right that's what I was thinking of, and as you pointed out, the functional part is all its own command | |
| 14:14:05 | gibi | https://bugs.launchpad.net/nova/+bug/2167088 | |
| 14:14:10 | gibi | yepp | |
| 14:14:13 | gibi | this is the bug | |
| 14:14:17 | gibi | it has some hints where the hang is | |
| 14:14:30 | dansmith | I'm walking up my series a bit to see if it still happens there | |
| 14:14:32 | gibi | or at least one hang I can sometimes reproduce locally | |
| 14:14:47 | dansmith | ah, good to know.. do you have to run cover or just functional alone? | |
| 14:14:56 | gibi | so far I needed to run cover | |
| 14:15:01 | gibi | and severy limit my cpu | |
| 14:15:05 | dansmith | isn't that really bizarre? | |
| 14:15:20 | dansmith | unless it's just the regex filter that is different there or somethin | |
| 14:15:21 | gibi | systemd-run --user --scope -p CPUQuota=10% tox -e cover -- --concurrency 1 nova.tests.functional.libvirt.test_numa_live_migration.NUMALiveMigrationRollbackTests --until-failure | |
| 14:15:32 | gibi | this reproduces to me semi frequently | |
| 14:15:47 | gibi | I haven't run much of this without cover | |
| 14:15:50 | gibi | it is on my todo list | |
| 14:16:08 | dansmith | I'm seeing it on arm64 with 10 asymmetric cores, fwiw | |
| 14:16:21 | dansmith | I'll try master if it repros earlier in my set | |
| 14:16:44 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 14:18:09 | gibi | btw you are touching live migration and my repro points to a live migration test class it cannot be a coincidence | |
| 14:19:14 | dansmith | perhaps | |
| 14:19:51 | dansmith | how did you determine it was that test class? | |
| 14:24:04 | gibi | I looked at multiple failing runs to see what test cases did not run at all and which was the last test case executed successfully by the hanging worker | |
| 14:24:12 | gibi | s/failing/hanging/ | |
| 14:24:29 | gibi | then made a guess | |
| 14:25:05 | gibi | if you reproduce locally you can see the hanging worker's assigned tests in a temp file | |
| 14:25:26 | gibi | if you look at the command line of the hanging stestr process | |
| 14:25:47 | gibi | in the CI we don't have that data saved | |
| 14:26:16 | gibi | so there I can only looked at the subunit partial result to see what was finished and the rest was assumed not started or hanging | |
| 14:27:35 | dansmith | ack | |
| 14:28:02 | dansmith | it didn't hang earlier in the series, so I suspect it's something I've done to the live migration path | |
| 14:28:16 | dansmith | either a total coincidence or maybe exacerbating whatever you're seeing | |
| 14:59:03 | Uggla | Reminder upstream triage in ~30mn. (https://meet.google.com/zjr-rxus-hzj) | |
| 15:01:02 | gibi | Uggla: see my note above for the triage | |
| 15:01:07 | gibi | ahh you sas | |
| 15:01:09 | gibi | saw | |
| 15:01:14 | gibi | I missed the ack | |
| 15:01:30 | Uggla | yep, nw | |
| 15:14:39 | elodilles | Uggla: i'll skip today's meeting as I'm still lagging behind since coming back from summit :/ | |
| 15:16:05 | Uggla | elodilles nw, I do not expect lots of progress due to the release. Anyway thx for letting me know. | |
| 15:16:21 | elodilles | +1 | |
| 15:48:20 | opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug 2162835 https://review.opendev.org/c/openstack/nova/+/999977 | |
| 15:48:20 | opendevreview | ribaudr proposed openstack/nova master: Fix evacuate race conditions with periodic task and task_state https://review.opendev.org/c/openstack/nova/+/999978 | |
| 17:19:08 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 17:33:54 | gmaan | gibi_: thanks for periodic_task fix for shutdown case, I commented in gerrit https://review.opendev.org/c/openstack/nova/+/1005880/comment/2facdbfa_bb30dd75/ | |
| 17:34:36 | gmaan | basically instead of raising LoopingCallDone(), I think we should return value which makes oslo.service to gracefully shutdown peropdic task thread | |
| 17:35:09 | gmaan | gibi_: another things, we should backport this fix to stable/2026.2 right? | |
| 17:36:19 | gmaan | btw, other fix for systemd notify in olso.service lgtm. | |
| 17:40:10 | opendevreview | Merged openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | |
| 21:06:04 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 21:13:36 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 21:46:13 | ashigupt | gmaan: gibi_ Does https://review.opendev.org/c/openstack/nova/+/1001369 looks good to you now ? | |
| #openstack-nova - 2026-09-17 | |||
| 07:03:10 | gibi_ | ashigupt: I will try to get to it but at the moment it is somewhere in the middle of my todo lit | |
| 07:03:13 | gibi_ | lis | |
| 07:03:13 | gibi_ | list | |
| 08:09:16 | bauzas | hey folks, back | |
| 08:16:08 | bauzas | gibi_: I mentioned https://review.opendev.org/c/openstack/nova/+/1005727 but this requires a bit of cleanup on my side per the last comments, fwiw | |
| 08:25:50 | gibi_ | ack | |
| 08:26:04 | gibi_ | I push what I have and then start looking at yours | |
| 09:00:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 09:01:38 | gibi_ | bauzas: that is mine ^^ it reuses an existing RPC call originally used for live migration during cold migration to clean up the leaked MoveClaim | |
| 09:01:54 | bauzas | ack | |
| 09:13:37 | gibi_ | bauzas: left questions in the fix | |
| 09:13:59 | bauzas | ty, still trying to untangle all the potential issues | |
| 09:14:08 | gibi_ | ack | |
| 09:14:22 | bauzas | as a reminder, the best solution is something persisted | |
| 09:14:44 | bauzas | but still trying to find a short-term fix for being backportable shortly | |
| 09:15:26 | gibi | ahh and I realized that I asked the same question as Dan | |
| 09:15:40 | gibi | about how the two parallel request can overlap | |
| 09:15:55 | bauzas | np | |
| 09:26:35 | gibi | OK moved my comment to the proper place | |
| 09:27:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 09:27:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 10:38:44 | opendevreview | Sylvain Bauza proposed openstack/nova master: Guard against deleting a concurrent request's BDM on timeout https://review.opendev.org/c/openstack/nova/+/1005727 | |
| 10:38:44 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reject stale reserve_block_device_name RPC on compute https://review.opendev.org/c/openstack/nova/+/1006011 | |
| 11:36:37 | opendevreview | Merged openstack/placement master: pre-commit: Add codespell https://review.opendev.org/c/openstack/placement/+/931806 | |
| 13:11:11 | gibi | gmaan: does this clear_events_for_instance needs to run for failed resizes as well https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/compute/manager.py#L6418 ? E.g. if driver.migrate_disk_and_power_off raises do we leak a pending resize task? | |
| 13:52:59 | bauzas | gibi: gmaan: I barely paid attention to the gate issues with threading, but I just saw that one https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c5e/openstack/c5e5e75ba3ba4ea1bbc056c4f9299476/testr_results.html | |
| 13:53:08 | bauzas | context : https://review.opendev.org/c/openstack/nova/+/1005727 | |
| 13:55:51 | gibi | https://bugs.launchpad.net/nova/+bug/2165149 | |
| 13:56:06 | gibi | https://review.opendev.org/c/openstack/nova/+/1003586 | |
| 13:56:10 | opendevreview | ribaudr proposed openstack/nova master: Fix swap disk creation skipped on NFS during cold migration https://review.opendev.org/c/openstack/nova/+/1006036 | |
| 13:56:24 | gibi | bauzas: that is also one of the todos I have | |
| 13:56:36 | bauzas | ack ok | |
| 13:56:46 | bauzas | I didn't wanted to blindly recheck | |
| 13:56:52 | gibi | we have open bug and a local fix that does not work on the gate :) | |
| 14:00:06 | bauzas | gibi: great questions by the way on my second patch for preveting BDM creation after RPC timeout | |
| 14:00:31 | bauzas | I wasn't having that level of expertise, you're pushing me into looking at all the oslo.service details :) | |
| 14:00:38 | gibi | yeah my feeling is that we are getting deeper and deeper into a rabbithole | |
| 14:01:24 | gibi | but I haven't had the brain power to offer a real backportable alternative | |
| 14:02:32 | bauzas | fwiw, I care about the first patch in the series, I just offered help for the other patch | |
| 14:03:00 | bauzas | if we don't find a short-term solution for the BDMs being created without calling Cinder, I'm all good | |
| 14:03:18 | bauzas | I just want nova to not delete the wrong BDM (the other case) :-) | |
| 14:03:33 | gibi | OK. we can refocus on the first | |
| 14:03:42 | gibi | I understand your priorities ;) | |
| 14:04:34 | gibi | btw mine is also more complicated than what I pushed. we might want to fixes on that prevent the problem to reapper(that I pushed) and a fix in the periodic to clear the old leaks | |
| 14:04:51 | gibi | *two fixes one that | |
| 14:05:05 | gibi | so I'm poking at the periodic | |