| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-16 | |||
| 14:19:14 | dansmith | perhaps | |
| 14:19:51 | dansmith | how did you determine it was that test class? | |
| 14:24:04 | gibi | I looked at multiple failing runs to see what test cases did not run at all and which was the last test case executed successfully by the hanging worker | |
| 14:24:12 | gibi | s/failing/hanging/ | |
| 14:24:29 | gibi | then made a guess | |
| 14:25:05 | gibi | if you reproduce locally you can see the hanging worker's assigned tests in a temp file | |
| 14:25:26 | gibi | if you look at the command line of the hanging stestr process | |
| 14:25:47 | gibi | in the CI we don't have that data saved | |
| 14:26:16 | gibi | so there I can only looked at the subunit partial result to see what was finished and the rest was assumed not started or hanging | |
| 14:27:35 | dansmith | ack | |
| 14:28:02 | dansmith | it didn't hang earlier in the series, so I suspect it's something I've done to the live migration path | |
| 14:28:16 | dansmith | either a total coincidence or maybe exacerbating whatever you're seeing | |
| 14:59:03 | Uggla | Reminder upstream triage in ~30mn. (https://meet.google.com/zjr-rxus-hzj) | |
| 15:01:02 | gibi | Uggla: see my note above for the triage | |
| 15:01:07 | gibi | ahh you sas | |
| 15:01:09 | gibi | saw | |
| 15:01:14 | gibi | I missed the ack | |
| 15:01:30 | Uggla | yep, nw | |
| 15:14:39 | elodilles | Uggla: i'll skip today's meeting as I'm still lagging behind since coming back from summit :/ | |
| 15:16:05 | Uggla | elodilles nw, I do not expect lots of progress due to the release. Anyway thx for letting me know. | |
| 15:16:21 | elodilles | +1 | |
| 15:48:20 | opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug 2162835 https://review.opendev.org/c/openstack/nova/+/999977 | |
| 15:48:20 | opendevreview | ribaudr proposed openstack/nova master: Fix evacuate race conditions with periodic task and task_state https://review.opendev.org/c/openstack/nova/+/999978 | |
| 17:19:08 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 17:33:54 | gmaan | gibi_: thanks for periodic_task fix for shutdown case, I commented in gerrit https://review.opendev.org/c/openstack/nova/+/1005880/comment/2facdbfa_bb30dd75/ | |
| 17:34:36 | gmaan | basically instead of raising LoopingCallDone(), I think we should return value which makes oslo.service to gracefully shutdown peropdic task thread | |
| 17:35:09 | gmaan | gibi_: another things, we should backport this fix to stable/2026.2 right? | |
| 17:36:19 | gmaan | btw, other fix for systemd notify in olso.service lgtm. | |
| 17:40:10 | opendevreview | Merged openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | |
| 21:06:04 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 21:13:36 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 21:46:13 | ashigupt | gmaan: gibi_ Does https://review.opendev.org/c/openstack/nova/+/1001369 looks good to you now ? | |
| #openstack-nova - 2026-09-17 | |||
| 07:03:10 | gibi_ | ashigupt: I will try to get to it but at the moment it is somewhere in the middle of my todo lit | |
| 07:03:13 | gibi_ | lis | |
| 07:03:13 | gibi_ | list | |
| 08:09:16 | bauzas | hey folks, back | |
| 08:16:08 | bauzas | gibi_: I mentioned https://review.opendev.org/c/openstack/nova/+/1005727 but this requires a bit of cleanup on my side per the last comments, fwiw | |
| 08:25:50 | gibi_ | ack | |
| 08:26:04 | gibi_ | I push what I have and then start looking at yours | |
| 09:00:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 09:01:38 | gibi_ | bauzas: that is mine ^^ it reuses an existing RPC call originally used for live migration during cold migration to clean up the leaked MoveClaim | |
| 09:01:54 | bauzas | ack | |
| 09:13:37 | gibi_ | bauzas: left questions in the fix | |
| 09:13:59 | bauzas | ty, still trying to untangle all the potential issues | |
| 09:14:08 | gibi_ | ack | |
| 09:14:22 | bauzas | as a reminder, the best solution is something persisted | |
| 09:14:44 | bauzas | but still trying to find a short-term fix for being backportable shortly | |
| 09:15:26 | gibi | ahh and I realized that I asked the same question as Dan | |
| 09:15:40 | gibi | about how the two parallel request can overlap | |
| 09:15:55 | bauzas | np | |
| 09:26:35 | gibi | OK moved my comment to the proper place | |
| 09:27:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 09:27:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 10:38:44 | opendevreview | Sylvain Bauza proposed openstack/nova master: Guard against deleting a concurrent request's BDM on timeout https://review.opendev.org/c/openstack/nova/+/1005727 | |
| 10:38:44 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reject stale reserve_block_device_name RPC on compute https://review.opendev.org/c/openstack/nova/+/1006011 | |
| 11:36:37 | opendevreview | Merged openstack/placement master: pre-commit: Add codespell https://review.opendev.org/c/openstack/placement/+/931806 | |
| 13:11:11 | gibi | gmaan: does this clear_events_for_instance needs to run for failed resizes as well https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/compute/manager.py#L6418 ? E.g. if driver.migrate_disk_and_power_off raises do we leak a pending resize task? | |
| 13:52:59 | bauzas | gibi: gmaan: I barely paid attention to the gate issues with threading, but I just saw that one https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c5e/openstack/c5e5e75ba3ba4ea1bbc056c4f9299476/testr_results.html | |
| 13:53:08 | bauzas | context : https://review.opendev.org/c/openstack/nova/+/1005727 | |
| 13:55:51 | gibi | https://bugs.launchpad.net/nova/+bug/2165149 | |
| 13:56:06 | gibi | https://review.opendev.org/c/openstack/nova/+/1003586 | |
| 13:56:10 | opendevreview | ribaudr proposed openstack/nova master: Fix swap disk creation skipped on NFS during cold migration https://review.opendev.org/c/openstack/nova/+/1006036 | |
| 13:56:24 | gibi | bauzas: that is also one of the todos I have | |
| 13:56:36 | bauzas | ack ok | |
| 13:56:46 | bauzas | I didn't wanted to blindly recheck | |
| 13:56:52 | gibi | we have open bug and a local fix that does not work on the gate :) | |
| 14:00:06 | bauzas | gibi: great questions by the way on my second patch for preveting BDM creation after RPC timeout | |
| 14:00:31 | bauzas | I wasn't having that level of expertise, you're pushing me into looking at all the oslo.service details :) | |
| 14:00:38 | gibi | yeah my feeling is that we are getting deeper and deeper into a rabbithole | |
| 14:01:24 | gibi | but I haven't had the brain power to offer a real backportable alternative | |
| 14:02:32 | bauzas | fwiw, I care about the first patch in the series, I just offered help for the other patch | |
| 14:03:00 | bauzas | if we don't find a short-term solution for the BDMs being created without calling Cinder, I'm all good | |
| 14:03:18 | bauzas | I just want nova to not delete the wrong BDM (the other case) :-) | |
| 14:03:33 | gibi | OK. we can refocus on the first | |
| 14:03:42 | gibi | I understand your priorities ;) | |
| 14:04:34 | gibi | btw mine is also more complicated than what I pushed. we might want to fixes on that prevent the problem to reapper(that I pushed) and a fix in the periodic to clear the old leaks | |
| 14:04:51 | gibi | *two fixes one that | |
| 14:05:05 | gibi | so I'm poking at the periodic | |
| 14:05:32 | gibi | while fighiting a race condition in the original reproduce as we revert stuff in an untestable order | |
| 14:06:10 | gibi | aaand I will drop for a wile in 25mins so running out of time too | |
| 14:06:51 | bauzas | yeah I need some brain cycles now to look at your patch | |
| 14:09:16 | gibi | if you look just focus on the RPC implications, the rest might change | |
| 14:09:35 | bauzas | I need to load a bit of context, that's it | |
| 15:48:23 | eduardomorais[m] | Could someone review this PR and give me some tips? I'm new to contributing, thanks in advance! https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 17:18:51 | dansmith | eduardomorais[m]: reviewing other people's submissions is a good way to get familiar with the codebase _and_ get yourself karma you can use for other purposes. i.e. find some (legitimate) thing to -1 :) | |
| 17:21:16 | gmaan | gibi: yes, resize has the sane issue of task leak in source resize_instance error in between. dest call resize_instance on source as 'RPC cast' so dest does not get to know if source failed as source does not call dest. As this is prep_resize time the new VM is not yet created on dest so we are good on dest host but task end is not done. | |
| 17:21:48 | gmaan | revert_resize has the same which not jsut leak task but make VM.host in unrecoverable state. i reported that bug https://bugs.launchpad.net/nova/+bug/2162090 | |
| 17:23:43 | gmaan | I am going to handle these error cases between source and dest as a genric way in part-3. | |
| 17:24:54 | gmaan | gibi: if you want I can make previously listed bug as more generic for all migration tasks instead of just live migration task leak at dest. this one https://review.opendev.org/c/openstack/nova/+/1003386/5/releasenotes/notes/nova-graceful-shutdown-task-tracking-13485c8346512ae4.yaml#23 | |
| 17:27:44 | dansmith | gmaan: do you have an easy way to disable waiting for graceful shutdown in functional workers? the thing I'm working on seems like it's getting stuck a *lot* and I suspect maybe it's because of graceful shutdown | |
| 17:27:54 | dansmith | maybe set the wait conf knob to zero by default as a test? | |
| 17:29:31 | gmaan | dansmith: we have config knob already set to 0 in configfixture so it should not stuck https://github.com/openstack/nova/blob/master/nova/tests/fixtures/conf.py#L65 | |
| 17:29:55 | dansmith | okay but I see lots of messages about it waiting | |
| 17:30:13 | dansmith | like he thread pool nova.tests.functional.test_servers.ServerMovingTestsWithNestedResourceRequests.test_resize_revert.default is shutting down | |
| 17:30:31 | dansmith | is that just it noting that it's shutting down but not waiting? (I haven't looked to verify) | |
| 17:30:51 | dansmith | it always seems to come after a long pause so I figured those were the ones that eventually did shut down after waiting | |
| 17:32:26 | gmaan | oh, are you seeing hang or just logs. because we have noisy logs of thread pool and I am fixing those for tests https://review.opendev.org/c/openstack/nova/+/1004160 | |
| 17:33:17 | gmaan | actual shutdown should not take time as we do not wait for in-progress tasks except in some graceful shutdown tests | |
| 17:34:13 | dansmith | I'm seeing hangs, trying to determine why they're hanging and suspecting graceful shutdown (not my glorious code of course!) | |
| 17:34:15 | dansmith | okay | |