Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-17
09:13:37 gibi_ bauzas: left questions in the fix
09:13:59 bauzas ty, still trying to untangle all the potential issues
09:14:08 gibi_ ack
09:14:22 bauzas as a reminder, the best solution is something persisted
09:14:44 bauzas but still trying to find a short-term fix for being backportable shortly
09:15:26 gibi ahh and I realized that I asked the same question as Dan
09:15:40 gibi about how the two parallel request can overlap
09:15:55 bauzas np
09:26:35 gibi OK moved my comment to the proper place
09:27:51 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658
09:27:51 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003
10:38:44 opendevreview Sylvain Bauza proposed openstack/nova master: Guard against deleting a concurrent request's BDM on timeout https://review.opendev.org/c/openstack/nova/+/1005727
10:38:44 opendevreview Sylvain Bauza proposed openstack/nova master: Reject stale reserve_block_device_name RPC on compute https://review.opendev.org/c/openstack/nova/+/1006011
11:36:37 opendevreview Merged openstack/placement master: pre-commit: Add codespell https://review.opendev.org/c/openstack/placement/+/931806
13:11:11 gibi gmaan: does this clear_events_for_instance needs to run for failed resizes as well https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/compute/manager.py#L6418 ? E.g. if driver.migrate_disk_and_power_off raises do we leak a pending resize task?
13:52:59 bauzas gibi: gmaan: I barely paid attention to the gate issues with threading, but I just saw that one https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c5e/openstack/c5e5e75ba3ba4ea1bbc056c4f9299476/testr_results.html
13:53:08 bauzas context : https://review.opendev.org/c/openstack/nova/+/1005727
13:55:51 gibi https://bugs.launchpad.net/nova/+bug/2165149
13:56:06 gibi https://review.opendev.org/c/openstack/nova/+/1003586
13:56:10 opendevreview ribaudr proposed openstack/nova master: Fix swap disk creation skipped on NFS during cold migration https://review.opendev.org/c/openstack/nova/+/1006036
13:56:24 gibi bauzas: that is also one of the todos I have
13:56:36 bauzas ack ok
13:56:46 bauzas I didn't wanted to blindly recheck
13:56:52 gibi we have open bug and a local fix that does not work on the gate :)
14:00:06 bauzas gibi: great questions by the way on my second patch for preveting BDM creation after RPC timeout
14:00:31 bauzas I wasn't having that level of expertise, you're pushing me into looking at all the oslo.service details :)
14:00:38 gibi yeah my feeling is that we are getting deeper and deeper into a rabbithole
14:01:24 gibi but I haven't had the brain power to offer a real backportable alternative
14:02:32 bauzas fwiw, I care about the first patch in the series, I just offered help for the other patch
14:03:00 bauzas if we don't find a short-term solution for the BDMs being created without calling Cinder, I'm all good
14:03:18 bauzas I just want nova to not delete the wrong BDM (the other case) :-)
14:03:33 gibi OK. we can refocus on the first
14:03:42 gibi I understand your priorities ;)
14:04:34 gibi btw mine is also more complicated than what I pushed. we might want to fixes on that prevent the problem to reapper(that I pushed) and a fix in the periodic to clear the old leaks
14:04:51 gibi *two fixes one that
14:05:05 gibi so I'm poking at the periodic
14:05:32 gibi while fighiting a race condition in the original reproduce as we revert stuff in an untestable order
14:06:10 gibi aaand I will drop for a wile in 25mins so running out of time too
14:06:51 bauzas yeah I need some brain cycles now to look at your patch
14:09:16 gibi if you look just focus on the RPC implications, the rest might change
14:09:35 bauzas I need to load a bit of context, that's it
15:48:23 eduardomorais[m] Could someone review this PR and give me some tips? I'm new to contributing, thanks in advance! https://review.opendev.org/c/openstack/nova/+/1005881
17:18:51 dansmith eduardomorais[m]: reviewing other people's submissions is a good way to get familiar with the codebase _and_ get yourself karma you can use for other purposes. i.e. find some (legitimate) thing to -1 :)
17:21:16 gmaan gibi: yes, resize has the sane issue of task leak in source resize_instance error in between. dest call resize_instance on source as 'RPC cast' so dest does not get to know if source failed as source does not call dest. As this is prep_resize time the new VM is not yet created on dest so we are good on dest host but task end is not done.
17:21:48 gmaan revert_resize has the same which not jsut leak task but make VM.host in unrecoverable state. i reported that bug https://bugs.launchpad.net/nova/+bug/2162090
17:23:43 gmaan I am going to handle these error cases between source and dest as a genric way in part-3.
17:24:54 gmaan gibi: if you want I can make previously listed bug as more generic for all migration tasks instead of just live migration task leak at dest. this one https://review.opendev.org/c/openstack/nova/+/1003386/5/releasenotes/notes/nova-graceful-shutdown-task-tracking-13485c8346512ae4.yaml#23
17:27:44 dansmith gmaan: do you have an easy way to disable waiting for graceful shutdown in functional workers? the thing I'm working on seems like it's getting stuck a *lot* and I suspect maybe it's because of graceful shutdown
17:27:54 dansmith maybe set the wait conf knob to zero by default as a test?
17:29:31 gmaan dansmith: we have config knob already set to 0 in configfixture so it should not stuck https://github.com/openstack/nova/blob/master/nova/tests/fixtures/conf.py#L65
17:29:55 dansmith okay but I see lots of messages about it waiting
17:30:13 dansmith like he thread pool nova.tests.functional.test_servers.ServerMovingTestsWithNestedResourceRequests.test_resize_revert.default is shutting down
17:30:31 dansmith is that just it noting that it's shutting down but not waiting? (I haven't looked to verify)
17:30:51 dansmith it always seems to come after a long pause so I figured those were the ones that eventually did shut down after waiting
17:32:26 gmaan oh, are you seeing hang or just logs. because we have noisy logs of thread pool and I am fixing those for tests https://review.opendev.org/c/openstack/nova/+/1004160
17:33:17 gmaan actual shutdown should not take time as we do not wait for in-progress tasks except in some graceful shutdown tests
17:34:13 dansmith I'm seeing hangs, trying to determine why they're hanging and suspecting graceful shutdown (not my glorious code of course!)
17:34:15 dansmith okay
17:36:32 gmaan humm, can you check what is manager_shutdown_time value there (by printing ro so). note: graceful_shutdown_timeout is kept default which is 180 sec i hope nothing stuck after manager return immidiatly
17:37:00 gmaan dansmith: one thing can take time is thead pool logs taking time, if anything still running in executors the it will wait
17:38:49 dansmith yeah I'm waiting well past 180s to determine they're stuck, but I was thinking maybe there was some deadlock we're hitting that made the timeout not relevant.. I haven't figured out which test is the problem yet because all the candidates seem to run fine when run alone
17:38:56 dansmith hence me suspecting some interaction
17:40:22 gmaan i see. i remember oslo.service does not timeout the shutdown based on graceful_shutdown_timeout as there is no releah SIGTERM handler called on test process between tests
17:41:06 gmaan so we basically retun from manager quickly with 0 sec wait but if something still running in exevcutor and hang then it will hang the shutdown also
17:42:27 gmaan we does not stop without wait as we want to cathc the bad tests (not your glorious code of course :)) who leak the things - https://github.com/openstack/nova/blob/master/nova/tests/fixtures/nova.py#L1353
17:44:22 gmaan we fixed a few tests which were hanging and not always but when they face some race so there might be some more but not seen yet in CI
17:53:16 dansmith reset_all_executors() will kill and continue not hang?
17:53:48 gmaan no, it stop executor with wait=True so anything in executro queue will be finished
17:54:03 dansmith but with timeout or no?
17:54:15 dansmith (heh I see the comment right above, sorry)
17:54:49 gmaan no timeout, because oslo.service SIGTERM handler does not timeout
17:55:57 gmaan I try to kill them during cleanup and not wait but then liked the gibi idea to wait and let test hang to see why test leaving the things in-progress
17:57:36 gmaan which exact tests hang or is it many tests take times
18:28:18 opendevreview Ghanshyam Maan proposed openstack/nova master: Update graceful shutdown known issue to list resize case also https://review.opendev.org/c/openstack/nova/+/1006095
18:37:40 gmaan gibi: I updated the existing bug description to cover the resize case - https://bugs.launchpad.net/nova/+bug/2166009
18:37:56 gmaan gibi: dansmith ^^ and updating the 'known issue' in releae notes
22:01:47 opendevreview Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881
22:11:35 eduardomorais[m] dansmithSorry for the late reply. I'll take a look at some other PRs to get a better understanding of how things work. Thanks for the help and the tips!
22:13:18 JayF eduardomorais[m]: it's worth noting the PTG (Project Teams Gathering) planning session is happening soon (ptg.opendev.org) -- and that would be a good place in many projects (I assume this is true for Nova as well) to find something to volunteer to work on
22:13:45 JayF eduardomorais[m]: IDK if you are just looking to drive by contribute or what, but if you want a longer engagement, attending PTG sessions of those projects is a GREAT way to get attention
22:18:21 eduardomorais[m] JayFI'd like to start contributing a bit more to OpenStack services. I currently work as an OpenStack operator, and I'd love to learn more about the codebase and help improve it. I'll definitely keep an eye out for the PTG, thanks for the tip!
#openstack-nova - 2026-09-18
03:07:00 opendevreview Ghanshyam Maan proposed openstack/nova master: Reduce the unnecessary executors shutdown logs in test jobs https://review.opendev.org/c/openstack/nova/+/1004160
07:39:40 gibi gmaan: re https://bugs.launchpad.net/nova/+bug/2166009 I think my question is about a different scenario. https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/compute/manager.py#L6418 runs on the source, and if the virt driver raises an exception then we wont hit this cleanup code so we potentially leak a task on the *source* node of the resize
07:40:59 gibi dansmith: yeah, this wait forever on the executor is intentional (and debatable) in the functional test. I want to avoid leaking running threads across test cases as that leads to very hard to debug unexpected behaviors in later test cases. native threads are not killable so I cannot force cleanup between tests
07:42:04 gibi you (we) can try to add some task tracking in the functional test and log and fail instead of wait forever if the executor is not empty to help debugging.
07:42:41 gibi I would love to work on this but swamped with important bugfixes at the moment
07:43:35 gibi you can try to locally flip the wait=true to wait=false in the executor shutdown and try to find a way to print the content of the excutor to help narroving which task is not finished in the test case
07:47:07 gibi OK, maybe a way out, mock the shutdown in the func test to wait a bit but not forever for the executor and if the executor is not empty after the wait then fail the test case cleanly, that should be easy to implement and would avoid the hard to debug hang and clearly show at least which test case is effected
08:13:19 opendevreview Balazs Gibizer proposed openstack/nova master: Add missing resize.error notification https://review.opendev.org/c/openstack/nova/+/1006163
08:30:09 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658
08:30:09 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003
08:35:12 opendevreview Sylvain Bauza proposed openstack/nova master: Reject stale reserve_block_device_name RPC on compute https://review.opendev.org/c/openstack/nova/+/1006011
08:59:59 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658
08:59:59 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003
10:24:05 zigo (Hibiscus, of course...)
10:24:47 zigo I very much love that I was able to see:
10:24:47 zigo setUpClass (tempest.api.compute.admin.test_live_migration.LiveMigrationWithVTPMTest) ... SKIPPED: LiveMigrationWithVTPMTest skipped as vTPM live migration is not enabled
10:24:47 zigo :)
10:25:01 zigo Super nice feature, thanks guys !!!
12:07:56 nicolairuckel Is anyone currently working on/thinking about this? https://bugs.launchpad.net/nova/+bug/1785123

Earlier   Later