| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-17 | |||
| 17:23:43 | gmaan | I am going to handle these error cases between source and dest as a genric way in part-3. | |
| 17:24:54 | gmaan | gibi: if you want I can make previously listed bug as more generic for all migration tasks instead of just live migration task leak at dest. this one https://review.opendev.org/c/openstack/nova/+/1003386/5/releasenotes/notes/nova-graceful-shutdown-task-tracking-13485c8346512ae4.yaml#23 | |
| 17:27:44 | dansmith | gmaan: do you have an easy way to disable waiting for graceful shutdown in functional workers? the thing I'm working on seems like it's getting stuck a *lot* and I suspect maybe it's because of graceful shutdown | |
| 17:27:54 | dansmith | maybe set the wait conf knob to zero by default as a test? | |
| 17:29:31 | gmaan | dansmith: we have config knob already set to 0 in configfixture so it should not stuck https://github.com/openstack/nova/blob/master/nova/tests/fixtures/conf.py#L65 | |
| 17:29:55 | dansmith | okay but I see lots of messages about it waiting | |
| 17:30:13 | dansmith | like he thread pool nova.tests.functional.test_servers.ServerMovingTestsWithNestedResourceRequests.test_resize_revert.default is shutting down | |
| 17:30:31 | dansmith | is that just it noting that it's shutting down but not waiting? (I haven't looked to verify) | |
| 17:30:51 | dansmith | it always seems to come after a long pause so I figured those were the ones that eventually did shut down after waiting | |
| 17:32:26 | gmaan | oh, are you seeing hang or just logs. because we have noisy logs of thread pool and I am fixing those for tests https://review.opendev.org/c/openstack/nova/+/1004160 | |
| 17:33:17 | gmaan | actual shutdown should not take time as we do not wait for in-progress tasks except in some graceful shutdown tests | |
| 17:34:13 | dansmith | I'm seeing hangs, trying to determine why they're hanging and suspecting graceful shutdown (not my glorious code of course!) | |
| 17:34:15 | dansmith | okay | |
| 17:36:32 | gmaan | humm, can you check what is manager_shutdown_time value there (by printing ro so). note: graceful_shutdown_timeout is kept default which is 180 sec i hope nothing stuck after manager return immidiatly | |
| 17:37:00 | gmaan | dansmith: one thing can take time is thead pool logs taking time, if anything still running in executors the it will wait | |
| 17:38:49 | dansmith | yeah I'm waiting well past 180s to determine they're stuck, but I was thinking maybe there was some deadlock we're hitting that made the timeout not relevant.. I haven't figured out which test is the problem yet because all the candidates seem to run fine when run alone | |
| 17:38:56 | dansmith | hence me suspecting some interaction | |
| 17:40:22 | gmaan | i see. i remember oslo.service does not timeout the shutdown based on graceful_shutdown_timeout as there is no releah SIGTERM handler called on test process between tests | |
| 17:41:06 | gmaan | so we basically retun from manager quickly with 0 sec wait but if something still running in exevcutor and hang then it will hang the shutdown also | |
| 17:42:27 | gmaan | we does not stop without wait as we want to cathc the bad tests (not your glorious code of course :)) who leak the things - https://github.com/openstack/nova/blob/master/nova/tests/fixtures/nova.py#L1353 | |
| 17:44:22 | gmaan | we fixed a few tests which were hanging and not always but when they face some race so there might be some more but not seen yet in CI | |
| 17:53:16 | dansmith | reset_all_executors() will kill and continue not hang? | |
| 17:53:48 | gmaan | no, it stop executor with wait=True so anything in executro queue will be finished | |
| 17:54:03 | dansmith | but with timeout or no? | |
| 17:54:15 | dansmith | (heh I see the comment right above, sorry) | |
| 17:54:49 | gmaan | no timeout, because oslo.service SIGTERM handler does not timeout | |
| 17:55:57 | gmaan | I try to kill them during cleanup and not wait but then liked the gibi idea to wait and let test hang to see why test leaving the things in-progress | |
| 17:57:36 | gmaan | which exact tests hang or is it many tests take times | |
| 18:28:18 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Update graceful shutdown known issue to list resize case also https://review.opendev.org/c/openstack/nova/+/1006095 | |
| 18:37:40 | gmaan | gibi: I updated the existing bug description to cover the resize case - https://bugs.launchpad.net/nova/+bug/2166009 | |
| 18:37:56 | gmaan | gibi: dansmith ^^ and updating the 'known issue' in releae notes | |
| 22:01:47 | opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | |
| 22:11:35 | eduardomorais[m] | dansmithSorry for the late reply. I'll take a look at some other PRs to get a better understanding of how things work. Thanks for the help and the tips! | |
| 22:13:18 | JayF | eduardomorais[m]: it's worth noting the PTG (Project Teams Gathering) planning session is happening soon (ptg.opendev.org) -- and that would be a good place in many projects (I assume this is true for Nova as well) to find something to volunteer to work on | |
| 22:13:45 | JayF | eduardomorais[m]: IDK if you are just looking to drive by contribute or what, but if you want a longer engagement, attending PTG sessions of those projects is a GREAT way to get attention | |
| 22:18:21 | eduardomorais[m] | JayFI'd like to start contributing a bit more to OpenStack services. I currently work as an OpenStack operator, and I'd love to learn more about the codebase and help improve it. I'll definitely keep an eye out for the PTG, thanks for the tip! | |
| #openstack-nova - 2026-09-18 | |||
| 03:07:00 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Reduce the unnecessary executors shutdown logs in test jobs https://review.opendev.org/c/openstack/nova/+/1004160 | |
| 07:39:40 | gibi | gmaan: re https://bugs.launchpad.net/nova/+bug/2166009 I think my question is about a different scenario. https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/compute/manager.py#L6418 runs on the source, and if the virt driver raises an exception then we wont hit this cleanup code so we potentially leak a task on the *source* node of the resize | |
| 07:40:59 | gibi | dansmith: yeah, this wait forever on the executor is intentional (and debatable) in the functional test. I want to avoid leaking running threads across test cases as that leads to very hard to debug unexpected behaviors in later test cases. native threads are not killable so I cannot force cleanup between tests | |
| 07:42:04 | gibi | you (we) can try to add some task tracking in the functional test and log and fail instead of wait forever if the executor is not empty to help debugging. | |
| 07:42:41 | gibi | I would love to work on this but swamped with important bugfixes at the moment | |
| 07:43:35 | gibi | you can try to locally flip the wait=true to wait=false in the executor shutdown and try to find a way to print the content of the excutor to help narroving which task is not finished in the test case | |
| 07:47:07 | gibi | OK, maybe a way out, mock the shutdown in the func test to wait a bit but not forever for the executor and if the executor is not empty after the wait then fail the test case cleanly, that should be easy to implement and would avoid the hard to debug hang and clearly show at least which test case is effected | |
| 08:13:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add missing resize.error notification https://review.opendev.org/c/openstack/nova/+/1006163 | |
| 08:30:09 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 08:30:09 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 08:35:12 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reject stale reserve_block_device_name RPC on compute https://review.opendev.org/c/openstack/nova/+/1006011 | |
| 08:59:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 08:59:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 10:24:05 | zigo | (Hibiscus, of course...) | |
| 10:24:47 | zigo | I very much love that I was able to see: | |
| 10:24:47 | zigo | setUpClass (tempest.api.compute.admin.test_live_migration.LiveMigrationWithVTPMTest) ... SKIPPED: LiveMigrationWithVTPMTest skipped as vTPM live migration is not enabled | |
| 10:24:47 | zigo | :) | |
| 10:25:01 | zigo | Super nice feature, thanks guys !!! | |
| 12:07:56 | nicolairuckel | Is anyone currently working on/thinking about this? https://bugs.launchpad.net/nova/+bug/1785123 | |
| 12:09:49 | gibi | nicolairuckel: isn't it fixed in https://review.opendev.org/c/openstack/nova/+/959682 ? | |
| 12:10:03 | gibi | or is that a partial fix? | |
| 12:10:30 | nicolairuckel | That was only a partial fix. That patch doesn't cover cold migration and shelve. | |
| 12:10:56 | nicolairuckel | We decided to deal with those in a separate patch (see https://review.opendev.org/c/openstack/nova/+/959682/comments/ae045d85_9e58a5a5?tab=comments) | |
| 12:11:42 | nicolairuckel | Unfortunately, we can't just use libvirt for that like I did in the patch you linked. | |
| 12:12:06 | gibi | nicolairuckel: I'm not aware of anybody working on the cold migration part | |
| 12:12:14 | gibi | sean-k-mooney: ^^ maybe you have more context | |
| 12:12:49 | sean-k-mooney | am | |
| 12:13:12 | sean-k-mooney | so i tought we had started on it but i think we mainly just fixed the reboot case so far | |
| 12:13:23 | sean-k-mooney | i dont recall if there was a draft for cold migragte | |
| 12:13:38 | sean-k-mooney | but we did say we likely need an api change for rezie eventually | |
| 12:14:00 | sean-k-mooney | so sate if the nvram shold be cleared or not. sorry that was for rebuild | |
| 12:14:17 | sean-k-mooney | for rebuild we need ot add an option to presserve/cleare the nvram | |
| 12:14:31 | sean-k-mooney | to cather for the rebuild form snapshot vs unrelated image case | |
| 12:14:59 | sean-k-mooney | there was https://review.opendev.org/c/openstack/nova/+/621646 | |
| 12:15:27 | sean-k-mooney | but no https://review.opendev.org/c/openstack/nova/+/959682 did not fix cold migrate and shleve | |
| 12:16:37 | sean-k-mooney | gibi: nicolairuckel i remember there was a reason we split it that way | |
| 12:16:42 | sean-k-mooney | but i dont recall what that was | |
| 12:17:16 | gibi | thanks sean-k-mooney | |
| 12:17:17 | nicolairuckel | Because of the API change we probably won't be able to backport the changes needed for cold migration. | |
| 12:17:29 | nicolairuckel | But for the other patch we were able to backport them | |
| 12:17:34 | sean-k-mooney | for cold migration we do not need an api change | |
| 12:17:38 | sean-k-mooney | that only for rebuild | |
| 12:17:40 | nicolairuckel | ah | |
| 12:17:44 | sean-k-mooney | i misspoke before | |
| 12:18:02 | sean-k-mooney | for cold migrate we whosul always preseve the nvram | |
| 12:18:14 | nicolairuckel | The other reason was complexity. My patch turned out to be quite simple since we were able to use libvirt. | |
| 12:18:36 | nicolairuckel | Is this the problem for rebuild? https://bugs.launchpad.net/nova/+bug/2139077 | |
| 12:18:59 | sean-k-mooney | no for rebuild we have 2 usecases | |
| 12:19:21 | sean-k-mooney | some people use rebuild for backup and restore or to upgrade the software in teh vm | |
| 12:19:36 | sean-k-mooney | so for that cohort we want to preserve the nvram | |
| 12:19:53 | sean-k-mooney | but you can also use rebuild to change the image entrily and even disable secure boot | |
| 12:20:08 | sean-k-mooney | so for that second group clearing the nvram is more desireable | |
| 12:20:27 | sean-k-mooney | my prefernce woudl be to preserve by default and have an new api option to ask for it to be cleared | |
| 12:20:46 | nicolairuckel | So I guess we should split that up even further: one patch for the cold migration and one for rebuild? | |
| 12:21:04 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2139077 is a third edgecase but its a less common one | |
| 12:21:21 | sean-k-mooney | yes i would keep it split | |
| 12:21:29 | sean-k-mooney | im not sure what the curret rebuild behivor is | |
| 12:22:04 | sean-k-mooney | shelve is also the final edgbecase we did not adress | |
| 12:22:17 | sean-k-mooney | which si it would be nice to save the nvram somewhere on shelve | |
| 12:22:29 | sean-k-mooney | we just didnt agree where | |
| 12:22:53 | sean-k-mooney | all of the edgecase for nvram also applies ot the tpm data too | |
| 12:23:27 | sean-k-mooney | for windows in partical preserving the nvram and loosing the tpm data on cold migrate will require bitlocker recovery | |
| 12:24:05 | sean-k-mooney | so if i was to work on this i would do cold-migrate/resize next for both nvram and tpm | |
| 13:13:47 | nicolairuckel | In one patch or separate patches? | |