| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-18 | |||
| 03:07:00 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Reduce the unnecessary executors shutdown logs in test jobs https://review.opendev.org/c/openstack/nova/+/1004160 | |
| 07:39:40 | gibi | gmaan: re https://bugs.launchpad.net/nova/+bug/2166009 I think my question is about a different scenario. https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/compute/manager.py#L6418 runs on the source, and if the virt driver raises an exception then we wont hit this cleanup code so we potentially leak a task on the *source* node of the resize | |
| 07:40:59 | gibi | dansmith: yeah, this wait forever on the executor is intentional (and debatable) in the functional test. I want to avoid leaking running threads across test cases as that leads to very hard to debug unexpected behaviors in later test cases. native threads are not killable so I cannot force cleanup between tests | |
| 07:42:04 | gibi | you (we) can try to add some task tracking in the functional test and log and fail instead of wait forever if the executor is not empty to help debugging. | |
| 07:42:41 | gibi | I would love to work on this but swamped with important bugfixes at the moment | |
| 07:43:35 | gibi | you can try to locally flip the wait=true to wait=false in the executor shutdown and try to find a way to print the content of the excutor to help narroving which task is not finished in the test case | |
| 07:47:07 | gibi | OK, maybe a way out, mock the shutdown in the func test to wait a bit but not forever for the executor and if the executor is not empty after the wait then fail the test case cleanly, that should be easy to implement and would avoid the hard to debug hang and clearly show at least which test case is effected | |
| 08:13:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add missing resize.error notification https://review.opendev.org/c/openstack/nova/+/1006163 | |
| 08:30:09 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 08:30:09 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 08:35:12 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reject stale reserve_block_device_name RPC on compute https://review.opendev.org/c/openstack/nova/+/1006011 | |
| 08:59:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 08:59:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 10:24:05 | zigo | (Hibiscus, of course...) | |
| 10:24:47 | zigo | I very much love that I was able to see: | |
| 10:24:47 | zigo | setUpClass (tempest.api.compute.admin.test_live_migration.LiveMigrationWithVTPMTest) ... SKIPPED: LiveMigrationWithVTPMTest skipped as vTPM live migration is not enabled | |
| 10:24:47 | zigo | :) | |
| 10:25:01 | zigo | Super nice feature, thanks guys !!! | |
| 12:07:56 | nicolairuckel | Is anyone currently working on/thinking about this? https://bugs.launchpad.net/nova/+bug/1785123 | |
| 12:09:49 | gibi | nicolairuckel: isn't it fixed in https://review.opendev.org/c/openstack/nova/+/959682 ? | |
| 12:10:03 | gibi | or is that a partial fix? | |
| 12:10:30 | nicolairuckel | That was only a partial fix. That patch doesn't cover cold migration and shelve. | |
| 12:10:56 | nicolairuckel | We decided to deal with those in a separate patch (see https://review.opendev.org/c/openstack/nova/+/959682/comments/ae045d85_9e58a5a5?tab=comments) | |
| 12:11:42 | nicolairuckel | Unfortunately, we can't just use libvirt for that like I did in the patch you linked. | |
| 12:12:06 | gibi | nicolairuckel: I'm not aware of anybody working on the cold migration part | |
| 12:12:14 | gibi | sean-k-mooney: ^^ maybe you have more context | |
| 12:12:49 | sean-k-mooney | am | |
| 12:13:12 | sean-k-mooney | so i tought we had started on it but i think we mainly just fixed the reboot case so far | |
| 12:13:23 | sean-k-mooney | i dont recall if there was a draft for cold migragte | |
| 12:13:38 | sean-k-mooney | but we did say we likely need an api change for rezie eventually | |
| 12:14:00 | sean-k-mooney | so sate if the nvram shold be cleared or not. sorry that was for rebuild | |
| 12:14:17 | sean-k-mooney | for rebuild we need ot add an option to presserve/cleare the nvram | |
| 12:14:31 | sean-k-mooney | to cather for the rebuild form snapshot vs unrelated image case | |
| 12:14:59 | sean-k-mooney | there was https://review.opendev.org/c/openstack/nova/+/621646 | |
| 12:15:27 | sean-k-mooney | but no https://review.opendev.org/c/openstack/nova/+/959682 did not fix cold migrate and shleve | |
| 12:16:37 | sean-k-mooney | gibi: nicolairuckel i remember there was a reason we split it that way | |
| 12:16:42 | sean-k-mooney | but i dont recall what that was | |
| 12:17:16 | gibi | thanks sean-k-mooney | |
| 12:17:17 | nicolairuckel | Because of the API change we probably won't be able to backport the changes needed for cold migration. | |
| 12:17:29 | nicolairuckel | But for the other patch we were able to backport them | |
| 12:17:34 | sean-k-mooney | for cold migration we do not need an api change | |
| 12:17:38 | sean-k-mooney | that only for rebuild | |
| 12:17:40 | nicolairuckel | ah | |
| 12:17:44 | sean-k-mooney | i misspoke before | |
| 12:18:02 | sean-k-mooney | for cold migrate we whosul always preseve the nvram | |
| 12:18:14 | nicolairuckel | The other reason was complexity. My patch turned out to be quite simple since we were able to use libvirt. | |
| 12:18:36 | nicolairuckel | Is this the problem for rebuild? https://bugs.launchpad.net/nova/+bug/2139077 | |
| 12:18:59 | sean-k-mooney | no for rebuild we have 2 usecases | |
| 12:19:21 | sean-k-mooney | some people use rebuild for backup and restore or to upgrade the software in teh vm | |
| 12:19:36 | sean-k-mooney | so for that cohort we want to preserve the nvram | |
| 12:19:53 | sean-k-mooney | but you can also use rebuild to change the image entrily and even disable secure boot | |
| 12:20:08 | sean-k-mooney | so for that second group clearing the nvram is more desireable | |
| 12:20:27 | sean-k-mooney | my prefernce woudl be to preserve by default and have an new api option to ask for it to be cleared | |
| 12:20:46 | nicolairuckel | So I guess we should split that up even further: one patch for the cold migration and one for rebuild? | |
| 12:21:04 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2139077 is a third edgecase but its a less common one | |
| 12:21:21 | sean-k-mooney | yes i would keep it split | |
| 12:21:29 | sean-k-mooney | im not sure what the curret rebuild behivor is | |
| 12:22:04 | sean-k-mooney | shelve is also the final edgbecase we did not adress | |
| 12:22:17 | sean-k-mooney | which si it would be nice to save the nvram somewhere on shelve | |
| 12:22:29 | sean-k-mooney | we just didnt agree where | |
| 12:22:53 | sean-k-mooney | all of the edgecase for nvram also applies ot the tpm data too | |
| 12:23:27 | sean-k-mooney | for windows in partical preserving the nvram and loosing the tpm data on cold migrate will require bitlocker recovery | |
| 12:24:05 | sean-k-mooney | so if i was to work on this i would do cold-migrate/resize next for both nvram and tpm | |
| 13:13:47 | nicolairuckel | In one patch or separate patches? | |
| 13:13:56 | bauzas | gibi: hope your laptop is back, can I start to look at your patches ? | |
| 13:14:34 | nicolairuckel | I'm not sure if we need TPM | |
| 13:15:57 | gibi | bauzas: give me 5 to push what I have (some missing tests) | |
| 13:16:40 | bauzas | ack, I'll stop in ~1.5hour | |
| 13:21:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 13:21:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Clean PCI claim of failed resize in periodic task https://review.opendev.org/c/openstack/nova/+/1006209 | |
| 13:22:09 | gibi | bauzas: so ^^ I have 3 patches | |
| 13:22:35 | gibi | 1. reproducer https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 13:22:52 | gibi | 2. clean old leaks from periodics https://review.opendev.org/c/openstack/nova/+/1006209 | |
| 13:23:03 | gibi | 3. prevent new leaks https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 13:23:06 | bauzas | ok | |
| 13:23:25 | gibi | the 3rd is RPC sensitve :) | |
| 13:23:46 | gibi | and I have to add unit test to the 2nd patch | |
| 13:36:17 | dansmith | gibi: I found a deadlock in my own stuff that was definitely related (but haven't solved yet) so that may have been my only issue, but other test workers are also hanging, so not sure | |
| 13:37:29 | dansmith | I guess it feels like maybe we should not wait there but fail if any tasks are left in some way instead of wait | |
| 13:40:20 | gibi | yeah that is a good idea to implement | |
| 15:55:38 | gmaan | yeah failing is also good, better than hanging. | |
| 16:00:12 | gmaan | gibi: RE:resize_instance: on source it will not hang the task as task on source resize_instance is tracked via RPC common wrapper and it will always end the task as soon RPC requsted is ended either by success or fail | |
| 16:00:28 | gmaan | gibi: but yes, event cleanup will be left in failure case | |
| 16:01:33 | gmaan | gibi: dansmith but on dest side we have case of task leakage same as live migration so what you think of it and we can backport it to stable/2026.2 https://review.opendev.org/c/openstack/nova/+/1006095 | |
| 20:09:45 | gmaan | dansmith: I think I am seeing it in my patch (https://review.opendev.org/c/openstack/nova/+/1004160) also where test are hanging due to wait on executors https://zuul.opendev.org/t/openstack/build/9263ab2ac8c244f3a81f94efd50ad1c7/log/job-output.txt | |
| 20:10:20 | gmaan | let me convert the wait to failure and see | |
| 21:22:35 | dansmith | gmaan: I need to go EOD soon but are you saying we can't make the functional test fail instead of wait because of the task leakage on live migration? | |
| 22:53:55 | gmaan | dansmith: we can, I am working on something to propose soon | |
| 22:54:37 | gmaan | dansmith: live migration task leakage is when we will bump the manager_shutdown_timeout which is 0 currently | |
| #openstack-nova - 2026-09-19 | |||
| 04:06:46 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Fail the test if threadpool executor hang during cleanup https://review.opendev.org/c/openstack/nova/+/1006351 | |
| #openstack-nova - 2026-09-21 | |||
| 09:19:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 2166786 https://review.opendev.org/c/openstack/nova/+/1004658 | |
| 09:19:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Clean PCI claim of failed resize in periodic task https://review.opendev.org/c/openstack/nova/+/1006209 | |
| 09:19:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI allocation leak on failed cold migration https://review.opendev.org/c/openstack/nova/+/1006003 | |
| 09:19:50 | gibi | bauzas: fixed your comment. I would appreciate some oppinion about the 3rd patch touching RPC if that is considered an OK move from you prespective | |
| 09:20:03 | bauzas | gibi: on my radar shortly | |
| 09:20:11 | gibi | thanks | |
| 09:41:35 | opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug #2120927 https://review.opendev.org/c/openstack/nova/+/991294 | |
| 09:41:36 | opendevreview | ribaudr proposed openstack/nova master: Fix shelve-offload/unshelve race wiping instance host https://review.opendev.org/c/openstack/nova/+/991295 | |
| 11:47:13 | gibi | bauzas: left comment on the BDM race https://review.opendev.org/c/openstack/nova/+/1005727/2#message-8b58cc364909d72a12803ec86a2e92a3d9841b71 | |
| 11:47:38 | gibi | I tink we can make it more safe and still backward compatible | |