Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-01
16:02:44 sean-k-mooney but we need to confirm
16:02:54 sean-k-mooney there is no way to od it localy however
16:04:22 gmaan yeah, RPC call will work but we have these kind of communication gap between source and dest, this live migration sand other i found in revert_resize and those can be handled in part-3 where we can try to make operations more self-healing or at least abort in good state
16:04:31 sean-k-mooney worst case we will have to add a new rpc jsut for this case to notify the dest
16:04:37 gmaan yeah
16:05:24 gmaan let me dig into it today if that is similar to revert_resize case and ok to move to part-3 or has to be handled in this part-2
16:05:26 sean-k-mooney if we added a new rpc we might be able to make it a littel mroe generic i.e notify_task_compelte
16:06:07 gmaan I will not add task complete or so but call per operation cleanup as actual operation is hanging in other node and it is not just task tracking
16:06:13 sean-k-mooney as long as we have a solution for the test senario for now movign the larger change to pahse 3 i think is ok
16:07:37 gmaan this is revert_resize case where dest can leave things in half way without notifying/doing revert at source https://bugs.launchpad.net/nova/+bug/2162090
16:08:36 gmaan and looks like this live migration issue is on similar issue. and i will not be surprise if we will have more such cases where source/dest are not talking to each other during failure and cleanup completely at both side
16:17:49 gibi ust to make sure that this is not production issue as that thread is daemon and will go away with process. but agree to fix it in functional tests
16:17:54 gibi I disagree ^^
16:18:21 gibi this will make the gracefull shutdown a lot slower than it should be as it will wait for the timeout value
16:19:26 gibi and it will be a warning log during graceful shutdown that sends support to a goose chase about a in progress migration that is not in progress
16:19:55 gmaan is it but, it will not wait for timeout as graceful shutdown will terminate the process and thread will be stopped right?
16:20:31 gibi if you sent the timeout to 300 then the test fill time out as it waits for the whole 300 seconds and that is more than the test case timeout
16:20:46 gibi so it is waiting for the the timeout as it thinks there is a task to wait for
16:20:53 gibi and when times out it exists
16:21:04 gibi but that means we wait a 180 seconds by default
16:21:08 gmaan yes for functional tests as process is not exit so i agree to fix the live migration task hanging.
16:22:35 gmaan the live migration task left is I agree is the production bug but I am talking about hthttps://bugs.launchpad.net/nova/+bug/2165991 though it is because of migration task hanging
16:22:37 gibi similarly the dest nova-compute will not exit until 180sec as it thinks there is an in progress live migration
16:22:48 gmaan I think I mixed both in my question
16:23:05 gibi I think that two things are the same :)
16:23:10 gibi feel free to rename the bug title
16:23:16 gibi to talk about the leaked task
16:23:24 gibi the hanging functional test is just a symptom
16:23:26 gmaan gibi: my question was if anything we need to fix on shutdown_thread creation and when it fail/timeout
16:23:33 gmaan gibi: agree on that
16:24:27 gmaan I am saying this part is all ok https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/service.py#L433-L435
16:24:45 gmaan and yes we need to fix that task leak
16:25:05 gibi it is a trap for functional test due to the thread can leak, but I can cirumwent that with setting a non 0 timeout to wait for that thread
16:26:52 gibi and when I set that timeout to 300 I saw the task leak in 3 functional tests I reported as the bug
16:27:21 gmaan ack
16:28:46 gibi also now I see that setting 300sec timeout also make a bunch of test sad in the eventlet job
16:29:32 gibi so we might have more leaks of we might have differently hanging shutdown in eventlet job
16:29:49 gibi https://review.opendev.org/c/openstack/nova/+/1003266/1
16:42:06 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
16:42:20 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
16:56:09 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
17:09:03 Uggla dansmith, melwitt, gouthamr comment about the virtiofs migration and FFE: https://review.opendev.org/c/openstack/nova/+/1001916/comments/965cff03_8b75df37
17:09:52 gouthamr \o/ ty Uggla
17:11:44 Uggla gouthamr, dansmith, melwitt I think Melanie is on PTO until tomorrow. I hope Melanie could jump on this review as soon as she will be back.
17:13:31 Uggla gouthamr, dansmith, melwitt also forget to mention that the msg above contains a short summary of the tests I did.
17:14:51 gouthamr ++ i read them, ty for testing those scenarios. I will try and add a multi-node job with cephfs to the manila gate, and one with LVM/NFS to the nova gate (and spread the test coverage between projects)
17:16:40 Uggla gouthamr, it will be great. Just to note I also tried several way to break it, and it looks solid so far.
17:41:29 gouthamr Uggla: ++ :) ty!
17:56:34 gmaan gibi: for eventlet functional job, you mean tox cover jobs or nova-tox-functional-py313 or both? bth failing though but I am seeing only one failure in nova-tox-functional-py313
19:33:33 melwitt Uggla: thanks for adding the comment, I will be reading over everything
20:16:19 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900
20:17:16 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349
21:14:56 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
21:16:39 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
21:21:39 opendevreview Ghanshyam Maan proposed openstack/nova master: Fix task leak for live migration rollback case https://review.opendev.org/c/openstack/nova/+/1003386
#openstack-nova - 2026-09-02
02:02:31 gmaan gibi: sean-k-mooney : this is fix for the live migration task leaking https://review.opendev.org/c/openstack/nova/+/1003386
05:19:56 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
06:12:01 opendevreview Adam McArthur proposed openstack/nova master: Use safe allocation deletion on instance delete https://review.opendev.org/c/openstack/nova/+/1003208
10:02:07 gibi can I get an core eye on https://review.opendev.org/c/openstack/nova/+/1003245 it is a functional test stabilization
10:12:32 sean-k-mooney so the intet is to set it to a largere value later but we are settign it to 10 for now so that if we do leak it does not cause the run to fail
10:15:58 sean-k-mooney i see in https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/functional/regressions/test_bug_1888395.py your bumping it to 300 and hten claping it to 10 on the tests that have know issues
10:16:20 sean-k-mooney any that are leaking a thread will timeout and you can clamp them until its fixed
10:17:54 sean-k-mooney gibi: we can technially start deleting evnetlet in like a weak to eliminate those leaks :)
10:19:47 sean-k-mooney gibi: by the way did you notice that the threading functional test now run in half the time of the eventlet ones
10:20:18 sean-k-mooney oh nope
10:20:30 sean-k-mooney thats the non funtionla treadign tests
10:21:49 gibi sean-k-mooney: we cannot delete them until post 2027.1
10:22:04 sean-k-mooney we technially can
10:22:26 sean-k-mooney but i know we are givign a very long tail in nova
10:22:29 gibi sc-tmp-lighbits-rhoso-certification
10:22:33 gibi https://governance.openstack.org/tc/goals/selected/remove-eventlet.html#completion-criteria
10:22:56 gibi u(2027.1) Deprecate the Eventlet support. To have a SLURP target release supporting both the old Eventlet and any new concurrency mode, this release keeps the Eventlet support.
10:23:08 sean-k-mooney those are the letes tg we can do those
10:23:22 sean-k-mooney not the first time for each milestone
10:23:52 sean-k-mooney several service just remove eventlet with no opertunity to have a singel slurp release that ran both
10:24:18 gibi but I think what is written in the goal is there because we wanted to do it that way :)
10:24:52 sean-k-mooney we did im just saying if it continue to be a maintiance isseu we can change that planb
10:24:57 sean-k-mooney on the ohter hand
10:25:08 sean-k-mooney if the eventlet test find leaks that we dont see in threaded
10:25:19 sean-k-mooney we probably shoudl fix those anyway
10:25:50 gibi yeah I haven't looked into the eventlet hangs in that patch
10:25:53 gibi but I will
10:47:27 opendevreview Balazs Gibizer proposed openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452
10:47:48 gibi sean-k-mooney: bauzas: ^^ another functional stabilization patch
13:21:29 opendevreview Balazs Gibizer proposed openstack/nova master: [functional]Do not patch req-id generation https://review.opendev.org/c/openstack/nova/+/1003024
13:21:45 gibi cores this is another stabilization ^^
14:16:56 stephenfin sean-k-mooney: did you already bring up https://review.opendev.org/c/openstack/releases/+/1003257?
14:17:21 stephenfin https://review.opendev.org/c/openstack/releases/+/1003311 is blocked by that
14:32:41 sean-k-mooney sorry no Uggla bauzas ^ we need to release os-vif again
14:33:13 sean-k-mooney elodilles: ^ fyi
14:33:45 Uggla sean-k-mooney ok I have seen the patches
14:41:49 elodilles sean-k-mooney: nova team needs to agree on FFE and an RFE needs to be requested from requirements team, otherwise LGTM
14:48:38 sean-k-mooney the config bug is a reguress intoduced in this cycle os i think it would qualify for a FFE adn enve after RC1 but idealy we do it before we do it
14:52:25 gibi I'm OK from nova perspective to accept this os_vif fix as FFE
14:52:33 gibi https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/fixtures/conf.py#73
14:52:36 gibi this is being the fix
15:37:49 opendevreview Balazs Gibizer proposed openstack/nova master: [DNM]reproduce "database is locked" error https://review.opendev.org/c/openstack/nova/+/1003554
15:38:51 gibi sean-k-mooney: ^^ finally after 2days and many deep rabbit holes this reproduces the DB locking issue pretty regularly to me locally
15:39:56 gibi there is something called checkpointing the WAL
15:40:08 gibi that happens after the last DB connection is closed

Earlier   Later