Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-01
11:18:27 gibi we need at least some help for devs to know what hangs when we log that
11:19:13 gibi as it was a pretty long goose chase for me to find out that pre_live_migration records a task named live_migration_at_dest but that task only finished when the whole live migration terminated.
11:19:37 gibi which make sense after I discovered all this but was pretty strange when I did not know about that logic
11:23:17 gibi btw I think we have a path that rolls back the live migration but never finishes the live_migration_at_dest task
11:23:39 gibi filing a bug and then we can discuss if it is a realistic scenario
11:23:52 gibi (it is hit by a functional test so it is at least triggerable)
11:34:59 gibi (I cannot open a bug as launchpad is dead)
11:35:18 gibi so https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10553-L10562 here if the code takes the else branch as do_cleanup is false then the task will never finish
12:00:23 gibi https://bugs.launchpad.net/nova/+bug/2166009
12:00:57 sean-k-mooney _rollback_live_migration can be called in _cleanup_pre_live_migration and 2 places in _do_live_migration before and after the migrtion is called
12:01:14 sean-k-mooney i dont really recall off the top of my head what the critia for do_cleanup is
12:01:32 sean-k-mooney looks like its determeidn by https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L9968 _live_migration_cleanup_flags
12:01:57 sean-k-mooney i guess its tryign to figure out if its on shared storage?
12:02:10 sean-k-mooney but we defintly dont want to alwasy do that
12:02:52 sean-k-mooney do you think its only a testing issue or a production one
12:04:25 gibi I feel like this is a real code path we can take
12:04:47 gibi and it result in a task kept pending forever in the graceful shutdown task tracker
12:04:48 sean-k-mooney yes if we are on shared storage i think we will take it (nfs)
12:06:01 sean-k-mooney or perhasp if we fail in pre-live migrtion
12:06:08 sean-k-mooney as there is nothign to clean up on the dest
12:07:26 gibi the complication here as far as I see is that this task is in the dest compute's tracker and the source computes rolls the migration back and never calls the dest compute so the dest compute cannot finish the task
12:07:30 sean-k-mooney it looks like do_cleanup, destroy_disks are only ever used inside that if
12:08:11 sean-k-mooney well should it be in the dest tracker at that point
12:08:52 sean-k-mooney i havent fully follosed you but if it roling back before the call to pre live migrate on dest it wont be there
12:09:45 gibi if you want to trace it there are at least two function test cases that hitting the codepath (noted in the bugreport)
12:09:51 sean-k-mooney so the edge case would have to haappend form pre live migratoin at soruce or the call to migrateToURI3 at the libivrt level
12:11:25 sean-k-mooney if we are seing live_migration_at_dest i think that impleise we are rooling back at the libvirt migrate call
12:11:29 gibi test nova.tests.functional.regressions.test_bug_1888395.TestLiveMigrationRollbackWithoutMultiplePortBindings this is the cleaner repro and it trigger dom.fail_job()
12:12:24 sean-k-mooney as in this is happen right at the point of no return since we do not supprot rollback if that call complete without error
12:13:36 sean-k-mooney thats still a little impresices we do not supprot calling rollabck if that libvirt job that was started by that call complete without failing
12:16:18 gibi this also triggers the path and it uses abort not rollback nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSampleWithMultipleCompute._test_live_migration_abort
12:17:55 sean-k-mooney gibi: so we are recordign the task start here https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L9426-L9432
12:18:09 sean-k-mooney but never callign https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L9441-L9444
12:18:29 gibi yeah
12:18:43 gibi I think rollback at dest and post at dest call that helper that finishes the task
12:18:55 gibi but in this scenario there is no rpc to the dest that can call that helper
12:19:19 sean-k-mooney if we just passed destory disks false when we woudl have had do_cleanup=false
12:19:21 gibi the source handles the rollback / abort alone as far as I see
12:19:21 sean-k-mooney it might be safe
12:20:00 sean-k-mooney i.e. if we just alwasy call cleanup at des when not pre_live_migation
12:20:26 sean-k-mooney actully no its recoreded in pre_live_migration
12:20:47 sean-k-mooney so we proably woudl want to alwasy call it if we can make sure its non distuctive for shared sotrage
12:21:22 sean-k-mooney after all this is being called or not called form the souce and the task is beint gract on the dest right
12:22:16 sean-k-mooney so we cant just do _end_pending_dest_live_migration_task since that woudl remvoe it form our local dictionatry
12:22:38 gibi yeah the livemigartion failure detected on the source but this task is pending at the dest
12:23:00 gibi if the source can alway send and rpc to the dest then that rpc can remove the pending task on the dest
12:24:55 sean-k-mooney i think instead of tghe if you could and do_cleanup with detstory_disks
12:25:41 sean-k-mooney the remainign question is are any of the actions in https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10658-L10731 soemthing that are unsafe to attpemt alwasys
12:26:05 sean-k-mooney i think the network cleanup is fine
12:26:24 sean-k-mooney we defintly dont want to leak portbindings and do want to make suere we have cleaned up any ports
12:27:02 sean-k-mooney freeing pci claims is also somthign we woudl prefr to do rather then waith for the resouce tracker
12:27:39 sean-k-mooney its possibel we failed before they were claimed btu i tough it and we coudl handel that anyway
12:27:56 sean-k-mooney so i think the manger level tasks all look ok
12:29:44 sean-k-mooney we also liekly do want to do most if not all of this https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/virt/libvirt/driver.py#L12040-L12085
12:30:08 sean-k-mooney the domain may or may not exist on the dest
12:30:17 sean-k-mooney but thats ok
12:30:18 opendevreview Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
12:30:35 gibi this is basically a poor man's reproducer ^^
12:32:08 gibi I did not dig deep enough to say for sure but it sounds like an unconditional rpc to the dest during rollback is a potential solution for this
12:32:13 gibi I let gmaan to chime in :)
12:32:55 gibi and now that https://review.opendev.org/c/openstack/nova/+/1003245 I can go back to try to reproduce the DB locked case
12:33:17 sean-k-mooney https://paste.opendev.org/show/borJKJA9ol8nnqWHqBru/
12:33:23 sean-k-mooney i would be tempted to do that
12:34:37 gibi yeah it is not a bad idea on the surface for sure but I did not dig deep enough to say it has no unwanted side effect
12:35:05 sean-k-mooney right i did skim pass torugh the manger eand driver but not enough to say it correct
12:35:18 sean-k-mooney but its proably worht a try when we have more time
12:35:40 gibi yepp
12:35:49 gibi I hope gmaan will have the time to dig deeper
12:36:51 sean-k-mooney ill comment on the bug and go back to post cybrog and watcehr release stuff
12:37:23 sean-k-mooney at least in nova we have enough fucntial testign of live migraton to validate much fo this
12:38:08 sean-k-mooney i.e. without going all the way to tempest and bespoke devstack deployment to verify all the edgecases
12:39:41 gibi bah one more functional-threading instability (seen 3 times now) https://bugs.launchpad.net/nova/+bug/2166014
12:40:15 gibi sean-k-mooney: thanks for checking it out
15:58:51 gmaan gibi: sean-k-mooney yes, live migration at DEST task start at "pre_live_migration" (which is on 2nd RPC server) and ends at two place 1. post_live_migration_at_destination 2. rollback_live_migration_at_destination so normal RPC rollbackl is covered (the if part)
15:59:21 gmaan but _rollback_live_migration() happening on source and (in else block) it does not notify the dest about it and task on dest is not marked as ended
16:00:35 gmaan so yea it is bug in this code path. solution is not easy because task is on dest (tracking list on dest process) and source cannot end that due to separate process
16:00:46 gmaan let me dig into it more today
16:01:37 gmaan btw thanks gibi for the functional tests bug https://bugs.launchpad.net/nova/+bug/2165991. just to make sure that this is not production issue as that thread is daemon and will go away with process. but agree to fix it in functional tests
16:02:24 gmaan i really wanted to move all unit/functional tests pass with defaults, I will keep checking your series of fixing hanging tests or spend my time also to find/fix a few
16:02:27 sean-k-mooney gmaan: well it can if it can safely call the rpc
16:02:40 sean-k-mooney i think it can safely call the rpc unconditionally
16:02:44 sean-k-mooney but we need to confirm
16:02:54 sean-k-mooney there is no way to od it localy however
16:04:22 gmaan yeah, RPC call will work but we have these kind of communication gap between source and dest, this live migration sand other i found in revert_resize and those can be handled in part-3 where we can try to make operations more self-healing or at least abort in good state
16:04:31 sean-k-mooney worst case we will have to add a new rpc jsut for this case to notify the dest
16:04:37 gmaan yeah
16:05:24 gmaan let me dig into it today if that is similar to revert_resize case and ok to move to part-3 or has to be handled in this part-2
16:05:26 sean-k-mooney if we added a new rpc we might be able to make it a littel mroe generic i.e notify_task_compelte
16:06:07 gmaan I will not add task complete or so but call per operation cleanup as actual operation is hanging in other node and it is not just task tracking
16:06:13 sean-k-mooney as long as we have a solution for the test senario for now movign the larger change to pahse 3 i think is ok
16:07:37 gmaan this is revert_resize case where dest can leave things in half way without notifying/doing revert at source https://bugs.launchpad.net/nova/+bug/2162090
16:08:36 gmaan and looks like this live migration issue is on similar issue. and i will not be surprise if we will have more such cases where source/dest are not talking to each other during failure and cleanup completely at both side
16:17:49 gibi ust to make sure that this is not production issue as that thread is daemon and will go away with process. but agree to fix it in functional tests
16:17:54 gibi I disagree ^^
16:18:21 gibi this will make the gracefull shutdown a lot slower than it should be as it will wait for the timeout value
16:19:26 gibi and it will be a warning log during graceful shutdown that sends support to a goose chase about a in progress migration that is not in progress
16:19:55 gmaan is it but, it will not wait for timeout as graceful shutdown will terminate the process and thread will be stopped right?
16:20:31 gibi if you sent the timeout to 300 then the test fill time out as it waits for the whole 300 seconds and that is more than the test case timeout
16:20:46 gibi so it is waiting for the the timeout as it thinks there is a task to wait for
16:20:53 gibi and when times out it exists

Earlier   Later