Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-01
12:18:09 sean-k-mooney but never callign https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L9441-L9444
12:18:29 gibi yeah
12:18:43 gibi I think rollback at dest and post at dest call that helper that finishes the task
12:18:55 gibi but in this scenario there is no rpc to the dest that can call that helper
12:19:19 sean-k-mooney if we just passed destory disks false when we woudl have had do_cleanup=false
12:19:21 sean-k-mooney it might be safe
12:19:21 gibi the source handles the rollback / abort alone as far as I see
12:20:00 sean-k-mooney i.e. if we just alwasy call cleanup at des when not pre_live_migation
12:20:26 sean-k-mooney actully no its recoreded in pre_live_migration
12:20:47 sean-k-mooney so we proably woudl want to alwasy call it if we can make sure its non distuctive for shared sotrage
12:21:22 sean-k-mooney after all this is being called or not called form the souce and the task is beint gract on the dest right
12:22:16 sean-k-mooney so we cant just do _end_pending_dest_live_migration_task since that woudl remvoe it form our local dictionatry
12:22:38 gibi yeah the livemigartion failure detected on the source but this task is pending at the dest
12:23:00 gibi if the source can alway send and rpc to the dest then that rpc can remove the pending task on the dest
12:24:55 sean-k-mooney i think instead of tghe if you could and do_cleanup with detstory_disks
12:25:41 sean-k-mooney the remainign question is are any of the actions in https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10658-L10731 soemthing that are unsafe to attpemt alwasys
12:26:05 sean-k-mooney i think the network cleanup is fine
12:26:24 sean-k-mooney we defintly dont want to leak portbindings and do want to make suere we have cleaned up any ports
12:27:02 sean-k-mooney freeing pci claims is also somthign we woudl prefr to do rather then waith for the resouce tracker
12:27:39 sean-k-mooney its possibel we failed before they were claimed btu i tough it and we coudl handel that anyway
12:27:56 sean-k-mooney so i think the manger level tasks all look ok
12:29:44 sean-k-mooney we also liekly do want to do most if not all of this https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/virt/libvirt/driver.py#L12040-L12085
12:30:08 sean-k-mooney the domain may or may not exist on the dest
12:30:17 sean-k-mooney but thats ok
12:30:18 opendevreview Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
12:30:35 gibi this is basically a poor man's reproducer ^^
12:32:08 gibi I did not dig deep enough to say for sure but it sounds like an unconditional rpc to the dest during rollback is a potential solution for this
12:32:13 gibi I let gmaan to chime in :)
12:32:55 gibi and now that https://review.opendev.org/c/openstack/nova/+/1003245 I can go back to try to reproduce the DB locked case
12:33:17 sean-k-mooney https://paste.opendev.org/show/borJKJA9ol8nnqWHqBru/
12:33:23 sean-k-mooney i would be tempted to do that
12:34:37 gibi yeah it is not a bad idea on the surface for sure but I did not dig deep enough to say it has no unwanted side effect
12:35:05 sean-k-mooney right i did skim pass torugh the manger eand driver but not enough to say it correct
12:35:18 sean-k-mooney but its proably worht a try when we have more time
12:35:40 gibi yepp
12:35:49 gibi I hope gmaan will have the time to dig deeper
12:36:51 sean-k-mooney ill comment on the bug and go back to post cybrog and watcehr release stuff
12:37:23 sean-k-mooney at least in nova we have enough fucntial testign of live migraton to validate much fo this
12:38:08 sean-k-mooney i.e. without going all the way to tempest and bespoke devstack deployment to verify all the edgecases
12:39:41 gibi bah one more functional-threading instability (seen 3 times now) https://bugs.launchpad.net/nova/+bug/2166014
12:40:15 gibi sean-k-mooney: thanks for checking it out
15:58:51 gmaan gibi: sean-k-mooney yes, live migration at DEST task start at "pre_live_migration" (which is on 2nd RPC server) and ends at two place 1. post_live_migration_at_destination 2. rollback_live_migration_at_destination so normal RPC rollbackl is covered (the if part)
15:59:21 gmaan but _rollback_live_migration() happening on source and (in else block) it does not notify the dest about it and task on dest is not marked as ended
16:00:35 gmaan so yea it is bug in this code path. solution is not easy because task is on dest (tracking list on dest process) and source cannot end that due to separate process
16:00:46 gmaan let me dig into it more today
16:01:37 gmaan btw thanks gibi for the functional tests bug https://bugs.launchpad.net/nova/+bug/2165991. just to make sure that this is not production issue as that thread is daemon and will go away with process. but agree to fix it in functional tests
16:02:24 gmaan i really wanted to move all unit/functional tests pass with defaults, I will keep checking your series of fixing hanging tests or spend my time also to find/fix a few
16:02:27 sean-k-mooney gmaan: well it can if it can safely call the rpc
16:02:40 sean-k-mooney i think it can safely call the rpc unconditionally
16:02:44 sean-k-mooney but we need to confirm
16:02:54 sean-k-mooney there is no way to od it localy however
16:04:22 gmaan yeah, RPC call will work but we have these kind of communication gap between source and dest, this live migration sand other i found in revert_resize and those can be handled in part-3 where we can try to make operations more self-healing or at least abort in good state
16:04:31 sean-k-mooney worst case we will have to add a new rpc jsut for this case to notify the dest
16:04:37 gmaan yeah
16:05:24 gmaan let me dig into it today if that is similar to revert_resize case and ok to move to part-3 or has to be handled in this part-2
16:05:26 sean-k-mooney if we added a new rpc we might be able to make it a littel mroe generic i.e notify_task_compelte
16:06:07 gmaan I will not add task complete or so but call per operation cleanup as actual operation is hanging in other node and it is not just task tracking
16:06:13 sean-k-mooney as long as we have a solution for the test senario for now movign the larger change to pahse 3 i think is ok
16:07:37 gmaan this is revert_resize case where dest can leave things in half way without notifying/doing revert at source https://bugs.launchpad.net/nova/+bug/2162090
16:08:36 gmaan and looks like this live migration issue is on similar issue. and i will not be surprise if we will have more such cases where source/dest are not talking to each other during failure and cleanup completely at both side
16:17:49 gibi ust to make sure that this is not production issue as that thread is daemon and will go away with process. but agree to fix it in functional tests
16:17:54 gibi I disagree ^^
16:18:21 gibi this will make the gracefull shutdown a lot slower than it should be as it will wait for the timeout value
16:19:26 gibi and it will be a warning log during graceful shutdown that sends support to a goose chase about a in progress migration that is not in progress
16:19:55 gmaan is it but, it will not wait for timeout as graceful shutdown will terminate the process and thread will be stopped right?
16:20:31 gibi if you sent the timeout to 300 then the test fill time out as it waits for the whole 300 seconds and that is more than the test case timeout
16:20:46 gibi so it is waiting for the the timeout as it thinks there is a task to wait for
16:20:53 gibi and when times out it exists
16:21:04 gibi but that means we wait a 180 seconds by default
16:21:08 gmaan yes for functional tests as process is not exit so i agree to fix the live migration task hanging.
16:22:35 gmaan the live migration task left is I agree is the production bug but I am talking about hthttps://bugs.launchpad.net/nova/+bug/2165991 though it is because of migration task hanging
16:22:37 gibi similarly the dest nova-compute will not exit until 180sec as it thinks there is an in progress live migration
16:22:48 gmaan I think I mixed both in my question
16:23:05 gibi I think that two things are the same :)
16:23:10 gibi feel free to rename the bug title
16:23:16 gibi to talk about the leaked task
16:23:24 gibi the hanging functional test is just a symptom
16:23:26 gmaan gibi: my question was if anything we need to fix on shutdown_thread creation and when it fail/timeout
16:23:33 gmaan gibi: agree on that
16:24:27 gmaan I am saying this part is all ok https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/service.py#L433-L435
16:24:45 gmaan and yes we need to fix that task leak
16:25:05 gibi it is a trap for functional test due to the thread can leak, but I can cirumwent that with setting a non 0 timeout to wait for that thread
16:26:52 gibi and when I set that timeout to 300 I saw the task leak in 3 functional tests I reported as the bug
16:27:21 gmaan ack
16:28:46 gibi also now I see that setting 300sec timeout also make a bunch of test sad in the eventlet job
16:29:32 gibi so we might have more leaks of we might have differently hanging shutdown in eventlet job
16:29:49 gibi https://review.opendev.org/c/openstack/nova/+/1003266/1
16:42:06 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
16:42:20 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
16:56:09 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
17:09:03 Uggla dansmith, melwitt, gouthamr comment about the virtiofs migration and FFE: https://review.opendev.org/c/openstack/nova/+/1001916/comments/965cff03_8b75df37
17:09:52 gouthamr \o/ ty Uggla
17:11:44 Uggla gouthamr, dansmith, melwitt I think Melanie is on PTO until tomorrow. I hope Melanie could jump on this review as soon as she will be back.
17:13:31 Uggla gouthamr, dansmith, melwitt also forget to mention that the msg above contains a short summary of the tests I did.
17:14:51 gouthamr ++ i read them, ty for testing those scenarios. I will try and add a multi-node job with cephfs to the manila gate, and one with LVM/NFS to the nova gate (and spread the test coverage between projects)
17:16:40 Uggla gouthamr, it will be great. Just to note I also tried several way to break it, and it looks solid so far.
17:41:29 gouthamr Uggla: ++ :) ty!
17:56:34 gmaan gibi: for eventlet functional job, you mean tox cover jobs or nova-tox-functional-py313 or both? bth failing though but I am seeing only one failure in nova-tox-functional-py313
19:33:33 melwitt Uggla: thanks for adding the comment, I will be reading over everything
20:16:19 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900

Earlier   Later