Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-09
17:13:16 bauzas all of the skim is for _post_live_migration, ie. once the instance is either fully migrated or rollbacked
17:13:30 dansmith actually, I guess not... it's any shared storage migration, even if it has already started
17:13:50 bauzas because we know we have remnants to cleanup, we have to modify the other compute to remove them
17:14:43 dansmith right, that's the case where we _do_ make the call
17:14:50 dansmith but the bug is in cases where we don't
17:15:14 dansmith as of the graceful shutdown stuff, we _always_ need to make the (or a) call to the dest because we left a pending task if nothing else
17:15:31 bauzas https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10557
17:15:45 dansmith so to me the right solution is to always make that call in the future and pass it the cleanup flag maybe, but we can only do that with a version bump
17:15:46 bauzas all of that is in _post_live_migrate
17:15:57 dansmith else we will ask n-1 computes to do a cleanup when there is none to do
17:16:41 gmaan its during cleanup of pre live migration not post
17:16:56 dansmith right
17:17:21 dansmith well, during cleanup of live migration... to clean up what was started in pre_live_migration
17:17:25 bauzas https://docs.openstack.org/nova/latest/reference/live-migration.html
17:17:33 gmaan dansmith: yeah, that is my preference too when i proposed change, if we can delay this bug and mentioned as known bug and in part-2 we can handle by flag or new call
17:17:57 gmaan dansmith: yes -> 'to clean up what was started in pre_live_migration'
17:18:08 dansmith bauzas: just to be clear, I'm looking for opinions on if we can/should bump the compute RPC to add this call/param right now, just before rc1
17:19:34 bauzas we're changing the live-migration workflow per se, not exactly the RPC interface
17:19:48 bauzas so that's kind of a grey area
17:20:03 bauzas but in case of a rolling upgrade, we have to ensure we're not breaking for sure
17:20:18 dansmith huh?
17:20:24 bauzas because old compute could expect a different workflow
17:20:27 dansmith it's 100% an RPC change :)
17:21:08 gmaan yeah, old dest will always do the rollback at dest when not needed and most probably fail
17:21:15 bauzas we don't change the RPC parameters for sure, but we changing the callers yes
17:21:17 dansmith not only are we expecting the new behavior of canceling the task, we need to be passing a parameter to instruct the old side not to try to clean up something it shouldn't be cleaning up
17:21:33 dansmith bauzas: we _need_ to add a parameter
17:21:51 bauzas lemme explain it differently
17:21:53 gmaan dansmith: flag can be avoided as as current code change where dest can calculate the cleanup is required or not
17:21:57 dansmith right now this will just generate (we think) a traceback
17:22:31 bauzas either we consider the existing patch good and then I have a concern due to rolling upgrades (old computes may expect the calls differently), or we assume we need to pass a new param and then I have concern witth a late RPC change
17:23:03 bauzas either way, I'm not happy with such behavioural change so close to the release
17:23:03 dansmith gmaan: but the flag also gives us the indication that we're making the new call so we can avoid doing it on the older versioon
17:23:48 dansmith bauzas: I don't think I'm okay with no bump and either change
17:23:58 dansmith it's very clearly just side-stepping RPC behavioral expectations
17:24:11 gmaan dansmith: that can but we can just check the same based on pin version and cleanup flag calculation. I mean flag is better way but if we want to avoid RPC signature change that is possible
17:25:06 bauzas ship has sailed for a RPC bump this release https://review.opendev.org/c/openstack/nova/+/1004424
17:25:12 dansmith gmaan: that would be compute manager inspecting the pin set in the rpcapi right?
17:25:39 bauzas dansmith: that was my "either" : the existing proposal has upgrade concners
17:26:05 dansmith bauzas: that hasn't merged so I don't think it has _sailed_ nor do I think landing that is terminal until we make the release :)
17:26:42 bauzas can't we just defer the fix to post-rc1 ?
17:27:10 dansmith meaning have this be broken in H? we can for sure, that's one of the options
17:27:14 gmaan dansmith: ah yeah. i get your point now
17:27:19 gmaan so what is main concern on stating it as known bug which is in scenario 1. live migration started at pre migration stage, source rolling it back , it is shared storage where cleanup not needed at dest 2. dest shutting down. Issue: it will hold dest graceful shutdown for full time as configured (manager_shutdown_time default 160 sec)
17:28:03 dansmith gmaan: yep, that's one option and was also something I think is worth surveying opinions on
17:28:04 gmaan only issue it will cause is to make shutdown taking full time instead of having task tracking benefits in this scenario
17:28:14 dansmith yup
17:28:17 gmaan which is nothing change than before this rlease
17:28:42 gmaan bcz I am not not 100% ok to bump RPC version or try some workaround which we need to chaneg again
17:28:52 gmaan *I am also not
17:29:19 bauzas because we're leaking threads as of now ?
17:29:47 dansmith not leaking threads really.. started a task that will wait for $timeout before allowing n-cpu to exit because it thinks a migration will still be coming
17:29:56 bauzas I see
17:29:59 gmaan bauzas: bcz we are leaking task from task tracking, live migration task started at dest willnot be ended as source do rollbacl and do not call dest
17:30:10 dansmith but it's very rare that this would be hit, and if you don't enable (or do disable) graceful shutdown it won't matter anyway
17:30:25 bauzas I think the stable policy still allows us to backport an RPC change provided certain constraints, but I can doublechecks
17:30:29 bauzas doublecheck*
17:30:29 dansmith so maybe just merge a reno with known-bug
17:30:42 dansmith bauzas: we can but it is majorly less good to do that than just merge it now
17:30:50 dansmith I think if we don't fix now, we won't fix in stable
17:31:58 bauzas I see your point, I don't have a particular opinion about it except that I don't want the existing patch to be merged without testing rolling upgrades
17:32:02 gmaan yeah, fix would not be backportable
17:33:32 bauzas we don't have a job that runs live-migration on a multinode grenade env, right?
17:34:41 dansmith I need to take a break
17:34:52 dansmith I think I'm most in favor of just #knownbug'ing it and fix it in I
17:35:32 dansmith if it absolutely explodes in our face, we can backport something like the current patch (or offer it to people out of band) but I doubt it will
17:35:37 gmaan dansmith: I have one more solution which does not need RPC bump or any ordering change of cleaup
17:36:03 bauzas https://zuul.opendev.org/t/openstack/job/nova-grenade-multinode is testing live-migration on a rolling upgrade fashion but we don't run graceful shutdown on it
17:36:40 gmaan dansmith: so this rollback at sourec happening after pre_live_migration at dest and where dest raise any error. and task is leaked beacuxse we start the live migration at dest task before dest's pre_live_migration
17:37:18 gmaan what if we start the live-migration_at_dest task after dest's pre_live_migration is completed so that this rollback at source will not leak the task at dest
17:37:49 gmaan I mean record lve migration at dest when we know it is actually going to start and not pre live migration failed
17:38:13 dansmith I dunno, that seems like a lot more change from what we have now and I'm probably too fried to think about it
17:38:38 dansmith I also thought about making the task on the dest just poll the migration object while it's waiting, but that is more work so late as well
17:39:07 gmaan I mean we are starting live migration at dest too early even we do not know if dest's pre live migration things are successful or not
17:39:36 gmaan poll task where and how to decide when to end it?
17:40:17 gmaan dansmith: for 'late' yes, but this sems backportable fix if we do it in next release right?
17:40:33 dansmith poll task in an executor task, end it when the migration goes to finished or error.. it was just a thought and something we'd have to look at
17:40:41 dansmith gmaan: potentially
17:41:51 gmaan dansmith: ohk so based on the migration status itself. yeah that is one possible way
18:11:46 bauzas I had to move errand, but I'd indeed prefer the third solution, which is to end the task when the migration finishes, instead of overloading our RPC interface for that need
18:27:07 gmaan yeah, i am finding that solution a better one. My solution of shifting the start_task is not perfect and does not solve issue when dest's pre live migration is successful but vif plugged event can cause rollback on source -
18:34:30 gmaan dansmith: bauzas i commented in gerrit about going for 'listing it as a known bug' path (please check), we can wait for gibi on his opinion and i can proceed accordingly
20:31:52 opendevreview ribaudr proposed openstack/nova master: Add Hibiscus prelude section https://review.opendev.org/c/openstack/nova/+/1004927
#openstack-nova - 2026-09-10
06:33:56 simondodsley gibi: Is there any chance that this patch (https://review.opendev.org/c/openstack/nova/+/973750) be relooked at again. It's a really simple fix for a big issue. Already have one +2
07:32:56 gibi simondodsley: I've approved it. I hope I did not risk RC1 with it. But I'm in a good mood so I taking my chances here. Btw next time feel free to ping me eariler if I reviewed someting before and never showed up re-reviewing the patch
09:15:26 bauzas gibi: would you agree with gmaan's comment on https://review.opendev.org/c/openstack/nova/+/1003386 that we go for a known bug reno instead of fixing it ?
09:15:57 bauzas I'm cool with that and if you look at the irc logs we discussed a nice alternative
09:53:06 gibi let me check
09:59:07 gibi I'v read the review comments. I'm OK to keep as a known bug for now. I agree that it is not fatal thing to have this task leaked as it only slows down the shutdown, does not prevent it
10:01:20 gibi Uggla: thanks I will jump there
10:02:24 gibi but I also need to read the tc channel logs to get the full context
10:42:57 bauzas given the lack of consensus yet, I'd propose to defer the application of nova-reviewers until a consequent discussion can happen (before PTG or in it) for at least being all of up agreeing for find some policy how to use this new group
12:23:14 sean-k-mooney ls
12:23:38 sean-k-mooney :)
12:25:30 sean-k-mooney gibi: gmaan: https://review.opendev.org/c/openstack/nova/+/1004687/1 and https://review.opendev.org/c/openstack/nova/+/966479/5 update nova next to ubuntu 26.04 and nova-alt-config to to debian 13, the latter was ment ot be done in 2026.1 so it woudl be nice if we can proceed with both this cycle, can ye take a look when ye have time
12:51:51 gibi Uggla: thanks for the email, I responded there and commented on the https://review.opendev.org/c/openstack/nova/+/986141 as well
12:52:01 gibi we need to start figure out a the way forward
13:04:40 opendevreview Kamil Sambor proposed openstack/nova master: Fix pickling error that forces oslo.service to fall back to fork https://review.opendev.org/c/openstack/nova/+/1000956
13:06:12 gablan hi gibi! I'm backporting the graceful shutdown work from master to stable/2026.1 downstream, and I hit the TODO you left in Host.cleanup() in `nova/virt/libvirt/host.py` about `stopping _event_handler` and `the _conn_event_thread` loop. do you have anything planned for it, or can I propose a patch for it?
13:17:35 gibi gablan: check master I think gmaan recent changes there are addressing it

Earlier   Later