Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-09
15:15:30 opendevreview ribaudr proposed openstack/nova master: Mark 2.104 as maximum API version for 2026.2 Hibiscus https://review.opendev.org/c/openstack/nova/+/1004872
15:22:35 gibi bauzas: it is ongoing but nothing important there
15:27:44 dansmith gmaan: sorry I just realized I never replied on your task leak thing
15:30:39 Uggla Reminder: upstream triage (https://meet.google.com/zjr-rxus-hzj)
15:31:36 gibi couple of mintues lte
15:31:37 gibi late
15:39:32 opendevreview Grzegorz Grasza proposed openstack/nova master: Match fixed_ip literally instead of partially escaping it as a regex https://review.opendev.org/c/openstack/nova/+/1004878
16:28:15 opendevreview Clif Houck proposed openstack/nova stable/2026.1: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/1004890
16:37:22 opendevreview Merged openstack/placement master: tox: Use constraints option https://review.opendev.org/c/openstack/placement/+/996893
16:43:00 dansmith gibi: are you still around/
16:52:07 gmaan dansmith: thanks for reply, reading
16:52:24 dansmith gmaan: I'm curious what gibi thinks about this
16:52:37 dansmith wondering if he was already okay with the RPC breakage or didn't realize when he +2d
16:53:52 gmaan dansmith: I think he mentioned about the revert if it fails but not specific to new vs old node
16:54:13 dansmith I know
17:03:41 opendevreview Clif Houck proposed openstack/nova stable/2025.2: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/1004899
17:05:52 gmaan dansmith: I think, adding a new RPC call (end_tracked_task or something) will be better here with RPC version bump. do you want me change that way or wanted to wait for gibi reply?
17:06:50 dansmith I guess I'd rather not have a new call just for the very rare "cancel before started" case... why do you like that better?
17:06:52 gmaan that is what i was thinking to bump the RPC for either change but was not sure if we can do the version bumo at this stage or not. if yes then no issue
17:07:26 gmaan dansmith: it will not cancel before started, it can check if task is recorded otherwise log warning
17:07:40 dansmith it's not great to bump for either for sure at this point
17:08:06 dansmith gmaan: it's a cancel to the destination because we basically never started the migration which now doesn't need to be cleaned up right?
17:08:12 gmaan I am thinking it generic way where this common RPC call can be used in other place, because I am not sure we handle all cases of error on each node for cross-node operations
17:08:13 dansmith bauzas: are you still around?
17:08:22 bauzas I am
17:08:34 dansmith bauzas: can you skim this? https://review.opendev.org/c/openstack/nova/+/1003386
17:09:00 dansmith just looking for another opinion.. I don't really like either option but they both require an RPC bump to fix (or we just ignore this case and leave it as a delayed shutdown bug I guess)
17:10:03 bauzas hah, ok, I can try to take a look but I don't have a lot of context for now
17:10:53 gmaan dansmith: but if rollback happening on source then it means migration started in dest right?
17:11:32 dansmith gmaan: started in nova but probably not libvirt yet right?
17:11:55 gmaan yes not till libvirt
17:12:35 bauzas wait
17:12:45 bauzas now I have a bit more of knowledge
17:13:16 bauzas all of the skim is for _post_live_migration, ie. once the instance is either fully migrated or rollbacked
17:13:30 dansmith actually, I guess not... it's any shared storage migration, even if it has already started
17:13:50 bauzas because we know we have remnants to cleanup, we have to modify the other compute to remove them
17:14:43 dansmith right, that's the case where we _do_ make the call
17:14:50 dansmith but the bug is in cases where we don't
17:15:14 dansmith as of the graceful shutdown stuff, we _always_ need to make the (or a) call to the dest because we left a pending task if nothing else
17:15:31 bauzas https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10557
17:15:45 dansmith so to me the right solution is to always make that call in the future and pass it the cleanup flag maybe, but we can only do that with a version bump
17:15:46 bauzas all of that is in _post_live_migrate
17:15:57 dansmith else we will ask n-1 computes to do a cleanup when there is none to do
17:16:41 gmaan its during cleanup of pre live migration not post
17:16:56 dansmith right
17:17:21 dansmith well, during cleanup of live migration... to clean up what was started in pre_live_migration
17:17:25 bauzas https://docs.openstack.org/nova/latest/reference/live-migration.html
17:17:33 gmaan dansmith: yeah, that is my preference too when i proposed change, if we can delay this bug and mentioned as known bug and in part-2 we can handle by flag or new call
17:17:57 gmaan dansmith: yes -> 'to clean up what was started in pre_live_migration'
17:18:08 dansmith bauzas: just to be clear, I'm looking for opinions on if we can/should bump the compute RPC to add this call/param right now, just before rc1
17:19:34 bauzas we're changing the live-migration workflow per se, not exactly the RPC interface
17:19:48 bauzas so that's kind of a grey area
17:20:03 bauzas but in case of a rolling upgrade, we have to ensure we're not breaking for sure
17:20:18 dansmith huh?
17:20:24 bauzas because old compute could expect a different workflow
17:20:27 dansmith it's 100% an RPC change :)
17:21:08 gmaan yeah, old dest will always do the rollback at dest when not needed and most probably fail
17:21:15 bauzas we don't change the RPC parameters for sure, but we changing the callers yes
17:21:17 dansmith not only are we expecting the new behavior of canceling the task, we need to be passing a parameter to instruct the old side not to try to clean up something it shouldn't be cleaning up
17:21:33 dansmith bauzas: we _need_ to add a parameter
17:21:51 bauzas lemme explain it differently
17:21:53 gmaan dansmith: flag can be avoided as as current code change where dest can calculate the cleanup is required or not
17:21:57 dansmith right now this will just generate (we think) a traceback
17:22:31 bauzas either we consider the existing patch good and then I have a concern due to rolling upgrades (old computes may expect the calls differently), or we assume we need to pass a new param and then I have concern witth a late RPC change
17:23:03 dansmith gmaan: but the flag also gives us the indication that we're making the new call so we can avoid doing it on the older versioon
17:23:03 bauzas either way, I'm not happy with such behavioural change so close to the release
17:23:48 dansmith bauzas: I don't think I'm okay with no bump and either change
17:23:58 dansmith it's very clearly just side-stepping RPC behavioral expectations
17:24:11 gmaan dansmith: that can but we can just check the same based on pin version and cleanup flag calculation. I mean flag is better way but if we want to avoid RPC signature change that is possible
17:25:06 bauzas ship has sailed for a RPC bump this release https://review.opendev.org/c/openstack/nova/+/1004424
17:25:12 dansmith gmaan: that would be compute manager inspecting the pin set in the rpcapi right?
17:25:39 bauzas dansmith: that was my "either" : the existing proposal has upgrade concners
17:26:05 dansmith bauzas: that hasn't merged so I don't think it has _sailed_ nor do I think landing that is terminal until we make the release :)
17:26:42 bauzas can't we just defer the fix to post-rc1 ?
17:27:10 dansmith meaning have this be broken in H? we can for sure, that's one of the options
17:27:14 gmaan dansmith: ah yeah. i get your point now
17:27:19 gmaan so what is main concern on stating it as known bug which is in scenario 1. live migration started at pre migration stage, source rolling it back , it is shared storage where cleanup not needed at dest 2. dest shutting down. Issue: it will hold dest graceful shutdown for full time as configured (manager_shutdown_time default 160 sec)
17:28:03 dansmith gmaan: yep, that's one option and was also something I think is worth surveying opinions on
17:28:04 gmaan only issue it will cause is to make shutdown taking full time instead of having task tracking benefits in this scenario
17:28:14 dansmith yup
17:28:17 gmaan which is nothing change than before this rlease
17:28:42 gmaan bcz I am not not 100% ok to bump RPC version or try some workaround which we need to chaneg again
17:28:52 gmaan *I am also not
17:29:19 bauzas because we're leaking threads as of now ?
17:29:47 dansmith not leaking threads really.. started a task that will wait for $timeout before allowing n-cpu to exit because it thinks a migration will still be coming
17:29:56 bauzas I see
17:29:59 gmaan bauzas: bcz we are leaking task from task tracking, live migration task started at dest willnot be ended as source do rollbacl and do not call dest
17:30:10 dansmith but it's very rare that this would be hit, and if you don't enable (or do disable) graceful shutdown it won't matter anyway
17:30:25 bauzas I think the stable policy still allows us to backport an RPC change provided certain constraints, but I can doublechecks
17:30:29 dansmith so maybe just merge a reno with known-bug
17:30:29 bauzas doublecheck*
17:30:42 dansmith bauzas: we can but it is majorly less good to do that than just merge it now
17:30:50 dansmith I think if we don't fix now, we won't fix in stable
17:31:58 bauzas I see your point, I don't have a particular opinion about it except that I don't want the existing patch to be merged without testing rolling upgrades
17:32:02 gmaan yeah, fix would not be backportable
17:33:32 bauzas we don't have a job that runs live-migration on a multinode grenade env, right?
17:34:41 dansmith I need to take a break
17:34:52 dansmith I think I'm most in favor of just #knownbug'ing it and fix it in I
17:35:32 dansmith if it absolutely explodes in our face, we can backport something like the current patch (or offer it to people out of band) but I doubt it will
17:35:37 gmaan dansmith: I have one more solution which does not need RPC bump or any ordering change of cleaup

Earlier   Later