Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-09
13:15:52 opendevreview Stephen Finucane proposed openstack/placement master: tox: Use constraints option https://review.opendev.org/c/openstack/placement/+/996893
13:15:53 opendevreview Stephen Finucane proposed openstack/placement master: Replace license classifier https://review.opendev.org/c/openstack/placement/+/1004422
13:15:53 opendevreview Stephen Finucane proposed openstack/placement master: Migrate requirements to pyproject.toml https://review.opendev.org/c/openstack/placement/+/1004423
13:17:24 Uggla stephenfin, I have just +1 the change above, sorry for the latency.
13:42:04 gibi Uggla: finally back to https://review.opendev.org/c/openstack/nova/+/999977/ I left some small suggestion inline but it is looks good pretty good now
13:43:18 Uggla gibi, thanks I will look at that.
14:12:13 gibi Uggla: also left some comments in the RC1 etherpad about api microversion history and prelude I guess those are still needs to be done
14:17:09 Uggla gibi, regarding prelude as it is a copy of highlight, I was waiting the HL to be settled which is the case now, so I'll create it. Thanks for the microversion history because I thought it was ok but missed it.
14:18:03 gibi cool
14:18:13 gibi I think the rest looks good
14:20:04 opendevreview Merged openstack/nova-specs master: Move Hibiscus implemented specs https://review.opendev.org/c/openstack/nova-specs/+/1004442
15:01:09 gibi Uggla: left feedback on https://review.opendev.org/c/openstack/nova/+/999978 I think this evac bug and my evac bug discussed on the Monday call are sharing a root case. I left suggestion how to combine the two fixes that is non-controversial from the Monday discussion perspective but probably useful for both bugs. It is pretty complicated so feel free to ask me to jump on a call to explain more
15:05:01 bauzas gibi: I guess we won't have eventlet meeting, right?
15:07:21 Uggla gibi, yeah I had the feeling that Monday topic was more or less linked and that a 2 steps approach is required. One short term to meet customer happy and one longer term for the PTG.
15:15:30 opendevreview ribaudr proposed openstack/nova master: Mark 2.104 as maximum API version for 2026.2 Hibiscus https://review.opendev.org/c/openstack/nova/+/1004872
15:22:35 gibi bauzas: it is ongoing but nothing important there
15:27:44 dansmith gmaan: sorry I just realized I never replied on your task leak thing
15:30:39 Uggla Reminder: upstream triage (https://meet.google.com/zjr-rxus-hzj)
15:31:36 gibi couple of mintues lte
15:31:37 gibi late
15:39:32 opendevreview Grzegorz Grasza proposed openstack/nova master: Match fixed_ip literally instead of partially escaping it as a regex https://review.opendev.org/c/openstack/nova/+/1004878
16:28:15 opendevreview Clif Houck proposed openstack/nova stable/2026.1: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/1004890
16:37:22 opendevreview Merged openstack/placement master: tox: Use constraints option https://review.opendev.org/c/openstack/placement/+/996893
16:43:00 dansmith gibi: are you still around/
16:52:07 gmaan dansmith: thanks for reply, reading
16:52:24 dansmith gmaan: I'm curious what gibi thinks about this
16:52:37 dansmith wondering if he was already okay with the RPC breakage or didn't realize when he +2d
16:53:52 gmaan dansmith: I think he mentioned about the revert if it fails but not specific to new vs old node
16:54:13 dansmith I know
17:03:41 opendevreview Clif Houck proposed openstack/nova stable/2025.2: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/1004899
17:05:52 gmaan dansmith: I think, adding a new RPC call (end_tracked_task or something) will be better here with RPC version bump. do you want me change that way or wanted to wait for gibi reply?
17:06:50 dansmith I guess I'd rather not have a new call just for the very rare "cancel before started" case... why do you like that better?
17:06:52 gmaan that is what i was thinking to bump the RPC for either change but was not sure if we can do the version bumo at this stage or not. if yes then no issue
17:07:26 gmaan dansmith: it will not cancel before started, it can check if task is recorded otherwise log warning
17:07:40 dansmith it's not great to bump for either for sure at this point
17:08:06 dansmith gmaan: it's a cancel to the destination because we basically never started the migration which now doesn't need to be cleaned up right?
17:08:12 gmaan I am thinking it generic way where this common RPC call can be used in other place, because I am not sure we handle all cases of error on each node for cross-node operations
17:08:13 dansmith bauzas: are you still around?
17:08:22 bauzas I am
17:08:34 dansmith bauzas: can you skim this? https://review.opendev.org/c/openstack/nova/+/1003386
17:09:00 dansmith just looking for another opinion.. I don't really like either option but they both require an RPC bump to fix (or we just ignore this case and leave it as a delayed shutdown bug I guess)
17:10:03 bauzas hah, ok, I can try to take a look but I don't have a lot of context for now
17:10:53 gmaan dansmith: but if rollback happening on source then it means migration started in dest right?
17:11:32 dansmith gmaan: started in nova but probably not libvirt yet right?
17:11:55 gmaan yes not till libvirt
17:12:35 bauzas wait
17:12:45 bauzas now I have a bit more of knowledge
17:13:16 bauzas all of the skim is for _post_live_migration, ie. once the instance is either fully migrated or rollbacked
17:13:30 dansmith actually, I guess not... it's any shared storage migration, even if it has already started
17:13:50 bauzas because we know we have remnants to cleanup, we have to modify the other compute to remove them
17:14:43 dansmith right, that's the case where we _do_ make the call
17:14:50 dansmith but the bug is in cases where we don't
17:15:14 dansmith as of the graceful shutdown stuff, we _always_ need to make the (or a) call to the dest because we left a pending task if nothing else
17:15:31 bauzas https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10557
17:15:45 dansmith so to me the right solution is to always make that call in the future and pass it the cleanup flag maybe, but we can only do that with a version bump
17:15:46 bauzas all of that is in _post_live_migrate
17:15:57 dansmith else we will ask n-1 computes to do a cleanup when there is none to do
17:16:41 gmaan its during cleanup of pre live migration not post
17:16:56 dansmith right
17:17:21 dansmith well, during cleanup of live migration... to clean up what was started in pre_live_migration
17:17:25 bauzas https://docs.openstack.org/nova/latest/reference/live-migration.html
17:17:33 gmaan dansmith: yeah, that is my preference too when i proposed change, if we can delay this bug and mentioned as known bug and in part-2 we can handle by flag or new call
17:17:57 gmaan dansmith: yes -> 'to clean up what was started in pre_live_migration'
17:18:08 dansmith bauzas: just to be clear, I'm looking for opinions on if we can/should bump the compute RPC to add this call/param right now, just before rc1
17:19:34 bauzas we're changing the live-migration workflow per se, not exactly the RPC interface
17:19:48 bauzas so that's kind of a grey area
17:20:03 bauzas but in case of a rolling upgrade, we have to ensure we're not breaking for sure
17:20:18 dansmith huh?
17:20:24 bauzas because old compute could expect a different workflow
17:20:27 dansmith it's 100% an RPC change :)
17:21:08 gmaan yeah, old dest will always do the rollback at dest when not needed and most probably fail
17:21:15 bauzas we don't change the RPC parameters for sure, but we changing the callers yes
17:21:17 dansmith not only are we expecting the new behavior of canceling the task, we need to be passing a parameter to instruct the old side not to try to clean up something it shouldn't be cleaning up
17:21:33 dansmith bauzas: we _need_ to add a parameter
17:21:51 bauzas lemme explain it differently
17:21:53 gmaan dansmith: flag can be avoided as as current code change where dest can calculate the cleanup is required or not
17:21:57 dansmith right now this will just generate (we think) a traceback
17:22:31 bauzas either we consider the existing patch good and then I have a concern due to rolling upgrades (old computes may expect the calls differently), or we assume we need to pass a new param and then I have concern witth a late RPC change
17:23:03 dansmith gmaan: but the flag also gives us the indication that we're making the new call so we can avoid doing it on the older versioon
17:23:03 bauzas either way, I'm not happy with such behavioural change so close to the release
17:23:48 dansmith bauzas: I don't think I'm okay with no bump and either change
17:23:58 dansmith it's very clearly just side-stepping RPC behavioral expectations
17:24:11 gmaan dansmith: that can but we can just check the same based on pin version and cleanup flag calculation. I mean flag is better way but if we want to avoid RPC signature change that is possible
17:25:06 bauzas ship has sailed for a RPC bump this release https://review.opendev.org/c/openstack/nova/+/1004424
17:25:12 dansmith gmaan: that would be compute manager inspecting the pin set in the rpcapi right?
17:25:39 bauzas dansmith: that was my "either" : the existing proposal has upgrade concners
17:26:05 dansmith bauzas: that hasn't merged so I don't think it has _sailed_ nor do I think landing that is terminal until we make the release :)
17:26:42 bauzas can't we just defer the fix to post-rc1 ?
17:27:10 dansmith meaning have this be broken in H? we can for sure, that's one of the options
17:27:14 gmaan dansmith: ah yeah. i get your point now
17:27:19 gmaan so what is main concern on stating it as known bug which is in scenario 1. live migration started at pre migration stage, source rolling it back , it is shared storage where cleanup not needed at dest 2. dest shutting down. Issue: it will hold dest graceful shutdown for full time as configured (manager_shutdown_time default 160 sec)
17:28:03 dansmith gmaan: yep, that's one option and was also something I think is worth surveying opinions on
17:28:04 gmaan only issue it will cause is to make shutdown taking full time instead of having task tracking benefits in this scenario
17:28:14 dansmith yup
17:28:17 gmaan which is nothing change than before this rlease
17:28:42 gmaan bcz I am not not 100% ok to bump RPC version or try some workaround which we need to chaneg again
17:28:52 gmaan *I am also not
17:29:19 bauzas because we're leaking threads as of now ?
17:29:47 dansmith not leaking threads really.. started a task that will wait for $timeout before allowing n-cpu to exit because it thinks a migration will still be coming
17:29:56 bauzas I see

Earlier   Later