| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-09 | |||
| 14:12:13 | gibi | Uggla: also left some comments in the RC1 etherpad about api microversion history and prelude I guess those are still needs to be done | |
| 14:17:09 | Uggla | gibi, regarding prelude as it is a copy of highlight, I was waiting the HL to be settled which is the case now, so I'll create it. Thanks for the microversion history because I thought it was ok but missed it. | |
| 14:18:03 | gibi | cool | |
| 14:18:13 | gibi | I think the rest looks good | |
| 14:20:04 | opendevreview | Merged openstack/nova-specs master: Move Hibiscus implemented specs https://review.opendev.org/c/openstack/nova-specs/+/1004442 | |
| 15:01:09 | gibi | Uggla: left feedback on https://review.opendev.org/c/openstack/nova/+/999978 I think this evac bug and my evac bug discussed on the Monday call are sharing a root case. I left suggestion how to combine the two fixes that is non-controversial from the Monday discussion perspective but probably useful for both bugs. It is pretty complicated so feel free to ask me to jump on a call to explain more | |
| 15:05:01 | bauzas | gibi: I guess we won't have eventlet meeting, right? | |
| 15:07:21 | Uggla | gibi, yeah I had the feeling that Monday topic was more or less linked and that a 2 steps approach is required. One short term to meet customer happy and one longer term for the PTG. | |
| 15:15:30 | opendevreview | ribaudr proposed openstack/nova master: Mark 2.104 as maximum API version for 2026.2 Hibiscus https://review.opendev.org/c/openstack/nova/+/1004872 | |
| 15:22:35 | gibi | bauzas: it is ongoing but nothing important there | |
| 15:27:44 | dansmith | gmaan: sorry I just realized I never replied on your task leak thing | |
| 15:30:39 | Uggla | Reminder: upstream triage (https://meet.google.com/zjr-rxus-hzj) | |
| 15:31:36 | gibi | couple of mintues lte | |
| 15:31:37 | gibi | late | |
| 15:39:32 | opendevreview | Grzegorz Grasza proposed openstack/nova master: Match fixed_ip literally instead of partially escaping it as a regex https://review.opendev.org/c/openstack/nova/+/1004878 | |
| 16:28:15 | opendevreview | Clif Houck proposed openstack/nova stable/2026.1: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/1004890 | |
| 16:37:22 | opendevreview | Merged openstack/placement master: tox: Use constraints option https://review.opendev.org/c/openstack/placement/+/996893 | |
| 16:43:00 | dansmith | gibi: are you still around/ | |
| 16:52:07 | gmaan | dansmith: thanks for reply, reading | |
| 16:52:24 | dansmith | gmaan: I'm curious what gibi thinks about this | |
| 16:52:37 | dansmith | wondering if he was already okay with the RPC breakage or didn't realize when he +2d | |
| 16:53:52 | gmaan | dansmith: I think he mentioned about the revert if it fails but not specific to new vs old node | |
| 16:54:13 | dansmith | I know | |
| 17:03:41 | opendevreview | Clif Houck proposed openstack/nova stable/2025.2: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/1004899 | |
| 17:05:52 | gmaan | dansmith: I think, adding a new RPC call (end_tracked_task or something) will be better here with RPC version bump. do you want me change that way or wanted to wait for gibi reply? | |
| 17:06:50 | dansmith | I guess I'd rather not have a new call just for the very rare "cancel before started" case... why do you like that better? | |
| 17:06:52 | gmaan | that is what i was thinking to bump the RPC for either change but was not sure if we can do the version bumo at this stage or not. if yes then no issue | |
| 17:07:26 | gmaan | dansmith: it will not cancel before started, it can check if task is recorded otherwise log warning | |
| 17:07:40 | dansmith | it's not great to bump for either for sure at this point | |
| 17:08:06 | dansmith | gmaan: it's a cancel to the destination because we basically never started the migration which now doesn't need to be cleaned up right? | |
| 17:08:12 | gmaan | I am thinking it generic way where this common RPC call can be used in other place, because I am not sure we handle all cases of error on each node for cross-node operations | |
| 17:08:13 | dansmith | bauzas: are you still around? | |
| 17:08:22 | bauzas | I am | |
| 17:08:34 | dansmith | bauzas: can you skim this? https://review.opendev.org/c/openstack/nova/+/1003386 | |
| 17:09:00 | dansmith | just looking for another opinion.. I don't really like either option but they both require an RPC bump to fix (or we just ignore this case and leave it as a delayed shutdown bug I guess) | |
| 17:10:03 | bauzas | hah, ok, I can try to take a look but I don't have a lot of context for now | |
| 17:10:53 | gmaan | dansmith: but if rollback happening on source then it means migration started in dest right? | |
| 17:11:32 | dansmith | gmaan: started in nova but probably not libvirt yet right? | |
| 17:11:55 | gmaan | yes not till libvirt | |
| 17:12:35 | bauzas | wait | |
| 17:12:45 | bauzas | now I have a bit more of knowledge | |
| 17:13:16 | bauzas | all of the skim is for _post_live_migration, ie. once the instance is either fully migrated or rollbacked | |
| 17:13:30 | dansmith | actually, I guess not... it's any shared storage migration, even if it has already started | |
| 17:13:50 | bauzas | because we know we have remnants to cleanup, we have to modify the other compute to remove them | |
| 17:14:43 | dansmith | right, that's the case where we _do_ make the call | |
| 17:14:50 | dansmith | but the bug is in cases where we don't | |
| 17:15:14 | dansmith | as of the graceful shutdown stuff, we _always_ need to make the (or a) call to the dest because we left a pending task if nothing else | |
| 17:15:31 | bauzas | https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10557 | |
| 17:15:45 | dansmith | so to me the right solution is to always make that call in the future and pass it the cleanup flag maybe, but we can only do that with a version bump | |
| 17:15:46 | bauzas | all of that is in _post_live_migrate | |
| 17:15:57 | dansmith | else we will ask n-1 computes to do a cleanup when there is none to do | |
| 17:16:41 | gmaan | its during cleanup of pre live migration not post | |
| 17:16:56 | dansmith | right | |
| 17:17:21 | dansmith | well, during cleanup of live migration... to clean up what was started in pre_live_migration | |
| 17:17:25 | bauzas | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 17:17:33 | gmaan | dansmith: yeah, that is my preference too when i proposed change, if we can delay this bug and mentioned as known bug and in part-2 we can handle by flag or new call | |
| 17:17:57 | gmaan | dansmith: yes -> 'to clean up what was started in pre_live_migration' | |
| 17:18:08 | dansmith | bauzas: just to be clear, I'm looking for opinions on if we can/should bump the compute RPC to add this call/param right now, just before rc1 | |
| 17:19:34 | bauzas | we're changing the live-migration workflow per se, not exactly the RPC interface | |
| 17:19:48 | bauzas | so that's kind of a grey area | |
| 17:20:03 | bauzas | but in case of a rolling upgrade, we have to ensure we're not breaking for sure | |
| 17:20:18 | dansmith | huh? | |
| 17:20:24 | bauzas | because old compute could expect a different workflow | |
| 17:20:27 | dansmith | it's 100% an RPC change :) | |
| 17:21:08 | gmaan | yeah, old dest will always do the rollback at dest when not needed and most probably fail | |
| 17:21:15 | bauzas | we don't change the RPC parameters for sure, but we changing the callers yes | |
| 17:21:17 | dansmith | not only are we expecting the new behavior of canceling the task, we need to be passing a parameter to instruct the old side not to try to clean up something it shouldn't be cleaning up | |
| 17:21:33 | dansmith | bauzas: we _need_ to add a parameter | |
| 17:21:51 | bauzas | lemme explain it differently | |
| 17:21:53 | gmaan | dansmith: flag can be avoided as as current code change where dest can calculate the cleanup is required or not | |
| 17:21:57 | dansmith | right now this will just generate (we think) a traceback | |
| 17:22:31 | bauzas | either we consider the existing patch good and then I have a concern due to rolling upgrades (old computes may expect the calls differently), or we assume we need to pass a new param and then I have concern witth a late RPC change | |
| 17:23:03 | bauzas | either way, I'm not happy with such behavioural change so close to the release | |
| 17:23:03 | dansmith | gmaan: but the flag also gives us the indication that we're making the new call so we can avoid doing it on the older versioon | |
| 17:23:48 | dansmith | bauzas: I don't think I'm okay with no bump and either change | |
| 17:23:58 | dansmith | it's very clearly just side-stepping RPC behavioral expectations | |
| 17:24:11 | gmaan | dansmith: that can but we can just check the same based on pin version and cleanup flag calculation. I mean flag is better way but if we want to avoid RPC signature change that is possible | |
| 17:25:06 | bauzas | ship has sailed for a RPC bump this release https://review.opendev.org/c/openstack/nova/+/1004424 | |
| 17:25:12 | dansmith | gmaan: that would be compute manager inspecting the pin set in the rpcapi right? | |
| 17:25:39 | bauzas | dansmith: that was my "either" : the existing proposal has upgrade concners | |
| 17:26:05 | dansmith | bauzas: that hasn't merged so I don't think it has _sailed_ nor do I think landing that is terminal until we make the release :) | |
| 17:26:42 | bauzas | can't we just defer the fix to post-rc1 ? | |
| 17:27:10 | dansmith | meaning have this be broken in H? we can for sure, that's one of the options | |
| 17:27:14 | gmaan | dansmith: ah yeah. i get your point now | |
| 17:27:19 | gmaan | so what is main concern on stating it as known bug which is in scenario 1. live migration started at pre migration stage, source rolling it back , it is shared storage where cleanup not needed at dest 2. dest shutting down. Issue: it will hold dest graceful shutdown for full time as configured (manager_shutdown_time default 160 sec) | |
| 17:28:03 | dansmith | gmaan: yep, that's one option and was also something I think is worth surveying opinions on | |
| 17:28:04 | gmaan | only issue it will cause is to make shutdown taking full time instead of having task tracking benefits in this scenario | |
| 17:28:14 | dansmith | yup | |
| 17:28:17 | gmaan | which is nothing change than before this rlease | |
| 17:28:42 | gmaan | bcz I am not not 100% ok to bump RPC version or try some workaround which we need to chaneg again | |
| 17:28:52 | gmaan | *I am also not | |
| 17:29:19 | bauzas | because we're leaking threads as of now ? | |
| 17:29:47 | dansmith | not leaking threads really.. started a task that will wait for $timeout before allowing n-cpu to exit because it thinks a migration will still be coming | |
| 17:29:56 | bauzas | I see | |
| 17:29:59 | gmaan | bauzas: bcz we are leaking task from task tracking, live migration task started at dest willnot be ended as source do rollbacl and do not call dest | |
| 17:30:10 | dansmith | but it's very rare that this would be hit, and if you don't enable (or do disable) graceful shutdown it won't matter anyway | |
| 17:30:25 | bauzas | I think the stable policy still allows us to backport an RPC change provided certain constraints, but I can doublechecks | |
| 17:30:29 | bauzas | doublecheck* | |
| 17:30:29 | dansmith | so maybe just merge a reno with known-bug | |
| 17:30:42 | dansmith | bauzas: we can but it is majorly less good to do that than just merge it now | |