| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-09 | |||
| 17:20:24 | bauzas | because old compute could expect a different workflow | |
| 17:20:27 | dansmith | it's 100% an RPC change :) | |
| 17:21:08 | gmaan | yeah, old dest will always do the rollback at dest when not needed and most probably fail | |
| 17:21:15 | bauzas | we don't change the RPC parameters for sure, but we changing the callers yes | |
| 17:21:17 | dansmith | not only are we expecting the new behavior of canceling the task, we need to be passing a parameter to instruct the old side not to try to clean up something it shouldn't be cleaning up | |
| 17:21:33 | dansmith | bauzas: we _need_ to add a parameter | |
| 17:21:51 | bauzas | lemme explain it differently | |
| 17:21:53 | gmaan | dansmith: flag can be avoided as as current code change where dest can calculate the cleanup is required or not | |
| 17:21:57 | dansmith | right now this will just generate (we think) a traceback | |
| 17:22:31 | bauzas | either we consider the existing patch good and then I have a concern due to rolling upgrades (old computes may expect the calls differently), or we assume we need to pass a new param and then I have concern witth a late RPC change | |
| 17:23:03 | dansmith | gmaan: but the flag also gives us the indication that we're making the new call so we can avoid doing it on the older versioon | |
| 17:23:03 | bauzas | either way, I'm not happy with such behavioural change so close to the release | |
| 17:23:48 | dansmith | bauzas: I don't think I'm okay with no bump and either change | |
| 17:23:58 | dansmith | it's very clearly just side-stepping RPC behavioral expectations | |
| 17:24:11 | gmaan | dansmith: that can but we can just check the same based on pin version and cleanup flag calculation. I mean flag is better way but if we want to avoid RPC signature change that is possible | |
| 17:25:06 | bauzas | ship has sailed for a RPC bump this release https://review.opendev.org/c/openstack/nova/+/1004424 | |
| 17:25:12 | dansmith | gmaan: that would be compute manager inspecting the pin set in the rpcapi right? | |
| 17:25:39 | bauzas | dansmith: that was my "either" : the existing proposal has upgrade concners | |
| 17:26:05 | dansmith | bauzas: that hasn't merged so I don't think it has _sailed_ nor do I think landing that is terminal until we make the release :) | |
| 17:26:42 | bauzas | can't we just defer the fix to post-rc1 ? | |
| 17:27:10 | dansmith | meaning have this be broken in H? we can for sure, that's one of the options | |
| 17:27:14 | gmaan | dansmith: ah yeah. i get your point now | |
| 17:27:19 | gmaan | so what is main concern on stating it as known bug which is in scenario 1. live migration started at pre migration stage, source rolling it back , it is shared storage where cleanup not needed at dest 2. dest shutting down. Issue: it will hold dest graceful shutdown for full time as configured (manager_shutdown_time default 160 sec) | |
| 17:28:03 | dansmith | gmaan: yep, that's one option and was also something I think is worth surveying opinions on | |
| 17:28:04 | gmaan | only issue it will cause is to make shutdown taking full time instead of having task tracking benefits in this scenario | |
| 17:28:14 | dansmith | yup | |
| 17:28:17 | gmaan | which is nothing change than before this rlease | |
| 17:28:42 | gmaan | bcz I am not not 100% ok to bump RPC version or try some workaround which we need to chaneg again | |
| 17:28:52 | gmaan | *I am also not | |
| 17:29:19 | bauzas | because we're leaking threads as of now ? | |
| 17:29:47 | dansmith | not leaking threads really.. started a task that will wait for $timeout before allowing n-cpu to exit because it thinks a migration will still be coming | |
| 17:29:56 | bauzas | I see | |
| 17:29:59 | gmaan | bauzas: bcz we are leaking task from task tracking, live migration task started at dest willnot be ended as source do rollbacl and do not call dest | |
| 17:30:10 | dansmith | but it's very rare that this would be hit, and if you don't enable (or do disable) graceful shutdown it won't matter anyway | |
| 17:30:25 | bauzas | I think the stable policy still allows us to backport an RPC change provided certain constraints, but I can doublechecks | |
| 17:30:29 | dansmith | so maybe just merge a reno with known-bug | |
| 17:30:29 | bauzas | doublecheck* | |
| 17:30:42 | dansmith | bauzas: we can but it is majorly less good to do that than just merge it now | |
| 17:30:50 | dansmith | I think if we don't fix now, we won't fix in stable | |
| 17:31:58 | bauzas | I see your point, I don't have a particular opinion about it except that I don't want the existing patch to be merged without testing rolling upgrades | |
| 17:32:02 | gmaan | yeah, fix would not be backportable | |
| 17:33:32 | bauzas | we don't have a job that runs live-migration on a multinode grenade env, right? | |
| 17:34:41 | dansmith | I need to take a break | |
| 17:34:52 | dansmith | I think I'm most in favor of just #knownbug'ing it and fix it in I | |
| 17:35:32 | dansmith | if it absolutely explodes in our face, we can backport something like the current patch (or offer it to people out of band) but I doubt it will | |
| 17:35:37 | gmaan | dansmith: I have one more solution which does not need RPC bump or any ordering change of cleaup | |
| 17:36:03 | bauzas | https://zuul.opendev.org/t/openstack/job/nova-grenade-multinode is testing live-migration on a rolling upgrade fashion but we don't run graceful shutdown on it | |
| 17:36:40 | gmaan | dansmith: so this rollback at sourec happening after pre_live_migration at dest and where dest raise any error. and task is leaked beacuxse we start the live migration at dest task before dest's pre_live_migration | |
| 17:37:18 | gmaan | what if we start the live-migration_at_dest task after dest's pre_live_migration is completed so that this rollback at source will not leak the task at dest | |
| 17:37:49 | gmaan | I mean record lve migration at dest when we know it is actually going to start and not pre live migration failed | |
| 17:38:13 | dansmith | I dunno, that seems like a lot more change from what we have now and I'm probably too fried to think about it | |
| 17:38:38 | dansmith | I also thought about making the task on the dest just poll the migration object while it's waiting, but that is more work so late as well | |
| 17:39:07 | gmaan | I mean we are starting live migration at dest too early even we do not know if dest's pre live migration things are successful or not | |
| 17:39:36 | gmaan | poll task where and how to decide when to end it? | |
| 17:40:17 | gmaan | dansmith: for 'late' yes, but this sems backportable fix if we do it in next release right? | |
| 17:40:33 | dansmith | poll task in an executor task, end it when the migration goes to finished or error.. it was just a thought and something we'd have to look at | |
| 17:40:41 | dansmith | gmaan: potentially | |
| 17:41:51 | gmaan | dansmith: ohk so based on the migration status itself. yeah that is one possible way | |
| 18:11:46 | bauzas | I had to move errand, but I'd indeed prefer the third solution, which is to end the task when the migration finishes, instead of overloading our RPC interface for that need | |
| 18:27:07 | gmaan | yeah, i am finding that solution a better one. My solution of shifting the start_task is not perfect and does not solve issue when dest's pre live migration is successful but vif plugged event can cause rollback on source - | |
| 18:34:30 | gmaan | dansmith: bauzas i commented in gerrit about going for 'listing it as a known bug' path (please check), we can wait for gibi on his opinion and i can proceed accordingly | |
| 20:31:52 | opendevreview | ribaudr proposed openstack/nova master: Add Hibiscus prelude section https://review.opendev.org/c/openstack/nova/+/1004927 | |
| #openstack-nova - 2026-09-10 | |||
| 06:33:56 | simondodsley | gibi: Is there any chance that this patch (https://review.opendev.org/c/openstack/nova/+/973750) be relooked at again. It's a really simple fix for a big issue. Already have one +2 | |
| 07:32:56 | gibi | simondodsley: I've approved it. I hope I did not risk RC1 with it. But I'm in a good mood so I taking my chances here. Btw next time feel free to ping me eariler if I reviewed someting before and never showed up re-reviewing the patch | |
| 09:15:26 | bauzas | gibi: would you agree with gmaan's comment on https://review.opendev.org/c/openstack/nova/+/1003386 that we go for a known bug reno instead of fixing it ? | |
| 09:15:57 | bauzas | I'm cool with that and if you look at the irc logs we discussed a nice alternative | |
| 09:53:06 | gibi | let me check | |
| 09:59:07 | gibi | I'v read the review comments. I'm OK to keep as a known bug for now. I agree that it is not fatal thing to have this task leaked as it only slows down the shutdown, does not prevent it | |
| 10:01:20 | gibi | Uggla: thanks I will jump there | |
| 10:02:24 | gibi | but I also need to read the tc channel logs to get the full context | |
| 10:42:57 | bauzas | given the lack of consensus yet, I'd propose to defer the application of nova-reviewers until a consequent discussion can happen (before PTG or in it) for at least being all of up agreeing for find some policy how to use this new group | |
| 12:23:14 | sean-k-mooney | ls | |
| 12:23:38 | sean-k-mooney | :) | |
| 12:25:30 | sean-k-mooney | gibi: gmaan: https://review.opendev.org/c/openstack/nova/+/1004687/1 and https://review.opendev.org/c/openstack/nova/+/966479/5 update nova next to ubuntu 26.04 and nova-alt-config to to debian 13, the latter was ment ot be done in 2026.1 so it woudl be nice if we can proceed with both this cycle, can ye take a look when ye have time | |
| 12:51:51 | gibi | Uggla: thanks for the email, I responded there and commented on the https://review.opendev.org/c/openstack/nova/+/986141 as well | |
| 12:52:01 | gibi | we need to start figure out a the way forward | |
| 13:04:40 | opendevreview | Kamil Sambor proposed openstack/nova master: Fix pickling error that forces oslo.service to fall back to fork https://review.opendev.org/c/openstack/nova/+/1000956 | |
| 13:06:12 | gablan | hi gibi! I'm backporting the graceful shutdown work from master to stable/2026.1 downstream, and I hit the TODO you left in Host.cleanup() in `nova/virt/libvirt/host.py` about `stopping _event_handler` and `the _conn_event_thread` loop. do you have anything planned for it, or can I propose a patch for it? | |
| 13:17:35 | gibi | gablan: check master I think gmaan recent changes there are addressing it | |
| 13:19:34 | gibi | sean-k-mooney: are you sure this is save to do on the day of RC1? If yes then we can land it but I need you to react and revert if something breaks on the gate after landing | |
| 13:19:40 | gibi | /save/safe/ | |
| 13:19:59 | gibi | cc gmaan, Uggla | |
| 13:20:18 | sean-k-mooney | so i think so but im ok to wait until after rc1 is cut and backport if reqruied. | |
| 13:20:24 | sean-k-mooney | i coudl also flip the order | |
| 13:20:35 | sean-k-mooney | the debian 12->13 change for alt config | |
| 13:20:52 | sean-k-mooney | shoudl be safe because our ceph jobs have been on debina 13 all cycle | |
| 13:21:24 | sean-k-mooney | we just didnt get aroudn to movaing nova-alt-conig last cycel or eailar becasue i forgot the patch was still open/in merge conflict | |
| 13:21:56 | gibi | when gmaan is up we can make the call I think | |
| 13:22:18 | gibi | I will be around for longer today to accomodate that (and due to other unrelated reasons) | |
| 13:22:25 | sean-k-mooney | unless we have a strong reason not to i woudl perfer to have some py3.14 testign on stable 2026.2 but im ok if perpoel prefer not too | |
| 13:22:35 | sean-k-mooney | ack works for me | |
| 13:39:50 | dansmith | my reasoning would be that we didn't go all of the cycle with that testing on, so flipping it right now seems much riskier than necessary | |
| 14:02:37 | bauzas | is it me or the level of changes we plan to deliver so close to RC1 is high today ? | |
| 14:03:14 | bauzas | as a reminder, we can make RC2 if there are some regressions that need to be fixed asap | |
| 14:04:00 | bauzas | but on the graceful shutdown case, I think we agreed yesterday to just amend the existing relnotes with a known bug file so people would know about the existing and decide whether to opt-in or not | |
| 14:05:14 | opendevreview | ribaudr proposed openstack/nova master: Add Hibiscus prelude section https://review.opendev.org/c/openstack/nova/+/1004927 | |
| 14:08:54 | Uggla | bauzas, gibi just to let you know I have updated the highlight with latest input from gmann/bauzas https://review.opendev.org/c/openstack/releases/+/1004465 also reflect that change to the prelude patch https://review.opendev.org/c/openstack/nova/+/1004927 | |
| 14:09:07 | bauzas | ack | |
| 14:09:50 | Uggla | and sorry it comes later than I what I expected. | |
| 14:10:33 | dansmith | bauzas: what else is proposed? we agreed to punt on the graceful shutdown one for sure | |