Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-09
17:26:05 dansmith bauzas: that hasn't merged so I don't think it has _sailed_ nor do I think landing that is terminal until we make the release :)
17:26:42 bauzas can't we just defer the fix to post-rc1 ?
17:27:10 dansmith meaning have this be broken in H? we can for sure, that's one of the options
17:27:14 gmaan dansmith: ah yeah. i get your point now
17:27:19 gmaan so what is main concern on stating it as known bug which is in scenario 1. live migration started at pre migration stage, source rolling it back , it is shared storage where cleanup not needed at dest 2. dest shutting down. Issue: it will hold dest graceful shutdown for full time as configured (manager_shutdown_time default 160 sec)
17:28:03 dansmith gmaan: yep, that's one option and was also something I think is worth surveying opinions on
17:28:04 gmaan only issue it will cause is to make shutdown taking full time instead of having task tracking benefits in this scenario
17:28:14 dansmith yup
17:28:17 gmaan which is nothing change than before this rlease
17:28:42 gmaan bcz I am not not 100% ok to bump RPC version or try some workaround which we need to chaneg again
17:28:52 gmaan *I am also not
17:29:19 bauzas because we're leaking threads as of now ?
17:29:47 dansmith not leaking threads really.. started a task that will wait for $timeout before allowing n-cpu to exit because it thinks a migration will still be coming
17:29:56 bauzas I see
17:29:59 gmaan bauzas: bcz we are leaking task from task tracking, live migration task started at dest willnot be ended as source do rollbacl and do not call dest
17:30:10 dansmith but it's very rare that this would be hit, and if you don't enable (or do disable) graceful shutdown it won't matter anyway
17:30:25 bauzas I think the stable policy still allows us to backport an RPC change provided certain constraints, but I can doublechecks
17:30:29 bauzas doublecheck*
17:30:29 dansmith so maybe just merge a reno with known-bug
17:30:42 dansmith bauzas: we can but it is majorly less good to do that than just merge it now
17:30:50 dansmith I think if we don't fix now, we won't fix in stable
17:31:58 bauzas I see your point, I don't have a particular opinion about it except that I don't want the existing patch to be merged without testing rolling upgrades
17:32:02 gmaan yeah, fix would not be backportable
17:33:32 bauzas we don't have a job that runs live-migration on a multinode grenade env, right?
17:34:41 dansmith I need to take a break
17:34:52 dansmith I think I'm most in favor of just #knownbug'ing it and fix it in I
17:35:32 dansmith if it absolutely explodes in our face, we can backport something like the current patch (or offer it to people out of band) but I doubt it will
17:35:37 gmaan dansmith: I have one more solution which does not need RPC bump or any ordering change of cleaup
17:36:03 bauzas https://zuul.opendev.org/t/openstack/job/nova-grenade-multinode is testing live-migration on a rolling upgrade fashion but we don't run graceful shutdown on it
17:36:40 gmaan dansmith: so this rollback at sourec happening after pre_live_migration at dest and where dest raise any error. and task is leaked beacuxse we start the live migration at dest task before dest's pre_live_migration
17:37:18 gmaan what if we start the live-migration_at_dest task after dest's pre_live_migration is completed so that this rollback at source will not leak the task at dest
17:37:49 gmaan I mean record lve migration at dest when we know it is actually going to start and not pre live migration failed
17:38:13 dansmith I dunno, that seems like a lot more change from what we have now and I'm probably too fried to think about it
17:38:38 dansmith I also thought about making the task on the dest just poll the migration object while it's waiting, but that is more work so late as well
17:39:07 gmaan I mean we are starting live migration at dest too early even we do not know if dest's pre live migration things are successful or not
17:39:36 gmaan poll task where and how to decide when to end it?
17:40:17 gmaan dansmith: for 'late' yes, but this sems backportable fix if we do it in next release right?
17:40:33 dansmith poll task in an executor task, end it when the migration goes to finished or error.. it was just a thought and something we'd have to look at
17:40:41 dansmith gmaan: potentially
17:41:51 gmaan dansmith: ohk so based on the migration status itself. yeah that is one possible way
18:11:46 bauzas I had to move errand, but I'd indeed prefer the third solution, which is to end the task when the migration finishes, instead of overloading our RPC interface for that need
18:27:07 gmaan yeah, i am finding that solution a better one. My solution of shifting the start_task is not perfect and does not solve issue when dest's pre live migration is successful but vif plugged event can cause rollback on source -
18:34:30 gmaan dansmith: bauzas i commented in gerrit about going for 'listing it as a known bug' path (please check), we can wait for gibi on his opinion and i can proceed accordingly
20:31:52 opendevreview ribaudr proposed openstack/nova master: Add Hibiscus prelude section https://review.opendev.org/c/openstack/nova/+/1004927
#openstack-nova - 2026-09-10
06:33:56 simondodsley gibi: Is there any chance that this patch (https://review.opendev.org/c/openstack/nova/+/973750) be relooked at again. It's a really simple fix for a big issue. Already have one +2
07:32:56 gibi simondodsley: I've approved it. I hope I did not risk RC1 with it. But I'm in a good mood so I taking my chances here. Btw next time feel free to ping me eariler if I reviewed someting before and never showed up re-reviewing the patch
09:15:26 bauzas gibi: would you agree with gmaan's comment on https://review.opendev.org/c/openstack/nova/+/1003386 that we go for a known bug reno instead of fixing it ?
09:15:57 bauzas I'm cool with that and if you look at the irc logs we discussed a nice alternative
09:53:06 gibi let me check
09:59:07 gibi I'v read the review comments. I'm OK to keep as a known bug for now. I agree that it is not fatal thing to have this task leaked as it only slows down the shutdown, does not prevent it
10:01:20 gibi Uggla: thanks I will jump there
10:02:24 gibi but I also need to read the tc channel logs to get the full context
10:42:57 bauzas given the lack of consensus yet, I'd propose to defer the application of nova-reviewers until a consequent discussion can happen (before PTG or in it) for at least being all of up agreeing for find some policy how to use this new group
12:23:14 sean-k-mooney ls
12:23:38 sean-k-mooney :)
12:25:30 sean-k-mooney gibi: gmaan: https://review.opendev.org/c/openstack/nova/+/1004687/1 and https://review.opendev.org/c/openstack/nova/+/966479/5 update nova next to ubuntu 26.04 and nova-alt-config to to debian 13, the latter was ment ot be done in 2026.1 so it woudl be nice if we can proceed with both this cycle, can ye take a look when ye have time
12:51:51 gibi Uggla: thanks for the email, I responded there and commented on the https://review.opendev.org/c/openstack/nova/+/986141 as well
12:52:01 gibi we need to start figure out a the way forward
13:04:40 opendevreview Kamil Sambor proposed openstack/nova master: Fix pickling error that forces oslo.service to fall back to fork https://review.opendev.org/c/openstack/nova/+/1000956
13:06:12 gablan hi gibi! I'm backporting the graceful shutdown work from master to stable/2026.1 downstream, and I hit the TODO you left in Host.cleanup() in `nova/virt/libvirt/host.py` about `stopping _event_handler` and `the _conn_event_thread` loop. do you have anything planned for it, or can I propose a patch for it?
13:17:35 gibi gablan: check master I think gmaan recent changes there are addressing it
13:19:34 gibi sean-k-mooney: are you sure this is save to do on the day of RC1? If yes then we can land it but I need you to react and revert if something breaks on the gate after landing
13:19:40 gibi /save/safe/
13:19:59 gibi cc gmaan, Uggla
13:20:18 sean-k-mooney so i think so but im ok to wait until after rc1 is cut and backport if reqruied.
13:20:24 sean-k-mooney i coudl also flip the order
13:20:35 sean-k-mooney the debian 12->13 change for alt config
13:20:52 sean-k-mooney shoudl be safe because our ceph jobs have been on debina 13 all cycle
13:21:24 sean-k-mooney we just didnt get aroudn to movaing nova-alt-conig last cycel or eailar becasue i forgot the patch was still open/in merge conflict
13:21:56 gibi when gmaan is up we can make the call I think
13:22:18 gibi I will be around for longer today to accomodate that (and due to other unrelated reasons)
13:22:25 sean-k-mooney unless we have a strong reason not to i woudl perfer to have some py3.14 testign on stable 2026.2 but im ok if perpoel prefer not too
13:22:35 sean-k-mooney ack works for me
13:39:50 dansmith my reasoning would be that we didn't go all of the cycle with that testing on, so flipping it right now seems much riskier than necessary
14:02:37 bauzas is it me or the level of changes we plan to deliver so close to RC1 is high today ?
14:03:14 bauzas as a reminder, we can make RC2 if there are some regressions that need to be fixed asap
14:04:00 bauzas but on the graceful shutdown case, I think we agreed yesterday to just amend the existing relnotes with a known bug file so people would know about the existing and decide whether to opt-in or not
14:05:14 opendevreview ribaudr proposed openstack/nova master: Add Hibiscus prelude section https://review.opendev.org/c/openstack/nova/+/1004927
14:08:54 Uggla bauzas, gibi just to let you know I have updated the highlight with latest input from gmann/bauzas https://review.opendev.org/c/openstack/releases/+/1004465 also reflect that change to the prelude patch https://review.opendev.org/c/openstack/nova/+/1004927
14:09:07 bauzas ack
14:09:50 Uggla and sorry it comes later than I what I expected.
14:10:33 dansmith bauzas: what else is proposed? we agreed to punt on the graceful shutdown one for sure
14:12:08 bauzas dansmith: I maybe misunderstood the above discussion, I thought two different things were discussed, the one about graceful shutdown leaking threads and the other one being a nova-next change
14:13:06 dansmith bauzas: oh yeah, the shutdown one is punted, and I agree, the nova-next change should wait.. I thought you meant something else
14:17:52 gibi Uggla: ack
14:37:09 gibi bauzas: this needs second code (max microversion) https://review.opendev.org/c/openstack/nova/+/1004872
14:37:18 gibi s/code/core/
14:37:22 bauzas oh I forgot that one
14:39:03 gibi Uggla: bauzas: what should we do with translation? https://review.opendev.org/c/openstack/nova/+/980544
14:39:44 bauzas we ship them by a single approver
14:40:05 gibi OK then I approve it
14:40:15 gibi ohh you did it good
14:40:15 bauzas I did it
14:40:16 gibi thanks
14:40:19 bauzas nw
14:40:33 bauzas and fwiw the i18n team works on a change from zanata
14:40:38 bauzas I dunno where they are
14:48:06 opendevreview Gabin Lanore proposed openstack/nova master: libvirt: stop the event dispatch loops on cleanup https://review.opendev.org/c/openstack/nova/+/1005022
14:50:20 gablan gibi: thanks! Checked both of gmaan's patches 1003386 is the task-registry leak, 1001572 is the executor ownership refactor. Neither seems to touch Host.cleanup(), so I pushed a patch for it :D
14:51:56 gibi gablan: thanks cc gmaan ^^

Earlier   Later