| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-01 | |||
| 16:22:35 | gmaan | the live migration task left is I agree is the production bug but I am talking about hthttps://bugs.launchpad.net/nova/+bug/2165991 though it is because of migration task hanging | |
| 16:22:37 | gibi | similarly the dest nova-compute will not exit until 180sec as it thinks there is an in progress live migration | |
| 16:22:48 | gmaan | I think I mixed both in my question | |
| 16:23:05 | gibi | I think that two things are the same :) | |
| 16:23:10 | gibi | feel free to rename the bug title | |
| 16:23:16 | gibi | to talk about the leaked task | |
| 16:23:24 | gibi | the hanging functional test is just a symptom | |
| 16:23:26 | gmaan | gibi: my question was if anything we need to fix on shutdown_thread creation and when it fail/timeout | |
| 16:23:33 | gmaan | gibi: agree on that | |
| 16:24:27 | gmaan | I am saying this part is all ok https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/service.py#L433-L435 | |
| 16:24:45 | gmaan | and yes we need to fix that task leak | |
| 16:25:05 | gibi | it is a trap for functional test due to the thread can leak, but I can cirumwent that with setting a non 0 timeout to wait for that thread | |
| 16:26:52 | gibi | and when I set that timeout to 300 I saw the task leak in 3 functional tests I reported as the bug | |
| 16:27:21 | gmaan | ack | |
| 16:28:46 | gibi | also now I see that setting 300sec timeout also make a bunch of test sad in the eventlet job | |
| 16:29:32 | gibi | so we might have more leaks of we might have differently hanging shutdown in eventlet job | |
| 16:29:49 | gibi | https://review.opendev.org/c/openstack/nova/+/1003266/1 | |
| 16:42:06 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 16:42:20 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 16:56:09 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 17:09:03 | Uggla | dansmith, melwitt, gouthamr comment about the virtiofs migration and FFE: https://review.opendev.org/c/openstack/nova/+/1001916/comments/965cff03_8b75df37 | |
| 17:09:52 | gouthamr | \o/ ty Uggla | |
| 17:11:44 | Uggla | gouthamr, dansmith, melwitt I think Melanie is on PTO until tomorrow. I hope Melanie could jump on this review as soon as she will be back. | |
| 17:13:31 | Uggla | gouthamr, dansmith, melwitt also forget to mention that the msg above contains a short summary of the tests I did. | |
| 17:14:51 | gouthamr | ++ i read them, ty for testing those scenarios. I will try and add a multi-node job with cephfs to the manila gate, and one with LVM/NFS to the nova gate (and spread the test coverage between projects) | |
| 17:16:40 | Uggla | gouthamr, it will be great. Just to note I also tried several way to break it, and it looks solid so far. | |
| 17:41:29 | gouthamr | Uggla: ++ :) ty! | |
| 17:56:34 | gmaan | gibi: for eventlet functional job, you mean tox cover jobs or nova-tox-functional-py313 or both? bth failing though but I am seeing only one failure in nova-tox-functional-py313 | |
| 19:33:33 | melwitt | Uggla: thanks for adding the comment, I will be reading over everything | |
| 20:16:19 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900 | |
| 20:17:16 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 21:14:56 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 21:16:39 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 21:21:39 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix task leak for live migration rollback case https://review.opendev.org/c/openstack/nova/+/1003386 | |
| #openstack-nova - 2026-09-02 | |||
| 02:02:31 | gmaan | gibi: sean-k-mooney : this is fix for the live migration task leaking https://review.opendev.org/c/openstack/nova/+/1003386 | |
| 05:19:56 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 06:12:01 | opendevreview | Adam McArthur proposed openstack/nova master: Use safe allocation deletion on instance delete https://review.opendev.org/c/openstack/nova/+/1003208 | |
| 10:02:07 | gibi | can I get an core eye on https://review.opendev.org/c/openstack/nova/+/1003245 it is a functional test stabilization | |
| 10:12:32 | sean-k-mooney | so the intet is to set it to a largere value later but we are settign it to 10 for now so that if we do leak it does not cause the run to fail | |
| 10:15:58 | sean-k-mooney | i see in https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/functional/regressions/test_bug_1888395.py your bumping it to 300 and hten claping it to 10 on the tests that have know issues | |
| 10:16:20 | sean-k-mooney | any that are leaking a thread will timeout and you can clamp them until its fixed | |
| 10:17:54 | sean-k-mooney | gibi: we can technially start deleting evnetlet in like a weak to eliminate those leaks :) | |
| 10:19:47 | sean-k-mooney | gibi: by the way did you notice that the threading functional test now run in half the time of the eventlet ones | |
| 10:20:18 | sean-k-mooney | oh nope | |
| 10:20:30 | sean-k-mooney | thats the non funtionla treadign tests | |
| 10:21:49 | gibi | sean-k-mooney: we cannot delete them until post 2027.1 | |
| 10:22:04 | sean-k-mooney | we technially can | |
| 10:22:26 | sean-k-mooney | but i know we are givign a very long tail in nova | |
| 10:22:29 | gibi | sc-tmp-lighbits-rhoso-certification | |
| 10:22:33 | gibi | https://governance.openstack.org/tc/goals/selected/remove-eventlet.html#completion-criteria | |
| 10:22:56 | gibi | u(2027.1) Deprecate the Eventlet support. To have a SLURP target release supporting both the old Eventlet and any new concurrency mode, this release keeps the Eventlet support. | |
| 10:23:08 | sean-k-mooney | those are the letes tg we can do those | |
| 10:23:22 | sean-k-mooney | not the first time for each milestone | |
| 10:23:52 | sean-k-mooney | several service just remove eventlet with no opertunity to have a singel slurp release that ran both | |
| 10:24:18 | gibi | but I think what is written in the goal is there because we wanted to do it that way :) | |
| 10:24:52 | sean-k-mooney | we did im just saying if it continue to be a maintiance isseu we can change that planb | |
| 10:24:57 | sean-k-mooney | on the ohter hand | |
| 10:25:08 | sean-k-mooney | if the eventlet test find leaks that we dont see in threaded | |
| 10:25:19 | sean-k-mooney | we probably shoudl fix those anyway | |
| 10:25:50 | gibi | yeah I haven't looked into the eventlet hangs in that patch | |
| 10:25:53 | gibi | but I will | |
| 10:47:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452 | |
| 10:47:48 | gibi | sean-k-mooney: bauzas: ^^ another functional stabilization patch | |
| 13:21:29 | opendevreview | Balazs Gibizer proposed openstack/nova master: [functional]Do not patch req-id generation https://review.opendev.org/c/openstack/nova/+/1003024 | |
| 13:21:45 | gibi | cores this is another stabilization ^^ | |
| 14:16:56 | stephenfin | sean-k-mooney: did you already bring up https://review.opendev.org/c/openstack/releases/+/1003257? | |
| 14:17:21 | stephenfin | https://review.opendev.org/c/openstack/releases/+/1003311 is blocked by that | |
| 14:32:41 | sean-k-mooney | sorry no Uggla bauzas ^ we need to release os-vif again | |
| 14:33:13 | sean-k-mooney | elodilles: ^ fyi | |
| 14:33:45 | Uggla | sean-k-mooney ok I have seen the patches | |
| 14:41:49 | elodilles | sean-k-mooney: nova team needs to agree on FFE and an RFE needs to be requested from requirements team, otherwise LGTM | |
| 14:48:38 | sean-k-mooney | the config bug is a reguress intoduced in this cycle os i think it would qualify for a FFE adn enve after RC1 but idealy we do it before we do it | |
| 14:52:25 | gibi | I'm OK from nova perspective to accept this os_vif fix as FFE | |
| 14:52:33 | gibi | https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/fixtures/conf.py#73 | |
| 14:52:36 | gibi | this is being the fix | |
| 15:37:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: [DNM]reproduce "database is locked" error https://review.opendev.org/c/openstack/nova/+/1003554 | |
| 15:38:51 | gibi | sean-k-mooney: ^^ finally after 2days and many deep rabbit holes this reproduces the DB locking issue pretty regularly to me locally | |
| 15:39:56 | gibi | there is something called checkpointing the WAL | |
| 15:40:08 | gibi | that happens after the last DB connection is closed | |
| 15:40:19 | gibi | happens outside of the db context manager | |
| 15:40:27 | gibi | and sqlite takes a DB write lock for it | |
| 15:41:06 | gibi | as we constantly open and close DB connections there is plenty of, very short, window when there is a last-connection-closed situation | |
| 15:41:40 | gibi | and if we are unlucky the next connection is a write transaction that hits within the window of the WAL checkpointing and seeing the DB as locked | |
| 15:42:11 | gmaan | gibi: https://review.opendev.org/c/openstack/nova/+/1003386 is green to fix the leaking task. can you rebase your manager_shutdown_timeout=300 change to this and we can see what all are fixed with this | |
| 15:42:20 | gibi | gmaan: on my list still for today | |
| 15:42:30 | gmaan | thanks | |
| 15:42:56 | melwitt | so that per something lock change did not help huh? | |
| 15:44:49 | gibi | melwitt: it helps | |
| 15:44:55 | gibi | we need to serialize writes | |
| 15:44:56 | gibi | but | |
| 15:45:08 | melwitt | oh ok | |
| 15:45:09 | gibi | there is this secondary case when sqlite takes a DB write lock | |
| 15:45:19 | gibi | outside of a write transaction | |
| 15:45:34 | gibi | the case being, the last connection to the DB is closed. | |
| 15:45:51 | gibi | so the immedate fix is, do not allow such situation to happen :) | |
| 15:46:04 | gibi | i.e. keep a connection open during the test | |
| 15:46:34 | melwitt | ah, yeah. that's how the in-memory db worked so it didn't come up in the past | |
| 15:46:56 | gibi | yes in memory DB used a StaticPool to have only a single connection to a DB all the time | |
| 15:47:08 | gibi | the file based uses a NullPool | |
| 15:47:08 | melwitt | right | |