Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-01
17:09:03 Uggla dansmith, melwitt, gouthamr comment about the virtiofs migration and FFE: https://review.opendev.org/c/openstack/nova/+/1001916/comments/965cff03_8b75df37
17:09:52 gouthamr \o/ ty Uggla
17:11:44 Uggla gouthamr, dansmith, melwitt I think Melanie is on PTO until tomorrow. I hope Melanie could jump on this review as soon as she will be back.
17:13:31 Uggla gouthamr, dansmith, melwitt also forget to mention that the msg above contains a short summary of the tests I did.
17:14:51 gouthamr ++ i read them, ty for testing those scenarios. I will try and add a multi-node job with cephfs to the manila gate, and one with LVM/NFS to the nova gate (and spread the test coverage between projects)
17:16:40 Uggla gouthamr, it will be great. Just to note I also tried several way to break it, and it looks solid so far.
17:41:29 gouthamr Uggla: ++ :) ty!
17:56:34 gmaan gibi: for eventlet functional job, you mean tox cover jobs or nova-tox-functional-py313 or both? bth failing though but I am seeing only one failure in nova-tox-functional-py313
19:33:33 melwitt Uggla: thanks for adding the comment, I will be reading over everything
20:16:19 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900
20:17:16 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349
21:14:56 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
21:16:39 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
21:21:39 opendevreview Ghanshyam Maan proposed openstack/nova master: Fix task leak for live migration rollback case https://review.opendev.org/c/openstack/nova/+/1003386
#openstack-nova - 2026-09-02
02:02:31 gmaan gibi: sean-k-mooney : this is fix for the live migration task leaking https://review.opendev.org/c/openstack/nova/+/1003386
05:19:56 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
06:12:01 opendevreview Adam McArthur proposed openstack/nova master: Use safe allocation deletion on instance delete https://review.opendev.org/c/openstack/nova/+/1003208
10:02:07 gibi can I get an core eye on https://review.opendev.org/c/openstack/nova/+/1003245 it is a functional test stabilization
10:12:32 sean-k-mooney so the intet is to set it to a largere value later but we are settign it to 10 for now so that if we do leak it does not cause the run to fail
10:15:58 sean-k-mooney i see in https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/functional/regressions/test_bug_1888395.py your bumping it to 300 and hten claping it to 10 on the tests that have know issues
10:16:20 sean-k-mooney any that are leaking a thread will timeout and you can clamp them until its fixed
10:17:54 sean-k-mooney gibi: we can technially start deleting evnetlet in like a weak to eliminate those leaks :)
10:19:47 sean-k-mooney gibi: by the way did you notice that the threading functional test now run in half the time of the eventlet ones
10:20:18 sean-k-mooney oh nope
10:20:30 sean-k-mooney thats the non funtionla treadign tests
10:21:49 gibi sean-k-mooney: we cannot delete them until post 2027.1
10:22:04 sean-k-mooney we technially can
10:22:26 sean-k-mooney but i know we are givign a very long tail in nova
10:22:29 gibi sc-tmp-lighbits-rhoso-certification
10:22:33 gibi https://governance.openstack.org/tc/goals/selected/remove-eventlet.html#completion-criteria
10:22:56 gibi u(2027.1) Deprecate the Eventlet support. To have a SLURP target release supporting both the old Eventlet and any new concurrency mode, this release keeps the Eventlet support.
10:23:08 sean-k-mooney those are the letes tg we can do those
10:23:22 sean-k-mooney not the first time for each milestone
10:23:52 sean-k-mooney several service just remove eventlet with no opertunity to have a singel slurp release that ran both
10:24:18 gibi but I think what is written in the goal is there because we wanted to do it that way :)
10:24:52 sean-k-mooney we did im just saying if it continue to be a maintiance isseu we can change that planb
10:24:57 sean-k-mooney on the ohter hand
10:25:08 sean-k-mooney if the eventlet test find leaks that we dont see in threaded
10:25:19 sean-k-mooney we probably shoudl fix those anyway
10:25:50 gibi yeah I haven't looked into the eventlet hangs in that patch
10:25:53 gibi but I will
10:47:27 opendevreview Balazs Gibizer proposed openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452
10:47:48 gibi sean-k-mooney: bauzas: ^^ another functional stabilization patch
13:21:29 opendevreview Balazs Gibizer proposed openstack/nova master: [functional]Do not patch req-id generation https://review.opendev.org/c/openstack/nova/+/1003024
13:21:45 gibi cores this is another stabilization ^^
14:16:56 stephenfin sean-k-mooney: did you already bring up https://review.opendev.org/c/openstack/releases/+/1003257?
14:17:21 stephenfin https://review.opendev.org/c/openstack/releases/+/1003311 is blocked by that
14:32:41 sean-k-mooney sorry no Uggla bauzas ^ we need to release os-vif again
14:33:13 sean-k-mooney elodilles: ^ fyi
14:33:45 Uggla sean-k-mooney ok I have seen the patches
14:41:49 elodilles sean-k-mooney: nova team needs to agree on FFE and an RFE needs to be requested from requirements team, otherwise LGTM
14:48:38 sean-k-mooney the config bug is a reguress intoduced in this cycle os i think it would qualify for a FFE adn enve after RC1 but idealy we do it before we do it
14:52:25 gibi I'm OK from nova perspective to accept this os_vif fix as FFE
14:52:33 gibi https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/fixtures/conf.py#73
14:52:36 gibi this is being the fix
15:37:49 opendevreview Balazs Gibizer proposed openstack/nova master: [DNM]reproduce "database is locked" error https://review.opendev.org/c/openstack/nova/+/1003554
15:38:51 gibi sean-k-mooney: ^^ finally after 2days and many deep rabbit holes this reproduces the DB locking issue pretty regularly to me locally
15:39:56 gibi there is something called checkpointing the WAL
15:40:08 gibi that happens after the last DB connection is closed
15:40:19 gibi happens outside of the db context manager
15:40:27 gibi and sqlite takes a DB write lock for it
15:41:06 gibi as we constantly open and close DB connections there is plenty of, very short, window when there is a last-connection-closed situation
15:41:40 gibi and if we are unlucky the next connection is a write transaction that hits within the window of the WAL checkpointing and seeing the DB as locked
15:42:11 gmaan gibi: https://review.opendev.org/c/openstack/nova/+/1003386 is green to fix the leaking task. can you rebase your manager_shutdown_timeout=300 change to this and we can see what all are fixed with this
15:42:20 gibi gmaan: on my list still for today
15:42:30 gmaan thanks
15:42:56 melwitt so that per something lock change did not help huh?
15:44:49 gibi melwitt: it helps
15:44:55 gibi we need to serialize writes
15:44:56 gibi but
15:45:08 melwitt oh ok
15:45:09 gibi there is this secondary case when sqlite takes a DB write lock
15:45:19 gibi outside of a write transaction
15:45:34 gibi the case being, the last connection to the DB is closed.
15:45:51 gibi so the immedate fix is, do not allow such situation to happen :)
15:46:04 gibi i.e. keep a connection open during the test
15:46:34 melwitt ah, yeah. that's how the in-memory db worked so it didn't come up in the past
15:46:56 gibi yes in memory DB used a StaticPool to have only a single connection to a DB all the time
15:47:08 gibi the file based uses a NullPool
15:47:08 melwitt right
15:47:33 gibi meaning one connection per transaction
15:48:00 gibi the StaticPool is not good for us in threading as that allows two parallel transaction to overlap on that single global connection
15:48:29 gibi but NullPool + WAL triggers this edge case of checkpointing the WAL
15:49:06 gibi my learning from the last two days: DB engines are complicated beasts
15:49:58 melwitt yes they are :( in my limited experience
15:56:31 sean-k-mooney gibi: nice find
15:57:22 sean-k-mooney the checkpointing this vagly rings a bell but only in that i think i came across that at some poihnt
15:57:39 sean-k-mooney but not that it happend on connection close/open
15:58:36 sean-k-mooney keeping the conenction open during the test is proably fine if we can do that cleanly
15:58:37 gibi I think this is part of the maintenance of the WAL itself
15:58:47 gibi > cleanly
15:58:51 gibi working on it :)
15:59:27 sean-k-mooney i was thinking of some kind fo per test singelton that we setup and tear down
16:00:04 sean-k-mooney the impornat thing is you have a repoducer that does not need an entire ci run now
16:03:45 gibi yepp by running with timeout=0 in the DB url (which is busy_timeout for SQLite) I can trigger the issue locally in my machine pretty frequently
16:03:48 gibi so I can test a fiz
16:03:49 gibi fix
16:04:30 sean-k-mooney if you push one ill let it run on my mac min for an hour or two in a loop while we wait on ci
16:05:35 gibi yeah working on the cleaniness
16:05:38 gibi of the fix

Earlier   Later