Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-02
15:38:51 gibi sean-k-mooney: ^^ finally after 2days and many deep rabbit holes this reproduces the DB locking issue pretty regularly to me locally
15:39:56 gibi there is something called checkpointing the WAL
15:40:08 gibi that happens after the last DB connection is closed
15:40:19 gibi happens outside of the db context manager
15:40:27 gibi and sqlite takes a DB write lock for it
15:41:06 gibi as we constantly open and close DB connections there is plenty of, very short, window when there is a last-connection-closed situation
15:41:40 gibi and if we are unlucky the next connection is a write transaction that hits within the window of the WAL checkpointing and seeing the DB as locked
15:42:11 gmaan gibi: https://review.opendev.org/c/openstack/nova/+/1003386 is green to fix the leaking task. can you rebase your manager_shutdown_timeout=300 change to this and we can see what all are fixed with this
15:42:20 gibi gmaan: on my list still for today
15:42:30 gmaan thanks
15:42:56 melwitt so that per something lock change did not help huh?
15:44:49 gibi melwitt: it helps
15:44:55 gibi we need to serialize writes
15:44:56 gibi but
15:45:08 melwitt oh ok
15:45:09 gibi there is this secondary case when sqlite takes a DB write lock
15:45:19 gibi outside of a write transaction
15:45:34 gibi the case being, the last connection to the DB is closed.
15:45:51 gibi so the immedate fix is, do not allow such situation to happen :)
15:46:04 gibi i.e. keep a connection open during the test
15:46:34 melwitt ah, yeah. that's how the in-memory db worked so it didn't come up in the past
15:46:56 gibi yes in memory DB used a StaticPool to have only a single connection to a DB all the time
15:47:08 melwitt right
15:47:08 gibi the file based uses a NullPool
15:47:33 gibi meaning one connection per transaction
15:48:00 gibi the StaticPool is not good for us in threading as that allows two parallel transaction to overlap on that single global connection
15:48:29 gibi but NullPool + WAL triggers this edge case of checkpointing the WAL
15:49:06 gibi my learning from the last two days: DB engines are complicated beasts
15:49:58 melwitt yes they are :( in my limited experience
15:56:31 sean-k-mooney gibi: nice find
15:57:22 sean-k-mooney the checkpointing this vagly rings a bell but only in that i think i came across that at some poihnt
15:57:39 sean-k-mooney but not that it happend on connection close/open
15:58:36 sean-k-mooney keeping the conenction open during the test is proably fine if we can do that cleanly
15:58:37 gibi I think this is part of the maintenance of the WAL itself
15:58:47 gibi > cleanly
15:58:51 gibi working on it :)
15:59:27 sean-k-mooney i was thinking of some kind fo per test singelton that we setup and tear down
16:00:04 sean-k-mooney the impornat thing is you have a repoducer that does not need an entire ci run now
16:03:45 gibi yepp by running with timeout=0 in the DB url (which is busy_timeout for SQLite) I can trigger the issue locally in my machine pretty frequently
16:03:48 gibi so I can test a fiz
16:03:49 gibi fix
16:04:30 sean-k-mooney if you push one ill let it run on my mac min for an hour or two in a loop while we wait on ci
16:05:35 gibi yeah working on the cleaniness
16:05:38 gibi of the fix
16:05:48 gibi but I saw having a connection open removing the issue locally
16:09:07 gibi closing the connection is a pain as the place where we need to open it is before the fixture setUp so I cannot register a function via addCleanup. Probably I need to create just an additional fixture for the connection and hope that opening the extra connection a bit after the DB creation is still OK to fix this
16:10:58 sean-k-mooney you can just directlly add it to tearDown
16:11:00 sean-k-mooney if needed
17:46:15 opendevreview Balazs Gibizer proposed openstack/nova master: [func]Avoid WAL checkpointing https://review.opendev.org/c/openstack/nova/+/1003586
17:46:41 gibi sean-k-mooney: ^^
17:47:42 sean-k-mooney cool, the DNM hung when i ran it so ill start running that
17:48:45 sean-k-mooney do you need to handel oterh escptions form the opertunistic test fixture
17:49:00 sean-k-mooney and or only do this when we are uing squlight
17:49:06 sean-k-mooney *sqlite
17:49:23 sean-k-mooney i guess no
17:49:39 sean-k-mooney because you rusign it from _bind_database_fixture_to_file_wal_sqlite
17:50:24 opendevreview Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
17:50:44 gibi yepp it is sqlite specific
17:51:11 gibi gmaan: cherry picked https://review.opendev.org/c/openstack/nova/+/1003266 top of you fix and removed the TODOs, I will check back on the result tomorrow
17:51:32 gibi aaand I drop for today o/
17:51:41 sean-k-mooney o/
17:56:48 gmaan gibi: thanks and o/
17:59:57 sean-k-mooney functional-py313-threading: OK (573.29=setup[8.02]+cmd[0.01,564.34,0.93] seconds)
18:00:12 sean-k-mooney ok im going to run that with --until failure for an hour or so
18:49:02 opendevreview Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
#openstack-nova - 2026-09-03
01:05:37 opendevreview Merged openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452
02:12:36 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349
02:12:36 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
02:12:37 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
03:23:30 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
06:08:08 opendevreview Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772
08:56:14 gibi can I get some eyes on a simple test stabilizition patch https://review.opendev.org/c/openstack/nova/+/1003024 ?
15:39:59 opendevreview Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834
15:40:33 opendevreview Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834
16:17:20 stephenfin Uggla: It's not hugely urgent, but I believe I need your +1 on the project-config change when you've time https://review.opendev.org/c/openstack/project-config/+/1003848
16:17:54 stephenfin Uggla: Context, in case you haven't seen it yet (I only just sent it) https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/EI44ZQG6UHIBNH3O26KHUVHIE576SCOC/
16:19:09 Uggla Hi stephenfin, ok I'll have a look probably tomorrow. Thx for letting me know.
16:20:49 stephenfin nw
18:40:04 opendevreview Merged openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834
19:13:52 dansmith gouthamr: are you around/
19:14:12 gouthamr hey dansmith, yes i am
19:14:49 dansmith gouthamr: question for you here: https://review.opendev.org/c/openstack/nova/+/1001900/comment/801dadc7_d5810626/
19:21:38 dansmith gouthamr: one more while you're looking: https://review.opendev.org/c/openstack/nova/+/1001900/comment/d75c5653_9253dded/
20:11:00 gouthamr ++ good questions, sorry my response took forever - distractions
20:21:57 dansmith gouthamr: might be quicker here as the clock is ticking and I'm running out of gas
20:22:54 dansmith gouthamr: I think I'm reading what you're saying as "this will leave per-host cephx keys stale after it's applied and things have moved enough to fully stop using the old ones? is that right?
20:23:09 dansmith meaning some years down the road, the operator will need to know to go back and clean all those up?
20:26:47 gouthamr dansmith: "per-host cephx keys" are what i'm proposing now (per-host-per-instance).. the old method was just one key: "nova"
20:27:24 gouthamr dansmith: yes, we'll forget that one after this fix is applied, intentionally so that an admin can clean it up later
20:27:32 dansmith perhaps I'm confusing two things here
20:28:03 dansmith currently we have per-host shares with each instance using the same deployment-wide key correct?
20:28:19 dansmith er, sharemapping I guess I should say
20:29:01 gouthamr yes
20:29:19 gouthamr true, one share mapping per host, no matter the number of instances you have
20:29:20 dansmith so there's one sharemapping for the host, and multiple access rules, one per instance
20:29:42 dansmith each access rule is currently using the shared key
20:29:49 gouthamr yes
20:30:21 dansmith after this patch is applied to a running deployment, instances will slowly start to move to the per-host-per-instance key as shares are added/removed, instances move around (I assume), etc right?
20:30:54 dansmith but there could be some long period where half the long-lived instances in a deployment are using the shared key such that the operator can't revoke/remove it right?
20:31:51 JayF If this is the model, it feels almost like you'd need a nova-manage command or other operator instruction on how to know it's safe to rotate the shared credential out

Earlier   Later