Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-02
15:42:11 gmaan gibi: https://review.opendev.org/c/openstack/nova/+/1003386 is green to fix the leaking task. can you rebase your manager_shutdown_timeout=300 change to this and we can see what all are fixed with this
15:42:20 gibi gmaan: on my list still for today
15:42:30 gmaan thanks
15:42:56 melwitt so that per something lock change did not help huh?
15:44:49 gibi melwitt: it helps
15:44:55 gibi we need to serialize writes
15:44:56 gibi but
15:45:08 melwitt oh ok
15:45:09 gibi there is this secondary case when sqlite takes a DB write lock
15:45:19 gibi outside of a write transaction
15:45:34 gibi the case being, the last connection to the DB is closed.
15:45:51 gibi so the immedate fix is, do not allow such situation to happen :)
15:46:04 gibi i.e. keep a connection open during the test
15:46:34 melwitt ah, yeah. that's how the in-memory db worked so it didn't come up in the past
15:46:56 gibi yes in memory DB used a StaticPool to have only a single connection to a DB all the time
15:47:08 gibi the file based uses a NullPool
15:47:08 melwitt right
15:47:33 gibi meaning one connection per transaction
15:48:00 gibi the StaticPool is not good for us in threading as that allows two parallel transaction to overlap on that single global connection
15:48:29 gibi but NullPool + WAL triggers this edge case of checkpointing the WAL
15:49:06 gibi my learning from the last two days: DB engines are complicated beasts
15:49:58 melwitt yes they are :( in my limited experience
15:56:31 sean-k-mooney gibi: nice find
15:57:22 sean-k-mooney the checkpointing this vagly rings a bell but only in that i think i came across that at some poihnt
15:57:39 sean-k-mooney but not that it happend on connection close/open
15:58:36 sean-k-mooney keeping the conenction open during the test is proably fine if we can do that cleanly
15:58:37 gibi I think this is part of the maintenance of the WAL itself
15:58:47 gibi > cleanly
15:58:51 gibi working on it :)
15:59:27 sean-k-mooney i was thinking of some kind fo per test singelton that we setup and tear down
16:00:04 sean-k-mooney the impornat thing is you have a repoducer that does not need an entire ci run now
16:03:45 gibi yepp by running with timeout=0 in the DB url (which is busy_timeout for SQLite) I can trigger the issue locally in my machine pretty frequently
16:03:48 gibi so I can test a fiz
16:03:49 gibi fix
16:04:30 sean-k-mooney if you push one ill let it run on my mac min for an hour or two in a loop while we wait on ci
16:05:35 gibi yeah working on the cleaniness
16:05:38 gibi of the fix
16:05:48 gibi but I saw having a connection open removing the issue locally
16:09:07 gibi closing the connection is a pain as the place where we need to open it is before the fixture setUp so I cannot register a function via addCleanup. Probably I need to create just an additional fixture for the connection and hope that opening the extra connection a bit after the DB creation is still OK to fix this
16:10:58 sean-k-mooney you can just directlly add it to tearDown
16:11:00 sean-k-mooney if needed
17:46:15 opendevreview Balazs Gibizer proposed openstack/nova master: [func]Avoid WAL checkpointing https://review.opendev.org/c/openstack/nova/+/1003586
17:46:41 gibi sean-k-mooney: ^^
17:47:42 sean-k-mooney cool, the DNM hung when i ran it so ill start running that
17:48:45 sean-k-mooney do you need to handel oterh escptions form the opertunistic test fixture
17:49:00 sean-k-mooney and or only do this when we are uing squlight
17:49:06 sean-k-mooney *sqlite
17:49:23 sean-k-mooney i guess no
17:49:39 sean-k-mooney because you rusign it from _bind_database_fixture_to_file_wal_sqlite
17:50:24 opendevreview Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
17:50:44 gibi yepp it is sqlite specific
17:51:11 gibi gmaan: cherry picked https://review.opendev.org/c/openstack/nova/+/1003266 top of you fix and removed the TODOs, I will check back on the result tomorrow
17:51:32 gibi aaand I drop for today o/
17:51:41 sean-k-mooney o/
17:56:48 gmaan gibi: thanks and o/
17:59:57 sean-k-mooney functional-py313-threading: OK (573.29=setup[8.02]+cmd[0.01,564.34,0.93] seconds)
18:00:12 sean-k-mooney ok im going to run that with --until failure for an hour or so
18:49:02 opendevreview Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
#openstack-nova - 2026-09-03
01:05:37 opendevreview Merged openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452
02:12:36 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
02:12:36 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349
02:12:37 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
03:23:30 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
06:08:08 opendevreview Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772
08:56:14 gibi can I get some eyes on a simple test stabilizition patch https://review.opendev.org/c/openstack/nova/+/1003024 ?
15:39:59 opendevreview Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834
15:40:33 opendevreview Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834
16:17:20 stephenfin Uggla: It's not hugely urgent, but I believe I need your +1 on the project-config change when you've time https://review.opendev.org/c/openstack/project-config/+/1003848
16:17:54 stephenfin Uggla: Context, in case you haven't seen it yet (I only just sent it) https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/EI44ZQG6UHIBNH3O26KHUVHIE576SCOC/
16:19:09 Uggla Hi stephenfin, ok I'll have a look probably tomorrow. Thx for letting me know.
16:20:49 stephenfin nw
18:40:04 opendevreview Merged openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834
19:13:52 dansmith gouthamr: are you around/
19:14:12 gouthamr hey dansmith, yes i am
19:14:49 dansmith gouthamr: question for you here: https://review.opendev.org/c/openstack/nova/+/1001900/comment/801dadc7_d5810626/
19:21:38 dansmith gouthamr: one more while you're looking: https://review.opendev.org/c/openstack/nova/+/1001900/comment/d75c5653_9253dded/
20:11:00 gouthamr ++ good questions, sorry my response took forever - distractions
20:21:57 dansmith gouthamr: might be quicker here as the clock is ticking and I'm running out of gas
20:22:54 dansmith gouthamr: I think I'm reading what you're saying as "this will leave per-host cephx keys stale after it's applied and things have moved enough to fully stop using the old ones? is that right?
20:23:09 dansmith meaning some years down the road, the operator will need to know to go back and clean all those up?
20:26:47 gouthamr dansmith: "per-host cephx keys" are what i'm proposing now (per-host-per-instance).. the old method was just one key: "nova"
20:27:24 gouthamr dansmith: yes, we'll forget that one after this fix is applied, intentionally so that an admin can clean it up later
20:27:32 dansmith perhaps I'm confusing two things here
20:28:03 dansmith currently we have per-host shares with each instance using the same deployment-wide key correct?
20:28:19 dansmith er, sharemapping I guess I should say
20:29:01 gouthamr yes
20:29:19 gouthamr true, one share mapping per host, no matter the number of instances you have
20:29:20 dansmith so there's one sharemapping for the host, and multiple access rules, one per instance
20:29:42 dansmith each access rule is currently using the shared key
20:29:49 gouthamr yes
20:30:21 dansmith after this patch is applied to a running deployment, instances will slowly start to move to the per-host-per-instance key as shares are added/removed, instances move around (I assume), etc right?
20:30:54 dansmith but there could be some long period where half the long-lived instances in a deployment are using the shared key such that the operator can't revoke/remove it right?
20:31:51 JayF If this is the model, it feels almost like you'd need a nova-manage command or other operator instruction on how to know it's safe to rotate the shared credential out
20:32:01 gouthamr (besides moving, you can restart the instances too)
20:32:04 dansmith this is why I'm asking
20:32:32 dansmith because there's not much visibility into when and by what deterministic process this would clean up
20:33:09 gouthamr on the manila side?
20:33:45 gouthamr JayF: yeah, i plan to write it out.. nova-manage isn't needed here because no cruft is left on nova
20:33:45 JayF gouthamr: here's the key question: in order for the operator to see security benefit, they have to drop the shared "nova" credential. In a large, public-cloud-sized nova deployment, are we giving folks the tools they need to determine they *can* do that?\
20:33:55 dansmith so I want to make sure that (a) we're not going to break existing deployments that use the old key (doesn't sound like we will) but that (b) we're not going to create a thing that will only be orphaned in five years when the last thing stops using it, and the operator won't be able to revoke between now and then nor set their alarm for when they can safely clean it up

Earlier   Later