| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-02 | |||
| 15:47:33 | gibi | meaning one connection per transaction | |
| 15:48:00 | gibi | the StaticPool is not good for us in threading as that allows two parallel transaction to overlap on that single global connection | |
| 15:48:29 | gibi | but NullPool + WAL triggers this edge case of checkpointing the WAL | |
| 15:49:06 | gibi | my learning from the last two days: DB engines are complicated beasts | |
| 15:49:58 | melwitt | yes they are :( in my limited experience | |
| 15:56:31 | sean-k-mooney | gibi: nice find | |
| 15:57:22 | sean-k-mooney | the checkpointing this vagly rings a bell but only in that i think i came across that at some poihnt | |
| 15:57:39 | sean-k-mooney | but not that it happend on connection close/open | |
| 15:58:36 | sean-k-mooney | keeping the conenction open during the test is proably fine if we can do that cleanly | |
| 15:58:37 | gibi | I think this is part of the maintenance of the WAL itself | |
| 15:58:47 | gibi | > cleanly | |
| 15:58:51 | gibi | working on it :) | |
| 15:59:27 | sean-k-mooney | i was thinking of some kind fo per test singelton that we setup and tear down | |
| 16:00:04 | sean-k-mooney | the impornat thing is you have a repoducer that does not need an entire ci run now | |
| 16:03:45 | gibi | yepp by running with timeout=0 in the DB url (which is busy_timeout for SQLite) I can trigger the issue locally in my machine pretty frequently | |
| 16:03:48 | gibi | so I can test a fiz | |
| 16:03:49 | gibi | fix | |
| 16:04:30 | sean-k-mooney | if you push one ill let it run on my mac min for an hour or two in a loop while we wait on ci | |
| 16:05:35 | gibi | yeah working on the cleaniness | |
| 16:05:38 | gibi | of the fix | |
| 16:05:48 | gibi | but I saw having a connection open removing the issue locally | |
| 16:09:07 | gibi | closing the connection is a pain as the place where we need to open it is before the fixture setUp so I cannot register a function via addCleanup. Probably I need to create just an additional fixture for the connection and hope that opening the extra connection a bit after the DB creation is still OK to fix this | |
| 16:10:58 | sean-k-mooney | you can just directlly add it to tearDown | |
| 16:11:00 | sean-k-mooney | if needed | |
| 17:46:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func]Avoid WAL checkpointing https://review.opendev.org/c/openstack/nova/+/1003586 | |
| 17:46:41 | gibi | sean-k-mooney: ^^ | |
| 17:47:42 | sean-k-mooney | cool, the DNM hung when i ran it so ill start running that | |
| 17:48:45 | sean-k-mooney | do you need to handel oterh escptions form the opertunistic test fixture | |
| 17:49:00 | sean-k-mooney | and or only do this when we are uing squlight | |
| 17:49:06 | sean-k-mooney | *sqlite | |
| 17:49:23 | sean-k-mooney | i guess no | |
| 17:49:39 | sean-k-mooney | because you rusign it from _bind_database_fixture_to_file_wal_sqlite | |
| 17:50:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | |
| 17:50:44 | gibi | yepp it is sqlite specific | |
| 17:51:11 | gibi | gmaan: cherry picked https://review.opendev.org/c/openstack/nova/+/1003266 top of you fix and removed the TODOs, I will check back on the result tomorrow | |
| 17:51:32 | gibi | aaand I drop for today o/ | |
| 17:51:41 | sean-k-mooney | o/ | |
| 17:56:48 | gmaan | gibi: thanks and o/ | |
| 17:59:57 | sean-k-mooney | functional-py313-threading: OK (573.29=setup[8.02]+cmd[0.01,564.34,0.93] seconds) | |
| 18:00:12 | sean-k-mooney | ok im going to run that with --until failure for an hour or so | |
| 18:49:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | |
| #openstack-nova - 2026-09-03 | |||
| 01:05:37 | opendevreview | Merged openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452 | |
| 02:12:36 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 02:12:36 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 02:12:37 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 03:23:30 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 06:08:08 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | |
| 08:56:14 | gibi | can I get some eyes on a simple test stabilizition patch https://review.opendev.org/c/openstack/nova/+/1003024 ? | |
| 15:39:59 | opendevreview | Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834 | |
| 15:40:33 | opendevreview | Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834 | |
| 16:17:20 | stephenfin | Uggla: It's not hugely urgent, but I believe I need your +1 on the project-config change when you've time https://review.opendev.org/c/openstack/project-config/+/1003848 | |
| 16:17:54 | stephenfin | Uggla: Context, in case you haven't seen it yet (I only just sent it) https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/EI44ZQG6UHIBNH3O26KHUVHIE576SCOC/ | |
| 16:19:09 | Uggla | Hi stephenfin, ok I'll have a look probably tomorrow. Thx for letting me know. | |
| 16:20:49 | stephenfin | nw | |
| 18:40:04 | opendevreview | Merged openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834 | |
| 19:13:52 | dansmith | gouthamr: are you around/ | |
| 19:14:12 | gouthamr | hey dansmith, yes i am | |
| 19:14:49 | dansmith | gouthamr: question for you here: https://review.opendev.org/c/openstack/nova/+/1001900/comment/801dadc7_d5810626/ | |
| 19:21:38 | dansmith | gouthamr: one more while you're looking: https://review.opendev.org/c/openstack/nova/+/1001900/comment/d75c5653_9253dded/ | |
| 20:11:00 | gouthamr | ++ good questions, sorry my response took forever - distractions | |
| 20:21:57 | dansmith | gouthamr: might be quicker here as the clock is ticking and I'm running out of gas | |
| 20:22:54 | dansmith | gouthamr: I think I'm reading what you're saying as "this will leave per-host cephx keys stale after it's applied and things have moved enough to fully stop using the old ones? is that right? | |
| 20:23:09 | dansmith | meaning some years down the road, the operator will need to know to go back and clean all those up? | |
| 20:26:47 | gouthamr | dansmith: "per-host cephx keys" are what i'm proposing now (per-host-per-instance).. the old method was just one key: "nova" | |
| 20:27:24 | gouthamr | dansmith: yes, we'll forget that one after this fix is applied, intentionally so that an admin can clean it up later | |
| 20:27:32 | dansmith | perhaps I'm confusing two things here | |
| 20:28:03 | dansmith | currently we have per-host shares with each instance using the same deployment-wide key correct? | |
| 20:28:19 | dansmith | er, sharemapping I guess I should say | |
| 20:29:01 | gouthamr | yes | |
| 20:29:19 | gouthamr | true, one share mapping per host, no matter the number of instances you have | |
| 20:29:20 | dansmith | so there's one sharemapping for the host, and multiple access rules, one per instance | |
| 20:29:42 | dansmith | each access rule is currently using the shared key | |
| 20:29:49 | gouthamr | yes | |
| 20:30:21 | dansmith | after this patch is applied to a running deployment, instances will slowly start to move to the per-host-per-instance key as shares are added/removed, instances move around (I assume), etc right? | |
| 20:30:54 | dansmith | but there could be some long period where half the long-lived instances in a deployment are using the shared key such that the operator can't revoke/remove it right? | |
| 20:31:51 | JayF | If this is the model, it feels almost like you'd need a nova-manage command or other operator instruction on how to know it's safe to rotate the shared credential out | |
| 20:32:01 | gouthamr | (besides moving, you can restart the instances too) | |
| 20:32:04 | dansmith | this is why I'm asking | |
| 20:32:32 | dansmith | because there's not much visibility into when and by what deterministic process this would clean up | |
| 20:33:09 | gouthamr | on the manila side? | |
| 20:33:45 | JayF | gouthamr: here's the key question: in order for the operator to see security benefit, they have to drop the shared "nova" credential. In a large, public-cloud-sized nova deployment, are we giving folks the tools they need to determine they *can* do that?\ | |
| 20:33:45 | gouthamr | JayF: yeah, i plan to write it out.. nova-manage isn't needed here because no cruft is left on nova | |
| 20:33:55 | dansmith | so I want to make sure that (a) we're not going to break existing deployments that use the old key (doesn't sound like we will) but that (b) we're not going to create a thing that will only be orphaned in five years when the last thing stops using it, and the operator won't be able to revoke between now and then nor set their alarm for when they can safely clean it up | |
| 20:34:15 | dansmith | JayF: exactamundo | |
| 20:34:38 | dansmith | gouthamr: the reference is stored in your object is it not? | |
| 20:35:16 | gouthamr | (b) is genuinely a worry - you can lose track.. that can happen today.. there's no real way of saying "are you using this credential" (its a universal "problem" with network-attached-storage if you want to generalize) | |
| 20:35:57 | gouthamr | dansmith: no we're actually not storing any metadata specific to an instance on manila.. we're locking the share from deletion, but the lock just says: "locked by nova" | |
| 20:36:21 | gouthamr | i actually think it'd be fine for the lock to say "locked by nova for instance: UUID" | |
| 20:36:39 | gouthamr | but i don't know if Uggla gibi thought that wasn't necessary when they wrote this up | |
| 20:36:42 | dansmith | gouthamr: it's stored in your access_to field in the nova object we store in the DB? | |
| 20:37:16 | dansmith | s/?/right?/ | |
| 20:37:59 | gouthamr | dansmith: yeah, manila knows what "access_to" is.. and today there's a shared hardcoded access_to: "nova".. you can filter on this stuff and isolate what all shares have this access | |
| 20:38:53 | dansmith | gouthamr: I'm saying.. there _is_ cruft left in nova (you asserted not above) in the form of the access_to in the object, _and_ that we _could_ provide what JayF and I are asking, which is a roadmap/status display of what has/has not migrated | |
| 20:40:06 | dansmith | right exactly | |
| 20:40:09 | dansmith | and that goes to my other question/concern, | |
| 20:40:38 | dansmith | which is that we're basically giving $specificName a "random" (or at least obfuscated) value which also makes it hard to track.. which is why I asked if it was secret | |
| 20:40:46 | dansmith | as the obfuscation makes it harder to roll your own toolage | |
| 20:40:48 | gouthamr | dansmith: where is this cruft? ShareMapping has no "access_to"... It was hardcoded in the code: https://github.com/openstack/nova/blob/ad5e2ea26e3bd80a84e0ab205ceef4200906ded1/nova/db/main/models.py#L756-L797 | |
| 20:41:52 | gouthamr | dansmith: is it hard though? the HOST field is known, instance IDs are known, you can query manila and get all the rules that map with this combination and the obfuscation logic | |
| 20:41:53 | dansmith | gouthamr: I hadn't looked at the lower layers.. so your object has the access_to field but you never persist it? | |