| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-02 | |||
| 10:25:08 | sean-k-mooney | if the eventlet test find leaks that we dont see in threaded | |
| 10:25:19 | sean-k-mooney | we probably shoudl fix those anyway | |
| 10:25:50 | gibi | yeah I haven't looked into the eventlet hangs in that patch | |
| 10:25:53 | gibi | but I will | |
| 10:47:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452 | |
| 10:47:48 | gibi | sean-k-mooney: bauzas: ^^ another functional stabilization patch | |
| 13:21:29 | opendevreview | Balazs Gibizer proposed openstack/nova master: [functional]Do not patch req-id generation https://review.opendev.org/c/openstack/nova/+/1003024 | |
| 13:21:45 | gibi | cores this is another stabilization ^^ | |
| 14:16:56 | stephenfin | sean-k-mooney: did you already bring up https://review.opendev.org/c/openstack/releases/+/1003257? | |
| 14:17:21 | stephenfin | https://review.opendev.org/c/openstack/releases/+/1003311 is blocked by that | |
| 14:32:41 | sean-k-mooney | sorry no Uggla bauzas ^ we need to release os-vif again | |
| 14:33:13 | sean-k-mooney | elodilles: ^ fyi | |
| 14:33:45 | Uggla | sean-k-mooney ok I have seen the patches | |
| 14:41:49 | elodilles | sean-k-mooney: nova team needs to agree on FFE and an RFE needs to be requested from requirements team, otherwise LGTM | |
| 14:48:38 | sean-k-mooney | the config bug is a reguress intoduced in this cycle os i think it would qualify for a FFE adn enve after RC1 but idealy we do it before we do it | |
| 14:52:25 | gibi | I'm OK from nova perspective to accept this os_vif fix as FFE | |
| 14:52:33 | gibi | https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/fixtures/conf.py#73 | |
| 14:52:36 | gibi | this is being the fix | |
| 15:37:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: [DNM]reproduce "database is locked" error https://review.opendev.org/c/openstack/nova/+/1003554 | |
| 15:38:51 | gibi | sean-k-mooney: ^^ finally after 2days and many deep rabbit holes this reproduces the DB locking issue pretty regularly to me locally | |
| 15:39:56 | gibi | there is something called checkpointing the WAL | |
| 15:40:08 | gibi | that happens after the last DB connection is closed | |
| 15:40:19 | gibi | happens outside of the db context manager | |
| 15:40:27 | gibi | and sqlite takes a DB write lock for it | |
| 15:41:06 | gibi | as we constantly open and close DB connections there is plenty of, very short, window when there is a last-connection-closed situation | |
| 15:41:40 | gibi | and if we are unlucky the next connection is a write transaction that hits within the window of the WAL checkpointing and seeing the DB as locked | |
| 15:42:11 | gmaan | gibi: https://review.opendev.org/c/openstack/nova/+/1003386 is green to fix the leaking task. can you rebase your manager_shutdown_timeout=300 change to this and we can see what all are fixed with this | |
| 15:42:20 | gibi | gmaan: on my list still for today | |
| 15:42:30 | gmaan | thanks | |
| 15:42:56 | melwitt | so that per something lock change did not help huh? | |
| 15:44:49 | gibi | melwitt: it helps | |
| 15:44:55 | gibi | we need to serialize writes | |
| 15:44:56 | gibi | but | |
| 15:45:08 | melwitt | oh ok | |
| 15:45:09 | gibi | there is this secondary case when sqlite takes a DB write lock | |
| 15:45:19 | gibi | outside of a write transaction | |
| 15:45:34 | gibi | the case being, the last connection to the DB is closed. | |
| 15:45:51 | gibi | so the immedate fix is, do not allow such situation to happen :) | |
| 15:46:04 | gibi | i.e. keep a connection open during the test | |
| 15:46:34 | melwitt | ah, yeah. that's how the in-memory db worked so it didn't come up in the past | |
| 15:46:56 | gibi | yes in memory DB used a StaticPool to have only a single connection to a DB all the time | |
| 15:47:08 | gibi | the file based uses a NullPool | |
| 15:47:08 | melwitt | right | |
| 15:47:33 | gibi | meaning one connection per transaction | |
| 15:48:00 | gibi | the StaticPool is not good for us in threading as that allows two parallel transaction to overlap on that single global connection | |
| 15:48:29 | gibi | but NullPool + WAL triggers this edge case of checkpointing the WAL | |
| 15:49:06 | gibi | my learning from the last two days: DB engines are complicated beasts | |
| 15:49:58 | melwitt | yes they are :( in my limited experience | |
| 15:56:31 | sean-k-mooney | gibi: nice find | |
| 15:57:22 | sean-k-mooney | the checkpointing this vagly rings a bell but only in that i think i came across that at some poihnt | |
| 15:57:39 | sean-k-mooney | but not that it happend on connection close/open | |
| 15:58:36 | sean-k-mooney | keeping the conenction open during the test is proably fine if we can do that cleanly | |
| 15:58:37 | gibi | I think this is part of the maintenance of the WAL itself | |
| 15:58:47 | gibi | > cleanly | |
| 15:58:51 | gibi | working on it :) | |
| 15:59:27 | sean-k-mooney | i was thinking of some kind fo per test singelton that we setup and tear down | |
| 16:00:04 | sean-k-mooney | the impornat thing is you have a repoducer that does not need an entire ci run now | |
| 16:03:45 | gibi | yepp by running with timeout=0 in the DB url (which is busy_timeout for SQLite) I can trigger the issue locally in my machine pretty frequently | |
| 16:03:48 | gibi | so I can test a fiz | |
| 16:03:49 | gibi | fix | |
| 16:04:30 | sean-k-mooney | if you push one ill let it run on my mac min for an hour or two in a loop while we wait on ci | |
| 16:05:35 | gibi | yeah working on the cleaniness | |
| 16:05:38 | gibi | of the fix | |
| 16:05:48 | gibi | but I saw having a connection open removing the issue locally | |
| 16:09:07 | gibi | closing the connection is a pain as the place where we need to open it is before the fixture setUp so I cannot register a function via addCleanup. Probably I need to create just an additional fixture for the connection and hope that opening the extra connection a bit after the DB creation is still OK to fix this | |
| 16:10:58 | sean-k-mooney | you can just directlly add it to tearDown | |
| 16:11:00 | sean-k-mooney | if needed | |
| 17:46:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func]Avoid WAL checkpointing https://review.opendev.org/c/openstack/nova/+/1003586 | |
| 17:46:41 | gibi | sean-k-mooney: ^^ | |
| 17:47:42 | sean-k-mooney | cool, the DNM hung when i ran it so ill start running that | |
| 17:48:45 | sean-k-mooney | do you need to handel oterh escptions form the opertunistic test fixture | |
| 17:49:00 | sean-k-mooney | and or only do this when we are uing squlight | |
| 17:49:06 | sean-k-mooney | *sqlite | |
| 17:49:23 | sean-k-mooney | i guess no | |
| 17:49:39 | sean-k-mooney | because you rusign it from _bind_database_fixture_to_file_wal_sqlite | |
| 17:50:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | |
| 17:50:44 | gibi | yepp it is sqlite specific | |
| 17:51:11 | gibi | gmaan: cherry picked https://review.opendev.org/c/openstack/nova/+/1003266 top of you fix and removed the TODOs, I will check back on the result tomorrow | |
| 17:51:32 | gibi | aaand I drop for today o/ | |
| 17:51:41 | sean-k-mooney | o/ | |
| 17:56:48 | gmaan | gibi: thanks and o/ | |
| 17:59:57 | sean-k-mooney | functional-py313-threading: OK (573.29=setup[8.02]+cmd[0.01,564.34,0.93] seconds) | |
| 18:00:12 | sean-k-mooney | ok im going to run that with --until failure for an hour or so | |
| 18:49:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | |
| #openstack-nova - 2026-09-03 | |||
| 01:05:37 | opendevreview | Merged openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452 | |
| 02:12:36 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 02:12:36 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 02:12:37 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 03:23:30 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 06:08:08 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | |
| 08:56:14 | gibi | can I get some eyes on a simple test stabilizition patch https://review.opendev.org/c/openstack/nova/+/1003024 ? | |
| 15:39:59 | opendevreview | Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834 | |
| 15:40:33 | opendevreview | Ivan Anfimov proposed openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834 | |
| 16:17:20 | stephenfin | Uggla: It's not hugely urgent, but I believe I need your +1 on the project-config change when you've time https://review.opendev.org/c/openstack/project-config/+/1003848 | |
| 16:17:54 | stephenfin | Uggla: Context, in case you haven't seen it yet (I only just sent it) https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/EI44ZQG6UHIBNH3O26KHUVHIE576SCOC/ | |
| 16:19:09 | Uggla | Hi stephenfin, ok I'll have a look probably tomorrow. Thx for letting me know. | |
| 16:20:49 | stephenfin | nw | |
| 18:40:04 | opendevreview | Merged openstack/os-vif master: Fix typo https://review.opendev.org/c/openstack/os-vif/+/1003834 | |
| 19:13:52 | dansmith | gouthamr: are you around/ | |
| 19:14:12 | gouthamr | hey dansmith, yes i am | |