| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-31 | |||
| 09:06:26 | gibi | OK, so you say we change oslo.db's list of exceptions to retry on | |
| 09:06:37 | gibi | :) | |
| 09:07:07 | gibi | I still cannot really justify that from nova perspective and the ironic case is something I cannot reproduce | |
| 09:07:08 | sean-k-mooney | i think that would be good in the long term yes but i think we shoudl be able to work aroudn this in the futnctionl tests | |
| 09:07:55 | sean-k-mooney | its just somethign i looked into when sqlite was segfaulting in cybrog | |
| 09:08:44 | sean-k-mooney | as a cleaner way then jsut hte locking | |
| 09:10:08 | sean-k-mooney | but ya if there are other tuneable or we can specficly tell sqlachemy to flush/release the db or something that might be better | |
| 09:10:39 | gibi | https://sqlite.org/c3ref/busy_timeout.html | |
| 09:10:50 | gibi | this could be an SQLite native retry | |
| 09:11:24 | gibi | https://sqlite.org/c3ref/busy_timeout.html | |
| 09:11:33 | gibi | https://sqlite.org/pragma.html#pragma_busy_timeout | |
| 09:13:41 | sean-k-mooney | """The SQLITE_LOCKED result code indicates that a write operation could not continue because of a conflict within the same database connection or a conflict with a different database connection that uses a shared cache. """ | |
| 09:14:20 | sean-k-mooney | so the vs """The SQLITE_BUSY result code indicates that the database file could not be written (or in some cases read) because of concurrent activity by some other database connection, usually a database connection in a separate process. """ | |
| 09:14:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: [functional]Do not patch req-id generation https://review.opendev.org/c/openstack/nova/+/1003024 | |
| 09:14:45 | sean-k-mooney | we are not using processes and we have a per db lock | |
| 09:15:08 | sean-k-mooney | so im no t sure the busy tunabels will help | |
| 09:15:14 | gibi | we use threads | |
| 09:15:18 | sean-k-mooney | are we using the cache file | |
| 09:15:36 | sean-k-mooney | i wonder if we are sharign the cache betwen dbs? | |
| 09:17:18 | gibi | I think it is a separate file per DB but I can probably check by injecting a long see into our tests and see the dir structure | |
| 09:17:35 | sean-k-mooney | ack | |
| 09:20:56 | sean-k-mooney | i think we create a seprate WAL,cache and db fiel per database each in there own temp dir for the test | |
| 09:21:30 | sean-k-mooney | but maybe only the db and wal file are separate | |
| 09:22:48 | gibi | https://paste.openstack.org/show/b8jN0kKZZWRZVUQ11xDP/ I don't see cache files | |
| 09:23:39 | sean-k-mooney | ya it looks like wal mode replaces it | |
| 09:23:49 | sean-k-mooney | and shared_cache mdoe is the legacy version | |
| 09:23:51 | sean-k-mooney | https://sqlite.org/wal.html | |
| 09:24:21 | gibi | we do set WAL | |
| 09:24:30 | sean-k-mooney | There is an additional quasi-persistent "-wal" file and "-shm" shared memory file associated with each database, | |
| 09:24:39 | sean-k-mooney | i was confusing it with the shared memory file | |
| 09:24:49 | gibi | but I don't see the WAL file either :) | |
| 09:25:11 | gibi | maybe it is deleted when the connection closed | |
| 09:27:06 | gibi | yepp | |
| 09:27:10 | gibi | they are there | |
| 09:27:14 | gibi | temporarily | |
| 09:27:18 | gibi | nova-test-api-5dclxobp.sqlite-wal | |
| 09:27:24 | gibi | nova-test-api-5dclxobp.sqlite-shm | |
| 09:27:35 | gibi | but that also means it is a unique file per DB | |
| 09:28:08 | gibi | now if two thread like nova-api and nova-conductor connects to the same api DB I assume they are sharing these | |
| 09:28:31 | sean-k-mooney | ack well if you find something that helps let me knwo https://sqlite.org/pragma.html#pragma_locking_mode might be intersting | |
| 09:28:52 | sean-k-mooney | gibi: they could but i thinkt hat is intended to be supproted | |
| 09:29:02 | gibi | anyhow I will push a patch with a PRAGMA busy_timeout = milliseconds; | |
| 09:29:12 | gibi | that is small and easy | |
| 09:29:24 | gibi | and functional test specific | |
| 09:32:02 | gibi | hm, maybe not. The WAL PRAGMA is persistent for the DB, but the busy_timeout is per connection so I cannot just add it to the same place :/ | |
| 09:32:18 | gibi | looking at oslo.db retry :) | |
| 09:36:51 | sean-k-mooney | the other thing you could try is set the lockign mode to exclusivne and back to normal in our currnet lock code https://sqlite.org/pragma.html#pragma_locking_mode but im not sure if that will really help or not | |
| 09:37:50 | sean-k-mooney | i feel like that hsoudl not be required but ya im goign to go get coffee and step away for a bit | |
| 09:40:19 | gibi | the hardness of this that I have no reproducer so whatever I try I have no proof it actually helps other than merging it and letting is run in the gate | |
| 09:41:12 | gibi | so trying random things is expensive | |
| 09:42:04 | gibi | the oslo.db retry code is pluged in a horrible way. I'm not even sure we use any retry at all by default | |
| 09:43:02 | gibi | we have decorators @oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) in some places | |
| 09:43:09 | gibi | so we do retry in some cases | |
| 09:43:11 | gibi | not all | |
| 09:43:28 | gibi | so even if I change what to retry on I would need to decorate everything | |
| 09:43:31 | gibi | that is uggly | |
| 09:48:51 | gibi | OK, looking at differently. We have the write lock per DB file. That should serialize all the writes. If I assume it works well, then the only thing that can happen is that the SQLite DB lock is not released when our functional fixture lock is released (as you said at the start) so we could try to flush, close, whatever the connection to help with that DB level lock | |
| 10:12:35 | opendevreview | Merged openstack/nova master: tests: Remove errant CONF mocks https://review.opendev.org/c/openstack/nova/+/1002852 | |
| 10:15:46 | tkajinam | could anyone give 2nd +2 to https://review.opendev.org/c/openstack/nova/+/995370 and https://review.opendev.org/c/openstack/nova/+/995370 to reduce deprecation warnings ? | |
| 10:52:36 | gibi | tkajinam: thanks I +Ad the https://review.opendev.org/c/openstack/nova/+/995370 but you linked that patch twice above so I assuem you have one more patch to look at | |
| 11:03:31 | tkajinam | oh sorry I've bee struggling to make my copy-paste work correctly | |
| 11:03:35 | tkajinam | gibi, https://review.opendev.org/c/openstack/nova/+/1002937 this is the 2nd one | |
| 11:03:47 | tkajinam | gibi, and thanks a lot ! | |
| 11:05:16 | tkajinam | oh wait I found a few remaining warnings | |
| 11:05:39 | tkajinam | will address these in follow-up | |
| 11:20:00 | opendevreview | Takashi Kajinami proposed openstack/nova master: ovo: Drop deprecated item_clas argument https://review.opendev.org/c/openstack/nova/+/1002937 | |
| 11:20:26 | opendevreview | Takashi Kajinami proposed openstack/nova master: ovo: Drop deprecated item_cls argument https://review.opendev.org/c/openstack/nova/+/1002937 | |
| 11:34:13 | gibi | thanks | |
| 12:25:52 | opendevreview | Merged openstack/nova master: Replace deprecated remotable_classmethod https://review.opendev.org/c/openstack/nova/+/995370 | |
| 12:41:11 | opendevreview | Takashi Kajinami proposed openstack/nova master: ovo: Drop deprecated item_cls argument https://review.opendev.org/c/openstack/nova/+/1002937 | |
| 13:39:55 | clif | sean-k-mooney: is there anything I can do to help that perf patch merge? | |
| 14:11:59 | sean-k-mooney | did it fail after the rebase? im on pto today so not really paying attaentio, gibi is lookign at some stablity impovment | |
| 14:12:17 | sean-k-mooney | i think you just geting unlucky with the test failures | |
| 14:12:48 | sean-k-mooney | the db locked error is oen of the knwo issues withthe func tests under threading mode currently | |
| 14:13:07 | sean-k-mooney | and it was in an entirly differnt area of the code so not related to that patch | |
| 14:40:54 | clif | gotcha | |
| 14:41:05 | clif | sorry to ping you on your day off, enjoy | |
| 14:41:34 | sean-k-mooney | its fien i would not be on irc if it bothered me | |
| 15:30:10 | Uggla | Reminder: Upstream meeting in ~30mn. | |
| 16:02:47 | Uggla | #startmeeting nova | |
| 16:02:47 | opendevmeet | Meeting started Mon Aug 31 16:02:47 2026 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:02:47 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:02:47 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:02:56 | Uggla | Hello everyone | |
| 16:02:59 | bauzas | o/ | |
| 16:03:02 | gmaan | o/ | |
| 16:03:42 | elodilles | o/ | |
| 16:04:37 | lajoskatona | o/ | |
| 16:05:14 | Uggla | Let's start | |
| 16:05:22 | Uggla | #topic Bugs (stuck/critical) | |
| 16:05:32 | Uggla | #info No Critical bug | |
| 16:05:43 | Uggla | #topic Gate status | |
| 16:05:46 | tkajinam | o/ | |
| 16:05:52 | Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:06:01 | Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:06:12 | Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | |
| 16:06:23 | Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:06:32 | Uggla | #info Please try to provide a meaningful comment when you recheck | |
| 16:06:33 | gmaan | we ,merged a lot of fixes last week, I will remove the bugs from that etherpad one we alrady fixed | |
| 16:07:13 | Uggla | #info last week we had gate instability. Any progress? | |