| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-31 | |||
| 09:32:02 | gibi | hm, maybe not. The WAL PRAGMA is persistent for the DB, but the busy_timeout is per connection so I cannot just add it to the same place :/ | |
| 09:32:18 | gibi | looking at oslo.db retry :) | |
| 09:36:51 | sean-k-mooney | the other thing you could try is set the lockign mode to exclusivne and back to normal in our currnet lock code https://sqlite.org/pragma.html#pragma_locking_mode but im not sure if that will really help or not | |
| 09:37:50 | sean-k-mooney | i feel like that hsoudl not be required but ya im goign to go get coffee and step away for a bit | |
| 09:40:19 | gibi | the hardness of this that I have no reproducer so whatever I try I have no proof it actually helps other than merging it and letting is run in the gate | |
| 09:41:12 | gibi | so trying random things is expensive | |
| 09:42:04 | gibi | the oslo.db retry code is pluged in a horrible way. I'm not even sure we use any retry at all by default | |
| 09:43:02 | gibi | we have decorators @oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) in some places | |
| 09:43:09 | gibi | so we do retry in some cases | |
| 09:43:11 | gibi | not all | |
| 09:43:28 | gibi | so even if I change what to retry on I would need to decorate everything | |
| 09:43:31 | gibi | that is uggly | |
| 09:48:51 | gibi | OK, looking at differently. We have the write lock per DB file. That should serialize all the writes. If I assume it works well, then the only thing that can happen is that the SQLite DB lock is not released when our functional fixture lock is released (as you said at the start) so we could try to flush, close, whatever the connection to help with that DB level lock | |
| 10:12:35 | opendevreview | Merged openstack/nova master: tests: Remove errant CONF mocks https://review.opendev.org/c/openstack/nova/+/1002852 | |
| 10:15:46 | tkajinam | could anyone give 2nd +2 to https://review.opendev.org/c/openstack/nova/+/995370 and https://review.opendev.org/c/openstack/nova/+/995370 to reduce deprecation warnings ? | |
| 10:52:36 | gibi | tkajinam: thanks I +Ad the https://review.opendev.org/c/openstack/nova/+/995370 but you linked that patch twice above so I assuem you have one more patch to look at | |
| 11:03:31 | tkajinam | oh sorry I've bee struggling to make my copy-paste work correctly | |
| 11:03:35 | tkajinam | gibi, https://review.opendev.org/c/openstack/nova/+/1002937 this is the 2nd one | |
| 11:03:47 | tkajinam | gibi, and thanks a lot ! | |
| 11:05:16 | tkajinam | oh wait I found a few remaining warnings | |
| 11:05:39 | tkajinam | will address these in follow-up | |
| 11:20:00 | opendevreview | Takashi Kajinami proposed openstack/nova master: ovo: Drop deprecated item_clas argument https://review.opendev.org/c/openstack/nova/+/1002937 | |
| 11:20:26 | opendevreview | Takashi Kajinami proposed openstack/nova master: ovo: Drop deprecated item_cls argument https://review.opendev.org/c/openstack/nova/+/1002937 | |
| 11:34:13 | gibi | thanks | |
| 12:25:52 | opendevreview | Merged openstack/nova master: Replace deprecated remotable_classmethod https://review.opendev.org/c/openstack/nova/+/995370 | |
| 12:41:11 | opendevreview | Takashi Kajinami proposed openstack/nova master: ovo: Drop deprecated item_cls argument https://review.opendev.org/c/openstack/nova/+/1002937 | |
| 13:39:55 | clif | sean-k-mooney: is there anything I can do to help that perf patch merge? | |
| 14:11:59 | sean-k-mooney | did it fail after the rebase? im on pto today so not really paying attaentio, gibi is lookign at some stablity impovment | |
| 14:12:17 | sean-k-mooney | i think you just geting unlucky with the test failures | |
| 14:12:48 | sean-k-mooney | the db locked error is oen of the knwo issues withthe func tests under threading mode currently | |
| 14:13:07 | sean-k-mooney | and it was in an entirly differnt area of the code so not related to that patch | |
| 14:40:54 | clif | gotcha | |
| 14:41:05 | clif | sorry to ping you on your day off, enjoy | |
| 14:41:34 | sean-k-mooney | its fien i would not be on irc if it bothered me | |
| 15:30:10 | Uggla | Reminder: Upstream meeting in ~30mn. | |
| 16:02:47 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:02:47 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:02:47 | opendevmeet | Meeting started Mon Aug 31 16:02:47 2026 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:02:47 | Uggla | #startmeeting nova | |
| 16:02:56 | Uggla | Hello everyone | |
| 16:02:59 | bauzas | o/ | |
| 16:03:02 | gmaan | o/ | |
| 16:03:42 | elodilles | o/ | |
| 16:04:37 | lajoskatona | o/ | |
| 16:05:14 | Uggla | Let's start | |
| 16:05:22 | Uggla | #topic Bugs (stuck/critical) | |
| 16:05:32 | Uggla | #info No Critical bug | |
| 16:05:43 | Uggla | #topic Gate status | |
| 16:05:46 | tkajinam | o/ | |
| 16:05:52 | Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:06:01 | Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:06:12 | Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | |
| 16:06:23 | Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:06:32 | Uggla | #info Please try to provide a meaningful comment when you recheck | |
| 16:06:33 | gmaan | we ,merged a lot of fixes last week, I will remove the bugs from that etherpad one we alrady fixed | |
| 16:07:13 | Uggla | #info last week we had gate instability. Any progress? | |
| 16:07:54 | Uggla | gmaan 👍 | |
| 16:08:00 | tkajinam | I found a few strange errors this evening like dead workers or zombie process causing job failure, but I'm not sure if that appears consistently | |
| 16:08:11 | gmaan | we merged almost all the fixes up for the frequent bugs happening but I dod not check today how gate is behaving | |
| 16:08:12 | tkajinam | (these are not specific to nova but look like infra problems | |
| 16:08:23 | gmaan | seems like, no change in CI pipeline | |
| 16:09:03 | tkajinam | gmaan, https://review.opendev.org/c/openstack/nova/+/1002223 is the one which is supposed to fix unstable threading job, right ? | |
| 16:09:05 | gmaan | tkajinam: ack, if you see them agian, please log the bug and we can check that if it happens more frequently | |
| 16:09:29 | gmaan | tkajinam: yes, this is one of that. but there were 5-6 totals | |
| 16:09:35 | tkajinam | gmaan, yeah or I'll report it to infra team, because that's a infra problem rather than a nova bug I believe. | |
| 16:09:38 | tkajinam | ok. good | |
| 16:09:43 | gmaan | cool, thanks | |
| 16:09:52 | tkajinam | will be careful about threading unit tests this week | |
| 16:10:53 | Uggla | I think we can move on | |
| 16:11:12 | Uggla | #topic Release Planning | |
| 16:11:18 | Uggla | #link https://releases.openstack.org/hibiscus/schedule.html | |
| 16:11:20 | tkajinam | yup | |
| 16:11:29 | Uggla | #info Nova deadlines are set in the above schedule | |
| 16:11:37 | Uggla | #info Feature freeze was last week. | |
| 16:12:09 | Uggla | #topic Review priorities | |
| 16:12:22 | Uggla | #info Starting: https://etherpad.opendev.org/p/nova-2026.2-status#L16 interesting bugs to review. | |
| 16:13:32 | Uggla | virtiofs cold migration needs focus too. I focused on testing it last week. | |
| 16:14:08 | Uggla | #topic Stable Branches | |
| 16:14:25 | elodilles | thx | |
| 16:14:29 | elodilles | #info stable gates should be OK | |
| 16:14:51 | elodilles | though there were not so many patches against stable branches in the past week o:) | |
| 16:15:13 | elodilles | anyway, if you see any stable gate issue, then please ping and/or: | |
| 16:15:16 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:15:22 | elodilles | add here ^^^ | |
| 16:17:06 | Uggla | thx elodilles | |
| 16:17:40 | Uggla | skipping vmware topic fwiesel is still on PTO I guess | |
| 16:18:39 | Uggla | #topic Kamil's news about eventlet removal | |
| 16:19:09 | Uggla | sambork is not available today. But he left this info about eventlet: btw only update regarding eventlet is perf testing, so we're done there.Scaling down to 9 scheduler/conductor pods gave native threading a lower, peak-free, steady-state memory footprint (still marginally above eventlet), while CPU usage and timing stayed on par with eventlet | |
| 16:19:10 | Uggla | and occasionally beat it thanks to native low-level thread scheduling. Setting executor_thread_pool_size to 32 for both the scheduler and conductor kept the queue depth at ~1, making those two the only default config changes required | |
| 16:19:38 | gibi | + Last week we worked on stabilizing the functional-py313-threading job | |
| 16:20:17 | Uggla | oh gibi, I thought you were not part of the meeting. My bad. | |
| 16:20:35 | tkajinam | I like hearing such "the new model works better than the old model" news :-) | |
| 16:20:51 | gibi | I think we eventually need to reopen https://bugs.launchpad.net/nova/+bug/2165149 as it might still happne | |
| 16:21:03 | gibi | also I saw TIMED_OUTs from this job | |
| 16:21:08 | gibi | that needs to be investiaget | |
| 16:21:11 | gibi | investigated | |
| 16:21:31 | tkajinam | gibi, is that what you discussed with sean today ? | |
| 16:21:31 | Uggla | tkajinam ++ | |
| 16:22:17 | gibi | yepp | |
| 16:22:27 | gibi | or mostly we discussed the DB locked issue | |