| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-07-30 | |||
| 15:29:46 | sean-k-mooney | 2026-07-30 08:08:57,991 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 21.99 sec | |
| 15:30:52 | sean-k-mooney | jgilaber: im going to pull your patch onto my macmin and let it run the funcitonal tests in a loop and see if it repoduces locally | |
| 15:31:53 | melwitt | the job fail I looked at was nova-tox-functional-py311 which afaik is eventlet | |
| 15:31:53 | jgilaber | thank sean-k-mooney, it's running in CI now with the cache clear | |
| 15:38:36 | sean-k-mooney | ya i think it should be eventlet as well | |
| 15:40:03 | sean-k-mooney | melwitt: there are som condering job wehn its calling placement however | |
| 15:40:06 | sean-k-mooney | 2026-07-30 08:07:18,669 DEBUG [placement.requestlog] Starting request: 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" | |
| 15:40:08 | sean-k-mooney | 2026-07-30 08:08:12,129 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 31.64 sec | |
| 15:40:45 | sean-k-mooney | that get completed at | |
| 15:40:47 | sean-k-mooney | 2026-07-30 08:08:24,892 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" status: 200 len: 317 microversion: 1.28 | |
| 15:41:24 | sean-k-mooney | but while it was happeing there are waring about timeout and such | |
| 15:41:44 | sean-k-mooney | i.e. https://paste.opendev.org/show/bVL566Wf6dqJEmComlVI/ | |
| 15:42:26 | sean-k-mooney | like that cell itmout looks sus | |
| 15:42:28 | sean-k-mooney | [nova.context] Timed out waiting for response from cell efc8e2e2-e5fb-446c-b0c4-03451f6aedc1. Left the cell worker thread to finish in the background. | |
| 15:43:04 | sean-k-mooney | melwitt: im partly wondering if the placement fixture is blockign nvoa db accesse or something while its servicing a request? | |
| 15:45:01 | sean-k-mooney | melwitt: it defintly looks liek somethign related tot he db for sure | |
| 15:45:46 | melwitt | I'm thinking about the file-based sqlite database but I'm pretty sure that's only used when nova is not monkey patched | |
| 15:46:57 | melwitt | basically wondering if one of the changes to cell database fixtures may be related | |
| 15:48:26 | melwitt | yeah file-based is guarded by "if utils.concurrency_mode_threading():" at least. but I think some of the other fixture changes may have been across the board. don't really have reason to suspect it yet but just something to keep in mind | |
| 15:56:14 | melwitt | (maybe some sort of lock race in the fixture) | |
| 16:02:54 | sean-k-mooney | so orginally i wanted to use filebacked alwasy | |
| 16:03:16 | sean-k-mooney | that what i tought we were doing for the unit tests and its what we are doning in cybrog and watcher | |
| 16:03:37 | sean-k-mooney | but i know that there were some issue with placmeent to get that to work | |
| 16:04:06 | sean-k-mooney | sambork and gibi woudl know more | |
| 16:36:04 | sean-k-mooney | jgilaber: AttributeError: module 'nova.scheduler.request_filter' has no attribute 'cache_clear' | |
| 16:40:21 | sean-k-mooney | im runing the previous version instead to see if it fails locally | |
| 16:40:35 | jgilaber | oh I used the module instead of the function | |
| 16:40:48 | jgilaber | I'll fix it and move it to the base TestCase | |
| 16:44:07 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 16:44:08 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 16:47:31 | gibi | sean-k-mooney: melwitt: so yeah the intention is to have the file backed SQL only for the threading version of the jobs | |
| 16:48:19 | gibi | but in threading mode we do have a global lock today | |
| 16:48:36 | sean-k-mooney | is that global lock hsared btween nova an placmeent? | |
| 16:48:44 | sean-k-mooney | or one per service | |
| 16:48:45 | gibi | and we have a patch to change that to a per DB lock https://review.opendev.org/c/openstack/nova/+/992862 | |
| 16:48:57 | gibi | yeas it is a shared global lock I think | |
| 16:49:02 | sean-k-mooney | ack | |
| 16:49:11 | gibi | but only should be in place for the threading tests | |
| 16:49:13 | sean-k-mooney | so that could expalin some of the warnings and slowness | |
| 16:49:16 | gibi | threading version of the test jobs | |
| 16:49:24 | sean-k-mooney | oh right | |
| 16:49:31 | gibi | we can land https://review.opendev.org/c/openstack/nova/+/992862 and see if it helps | |
| 16:49:31 | sean-k-mooney | this was in the eventlet based job | |
| 16:50:17 | sean-k-mooney | i mena i can go review that now in either case | |
| 16:50:28 | gibi | we could add some tracing to the lock handling and see if that shows that we somehow introduced the lock to the eventlet job as well | |
| 16:50:30 | sean-k-mooney | runing jgilaber's patch locally its not repoduceing for me | |
| 16:51:35 | sean-k-mooney | hum | |
| 16:52:03 | sean-k-mooney | gibi: so this woudl be a good case to use the FiarLockGuarde i wrote | |
| 16:52:49 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/utils.py#L1149 | |
| 16:59:48 | gibi | I'm not following how | |
| 17:00:12 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/992862/comment/6022582b_82867746/ | |
| 17:01:06 | gibi | but fairlockguard is to handle when multiple locks is being taken in an order | |
| 17:01:13 | gibi | here we only take one lock | |
| 17:01:16 | sean-k-mooney | no | |
| 17:01:22 | sean-k-mooney | that a usecase | |
| 17:01:38 | sean-k-mooney | it was intened to be used anytime you need to aquire the same named lock in diffent context | |
| 17:01:46 | sean-k-mooney | not just multipel locks | |
| 17:03:42 | gibi | I feel like here we have a simpler use case than what the fairlockguard is created for. Sure fairlockguard is also capable of doing this but to do this we don't necessary need the complexity of the fairlockguard | |
| 17:04:04 | sean-k-mooney | right but i wnated ot creat that so we would stop using one of version of it | |
| 17:04:24 | sean-k-mooney | this is reimpelnted a new lock guard | |
| 17:05:36 | sean-k-mooney | we dont have to use it i think it woudl be good to use it when we need soemthing like this in the futre | |
| 17:06:40 | gibi | it was a long day so I might not capable of seeing the whole picture any more but fairlockguard could be used in any place where a single lock is taken, but we don't want to replace all the locks in our code with a fairlockguard as that would be overkill | |
| 17:07:20 | gibi | fairlockguard shines where you have a list of lock to be taken | |
| 17:07:26 | gibi | and you want to do that safely | |
| 17:07:50 | gibi | if you have just a single lock to take at a time then it is overkill in my eyes | |
| 17:08:23 | sean-k-mooney | the bit i dont like is the fact your creating collections.defaultdict(threading.RLock) | |
| 17:08:31 | sean-k-mooney | and locking that collection access | |
| 17:08:48 | sean-k-mooney | that changed it form a singel lock to a set of related locks | |
| 17:09:40 | sean-k-mooney | even if any one invoction of the _locked_scope context manager is only useing one lock | |
| 17:12:40 | gibi | we could monkey patch the lock object into transaction_context_manager._root for each DB then the DatabaseWriteLock does not need to look up the lock in a dict that maps the context manager to a lock | |
| 17:13:15 | sean-k-mooney | we could im not aginst merging the patch as it is | |
| 17:13:24 | gibi | but as we already hooking into sqla here it would be extra complexity to hook this deep into it again at DB creation | |
| 17:13:47 | sean-k-mooney | but we spent a lot of effort addign nesting suprot and other infra in the fairlockgurard and testing to verify it for complex lockign cases like this | |
| 17:15:31 | gibi | but exactly those things are not needed in this case | |
| 17:15:37 | gibi | (I think) | |
| 17:15:59 | gibi | it is not such a complex case | |
| 17:16:00 | sean-k-mooney | you have muyltiple related locks, nesting / rentrancy | |
| 17:16:11 | sean-k-mooney | hence the callout why you needed to RLOCKS | |
| 17:16:16 | gibi | these locks are not related each is for its own DB | |
| 17:16:59 | gibi | they are not guarding the same shared resouce it is one lock - one DB | |
| 17:17:07 | sean-k-mooney | i know | |
| 17:17:28 | sean-k-mooney | but you all out thing like "An RLock is used per database because a single thread may nest writer | |
| 17:17:30 | sean-k-mooney | transactions against the same engine facade." | |
| 17:18:20 | gibi | and that is exactly what an RLock is for :) | |
| 17:18:48 | sean-k-mooney | and its also one of the thing build into the FairlockGurads constaitns | |
| 17:18:54 | sean-k-mooney | ill upgarde to a +2 for now | |
| 17:19:19 | sean-k-mooney | but if this proves unstabel i would prefrer ot adotp the FairLockGuard instead fo theying ot fix this implemetion | |
| 17:19:28 | sean-k-mooney | gibi: does that work for you | |
| 17:20:54 | gibi | if the instability is due to a problem that is solved by the fairlockguard then yes it is good for me | |
| 17:22:28 | sean-k-mooney | patchset 8 was created 21 days ago | |
| 17:22:43 | sean-k-mooney | so that before your shutdown speedup patch landed yes | |
| 17:23:11 | sean-k-mooney | so we woudl execte a new run to also be faster | |
| 17:23:47 | sean-k-mooney | or rather the thereaded and eventet version shoudl be closer now | |
| 17:41:23 | jgilaber | sean-k-mooney, so after clearing the cache the functional tests are green again https://zuul.opendev.org/t/openstack/buildset/6a7d3edc4ae84ba7b33487e037f1acc5 | |
| 17:41:50 | sean-k-mooney | jgilaber: with an without that change i was not able to cause it to fial locally | |
| 17:41:56 | sean-k-mooney | i still have it runingin the backgorund | |
| 17:42:20 | sean-k-mooney | but if ci is happy i guess that is progress | |
| 17:43:31 | jgilaber | yeah, I don't like that I don't know what was wrong there, but good to advance | |
| 17:44:15 | jgilaber | I guess CI harware is slower than our local machines and caused the problem to surface somehow | |
| 17:44:58 | sean-k-mooney | yes that happens | |