| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-07-30 | |||
| 15:20:19 | sean-k-mooney | melwitt: perhaps in the resouce translation logic | |
| 15:20:21 | melwitt | mainly I wanted to mention that there is a possibility it could be related to the random failures if there is something with the database session nesting, that's all | |
| 15:20:38 | jgilaber | ack, thanks, I'll respin with the cache_clear for now to see if that helps | |
| 15:20:48 | melwitt | sounds good | |
| 15:21:00 | jgilaber | thanks for looking! | |
| 15:22:47 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 15:22:47 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 15:22:53 | sean-k-mooney | so what failing initally is a server show request | |
| 15:23:08 | sean-k-mooney | GET /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 500 | |
| 15:24:23 | sean-k-mooney | but that is just us pooling for the delete | |
| 15:24:25 | sean-k-mooney | 2026-07-30 08:06:21,544 INFO [nova.api.openstack.requestlog] 127.0.0.1 "DELETE /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 204 len: 0 microversion: 2.104 time: 0.071912 | |
| 15:24:28 | sean-k-mooney | to complete | |
| 15:24:52 | melwitt | on one of the HTTP 500 errors I saw that it got stuck in a sql connection code and then timed out | |
| 15:25:05 | melwitt | that's what made me think about database | |
| 15:25:11 | sean-k-mooney | yes | |
| 15:25:19 | sean-k-mooney | its coming form | |
| 15:25:21 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 469, in show | |
| 15:25:23 | sean-k-mooney | server['server']['scheduler_hints'] = self._get_scheduler_hints( | |
| 15:25:25 | sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 15:25:27 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 256, in _get_scheduler_hints | |
| 15:25:29 | sean-k-mooney | req_spec = objects.RequestSpec.get_by_instance_uuid( | |
| 15:25:31 | sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 15:25:44 | sean-k-mooney | we are pullign the schduelr hinits because of the newer microver which is gettign the requst spec | |
| 15:26:16 | sean-k-mooney | and that call is getting stuck | |
| 15:26:25 | sean-k-mooney | but it might be because of os-brick | |
| 15:26:29 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py311/lib/python3.11/site-packages/os_brick/utils.py", line 47, in _sleep | |
| 15:26:31 | sean-k-mooney | _time_sleep(secs) | |
| 15:27:22 | sean-k-mooney | sqlachme is calling _thread_yeild which is time.sleep(0) then os-birck is calling time_sleep(secs) | |
| 15:28:09 | sean-k-mooney | and then we basily end up timign out, so not sure if osbick is the problem or not | |
| 15:28:24 | sean-k-mooney | im looking at test_resize_revert | |
| 15:28:29 | sean-k-mooney | in https://5bf0723027f4ff260a96-e8d1ca7bab9532ed61be7cb4aac9c3f9.ssl.cf5.rackcdn.com/openstack/1b76d14e3a7a48278e4d8b84027e546b/testr_results.html | |
| 15:29:12 | sean-k-mooney | when i see a time out after som eslep calls i wonder if that sleep was a real thread sleep or not | |
| 15:29:44 | sean-k-mooney | espically when we see warnding like | |
| 15:29:46 | sean-k-mooney | 2026-07-30 08:08:57,991 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 21.99 sec | |
| 15:30:52 | sean-k-mooney | jgilaber: im going to pull your patch onto my macmin and let it run the funcitonal tests in a loop and see if it repoduces locally | |
| 15:31:53 | melwitt | the job fail I looked at was nova-tox-functional-py311 which afaik is eventlet | |
| 15:31:53 | jgilaber | thank sean-k-mooney, it's running in CI now with the cache clear | |
| 15:38:36 | sean-k-mooney | ya i think it should be eventlet as well | |
| 15:40:03 | sean-k-mooney | melwitt: there are som condering job wehn its calling placement however | |
| 15:40:06 | sean-k-mooney | 2026-07-30 08:07:18,669 DEBUG [placement.requestlog] Starting request: 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" | |
| 15:40:08 | sean-k-mooney | 2026-07-30 08:08:12,129 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 31.64 sec | |
| 15:40:45 | sean-k-mooney | that get completed at | |
| 15:40:47 | sean-k-mooney | 2026-07-30 08:08:24,892 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" status: 200 len: 317 microversion: 1.28 | |
| 15:41:24 | sean-k-mooney | but while it was happeing there are waring about timeout and such | |
| 15:41:44 | sean-k-mooney | i.e. https://paste.opendev.org/show/bVL566Wf6dqJEmComlVI/ | |
| 15:42:26 | sean-k-mooney | like that cell itmout looks sus | |
| 15:42:28 | sean-k-mooney | [nova.context] Timed out waiting for response from cell efc8e2e2-e5fb-446c-b0c4-03451f6aedc1. Left the cell worker thread to finish in the background. | |
| 15:43:04 | sean-k-mooney | melwitt: im partly wondering if the placement fixture is blockign nvoa db accesse or something while its servicing a request? | |
| 15:45:01 | sean-k-mooney | melwitt: it defintly looks liek somethign related tot he db for sure | |
| 15:45:46 | melwitt | I'm thinking about the file-based sqlite database but I'm pretty sure that's only used when nova is not monkey patched | |
| 15:46:57 | melwitt | basically wondering if one of the changes to cell database fixtures may be related | |
| 15:48:26 | melwitt | yeah file-based is guarded by "if utils.concurrency_mode_threading():" at least. but I think some of the other fixture changes may have been across the board. don't really have reason to suspect it yet but just something to keep in mind | |
| 15:56:14 | melwitt | (maybe some sort of lock race in the fixture) | |
| 16:02:54 | sean-k-mooney | so orginally i wanted to use filebacked alwasy | |
| 16:03:16 | sean-k-mooney | that what i tought we were doing for the unit tests and its what we are doning in cybrog and watcher | |
| 16:03:37 | sean-k-mooney | but i know that there were some issue with placmeent to get that to work | |
| 16:04:06 | sean-k-mooney | sambork and gibi woudl know more | |
| 16:36:04 | sean-k-mooney | jgilaber: AttributeError: module 'nova.scheduler.request_filter' has no attribute 'cache_clear' | |
| 16:40:21 | sean-k-mooney | im runing the previous version instead to see if it fails locally | |
| 16:40:35 | jgilaber | oh I used the module instead of the function | |
| 16:40:48 | jgilaber | I'll fix it and move it to the base TestCase | |
| 16:44:07 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 16:44:08 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 16:47:31 | gibi | sean-k-mooney: melwitt: so yeah the intention is to have the file backed SQL only for the threading version of the jobs | |
| 16:48:19 | gibi | but in threading mode we do have a global lock today | |
| 16:48:36 | sean-k-mooney | is that global lock hsared btween nova an placmeent? | |
| 16:48:44 | sean-k-mooney | or one per service | |
| 16:48:45 | gibi | and we have a patch to change that to a per DB lock https://review.opendev.org/c/openstack/nova/+/992862 | |
| 16:48:57 | gibi | yeas it is a shared global lock I think | |
| 16:49:02 | sean-k-mooney | ack | |
| 16:49:11 | gibi | but only should be in place for the threading tests | |
| 16:49:13 | sean-k-mooney | so that could expalin some of the warnings and slowness | |
| 16:49:16 | gibi | threading version of the test jobs | |
| 16:49:24 | sean-k-mooney | oh right | |
| 16:49:31 | gibi | we can land https://review.opendev.org/c/openstack/nova/+/992862 and see if it helps | |
| 16:49:31 | sean-k-mooney | this was in the eventlet based job | |
| 16:50:17 | sean-k-mooney | i mena i can go review that now in either case | |
| 16:50:28 | gibi | we could add some tracing to the lock handling and see if that shows that we somehow introduced the lock to the eventlet job as well | |
| 16:50:30 | sean-k-mooney | runing jgilaber's patch locally its not repoduceing for me | |
| 16:51:35 | sean-k-mooney | hum | |
| 16:52:03 | sean-k-mooney | gibi: so this woudl be a good case to use the FiarLockGuarde i wrote | |
| 16:52:49 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/utils.py#L1149 | |
| 16:59:48 | gibi | I'm not following how | |
| 17:00:12 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/992862/comment/6022582b_82867746/ | |
| 17:01:06 | gibi | but fairlockguard is to handle when multiple locks is being taken in an order | |
| 17:01:13 | gibi | here we only take one lock | |
| 17:01:16 | sean-k-mooney | no | |
| 17:01:22 | sean-k-mooney | that a usecase | |
| 17:01:38 | sean-k-mooney | it was intened to be used anytime you need to aquire the same named lock in diffent context | |
| 17:01:46 | sean-k-mooney | not just multipel locks | |
| 17:03:42 | gibi | I feel like here we have a simpler use case than what the fairlockguard is created for. Sure fairlockguard is also capable of doing this but to do this we don't necessary need the complexity of the fairlockguard | |
| 17:04:04 | sean-k-mooney | right but i wnated ot creat that so we would stop using one of version of it | |
| 17:04:24 | sean-k-mooney | this is reimpelnted a new lock guard | |
| 17:05:36 | sean-k-mooney | we dont have to use it i think it woudl be good to use it when we need soemthing like this in the futre | |
| 17:06:40 | gibi | it was a long day so I might not capable of seeing the whole picture any more but fairlockguard could be used in any place where a single lock is taken, but we don't want to replace all the locks in our code with a fairlockguard as that would be overkill | |
| 17:07:20 | gibi | fairlockguard shines where you have a list of lock to be taken | |
| 17:07:26 | gibi | and you want to do that safely | |
| 17:07:50 | gibi | if you have just a single lock to take at a time then it is overkill in my eyes | |
| 17:08:23 | sean-k-mooney | the bit i dont like is the fact your creating collections.defaultdict(threading.RLock) | |
| 17:08:31 | sean-k-mooney | and locking that collection access | |