| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-07-30 | |||
| 15:08:15 | sean-k-mooney | jgilaber: you removed resting the cach | |
| 15:08:28 | sean-k-mooney | in https://review.opendev.org/c/openstack/nova/+/994299/10..13/nova/tests/functional/integrated_helpers.py | |
| 15:08:37 | melwitt | and there is a possibility that moving that part might change whatever is going wrong. BUT this is just my preliminary thoughts and I need to continue to look at it | |
| 15:08:45 | sean-k-mooney | you were clearign the cash but with the decoreateo you need ot call clear on the funtion | |
| 15:09:32 | jgilaber | right, but even then, that should only skip using the filter | |
| 15:09:49 | jgilaber | the scheduler should not hang because of that, right? | |
| 15:10:00 | sean-k-mooney | jgilaber: also what melwitt said but i woudl try clearing the cash like thi https://github.com/openstack/nova/blob/master/nova/test.py#L319 | |
| 15:10:36 | sean-k-mooney | well the filter is adding som eaddtional db calls | |
| 15:10:49 | sean-k-mooney | to get the min version | |
| 15:11:21 | sean-k-mooney | actully that is a bad exampel that using the run_once decortor not cache | |
| 15:12:02 | sean-k-mooney | https://docs.python.org/3/library/functools.html#:~:text=cache%5Fclear%28%29 | |
| 15:12:23 | sean-k-mooney | but ya we normally clear thsoe | |
| 15:12:25 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/test.py#L201 | |
| 15:13:45 | jgilaber | ack, thanks I'll try that | |
| 15:17:41 | sean-k-mooney | so lookign at the fialures more closely | |
| 15:18:28 | sean-k-mooney | the failrue seam to be in in the clean up | |
| 15:18:28 | jgilaber | melwitt, I'm trying to understand your comment, I'm still not too familiar with most of the nova code | |
| 15:18:45 | jgilaber | so you think that I should not add the version check in the request spec object? | |
| 15:19:56 | melwitt | jgilaber: I think it should be elsewhere yes but I will add detailed comments to the review on this once I go through it properly. I just wanted to give you a quick heads up, it is ok to ignore for now | |
| 15:20:19 | sean-k-mooney | melwitt: perhaps in the resouce translation logic | |
| 15:20:21 | melwitt | mainly I wanted to mention that there is a possibility it could be related to the random failures if there is something with the database session nesting, that's all | |
| 15:20:38 | jgilaber | ack, thanks, I'll respin with the cache_clear for now to see if that helps | |
| 15:20:48 | melwitt | sounds good | |
| 15:21:00 | jgilaber | thanks for looking! | |
| 15:22:47 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 15:22:47 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 15:22:53 | sean-k-mooney | so what failing initally is a server show request | |
| 15:23:08 | sean-k-mooney | GET /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 500 | |
| 15:24:23 | sean-k-mooney | but that is just us pooling for the delete | |
| 15:24:25 | sean-k-mooney | 2026-07-30 08:06:21,544 INFO [nova.api.openstack.requestlog] 127.0.0.1 "DELETE /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 204 len: 0 microversion: 2.104 time: 0.071912 | |
| 15:24:28 | sean-k-mooney | to complete | |
| 15:24:52 | melwitt | on one of the HTTP 500 errors I saw that it got stuck in a sql connection code and then timed out | |
| 15:25:05 | melwitt | that's what made me think about database | |
| 15:25:11 | sean-k-mooney | yes | |
| 15:25:19 | sean-k-mooney | its coming form | |
| 15:25:21 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 469, in show | |
| 15:25:23 | sean-k-mooney | server['server']['scheduler_hints'] = self._get_scheduler_hints( | |
| 15:25:25 | sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 15:25:27 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 256, in _get_scheduler_hints | |
| 15:25:29 | sean-k-mooney | req_spec = objects.RequestSpec.get_by_instance_uuid( | |
| 15:25:31 | sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 15:25:44 | sean-k-mooney | we are pullign the schduelr hinits because of the newer microver which is gettign the requst spec | |
| 15:26:16 | sean-k-mooney | and that call is getting stuck | |
| 15:26:25 | sean-k-mooney | but it might be because of os-brick | |
| 15:26:29 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py311/lib/python3.11/site-packages/os_brick/utils.py", line 47, in _sleep | |
| 15:26:31 | sean-k-mooney | _time_sleep(secs) | |
| 15:27:22 | sean-k-mooney | sqlachme is calling _thread_yeild which is time.sleep(0) then os-birck is calling time_sleep(secs) | |
| 15:28:09 | sean-k-mooney | and then we basily end up timign out, so not sure if osbick is the problem or not | |
| 15:28:24 | sean-k-mooney | im looking at test_resize_revert | |
| 15:28:29 | sean-k-mooney | in https://5bf0723027f4ff260a96-e8d1ca7bab9532ed61be7cb4aac9c3f9.ssl.cf5.rackcdn.com/openstack/1b76d14e3a7a48278e4d8b84027e546b/testr_results.html | |
| 15:29:12 | sean-k-mooney | when i see a time out after som eslep calls i wonder if that sleep was a real thread sleep or not | |
| 15:29:44 | sean-k-mooney | espically when we see warnding like | |
| 15:29:46 | sean-k-mooney | 2026-07-30 08:08:57,991 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 21.99 sec | |
| 15:30:52 | sean-k-mooney | jgilaber: im going to pull your patch onto my macmin and let it run the funcitonal tests in a loop and see if it repoduces locally | |
| 15:31:53 | melwitt | the job fail I looked at was nova-tox-functional-py311 which afaik is eventlet | |
| 15:31:53 | jgilaber | thank sean-k-mooney, it's running in CI now with the cache clear | |
| 15:38:36 | sean-k-mooney | ya i think it should be eventlet as well | |
| 15:40:03 | sean-k-mooney | melwitt: there are som condering job wehn its calling placement however | |
| 15:40:06 | sean-k-mooney | 2026-07-30 08:07:18,669 DEBUG [placement.requestlog] Starting request: 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" | |
| 15:40:08 | sean-k-mooney | 2026-07-30 08:08:12,129 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 31.64 sec | |
| 15:40:45 | sean-k-mooney | that get completed at | |
| 15:40:47 | sean-k-mooney | 2026-07-30 08:08:24,892 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" status: 200 len: 317 microversion: 1.28 | |
| 15:41:24 | sean-k-mooney | but while it was happeing there are waring about timeout and such | |
| 15:41:44 | sean-k-mooney | i.e. https://paste.opendev.org/show/bVL566Wf6dqJEmComlVI/ | |
| 15:42:26 | sean-k-mooney | like that cell itmout looks sus | |
| 15:42:28 | sean-k-mooney | [nova.context] Timed out waiting for response from cell efc8e2e2-e5fb-446c-b0c4-03451f6aedc1. Left the cell worker thread to finish in the background. | |
| 15:43:04 | sean-k-mooney | melwitt: im partly wondering if the placement fixture is blockign nvoa db accesse or something while its servicing a request? | |
| 15:45:01 | sean-k-mooney | melwitt: it defintly looks liek somethign related tot he db for sure | |
| 15:45:46 | melwitt | I'm thinking about the file-based sqlite database but I'm pretty sure that's only used when nova is not monkey patched | |
| 15:46:57 | melwitt | basically wondering if one of the changes to cell database fixtures may be related | |
| 15:48:26 | melwitt | yeah file-based is guarded by "if utils.concurrency_mode_threading():" at least. but I think some of the other fixture changes may have been across the board. don't really have reason to suspect it yet but just something to keep in mind | |
| 15:56:14 | melwitt | (maybe some sort of lock race in the fixture) | |
| 16:02:54 | sean-k-mooney | so orginally i wanted to use filebacked alwasy | |
| 16:03:16 | sean-k-mooney | that what i tought we were doing for the unit tests and its what we are doning in cybrog and watcher | |
| 16:03:37 | sean-k-mooney | but i know that there were some issue with placmeent to get that to work | |
| 16:04:06 | sean-k-mooney | sambork and gibi woudl know more | |
| 16:36:04 | sean-k-mooney | jgilaber: AttributeError: module 'nova.scheduler.request_filter' has no attribute 'cache_clear' | |
| 16:40:21 | sean-k-mooney | im runing the previous version instead to see if it fails locally | |
| 16:40:35 | jgilaber | oh I used the module instead of the function | |
| 16:40:48 | jgilaber | I'll fix it and move it to the base TestCase | |
| 16:44:07 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 16:44:08 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 16:47:31 | gibi | sean-k-mooney: melwitt: so yeah the intention is to have the file backed SQL only for the threading version of the jobs | |
| 16:48:19 | gibi | but in threading mode we do have a global lock today | |
| 16:48:36 | sean-k-mooney | is that global lock hsared btween nova an placmeent? | |
| 16:48:44 | sean-k-mooney | or one per service | |
| 16:48:45 | gibi | and we have a patch to change that to a per DB lock https://review.opendev.org/c/openstack/nova/+/992862 | |
| 16:48:57 | gibi | yeas it is a shared global lock I think | |
| 16:49:02 | sean-k-mooney | ack | |
| 16:49:11 | gibi | but only should be in place for the threading tests | |
| 16:49:13 | sean-k-mooney | so that could expalin some of the warnings and slowness | |
| 16:49:16 | gibi | threading version of the test jobs | |
| 16:49:24 | sean-k-mooney | oh right | |
| 16:49:31 | gibi | we can land https://review.opendev.org/c/openstack/nova/+/992862 and see if it helps | |
| 16:49:31 | sean-k-mooney | this was in the eventlet based job | |
| 16:50:17 | sean-k-mooney | i mena i can go review that now in either case | |
| 16:50:28 | gibi | we could add some tracing to the lock handling and see if that shows that we somehow introduced the lock to the eventlet job as well | |
| 16:50:30 | sean-k-mooney | runing jgilaber's patch locally its not repoduceing for me | |
| 16:51:35 | sean-k-mooney | hum | |
| 16:52:03 | sean-k-mooney | gibi: so this woudl be a good case to use the FiarLockGuarde i wrote | |