| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-07-30 | |||
| 13:37:29 | bauzas | it was queued | |
| 13:37:36 | bauzas | but not started AFAICS | |
| 13:39:27 | bauzas | with the controller, we see the thread awaken up here https://zuul.opendev.org/t/openstack/build/db30a8569196402898d754be9ce85426/log/job-output.txt#16760 | |
| 15:04:59 | jgilaber | melwitt, sean-k-mooney hi I'm seeing some failures on the functional test from my patch adding the OWNER_NOVA trait that I don't quite understand | |
| 15:05:33 | jgilaber | the tests worked fine in patchset 10, and after what I thought were mostly cosmetic change they are failing apparently randomly https://review.opendev.org/c/openstack/nova/+/994299/10..13 | |
| 15:05:52 | jgilaber | e.g https://zuul.opendev.org/t/openstack/build/46982818a76243dfa64f66aaa34c2ef3 only one test fails but I see no error | |
| 15:06:34 | jgilaber | and in the same buildset the py313 job failed with different tests https://zuul.opendev.org/t/openstack/build/46982818a76243dfa64f66aaa34c2ef3 | |
| 15:07:49 | jgilaber | sorry wrong link, I wanted to paste https://zuul.opendev.org/t/openstack/build/1b76d14e3a7a48278e4d8b84027e546b | |
| 15:07:55 | melwitt | jgilaber: I was just starting to look at that. so far I am suspecting something with database access (perhaps improper transaction context manager nesting or something of that nature). I began to look through the bottom patch and I'm not sure about the location of the minimum service version check and trait addition, my inclination is it shouldn't be in the request spec object there, | |
| 15:08:15 | sean-k-mooney | jgilaber: you removed resting the cach | |
| 15:08:28 | sean-k-mooney | in https://review.opendev.org/c/openstack/nova/+/994299/10..13/nova/tests/functional/integrated_helpers.py | |
| 15:08:37 | melwitt | and there is a possibility that moving that part might change whatever is going wrong. BUT this is just my preliminary thoughts and I need to continue to look at it | |
| 15:08:45 | sean-k-mooney | you were clearign the cash but with the decoreateo you need ot call clear on the funtion | |
| 15:09:32 | jgilaber | right, but even then, that should only skip using the filter | |
| 15:09:49 | jgilaber | the scheduler should not hang because of that, right? | |
| 15:10:00 | sean-k-mooney | jgilaber: also what melwitt said but i woudl try clearing the cash like thi https://github.com/openstack/nova/blob/master/nova/test.py#L319 | |
| 15:10:36 | sean-k-mooney | well the filter is adding som eaddtional db calls | |
| 15:10:49 | sean-k-mooney | to get the min version | |
| 15:11:21 | sean-k-mooney | actully that is a bad exampel that using the run_once decortor not cache | |
| 15:12:02 | sean-k-mooney | https://docs.python.org/3/library/functools.html#:~:text=cache%5Fclear%28%29 | |
| 15:12:23 | sean-k-mooney | but ya we normally clear thsoe | |
| 15:12:25 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/test.py#L201 | |
| 15:13:45 | jgilaber | ack, thanks I'll try that | |
| 15:17:41 | sean-k-mooney | so lookign at the fialures more closely | |
| 15:18:28 | sean-k-mooney | the failrue seam to be in in the clean up | |
| 15:18:28 | jgilaber | melwitt, I'm trying to understand your comment, I'm still not too familiar with most of the nova code | |
| 15:18:45 | jgilaber | so you think that I should not add the version check in the request spec object? | |
| 15:19:56 | melwitt | jgilaber: I think it should be elsewhere yes but I will add detailed comments to the review on this once I go through it properly. I just wanted to give you a quick heads up, it is ok to ignore for now | |
| 15:20:19 | sean-k-mooney | melwitt: perhaps in the resouce translation logic | |
| 15:20:21 | melwitt | mainly I wanted to mention that there is a possibility it could be related to the random failures if there is something with the database session nesting, that's all | |
| 15:20:38 | jgilaber | ack, thanks, I'll respin with the cache_clear for now to see if that helps | |
| 15:20:48 | melwitt | sounds good | |
| 15:21:00 | jgilaber | thanks for looking! | |
| 15:22:47 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 15:22:47 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 15:22:53 | sean-k-mooney | so what failing initally is a server show request | |
| 15:23:08 | sean-k-mooney | GET /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 500 | |
| 15:24:23 | sean-k-mooney | but that is just us pooling for the delete | |
| 15:24:25 | sean-k-mooney | 2026-07-30 08:06:21,544 INFO [nova.api.openstack.requestlog] 127.0.0.1 "DELETE /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 204 len: 0 microversion: 2.104 time: 0.071912 | |
| 15:24:28 | sean-k-mooney | to complete | |
| 15:24:52 | melwitt | on one of the HTTP 500 errors I saw that it got stuck in a sql connection code and then timed out | |
| 15:25:05 | melwitt | that's what made me think about database | |
| 15:25:11 | sean-k-mooney | yes | |
| 15:25:19 | sean-k-mooney | its coming form | |
| 15:25:21 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 469, in show | |
| 15:25:23 | sean-k-mooney | server['server']['scheduler_hints'] = self._get_scheduler_hints( | |
| 15:25:25 | sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 15:25:27 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 256, in _get_scheduler_hints | |
| 15:25:29 | sean-k-mooney | req_spec = objects.RequestSpec.get_by_instance_uuid( | |
| 15:25:31 | sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 15:25:44 | sean-k-mooney | we are pullign the schduelr hinits because of the newer microver which is gettign the requst spec | |
| 15:26:16 | sean-k-mooney | and that call is getting stuck | |
| 15:26:25 | sean-k-mooney | but it might be because of os-brick | |
| 15:26:29 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py311/lib/python3.11/site-packages/os_brick/utils.py", line 47, in _sleep | |
| 15:26:31 | sean-k-mooney | _time_sleep(secs) | |
| 15:27:22 | sean-k-mooney | sqlachme is calling _thread_yeild which is time.sleep(0) then os-birck is calling time_sleep(secs) | |
| 15:28:09 | sean-k-mooney | and then we basily end up timign out, so not sure if osbick is the problem or not | |
| 15:28:24 | sean-k-mooney | im looking at test_resize_revert | |
| 15:28:29 | sean-k-mooney | in https://5bf0723027f4ff260a96-e8d1ca7bab9532ed61be7cb4aac9c3f9.ssl.cf5.rackcdn.com/openstack/1b76d14e3a7a48278e4d8b84027e546b/testr_results.html | |
| 15:29:12 | sean-k-mooney | when i see a time out after som eslep calls i wonder if that sleep was a real thread sleep or not | |
| 15:29:44 | sean-k-mooney | espically when we see warnding like | |
| 15:29:46 | sean-k-mooney | 2026-07-30 08:08:57,991 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 21.99 sec | |
| 15:30:52 | sean-k-mooney | jgilaber: im going to pull your patch onto my macmin and let it run the funcitonal tests in a loop and see if it repoduces locally | |
| 15:31:53 | melwitt | the job fail I looked at was nova-tox-functional-py311 which afaik is eventlet | |
| 15:31:53 | jgilaber | thank sean-k-mooney, it's running in CI now with the cache clear | |
| 15:38:36 | sean-k-mooney | ya i think it should be eventlet as well | |
| 15:40:03 | sean-k-mooney | melwitt: there are som condering job wehn its calling placement however | |
| 15:40:06 | sean-k-mooney | 2026-07-30 08:07:18,669 DEBUG [placement.requestlog] Starting request: 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" | |
| 15:40:08 | sean-k-mooney | 2026-07-30 08:08:12,129 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 31.64 sec | |
| 15:40:45 | sean-k-mooney | that get completed at | |
| 15:40:47 | sean-k-mooney | 2026-07-30 08:08:24,892 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" status: 200 len: 317 microversion: 1.28 | |
| 15:41:24 | sean-k-mooney | but while it was happeing there are waring about timeout and such | |
| 15:41:44 | sean-k-mooney | i.e. https://paste.opendev.org/show/bVL566Wf6dqJEmComlVI/ | |
| 15:42:26 | sean-k-mooney | like that cell itmout looks sus | |
| 15:42:28 | sean-k-mooney | [nova.context] Timed out waiting for response from cell efc8e2e2-e5fb-446c-b0c4-03451f6aedc1. Left the cell worker thread to finish in the background. | |
| 15:43:04 | sean-k-mooney | melwitt: im partly wondering if the placement fixture is blockign nvoa db accesse or something while its servicing a request? | |
| 15:45:01 | sean-k-mooney | melwitt: it defintly looks liek somethign related tot he db for sure | |
| 15:45:46 | melwitt | I'm thinking about the file-based sqlite database but I'm pretty sure that's only used when nova is not monkey patched | |
| 15:46:57 | melwitt | basically wondering if one of the changes to cell database fixtures may be related | |
| 15:48:26 | melwitt | yeah file-based is guarded by "if utils.concurrency_mode_threading():" at least. but I think some of the other fixture changes may have been across the board. don't really have reason to suspect it yet but just something to keep in mind | |
| 15:56:14 | melwitt | (maybe some sort of lock race in the fixture) | |
| 16:02:54 | sean-k-mooney | so orginally i wanted to use filebacked alwasy | |
| 16:03:16 | sean-k-mooney | that what i tought we were doing for the unit tests and its what we are doning in cybrog and watcher | |
| 16:03:37 | sean-k-mooney | but i know that there were some issue with placmeent to get that to work | |
| 16:04:06 | sean-k-mooney | sambork and gibi woudl know more | |
| 16:36:04 | sean-k-mooney | jgilaber: AttributeError: module 'nova.scheduler.request_filter' has no attribute 'cache_clear' | |
| 16:40:21 | sean-k-mooney | im runing the previous version instead to see if it fails locally | |
| 16:40:35 | jgilaber | oh I used the module instead of the function | |
| 16:40:48 | jgilaber | I'll fix it and move it to the base TestCase | |
| 16:44:07 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 16:44:08 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 16:47:31 | gibi | sean-k-mooney: melwitt: so yeah the intention is to have the file backed SQL only for the threading version of the jobs | |
| 16:48:19 | gibi | but in threading mode we do have a global lock today | |
| 16:48:36 | sean-k-mooney | is that global lock hsared btween nova an placmeent? | |
| 16:48:44 | sean-k-mooney | or one per service | |
| 16:48:45 | gibi | and we have a patch to change that to a per DB lock https://review.opendev.org/c/openstack/nova/+/992862 | |
| 16:48:57 | gibi | yeas it is a shared global lock I think | |
| 16:49:02 | sean-k-mooney | ack | |
| 16:49:11 | gibi | but only should be in place for the threading tests | |
| 16:49:13 | sean-k-mooney | so that could expalin some of the warnings and slowness | |