Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-07-30
15:08:28 sean-k-mooney in https://review.opendev.org/c/openstack/nova/+/994299/10..13/nova/tests/functional/integrated_helpers.py
15:08:37 melwitt and there is a possibility that moving that part might change whatever is going wrong. BUT this is just my preliminary thoughts and I need to continue to look at it
15:08:45 sean-k-mooney you were clearign the cash but with the decoreateo you need ot call clear on the funtion
15:09:32 jgilaber right, but even then, that should only skip using the filter
15:09:49 jgilaber the scheduler should not hang because of that, right?
15:10:00 sean-k-mooney jgilaber: also what melwitt said but i woudl try clearing the cash like thi https://github.com/openstack/nova/blob/master/nova/test.py#L319
15:10:36 sean-k-mooney well the filter is adding som eaddtional db calls
15:10:49 sean-k-mooney to get the min version
15:11:21 sean-k-mooney actully that is a bad exampel that using the run_once decortor not cache
15:12:02 sean-k-mooney https://docs.python.org/3/library/functools.html#:~:text=cache%5Fclear%28%29
15:12:23 sean-k-mooney but ya we normally clear thsoe
15:12:25 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/test.py#L201
15:13:45 jgilaber ack, thanks I'll try that
15:17:41 sean-k-mooney so lookign at the fialures more closely
15:18:28 jgilaber melwitt, I'm trying to understand your comment, I'm still not too familiar with most of the nova code
15:18:28 sean-k-mooney the failrue seam to be in in the clean up
15:18:45 jgilaber so you think that I should not add the version check in the request spec object?
15:19:56 melwitt jgilaber: I think it should be elsewhere yes but I will add detailed comments to the review on this once I go through it properly. I just wanted to give you a quick heads up, it is ok to ignore for now
15:20:19 sean-k-mooney melwitt: perhaps in the resouce translation logic
15:20:21 melwitt mainly I wanted to mention that there is a possibility it could be related to the random failures if there is something with the database session nesting, that's all
15:20:38 jgilaber ack, thanks, I'll respin with the cache_clear for now to see if that helps
15:20:48 melwitt sounds good
15:21:00 jgilaber thanks for looking!
15:22:47 opendevreview Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299
15:22:47 opendevreview Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840
15:22:53 sean-k-mooney so what failing initally is a server show request
15:23:08 sean-k-mooney GET /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 500
15:24:23 sean-k-mooney but that is just us pooling for the delete
15:24:25 sean-k-mooney 2026-07-30 08:06:21,544 INFO [nova.api.openstack.requestlog] 127.0.0.1 "DELETE /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 204 len: 0 microversion: 2.104 time: 0.071912
15:24:28 sean-k-mooney to complete
15:24:52 melwitt on one of the HTTP 500 errors I saw that it got stuck in a sql connection code and then timed out
15:25:05 melwitt that's what made me think about database
15:25:11 sean-k-mooney yes
15:25:19 sean-k-mooney its coming form
15:25:21 sean-k-mooney File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 469, in show
15:25:23 sean-k-mooney server['server']['scheduler_hints'] = self._get_scheduler_hints(
15:25:25 sean-k-mooney ^^^^^^^^^^^^^^^^^^^^^^^^^^
15:25:27 sean-k-mooney File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 256, in _get_scheduler_hints
15:25:29 sean-k-mooney req_spec = objects.RequestSpec.get_by_instance_uuid(
15:25:31 sean-k-mooney ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:25:44 sean-k-mooney we are pullign the schduelr hinits because of the newer microver which is gettign the requst spec
15:26:16 sean-k-mooney and that call is getting stuck
15:26:25 sean-k-mooney but it might be because of os-brick
15:26:29 sean-k-mooney File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py311/lib/python3.11/site-packages/os_brick/utils.py", line 47, in _sleep
15:26:31 sean-k-mooney _time_sleep(secs)
15:27:22 sean-k-mooney sqlachme is calling _thread_yeild which is time.sleep(0) then os-birck is calling time_sleep(secs)
15:28:09 sean-k-mooney and then we basily end up timign out, so not sure if osbick is the problem or not
15:28:24 sean-k-mooney im looking at test_resize_revert
15:28:29 sean-k-mooney in https://5bf0723027f4ff260a96-e8d1ca7bab9532ed61be7cb4aac9c3f9.ssl.cf5.rackcdn.com/openstack/1b76d14e3a7a48278e4d8b84027e546b/testr_results.html
15:29:12 sean-k-mooney when i see a time out after som eslep calls i wonder if that sleep was a real thread sleep or not
15:29:44 sean-k-mooney espically when we see warnding like
15:29:46 sean-k-mooney 2026-07-30 08:08:57,991 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 21.99 sec
15:30:52 sean-k-mooney jgilaber: im going to pull your patch onto my macmin and let it run the funcitonal tests in a loop and see if it repoduces locally
15:31:53 jgilaber thank sean-k-mooney, it's running in CI now with the cache clear
15:31:53 melwitt the job fail I looked at was nova-tox-functional-py311 which afaik is eventlet
15:38:36 sean-k-mooney ya i think it should be eventlet as well
15:40:03 sean-k-mooney melwitt: there are som condering job wehn its calling placement however
15:40:06 sean-k-mooney 2026-07-30 08:07:18,669 DEBUG [placement.requestlog] Starting request: 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce"
15:40:08 sean-k-mooney 2026-07-30 08:08:12,129 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 31.64 sec
15:40:45 sean-k-mooney that get completed at
15:40:47 sean-k-mooney 2026-07-30 08:08:24,892 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" status: 200 len: 317 microversion: 1.28
15:41:24 sean-k-mooney but while it was happeing there are waring about timeout and such
15:41:44 sean-k-mooney i.e. https://paste.opendev.org/show/bVL566Wf6dqJEmComlVI/
15:42:26 sean-k-mooney like that cell itmout looks sus
15:42:28 sean-k-mooney [nova.context] Timed out waiting for response from cell efc8e2e2-e5fb-446c-b0c4-03451f6aedc1. Left the cell worker thread to finish in the background.
15:43:04 sean-k-mooney melwitt: im partly wondering if the placement fixture is blockign nvoa db accesse or something while its servicing a request?
15:45:01 sean-k-mooney melwitt: it defintly looks liek somethign related tot he db for sure
15:45:46 melwitt I'm thinking about the file-based sqlite database but I'm pretty sure that's only used when nova is not monkey patched
15:46:57 melwitt basically wondering if one of the changes to cell database fixtures may be related
15:48:26 melwitt yeah file-based is guarded by "if utils.concurrency_mode_threading():" at least. but I think some of the other fixture changes may have been across the board. don't really have reason to suspect it yet but just something to keep in mind
15:56:14 melwitt (maybe some sort of lock race in the fixture)
16:02:54 sean-k-mooney so orginally i wanted to use filebacked alwasy
16:03:16 sean-k-mooney that what i tought we were doing for the unit tests and its what we are doning in cybrog and watcher
16:03:37 sean-k-mooney but i know that there were some issue with placmeent to get that to work
16:04:06 sean-k-mooney sambork and gibi woudl know more
16:36:04 sean-k-mooney jgilaber: AttributeError: module 'nova.scheduler.request_filter' has no attribute 'cache_clear'
16:40:21 sean-k-mooney im runing the previous version instead to see if it fails locally
16:40:35 jgilaber oh I used the module instead of the function
16:40:48 jgilaber I'll fix it and move it to the base TestCase
16:44:07 opendevreview Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299
16:44:08 opendevreview Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840
16:47:31 gibi sean-k-mooney: melwitt: so yeah the intention is to have the file backed SQL only for the threading version of the jobs
16:48:19 gibi but in threading mode we do have a global lock today
16:48:36 sean-k-mooney is that global lock hsared btween nova an placmeent?
16:48:44 sean-k-mooney or one per service
16:48:45 gibi and we have a patch to change that to a per DB lock https://review.opendev.org/c/openstack/nova/+/992862
16:48:57 gibi yeas it is a shared global lock I think
16:49:02 sean-k-mooney ack
16:49:11 gibi but only should be in place for the threading tests
16:49:13 sean-k-mooney so that could expalin some of the warnings and slowness
16:49:16 gibi threading version of the test jobs
16:49:24 sean-k-mooney oh right
16:49:31 sean-k-mooney this was in the eventlet based job
16:49:31 gibi we can land https://review.opendev.org/c/openstack/nova/+/992862 and see if it helps
16:50:17 sean-k-mooney i mena i can go review that now in either case
16:50:28 gibi we could add some tracing to the lock handling and see if that shows that we somehow introduced the lock to the eventlet job as well
16:50:30 sean-k-mooney runing jgilaber's patch locally its not repoduceing for me
16:51:35 sean-k-mooney hum
16:52:03 sean-k-mooney gibi: so this woudl be a good case to use the FiarLockGuarde i wrote
16:52:49 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/utils.py#L1149

Earlier   Later