Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-07-08
16:24:01 sean-k-mooney but in your lamda case that should not be costly
16:24:16 gibi is it due to https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L103
16:24:19 gibi ?
16:24:27 gibi there is a clear 1 sec timeout passed there
16:25:00 sean-k-mooney that what i was wondering too
16:25:13 sean-k-mooney but that get is in blockign mode with a max of 1 second
16:25:26 sean-k-mooney if the queue has a value it wont wait for one second
16:26:06 sean-k-mooney i was wondering if you coudl jsut monkeypatch https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L142 to 0.1
16:26:14 sean-k-mooney if that woudl have any effect on the timeings
16:26:19 sean-k-mooney but i dont think it should
16:26:50 gibi yeah I'm about to try it :)
16:27:35 sean-k-mooney jsut looking at the code im not seeing nayting obviouly dumb like a hardcoded sleep or similar
16:29:28 sean-k-mooney so your tesitn calling shudown but do we call stop before we do that. i assume shutdown sill do that interenally
16:29:59 sean-k-mooney oh well stop is on the threadworker and shutdown is on the executor
16:30:04 gibi it is the problem
16:30:15 gibi if I set it to 0.1 then the shutdown is 10 times faster
16:30:33 gibi added a comment to the bug
16:31:30 gibi I will propose a patch top of Ashish's functional series to monkey patch that value in the test
16:31:32 sean-k-mooney oh
16:31:39 sean-k-mooney i know what happenign i think
16:31:51 sean-k-mooney so each tradwroker is waiting up to a secodn but we have 10 workers
16:32:05 sean-k-mooney and i bet when we call shutdown it loops over all of them serially
16:32:23 sean-k-mooney i.e. the executor is calling stop on each threqad worker one at a time
16:32:46 sean-k-mooney gibi: i bet if you double the worker count you will see it doubel coorect?
16:33:07 gibi nope I tried that adding more worker does not change the timing
16:34:01 sean-k-mooney huh weird
16:34:05 sean-k-mooney https://docs.python.org/3/library/queue.html#queue.Queue.get
16:35:41 sean-k-mooney it really is the upper bound
16:36:30 gibi but the queue is empty in this case as there is no work left. So it does wait 1 sec before it raises Empty
16:37:02 sean-k-mooney yes
16:37:02 gibi and the shutdown logic only checked in the empty exception handler
16:37:59 gibi so we need better logic. We need to wake up the queue during shutdown with a poison added to the queue
16:38:30 gibi or ignore this and just monkey patch the 1 sec to a small number from our test :) In production that 1 sec is totally OK during shutdown
16:39:33 sean-k-mooney https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L224-L240
16:40:23 sean-k-mooney gibi: so yes in the tests we can set ti to somethign smal for now
16:40:33 sean-k-mooney but that for over the woekers
16:40:53 sean-k-mooney where we call join is why i tought it would scale with teh worksers
16:41:45 gibi I think we wait that 1 sec per worker in parallel for each worker
16:41:50 opendevreview Merged openstack/nova master: Fix intermittent 409 in TestGracefulShutdown tests https://review.opendev.org/c/openstack/nova/+/996381
16:42:30 sean-k-mooney sortof each worker will do that
16:42:33 sean-k-mooney but https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L237
16:42:45 sean-k-mooney when we call join there is seriailised
16:44:00 sean-k-mooney so we are taking the shutdwon lock, setting studwo to ture and waiting for the queu to drain,
16:44:01 gibi yes but join is not the one that triggers the wait for 1 sec. Join is just checks if the given thread is finished or not. At the first join we wait for 1 sec, then we see that the first thread finished, at the second join we see that the second thread already finished. etc
16:44:08 sean-k-mooney then notifying all workser to stop
16:44:13 sean-k-mooney and finally joining them all
16:44:46 gibi we join serially but the thread wait for their own 1 sec in parallel
16:44:54 gibi *threads
16:46:32 sean-k-mooney i think your correct
16:46:48 sean-k-mooney im just not sure why we see the wait
16:47:29 gibi anyhow I will dig deeper tomorrow. The last thing I do today is put a patch up to nova to nuke this 1 sec in the test
16:57:11 opendevreview Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500
16:57:12 sean-k-mooney gibi: i might quickly hack on somehting, if i push anything ill add you to it and you can free free to udpate it or push your own fix
16:57:20 gibi OK thanks
16:57:40 sean-k-mooney im basiclly goign to ask ai to reporduce your finding in a test case and then see if it can rootcause it
16:57:42 gibi https://review.opendev.org/c/openstack/nova/+/996500 for me locally re-gains all the lost time in the threading mode
16:57:50 gibi sean-k-mooney: OK cool
16:58:22 opendevreview Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500
16:58:31 sean-k-mooney also cool lets see how that ^ goes in ci over night
16:58:38 gibi yepp
16:58:42 gibi dropping now
16:58:44 gibi see you tomorrow
17:42:56 opendevreview Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693
18:45:38 melwitt I have a patch that has been up for a while with one +2, to make QEMU_IMG_LIMITS configurable if anyone might be interested https://review.opendev.org/c/openstack/nova/+/969538 this came from an old upstream CI issue where the image limit was too low for an encrypted disk with ceph
20:22:24 sean-k-mooney melwitt: so while that helps
20:22:39 sean-k-mooney we proably shoudl fix the memory limit if we are doign this
20:22:56 sean-k-mooney melwitt: currently we are limiting on adress space but we shoudl be limiting on RSS
20:23:39 melwitt hm ok .. I am not yet familiar with that
20:24:06 sean-k-mooney address space is how much well adrsss space that is mapped into the process with or without actual memory usage
20:24:15 sean-k-mooney RSS is residnet set size
20:24:24 sean-k-mooney i.e. how much actual memory is being used
20:25:05 sean-k-mooney in ci with some config we were seing newver version fo ceph/rbd map alot of memory without actully usign a lot of memroy
20:25:40 melwitt oh I see
20:25:56 sean-k-mooney https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/processutils.py#L162
20:26:19 melwitt a-ha
20:27:15 melwitt so you think if we just s/address_space/resident_set_size/
20:27:43 melwitt looks like rss has been a choice as long as address space ... so it is not a newly available choice. yet it has not been used
20:27:51 melwitt I wonder if that was deliberate
20:28:26 sean-k-mooney i would just sed it yes
20:29:54 melwitt looks like no one is using it, huh. https://codesearch.openstack.org/?q=resident_set_size&i=nope&literal=nope&files=&excludeFiles=&repos=
20:31:25 sean-k-mooney so i moved the ceph jobs to debian to work around it https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/955714 and proposed a reviert of the tempest skip https://review.opendev.org/c/openstack/tempest/+/955177
20:31:54 sean-k-mooney but we never actully merge the tempest skip reviert
20:32:11 melwitt yeah, I remembered that. I had wondered what the "real fix" should be
20:33:48 sean-k-mooney i +1 your patch becasue we coudl proceed with it, i think movign to rss instead fo adresss space woudl make sense btu let see if gibi has or somoen else has an opion
20:34:20 sean-k-mooney melwitt: im not agaisnt makign it configurabl i just dont think that the correct limit to use in this case in general
20:34:31 sean-k-mooney but we coudl change that in a followup
20:35:14 melwitt sure. I'm also happy to just change it to resident_set_size if there is no potential bad thing about that haha
20:35:31 melwitt I am a bit wondering why no one has made use of it before
20:36:36 sean-k-mooney well almost nothign uses this in general https://codesearch.openstack.org/?q=ProcessLimits&i=nope&literal=nope&files=&excludeFiles=&repos=
20:36:45 sean-k-mooney and i think the code was cargo culted form nova to the rest
20:37:31 sean-k-mooney tweaking the adress space limit will allow more meory to be used like rss would
20:37:44 melwitt nova trend setter
20:37:47 sean-k-mooney but you may have to set it higher then otherwise
20:38:26 melwitt yeah. I mostly saw how cinder and ironic both doing it
20:38:44 sean-k-mooney you know what ill appove your current patch but lets chat about this more
20:38:53 sean-k-mooney your currrent patch matches cinder and ironics approch
20:39:06 sean-k-mooney so if it works for them it will proably be fine
20:39:11 melwitt it's ok, I don't mind chatting for rss
20:39:32 sean-k-mooney ok but lets check back tomorrow so we dont forget :)
20:40:12 melwitt I feel like dan would have a good input for this but I think he's away until next week

Earlier   Later