Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-04-29
08:52:23 elodilles Uggla: just a quick double check: do you want to wait for reviews with the schedule patch (review + freeze days) or is it ready to merge already? https://review.opendev.org/c/openstack/releases/+/948077
09:02:30 gibi elodilles: Uggla is on PTO the whole week as far as I remember
09:04:22 elodilles ahh, right.
09:04:43 elodilles then I'll wait maybe until the meeting :)
09:21:00 opendevreview Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100
09:21:01 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140
10:05:39 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet.queue.LightQueue with queue.SimpleQueue https://review.opendev.org/c/openstack/nova/+/948417
10:57:28 gibi sean-k-mooney: bauzas: we need this to unblock the nova gate https://review.opendev.org/c/openstack/nova/+/948392
10:58:39 sean-k-mooney ...
10:59:03 sean-k-mooney damb i new trhis was going to happen but i hoped we would have a few more months
11:03:24 sean-k-mooney +2... i was hoping to keep watcher compaitble with 3.9 until we have an alternitive but if the requiremetns repo has droped validation fo 3.9 deps we are going to have to do something similar
11:03:48 gibi yeah unfortunately they pulled the trigger
11:04:16 gibi not having a py39 upper constraints causes version conflict in py39 jobs
11:05:17 sean-k-mooney ya so for wathcer ill create a patch to do the same. im not going to bump required_python initally but ill do that in a followup patch
11:05:28 sean-k-mooney that we will have to merge before the end of the cycle
11:06:16 sean-k-mooney our downstream container build does not use the latest version of lib that UC allows so i want to see if it can still build on 3.9 with those deps before i fully block it
11:06:23 sean-k-mooney but that is only a tempory hack.
11:06:52 sean-k-mooney actully by downstrema in this context i just mean the operator jobs whcih are technially upstream using continers built form rdo content
11:07:19 sean-k-mooney we are working on moving those to centos 10 stream instead which will upgrade them to 3.12
11:07:55 sean-k-mooney i was hoping ot have another month or two to finish that but its fine
11:15:42 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
11:15:42 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
11:16:14 gibi sean-k-mooney: yeah
11:29:52 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
11:29:53 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
11:51:37 sean-k-mooney https://review.opendev.org/c/openstack/devstack/+/948408/1/lib/nova#b1118 is fine but it still might be cleaner to have either the nova devstack plugin or the job create a droping file for the systemd serivce in the long term
11:52:17 sean-k-mooney gibi: if we ever land your patch to make the [[post-config|path]] thing work for non exisitent directories
11:52:28 sean-k-mooney we coudl do that via that in the job too
11:54:10 gibi sean-k-mooney: yeah the current devstack patch is a hardocded hack. I planning to have a devstack local conf variable to list the nova services that should run in threading mode then use that to pass the env var to systemd
11:54:13 sean-k-mooney what im kind of hoping we can avoid si a patch to devstack for every binary we need to swap between modes so im hopign we can come up with somethign generic and reusable
11:55:34 sean-k-mooney ya that woudl work. i was thinking more generic have a map of systemd unit name to a list of extr env vars
11:55:56 sean-k-mooney so it can work for any of the services so it woudl work for neutron or glance too
11:56:11 sean-k-mooney but we can keep it nova speicifc not to inflate the scope too much
12:16:33 gibi I will look into the generic mapping, that sounds useful
12:25:22 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
12:25:22 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
12:28:19 sean-k-mooney by the way could we land https://review.opendev.org/c/openstack/placement/+/945487 i would like to get that off my list of open patches while its not in merge conflict
12:31:19 gibi +2 from me
12:34:20 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140
12:35:26 gibi dansmith: sean-k-mooney: I had some local stress testing of the threaded scheduler. In the happy path scatter-gather worker threads are properly returned to the pool and reused (I scheduled 120 VMs with the pool of 20 threads x 2 worker processes).
12:37:30 sean-k-mooney cool and did that taek a lot of ram and or time or was it reasonable
12:37:39 sean-k-mooney im just askign for a gut feeling by the way
12:37:52 sean-k-mooney i assume that was not one multi create by the way
12:38:04 sean-k-mooney you did parallel requests
12:40:15 gibi the nova-scheduler workers are at 130MB RSS each
12:40:40 gibi I did parallel create requests not a single multicreate
13:04:40 gibi aand we have the first CI result where n-sch run with threading https://zuul.opendev.org/t/openstack/build/c2ec6bc4ae4946468d5b2b4f8468d12a/log/controller/logs/screen-n-sch.txt#2855-2856
13:26:12 dansmith gibi: nice
13:40:29 sean-k-mooney gibi: that all sounds pretty reasonable. out side of a small number of thread local varablie and stack frames we woudl not expect the worker size to grow much as we increase the thread outside of the data needed for the schduler ot schdule
13:40:54 sean-k-mooney so 130MB RSS per worker does not sound bad at all
13:43:56 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet semaphores with threading one https://review.opendev.org/c/openstack/nova/+/948437
13:58:26 gibi the nova-lvm job run 70 worker https://zuul.opendev.org/t/openstack/build/b5d70ad645514181a1fda52bba8187ce/log/controller/logs/screen-n-sch.txt#5099-5100 and n-sch had 120MB RSS https://zuul.opendev.org/t/openstack/build/b5d70ad645514181a1fda52bba8187ce/log/controller/logs/screen-memory_tracker.txt#2507-2508 on the gate
13:58:46 gibi 70 task with 20 worker threads
13:59:18 gibi 2 worker processes
14:00:47 sean-k-mooney how are you calulating the defautl therad pool size by the way. are you jsut levaing futurist decide currently
14:01:54 sean-k-mooney (failures=0, executed=70, runtime=1.90, cancelled=0)>
14:02:07 sean-k-mooney does that mean it ran for 1.9 sconds total?
14:02:32 sean-k-mooney i gues not
14:02:34 sean-k-mooney 0x7cab7c4de4c0 (failures=0, executed=69, runtime=2.33, cancelled=0)>
14:02:54 sean-k-mooney perhaps its since the last tiem you got the stats
14:03:43 gibi futurist decides today
14:03:48 gibi but we can have a config
14:04:03 sean-k-mooney i belvie it default to 2.5*nproc
14:04:15 gibi I can run some tests to see what is the runtime stats there
14:04:32 sean-k-mooney i reused one fo the existing config option but i would proably default this to 4
14:04:49 sean-k-mooney and allow peopel to overried if needed
14:04:50 gibi https://docs.openstack.org/futurist/latest/reference/index.html#futurist.ExecutorStatistics.runtime doc says total runtime of the pool
14:05:14 sean-k-mooney hum well it went down form 69 to 70
14:06:15 gibi we have two worker processes so you need to check the pid to see if this is the same worker
14:06:25 sean-k-mooney oh good point
14:06:32 sean-k-mooney are you usign https://docs.openstack.org/futurist/latest/reference/index.html#futurist.periodics.Watcher
14:06:34 gibi futurist creates cpu * 5 threads per executor by default https://github.com/openstack/futurist/blob/master/futurist/_utils.py#L126
14:06:57 sean-k-mooney gibi: ack thats proably more then we need
14:08:07 gibi I don't use periodics, I log the stats when new work is submitted and if the last report was older than 60 sec
14:08:22 gibi we can change it to periodic
14:08:33 gibi and or we can have a config option for the age
14:09:12 gibi yeah, 20 worker for scatter gather is too much, we can aim for 4 ish, but it depeds on the amount of parallel scheduling we need to serve
14:11:02 sean-k-mooney right but we have 2 ways to scale workers and treads in each workser thread pool. and we can make it configurable
14:11:43 sean-k-mooney well technially 3 you can just run more schdulers
14:12:36 sean-k-mooney i think tying the count to the cpu cores will bit ues down the line espcially if this is deployed on k8s as that could change based on where the pod runs so that why i would prefer to have a small integer default
14:16:55 gibi yeah I will expose a config option with a small default
14:17:11 gibi and document that you can ask for more worker processes as well as a scaling mechanism
14:19:47 dansmith I think it will take some time to figure out what the real default should be
14:20:16 dansmith greenthreads are nice, but they also kinda avoid the resource exhaustion question such that we don't really have a good idea of what it needs to be, IMHO
14:54:45 gibi yepp
14:54:51 sean-k-mooney gibi: https://zuul.opendev.org/t/openstack/buildset/f495c3661ca346a198c80a01387d6ba4 so that not terible for a first run
14:55:06 gibi nova-next is green that is enough for me to declare success
14:55:19 sean-k-mooney im not sure why grenade is unhappy
14:55:28 sean-k-mooney maybe the devstack change
14:55:32 gibi some tempest jobs does not have oslo.service as required project hence the oslo.service depends on is ignored
14:55:45 sean-k-mooney ah ya that too
14:56:49 gibi also nova-multi-cell is green
14:57:02 sean-k-mooney yep most of them are
14:57:20 opendevreview Dan Smith proposed openstack/nova master: WIP Example NVMe cleaning script for one-time-use https://review.opendev.org/c/openstack/nova/+/948446
14:57:52 sean-k-mooney gibi: the cyborg job is broken because they do not have pyproject.toml supprot so there wsgi script are not gerneated
14:57:54 gibi nova-multi-cell executed the most scatter-gathers and the nova-scheduler RSS is still around 122MB
14:58:24 gibi sean-k-mooney: yeah I ended up at the same conclusion this morning, as well as barbican and designate
14:58:30 sean-k-mooney im fixing that on watcher but that will impact other proejct shortly
14:58:36 gibi I filed bugs for barbican and cyborg, ralonsoh has bug for designate
14:58:54 sean-k-mooney ack there was a pip release 3 days ago

Earlier   Later