| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-29 | |||
| 07:32:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:32:51 | frickler | the former looks similar to what ralonsoh found https://bugs.launchpad.net/designate/+bug/2109577 | |
| 07:33:27 | gibi | frickler: indeed | |
| 07:34:01 | gibi | and also started failing on the 27th | |
| 07:34:22 | gibi | so probably the same common but unknown cause | |
| 07:35:02 | frickler | well setuptools==80 released on the 27th, looks very likely | |
| 07:35:36 | gibi | ohh | |
| 07:38:52 | ralonsoh | in designate, the wsgi binary is not being created | |
| 07:38:59 | ralonsoh | but I can´t reproduce it locally | |
| 07:39:33 | gibi | is it the missing pyproject.toml? | |
| 07:39:43 | gibi | 2025-04-29 05:43:24.960850 | controller | DEPRECATION: Legacy editable install of keystone==27.1.0.dev20 from file:///opt/stack/keystone (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using | |
| 07:39:49 | gibi | --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457 | |
| 07:39:53 | gibi | 7010 | |
| 07:40:15 | gibi | https://zuul.opendev.org/t/openstack/build/af7efda68d204f1c99f8eeef77359ebf/log/job-output.txt#7009 | |
| 07:40:35 | gibi | as setuptool 80 changelog says | |
| 07:40:36 | gibi | Develop command no longer uses easy_install, but instead defers execution to pip (which then will re-invoke Setuptools via PEP 517 to build the editable wheel). Most of the options to develop are dropped. This is the final warning before the command is dropped completely in a few months. Use-cases relying on ‘setup.py develop’ should pin to older Setuptools version or migrate to modern build | |
| 07:40:42 | gibi | tooling. (#4955) | |
| 07:40:43 | gibi | https://setuptools.pypa.io/en/stable/history.html | |
| 07:41:30 | gibi | yeah barbican does not have pyproject.toml but nova has | |
| 07:42:26 | gibi | designate neither | |
| 07:42:33 | ralonsoh | neutron doesn't have it | |
| 07:42:38 | ralonsoh | but seems to run fine | |
| 07:43:27 | gibi | maybe neutron does not use setup.py develop | |
| 07:44:23 | gibi | https://review.opendev.org/c/openstack/nova/+/899753 this added pyproject.toml to nova | |
| 07:45:35 | ralonsoh | but, if I'm not wrong, we use editable installs in neutron, at least in neutron-tempest-plugin | |
| 07:45:38 | ralonsoh | maybe I'm wrong | |
| 07:56:12 | johnsom | https://review.opendev.org/c/openstack/designate/+/902846 | |
| 07:57:28 | ralonsoh | johnsom, can you add this info in the designate bug? | |
| 07:58:04 | ralonsoh | johnsom, also this is blocking now the Neutron CI. Could be possible to have it rebased and reviewed? | |
| 07:58:26 | johnsom | Yea, I am about to turn in for the night, but can add a comment | |
| 07:58:58 | johnsom | I would hope so. Maybe also ping in the designate channel | |
| 07:59:09 | ralonsoh | johnsom, thanks a lot! | |
| 07:59:13 | frickler | I can take a look after a bunch of meetings that I've got coming up | |
| 07:59:47 | ralonsoh | gibi, ok, now I realize that Neutron is actually using the wsgi module and no longer using the script. This is why is working | |
| 08:03:30 | gibi | thanks folks | |
| 08:46:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 08:46:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 08:52:23 | elodilles | Uggla: just a quick double check: do you want to wait for reviews with the schedule patch (review + freeze days) or is it ready to merge already? https://review.opendev.org/c/openstack/releases/+/948077 | |
| 09:02:30 | gibi | elodilles: Uggla is on PTO the whole week as far as I remember | |
| 09:04:22 | elodilles | ahh, right. | |
| 09:04:43 | elodilles | then I'll wait maybe until the meeting :) | |
| 09:21:00 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 09:21:01 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 10:05:39 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet.queue.LightQueue with queue.SimpleQueue https://review.opendev.org/c/openstack/nova/+/948417 | |
| 10:57:28 | gibi | sean-k-mooney: bauzas: we need this to unblock the nova gate https://review.opendev.org/c/openstack/nova/+/948392 | |
| 10:58:39 | sean-k-mooney | ... | |
| 10:59:03 | sean-k-mooney | damb i new trhis was going to happen but i hoped we would have a few more months | |
| 11:03:24 | sean-k-mooney | +2... i was hoping to keep watcher compaitble with 3.9 until we have an alternitive but if the requiremetns repo has droped validation fo 3.9 deps we are going to have to do something similar | |
| 11:03:48 | gibi | yeah unfortunately they pulled the trigger | |
| 11:04:16 | gibi | not having a py39 upper constraints causes version conflict in py39 jobs | |
| 11:05:17 | sean-k-mooney | ya so for wathcer ill create a patch to do the same. im not going to bump required_python initally but ill do that in a followup patch | |
| 11:05:28 | sean-k-mooney | that we will have to merge before the end of the cycle | |
| 11:06:16 | sean-k-mooney | our downstream container build does not use the latest version of lib that UC allows so i want to see if it can still build on 3.9 with those deps before i fully block it | |
| 11:06:23 | sean-k-mooney | but that is only a tempory hack. | |
| 11:06:52 | sean-k-mooney | actully by downstrema in this context i just mean the operator jobs whcih are technially upstream using continers built form rdo content | |
| 11:07:19 | sean-k-mooney | we are working on moving those to centos 10 stream instead which will upgrade them to 3.12 | |
| 11:07:55 | sean-k-mooney | i was hoping ot have another month or two to finish that but its fine | |
| 11:15:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 11:15:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 11:16:14 | gibi | sean-k-mooney: yeah | |
| 11:29:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 11:29:53 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 11:51:37 | sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/948408/1/lib/nova#b1118 is fine but it still might be cleaner to have either the nova devstack plugin or the job create a droping file for the systemd serivce in the long term | |
| 11:52:17 | sean-k-mooney | gibi: if we ever land your patch to make the [[post-config|path]] thing work for non exisitent directories | |
| 11:52:28 | sean-k-mooney | we coudl do that via that in the job too | |
| 11:54:10 | gibi | sean-k-mooney: yeah the current devstack patch is a hardocded hack. I planning to have a devstack local conf variable to list the nova services that should run in threading mode then use that to pass the env var to systemd | |
| 11:54:13 | sean-k-mooney | what im kind of hoping we can avoid si a patch to devstack for every binary we need to swap between modes so im hopign we can come up with somethign generic and reusable | |
| 11:55:34 | sean-k-mooney | ya that woudl work. i was thinking more generic have a map of systemd unit name to a list of extr env vars | |
| 11:55:56 | sean-k-mooney | so it can work for any of the services so it woudl work for neutron or glance too | |
| 11:56:11 | sean-k-mooney | but we can keep it nova speicifc not to inflate the scope too much | |
| 12:16:33 | gibi | I will look into the generic mapping, that sounds useful | |
| 12:25:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 12:25:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 12:28:19 | sean-k-mooney | by the way could we land https://review.opendev.org/c/openstack/placement/+/945487 i would like to get that off my list of open patches while its not in merge conflict | |
| 12:31:19 | gibi | +2 from me | |
| 12:34:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 12:35:26 | gibi | dansmith: sean-k-mooney: I had some local stress testing of the threaded scheduler. In the happy path scatter-gather worker threads are properly returned to the pool and reused (I scheduled 120 VMs with the pool of 20 threads x 2 worker processes). | |
| 12:37:30 | sean-k-mooney | cool and did that taek a lot of ram and or time or was it reasonable | |
| 12:37:39 | sean-k-mooney | im just askign for a gut feeling by the way | |
| 12:37:52 | sean-k-mooney | i assume that was not one multi create by the way | |
| 12:38:04 | sean-k-mooney | you did parallel requests | |
| 12:40:15 | gibi | the nova-scheduler workers are at 130MB RSS each | |
| 12:40:40 | gibi | I did parallel create requests not a single multicreate | |
| 13:04:40 | gibi | aand we have the first CI result where n-sch run with threading https://zuul.opendev.org/t/openstack/build/c2ec6bc4ae4946468d5b2b4f8468d12a/log/controller/logs/screen-n-sch.txt#2855-2856 | |
| 13:26:12 | dansmith | gibi: nice | |
| 13:40:29 | sean-k-mooney | gibi: that all sounds pretty reasonable. out side of a small number of thread local varablie and stack frames we woudl not expect the worker size to grow much as we increase the thread outside of the data needed for the schduler ot schdule | |
| 13:40:54 | sean-k-mooney | so 130MB RSS per worker does not sound bad at all | |
| 13:43:56 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet semaphores with threading one https://review.opendev.org/c/openstack/nova/+/948437 | |
| 13:58:26 | gibi | the nova-lvm job run 70 worker https://zuul.opendev.org/t/openstack/build/b5d70ad645514181a1fda52bba8187ce/log/controller/logs/screen-n-sch.txt#5099-5100 and n-sch had 120MB RSS https://zuul.opendev.org/t/openstack/build/b5d70ad645514181a1fda52bba8187ce/log/controller/logs/screen-memory_tracker.txt#2507-2508 on the gate | |
| 13:58:46 | gibi | 70 task with 20 worker threads | |
| 13:59:18 | gibi | 2 worker processes | |
| 14:00:47 | sean-k-mooney | how are you calulating the defautl therad pool size by the way. are you jsut levaing futurist decide currently | |
| 14:01:54 | sean-k-mooney | (failures=0, executed=70, runtime=1.90, cancelled=0)> | |
| 14:02:07 | sean-k-mooney | does that mean it ran for 1.9 sconds total? | |
| 14:02:32 | sean-k-mooney | i gues not | |
| 14:02:34 | sean-k-mooney | 0x7cab7c4de4c0 (failures=0, executed=69, runtime=2.33, cancelled=0)> | |
| 14:02:54 | sean-k-mooney | perhaps its since the last tiem you got the stats | |
| 14:03:43 | gibi | futurist decides today | |
| 14:03:48 | gibi | but we can have a config | |
| 14:04:03 | sean-k-mooney | i belvie it default to 2.5*nproc | |