Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-04-29
01:54:44 melwitt I also notice cyborg-tempest and barbican-tempest-plugin-simple-crypto started failing recently, 1-2 days ago
07:00:10 opendevreview Ivan Anfimov proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426
07:00:53 gibi melwitt: the cyborg-tempest failure is caused by cyborg-api not starting: --- no python application found, check your startup logs for errors ---
07:06:05 gibi melwitt: reported bug https://bugs.launchpad.net/openstack-cyborg/+bug/2109583
07:07:24 gibi melwitt: barbican has very similar failure and similar data of breakdown
07:07:31 gibi so there should be a common cause
07:10:28 gibi filed a barbican bug too https://bugs.launchpad.net/barbican/+bug/2109584
07:12:14 gibi it all started on 04-27 but I don't see devstack/tempest changes merged around that time
07:12:29 gibi there are requirements changes but nothing suspicious to me
07:15:23 gibi melwitt: but I think the reason functional-py39 fails is that we dropped that python version from global requirements. I will propose a patch to remove the job from nova
07:23:01 opendevreview Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392
07:23:12 gibi melwitt, bauzas, sean-k-mooney ^^
07:24:48 opendevreview Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392
07:25:10 gibi this is needed to unblock the nova gate
07:32:39 opendevreview Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392
07:32:51 frickler the former looks similar to what ralonsoh found https://bugs.launchpad.net/designate/+bug/2109577
07:33:27 gibi frickler: indeed
07:34:01 gibi and also started failing on the 27th
07:34:22 gibi so probably the same common but unknown cause
07:35:02 frickler well setuptools==80 released on the 27th, looks very likely
07:35:36 gibi ohh
07:38:52 ralonsoh in designate, the wsgi binary is not being created
07:38:59 ralonsoh but I can´t reproduce it locally
07:39:33 gibi is it the missing pyproject.toml?
07:39:43 gibi 2025-04-29 05:43:24.960850 | controller | DEPRECATION: Legacy editable install of keystone==27.1.0.dev20 from file:///opt/stack/keystone (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using
07:39:49 gibi --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457
07:39:53 gibi 7010
07:40:15 gibi https://zuul.opendev.org/t/openstack/build/af7efda68d204f1c99f8eeef77359ebf/log/job-output.txt#7009
07:40:35 gibi as setuptool 80 changelog says
07:40:36 gibi Develop command no longer uses easy_install, but instead defers execution to pip (which then will re-invoke Setuptools via PEP 517 to build the editable wheel). Most of the options to develop are dropped. This is the final warning before the command is dropped completely in a few months. Use-cases relying on ‘setup.py develop’ should pin to older Setuptools version or migrate to modern build
07:40:42 gibi tooling. (#4955)
07:40:43 gibi https://setuptools.pypa.io/en/stable/history.html
07:41:30 gibi yeah barbican does not have pyproject.toml but nova has
07:42:26 gibi designate neither
07:42:33 ralonsoh neutron doesn't have it
07:42:38 ralonsoh but seems to run fine
07:43:27 gibi maybe neutron does not use setup.py develop
07:44:23 gibi https://review.opendev.org/c/openstack/nova/+/899753 this added pyproject.toml to nova
07:45:35 ralonsoh but, if I'm not wrong, we use editable installs in neutron, at least in neutron-tempest-plugin
07:45:38 ralonsoh maybe I'm wrong
07:56:12 johnsom https://review.opendev.org/c/openstack/designate/+/902846
07:57:28 ralonsoh johnsom, can you add this info in the designate bug?
07:58:04 ralonsoh johnsom, also this is blocking now the Neutron CI. Could be possible to have it rebased and reviewed?
07:58:26 johnsom Yea, I am about to turn in for the night, but can add a comment
07:58:58 johnsom I would hope so. Maybe also ping in the designate channel
07:59:09 ralonsoh johnsom, thanks a lot!
07:59:13 frickler I can take a look after a bunch of meetings that I've got coming up
07:59:47 ralonsoh gibi, ok, now I realize that Neutron is actually using the wsgi module and no longer using the script. This is why is working
08:03:30 gibi thanks folks
08:46:57 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
08:46:58 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
08:52:23 elodilles Uggla: just a quick double check: do you want to wait for reviews with the schedule patch (review + freeze days) or is it ready to merge already? https://review.opendev.org/c/openstack/releases/+/948077
09:02:30 gibi elodilles: Uggla is on PTO the whole week as far as I remember
09:04:22 elodilles ahh, right.
09:04:43 elodilles then I'll wait maybe until the meeting :)
09:21:00 opendevreview Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100
09:21:01 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140
10:05:39 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet.queue.LightQueue with queue.SimpleQueue https://review.opendev.org/c/openstack/nova/+/948417
10:57:28 gibi sean-k-mooney: bauzas: we need this to unblock the nova gate https://review.opendev.org/c/openstack/nova/+/948392
10:58:39 sean-k-mooney ...
10:59:03 sean-k-mooney damb i new trhis was going to happen but i hoped we would have a few more months
11:03:24 sean-k-mooney +2... i was hoping to keep watcher compaitble with 3.9 until we have an alternitive but if the requiremetns repo has droped validation fo 3.9 deps we are going to have to do something similar
11:03:48 gibi yeah unfortunately they pulled the trigger
11:04:16 gibi not having a py39 upper constraints causes version conflict in py39 jobs
11:05:17 sean-k-mooney ya so for wathcer ill create a patch to do the same. im not going to bump required_python initally but ill do that in a followup patch
11:05:28 sean-k-mooney that we will have to merge before the end of the cycle
11:06:16 sean-k-mooney our downstream container build does not use the latest version of lib that UC allows so i want to see if it can still build on 3.9 with those deps before i fully block it
11:06:23 sean-k-mooney but that is only a tempory hack.
11:06:52 sean-k-mooney actully by downstrema in this context i just mean the operator jobs whcih are technially upstream using continers built form rdo content
11:07:19 sean-k-mooney we are working on moving those to centos 10 stream instead which will upgrade them to 3.12
11:07:55 sean-k-mooney i was hoping ot have another month or two to finish that but its fine
11:15:42 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
11:15:42 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
11:16:14 gibi sean-k-mooney: yeah
11:29:52 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
11:29:53 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
11:51:37 sean-k-mooney https://review.opendev.org/c/openstack/devstack/+/948408/1/lib/nova#b1118 is fine but it still might be cleaner to have either the nova devstack plugin or the job create a droping file for the systemd serivce in the long term
11:52:17 sean-k-mooney gibi: if we ever land your patch to make the [[post-config|path]] thing work for non exisitent directories
11:52:28 sean-k-mooney we coudl do that via that in the job too
11:54:10 gibi sean-k-mooney: yeah the current devstack patch is a hardocded hack. I planning to have a devstack local conf variable to list the nova services that should run in threading mode then use that to pass the env var to systemd
11:54:13 sean-k-mooney what im kind of hoping we can avoid si a patch to devstack for every binary we need to swap between modes so im hopign we can come up with somethign generic and reusable
11:55:34 sean-k-mooney ya that woudl work. i was thinking more generic have a map of systemd unit name to a list of extr env vars
11:55:56 sean-k-mooney so it can work for any of the services so it woudl work for neutron or glance too
11:56:11 sean-k-mooney but we can keep it nova speicifc not to inflate the scope too much
12:16:33 gibi I will look into the generic mapping, that sounds useful
12:25:22 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
12:25:22 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
12:28:19 sean-k-mooney by the way could we land https://review.opendev.org/c/openstack/placement/+/945487 i would like to get that off my list of open patches while its not in merge conflict
12:31:19 gibi +2 from me
12:34:20 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140
12:35:26 gibi dansmith: sean-k-mooney: I had some local stress testing of the threaded scheduler. In the happy path scatter-gather worker threads are properly returned to the pool and reused (I scheduled 120 VMs with the pool of 20 threads x 2 worker processes).
12:37:30 sean-k-mooney cool and did that taek a lot of ram and or time or was it reasonable
12:37:39 sean-k-mooney im just askign for a gut feeling by the way
12:37:52 sean-k-mooney i assume that was not one multi create by the way
12:38:04 sean-k-mooney you did parallel requests
12:40:15 gibi the nova-scheduler workers are at 130MB RSS each
12:40:40 gibi I did parallel create requests not a single multicreate
13:04:40 gibi aand we have the first CI result where n-sch run with threading https://zuul.opendev.org/t/openstack/build/c2ec6bc4ae4946468d5b2b4f8468d12a/log/controller/logs/screen-n-sch.txt#2855-2856
13:26:12 dansmith gibi: nice
13:40:29 sean-k-mooney gibi: that all sounds pretty reasonable. out side of a small number of thread local varablie and stack frames we woudl not expect the worker size to grow much as we increase the thread outside of the data needed for the schduler ot schdule

Earlier   Later