| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-29 | |||
| 01:53:23 | melwitt | looks like nova-tox-functional-py39 is failing 100% due to a package dependency conflict https://zuul.opendev.org/t/openstack/builds?job_name=nova-tox-functional-py39&project=openstack/nova | |
| 01:54:44 | melwitt | I also notice cyborg-tempest and barbican-tempest-plugin-simple-crypto started failing recently, 1-2 days ago | |
| 07:00:10 | opendevreview | Ivan Anfimov proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| 07:00:53 | gibi | melwitt: the cyborg-tempest failure is caused by cyborg-api not starting: --- no python application found, check your startup logs for errors --- | |
| 07:06:05 | gibi | melwitt: reported bug https://bugs.launchpad.net/openstack-cyborg/+bug/2109583 | |
| 07:07:24 | gibi | melwitt: barbican has very similar failure and similar data of breakdown | |
| 07:07:31 | gibi | so there should be a common cause | |
| 07:10:28 | gibi | filed a barbican bug too https://bugs.launchpad.net/barbican/+bug/2109584 | |
| 07:12:14 | gibi | it all started on 04-27 but I don't see devstack/tempest changes merged around that time | |
| 07:12:29 | gibi | there are requirements changes but nothing suspicious to me | |
| 07:15:23 | gibi | melwitt: but I think the reason functional-py39 fails is that we dropped that python version from global requirements. I will propose a patch to remove the job from nova | |
| 07:23:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:23:12 | gibi | melwitt, bauzas, sean-k-mooney ^^ | |
| 07:24:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:25:10 | gibi | this is needed to unblock the nova gate | |
| 07:32:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:32:51 | frickler | the former looks similar to what ralonsoh found https://bugs.launchpad.net/designate/+bug/2109577 | |
| 07:33:27 | gibi | frickler: indeed | |
| 07:34:01 | gibi | and also started failing on the 27th | |
| 07:34:22 | gibi | so probably the same common but unknown cause | |
| 07:35:02 | frickler | well setuptools==80 released on the 27th, looks very likely | |
| 07:35:36 | gibi | ohh | |
| 07:38:52 | ralonsoh | in designate, the wsgi binary is not being created | |
| 07:38:59 | ralonsoh | but I can´t reproduce it locally | |
| 07:39:33 | gibi | is it the missing pyproject.toml? | |
| 07:39:43 | gibi | 2025-04-29 05:43:24.960850 | controller | DEPRECATION: Legacy editable install of keystone==27.1.0.dev20 from file:///opt/stack/keystone (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using | |
| 07:39:49 | gibi | --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457 | |
| 07:39:53 | gibi | 7010 | |
| 07:40:15 | gibi | https://zuul.opendev.org/t/openstack/build/af7efda68d204f1c99f8eeef77359ebf/log/job-output.txt#7009 | |
| 07:40:35 | gibi | as setuptool 80 changelog says | |
| 07:40:36 | gibi | Develop command no longer uses easy_install, but instead defers execution to pip (which then will re-invoke Setuptools via PEP 517 to build the editable wheel). Most of the options to develop are dropped. This is the final warning before the command is dropped completely in a few months. Use-cases relying on ‘setup.py develop’ should pin to older Setuptools version or migrate to modern build | |
| 07:40:42 | gibi | tooling. (#4955) | |
| 07:40:43 | gibi | https://setuptools.pypa.io/en/stable/history.html | |
| 07:41:30 | gibi | yeah barbican does not have pyproject.toml but nova has | |
| 07:42:26 | gibi | designate neither | |
| 07:42:33 | ralonsoh | neutron doesn't have it | |
| 07:42:38 | ralonsoh | but seems to run fine | |
| 07:43:27 | gibi | maybe neutron does not use setup.py develop | |
| 07:44:23 | gibi | https://review.opendev.org/c/openstack/nova/+/899753 this added pyproject.toml to nova | |
| 07:45:35 | ralonsoh | but, if I'm not wrong, we use editable installs in neutron, at least in neutron-tempest-plugin | |
| 07:45:38 | ralonsoh | maybe I'm wrong | |
| 07:56:12 | johnsom | https://review.opendev.org/c/openstack/designate/+/902846 | |
| 07:57:28 | ralonsoh | johnsom, can you add this info in the designate bug? | |
| 07:58:04 | ralonsoh | johnsom, also this is blocking now the Neutron CI. Could be possible to have it rebased and reviewed? | |
| 07:58:26 | johnsom | Yea, I am about to turn in for the night, but can add a comment | |
| 07:58:58 | johnsom | I would hope so. Maybe also ping in the designate channel | |
| 07:59:09 | ralonsoh | johnsom, thanks a lot! | |
| 07:59:13 | frickler | I can take a look after a bunch of meetings that I've got coming up | |
| 07:59:47 | ralonsoh | gibi, ok, now I realize that Neutron is actually using the wsgi module and no longer using the script. This is why is working | |
| 08:03:30 | gibi | thanks folks | |
| 08:46:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 08:46:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 08:52:23 | elodilles | Uggla: just a quick double check: do you want to wait for reviews with the schedule patch (review + freeze days) or is it ready to merge already? https://review.opendev.org/c/openstack/releases/+/948077 | |
| 09:02:30 | gibi | elodilles: Uggla is on PTO the whole week as far as I remember | |
| 09:04:22 | elodilles | ahh, right. | |
| 09:04:43 | elodilles | then I'll wait maybe until the meeting :) | |
| 09:21:00 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 09:21:01 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 10:05:39 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet.queue.LightQueue with queue.SimpleQueue https://review.opendev.org/c/openstack/nova/+/948417 | |
| 10:57:28 | gibi | sean-k-mooney: bauzas: we need this to unblock the nova gate https://review.opendev.org/c/openstack/nova/+/948392 | |
| 10:58:39 | sean-k-mooney | ... | |
| 10:59:03 | sean-k-mooney | damb i new trhis was going to happen but i hoped we would have a few more months | |
| 11:03:24 | sean-k-mooney | +2... i was hoping to keep watcher compaitble with 3.9 until we have an alternitive but if the requiremetns repo has droped validation fo 3.9 deps we are going to have to do something similar | |
| 11:03:48 | gibi | yeah unfortunately they pulled the trigger | |
| 11:04:16 | gibi | not having a py39 upper constraints causes version conflict in py39 jobs | |
| 11:05:17 | sean-k-mooney | ya so for wathcer ill create a patch to do the same. im not going to bump required_python initally but ill do that in a followup patch | |
| 11:05:28 | sean-k-mooney | that we will have to merge before the end of the cycle | |
| 11:06:16 | sean-k-mooney | our downstream container build does not use the latest version of lib that UC allows so i want to see if it can still build on 3.9 with those deps before i fully block it | |
| 11:06:23 | sean-k-mooney | but that is only a tempory hack. | |
| 11:06:52 | sean-k-mooney | actully by downstrema in this context i just mean the operator jobs whcih are technially upstream using continers built form rdo content | |
| 11:07:19 | sean-k-mooney | we are working on moving those to centos 10 stream instead which will upgrade them to 3.12 | |
| 11:07:55 | sean-k-mooney | i was hoping ot have another month or two to finish that but its fine | |
| 11:15:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 11:15:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 11:16:14 | gibi | sean-k-mooney: yeah | |
| 11:29:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 11:29:53 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 11:51:37 | sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/948408/1/lib/nova#b1118 is fine but it still might be cleaner to have either the nova devstack plugin or the job create a droping file for the systemd serivce in the long term | |
| 11:52:17 | sean-k-mooney | gibi: if we ever land your patch to make the [[post-config|path]] thing work for non exisitent directories | |
| 11:52:28 | sean-k-mooney | we coudl do that via that in the job too | |
| 11:54:10 | gibi | sean-k-mooney: yeah the current devstack patch is a hardocded hack. I planning to have a devstack local conf variable to list the nova services that should run in threading mode then use that to pass the env var to systemd | |
| 11:54:13 | sean-k-mooney | what im kind of hoping we can avoid si a patch to devstack for every binary we need to swap between modes so im hopign we can come up with somethign generic and reusable | |
| 11:55:34 | sean-k-mooney | ya that woudl work. i was thinking more generic have a map of systemd unit name to a list of extr env vars | |
| 11:55:56 | sean-k-mooney | so it can work for any of the services so it woudl work for neutron or glance too | |
| 11:56:11 | sean-k-mooney | but we can keep it nova speicifc not to inflate the scope too much | |
| 12:16:33 | gibi | I will look into the generic mapping, that sounds useful | |
| 12:25:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 12:25:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 12:28:19 | sean-k-mooney | by the way could we land https://review.opendev.org/c/openstack/placement/+/945487 i would like to get that off my list of open patches while its not in merge conflict | |
| 12:31:19 | gibi | +2 from me | |
| 12:34:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 12:35:26 | gibi | dansmith: sean-k-mooney: I had some local stress testing of the threaded scheduler. In the happy path scatter-gather worker threads are properly returned to the pool and reused (I scheduled 120 VMs with the pool of 20 threads x 2 worker processes). | |
| 12:37:30 | sean-k-mooney | cool and did that taek a lot of ram and or time or was it reasonable | |
| 12:37:39 | sean-k-mooney | im just askign for a gut feeling by the way | |
| 12:37:52 | sean-k-mooney | i assume that was not one multi create by the way | |
| 12:38:04 | sean-k-mooney | you did parallel requests | |
| 12:40:15 | gibi | the nova-scheduler workers are at 130MB RSS each | |
| 12:40:40 | gibi | I did parallel create requests not a single multicreate | |
| 13:04:40 | gibi | aand we have the first CI result where n-sch run with threading https://zuul.opendev.org/t/openstack/build/c2ec6bc4ae4946468d5b2b4f8468d12a/log/controller/logs/screen-n-sch.txt#2855-2856 | |
| 13:26:12 | dansmith | gibi: nice | |