| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-28 | |||
| 14:59:26 | sean-k-mooney | i know there will be run to run variance but we can at least see if it massively changed things | |
| 14:59:42 | sean-k-mooney | it would be good to look at the memory usage form those runs too | |
| 15:00:06 | sean-k-mooney | im not expecting anything dramatic in either case but it will be a nice datapoint | |
| 15:01:06 | gibi | I need to extend devstack to be able to pass an env variable to nova-scheduler service from the zuul config to enable the threaded mode | |
| 15:01:27 | sean-k-mooney | ah well you could hardcode it for now to on | |
| 15:01:33 | gibi | ahh true | |
| 15:01:42 | gibi | a DNM patch with hardcoded logic | |
| 15:01:42 | sean-k-mooney | but ok ya we need to be able to pass it via the systemd service file | |
| 15:01:51 | gibi | ahh meeting :/ | |
| 15:30:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:37:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:39:43 | gibi | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/948311/comment/0de434cf_ac26e3db/ this should trigger CI with nova-scheduler running in threading mode. | |
| 15:40:42 | sean-k-mooney | dansmith: looks like it passed unit tests... at least on arm | |
| 15:42:31 | sean-k-mooney | it failed the functional test os at least we have some testign for that | |
| 15:42:49 | sean-k-mooney | i guess testing online data migration does make more sense to do in fucntional test | |
| 15:43:43 | sean-k-mooney | although we could have had a test that just mocked each of the migration funciton and asserted all of the expected one were called but we evidently dont | |
| 15:47:13 | sean-k-mooney | ... locks like the cyborg job i sbroken its not able to find the wsgi console script | |
| 15:47:42 | sean-k-mooney | partly becasue they have not converted to pyporject.toml + havign the wsgi entryp point being called via a module path | |
| 16:07:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 17:06:05 | sean-k-mooney | gibi: the ovs hybrid plug job passed so that promising https://zuul.opendev.org/t/openstack/build/4770c559c7514ad48463b7c198f31fa1 | |
| 17:07:06 | sean-k-mooney | https://zuul.opendev.org/t/openstack/buildset/1cd96c3f1a6e405eba4934bc783382db some of the other fast tempet test have also passed | |
| 17:08:47 | sean-k-mooney | gibi: is there any logmessage or anything like that that we can point to to knwo its in threadign mode | |
| 17:22:31 | sean-k-mooney | gibi: ah we shoudl see https://review.opendev.org/c/openstack/nova/+/948311/4/nova/monkey_patch.py#89 if we are monkey patched | |
| 17:22:53 | sean-k-mooney | and this warning if we are not https://review.opendev.org/c/openstack/nova/+/948311/4/nova/monkey_patch.py#119 | |
| 17:24:38 | sean-k-mooney | i do not see either printed | |
| 17:25:01 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 17:31:27 | gibi | sean-k-mooney: yeah I think somehow logging is not properly initialized. If I run then nova-scheduler in devstack from a terminal I see the warning. I will look into this. In parellel I put together a extra logging about the stats of our pools during use. | |
| 17:31:53 | sean-k-mooney | ack | |
| 17:32:08 | sean-k-mooney | when we are monkey patching i dont think we hav initalised oslo | |
| 17:32:15 | sean-k-mooney | or logign fully | |
| 17:32:28 | sean-k-mooney | * or rather logging fullyt | |
| 17:33:39 | sean-k-mooney | gibi: what prorbaly happening is this is likely going to stdout | |
| 17:33:56 | sean-k-mooney | because we have not setup the redirection to the journal yet | |
| 17:34:39 | sean-k-mooney | im not sure we have parsed or cofnig yet when we are monkey patching like this | |
| 17:35:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 17:36:07 | gibi | this will print semi periodically ^^ | |
| 17:36:16 | gibi | so we will see the type and the state of the executors | |
| 17:36:23 | sean-k-mooney | ack | |
| 17:37:12 | sean-k-mooney | ok that include the executor name in the stats | |
| 17:37:19 | sean-k-mooney | ya that shoudl work | |
| 17:37:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 17:38:14 | gibi | I stop here today | |
| 17:38:19 | gibi | it was a good day :) | |
| 17:39:39 | sean-k-mooney | gibi: ack o/ | |
| 17:39:53 | gibi | o/ | |
| 17:39:54 | sean-k-mooney | we shoudl have som enice logs to reviw in the morning | |
| 18:33:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 18:59:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 19:17:25 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 20:04:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 21:49:56 | opendevreview | Ivan Anfimov proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| #openstack-nova - 2025-04-29 | |||
| 01:18:59 | opendevreview | melanie witt proposed openstack/nova master: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/946600 | |
| 01:53:23 | melwitt | looks like nova-tox-functional-py39 is failing 100% due to a package dependency conflict https://zuul.opendev.org/t/openstack/builds?job_name=nova-tox-functional-py39&project=openstack/nova | |
| 01:54:44 | melwitt | I also notice cyborg-tempest and barbican-tempest-plugin-simple-crypto started failing recently, 1-2 days ago | |
| 07:00:10 | opendevreview | Ivan Anfimov proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| 07:00:53 | gibi | melwitt: the cyborg-tempest failure is caused by cyborg-api not starting: --- no python application found, check your startup logs for errors --- | |
| 07:06:05 | gibi | melwitt: reported bug https://bugs.launchpad.net/openstack-cyborg/+bug/2109583 | |
| 07:07:24 | gibi | melwitt: barbican has very similar failure and similar data of breakdown | |
| 07:07:31 | gibi | so there should be a common cause | |
| 07:10:28 | gibi | filed a barbican bug too https://bugs.launchpad.net/barbican/+bug/2109584 | |
| 07:12:14 | gibi | it all started on 04-27 but I don't see devstack/tempest changes merged around that time | |
| 07:12:29 | gibi | there are requirements changes but nothing suspicious to me | |
| 07:15:23 | gibi | melwitt: but I think the reason functional-py39 fails is that we dropped that python version from global requirements. I will propose a patch to remove the job from nova | |
| 07:23:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:23:12 | gibi | melwitt, bauzas, sean-k-mooney ^^ | |
| 07:24:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:25:10 | gibi | this is needed to unblock the nova gate | |
| 07:32:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:32:51 | frickler | the former looks similar to what ralonsoh found https://bugs.launchpad.net/designate/+bug/2109577 | |
| 07:33:27 | gibi | frickler: indeed | |
| 07:34:01 | gibi | and also started failing on the 27th | |
| 07:34:22 | gibi | so probably the same common but unknown cause | |
| 07:35:02 | frickler | well setuptools==80 released on the 27th, looks very likely | |
| 07:35:36 | gibi | ohh | |
| 07:38:52 | ralonsoh | in designate, the wsgi binary is not being created | |
| 07:38:59 | ralonsoh | but I can´t reproduce it locally | |
| 07:39:33 | gibi | is it the missing pyproject.toml? | |
| 07:39:43 | gibi | 2025-04-29 05:43:24.960850 | controller | DEPRECATION: Legacy editable install of keystone==27.1.0.dev20 from file:///opt/stack/keystone (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using | |
| 07:39:49 | gibi | --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457 | |
| 07:39:53 | gibi | 7010 | |
| 07:40:15 | gibi | https://zuul.opendev.org/t/openstack/build/af7efda68d204f1c99f8eeef77359ebf/log/job-output.txt#7009 | |
| 07:40:35 | gibi | as setuptool 80 changelog says | |
| 07:40:36 | gibi | Develop command no longer uses easy_install, but instead defers execution to pip (which then will re-invoke Setuptools via PEP 517 to build the editable wheel). Most of the options to develop are dropped. This is the final warning before the command is dropped completely in a few months. Use-cases relying on ‘setup.py develop’ should pin to older Setuptools version or migrate to modern build | |
| 07:40:42 | gibi | tooling. (#4955) | |
| 07:40:43 | gibi | https://setuptools.pypa.io/en/stable/history.html | |
| 07:41:30 | gibi | yeah barbican does not have pyproject.toml but nova has | |
| 07:42:26 | gibi | designate neither | |
| 07:42:33 | ralonsoh | neutron doesn't have it | |
| 07:42:38 | ralonsoh | but seems to run fine | |
| 07:43:27 | gibi | maybe neutron does not use setup.py develop | |
| 07:44:23 | gibi | https://review.opendev.org/c/openstack/nova/+/899753 this added pyproject.toml to nova | |
| 07:45:35 | ralonsoh | but, if I'm not wrong, we use editable installs in neutron, at least in neutron-tempest-plugin | |
| 07:45:38 | ralonsoh | maybe I'm wrong | |
| 07:56:12 | johnsom | https://review.opendev.org/c/openstack/designate/+/902846 | |
| 07:57:28 | ralonsoh | johnsom, can you add this info in the designate bug? | |
| 07:58:04 | ralonsoh | johnsom, also this is blocking now the Neutron CI. Could be possible to have it rebased and reviewed? | |
| 07:58:26 | johnsom | Yea, I am about to turn in for the night, but can add a comment | |
| 07:58:58 | johnsom | I would hope so. Maybe also ping in the designate channel | |
| 07:59:09 | ralonsoh | johnsom, thanks a lot! | |
| 07:59:13 | frickler | I can take a look after a bunch of meetings that I've got coming up | |