| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-28 | |||
| 14:54:33 | sean-k-mooney | gibi: basicaly this https://github.com/openstack/nova/blob/master/nova/rpc.py#L220-L226 | |
| 14:54:36 | gibi | sean-k-mooney: not yet explicitly. I thought that is automatic | |
| 14:54:38 | gibi | ohh | |
| 14:54:46 | sean-k-mooney | it is if we dont pass one to use | |
| 14:54:48 | gibi | hehe | |
| 14:54:50 | sean-k-mooney | but nova does | |
| 14:54:52 | gibi | then that is the next step | |
| 14:55:00 | gibi | thanks for the pointer | |
| 14:55:09 | sean-k-mooney | i woudl personally prefer to continue to pass one explcitly | |
| 14:55:19 | sean-k-mooney | but in theory either works | |
| 14:55:44 | sean-k-mooney | i think the automatic switch can fail in some cases but im unsure | |
| 14:57:00 | gibi | woot, scheduling works. VM booted with threaded scheduler | |
| 14:57:07 | sean-k-mooney | nice | |
| 14:58:04 | gibi | now I have a long list of cleanups and a long list of testing to make in the series while the oslo folks cleans up the their patch | |
| 14:58:55 | sean-k-mooney | if you can patch the new revision with that chagne we can get some ci ressult and see if 1 it work in ci and 2 if there is any obvious timing differences | |
| 14:59:26 | sean-k-mooney | i know there will be run to run variance but we can at least see if it massively changed things | |
| 14:59:42 | sean-k-mooney | it would be good to look at the memory usage form those runs too | |
| 15:00:06 | sean-k-mooney | im not expecting anything dramatic in either case but it will be a nice datapoint | |
| 15:01:06 | gibi | I need to extend devstack to be able to pass an env variable to nova-scheduler service from the zuul config to enable the threaded mode | |
| 15:01:27 | sean-k-mooney | ah well you could hardcode it for now to on | |
| 15:01:33 | gibi | ahh true | |
| 15:01:42 | gibi | a DNM patch with hardcoded logic | |
| 15:01:42 | sean-k-mooney | but ok ya we need to be able to pass it via the systemd service file | |
| 15:01:51 | gibi | ahh meeting :/ | |
| 15:30:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:37:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:39:43 | gibi | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/948311/comment/0de434cf_ac26e3db/ this should trigger CI with nova-scheduler running in threading mode. | |
| 15:40:42 | sean-k-mooney | dansmith: looks like it passed unit tests... at least on arm | |
| 15:42:31 | sean-k-mooney | it failed the functional test os at least we have some testign for that | |
| 15:42:49 | sean-k-mooney | i guess testing online data migration does make more sense to do in fucntional test | |
| 15:43:43 | sean-k-mooney | although we could have had a test that just mocked each of the migration funciton and asserted all of the expected one were called but we evidently dont | |
| 15:47:13 | sean-k-mooney | ... locks like the cyborg job i sbroken its not able to find the wsgi console script | |
| 15:47:42 | sean-k-mooney | partly becasue they have not converted to pyporject.toml + havign the wsgi entryp point being called via a module path | |
| 16:07:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 17:06:05 | sean-k-mooney | gibi: the ovs hybrid plug job passed so that promising https://zuul.opendev.org/t/openstack/build/4770c559c7514ad48463b7c198f31fa1 | |
| 17:07:06 | sean-k-mooney | https://zuul.opendev.org/t/openstack/buildset/1cd96c3f1a6e405eba4934bc783382db some of the other fast tempet test have also passed | |
| 17:08:47 | sean-k-mooney | gibi: is there any logmessage or anything like that that we can point to to knwo its in threadign mode | |
| 17:22:31 | sean-k-mooney | gibi: ah we shoudl see https://review.opendev.org/c/openstack/nova/+/948311/4/nova/monkey_patch.py#89 if we are monkey patched | |
| 17:22:53 | sean-k-mooney | and this warning if we are not https://review.opendev.org/c/openstack/nova/+/948311/4/nova/monkey_patch.py#119 | |
| 17:24:38 | sean-k-mooney | i do not see either printed | |
| 17:25:01 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 17:31:27 | gibi | sean-k-mooney: yeah I think somehow logging is not properly initialized. If I run then nova-scheduler in devstack from a terminal I see the warning. I will look into this. In parellel I put together a extra logging about the stats of our pools during use. | |
| 17:31:53 | sean-k-mooney | ack | |
| 17:32:08 | sean-k-mooney | when we are monkey patching i dont think we hav initalised oslo | |
| 17:32:15 | sean-k-mooney | or logign fully | |
| 17:32:28 | sean-k-mooney | * or rather logging fullyt | |
| 17:33:39 | sean-k-mooney | gibi: what prorbaly happening is this is likely going to stdout | |
| 17:33:56 | sean-k-mooney | because we have not setup the redirection to the journal yet | |
| 17:34:39 | sean-k-mooney | im not sure we have parsed or cofnig yet when we are monkey patching like this | |
| 17:35:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 17:36:07 | gibi | this will print semi periodically ^^ | |
| 17:36:16 | gibi | so we will see the type and the state of the executors | |
| 17:36:23 | sean-k-mooney | ack | |
| 17:37:12 | sean-k-mooney | ok that include the executor name in the stats | |
| 17:37:19 | sean-k-mooney | ya that shoudl work | |
| 17:37:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 17:38:14 | gibi | I stop here today | |
| 17:38:19 | gibi | it was a good day :) | |
| 17:39:39 | sean-k-mooney | gibi: ack o/ | |
| 17:39:53 | gibi | o/ | |
| 17:39:54 | sean-k-mooney | we shoudl have som enice logs to reviw in the morning | |
| 18:33:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 18:59:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 19:17:25 | opendevreview | Dan Smith proposed openstack/nova master: Remove buggy VIF migration from Stein https://review.opendev.org/c/openstack/nova/+/948328 | |
| 20:04:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 21:49:56 | opendevreview | Ivan Anfimov proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| #openstack-nova - 2025-04-29 | |||
| 01:18:59 | opendevreview | melanie witt proposed openstack/nova master: live migration: Avoid volume rollback mismatches https://review.opendev.org/c/openstack/nova/+/946600 | |
| 01:53:23 | melwitt | looks like nova-tox-functional-py39 is failing 100% due to a package dependency conflict https://zuul.opendev.org/t/openstack/builds?job_name=nova-tox-functional-py39&project=openstack/nova | |
| 01:54:44 | melwitt | I also notice cyborg-tempest and barbican-tempest-plugin-simple-crypto started failing recently, 1-2 days ago | |
| 07:00:10 | opendevreview | Ivan Anfimov proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| 07:00:53 | gibi | melwitt: the cyborg-tempest failure is caused by cyborg-api not starting: --- no python application found, check your startup logs for errors --- | |
| 07:06:05 | gibi | melwitt: reported bug https://bugs.launchpad.net/openstack-cyborg/+bug/2109583 | |
| 07:07:24 | gibi | melwitt: barbican has very similar failure and similar data of breakdown | |
| 07:07:31 | gibi | so there should be a common cause | |
| 07:10:28 | gibi | filed a barbican bug too https://bugs.launchpad.net/barbican/+bug/2109584 | |
| 07:12:14 | gibi | it all started on 04-27 but I don't see devstack/tempest changes merged around that time | |
| 07:12:29 | gibi | there are requirements changes but nothing suspicious to me | |
| 07:15:23 | gibi | melwitt: but I think the reason functional-py39 fails is that we dropped that python version from global requirements. I will propose a patch to remove the job from nova | |
| 07:23:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:23:12 | gibi | melwitt, bauzas, sean-k-mooney ^^ | |
| 07:24:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:25:10 | gibi | this is needed to unblock the nova gate | |
| 07:32:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove python 3.9 support https://review.opendev.org/c/openstack/nova/+/948392 | |
| 07:32:51 | frickler | the former looks similar to what ralonsoh found https://bugs.launchpad.net/designate/+bug/2109577 | |
| 07:33:27 | gibi | frickler: indeed | |
| 07:34:01 | gibi | and also started failing on the 27th | |
| 07:34:22 | gibi | so probably the same common but unknown cause | |
| 07:35:02 | frickler | well setuptools==80 released on the 27th, looks very likely | |
| 07:35:36 | gibi | ohh | |
| 07:38:52 | ralonsoh | in designate, the wsgi binary is not being created | |
| 07:38:59 | ralonsoh | but I can´t reproduce it locally | |
| 07:39:33 | gibi | is it the missing pyproject.toml? | |
| 07:39:43 | gibi | 2025-04-29 05:43:24.960850 | controller | DEPRECATION: Legacy editable install of keystone==27.1.0.dev20 from file:///opt/stack/keystone (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using | |
| 07:39:49 | gibi | --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457 | |
| 07:39:53 | gibi | 7010 | |
| 07:40:15 | gibi | https://zuul.opendev.org/t/openstack/build/af7efda68d204f1c99f8eeef77359ebf/log/job-output.txt#7009 | |
| 07:40:35 | gibi | as setuptool 80 changelog says | |
| 07:40:36 | gibi | Develop command no longer uses easy_install, but instead defers execution to pip (which then will re-invoke Setuptools via PEP 517 to build the editable wheel). Most of the options to develop are dropped. This is the final warning before the command is dropped completely in a few months. Use-cases relying on ‘setup.py develop’ should pin to older Setuptools version or migrate to modern build | |
| 07:40:42 | gibi | tooling. (#4955) | |
| 07:40:43 | gibi | https://setuptools.pypa.io/en/stable/history.html | |