| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-30 | |||
| 12:55:47 | zigo | Except Eventlet and Python 3.13 ! | |
| 12:55:51 | zigo | I hope we can find a solution. | |
| 12:55:57 | sean-k-mooney | zigo: i was going to ask | |
| 12:55:59 | zigo | Not sure if Herve is still working on that. | |
| 12:56:08 | sean-k-mooney | i tought 3.13 was supper broken | |
| 12:56:29 | zigo | I haven't had a chance to test it yet. | |
| 12:56:38 | sean-k-mooney | well maybe not super broken but the whoel thread id stuff was broken so... | |
| 12:56:40 | zigo | That's on my plate for when I return from holidays. | |
| 12:56:57 | zigo | I need all in Trixie to be able to test though. | |
| 12:57:00 | sean-k-mooney | zigo: so as far as i know eventlet work well enough to pass our unit test and funcitonal | |
| 12:57:09 | zigo | Oh, ok. | |
| 12:57:16 | sean-k-mooney | but fails in reality when mokeypatching due to some changies in the threading module | |
| 12:57:24 | bauzas | gibi: cool, thanks for the info, I'm now myself hardly working on mtty effort | |
| 12:57:25 | zigo | I do unit tests for all packages at build time and autopkgtest, so I know that at least isn't broken. | |
| 12:57:32 | sean-k-mooney | not sure if that is still a the case | |
| 12:57:37 | zigo | What I'm missing is a full setup + tempest run. | |
| 12:57:44 | sean-k-mooney | btu that is where it was at before the ptg | |
| 12:58:06 | zigo | With my Bookworm backports setup, I have 1900+ tempest tests passing. | |
| 12:58:21 | zigo | I'll see how far I can go with Trixie. | |
| 12:58:23 | sean-k-mooney | zigo: ack. that all promising | |
| 12:58:59 | sean-k-mooney | that remind me i need to go update our pre-commit versions for py3.13 | |
| 13:00:01 | sean-k-mooney | zigo:anyway thanks for looking at this while on vacation but please dont spend too much time on it | |
| 13:00:41 | zigo | No worries. | |
| 13:01:35 | zigo | If I put the novnc package somewhere, can you test it? | |
| 13:01:47 | zigo | I did test 1.6.0 already... | |
| 13:02:00 | zigo | Worked, though had to fix some CSS. | |
| 13:02:21 | sean-k-mooney | frickler: ^ would you be able to test it in the test env you had set up | |
| 13:02:58 | frickler | I've now installed novnc from source already, better setup a new host I guess | |
| 13:03:00 | sean-k-mooney | zigo: i could proably install it into a devstack too i just dont have one set up for this at the moment | |
| 13:03:30 | zigo | http://shade.infomaniak.ch/novnc_1.6.0-2_all.deb | |
| 13:03:37 | frickler | (that is a held node, not a local env) | |
| 13:03:42 | zigo | It's ok, I'll just upload like that ... | |
| 13:05:34 | zigo | uploaded | |
| 13:06:05 | zigo | To get the fixed novnc in Ubuntu, you may have to ping some Canonical guys ... jamespage (in here) for example! :) | |
| 13:06:31 | sean-k-mooney | ack, it would be nice to fix in 24.04 | |
| 13:06:41 | zigo | jamespage: Could you get novnc 1:1.6.0-2 pushed to Ubuntu? | |
| 13:07:04 | zigo | sean-k-mooney: Not sure they will push a new upstream release to an already released Ubuntu though, but maybe in their PPA. | |
| 13:07:21 | sean-k-mooney | ya we have a workaround anywya | |
| 13:07:26 | sean-k-mooney | which is just install form source | |
| 13:07:30 | sean-k-mooney | and it only affects jobs | |
| 13:07:35 | sean-k-mooney | that are using unreleased oslo | |
| 13:07:42 | sean-k-mooney | which is a very small minority | |
| 13:07:48 | sean-k-mooney | so we are likey ok if they cant | |
| 13:08:27 | sean-k-mooney | we could also jsut swap to debian :) | |
| 13:11:33 | gibi | bauzas: blogpost with more data https://gibizer.github.io/posts/Eventlet-Removal-Scheduler-First-Run/ | |
| 13:13:07 | bauzas | ta a lot | |
| 13:23:03 | opendevreview | LiuWenYu proposed openstack/nova master: Implements: Fixed the issue where the code logic would throw an error when reading a variable after NUMA node 0’s CPUs were fully utilized. Related-Bug: #2109675 https://review.opendev.org/c/openstack/nova/+/948540 | |
| 13:24:48 | zigo | sean-k-mooney: By all means, please get jobs running on Debian 13 and Py 3.13 ! :) | |
| 13:25:59 | sean-k-mooney | zigo: we moved nova-hybrid-plug to debian 12 to test with spice | |
| 13:26:29 | sean-k-mooney | zigo: so we will proably move that to debain 13 once thats avaiable provided spice is not compiled out | |
| 13:33:56 | frickler | zigo: fyi jamespage is no longer with canonical | |
| 13:34:14 | zigo | Oh, I didn't know. | |
| 13:34:44 | frickler | that's why I mention it ;) happened pretty recently | |
| 13:35:45 | zigo | Who's in charge of OpenStack stuff now at Canonical then? | |
| 13:36:35 | frickler | I'm not sure, I think all the contacts I knew earlier are now gone | |
| 13:40:13 | zigo | Only me stays ... forever ! :) | |
| 14:58:55 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 14:58:55 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 16:20:15 | sean-k-mooney | bauzas: so you can just install mdevctl on ubuntu | |
| 16:20:25 | sean-k-mooney | its not there by default but it should be packaged | |
| 16:20:35 | bauzas | melwitt: fun stuff, ubuntu noble doesn't have mdevctl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_92e/openstack/92e4f13448c94210af9f36f145e4a58c/controller/logs/screen-n-cpu.txt | |
| 16:20:38 | sean-k-mooney | https://manpages.ubuntu.com/manpages/focal/man8/mdevctl.8.html | |
| 16:20:52 | sean-k-mooney | bauzas: it does its just not instlaled by defualt | |
| 16:20:56 | bauzas | sean-k-mooney: yeah, we need to install it in nova-next | |
| 16:21:13 | sean-k-mooney | you could add it to bindep in the test profile | |
| 16:21:17 | bauzas | but honestly, I hate that silent exception | |
| 16:21:20 | sean-k-mooney | to keep it optional | |
| 16:21:54 | sean-k-mooney | bauzas: well the better fix would be to fall back to the old way of not using libvirt persitnace if mdevctl is not there | |
| 16:22:05 | bauzas | sean-k-mooney: I honestly don't know how to do it, examples would be welcomed | |
| 16:22:28 | sean-k-mooney | sure just add the package to https://github.com/openstack/nova/blob/master/bindep.txt | |
| 16:22:28 | bauzas | sean-k-mooney: yup, we should failback on a libvirt exception | |
| 16:22:48 | bauzas | I know what bindep is, I don't know how to make it optional | |
| 16:22:56 | sean-k-mooney | [platform:dpkg test] | |
| 16:23:04 | sean-k-mooney | install on debing but only for the test profile | |
| 16:23:11 | sean-k-mooney | we use test in devstack | |
| 16:23:21 | bauzas | k | |
| 16:23:30 | sean-k-mooney | we could jsut list it as required | |
| 16:23:46 | sean-k-mooney | i.e. without test | |
| 16:23:58 | sean-k-mooney | if we want to always use the new persitance api | |
| 16:24:04 | bauzas | that's what we do | |
| 16:24:21 | bauzas | anyway, I'll quickly add it to bindep before I go off | |
| 16:26:03 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 16:26:04 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 17:03:29 | haleyb | zigo: regarding Canonical contact, i would start with Billy (wolsen) who isn't usually on irc, but you could try me as a proxy, i meet with them every Monday | |
| 17:38:18 | EugenMayer440180 | after my instance stopped restart properly, i restarted the cluster. Now i see that the docker container nova-compute did not start (restarted the controller/computes entirely), with the message '3 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules.' | |
| 17:41:20 | EugenMayer440180 | it seems like the controller shows all the VM 'active' and running, but they are not. Seems like the state cannot be updated | |
| 17:49:35 | EugenMayer440180 | the actual issue seems to be 'My compute node 9946da6f-9a3d-45c7-a9e3-e7e8f99811fc has hypervisor_hostname compute1.cluster.kontextwork.net but virt driver reports it should be compute1. Possible rename detected, refusing to start!' | |
| 17:54:22 | EugenMayer440180 | we recently had issues with 2024.2 with hostname != hostname -f and thus we changed that the computes nor longer have a FQDN like 'compute1.cluster.kontextwork.net' but rather compute1 | |
| 17:55:01 | EugenMayer440180 | this seems to be an issue now. Is there any way to re-register the computes with the controller using the new hostname without loosing any data? | |
| 18:13:32 | dansmith | EugenMayer440180: you cannot change the hostname of the computes, ever, for any reason | |
| 18:24:09 | EugenMayer440180 | well i just did. | |
| 18:29:34 | EugenMayer440180 | changed it in the database, restarted the computes and everything is up and running again | |
| 18:30:30 | sean-k-mooney | EugenMayer440180: the problem is the hostname is also in teh cinder and neutron dbs in several tables | |
| 18:31:01 | EugenMayer440180 | not using cinder, so this one i dodged. Neutron and the network agents already think about compute1 | |
| 18:31:02 | sean-k-mooney | you would have to consitently change it in 10-20 tables | |
| 18:31:37 | EugenMayer440180 | at least when quering the network agents and so forth, they all use the hostname already (or lets say the hostname which is the new FQDN) | |
| 18:31:43 | sean-k-mooney | changing the hypervior_hostname will break all of the placement allcoations too | |
| 18:32:56 | sean-k-mooney | neutron port bindign if they are out of sync can be fixed by detachign the ports and reattachign them or live migration/shleve/resize | |
| 18:33:37 | sean-k-mooney | EugenMayer440180: the thing you hit preventign the agent form starting was likely https://specs.openstack.org/openstack/nova-specs/specs/2023.1/implemented/stable-compute-uuid.html | |
| 18:34:02 | sean-k-mooney | after years fo saying renaming compute nodes is not supproted we finally started enforcing it | |