| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-30 | |||
| 13:07:35 | sean-k-mooney | that are using unreleased oslo | |
| 13:07:42 | sean-k-mooney | which is a very small minority | |
| 13:07:48 | sean-k-mooney | so we are likey ok if they cant | |
| 13:08:27 | sean-k-mooney | we could also jsut swap to debian :) | |
| 13:11:33 | gibi | bauzas: blogpost with more data https://gibizer.github.io/posts/Eventlet-Removal-Scheduler-First-Run/ | |
| 13:13:07 | bauzas | ta a lot | |
| 13:23:03 | opendevreview | LiuWenYu proposed openstack/nova master: Implements: Fixed the issue where the code logic would throw an error when reading a variable after NUMA node 0’s CPUs were fully utilized. Related-Bug: #2109675 https://review.opendev.org/c/openstack/nova/+/948540 | |
| 13:24:48 | zigo | sean-k-mooney: By all means, please get jobs running on Debian 13 and Py 3.13 ! :) | |
| 13:25:59 | sean-k-mooney | zigo: we moved nova-hybrid-plug to debian 12 to test with spice | |
| 13:26:29 | sean-k-mooney | zigo: so we will proably move that to debain 13 once thats avaiable provided spice is not compiled out | |
| 13:33:56 | frickler | zigo: fyi jamespage is no longer with canonical | |
| 13:34:14 | zigo | Oh, I didn't know. | |
| 13:34:44 | frickler | that's why I mention it ;) happened pretty recently | |
| 13:35:45 | zigo | Who's in charge of OpenStack stuff now at Canonical then? | |
| 13:36:35 | frickler | I'm not sure, I think all the contacts I knew earlier are now gone | |
| 13:40:13 | zigo | Only me stays ... forever ! :) | |
| 14:58:55 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 14:58:55 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 16:20:15 | sean-k-mooney | bauzas: so you can just install mdevctl on ubuntu | |
| 16:20:25 | sean-k-mooney | its not there by default but it should be packaged | |
| 16:20:35 | bauzas | melwitt: fun stuff, ubuntu noble doesn't have mdevctl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_92e/openstack/92e4f13448c94210af9f36f145e4a58c/controller/logs/screen-n-cpu.txt | |
| 16:20:38 | sean-k-mooney | https://manpages.ubuntu.com/manpages/focal/man8/mdevctl.8.html | |
| 16:20:52 | sean-k-mooney | bauzas: it does its just not instlaled by defualt | |
| 16:20:56 | bauzas | sean-k-mooney: yeah, we need to install it in nova-next | |
| 16:21:13 | sean-k-mooney | you could add it to bindep in the test profile | |
| 16:21:17 | bauzas | but honestly, I hate that silent exception | |
| 16:21:20 | sean-k-mooney | to keep it optional | |
| 16:21:54 | sean-k-mooney | bauzas: well the better fix would be to fall back to the old way of not using libvirt persitnace if mdevctl is not there | |
| 16:22:05 | bauzas | sean-k-mooney: I honestly don't know how to do it, examples would be welcomed | |
| 16:22:28 | bauzas | sean-k-mooney: yup, we should failback on a libvirt exception | |
| 16:22:28 | sean-k-mooney | sure just add the package to https://github.com/openstack/nova/blob/master/bindep.txt | |
| 16:22:48 | bauzas | I know what bindep is, I don't know how to make it optional | |
| 16:22:56 | sean-k-mooney | [platform:dpkg test] | |
| 16:23:04 | sean-k-mooney | install on debing but only for the test profile | |
| 16:23:11 | sean-k-mooney | we use test in devstack | |
| 16:23:21 | bauzas | k | |
| 16:23:30 | sean-k-mooney | we could jsut list it as required | |
| 16:23:46 | sean-k-mooney | i.e. without test | |
| 16:23:58 | sean-k-mooney | if we want to always use the new persitance api | |
| 16:24:04 | bauzas | that's what we do | |
| 16:24:21 | bauzas | anyway, I'll quickly add it to bindep before I go off | |
| 16:26:03 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 16:26:04 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 17:03:29 | haleyb | zigo: regarding Canonical contact, i would start with Billy (wolsen) who isn't usually on irc, but you could try me as a proxy, i meet with them every Monday | |
| 17:38:18 | EugenMayer440180 | after my instance stopped restart properly, i restarted the cluster. Now i see that the docker container nova-compute did not start (restarted the controller/computes entirely), with the message '3 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules.' | |
| 17:41:20 | EugenMayer440180 | it seems like the controller shows all the VM 'active' and running, but they are not. Seems like the state cannot be updated | |
| 17:49:35 | EugenMayer440180 | the actual issue seems to be 'My compute node 9946da6f-9a3d-45c7-a9e3-e7e8f99811fc has hypervisor_hostname compute1.cluster.kontextwork.net but virt driver reports it should be compute1. Possible rename detected, refusing to start!' | |
| 17:54:22 | EugenMayer440180 | we recently had issues with 2024.2 with hostname != hostname -f and thus we changed that the computes nor longer have a FQDN like 'compute1.cluster.kontextwork.net' but rather compute1 | |
| 17:55:01 | EugenMayer440180 | this seems to be an issue now. Is there any way to re-register the computes with the controller using the new hostname without loosing any data? | |
| 18:13:32 | dansmith | EugenMayer440180: you cannot change the hostname of the computes, ever, for any reason | |
| 18:24:09 | EugenMayer440180 | well i just did. | |
| 18:29:34 | EugenMayer440180 | changed it in the database, restarted the computes and everything is up and running again | |
| 18:30:30 | sean-k-mooney | EugenMayer440180: the problem is the hostname is also in teh cinder and neutron dbs in several tables | |
| 18:31:01 | EugenMayer440180 | not using cinder, so this one i dodged. Neutron and the network agents already think about compute1 | |
| 18:31:02 | sean-k-mooney | you would have to consitently change it in 10-20 tables | |
| 18:31:37 | EugenMayer440180 | at least when quering the network agents and so forth, they all use the hostname already (or lets say the hostname which is the new FQDN) | |
| 18:31:43 | sean-k-mooney | changing the hypervior_hostname will break all of the placement allcoations too | |
| 18:32:56 | sean-k-mooney | neutron port bindign if they are out of sync can be fixed by detachign the ports and reattachign them or live migration/shleve/resize | |
| 18:33:37 | sean-k-mooney | EugenMayer440180: the thing you hit preventign the agent form starting was likely https://specs.openstack.org/openstack/nova-specs/specs/2023.1/implemented/stable-compute-uuid.html | |
| 18:34:02 | sean-k-mooney | after years fo saying renaming compute nodes is not supproted we finally started enforcing it | |
| 18:34:18 | EugenMayer440180 | checked the allocations tables in nova/nova_api/nova_cell0, they are all empty | |
| 18:34:30 | EugenMayer440180 | sean-k-mooney i did not choose to rename those | |
| 18:34:31 | sean-k-mooney | that not where they are stored | |
| 18:34:43 | sean-k-mooney | placement has its own db | |
| 18:35:46 | sean-k-mooney | EugenMayer440180: has the value of hypervior_hostname changed in the db? if so that is the name used for the compute_node resouce provider in placement | |
| 18:37:14 | EugenMayer440180 | found them there, as you expected. Will try to fix them there too. Thanks for the hint | |
| 18:41:07 | EugenMayer440180 | sean-k-mooney i understand that i tapped into the internals. But there was no other choice for me here and i will clean up the mess that this created one by one until we migrate away i assume. | |
| 18:43:33 | EugenMayer440180 | The reason i changed the FQDN was the migration to 2024.2 where nova stopped scheduling instances since in my case, the compute nodes always had the 'hostname' (compute1) while the hypervisors had the FQDN (hostname -f) ... it's nothing i have chosen and worked since not sure, 5 years - but then stopped here. Then i had to change the FQDN work | |
| 18:43:33 | EugenMayer440180 | again, so i could e.g. restore instances from glance again. What i did not notice then is that this then broke the hypervisors / nova_libvirt (which happened now) | |
| 18:44:48 | EugenMayer440180 | i was told then 'openstack does not like when 'hostname != hostname -f' and this can lead to issues ... which it did with 2024. Kind of suprising that something of this size cannot handle what is a usual setup for me since decades. | |
| 18:45:10 | sean-k-mooney | we have not chnage how the logic in nova works | |
| 18:45:21 | sean-k-mooney | you do not have to have host and hypervior_hostname match | |
| 18:45:30 | sean-k-mooney | they can be differnt they jsut can change over time | |
| 18:45:39 | sean-k-mooney | hypervior hostname comes form libvirt | |
| 18:45:49 | sean-k-mooney | you can see what it returns with virish hostname | |
| 18:46:00 | EugenMayer440180 | well not sure who broke it, might have been neutron, since it was all about the chassis name | |
| 18:46:05 | sean-k-mooney | the host value comes form the config and defautl to socket.gethostname)( | |
| 18:46:09 | EugenMayer440180 | 'due to no OVN chassis for host: compute3' | |
| 18:46:50 | sean-k-mooney | neutron have ways to cofnigure that via data in the ovs db in the case of ovn or config options for ml2/ovs | |
| 18:47:06 | EugenMayer440180 | with 2024.2 the chassis name (which was compute1 and so forth in my case) was no longer how nova expected (since what nova expected was compute1.cluster...) - this was the change that broke it | |
| 18:48:02 | sean-k-mooney | the value we pass for nova to neutron in the port biding is the valuce form conf.host | |
| 18:48:18 | sean-k-mooney | which default to socket.gethostname() | |
| 18:48:34 | sean-k-mooney | so that implies either your dhcp server started handign out domians | |
| 18:48:43 | EugenMayer440180 | well then it either changed with neitron or kolla does no longer properly configure those - cannot say. | |
| 18:48:47 | sean-k-mooney | you /etc/host change teh fqdn to be the canonical hostname | |
| 18:48:54 | sean-k-mooney | or /etc/hostname now has the fqdn | |
| 18:48:56 | EugenMayer440180 | My DNS was always resolving the FQDN properly, yes | |
| 18:49:37 | EugenMayer440180 | yes hosts where properly setup, as usual, and also the hostname. I mean, it's the 101 and it was not changed, and after it broke, i tripple checked it. | |
| 18:50:16 | sean-k-mooney | ya so i know that the ovs-ctl script that is used to start ovs can also have a hostnaem parmater | |
| 18:50:39 | sean-k-mooney | i would assume it has to be either a change in that, in ovs or in how ovn is configured by kolla | |
| 18:50:56 | sean-k-mooney | EugenMayer440180: did you report an issue to the kolla team? | |
| 18:51:00 | EugenMayer440180 | yes, kolla might be a source for that trouble too. | |
| 18:51:15 | sean-k-mooney | im wondering if it could be a regresion on there end and if other will be impacted | |
| 18:51:58 | EugenMayer440180 | Yes / no. It basically ended up to be noones 'topic'. Neutron was insisting on 'that is how chasis are', nova was 'well it must much, this is how nova always looked for the chases' and kolla was 'well we did not change anything, do not look here' | |
| 18:55:35 | EugenMayer440180 | If i may ask, a different topic. The nova cli is deprecated as i have seen, but the openstack cli has no replacement for 'backup' - does this mean that it will be dropped in the (when?) near future? | |
| 18:55:56 | EugenMayer440180 | is there an official EOL for the nova cli? | |
| 18:57:03 | sean-k-mooney | yes and no. its been feature frozen for about 2 years | |
| 18:57:15 | sean-k-mooney | the python bindign are also frozen | |
| 18:57:35 | sean-k-mooney | we plan to remove it once heat/watcher/horizon stop usign it | |
| 18:58:30 | sean-k-mooney | EugenMayer440180: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-backup.html | |