| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-30 | |||
| 17:41:20 | EugenMayer440180 | it seems like the controller shows all the VM 'active' and running, but they are not. Seems like the state cannot be updated | |
| 17:49:35 | EugenMayer440180 | the actual issue seems to be 'My compute node 9946da6f-9a3d-45c7-a9e3-e7e8f99811fc has hypervisor_hostname compute1.cluster.kontextwork.net but virt driver reports it should be compute1. Possible rename detected, refusing to start!' | |
| 17:54:22 | EugenMayer440180 | we recently had issues with 2024.2 with hostname != hostname -f and thus we changed that the computes nor longer have a FQDN like 'compute1.cluster.kontextwork.net' but rather compute1 | |
| 17:55:01 | EugenMayer440180 | this seems to be an issue now. Is there any way to re-register the computes with the controller using the new hostname without loosing any data? | |
| 18:13:32 | dansmith | EugenMayer440180: you cannot change the hostname of the computes, ever, for any reason | |
| 18:24:09 | EugenMayer440180 | well i just did. | |
| 18:29:34 | EugenMayer440180 | changed it in the database, restarted the computes and everything is up and running again | |
| 18:30:30 | sean-k-mooney | EugenMayer440180: the problem is the hostname is also in teh cinder and neutron dbs in several tables | |
| 18:31:01 | EugenMayer440180 | not using cinder, so this one i dodged. Neutron and the network agents already think about compute1 | |
| 18:31:02 | sean-k-mooney | you would have to consitently change it in 10-20 tables | |
| 18:31:37 | EugenMayer440180 | at least when quering the network agents and so forth, they all use the hostname already (or lets say the hostname which is the new FQDN) | |
| 18:31:43 | sean-k-mooney | changing the hypervior_hostname will break all of the placement allcoations too | |
| 18:32:56 | sean-k-mooney | neutron port bindign if they are out of sync can be fixed by detachign the ports and reattachign them or live migration/shleve/resize | |
| 18:33:37 | sean-k-mooney | EugenMayer440180: the thing you hit preventign the agent form starting was likely https://specs.openstack.org/openstack/nova-specs/specs/2023.1/implemented/stable-compute-uuid.html | |
| 18:34:02 | sean-k-mooney | after years fo saying renaming compute nodes is not supproted we finally started enforcing it | |
| 18:34:18 | EugenMayer440180 | checked the allocations tables in nova/nova_api/nova_cell0, they are all empty | |
| 18:34:30 | EugenMayer440180 | sean-k-mooney i did not choose to rename those | |
| 18:34:31 | sean-k-mooney | that not where they are stored | |
| 18:34:43 | sean-k-mooney | placement has its own db | |
| 18:35:46 | sean-k-mooney | EugenMayer440180: has the value of hypervior_hostname changed in the db? if so that is the name used for the compute_node resouce provider in placement | |
| 18:37:14 | EugenMayer440180 | found them there, as you expected. Will try to fix them there too. Thanks for the hint | |
| 18:41:07 | EugenMayer440180 | sean-k-mooney i understand that i tapped into the internals. But there was no other choice for me here and i will clean up the mess that this created one by one until we migrate away i assume. | |
| 18:43:33 | EugenMayer440180 | again, so i could e.g. restore instances from glance again. What i did not notice then is that this then broke the hypervisors / nova_libvirt (which happened now) | |
| 18:43:33 | EugenMayer440180 | The reason i changed the FQDN was the migration to 2024.2 where nova stopped scheduling instances since in my case, the compute nodes always had the 'hostname' (compute1) while the hypervisors had the FQDN (hostname -f) ... it's nothing i have chosen and worked since not sure, 5 years - but then stopped here. Then i had to change the FQDN work | |
| 18:44:48 | EugenMayer440180 | i was told then 'openstack does not like when 'hostname != hostname -f' and this can lead to issues ... which it did with 2024. Kind of suprising that something of this size cannot handle what is a usual setup for me since decades. | |
| 18:45:10 | sean-k-mooney | we have not chnage how the logic in nova works | |
| 18:45:21 | sean-k-mooney | you do not have to have host and hypervior_hostname match | |
| 18:45:30 | sean-k-mooney | they can be differnt they jsut can change over time | |
| 18:45:39 | sean-k-mooney | hypervior hostname comes form libvirt | |
| 18:45:49 | sean-k-mooney | you can see what it returns with virish hostname | |
| 18:46:00 | EugenMayer440180 | well not sure who broke it, might have been neutron, since it was all about the chassis name | |
| 18:46:05 | sean-k-mooney | the host value comes form the config and defautl to socket.gethostname)( | |
| 18:46:09 | EugenMayer440180 | 'due to no OVN chassis for host: compute3' | |
| 18:46:50 | sean-k-mooney | neutron have ways to cofnigure that via data in the ovs db in the case of ovn or config options for ml2/ovs | |
| 18:47:06 | EugenMayer440180 | with 2024.2 the chassis name (which was compute1 and so forth in my case) was no longer how nova expected (since what nova expected was compute1.cluster...) - this was the change that broke it | |
| 18:48:02 | sean-k-mooney | the value we pass for nova to neutron in the port biding is the valuce form conf.host | |
| 18:48:18 | sean-k-mooney | which default to socket.gethostname() | |
| 18:48:34 | sean-k-mooney | so that implies either your dhcp server started handign out domians | |
| 18:48:43 | EugenMayer440180 | well then it either changed with neitron or kolla does no longer properly configure those - cannot say. | |
| 18:48:47 | sean-k-mooney | you /etc/host change teh fqdn to be the canonical hostname | |
| 18:48:54 | sean-k-mooney | or /etc/hostname now has the fqdn | |
| 18:48:56 | EugenMayer440180 | My DNS was always resolving the FQDN properly, yes | |
| 18:49:37 | EugenMayer440180 | yes hosts where properly setup, as usual, and also the hostname. I mean, it's the 101 and it was not changed, and after it broke, i tripple checked it. | |
| 18:50:16 | sean-k-mooney | ya so i know that the ovs-ctl script that is used to start ovs can also have a hostnaem parmater | |
| 18:50:39 | sean-k-mooney | i would assume it has to be either a change in that, in ovs or in how ovn is configured by kolla | |
| 18:50:56 | sean-k-mooney | EugenMayer440180: did you report an issue to the kolla team? | |
| 18:51:00 | EugenMayer440180 | yes, kolla might be a source for that trouble too. | |
| 18:51:15 | sean-k-mooney | im wondering if it could be a regresion on there end and if other will be impacted | |
| 18:51:58 | EugenMayer440180 | Yes / no. It basically ended up to be noones 'topic'. Neutron was insisting on 'that is how chasis are', nova was 'well it must much, this is how nova always looked for the chases' and kolla was 'well we did not change anything, do not look here' | |
| 18:55:35 | EugenMayer440180 | If i may ask, a different topic. The nova cli is deprecated as i have seen, but the openstack cli has no replacement for 'backup' - does this mean that it will be dropped in the (when?) near future? | |
| 18:55:56 | EugenMayer440180 | is there an official EOL for the nova cli? | |
| 18:57:03 | sean-k-mooney | yes and no. its been feature frozen for about 2 years | |
| 18:57:15 | sean-k-mooney | the python bindign are also frozen | |
| 18:57:35 | sean-k-mooney | we plan to remove it once heat/watcher/horizon stop usign it | |
| 18:58:30 | sean-k-mooney | EugenMayer440180: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-backup.html | |
| 18:58:31 | EugenMayer440180 | any time horizon here? years, months? 2025.x or 2026? | |
| 18:59:05 | sean-k-mooney | my hope would be in the next 12-24 months but eventlet removal is a higher priority | |
| 18:59:56 | EugenMayer440180 | i tried to onboard the above, but it always failed with 'More than one Image exists with the name' | |
| 19:00:02 | sean-k-mooney | EugenMayer440180: osc has backup and we reached feature partiy with nova client a few years ago. teh only comamnd we did nto port are the host evacuate ones | |
| 19:00:13 | EugenMayer440180 | last time i tried was Nov 2023 though | |
| 19:02:12 | EugenMayer440180 | just tried again and: | |
| 19:02:13 | EugenMayer440180 | More than one Image exists with the name 'backup-vpn'. | |
| 19:02:13 | EugenMayer440180 | openstack --insecure server backup create --os-compute-api-version 2.64 --name "backup-vpn" --type daily --rotate 5 2ba904fb-47e3-4ee8-92b4-66e41bcf2a96 | |
| 19:02:39 | sean-k-mooney | well you ment to provide unique names for each backup i belive | |
| 19:02:44 | sean-k-mooney | when passing --name | |
| 19:03:10 | EugenMayer440180 | well it creates a backup, it just fails | |
| 19:03:32 | EugenMayer440180 | (and nova backup actually needs that name to rotate .. so i thing osc will need that too;v | |
| 19:04:01 | EugenMayer440180 | i would love to use a name including a date, but this does not work for rotate - the name has to match the older backups. | |
| 19:04:06 | sean-k-mooney | well osc is using the sdk | |
| 19:04:15 | sean-k-mooney | but lo9oking at the api | |
| 19:04:21 | sean-k-mooney | https://docs.openstack.org/api-ref/compute/#id57 | |
| 19:04:27 | sean-k-mooney | the name is not optional | |
| 19:05:07 | sean-k-mooney | EugenMayer440180: its not a bfv instance is it | |
| 19:05:26 | EugenMayer440180 | my point is, nova cli forced the name to be 'always the same' to be able to group the rotation - osc dropts that issue but ends up beeing in need for that too anyway. so with osc you are basically in a no-go spot | |
| 19:05:35 | EugenMayer440180 | bfv? | |
| 19:05:45 | sean-k-mooney | boot form volume | |
| 19:06:29 | sean-k-mooney | osc will default to using the server display name if you dont pass a name for the backup | |
| 19:06:49 | sean-k-mooney | wich is reasonble but do you know if the error is coming form nova or form the client | |
| 19:06:57 | EugenMayer440180 | if osc defaults to that, it will drop an error, since the server-name should be the same every time | |
| 19:07:22 | EugenMayer440180 | oh you mean i should try that once | |
| 19:07:59 | sean-k-mooney | the rotaion i belive is ment ot be server side | |
| 19:08:40 | sean-k-mooney | but im wonderign if your geting a error form nova or if its coming form the client or sdk | |
| 19:08:55 | EugenMayer440180 | the rotation is based on the name - that is no assumption. Tried and tested that, documented that | |
| 19:09:03 | sean-k-mooney | it might be form here https://github.com/openstack/python-openstackclient/blob/master/openstackclient/compute/v2/server_backup.py#L102 | |
| 19:09:27 | sean-k-mooney | so they should be saving the uuid that is returned form the backup call and then using that not the name to look upt the image | |
| 19:09:49 | EugenMayer440180 | well is that new on osc? because nova does not do that, it used the name | |
| 19:10:08 | sean-k-mooney | this is an osc bug | |
| 19:10:17 | EugenMayer440180 | or, nova changed that recently, i'am not reading the commits | |
| 19:10:25 | sean-k-mooney | its a regression form when they swaped form the novaclint to the sdk | |
| 19:10:39 | sean-k-mooney | EugenMayer440180: this has not change in nova in many many years | |
| 19:10:57 | EugenMayer440180 | well then i can assure you, the 'uuid lookup' is broken too. | |
| 19:11:05 | sean-k-mooney | but the openstackclinet moved all the nova command to the openstacksdk | |
| 19:11:16 | sean-k-mooney | EugenMayer440180: its not using uuid lookup at all | |
| 19:11:22 | sean-k-mooney | its finding the backup image by name | |
| 19:11:49 | EugenMayer440180 | ok you confused me / i got confused. If the name is required (you mean the server name or the backup name?) | |
| 19:12:11 | sean-k-mooney | in novas api the name of the backup image is required | |
| 19:12:41 | sean-k-mooney | osc is passing that to the backup_server functon | |
| 19:12:51 | sean-k-mooney | but its alos doing image_client.find_image(backup_name, ignore_missing=False) | |
| 19:12:52 | EugenMayer440180 | ok then we are actually on the same spot. The name is required and is used to find the older backups, the lookup group for the rotate, the PK or whatever | |
| 19:13:19 | sean-k-mooney | i htink your error is happening here https://github.com/openstack/python-openstackclient/blob/master/openstackclient/compute/v2/server_backup.py#L102 | |