| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-30 | |||
| 18:34:31 | sean-k-mooney | that not where they are stored | |
| 18:34:43 | sean-k-mooney | placement has its own db | |
| 18:35:46 | sean-k-mooney | EugenMayer440180: has the value of hypervior_hostname changed in the db? if so that is the name used for the compute_node resouce provider in placement | |
| 18:37:14 | EugenMayer440180 | found them there, as you expected. Will try to fix them there too. Thanks for the hint | |
| 18:41:07 | EugenMayer440180 | sean-k-mooney i understand that i tapped into the internals. But there was no other choice for me here and i will clean up the mess that this created one by one until we migrate away i assume. | |
| 18:43:33 | EugenMayer440180 | The reason i changed the FQDN was the migration to 2024.2 where nova stopped scheduling instances since in my case, the compute nodes always had the 'hostname' (compute1) while the hypervisors had the FQDN (hostname -f) ... it's nothing i have chosen and worked since not sure, 5 years - but then stopped here. Then i had to change the FQDN work | |
| 18:43:33 | EugenMayer440180 | again, so i could e.g. restore instances from glance again. What i did not notice then is that this then broke the hypervisors / nova_libvirt (which happened now) | |
| 18:44:48 | EugenMayer440180 | i was told then 'openstack does not like when 'hostname != hostname -f' and this can lead to issues ... which it did with 2024. Kind of suprising that something of this size cannot handle what is a usual setup for me since decades. | |
| 18:45:10 | sean-k-mooney | we have not chnage how the logic in nova works | |
| 18:45:21 | sean-k-mooney | you do not have to have host and hypervior_hostname match | |
| 18:45:30 | sean-k-mooney | they can be differnt they jsut can change over time | |
| 18:45:39 | sean-k-mooney | hypervior hostname comes form libvirt | |
| 18:45:49 | sean-k-mooney | you can see what it returns with virish hostname | |
| 18:46:00 | EugenMayer440180 | well not sure who broke it, might have been neutron, since it was all about the chassis name | |
| 18:46:05 | sean-k-mooney | the host value comes form the config and defautl to socket.gethostname)( | |
| 18:46:09 | EugenMayer440180 | 'due to no OVN chassis for host: compute3' | |
| 18:46:50 | sean-k-mooney | neutron have ways to cofnigure that via data in the ovs db in the case of ovn or config options for ml2/ovs | |
| 18:47:06 | EugenMayer440180 | with 2024.2 the chassis name (which was compute1 and so forth in my case) was no longer how nova expected (since what nova expected was compute1.cluster...) - this was the change that broke it | |
| 18:48:02 | sean-k-mooney | the value we pass for nova to neutron in the port biding is the valuce form conf.host | |
| 18:48:18 | sean-k-mooney | which default to socket.gethostname() | |
| 18:48:34 | sean-k-mooney | so that implies either your dhcp server started handign out domians | |
| 18:48:43 | EugenMayer440180 | well then it either changed with neitron or kolla does no longer properly configure those - cannot say. | |
| 18:48:47 | sean-k-mooney | you /etc/host change teh fqdn to be the canonical hostname | |
| 18:48:54 | sean-k-mooney | or /etc/hostname now has the fqdn | |
| 18:48:56 | EugenMayer440180 | My DNS was always resolving the FQDN properly, yes | |
| 18:49:37 | EugenMayer440180 | yes hosts where properly setup, as usual, and also the hostname. I mean, it's the 101 and it was not changed, and after it broke, i tripple checked it. | |
| 18:50:16 | sean-k-mooney | ya so i know that the ovs-ctl script that is used to start ovs can also have a hostnaem parmater | |
| 18:50:39 | sean-k-mooney | i would assume it has to be either a change in that, in ovs or in how ovn is configured by kolla | |
| 18:50:56 | sean-k-mooney | EugenMayer440180: did you report an issue to the kolla team? | |
| 18:51:00 | EugenMayer440180 | yes, kolla might be a source for that trouble too. | |
| 18:51:15 | sean-k-mooney | im wondering if it could be a regresion on there end and if other will be impacted | |
| 18:51:58 | EugenMayer440180 | Yes / no. It basically ended up to be noones 'topic'. Neutron was insisting on 'that is how chasis are', nova was 'well it must much, this is how nova always looked for the chases' and kolla was 'well we did not change anything, do not look here' | |
| 18:55:35 | EugenMayer440180 | If i may ask, a different topic. The nova cli is deprecated as i have seen, but the openstack cli has no replacement for 'backup' - does this mean that it will be dropped in the (when?) near future? | |
| 18:55:56 | EugenMayer440180 | is there an official EOL for the nova cli? | |
| 18:57:03 | sean-k-mooney | yes and no. its been feature frozen for about 2 years | |
| 18:57:15 | sean-k-mooney | the python bindign are also frozen | |
| 18:57:35 | sean-k-mooney | we plan to remove it once heat/watcher/horizon stop usign it | |
| 18:58:30 | sean-k-mooney | EugenMayer440180: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-backup.html | |
| 18:58:31 | EugenMayer440180 | any time horizon here? years, months? 2025.x or 2026? | |
| 18:59:05 | sean-k-mooney | my hope would be in the next 12-24 months but eventlet removal is a higher priority | |
| 18:59:56 | EugenMayer440180 | i tried to onboard the above, but it always failed with 'More than one Image exists with the name' | |
| 19:00:02 | sean-k-mooney | EugenMayer440180: osc has backup and we reached feature partiy with nova client a few years ago. teh only comamnd we did nto port are the host evacuate ones | |
| 19:00:13 | EugenMayer440180 | last time i tried was Nov 2023 though | |
| 19:02:12 | EugenMayer440180 | just tried again and: | |
| 19:02:13 | EugenMayer440180 | openstack --insecure server backup create --os-compute-api-version 2.64 --name "backup-vpn" --type daily --rotate 5 2ba904fb-47e3-4ee8-92b4-66e41bcf2a96 | |
| 19:02:13 | EugenMayer440180 | More than one Image exists with the name 'backup-vpn'. | |
| 19:02:39 | sean-k-mooney | well you ment to provide unique names for each backup i belive | |
| 19:02:44 | sean-k-mooney | when passing --name | |
| 19:03:10 | EugenMayer440180 | well it creates a backup, it just fails | |
| 19:03:32 | EugenMayer440180 | (and nova backup actually needs that name to rotate .. so i thing osc will need that too;v | |
| 19:04:01 | EugenMayer440180 | i would love to use a name including a date, but this does not work for rotate - the name has to match the older backups. | |
| 19:04:06 | sean-k-mooney | well osc is using the sdk | |
| 19:04:15 | sean-k-mooney | but lo9oking at the api | |
| 19:04:21 | sean-k-mooney | https://docs.openstack.org/api-ref/compute/#id57 | |
| 19:04:27 | sean-k-mooney | the name is not optional | |
| 19:05:07 | sean-k-mooney | EugenMayer440180: its not a bfv instance is it | |
| 19:05:26 | EugenMayer440180 | my point is, nova cli forced the name to be 'always the same' to be able to group the rotation - osc dropts that issue but ends up beeing in need for that too anyway. so with osc you are basically in a no-go spot | |
| 19:05:35 | EugenMayer440180 | bfv? | |
| 19:05:45 | sean-k-mooney | boot form volume | |
| 19:06:29 | sean-k-mooney | osc will default to using the server display name if you dont pass a name for the backup | |
| 19:06:49 | sean-k-mooney | wich is reasonble but do you know if the error is coming form nova or form the client | |
| 19:06:57 | EugenMayer440180 | if osc defaults to that, it will drop an error, since the server-name should be the same every time | |
| 19:07:22 | EugenMayer440180 | oh you mean i should try that once | |
| 19:07:59 | sean-k-mooney | the rotaion i belive is ment ot be server side | |
| 19:08:40 | sean-k-mooney | but im wonderign if your geting a error form nova or if its coming form the client or sdk | |
| 19:08:55 | EugenMayer440180 | the rotation is based on the name - that is no assumption. Tried and tested that, documented that | |
| 19:09:03 | sean-k-mooney | it might be form here https://github.com/openstack/python-openstackclient/blob/master/openstackclient/compute/v2/server_backup.py#L102 | |
| 19:09:27 | sean-k-mooney | so they should be saving the uuid that is returned form the backup call and then using that not the name to look upt the image | |
| 19:09:49 | EugenMayer440180 | well is that new on osc? because nova does not do that, it used the name | |
| 19:10:08 | sean-k-mooney | this is an osc bug | |
| 19:10:17 | EugenMayer440180 | or, nova changed that recently, i'am not reading the commits | |
| 19:10:25 | sean-k-mooney | its a regression form when they swaped form the novaclint to the sdk | |
| 19:10:39 | sean-k-mooney | EugenMayer440180: this has not change in nova in many many years | |
| 19:10:57 | EugenMayer440180 | well then i can assure you, the 'uuid lookup' is broken too. | |
| 19:11:05 | sean-k-mooney | but the openstackclinet moved all the nova command to the openstacksdk | |
| 19:11:16 | sean-k-mooney | EugenMayer440180: its not using uuid lookup at all | |
| 19:11:22 | sean-k-mooney | its finding the backup image by name | |
| 19:11:49 | EugenMayer440180 | ok you confused me / i got confused. If the name is required (you mean the server name or the backup name?) | |
| 19:12:11 | sean-k-mooney | in novas api the name of the backup image is required | |
| 19:12:41 | sean-k-mooney | osc is passing that to the backup_server functon | |
| 19:12:51 | sean-k-mooney | but its alos doing image_client.find_image(backup_name, ignore_missing=False) | |
| 19:12:52 | EugenMayer440180 | ok then we are actually on the same spot. The name is required and is used to find the older backups, the lookup group for the rotate, the PK or whatever | |
| 19:13:19 | sean-k-mooney | i htink your error is happening here https://github.com/openstack/python-openstackclient/blob/master/openstackclient/compute/v2/server_backup.py#L102 | |
| 19:13:32 | sean-k-mooney | i have an devstafck actully let me check | |
| 19:13:47 | EugenMayer440180 | so changing the name for every backup would be wrong. Thus wht osc requires of me (using a unique name) is wrong (and this is what you pointed out to be a bug) | |
| 19:15:22 | sean-k-mooney | ya it would | |
| 19:15:27 | EugenMayer440180 | not sure how to take it, that the centralized, only way to backup an instance (openstack server backup) is broken for over 1.5 years and nobody has noticed that | |
| 19:15:29 | sean-k-mooney | it will need to be sable | |
| 19:15:49 | sean-k-mooney | EugenMayer440180: you realsise we condiered removing the backup api entirely | |
| 19:15:54 | sean-k-mooney | instend of porting it | |
| 19:16:09 | sean-k-mooney | it is not really used much | |
| 19:16:30 | sean-k-mooney | most people just use snaphost directly instead of backup | |
| 19:16:40 | sean-k-mooney | because backup has never workd form server booted form cinder volume | |
| 19:16:50 | sean-k-mooney | where as the snapshot command do | |
| 19:17:49 | sean-k-mooney | im not saying this should not be fix im saying its a rearly used api/command | |
| 19:18:13 | EugenMayer440180 | yeah, this would explain that. Can you safely say that both 2025.x releases will not remove nova backup? or is this nothing that can be said | |
| 19:19:30 | sean-k-mooney | more or less yes. i expect it to still be a thing in 2026.1 | |
| 19:20:01 | sean-k-mooney | it might not exist in 2027.1 or 2027.2 if we mange to get everyone over to the sdk | |
| 19:21:04 | EugenMayer440180 | well i will not get to 2026 anyway, so this is not what i would worry then. I plan to migrate away from openstack in Dec 2025 and want to be sure the backup strategy works as long as that. | |
| 19:21:28 | sean-k-mooney | EugenMayer440180: https://paste.opendev.org/show/bZLdha8p7l5juf4jF8Dw/ | |