Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-11-23
11:00:13 sean-k-mooney indicate that something deleted the disk image from behind nova's back
11:00:34 sean-k-mooney soft reboot escalate to hard reboot if it fails
11:00:43 opendevreview Amit Uniyal proposed openstack/nova stable/train: func: Introduce a server_expected_state kwarg to InstanceHelperMixin._live_migrate https://review.opendev.org/c/openstack/nova/+/865382
11:00:44 opendevreview Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670
11:00:45 opendevreview Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671
11:00:46 opendevreview Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806
11:00:47 opendevreview Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055
11:01:13 sean-k-mooney gokhani: we might not be passign the auth token propelry in the context when we do that
11:01:42 sean-k-mooney gokhani: the code path that it is taking woudl repair the issue that it detected in soft reboot
11:01:48 sean-k-mooney if the image still existed in glance
11:02:18 sean-k-mooney if its deltete tehre is no way to fix the vm without copying the backing file form somewhere else in teh cloud that still has it
11:02:57 sean-k-mooney if /var/lib/nova/instances/_base/9d36e2c635ce070d95805f64f4b34655f3eae96b exist somewhere else on your cloud and you copy it to that location on this host
11:03:16 sean-k-mooney and ensure it has the corret user permissions as the other images there
11:03:26 sean-k-mooney then you might be able to recover the vm
11:03:34 sean-k-mooney with another hard/soft reboot
11:04:07 gokhani sean-k-mooney, I deleted image of this instance in glance.
11:04:21 sean-k-mooney sure that should be fine
11:04:37 sean-k-mooney but something also deleted the image backing file on the compute node
11:04:58 sean-k-mooney that is not ok and should not happen if there is an instance based on that image on the host
11:06:14 sean-k-mooney you have configured your deployment to use qcow images for the vm with a backign file
11:06:22 gokhani sean-k-mooney, what can be the reason of deleting image backing file ? Is there any config on nova?
11:06:49 sean-k-mooney this will only be deleted by nova if there are no vms on this host based on that glance image
11:07:10 sean-k-mooney you are not mounting this on a nfs share or soemthign liek that?
11:08:00 gokhani sean-k-mooney, yes I have netapp storage and Instance disks are on nfs share
11:08:13 sean-k-mooney nfsv3...?
11:09:46 gokhani nfs4
11:09:49 sean-k-mooney nfs v3 has consitey/locking issues and we stongly discurage using it. ideally you woudl use v4.2+ with nova.
11:09:52 gokhani nfsv4
11:09:53 sean-k-mooney ok
11:10:27 sean-k-mooney so the only thing that comes to mind is if you are not using a sepeate share or directory per host
11:11:02 sean-k-mooney then if the /var/lib/nova/instances/_base is shared one of the other nova compute could have deleted it if it did not detect its on a shared file system properly
11:11:30 gokhani sean-k-mooney, I am using same share for all compute nodes
11:11:56 sean-k-mooney in generall we discorgage puting the instance directory on nfs by the way.
11:12:37 sean-k-mooney you can do that and we know operators do but its not well tested and there are defintly more bugs in that config.
11:12:56 sean-k-mooney can you check if you ahve a file for me
11:13:03 sean-k-mooney one sec while i look for it
11:13:55 gokhani sean-k-mooney, I have also instance disk under /var/lib/nova/instances/xxxxxxxxx/disk
11:14:13 sean-k-mooney yes thats where they are stored by default
11:14:22 sean-k-mooney you have /var/lib/nova/instances/compute_nodes
11:14:27 sean-k-mooney *do you have
11:14:55 gokhani yes I have
11:15:08 sean-k-mooney does it have muliple entries
11:15:32 gokhani I will check it
11:15:45 sean-k-mooney mine looks like this
11:15:47 sean-k-mooney nova-compute)[nova@cloud instances]$ cat /var/lib/nova/instances/compute_nodes
11:15:49 sean-k-mooney {"cloud": 1669200576.0961165}
11:16:03 sean-k-mooney im not sure if that should have muliple entries for nfs
11:16:30 sean-k-mooney but that is created by the image cache code and i think its related ot deleteion on shared filesystmes
11:16:56 sean-k-mooney im just wondering if it exists and if it has multipel entires the content itself is not really that important
11:18:00 gokhani sean-k-mooney, https://paste.openstack.org/show/bldgeGSIq8BoLjp0u6sx/
11:18:28 sean-k-mooney ack that is what i was expecting to see
11:19:15 sean-k-mooney that generated here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/storage_users.py#L45-L72
11:19:46 sean-k-mooney so each compute on shared storage adds themselves to that list
11:21:24 sean-k-mooney https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L10901-L10916
11:21:50 sean-k-mooney we use that file to get the instnace for all host on the shared storage
11:21:59 sean-k-mooney and tehn we clean the cache based on that
11:22:14 sean-k-mooney on a normal non shared deployment like mine that only has one host
11:22:20 sean-k-mooney in your case it has many
11:23:22 sean-k-mooney i see prod-compute3 but not compute03 in that list
11:23:40 gokhani sean-k-mooney, we need to find how image backing file is deleted . I have this problem on instances whose images are deleted on glance
11:24:15 sean-k-mooney the delete proably happen because of the perodic cleanup
11:24:44 sean-k-mooney but can you verify if the compute node is compute03 or prod-compute3 in the hypervior api or compute_node db table
11:25:29 sean-k-mooney we are using conf.host to populate that file
11:25:42 sean-k-mooney im wondering if you perhaps changed that at some point?
11:25:48 sean-k-mooney or change the hostname
11:26:02 sean-k-mooney to remove the prod- prefix
11:27:20 sean-k-mooney my guess is that currently eitehr the [DEFAULT]/host value does not match the value in the compute_nodes file or its unset and your hostname nolonger matches because it was change somehow
11:28:01 sean-k-mooney that file is updated before every time we cleanup the image cache
11:28:13 gokhani sean-k-mooney, https://paste.openstack.org/show/bVzhWTyWsA3f1Umb5YGM/
11:28:20 sean-k-mooney so the compute agents think they shoudl be usingthe prod- prefix
11:28:36 sean-k-mooney ok
11:28:50 sean-k-mooney that atleast alines to whats in the file
11:28:54 sean-k-mooney althoguh
11:29:06 sean-k-mooney thats showing the hypervior hostname
11:29:14 gokhani sean-k-mooney, do ı need to also verify on db ?
11:29:39 sean-k-mooney what i actully need is the host value rather then hypervior hostname i think
11:30:26 sean-k-mooney that would be the host value in the compute service entry for example although its also in teh compute nodes table
11:30:37 sean-k-mooney im just trying to verify that they match what is in the file
11:31:12 sean-k-mooney on never mind
11:31:15 sean-k-mooney its fine
11:31:17 sean-k-mooney https://paste.openstack.org/show/b7FSkdeQOxcuWu296tfq/
11:31:33 sean-k-mooney i had scolled that over and th start of the lines were cut off
11:31:41 sean-k-mooney so its prod-compute03
11:31:48 gokhani yes
11:36:07 sean-k-mooney do you see this warning any of the logs https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/imagecache.py#L331-L334
11:41:07 gokhani sean-k-mooney, I dont see any warning like that
11:43:41 gokhani there are periodic checks like that https://paste.openstack.org/show/bGWC1FfgR5R7nXqiCOuQ/
11:48:06 sean-k-mooney ya that what we expect to see
11:48:22 sean-k-mooney when its workign correctly
11:49:48 gokhani sean-k-mooney, for rescuing these vms it seems there is only one option create glance images from /var/lib/nova/instances/xxx/disk
11:50:28 sean-k-mooney glance does not allow you to create and image with a specifc uuid
11:50:53 sean-k-mooney breifly looking at the code i done se where the bug is
11:50:59 sean-k-mooney https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/imagecache.py#L43
11:51:27 sean-k-mooney the list_running_isntnace code seams to take into account local and remote instnace
11:57:32 sean-k-mooney i have a meeting now so unfortuetly i cant continue to look at this now
11:57:52 sean-k-mooney the root cause so far is something deleted the backing file
11:58:18 sean-k-mooney since its deleted in glance and its deleted form the nfs share im not sure there is a way to correct this
11:59:45 gokhani thanks sean-k-mooney for your help :)
14:12:33 opendevreview Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432
14:14:33 opendevreview Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432

Earlier   Later