| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-11-23 | |||
| 11:03:26 | sean-k-mooney | then you might be able to recover the vm | |
| 11:03:34 | sean-k-mooney | with another hard/soft reboot | |
| 11:04:07 | gokhani | sean-k-mooney, I deleted image of this instance in glance. | |
| 11:04:21 | sean-k-mooney | sure that should be fine | |
| 11:04:37 | sean-k-mooney | but something also deleted the image backing file on the compute node | |
| 11:04:58 | sean-k-mooney | that is not ok and should not happen if there is an instance based on that image on the host | |
| 11:06:14 | sean-k-mooney | you have configured your deployment to use qcow images for the vm with a backign file | |
| 11:06:22 | gokhani | sean-k-mooney, what can be the reason of deleting image backing file ? Is there any config on nova? | |
| 11:06:49 | sean-k-mooney | this will only be deleted by nova if there are no vms on this host based on that glance image | |
| 11:07:10 | sean-k-mooney | you are not mounting this on a nfs share or soemthign liek that? | |
| 11:08:00 | gokhani | sean-k-mooney, yes I have netapp storage and Instance disks are on nfs share | |
| 11:08:13 | sean-k-mooney | nfsv3...? | |
| 11:09:46 | gokhani | nfs4 | |
| 11:09:49 | sean-k-mooney | nfs v3 has consitey/locking issues and we stongly discurage using it. ideally you woudl use v4.2+ with nova. | |
| 11:09:52 | gokhani | nfsv4 | |
| 11:09:53 | sean-k-mooney | ok | |
| 11:10:27 | sean-k-mooney | so the only thing that comes to mind is if you are not using a sepeate share or directory per host | |
| 11:11:02 | sean-k-mooney | then if the /var/lib/nova/instances/_base is shared one of the other nova compute could have deleted it if it did not detect its on a shared file system properly | |
| 11:11:30 | gokhani | sean-k-mooney, I am using same share for all compute nodes | |
| 11:11:56 | sean-k-mooney | in generall we discorgage puting the instance directory on nfs by the way. | |
| 11:12:37 | sean-k-mooney | you can do that and we know operators do but its not well tested and there are defintly more bugs in that config. | |
| 11:12:56 | sean-k-mooney | can you check if you ahve a file for me | |
| 11:13:03 | sean-k-mooney | one sec while i look for it | |
| 11:13:55 | gokhani | sean-k-mooney, I have also instance disk under /var/lib/nova/instances/xxxxxxxxx/disk | |
| 11:14:13 | sean-k-mooney | yes thats where they are stored by default | |
| 11:14:22 | sean-k-mooney | you have /var/lib/nova/instances/compute_nodes | |
| 11:14:27 | sean-k-mooney | *do you have | |
| 11:14:55 | gokhani | yes I have | |
| 11:15:08 | sean-k-mooney | does it have muliple entries | |
| 11:15:32 | gokhani | I will check it | |
| 11:15:45 | sean-k-mooney | mine looks like this | |
| 11:15:47 | sean-k-mooney | nova-compute)[nova@cloud instances]$ cat /var/lib/nova/instances/compute_nodes | |
| 11:15:49 | sean-k-mooney | {"cloud": 1669200576.0961165} | |
| 11:16:03 | sean-k-mooney | im not sure if that should have muliple entries for nfs | |
| 11:16:30 | sean-k-mooney | but that is created by the image cache code and i think its related ot deleteion on shared filesystmes | |
| 11:16:56 | sean-k-mooney | im just wondering if it exists and if it has multipel entires the content itself is not really that important | |
| 11:18:00 | gokhani | sean-k-mooney, https://paste.openstack.org/show/bldgeGSIq8BoLjp0u6sx/ | |
| 11:18:28 | sean-k-mooney | ack that is what i was expecting to see | |
| 11:19:15 | sean-k-mooney | that generated here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/storage_users.py#L45-L72 | |
| 11:19:46 | sean-k-mooney | so each compute on shared storage adds themselves to that list | |
| 11:21:24 | sean-k-mooney | https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L10901-L10916 | |
| 11:21:50 | sean-k-mooney | we use that file to get the instnace for all host on the shared storage | |
| 11:21:59 | sean-k-mooney | and tehn we clean the cache based on that | |
| 11:22:14 | sean-k-mooney | on a normal non shared deployment like mine that only has one host | |
| 11:22:20 | sean-k-mooney | in your case it has many | |
| 11:23:22 | sean-k-mooney | i see prod-compute3 but not compute03 in that list | |
| 11:23:40 | gokhani | sean-k-mooney, we need to find how image backing file is deleted . I have this problem on instances whose images are deleted on glance | |
| 11:24:15 | sean-k-mooney | the delete proably happen because of the perodic cleanup | |
| 11:24:44 | sean-k-mooney | but can you verify if the compute node is compute03 or prod-compute3 in the hypervior api or compute_node db table | |
| 11:25:29 | sean-k-mooney | we are using conf.host to populate that file | |
| 11:25:42 | sean-k-mooney | im wondering if you perhaps changed that at some point? | |
| 11:25:48 | sean-k-mooney | or change the hostname | |
| 11:26:02 | sean-k-mooney | to remove the prod- prefix | |
| 11:27:20 | sean-k-mooney | my guess is that currently eitehr the [DEFAULT]/host value does not match the value in the compute_nodes file or its unset and your hostname nolonger matches because it was change somehow | |
| 11:28:01 | sean-k-mooney | that file is updated before every time we cleanup the image cache | |
| 11:28:13 | gokhani | sean-k-mooney, https://paste.openstack.org/show/bVzhWTyWsA3f1Umb5YGM/ | |
| 11:28:20 | sean-k-mooney | so the compute agents think they shoudl be usingthe prod- prefix | |
| 11:28:36 | sean-k-mooney | ok | |
| 11:28:50 | sean-k-mooney | that atleast alines to whats in the file | |
| 11:28:54 | sean-k-mooney | althoguh | |
| 11:29:06 | sean-k-mooney | thats showing the hypervior hostname | |
| 11:29:14 | gokhani | sean-k-mooney, do ı need to also verify on db ? | |
| 11:29:39 | sean-k-mooney | what i actully need is the host value rather then hypervior hostname i think | |
| 11:30:26 | sean-k-mooney | that would be the host value in the compute service entry for example although its also in teh compute nodes table | |
| 11:30:37 | sean-k-mooney | im just trying to verify that they match what is in the file | |
| 11:31:12 | sean-k-mooney | on never mind | |
| 11:31:15 | sean-k-mooney | its fine | |
| 11:31:17 | sean-k-mooney | https://paste.openstack.org/show/b7FSkdeQOxcuWu296tfq/ | |
| 11:31:33 | sean-k-mooney | i had scolled that over and th start of the lines were cut off | |
| 11:31:41 | sean-k-mooney | so its prod-compute03 | |
| 11:31:48 | gokhani | yes | |
| 11:36:07 | sean-k-mooney | do you see this warning any of the logs https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/imagecache.py#L331-L334 | |
| 11:41:07 | gokhani | sean-k-mooney, I dont see any warning like that | |
| 11:43:41 | gokhani | there are periodic checks like that https://paste.openstack.org/show/bGWC1FfgR5R7nXqiCOuQ/ | |
| 11:48:06 | sean-k-mooney | ya that what we expect to see | |
| 11:48:22 | sean-k-mooney | when its workign correctly | |
| 11:49:48 | gokhani | sean-k-mooney, for rescuing these vms it seems there is only one option create glance images from /var/lib/nova/instances/xxx/disk | |
| 11:50:28 | sean-k-mooney | glance does not allow you to create and image with a specifc uuid | |
| 11:50:53 | sean-k-mooney | breifly looking at the code i done se where the bug is | |
| 11:50:59 | sean-k-mooney | https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/imagecache.py#L43 | |
| 11:51:27 | sean-k-mooney | the list_running_isntnace code seams to take into account local and remote instnace | |
| 11:57:32 | sean-k-mooney | i have a meeting now so unfortuetly i cant continue to look at this now | |
| 11:57:52 | sean-k-mooney | the root cause so far is something deleted the backing file | |
| 11:58:18 | sean-k-mooney | since its deleted in glance and its deleted form the nfs share im not sure there is a way to correct this | |
| 11:59:45 | gokhani | thanks sean-k-mooney for your help :) | |
| 14:12:33 | opendevreview | Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432 | |
| 14:14:33 | opendevreview | Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432 | |
| 15:10:43 | opendevreview | Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432 | |
| 16:02:16 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 16:02:17 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 16:58:11 | bauzas | man, the gate is super flakey those days | |
| 16:58:29 | bauzas | lots of rechecks on networing and volume detachs :( | |
| 17:11:30 | gibi | probably the switch to jammy | |
| 17:11:51 | gibi | but this time I have not time to look at what changed with the detach logic again | |
| 17:12:04 | gibi | *I don't have time | |
| 18:28:42 | sean-k-mooney | gibi: im not sure that it really has changed | |
| 18:28:57 | sean-k-mooney | gibi: i think they still have not fixed it | |
| 18:30:42 | sean-k-mooney | gibi: bauzas by the way if eitehr of ye can approve https://review.opendev.org/c/openstack/nova/+/865031 it will simplfy ralonsoh life and help with the trunk port issue | |
| 18:31:12 | sean-k-mooney | https://review.opendev.org/c/openstack/neutron/+/837780 cuurntly need a cofnig option because of our min version | |
| 18:31:26 | sean-k-mooney | but that can be removed if we increase the min version of os-vif | |