| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-11-23 | |||
| 09:57:33 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 09:57:34 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 09:57:35 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 09:57:36 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 09:57:38 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Add _live_migrate helper to InstanceHelperMixin https://review.opendev.org/c/openstack/nova/+/865381 | |
| 09:57:40 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Introduce a server_expected_state kwarg to InstanceHelperMixin._live_migrate https://review.opendev.org/c/openstack/nova/+/865382 | |
| 10:20:48 | auniyal | Hi sean-k-mooney | |
| 10:21:06 | auniyal | can you please review these patches - https://review.opendev.org/c/openstack/nova/+/864055 | |
| 10:23:14 | opendevreview | Anton Kurbatov proposed openstack/nova master: Fix VMs sorting fail in case of comparison with None https://review.opendev.org/c/openstack/nova/+/865037 | |
| 10:23:15 | sean-k-mooney | i can look quickly but i dont really have time for nova work this week outside of the vdpa rebases. | |
| 10:23:53 | sean-k-mooney | oh that the train backport | |
| 10:24:54 | auniyal | yeah, the last patch is failing before migration | |
| 10:42:02 | gokhani | sean-k-mooney, you are right, instance is in unconsistent state and I tried reboot it. Glance image is already deleted and so base image is also deleted. I didn't activate resume_state_on_host_boot option, it is false. | |
| 10:43:32 | sean-k-mooney | this code path is only taken if you dont have an authtoken | |
| 10:43:42 | sean-k-mooney | so you should not be able to get here manually | |
| 10:43:56 | sean-k-mooney | gokhani: how did you trigger the hard reboot | |
| 10:46:14 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 10:46:15 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 10:46:16 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 10:46:17 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 10:47:05 | gokhani | sean-k-mooney, firstly run "nova reset-state --active 3be526a1-664c-4166-8212-a0b979259ddf" and after that "openstack server reboot --hard 3be526a1-664c-4166-8212-a0b979259ddf" | |
| 10:47:32 | sean-k-mooney | the second command should not have taken this code path | |
| 10:48:03 | sean-k-mooney | its guarded by "if context.auth_token is not None:" | |
| 10:49:44 | sean-k-mooney | if hard_reboot was trigged form the api then there should always be an authtoken | |
| 10:49:51 | bauzas | sean-k-mooney: can you drop your -1 on https://review.opendev.org/c/openstack/nova/+/864418 now that I flipped the patches in the series ? | |
| 10:50:05 | sean-k-mooney | is that the gpu stuff | |
| 10:50:09 | sean-k-mooney | yep | |
| 10:50:10 | bauzas | yes | |
| 10:50:54 | sean-k-mooney | ill swap it to a review priorty +1 but i wont get to it today | |
| 10:51:21 | sean-k-mooney | ill see if i can loop back to it later in the week | |
| 10:52:14 | gokhani | sean-k-mooney, I will try to reproduce this problem with trying to reboot another instance | |
| 10:53:01 | sean-k-mooney | gokhani: ack if you can repoduced it with a knwon good test instance then that would help | |
| 10:53:16 | sean-k-mooney | gokhani: your using victoria correct | |
| 10:53:34 | sean-k-mooney | i noticed nova 22.somehting in the trace | |
| 10:54:03 | gokhani | sean-k-mooney, yes our env is victoria | |
| 10:54:08 | sean-k-mooney | i didnt check the same section of code in master but its possible that this was a bug that was fix in between | |
| 10:55:08 | sean-k-mooney | no that funciton is the same on master | |
| 10:55:15 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3931 | |
| 10:55:26 | sean-k-mooney | so if there is a bug it should happen there too. | |
| 10:58:49 | gokhani | sean-k-mooney, I tried to soft reboot an instance and it throws error again | |
| 10:58:51 | gokhani | https://paste.openstack.org/show/b7FSkdeQOxcuWu296tfq/ | |
| 10:59:12 | gokhani | rebooting is triggered with horizon | |
| 10:59:49 | sean-k-mooney | so the inital error Cannot access backing file '/var/lib/nova/instances/_base/9d36e2c635ce070d95805f64f4b34655f3eae96b' of storage file '/var/lib/nova/instances/4a34a2f4-98a4-4cea-8d80-96d28f12edd5/disk' | |
| 11:00:13 | sean-k-mooney | indicate that something deleted the disk image from behind nova's back | |
| 11:00:34 | sean-k-mooney | soft reboot escalate to hard reboot if it fails | |
| 11:00:43 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Introduce a server_expected_state kwarg to InstanceHelperMixin._live_migrate https://review.opendev.org/c/openstack/nova/+/865382 | |
| 11:00:44 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 11:00:45 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 11:00:46 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 11:00:47 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 11:01:13 | sean-k-mooney | gokhani: we might not be passign the auth token propelry in the context when we do that | |
| 11:01:42 | sean-k-mooney | gokhani: the code path that it is taking woudl repair the issue that it detected in soft reboot | |
| 11:01:48 | sean-k-mooney | if the image still existed in glance | |
| 11:02:18 | sean-k-mooney | if its deltete tehre is no way to fix the vm without copying the backing file form somewhere else in teh cloud that still has it | |
| 11:02:57 | sean-k-mooney | if /var/lib/nova/instances/_base/9d36e2c635ce070d95805f64f4b34655f3eae96b exist somewhere else on your cloud and you copy it to that location on this host | |
| 11:03:16 | sean-k-mooney | and ensure it has the corret user permissions as the other images there | |
| 11:03:26 | sean-k-mooney | then you might be able to recover the vm | |
| 11:03:34 | sean-k-mooney | with another hard/soft reboot | |
| 11:04:07 | gokhani | sean-k-mooney, I deleted image of this instance in glance. | |
| 11:04:21 | sean-k-mooney | sure that should be fine | |
| 11:04:37 | sean-k-mooney | but something also deleted the image backing file on the compute node | |
| 11:04:58 | sean-k-mooney | that is not ok and should not happen if there is an instance based on that image on the host | |
| 11:06:14 | sean-k-mooney | you have configured your deployment to use qcow images for the vm with a backign file | |
| 11:06:22 | gokhani | sean-k-mooney, what can be the reason of deleting image backing file ? Is there any config on nova? | |
| 11:06:49 | sean-k-mooney | this will only be deleted by nova if there are no vms on this host based on that glance image | |
| 11:07:10 | sean-k-mooney | you are not mounting this on a nfs share or soemthign liek that? | |
| 11:08:00 | gokhani | sean-k-mooney, yes I have netapp storage and Instance disks are on nfs share | |
| 11:08:13 | sean-k-mooney | nfsv3...? | |
| 11:09:46 | gokhani | nfs4 | |
| 11:09:49 | sean-k-mooney | nfs v3 has consitey/locking issues and we stongly discurage using it. ideally you woudl use v4.2+ with nova. | |
| 11:09:52 | gokhani | nfsv4 | |
| 11:09:53 | sean-k-mooney | ok | |
| 11:10:27 | sean-k-mooney | so the only thing that comes to mind is if you are not using a sepeate share or directory per host | |
| 11:11:02 | sean-k-mooney | then if the /var/lib/nova/instances/_base is shared one of the other nova compute could have deleted it if it did not detect its on a shared file system properly | |
| 11:11:30 | gokhani | sean-k-mooney, I am using same share for all compute nodes | |
| 11:11:56 | sean-k-mooney | in generall we discorgage puting the instance directory on nfs by the way. | |
| 11:12:37 | sean-k-mooney | you can do that and we know operators do but its not well tested and there are defintly more bugs in that config. | |
| 11:12:56 | sean-k-mooney | can you check if you ahve a file for me | |
| 11:13:03 | sean-k-mooney | one sec while i look for it | |
| 11:13:55 | gokhani | sean-k-mooney, I have also instance disk under /var/lib/nova/instances/xxxxxxxxx/disk | |
| 11:14:13 | sean-k-mooney | yes thats where they are stored by default | |
| 11:14:22 | sean-k-mooney | you have /var/lib/nova/instances/compute_nodes | |
| 11:14:27 | sean-k-mooney | *do you have | |
| 11:14:55 | gokhani | yes I have | |
| 11:15:08 | sean-k-mooney | does it have muliple entries | |
| 11:15:32 | gokhani | I will check it | |
| 11:15:45 | sean-k-mooney | mine looks like this | |
| 11:15:47 | sean-k-mooney | nova-compute)[nova@cloud instances]$ cat /var/lib/nova/instances/compute_nodes | |
| 11:15:49 | sean-k-mooney | {"cloud": 1669200576.0961165} | |
| 11:16:03 | sean-k-mooney | im not sure if that should have muliple entries for nfs | |
| 11:16:30 | sean-k-mooney | but that is created by the image cache code and i think its related ot deleteion on shared filesystmes | |
| 11:16:56 | sean-k-mooney | im just wondering if it exists and if it has multipel entires the content itself is not really that important | |
| 11:18:00 | gokhani | sean-k-mooney, https://paste.openstack.org/show/bldgeGSIq8BoLjp0u6sx/ | |
| 11:18:28 | sean-k-mooney | ack that is what i was expecting to see | |
| 11:19:15 | sean-k-mooney | that generated here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/storage_users.py#L45-L72 | |
| 11:19:46 | sean-k-mooney | so each compute on shared storage adds themselves to that list | |
| 11:21:24 | sean-k-mooney | https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L10901-L10916 | |
| 11:21:50 | sean-k-mooney | we use that file to get the instnace for all host on the shared storage | |
| 11:21:59 | sean-k-mooney | and tehn we clean the cache based on that | |
| 11:22:14 | sean-k-mooney | on a normal non shared deployment like mine that only has one host | |