| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-11-23 | |||
| 09:03:14 | sean-k-mooney | so that implies that the instance was in an inconsitent state before the reboot | |
| 09:04:08 | sean-k-mooney | ile "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9930, in _create_images_and_backing | |
| 09:07:27 | sean-k-mooney | so this should only be invokded if the vm is started via due to the resume_state_on_host_boot config option after a host reboot | |
| 09:07:37 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L3368-L3377 | |
| 09:08:35 | sean-k-mooney | gokhani: based on the trace back the instnace disk does not exist anymore | |
| 09:08:43 | sean-k-mooney | since it tokk this branch https://github.com/openstack/nova/blob/3224ceb3fffc57d2375e5163d8ffbbb77529bc38/nova/virt/libvirt/driver.py#L9949-L9951 | |
| 09:10:34 | sean-k-mooney | actullly no sorry https://github.com/openstack/nova/blob/3224ceb3fffc57d2375e5163d8ffbbb77529bc38/nova/virt/libvirt/driver.py#L9980-L9986 is the path its taking | |
| 09:11:11 | sean-k-mooney | so info['backing_file'] is equivalent to true | |
| 09:11:37 | sean-k-mooney | and to take the else branch that means that this si not the swap or ephmeral disk | |
| 09:16:49 | sean-k-mooney | anyway as the comment suggest https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L3368-L3377 | |
| 09:17:15 | sean-k-mooney | that code path is there to endure that any backing files that are shared between teh vms are still present on the host | |
| 09:17:49 | sean-k-mooney | so this implies taht the backing file is not presnt on the host and it has been deleted from glance | |
| 09:18:41 | sean-k-mooney | in a normal hard reboot triggered by a human that code path shoudl not be taken and we will just redefine the domain | |
| 09:36:20 | kashyap | sean-k-mooney: Morning. Do we document any of this anywhere, at a high-level? | |
| 09:40:07 | sean-k-mooney | that hard reboot does not redownload the image | |
| 09:40:33 | sean-k-mooney | i dont think so but thats more of a impemation detail | |
| 09:41:07 | sean-k-mooney | it should not be required in a normal workflow sincew we are simulating rebooting a physical server | |
| 09:41:23 | sean-k-mooney | and you would not reinstall the os on a phsyical server | |
| 09:41:36 | sean-k-mooney | the fact we have code to do it after a host reboot is a little odd | |
| 09:41:52 | sean-k-mooney | but i would guess it is tehre because of a bug | |
| 09:42:09 | kashyap | Yeah, that's fair. (I'm not saying impl detail should be be documented. Just maybe somewhere in a debugging guide. I admit I can't find an appropriate place for it, though) | |
| 09:42:22 | sean-k-mooney | i.e. to work around a bug where perhaps at some point in the past the image casche got lost or something | |
| 09:42:41 | kashyap | "Image cache" ... /me runs for the hills | |
| 09:42:54 | kashyap | (Except no hills here in this part of the low lands :P) | |
| 09:43:30 | sean-k-mooney | ya ill admit im kind of surprised we have that code path to try and heal broken/missing backing files | |
| 09:43:56 | kashyap | TIL, too | |
| 09:44:04 | sean-k-mooney | the fallback to using it form the image cache if its not in glance is interesting... | |
| 09:44:13 | sean-k-mooney | it makes sense i guess | |
| 09:44:32 | sean-k-mooney | but this is and edgecase of an edgecase | |
| 09:44:36 | kashyap | Yeah, it definitely does to me. | |
| 09:45:13 | sean-k-mooney | i.e. after a host reboot the backing file need to have been delted and the image need to have been delted form glance but still exist in the host image cache for that to work | |
| 09:46:11 | sean-k-mooney | im sure thats the exact case someone hit and they added this to fix it however | |
| 09:57:33 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 09:57:34 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 09:57:35 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 09:57:36 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 09:57:38 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Add _live_migrate helper to InstanceHelperMixin https://review.opendev.org/c/openstack/nova/+/865381 | |
| 09:57:40 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Introduce a server_expected_state kwarg to InstanceHelperMixin._live_migrate https://review.opendev.org/c/openstack/nova/+/865382 | |
| 10:20:48 | auniyal | Hi sean-k-mooney | |
| 10:21:06 | auniyal | can you please review these patches - https://review.opendev.org/c/openstack/nova/+/864055 | |
| 10:23:14 | opendevreview | Anton Kurbatov proposed openstack/nova master: Fix VMs sorting fail in case of comparison with None https://review.opendev.org/c/openstack/nova/+/865037 | |
| 10:23:15 | sean-k-mooney | i can look quickly but i dont really have time for nova work this week outside of the vdpa rebases. | |
| 10:23:53 | sean-k-mooney | oh that the train backport | |
| 10:24:54 | auniyal | yeah, the last patch is failing before migration | |
| 10:42:02 | gokhani | sean-k-mooney, you are right, instance is in unconsistent state and I tried reboot it. Glance image is already deleted and so base image is also deleted. I didn't activate resume_state_on_host_boot option, it is false. | |
| 10:43:32 | sean-k-mooney | this code path is only taken if you dont have an authtoken | |
| 10:43:42 | sean-k-mooney | so you should not be able to get here manually | |
| 10:43:56 | sean-k-mooney | gokhani: how did you trigger the hard reboot | |
| 10:46:14 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 10:46:15 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 10:46:16 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 10:46:17 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 10:47:05 | gokhani | sean-k-mooney, firstly run "nova reset-state --active 3be526a1-664c-4166-8212-a0b979259ddf" and after that "openstack server reboot --hard 3be526a1-664c-4166-8212-a0b979259ddf" | |
| 10:47:32 | sean-k-mooney | the second command should not have taken this code path | |
| 10:48:03 | sean-k-mooney | its guarded by "if context.auth_token is not None:" | |
| 10:49:44 | sean-k-mooney | if hard_reboot was trigged form the api then there should always be an authtoken | |
| 10:49:51 | bauzas | sean-k-mooney: can you drop your -1 on https://review.opendev.org/c/openstack/nova/+/864418 now that I flipped the patches in the series ? | |
| 10:50:05 | sean-k-mooney | is that the gpu stuff | |
| 10:50:09 | sean-k-mooney | yep | |
| 10:50:10 | bauzas | yes | |
| 10:50:54 | sean-k-mooney | ill swap it to a review priorty +1 but i wont get to it today | |
| 10:51:21 | sean-k-mooney | ill see if i can loop back to it later in the week | |
| 10:52:14 | gokhani | sean-k-mooney, I will try to reproduce this problem with trying to reboot another instance | |
| 10:53:01 | sean-k-mooney | gokhani: ack if you can repoduced it with a knwon good test instance then that would help | |
| 10:53:16 | sean-k-mooney | gokhani: your using victoria correct | |
| 10:53:34 | sean-k-mooney | i noticed nova 22.somehting in the trace | |
| 10:54:03 | gokhani | sean-k-mooney, yes our env is victoria | |
| 10:54:08 | sean-k-mooney | i didnt check the same section of code in master but its possible that this was a bug that was fix in between | |
| 10:55:08 | sean-k-mooney | no that funciton is the same on master | |
| 10:55:15 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3931 | |
| 10:55:26 | sean-k-mooney | so if there is a bug it should happen there too. | |
| 10:58:49 | gokhani | sean-k-mooney, I tried to soft reboot an instance and it throws error again | |
| 10:58:51 | gokhani | https://paste.openstack.org/show/b7FSkdeQOxcuWu296tfq/ | |
| 10:59:12 | gokhani | rebooting is triggered with horizon | |
| 10:59:49 | sean-k-mooney | so the inital error Cannot access backing file '/var/lib/nova/instances/_base/9d36e2c635ce070d95805f64f4b34655f3eae96b' of storage file '/var/lib/nova/instances/4a34a2f4-98a4-4cea-8d80-96d28f12edd5/disk' | |
| 11:00:13 | sean-k-mooney | indicate that something deleted the disk image from behind nova's back | |
| 11:00:34 | sean-k-mooney | soft reboot escalate to hard reboot if it fails | |
| 11:00:43 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Introduce a server_expected_state kwarg to InstanceHelperMixin._live_migrate https://review.opendev.org/c/openstack/nova/+/865382 | |
| 11:00:44 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 11:00:45 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 11:00:46 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 11:00:47 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 11:01:13 | sean-k-mooney | gokhani: we might not be passign the auth token propelry in the context when we do that | |
| 11:01:42 | sean-k-mooney | gokhani: the code path that it is taking woudl repair the issue that it detected in soft reboot | |
| 11:01:48 | sean-k-mooney | if the image still existed in glance | |
| 11:02:18 | sean-k-mooney | if its deltete tehre is no way to fix the vm without copying the backing file form somewhere else in teh cloud that still has it | |
| 11:02:57 | sean-k-mooney | if /var/lib/nova/instances/_base/9d36e2c635ce070d95805f64f4b34655f3eae96b exist somewhere else on your cloud and you copy it to that location on this host | |
| 11:03:16 | sean-k-mooney | and ensure it has the corret user permissions as the other images there | |
| 11:03:26 | sean-k-mooney | then you might be able to recover the vm | |
| 11:03:34 | sean-k-mooney | with another hard/soft reboot | |
| 11:04:07 | gokhani | sean-k-mooney, I deleted image of this instance in glance. | |
| 11:04:21 | sean-k-mooney | sure that should be fine | |
| 11:04:37 | sean-k-mooney | but something also deleted the image backing file on the compute node | |
| 11:04:58 | sean-k-mooney | that is not ok and should not happen if there is an instance based on that image on the host | |
| 11:06:14 | sean-k-mooney | you have configured your deployment to use qcow images for the vm with a backign file | |
| 11:06:22 | gokhani | sean-k-mooney, what can be the reason of deleting image backing file ? Is there any config on nova? | |
| 11:06:49 | sean-k-mooney | this will only be deleted by nova if there are no vms on this host based on that glance image | |
| 11:07:10 | sean-k-mooney | you are not mounting this on a nfs share or soemthign liek that? | |
| 11:08:00 | gokhani | sean-k-mooney, yes I have netapp storage and Instance disks are on nfs share | |
| 11:08:13 | sean-k-mooney | nfsv3...? | |