| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-08 | |||
| 16:06:11 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372 | |
| 16:44:23 | ygk_12345 | @gibi hi | |
| 16:54:11 | ygk_12345 | hi all | |
| 16:54:52 | ygk_12345 | i have upgraded the nova from stein to train and I have rebooted the computes. but after the compute is up , it is not listing all the vms in the virsh list | |
| 16:55:17 | ygk_12345 | but the vm's disk files are intact as /var/lib/nova/instances folder | |
| 16:55:23 | ygk_12345 | why is it so ? | |
| 16:55:30 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Suppress remaining policy warnings in unit tests https://review.opendev.org/726272 | |
| 16:56:17 | ygk_12345 | but the vm's disk files are intact in /var/lib/nova/instances folder | |
| 16:57:30 | ygk_12345 | can someone help me please | |
| 16:58:02 | stephenfin | ygk_12345: are the instances appearing if you do 'nova list'? | |
| 16:58:24 | ygk_12345 | stephenfin yes in ERROR state | |
| 16:58:35 | ygk_12345 | i tried reset and rebooted bbut still not coming in the virsh list | |
| 16:59:06 | stephenfin | ygk_12345: do you have 'resume_guests_state_on_host_boot' configured? I assume not? | |
| 16:59:24 | ygk_12345 | stephenfin where to check that ? | |
| 16:59:27 | stephenfin | nova.conf | |
| 16:59:32 | ygk_12345 | let me check | |
| 16:59:33 | stephenfin | If that is left to it's default value, nova won't restart VMs for you | |
| 16:59:50 | openstackgerrit | Takashi Natsume proposed openstack/os-vif master: Remove translation sections from setup.cfg https://review.opendev.org/724974 | |
| 17:00:02 | stephenfin | though the domains should still be there, I would have thought | |
| 17:00:03 | ygk_12345 | so how do I recover those now ? the disk files are intact in the /var/lib/nova folder though | |
| 17:00:19 | ygk_12345 | stephenfin the virsh is not listing all the domain though | |
| 17:01:31 | ygk_12345 | stephenfin yes it is configured "resume_guests_state_on_host_boot = True" | |
| 17:01:52 | stephenfin | Odd. What do the logs say? | |
| 17:02:38 | ygk_12345 | stephenfin the computes log is complaining about just one missing disk of a vm at "nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/e7fefd4e-f2cf-4048-9b37-4ff807243291/disk" | |
| 17:03:22 | stephenfin | can you rebuild that instance? | |
| 17:03:31 | ygk_12345 | stephenfin how ? | |
| 17:04:18 | stephenfin | nova rebuild SERVER_ID IMAGE_ID | |
| 17:04:59 | ygk_12345 | stephenfin ok let me try. please hold on | |
| 17:05:28 | ygk_12345 | stephenfin will I loose any data on this vm ? | |
| 17:05:45 | stephenfin | Yes. Rebuild is a destructive operation | |
| 17:06:19 | ygk_12345 | in that case I can also extract the disk file as image and spawn the vm right ? | |
| 17:06:42 | ygk_12345 | i dont want ot loose data | |
| 17:07:16 | ygk_12345 | stephenfin is there a way to restore those vms ? | |
| 17:07:58 | ygk_12345 | stephenfin I see lot of disk files in the location /var/lib/nova with the uudi's folders | |
| 17:08:07 | ygk_12345 | vm uuids | |
| 17:10:46 | ygk_12345 | can someone help me with this issue ? | |
| 17:10:58 | ygk_12345 | i see active base files still at the folder | |
| 17:12:24 | sean-k-mooney | gmann: ah yes that is the spec i was tinking of. | |
| 17:12:58 | sean-k-mooney | ygk_12345: you can snapshot the vm | |
| 17:13:03 | ygk_12345 | somehow it is not lsiting other virsh domains | |
| 17:13:30 | ygk_12345 | sean-k-mooney even when it is in error state u mean ? | |
| 17:13:45 | sean-k-mooney | oh no im justre reading back now | |
| 17:14:08 | sean-k-mooney | so in its curent state i dont think snapshot would work. | |
| 17:14:26 | sean-k-mooney | but that is generaly how you would preseve the content if you were going to rebuild | |
| 17:15:20 | ygk_12345 | sean-k-mooney why is it not listing the other virsh domains ? | |
| 17:15:31 | sean-k-mooney | ygk_12345: do you have your /var/lib/nova on nfs or other shared storage? | |
| 17:15:45 | sean-k-mooney | ygk_12345: are you using sudo | |
| 17:15:54 | ygk_12345 | sean-k-mooney yes it is an lvm share | |
| 17:15:58 | ygk_12345 | we mounted it | |
| 17:16:10 | sean-k-mooney | lvm is not a shared file system | |
| 17:16:26 | sean-k-mooney | as in shared over the netwrok between multiple hosts | |
| 17:16:27 | ygk_12345 | sean-k-mooney yes | |
| 17:16:36 | ygk_12345 | not in that sense | |
| 17:16:43 | ygk_12345 | it is deidcted to taht server | |
| 17:16:48 | ygk_12345 | *that host | |
| 17:17:11 | sean-k-mooney | there was a bug related to sometimes incorrectly deleting the vm disk if migration or evac failed on nfs not too loong ago | |
| 17:17:24 | sean-k-mooney | i was wondering if that is who you got into this state | |
| 17:17:32 | sean-k-mooney | it woudl appear not to be relatecxd | |
| 17:18:09 | ygk_12345 | sean-k-mooney this is just lvm and the uuid folders are still present in the /var/lib/nova folder which is an lvm | |
| 17:18:31 | sean-k-mooney | yep but the disk file is not correct | |
| 17:18:54 | ygk_12345 | sean-k-mooney what do u mean by correct ? | |
| 17:19:20 | sean-k-mooney | /var/lib/nova/instances/e7fefd4e-f2cf-4048-9b37-4ff807243291/disk is not present wehn you check | |
| 17:19:33 | ygk_12345 | sean-k-mooney yes its there | |
| 17:19:49 | sean-k-mooney | oh its there but nova thinks its not there | |
| 17:19:51 | ygk_12345 | oh yes that only one file is absent | |
| 17:19:58 | ygk_12345 | rest are there | |
| 17:20:00 | sean-k-mooney | ok | |
| 17:20:06 | ygk_12345 | but the domains have vanished | |
| 17:20:21 | ygk_12345 | thats what I am trying to say | |
| 17:20:36 | sean-k-mooney | sudo if you do "sudo virsh list -all" | |
| 17:20:41 | sean-k-mooney | there is nothing | |
| 17:20:42 | ygk_12345 | let me check | |
| 17:20:52 | sean-k-mooney | its either -all or --all i cant rememeber | |
| 17:21:12 | sean-k-mooney | nova creates the vms in the system namespace | |
| 17:21:35 | ygk_12345 | sean-k-mooney it is showing only two domains, but it shol;d have 30 + domains | |
| 17:21:36 | sean-k-mooney | if you dont use sudo on some operating sytems be default you wont see domains in the system namespace | |
| 17:22:17 | sean-k-mooney | oh just read back you just upgraded? | |
| 17:22:39 | sean-k-mooney | i need to check one thing | |
| 17:22:44 | ygk_12345 | yes | |
| 17:22:49 | ygk_12345 | from stein to train | |
| 17:22:54 | ygk_12345 | one compute node is fine | |
| 17:23:02 | ygk_12345 | the rest three are having this problem | |
| 17:23:10 | ygk_12345 | not listing all the domains | |
| 17:23:28 | ygk_12345 | after a reboot | |
| 17:23:41 | ygk_12345 | i have to manually mount the lvm | |
| 17:23:54 | sean-k-mooney | ya so im not sure which release we changed it in but we fixed a bug with evacuation a couple of release ago where we did not properly clean up domains. but the side effect of that is if you did an evacuate and it failed and you did a reset state | |
| 17:23:55 | ygk_12345 | on to the /var/lib/nova/instances | |
| 17:24:13 | sean-k-mooney | then after upgrade the domain could be deleted | |
| 17:24:28 | ygk_12345 | but they are intact in one compute | |
| 17:24:47 | ygk_12345 | i still see the disk files with nova uuids present | |
| 17:24:53 | sean-k-mooney | it only affected instance where the operator incoorect did reset state | |
| 17:24:59 | ygk_12345 | but onl;;y virsh domains vanished | |
| 17:25:19 | sean-k-mooney | basically im wondering if the period jobs deleted the vms | |
| 17:25:25 | ygk_12345 | many vms went into ERROR state | |
| 17:25:35 | ygk_12345 | no such crons | |
| 17:26:06 | sean-k-mooney | ygk_12345: its not a cron job we have a period task in the nova compute agent | |
| 17:26:16 | ygk_12345 | ok | |
| 17:26:24 | sean-k-mooney | that looks for stale domains form evacuation or migratiosn and deletes them | |
| 17:26:25 | ygk_12345 | can we restore these vms now ? | |
| 17:26:44 | ygk_12345 | these were active vms before the upgrade | |