| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-08 | |||
| 17:16:36 | ygk_12345 | not in that sense | |
| 17:16:43 | ygk_12345 | it is deidcted to taht server | |
| 17:16:48 | ygk_12345 | *that host | |
| 17:17:11 | sean-k-mooney | there was a bug related to sometimes incorrectly deleting the vm disk if migration or evac failed on nfs not too loong ago | |
| 17:17:24 | sean-k-mooney | i was wondering if that is who you got into this state | |
| 17:17:32 | sean-k-mooney | it woudl appear not to be relatecxd | |
| 17:18:09 | ygk_12345 | sean-k-mooney this is just lvm and the uuid folders are still present in the /var/lib/nova folder which is an lvm | |
| 17:18:31 | sean-k-mooney | yep but the disk file is not correct | |
| 17:18:54 | ygk_12345 | sean-k-mooney what do u mean by correct ? | |
| 17:19:20 | sean-k-mooney | /var/lib/nova/instances/e7fefd4e-f2cf-4048-9b37-4ff807243291/disk is not present wehn you check | |
| 17:19:33 | ygk_12345 | sean-k-mooney yes its there | |
| 17:19:49 | sean-k-mooney | oh its there but nova thinks its not there | |
| 17:19:51 | ygk_12345 | oh yes that only one file is absent | |
| 17:19:58 | ygk_12345 | rest are there | |
| 17:20:00 | sean-k-mooney | ok | |
| 17:20:06 | ygk_12345 | but the domains have vanished | |
| 17:20:21 | ygk_12345 | thats what I am trying to say | |
| 17:20:36 | sean-k-mooney | sudo if you do "sudo virsh list -all" | |
| 17:20:41 | sean-k-mooney | there is nothing | |
| 17:20:42 | ygk_12345 | let me check | |
| 17:20:52 | sean-k-mooney | its either -all or --all i cant rememeber | |
| 17:21:12 | sean-k-mooney | nova creates the vms in the system namespace | |
| 17:21:35 | ygk_12345 | sean-k-mooney it is showing only two domains, but it shol;d have 30 + domains | |
| 17:21:36 | sean-k-mooney | if you dont use sudo on some operating sytems be default you wont see domains in the system namespace | |
| 17:22:17 | sean-k-mooney | oh just read back you just upgraded? | |
| 17:22:39 | sean-k-mooney | i need to check one thing | |
| 17:22:44 | ygk_12345 | yes | |
| 17:22:49 | ygk_12345 | from stein to train | |
| 17:22:54 | ygk_12345 | one compute node is fine | |
| 17:23:02 | ygk_12345 | the rest three are having this problem | |
| 17:23:10 | ygk_12345 | not listing all the domains | |
| 17:23:28 | ygk_12345 | after a reboot | |
| 17:23:41 | ygk_12345 | i have to manually mount the lvm | |
| 17:23:54 | sean-k-mooney | ya so im not sure which release we changed it in but we fixed a bug with evacuation a couple of release ago where we did not properly clean up domains. but the side effect of that is if you did an evacuate and it failed and you did a reset state | |
| 17:23:55 | ygk_12345 | on to the /var/lib/nova/instances | |
| 17:24:13 | sean-k-mooney | then after upgrade the domain could be deleted | |
| 17:24:28 | ygk_12345 | but they are intact in one compute | |
| 17:24:47 | ygk_12345 | i still see the disk files with nova uuids present | |
| 17:24:53 | sean-k-mooney | it only affected instance where the operator incoorect did reset state | |
| 17:24:59 | ygk_12345 | but onl;;y virsh domains vanished | |
| 17:25:19 | sean-k-mooney | basically im wondering if the period jobs deleted the vms | |
| 17:25:25 | ygk_12345 | many vms went into ERROR state | |
| 17:25:35 | ygk_12345 | no such crons | |
| 17:26:06 | sean-k-mooney | ygk_12345: its not a cron job we have a period task in the nova compute agent | |
| 17:26:16 | ygk_12345 | ok | |
| 17:26:24 | sean-k-mooney | that looks for stale domains form evacuation or migratiosn and deletes them | |
| 17:26:25 | ygk_12345 | can we restore these vms now ? | |
| 17:26:44 | ygk_12345 | these were active vms before the upgrade | |
| 17:28:03 | ygk_12345 | i still see the active base files in the log for these vanished domaisn | |
| 17:28:59 | sean-k-mooney | yes but there are cases where they can be active because you did "nova reset-state" to put them in active but did not clean up migrations or evac | |
| 17:29:21 | ygk_12345 | no ididn't do any reset either | |
| 17:29:30 | ygk_12345 | i still see the base disk files intact | |
| 17:30:32 | sean-k-mooney | ygk_12345: you dont see a log message like this do you https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9851-L9853 | |
| 17:31:15 | ygk_12345 | which message to be exact ? | |
| 17:32:13 | ygk_12345 | i see these vms in ERROR state | |
| 17:32:32 | sean-k-mooney | "Destroying instance as it has been evacuated from this host but still exists in the hypervisor ..." | |
| 17:32:57 | sean-k-mooney | or "Destroying instance with name label " | |
| 17:32:58 | sean-k-mooney | "'%s' which is marked as " | |
| 17:33:00 | sean-k-mooney | "DELETED but still present on host." | |
| 17:33:07 | ygk_12345 | nope | |
| 17:33:23 | ygk_12345 | its just in ERROR state | |
| 17:36:42 | sean-k-mooney | ok i was makeing sure it was not related too https://github.com/openstack/nova/commit/c4988cdabf311d29cf64af732091068cfabeedaa | |
| 17:36:58 | openstack | Launchpad bug 1764883 in OpenStack Compute (nova) queens "Evacuation fails if the source host returns while the migration is still in progress" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood) | |
| 17:36:58 | sean-k-mooney | which fixes https://bugs.launchpad.net/nova/+bug/1764883 | |
| 17:37:22 | sean-k-mooney | but its not | |
| 17:38:22 | sean-k-mooney | that cause some vms in the active state to be deleted if operators incorrectly started an evacuation which failed and then reset the vm state. | |
| 17:38:57 | sean-k-mooney | ygk_12345: in this case im not aware of a bug that would result in the domains being deleted as part of a upgrade | |
| 17:39:43 | ygk_12345 | we did not evacuate. just upgraded from stein to train. one of the four computes is fine | |
| 17:40:09 | ygk_12345 | only virsh domains on these three computes vanished | |
| 17:40:14 | ygk_12345 | but the disk files are there | |
| 17:41:52 | artom | ygk_12345, so `virsh list` is empty? What about `virsh list --all`? | |
| 18:33:21 | sean-k-mooney | artom: it only has 2 domains out of like 30 | |
| 18:33:30 | sean-k-mooney | so something undefiend all the domains | |
| 18:33:59 | sean-k-mooney | and since the log message for the only code that i know of in nova that will do that is not present i dont really know why that happened | |
| 18:34:25 | sean-k-mooney | e.g. it was not the evac cleanup code or the preiodic take for deleted instances | |
| 18:35:02 | sean-k-mooney | so something undefiend the domains | |
| 18:39:08 | artom | sean-k-mooney, and since ygk are gone, we'll never know what happened :P | |
| 18:39:45 | sean-k-mooney | ya i dont really know how you woudl fix this either | |
| 18:40:03 | sean-k-mooney | i think the root disk files are already deleted | |
| 18:40:21 | sean-k-mooney | so im not sure there is much you can do at this point | |
| 21:59:15 | jharris | hey, I am trying to get a nova profile working, and having trouble with blockmapping | |
| 21:59:18 | jharris | do I need to put the image in the block mapping as well as the volume I want to mount? | |
| 22:07:36 | openstackgerrit | Merged openstack/os-vif master: Remove translation sections from setup.cfg https://review.opendev.org/724974 | |
| #openstack-nova - 2020-05-09 | |||
| 01:35:24 | openstackgerrit | xuyuanhao proposed openstack/nova master: the vms can not be force deleted when vm_status is soft-delete and task-state=deleting https://review.opendev.org/724260 | |
| 06:30:15 | openstackgerrit | Takashi Natsume proposed openstack/os-vif master: Use unittest.mock instead of third party mock https://review.opendev.org/724965 | |
| 09:14:47 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Add retry to cinder API calls related to volume detach https://review.opendev.org/726508 | |
| 09:54:42 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3 https://review.opendev.org/726512 | |
| 14:15:16 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3 https://review.opendev.org/726512 | |
| #openstack-nova - 2020-05-10 | |||
| 01:19:47 | openstackgerrit | Brin Zhang proposed openstack/nova master: Optimize _create_and_bind_arqs logic in conducor https://review.opendev.org/726564 | |
| 03:10:53 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.add_metaclass https://review.opendev.org/726565 | |
| #openstack-nova - 2020-05-11 | |||
| 04:44:14 | openstackgerrit | Elod Illes proposed openstack/nova stable/rocky: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.opendev.org/711233 | |
| 06:40:31 | openstackgerrit | Xinran WANG proposed openstack/os-resource-classes master: Add new resource class for QAT card. https://review.opendev.org/726314 | |
| 08:05:23 | gibi | good morning Nova | |
| 08:07:31 | lyarwood | Morning \o | |
| 08:12:03 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439 | |
| 08:24:09 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439 | |
| 08:37:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'Migration' https://review.opendev.org/723572 | |
| 08:37:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'InstancePCIRequest' https://review.opendev.org/723573 | |
| 09:07:44 | brinzhang | submmit a patch to optimize it, would you like to review https://review.opendev.org/#/c/726564/ | |
| 09:07:44 | brinzhang | gibi, bauzas: When I researching the define nova cyborg interaction notification, I found some code logical not suitable, so I | |