| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-09-30 | |||
| 15:37:24 | bauzas | now the problem is how to get this | |
| 15:37:27 | bauzas | and how to present this | |
| 15:44:15 | clarkb | yup, I think the raeson it hasn't been done is that it is complicated | |
| 22:06:01 | atmark | how to properly to delete a VM stuck on BUILD status? openstack server delete throws no server with name or ID exists | |
| 22:07:07 | sean-k-mooney | that is the correct way if we have gotten to the point of creating a instance object in the db | |
| 22:07:30 | sean-k-mooney | if its stuck before that point then there should be no recored to delete | |
| 22:08:14 | atmark | server list still list the instance | |
| 22:08:35 | atmark | how can I get rid of the record? | |
| 22:08:58 | sean-k-mooney | you should be able to just do a server delete | |
| 22:09:16 | sean-k-mooney | if you cant then you have a broken deployment and you need to triage why the request is failing | |
| 22:09:45 | sean-k-mooney | did you try deleteing with the uuid | |
| 22:09:52 | sean-k-mooney | rahter then the name | |
| 22:10:12 | sean-k-mooney | if that fails you should check the nova-api to see if there is an internal excption | |
| 22:10:37 | atmark | i'm deleting by uuid but response is no name or ID or exists | |
| 22:11:07 | atmark | i think this an old instance from last year | |
| 22:11:19 | sean-k-mooney | ack there is obviouly an error internally without knowing what that is there is not much more i can suggest | |
| 22:11:32 | sean-k-mooney | if you take the request id you shoudl be able to find it in the api logs | |
| 22:11:47 | atmark | i'm doing cleanup atm and just caught one on weird state | |
| 22:12:07 | sean-k-mooney | its possible that it exists in teh api db but not in the cell db | |
| 22:12:22 | sean-k-mooney | if you do a server show does it have a host | |
| 22:12:29 | atmark | no | |
| 22:12:57 | sean-k-mooney | ok so its before the schuiler has selected one most likely | |
| 22:13:43 | atmark | https://paste.openstack.org/show/bXZv2CMnmfdGeswoUKUC/ | |
| 22:14:23 | atmark | VM is octavia amphorae | |
| 22:14:48 | sean-k-mooney | --all is all tenants | |
| 22:15:04 | sean-k-mooney | is that owned by your current project | |
| 22:15:18 | johnsom | Yep, the instance is owned by the Octavia service account. | |
| 22:15:49 | sean-k-mooney | no what i mean is when openstack server delete 6fb27c67-539c-4525-8747-e26487d15e75 is run | |
| 22:15:49 | atmark | yes, i have admin access to all tenants | |
| 22:15:52 | sean-k-mooney | what user is that | |
| 22:16:04 | sean-k-mooney | ok so thats an admin user | |
| 22:16:41 | sean-k-mooney | i think you should be able to delete the vm then even if your current token is not for the correct project due to admin right if you are not using new policy | |
| 22:17:30 | sean-k-mooney | the server show is implying that its not in the current project | |
| 22:17:46 | sean-k-mooney | so i was wondering if the delete was failing for the same reason | |
| 22:18:04 | johnsom | There is a --all-projects for the server delete command too, just like for list. | |
| 22:18:13 | sean-k-mooney | i tought admins could bypass that check | |
| 22:18:25 | johnsom | I thought so too honestly | |
| 22:19:04 | sean-k-mooney | its been a while since i have done that so cant recall if you need to set the project id somehow or not | |
| 22:20:17 | atmark | openstack server --os-project-id 848fe125a93a408ba8a8044fb87e9cdf delete 6fb27c67-539c-4525-8747-e26487d15e75 | |
| 22:20:48 | atmark | 848.. is tenant ID of octavia | |
| 22:22:19 | sean-k-mooney | that would require your current user to have the member or admin role on that project for keystone to be able to issue the token | |
| 22:22:40 | sean-k-mooney | the uuid shoudl be enough if your an admin | |
| 22:22:51 | sean-k-mooney | --all-project is only required to delete by name | |
| 22:23:06 | sean-k-mooney | on a differnt project | |
| 22:23:15 | atmark | i'm able to list VMs `openstack server list --os-project-id 848fe125a93a408ba8a8044fb87e9cdf` | |
| 22:23:27 | sean-k-mooney | ack | |
| 22:23:33 | sean-k-mooney | and i assume the delete didnt work | |
| 22:23:38 | atmark | it didn't | |
| 22:23:51 | sean-k-mooney | you could try --force | |
| 22:23:56 | sean-k-mooney | but that is not for this usecase | |
| 22:24:10 | sean-k-mooney | its for forceing the delete now if you have soft delete enabled | |
| 22:24:26 | sean-k-mooney | you could also try reset-state | |
| 22:24:28 | sean-k-mooney | then delete | |
| 22:24:41 | sean-k-mooney | so reset the state to error | |
| 22:24:44 | sean-k-mooney | then delete form error | |
| 22:25:46 | atmark | throws an no ID found on reset-state | |
| 22:26:42 | sean-k-mooney | ya so i think what is happenign is there is a record in teh api db | |
| 22:26:58 | sean-k-mooney | but there is no cell mapping for the instnace and no recored in teh cell db | |
| 22:27:18 | sean-k-mooney | reset-state woudl try and update the instance in the cell db | |
| 22:27:32 | sean-k-mooney | but that fails because it was not created | |
| 22:28:01 | atmark | this is what's in the db https://paste.openstack.org/show/bbS8V1bZK0VcFO7WBD7p/ | |
| 22:28:15 | sean-k-mooney | if you have db access your could confirm that by checkign the api db to see if the build request exists and/or instance_mapping | |
| 22:28:45 | sean-k-mooney | ok thats in which db | |
| 22:28:48 | sean-k-mooney | cell0 | |
| 22:28:51 | sean-k-mooney | or api | |
| 22:29:00 | sean-k-mooney | nova has 3 databases by default | |
| 22:29:02 | atmark | full columns https://paste.openstack.org/show/bsnbvSiQ7ieLLCW7zPyF/ | |
| 22:29:07 | atmark | nova | |
| 22:29:28 | sean-k-mooney | that does not tell me anything what installer did you use | |
| 22:29:35 | atmark | kolla-ansible | |
| 22:29:49 | sean-k-mooney | ok i have that deploy locally let me see which db that is | |
| 22:29:55 | sean-k-mooney | which version? | |
| 22:30:00 | atmark | ussuri | |
| 22:30:05 | sean-k-mooney | ack | |
| 22:30:27 | atmark | there 3 nova db, nova, nova_api and nova_cell0 | |
| 22:30:36 | atmark | the paste came from nova db | |
| 22:30:39 | sean-k-mooney | ack ok so nova i the nova cell1 db | |
| 22:31:08 | sean-k-mooney | so an instance should only end up in cell1 after it has been asigned a host | |
| 22:31:51 | sean-k-mooney | the only time after its booted that it can not have a host in cell 1 is if its shleved | |
| 22:36:14 | atmark | so it didn't end up in any cell since it doesn't have a host? | |
| 22:36:27 | atmark | host is NULL | |
| 22:36:42 | sean-k-mooney | select * from nova_api.instance_mappings where instance_uuid = "ded221d0-8410-4f89-b322-b9ff0207a0dd"; | |
| 22:36:50 | sean-k-mooney | can you run that but with your instance uuid | |
| 22:37:19 | sean-k-mooney | you should see somethign like this https://paste.openstack.org/show/bAUqQT5Al2Q9GoF9Nln2/ | |
| 22:38:15 | atmark | https://paste.openstack.org/show/b9tx6I2rQ3tUtsGBR98g/ | |
| 22:38:20 | sean-k-mooney | what im wondering is does that exist and is the cell_id set | |
| 22:38:32 | sean-k-mooney | and if so is that cell in your cell_mappings table | |
| 22:38:54 | sean-k-mooney | ok so cell_id is 6 | |
| 22:39:11 | sean-k-mooney | the cell mappings | |
| 22:39:16 | sean-k-mooney | has potitally password inf | |
| 22:39:22 | sean-k-mooney | so dont paste that | |
| 22:39:46 | atmark | 6 exist | |
| 22:39:49 | sean-k-mooney | but does the db connection string for cell id 6 end in nova | |
| 22:39:49 | atmark | https://paste.openstack.org/show/bh9PvsUNrxNuYSRsATga/ | |
| 22:40:08 | atmark | 6 and 9 | |
| 22:40:13 | sean-k-mooney | yep | |
| 22:40:20 | sean-k-mooney | does 6 point to the db where the instance iss | |
| 22:40:25 | sean-k-mooney | i.e. nova in your case | |
| 22:40:48 | sean-k-mooney | it shoudl end in :3306/nova | |
| 22:41:10 | sean-k-mooney | this is how we know which database to look in to delete the instace at the cell level | |