| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-27 | |||
| 20:55:30 | mriedem | mnaser: maybe - if the build request exists, the instance shouldn't be in a cell | |
| 20:55:49 | mriedem | so L42 in your script is where i'd look for a build request | |
| 20:55:51 | mriedem | as a sanity check | |
| 20:56:24 | mnaser | mriedem: yeah i was planning to just run the mysql till a certain point and assume the rest was just unscheduled stuff but it could be confusing to hand off to others | |
| 20:57:10 | mnaser | i'm feeling to check if a build request exists at L27 so a) i dont hit the cells and b) if a build requests exists, technically there shouldn't be an issue because api calls will interact with that build request | |
| 20:57:24 | mnaser | i think the problem is there when a build request AND cell mapping is missing | |
| 20:57:44 | mnaser | but i believe if build request is there but cell mapping is missing, it'll work just fine and not do any weird 404s on instances | |
| 20:57:49 | mriedem | correct | |
| 20:58:07 | mriedem | this was the case i was worried about last week https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/conductor/manager.py#L1243 | |
| 20:58:23 | mriedem | in that case, the api has deleted the build request, and we haven't updated the instance mapping | |
| 20:58:56 | mriedem | but, we wouldn't put the instance in cell0 b/c the user deleted the instance before we created it (via build request) | |
| 20:59:04 | mriedem | mnaser: might be nice info to know if these unmapped instances are deleted | |
| 20:59:15 | melwitt | one thing that's interesting that I learned recently is that if, for some reason, there is a case where a build request exists but *no* instance mapping exists, the API does not handle it in that, the "instance" will show up in a 'nova list' but it can't be deleted because delete will raise NotFound | |
| 20:59:46 | mriedem | i don't know how that could happen | |
| 20:59:56 | mriedem | we create the build request and the instance mapping in _provision_instances | |
| 21:00:00 | mriedem | *and request spec | |
| 21:00:18 | melwitt | and via code inspection, I don't know how that state could be gotten into other than nova-api restarting at precisely the moment after the build request is created but before the instance mapping was | |
| 21:00:34 | mriedem | https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/api.py#L930 and then https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/api.py#L942 | |
| 21:00:58 | mriedem | or the db failing the instance mapping insert | |
| 21:00:58 | mnaser | melwitt: yeah that's essentially the state that these vms are in | |
| 21:01:17 | melwitt | mnaser: I thought you had instance mappings though, right? | |
| 21:01:20 | melwitt | yeah, or that | |
| 21:01:30 | mnaser | melwitt: instance_mapping is there sure, but cell_id=NONE | |
| 21:01:43 | melwitt | yeah, that's different than what I said. your case will let a delete work | |
| 21:01:43 | mnaser | so some of those are list-able, but not delete-able | |
| 21:01:52 | mriedem | mnaser: are you listing as admin? | |
| 21:01:53 | melwitt | oh really? | |
| 21:01:55 | mriedem | to list out deleted instances? | |
| 21:02:08 | mriedem | i have to think you're hitting this https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/conductor/manager.py#L1243 | |
| 21:02:08 | mnaser | nope, i had a user complain they could list an instance but could not delete it | |
| 21:02:10 | mnaser | hell i cant even delete it | |
| 21:02:22 | mnaser | let me dig th eticket | |
| 21:02:22 | melwitt | hm, okay, that is a new case I didn't know | |
| 21:03:08 | melwitt | I guess what it must do is, get the instance mapping, see cell_id=None and then think "I can't lookup the instance, therefore I can't delete it" | |
| 21:03:19 | mriedem | well, | |
| 21:03:31 | mnaser | ok so confirmed here | |
| 21:03:35 | mriedem | it will fallback to trying to lookup the instance from the locally configured (in the api) [database]/connection | |
| 21:03:39 | mnaser | nova list --all-tenants | grep 1812c2eb-cfbc-4659-9817-4694ad3d2c37 < returns the instance with ERROR/NOSTATE | |
| 21:03:50 | mnaser | nova show 1812c2eb-cfbc-4659-9817-4694ad3d2c37 => ERROR (CommandError): No server with a name or ID of '1812c2eb-cfbc-4659-9817-4694ad3d2c37' exists. | |
| 21:03:57 | mriedem | mnaser: is that instance deleted? | |
| 21:04:03 | mriedem | instances.deleted != 0 | |
| 21:04:16 | mnaser | let me double check | |
| 21:04:36 | mnaser | fwiw though cell_id=NULL | |
| 21:04:37 | mnaser | checking instances | |
| 21:05:06 | mnaser | deleted=0 but this one is in cell0 | |
| 21:05:09 | mriedem | melwitt: this is what i'm thinking of https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/api.py#L1768 | |
| 21:05:27 | mriedem | mnaser: hmm, ok so the instance was created in cell0 but the instance mapping update failed | |
| 21:05:35 | mnaser | in this case yes | |
| 21:05:38 | melwitt | that's not what runs for a delete though | |
| 21:05:49 | mriedem | melwitt: it has to lookup the instance right? | |
| 21:05:57 | mnaser | yeah i cant even look it up, it just 404s | |
| 21:05:57 | mriedem | _lookup_instance is called via API.get() | |
| 21:05:58 | melwitt | yeah but it goes here https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/api.py#L2333 | |
| 21:06:28 | mnaser | let me check | |
| 21:06:32 | mnaser | it probably doesnt have a build request | |
| 21:07:01 | mnaser | no build request indeed | |
| 21:07:25 | mnaser | https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/api.py#L2353 | |
| 21:07:27 | mnaser | so ending up here afaik | |
| 21:08:28 | mriedem | how are we listing it then... | |
| 21:09:02 | mnaser | maybe list just hits the cells and ignores api stuff? | |
| 21:09:31 | mnaser | i can help if i knew where the list code is :p | |
| 21:09:46 | mriedem | https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/instance_list.py#L98 | |
| 21:10:25 | melwitt | _lookup_instance is called via API().delete, _get_instance is called via API().get | |
| 21:10:48 | melwitt | and the API (nova/api/openstack/compute/servers.py) does a API().get first before doing anything with an instance | |
| 21:10:52 | mriedem | mnaser: you're right, we'll just iterate the cells | |
| 21:11:48 | mnaser | i guess in an ideal world you retrieve list of vms from nova_api, and then generate a subsequent list to each cell with a list of instance uuids to request | |
| 21:12:08 | mnaser | which might even eliminate extra calls if a user is located in one cell | |
| 21:12:40 | melwitt | so in the case of a build request with a instance mapping with cell_mapping = None, it will return build_request.instance, which I'm not sure what will happen if you try to delete that | |
| 21:12:54 | mriedem | mnaser: that's what this is for https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/instance_list.py#L101 | |
| 21:12:56 | melwitt | presumably it fails | |
| 21:12:57 | mriedem | and that's what cern uses | |
| 21:13:39 | mnaser | wouldn't it be safer to only delete the build request once the cell has been set? | |
| 21:13:54 | melwitt | so that means build_request.instance gets passed to compute API().delete | |
| 21:14:54 | mriedem | melwitt: in that case we should go through here https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/api.py#L1877 | |
| 21:15:39 | mriedem | mnaser: the idea is if the user deletes the build request before the instance has been scheduled to a cell, we never create the instance in the cell, | |
| 21:15:47 | mriedem | so there is nothing to do with the instance mapping b/c it's not in a cell | |
| 21:16:00 | mriedem | and shouldn't get listed either b/c it's (1) not a build request and (2) not in a cell | |
| 21:16:28 | mnaser | yeah so maybe the issue here really inside list? | |
| 21:16:40 | melwitt | right, so the delete of the build request would succeed, but then the lookup of the instance will fail because it was just a build_request.instance shell | |
| 21:16:41 | mriedem | which if that is really working, we get here in conductor after the build request was deleted in api https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/conductor/manager.py#L1243 | |
| 21:17:39 | melwitt | or well, maybe not. _lookup_instance would return None, None in the cell_mapping = None case | |
| 21:17:45 | mriedem | i wonder why we don't update the instance mapping right after this https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/conductor/manager.py#L1257 | |
| 21:18:57 | mriedem | melwitt: right, if _delete_while_booting returns True, we exit https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/api.py#L1877 | |
| 21:19:10 | melwitt | hm, so I'm not seeing how delete would fail in that case | |
| 21:22:28 | melwitt | mnaser: is there any chance the service version in one of the records in the 'services' tables is < 15? | |
| 21:22:49 | mriedem | heh, i asked that last week too :) | |
| 21:22:50 | melwitt | I guess that wouldn't make sense. all of your instance GET would fail in that case | |
| 21:22:50 | mnaser | melwitt: i checked that with mriedem last time we tried to look into this and no, none | |
| 21:22:56 | mriedem | btw, i thin kwe should probably remove that service version check now | |
| 21:23:12 | openstack | Launchpad bug 1784074 in OpenStack Compute (nova) "Instances end up with no cell assigned in instance_mappings" [Undecided,New] | |
| 21:23:12 | mriedem | commented on the bug https://bugs.launchpad.net/nova/+bug/1784074/comments/1 | |
| 21:23:19 | mriedem | with what *might* be happening | |
| 21:23:22 | mriedem | but you'd have errors in the logs | |
| 21:23:25 | melwitt | this doesn't make any sense how delete returns 404 | |
| 21:23:50 | mriedem | melwitt: read ^ that comment in the bug because i think that could explain a window where it could happen | |
| 21:24:25 | mriedem | mnaser: i wonder if these are instances getting created as part of a multi-create request where they all get created in a cell, then when we go to update mappings, something fails and then the rest are left unmapped | |
| 21:24:46 | mriedem | the user attempts to delete the instance, they delete the build request, but then they can still list it, | |
| 21:24:55 | mriedem | but can't delete it b/c the build request is gone and the instance mapping isn't poining at a cell | |
| 21:24:58 | mriedem | hence your fix up script | |
| 21:25:03 | melwitt | ohhh | |