| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 20:19:19 | dansmith | if the mapping is set, then BR is ignored | |
| 20:19:27 | dansmith | so we can delete it late | |
| 20:19:50 | dansmith | and that's what we're doing | |
| 20:19:57 | mriedem | schedule_and_build_instances in the conductor does set the cell in the instance mapping and then attempt to delete the build request, | |
| 20:20:04 | mriedem | but the build request can be deleted by the api first | |
| 20:20:05 | dansmith | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155-L1182 | |
| 20:20:09 | dansmith | that's for cell0 | |
| 20:20:20 | dansmith | which presumably is what his situation is if it never scheduled | |
| 20:20:53 | mriedem | not sure why we'd pass here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1178 | |
| 20:21:13 | dansmith | if it got deleted | |
| 20:21:24 | mriedem | the instance mapping? | |
| 20:21:28 | dansmith | on the non-cell0 case we also target the mapping before deleting the BR | |
| 20:21:29 | mriedem | why would that get deleted during scheduling | |
| 20:21:31 | mnaser | is it possible this happened when a vm was quickly deleted while it was scheduled | |
| 20:21:37 | dansmith | mriedem: if it got deleted by the user I mean | |
| 20:21:44 | mriedem | the instance mapping isn't deleted when the instance is deleted | |
| 20:21:47 | dansmith | mriedem: and we did a local delete because it wasn't scheudled | |
| 20:21:51 | dansmith | on local delete? | |
| 20:21:52 | mriedem | no | |
| 20:21:55 | mriedem | only on archive | |
| 20:22:15 | mnaser | let me check the actual cell db | |
| 20:22:33 | dansmith | mriedem: I think we delete them somewhere else | |
| 20:22:36 | dansmith | multi-create maube? | |
| 20:22:36 | mnaser | so this went to cell0 | |
| 20:22:47 | mnaser | and it's state in cell0 is 'scheduling' | |
| 20:23:02 | mriedem | dansmith: i'm pretty sure not, because that's what bauzas had originally written and we were against it in case the actual delete in the compute failed | |
| 20:23:06 | mnaser | deleted=0 in cell0 | |
| 20:23:10 | mnaser | so it's not even deleted | |
| 20:23:11 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1271 | |
| 20:23:22 | mnaser | maybe this is something in the bury in cell0 code | |
| 20:23:33 | mriedem | dansmith: ok so before we even get to conductor | |
| 20:23:54 | dansmith | yeah fair | |
| 20:23:56 | mriedem | if the instance mapping is gone and the build request is gone, shouldn't be able to list it | |
| 20:24:05 | dansmith | we will though | |
| 20:24:22 | dansmith | becase we'll just hit each db and list anything we find, mappings be damned | |
| 20:24:47 | mriedem | but if we deleted the build request and instance mapping in ^ | |
| 20:24:55 | mriedem | we don't get to conductor to schedule | |
| 20:25:02 | mriedem | so we don't create the instance in any cell | |
| 20:25:26 | dansmith | yeah, in the delete_build_artifacts case yep, I meant if we end up with an instance in a cell with no mapping, | |
| 20:25:29 | dansmith | we'll still list it | |
| 20:25:43 | mriedem | sure, i just don't know how that could happen | |
| 20:26:04 | dansmith | well, I'm not sure how we could end up with a deleted BR and untargeted instance mapping | |
| 20:30:01 | mriedem | so we could delete a build request here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1848 | |
| 20:30:04 | mriedem | while scheduling | |
| 20:30:24 | mnaser | this was deleted via the api (while scheduling) | |
| 20:30:29 | mnaser | it def didnt fail to schedule | |
| 20:31:20 | mriedem | assuming we deleted the build request in the api, the api would check to see if the instance mapping is pointing at a cell yet https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1760 | |
| 20:31:46 | dansmith | if we do that, | |
| 20:31:47 | dansmith | we should also delete the instance in the cell db if we race with the conversion from BR to instance | |
| 20:31:52 | dansmith | but maybe we're slipping through those cracks | |
| 20:32:05 | dansmith | here: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1184 | |
| 20:32:53 | dansmith | not really sure how that could be wrong tbh | |
| 20:33:12 | dansmith | either we fail to delete it "during booting" and then fall through to regular delete, | |
| 20:33:23 | mriedem | mnaser: so you have any old nova-osapi_compute services? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1803 | |
| 20:33:23 | dansmith | or we succeed, conductor tries/fails to delete the BR, and destroys the instance | |
| 20:33:30 | mriedem | *do you have | |
| 20:33:36 | dansmith | or old service records | |
| 20:33:42 | mnaser | let me check the db rather than service-list | |
| 20:33:55 | mriedem | service list doesn't show api servies | |
| 20:33:58 | mriedem | so yeah you have to check the db | |
| 20:34:22 | mriedem | you could have some api services with version 0 from running with wsgi before we fixed that? | |
| 20:34:57 | mnaser | there are some stale records but i see 16/22/30 | |
| 20:35:05 | mnaser | can i safely delete those older ones in the db | |
| 20:35:11 | mnaser | (no 0's though) | |
| 20:35:33 | mnaser | haha | |
| 20:35:41 | mriedem | it's checking for version < 15 so those shouldn't be a problem for this issue | |
| 20:35:43 | mnaser | truncate table services; | |
| 20:35:46 | mnaser | "it'll autopopulate" | |
| 20:36:45 | mriedem | dansmith: umm, couldn't we have an instance mapping with no cell mapping if we hit this? https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1244 | |
| 20:37:03 | mriedem | api deletes the build request, we catch it and just continue | |
| 20:37:33 | dansmith | instance mapping with no cell is not a problem if the instance is deleted from the cell | |
| 20:37:48 | mriedem | in ^ the instance wouldn't be created in a cell | |
| 20:38:04 | mnaser | this instance was created in cell0 though (i checked via the db) | |
| 20:38:12 | dansmith | mriedem: then it wouldn't fit his problem | |
| 20:38:59 | mnaser | is _bury_in_cell0 the only codebase that puts things into cell0? | |
| 20:39:04 | dansmith | yeah | |
| 20:39:30 | mnaser | so we know it got at least to here - https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155 | |
| 20:39:34 | mnaser | let me check if it created bdms | |
| 20:40:08 | dansmith | should have, | |
| 20:40:11 | dansmith | nothing to stop it there I think | |
| 20:40:35 | dansmith | it must not have hit 1183 because if so, the instance would be deleted | |
| 20:40:53 | mnaser | are bdms created when one is given? | |
| 20:41:18 | mnaser | cause: select * from block_device_mapping where instance_uuid='f5ce0c5f-0484-496f-a497-6d1a1fb370d3'; => nada | |
| 20:41:41 | dansmith | when one is _not_ given? | |
| 20:41:47 | dansmith | I think we can have no BDMs here | |
| 20:42:01 | mnaser | yes sorry, when one is not given | |
| 20:42:23 | mriedem | when you're not booting from volume, | |
| 20:42:24 | mnaser | hmm | |
| 20:42:30 | mriedem | i want to say we create a bdm record for the local image | |
| 20:42:33 | mnaser | the status should be ERROR but the status in cell0 is not error | |
| 20:43:07 | mnaser | state in cell0 is building/scheduling | |
| 20:43:12 | mnaser | which it should not be? | |
| 20:43:27 | mriedem | _set_vm_state_and_notify probably failed then | |
| 20:43:29 | dansmith | I'm not sure where/when it goes to error | |
| 20:43:36 | mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1166 | |
| 20:43:42 | mriedem | updates = {'vm_state': vm_states.ERROR, 'task_state': None} | |
| 20:43:43 | dansmith | maybe it failed on a task_state conflict? | |
| 20:43:59 | mriedem | scheduler_utils.set_vm_state_and_notify does a few things... | |
| 20:44:15 | mnaser | this should all be in the conductor right? | |
| 20:44:20 | dansmith | yeah | |