| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 20:17:30 | dansmith | this patch doesn't address his problem, | |
| 20:17:32 | mnaser | rabbitmq had a field day a few days ago for us | |
| 20:17:38 | dansmith | but papers over it for list I guess | |
| 20:17:41 | mnaser | so i would totally attribute it to anything related to that | |
| 20:17:54 | mriedem | i believe jroll was seeing similar | |
| 20:18:00 | dansmith | mnaser: mm, shouldn't really as we don't use rabbit to create the instance in the cell db | |
| 20:18:03 | dansmith | well, | |
| 20:18:05 | mnaser | oh true | |
| 20:18:19 | dansmith | unless we dropped something between superconductors or something | |
| 20:18:25 | mnaser | no superconductor here | |
| 20:18:29 | mnaser | 1 cell, 1 conductor | |
| 20:18:37 | dansmith | s/super// | |
| 20:18:42 | mriedem | last i looked at this, we do have a window between the time we delete the build request and set the instance mapping cell | |
| 20:19:05 | dansmith | mriedem: we should be setting the mapping and then deleting the BR | |
| 20:19:07 | mnaser | wouldn't it be safer to set the instance mapping first and then delete the br? | |
| 20:19:09 | mnaser | yeah ^ | |
| 20:19:12 | dansmith | because that's how the get logic expects it to work | |
| 20:19:19 | dansmith | if the mapping is set, then BR is ignored | |
| 20:19:27 | dansmith | so we can delete it late | |
| 20:19:50 | dansmith | and that's what we're doing | |
| 20:19:57 | mriedem | schedule_and_build_instances in the conductor does set the cell in the instance mapping and then attempt to delete the build request, | |
| 20:20:04 | mriedem | but the build request can be deleted by the api first | |
| 20:20:05 | dansmith | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155-L1182 | |
| 20:20:09 | dansmith | that's for cell0 | |
| 20:20:20 | dansmith | which presumably is what his situation is if it never scheduled | |
| 20:20:53 | mriedem | not sure why we'd pass here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1178 | |
| 20:21:13 | dansmith | if it got deleted | |
| 20:21:24 | mriedem | the instance mapping? | |
| 20:21:28 | dansmith | on the non-cell0 case we also target the mapping before deleting the BR | |
| 20:21:29 | mriedem | why would that get deleted during scheduling | |
| 20:21:31 | mnaser | is it possible this happened when a vm was quickly deleted while it was scheduled | |
| 20:21:37 | dansmith | mriedem: if it got deleted by the user I mean | |
| 20:21:44 | mriedem | the instance mapping isn't deleted when the instance is deleted | |
| 20:21:47 | dansmith | mriedem: and we did a local delete because it wasn't scheudled | |
| 20:21:51 | dansmith | on local delete? | |
| 20:21:52 | mriedem | no | |
| 20:21:55 | mriedem | only on archive | |
| 20:22:15 | mnaser | let me check the actual cell db | |
| 20:22:33 | dansmith | mriedem: I think we delete them somewhere else | |
| 20:22:36 | dansmith | multi-create maube? | |
| 20:22:36 | mnaser | so this went to cell0 | |
| 20:22:47 | mnaser | and it's state in cell0 is 'scheduling' | |
| 20:23:02 | mriedem | dansmith: i'm pretty sure not, because that's what bauzas had originally written and we were against it in case the actual delete in the compute failed | |
| 20:23:06 | mnaser | deleted=0 in cell0 | |
| 20:23:10 | mnaser | so it's not even deleted | |
| 20:23:11 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1271 | |
| 20:23:22 | mnaser | maybe this is something in the bury in cell0 code | |
| 20:23:33 | mriedem | dansmith: ok so before we even get to conductor | |
| 20:23:54 | dansmith | yeah fair | |
| 20:23:56 | mriedem | if the instance mapping is gone and the build request is gone, shouldn't be able to list it | |
| 20:24:05 | dansmith | we will though | |
| 20:24:22 | dansmith | becase we'll just hit each db and list anything we find, mappings be damned | |
| 20:24:47 | mriedem | but if we deleted the build request and instance mapping in ^ | |
| 20:24:55 | mriedem | we don't get to conductor to schedule | |
| 20:25:02 | mriedem | so we don't create the instance in any cell | |
| 20:25:26 | dansmith | yeah, in the delete_build_artifacts case yep, I meant if we end up with an instance in a cell with no mapping, | |
| 20:25:29 | dansmith | we'll still list it | |
| 20:25:43 | mriedem | sure, i just don't know how that could happen | |
| 20:26:04 | dansmith | well, I'm not sure how we could end up with a deleted BR and untargeted instance mapping | |
| 20:30:01 | mriedem | so we could delete a build request here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1848 | |
| 20:30:04 | mriedem | while scheduling | |
| 20:30:24 | mnaser | this was deleted via the api (while scheduling) | |
| 20:30:29 | mnaser | it def didnt fail to schedule | |
| 20:31:20 | mriedem | assuming we deleted the build request in the api, the api would check to see if the instance mapping is pointing at a cell yet https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1760 | |
| 20:31:46 | dansmith | if we do that, | |
| 20:31:47 | dansmith | we should also delete the instance in the cell db if we race with the conversion from BR to instance | |
| 20:31:52 | dansmith | but maybe we're slipping through those cracks | |
| 20:32:05 | dansmith | here: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1184 | |
| 20:32:53 | dansmith | not really sure how that could be wrong tbh | |
| 20:33:12 | dansmith | either we fail to delete it "during booting" and then fall through to regular delete, | |
| 20:33:23 | mriedem | mnaser: so you have any old nova-osapi_compute services? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1803 | |
| 20:33:23 | dansmith | or we succeed, conductor tries/fails to delete the BR, and destroys the instance | |
| 20:33:30 | mriedem | *do you have | |
| 20:33:36 | dansmith | or old service records | |
| 20:33:42 | mnaser | let me check the db rather than service-list | |
| 20:33:55 | mriedem | service list doesn't show api servies | |
| 20:33:58 | mriedem | so yeah you have to check the db | |
| 20:34:22 | mriedem | you could have some api services with version 0 from running with wsgi before we fixed that? | |
| 20:34:57 | mnaser | there are some stale records but i see 16/22/30 | |
| 20:35:05 | mnaser | can i safely delete those older ones in the db | |
| 20:35:11 | mnaser | (no 0's though) | |
| 20:35:33 | mnaser | haha | |
| 20:35:41 | mriedem | it's checking for version < 15 so those shouldn't be a problem for this issue | |
| 20:35:43 | mnaser | truncate table services; | |
| 20:35:46 | mnaser | "it'll autopopulate" | |
| 20:36:45 | mriedem | dansmith: umm, couldn't we have an instance mapping with no cell mapping if we hit this? https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1244 | |
| 20:37:03 | mriedem | api deletes the build request, we catch it and just continue | |
| 20:37:33 | dansmith | instance mapping with no cell is not a problem if the instance is deleted from the cell | |
| 20:37:48 | mriedem | in ^ the instance wouldn't be created in a cell | |
| 20:38:04 | mnaser | this instance was created in cell0 though (i checked via the db) | |
| 20:38:12 | dansmith | mriedem: then it wouldn't fit his problem | |
| 20:38:59 | mnaser | is _bury_in_cell0 the only codebase that puts things into cell0? | |
| 20:39:04 | dansmith | yeah | |
| 20:39:30 | mnaser | so we know it got at least to here - https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155 | |
| 20:39:34 | mnaser | let me check if it created bdms | |
| 20:40:08 | dansmith | should have, | |
| 20:40:11 | dansmith | nothing to stop it there I think | |
| 20:40:35 | dansmith | it must not have hit 1183 because if so, the instance would be deleted | |
| 20:40:53 | mnaser | are bdms created when one is given? | |
| 20:41:18 | mnaser | cause: select * from block_device_mapping where instance_uuid='f5ce0c5f-0484-496f-a497-6d1a1fb370d3'; => nada | |