| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 20:10:56 | mnaser | or even if the db is mucked it shouldn't be like this? | |
| 20:11:18 | dansmith | well, depends on the level of mucking I guess, but.. it should never be like this | |
| 20:11:37 | mnaser | so as i understand it, list just grabs stuff from the 'api' db, but the 'get' actually hits the cell | |
| 20:11:43 | dansmith | no | |
| 20:11:58 | dansmith | list grabs stuff from the api and cell databases together | |
| 20:12:11 | mnaser | so it must know the mapping too i guess | |
| 20:12:15 | dansmith | get will pull the mapping from the api db, and if it looks like it is scheduled, will talk to the cell, and if not will return the build request data | |
| 20:12:21 | mnaser | ah okay | |
| 20:12:38 | mnaser | so it's in a state where it's doing neither talk to cell or return build request data | |
| 20:12:50 | dansmith | if it' | |
| 20:12:57 | dansmith | if it is 404'ing a scheduling instance, | |
| 20:13:11 | dansmith | then it's getting the mapping and then not returning the buildrequest or something | |
| 20:14:01 | dansmith | mnaser: all this logic is fairly straightforward here: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2298-L2357 | |
| 20:14:09 | dansmith | start with L2328 | |
| 20:14:26 | dansmith | we get the mapping (which should always exist once we've returned a uuid to the user) | |
| 20:14:36 | dansmith | maybe check to see if that failed to get written? | |
| 20:14:41 | mnaser | let me check the db mapping | |
| 20:14:56 | dansmith | if there's no mapping but it did get created in a cell, that would cause it to show up in list and not get | |
| 20:15:04 | dansmith | because of L2355 there | |
| 20:15:37 | mnaser | cell_id=NULL in the db | |
| 20:15:51 | dansmith | that's what it should be for scheduling | |
| 20:15:54 | mnaser | so it should def go into the elif | |
| 20:16:08 | dansmith | that triggers L2333 to get it from the api db's buildrequest | |
| 20:16:14 | mnaser | let me check the build request | |
| 20:16:35 | mnaser | no build_request | |
| 20:16:42 | mriedem | i have a patch for this | |
| 20:16:52 | mriedem | but that's listing, not showing | |
| 20:16:52 | mriedem | https://review.openstack.org/#/c/575556/ ? | |
| 20:16:55 | mriedem | sounds similar though | |
| 20:17:03 | mriedem | build request is gone, instance mapping isn't pointing at a cell | |
| 20:17:05 | dansmith | and it's marker | |
| 20:17:16 | dansmith | how does that situation happen? | |
| 20:17:24 | mriedem | do not know | |
| 20:17:27 | mnaser | i mean | |
| 20:17:30 | dansmith | this patch doesn't address his problem, | |
| 20:17:32 | mnaser | rabbitmq had a field day a few days ago for us | |
| 20:17:38 | dansmith | but papers over it for list I guess | |
| 20:17:41 | mnaser | so i would totally attribute it to anything related to that | |
| 20:17:54 | mriedem | i believe jroll was seeing similar | |
| 20:18:00 | dansmith | mnaser: mm, shouldn't really as we don't use rabbit to create the instance in the cell db | |
| 20:18:03 | dansmith | well, | |
| 20:18:05 | mnaser | oh true | |
| 20:18:19 | dansmith | unless we dropped something between superconductors or something | |
| 20:18:25 | mnaser | no superconductor here | |
| 20:18:29 | mnaser | 1 cell, 1 conductor | |
| 20:18:37 | dansmith | s/super// | |
| 20:18:42 | mriedem | last i looked at this, we do have a window between the time we delete the build request and set the instance mapping cell | |
| 20:19:05 | dansmith | mriedem: we should be setting the mapping and then deleting the BR | |
| 20:19:07 | mnaser | wouldn't it be safer to set the instance mapping first and then delete the br? | |
| 20:19:09 | mnaser | yeah ^ | |
| 20:19:12 | dansmith | because that's how the get logic expects it to work | |
| 20:19:19 | dansmith | if the mapping is set, then BR is ignored | |
| 20:19:27 | dansmith | so we can delete it late | |
| 20:19:50 | dansmith | and that's what we're doing | |
| 20:19:57 | mriedem | schedule_and_build_instances in the conductor does set the cell in the instance mapping and then attempt to delete the build request, | |
| 20:20:04 | mriedem | but the build request can be deleted by the api first | |
| 20:20:05 | dansmith | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155-L1182 | |
| 20:20:09 | dansmith | that's for cell0 | |
| 20:20:20 | dansmith | which presumably is what his situation is if it never scheduled | |
| 20:20:53 | mriedem | not sure why we'd pass here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1178 | |
| 20:21:13 | dansmith | if it got deleted | |
| 20:21:24 | mriedem | the instance mapping? | |
| 20:21:28 | dansmith | on the non-cell0 case we also target the mapping before deleting the BR | |
| 20:21:29 | mriedem | why would that get deleted during scheduling | |
| 20:21:31 | mnaser | is it possible this happened when a vm was quickly deleted while it was scheduled | |
| 20:21:37 | dansmith | mriedem: if it got deleted by the user I mean | |
| 20:21:44 | mriedem | the instance mapping isn't deleted when the instance is deleted | |
| 20:21:47 | dansmith | mriedem: and we did a local delete because it wasn't scheudled | |
| 20:21:51 | dansmith | on local delete? | |
| 20:21:52 | mriedem | no | |
| 20:21:55 | mriedem | only on archive | |
| 20:22:15 | mnaser | let me check the actual cell db | |
| 20:22:33 | dansmith | mriedem: I think we delete them somewhere else | |
| 20:22:36 | mnaser | so this went to cell0 | |
| 20:22:36 | dansmith | multi-create maube? | |
| 20:22:47 | mnaser | and it's state in cell0 is 'scheduling' | |
| 20:23:02 | mriedem | dansmith: i'm pretty sure not, because that's what bauzas had originally written and we were against it in case the actual delete in the compute failed | |
| 20:23:06 | mnaser | deleted=0 in cell0 | |
| 20:23:10 | mnaser | so it's not even deleted | |
| 20:23:11 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1271 | |
| 20:23:22 | mnaser | maybe this is something in the bury in cell0 code | |
| 20:23:33 | mriedem | dansmith: ok so before we even get to conductor | |
| 20:23:54 | dansmith | yeah fair | |
| 20:23:56 | mriedem | if the instance mapping is gone and the build request is gone, shouldn't be able to list it | |
| 20:24:05 | dansmith | we will though | |
| 20:24:22 | dansmith | becase we'll just hit each db and list anything we find, mappings be damned | |
| 20:24:47 | mriedem | but if we deleted the build request and instance mapping in ^ | |
| 20:24:55 | mriedem | we don't get to conductor to schedule | |
| 20:25:02 | mriedem | so we don't create the instance in any cell | |
| 20:25:26 | dansmith | yeah, in the delete_build_artifacts case yep, I meant if we end up with an instance in a cell with no mapping, | |
| 20:25:29 | dansmith | we'll still list it | |
| 20:25:43 | mriedem | sure, i just don't know how that could happen | |
| 20:26:04 | dansmith | well, I'm not sure how we could end up with a deleted BR and untargeted instance mapping | |
| 20:30:01 | mriedem | so we could delete a build request here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1848 | |
| 20:30:04 | mriedem | while scheduling | |
| 20:30:24 | mnaser | this was deleted via the api (while scheduling) | |
| 20:30:29 | mnaser | it def didnt fail to schedule | |
| 20:31:20 | mriedem | assuming we deleted the build request in the api, the api would check to see if the instance mapping is pointing at a cell yet https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1760 | |
| 20:31:46 | dansmith | if we do that, | |
| 20:31:47 | dansmith | we should also delete the instance in the cell db if we race with the conversion from BR to instance | |