| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 14:13:41 | mriedem | jroll: damn | |
| 14:13:52 | jroll | it might actually be easy | |
| 14:13:53 | mriedem | jroll: will need cdent, edleafe and/or jaypipes to look at that one | |
| 14:14:02 | jroll | mriedem: thank you for pinging them :P | |
| 14:14:17 | mriedem | i was thinking the periodic update would remove the inventory for the removed node but vdrok said in the bug that it didn't | |
| 14:14:22 | cdent | joy | |
| 14:15:34 | edleafe | mriedem: I can take a look at that | |
| 14:15:57 | jroll | right, I assume we're missing something like this for inventory https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6577 | |
| 14:18:50 | mriedem | jroll: yeah that looks reasonable | |
| 14:19:02 | mriedem | jaypipes refactored a ton of that code so he'll know best | |
| 14:19:15 | jaypipes | jroll, vdrok, mriedem, edleafe, cdent: I have a suspicion on cause of that bug. | |
| 14:19:31 | edleafe | jaypipes: do tell | |
| 14:20:11 | jaypipes | jroll, vdrok, mriedem, edleafe, cdent: Ironic virt driver is probably returning False for node_is_available() and therefore when update_available_resource() runs, we never even get to the call for InstanceList.get_by_host_and_node() and therefore never clean up the now-deleted instance. | |
| 14:20:48 | mriedem | sdague: how do you feel about this? https://review.openstack.org/#/c/426926/ | |
| 14:20:49 | jaypipes | mriedem: remember when you asked me to change RT.disabled(nodename) to account for the Ironic-only node_is_available() call? :) | |
| 14:20:57 | mriedem | sdague: because i'd like to get the dependent nova change in for ocata | |
| 14:21:15 | mriedem | jaypipes: vaguely | |
| 14:21:35 | mriedem | jaypipes: i'm pretty sure i was drunk reviewing most of that refactor series :) | |
| 14:21:45 | jroll | jaypipes: this is about deleted nodes, not deleted instances, to be clear | |
| 14:22:11 | jroll | jaypipes: so it wouldn't even be returned in get_available_nodes() | |
| 14:22:18 | jaypipes | vdrok: I asked for you to do a flash-patch in the bug above. | |
| 14:22:34 | cdent | is running two nova-computes on the same machine normal and/or an important consideration in this bug? | |
| 14:23:01 | jaypipes | jroll: right, but the inventory won't get cleaned up in the placement DB if the node_is_available() returns False here: | |
| 14:23:02 | jaypipes | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L414 | |
| 14:23:02 | vdrok | jaypipes: ok, thanks, will do after lunch | |
| 14:23:12 | jaypipes | cdent: no. | |
| 14:23:26 | sdague | mriedem: yeh, that seems fine. | |
| 14:23:41 | sdague | mriedem: under what circumstances does that error out? | |
| 14:24:09 | jroll | jaypipes: yeah, it'll definitely return False there if the node is deleted | |
| 14:24:40 | mriedem | sdague: would have to dig into nova-status code for specifics, but things like not being able to talk to the placement api | |
| 14:24:49 | sdague | mriedem: yeh, seems fine | |
| 14:25:42 | mriedem | no cell mappings, no cell0, no host mappings when there are computes, etc | |
| 14:25:43 | mriedem | thanks | |
| 14:26:06 | edleafe | jroll: So if the node is deleted, the problem is just orphaned invntory, right? IOW, it isn't in the DB anymore, so it won't get returned from a placement GET | |
| 14:26:38 | jroll | edleafe: AIUI it *is* being returned from the placement API | |
| 14:27:10 | mriedem | the error in the bug report was missing some details, like what the actual build failure was | |
| 14:27:33 | jroll | mriedem: fault | {"message": "Node 6cc8803d-4e77-4948-b653-663d8d5e52b7 could not be found. (HTTP 404)", "code": 500, "details": " File \"/opt/stack/nova/....... | |
| 14:27:38 | jroll | right at the top | |
| 14:27:57 | mriedem | yeah, but | |
| 14:28:08 | mriedem | that doesn't really have anything to do with placement... | |
| 14:28:22 | mriedem | it's in _build_and_run_instance | |
| 14:28:41 | jroll | well | |
| 14:28:47 | jroll | it selected a node that doesn't exist in ironic | |
| 14:29:23 | jroll | it also doesn't exist in the compute_nodes table, but does exist in inventory | |
| 14:29:37 | mriedem | unless it's blowing up here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1905 | |
| 14:30:26 | edleafe | jroll: so it's gone from the compute_nodes table in the cell, but not in the resource_providers table in the API/placement db | |
| 14:30:39 | edleafe | jroll: So placement still thinks it's there | |
| 14:30:40 | jroll | edleafe: AIUI, yes | |
| 14:30:44 | jroll | keep in mind I did not file this bug | |
| 14:30:50 | edleafe | jroll: ack | |
| 14:30:53 | jroll | I have not independently reproduced it | |
| 14:31:01 | jroll | I'm just trying to answer question while vdrok is at lunch :) | |
| 14:31:27 | vdrok | jaypipes: so, I tried that, return (nodename not in self.compute_nodes) only in the disabled, restarted both computes, still the same | |
| 14:31:57 | jroll | mriedem: nah, I think all the placement/allocations stuff is working fine, except that the node being selected doesn't exist, so spawn() explodes | |
| 14:32:49 | mriedem | what is selecting the node? the scheduler? | |
| 14:33:31 | jroll | I assume so | |
| 14:33:36 | jroll | that's what selects things, right | |
| 14:33:50 | vdrok | mriedem: I added a bit more log from the n-cpu | |
| 14:33:50 | edleafe | jroll: jaypipes: so maybe we need to also add a call to the placement API after https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L324 to delete the compute node as a resource provider? | |
| 14:33:52 | mriedem | i guess if everything is fully master then the filter scheduler will use placement | |
| 14:34:40 | jroll | yeah, looks recent enough for placement api, that's why I bring it up :) | |
| 14:35:11 | jroll | edleafe: that seems sensible to my feeble brain | |
| 14:35:14 | mriedem | edleafe: then the compute node object has to use the scheduler client... | |
| 14:35:21 | vdrok | now going to lunch for real :) | |
| 14:35:34 | mriedem | i'd rather we just call the scheduler client to delete the RP after deleting the CN here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6585 | |
| 14:35:44 | mriedem | since the compute manager already has it's grubby hands into that code | |
| 14:36:58 | edleafe | mriedem: yeah, makes sense | |
| 14:40:32 | mriedem | as far as i can tell, we only ever delete inventories or allocations, but never the actual resource provider | |
| 14:40:53 | mriedem | and since the CN is the RP, and we delete the CN, i guess we need to also delete the RP | |
| 14:41:47 | mriedem | i'm going to head into the office, bbiab | |
| 14:41:52 | cdent | yes | |
| 14:42:07 | cdent | (the delete the rp) | |
| 14:56:13 | mdbooth | lyarwood: I've reviewed the swap_volume change. Haven't looked at the tests yet, btw. Doing that now. | |
| 14:59:35 | lyarwood | mdbooth: thanks again, working on it now. | |
| 15:07:27 | mriedem | please all be fixed, please all be fixed... | |
| 15:07:29 | mriedem | no whammies | |
| 15:07:57 | lyarwood | mriedem: my fix is failing in check, looking now | |
| 15:10:59 | lyarwood | mriedem: http://logs.openstack.org/67/427267/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0f048f1/console.html#_2017-02-02_15_03_33_671655 - sorry if I've missed a previous discussion about this still being broken btw | |
| 15:12:37 | mriedem | lyarwood: another fix for that is in the gate | |
| 15:13:03 | lyarwood | mriedem: ah great, sorry I missed that fly past | |
| 15:13:35 | mriedem | lyarwood: this one https://review.openstack.org/#/c/427775/ | |
| 15:13:49 | mriedem | 2 in the gate | |
| 15:17:34 | mriedem | do we have a definition of barber-pole sorting anywhere? | |
| 15:18:49 | mriedem | just means it wraps around from cell to cell right? | |
| 15:19:05 | mriedem | "striped across the cell boundaries" | |
| 15:19:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix typo in cells v2 ocata reno https://review.openstack.org/428198 | |
| 15:22:04 | jaypipes | edleafe: when (if ever) do compute_nodes table records get deleted? | |
| 15:24:12 | jaypipes | mriedem: more generally, I wonder when we ever delete compute nodes... | |
| 15:24:48 | mriedem | in the case of ironic, i guess when they stop reporting available nodes | |
| 15:25:21 | mriedem | jaypipes: also when the nova-compute service is deleted via the api i think | |
| 15:25:25 | mriedem | will have to look | |
| 15:25:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve assertJsonEqual error reporting https://review.openstack.org/428199 | |
| 15:25:40 | mriedem | i remember fixinga 404 in the api around that recently | |
| 15:25:53 | jaypipes | mriedem: that would be a big mistake in the case of Ironic then... (since service != compute_node for Ironic) | |
| 15:26:15 | mriedem | https://review.openstack.org/#/c/406627/ was the 404 i was thinking of | |
| 15:26:25 | mriedem | "Compute node resources must currently be deleted manually in the database" | |
| 15:27:45 | edleafe | jaypipes: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6585 | |
| 15:27:47 | mriedem | yeah | |
| 15:27:47 | mriedem | but we do have this https://github.com/openstack/nova/blob/13.1.2/nova/compute/manager.py#L6513 | |
| 15:27:51 | edleafe | jaypipes: which in turn calls https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L324 | |