| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 19:14:47 | melwitt | yeah | |
| 19:14:49 | mriedem | but, we should see it in the logs somewhere, | |
| 19:14:49 | dansmith | melwitt: I know, but they've done at least one get_all by that point right? | |
| 19:14:53 | mriedem | because tempest will list instances at some point | |
| 19:14:56 | dansmith | right | |
| 19:15:09 | melwitt | hm | |
| 19:16:30 | melwitt | it's possible it didn't yet, but let me see if I can locate a list all in the log | |
| 19:17:00 | dansmith | am I missing where our requests are logged in n-api log? | |
| 19:17:49 | melwitt | I'm not seeing it them yet. this doesn't look usual to me | |
| 19:18:06 | dansmith | melwitt: http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/logs/tempest.txt.gz#_2017-02-02_12_53_52_735 | |
| 19:18:13 | dansmith | melwitt: yeah, something seems wrong about it | |
| 19:18:21 | dansmith | melwitt: but the above should have triggered a list all right? | |
| 19:18:29 | jaypipes | johnthetubaguy: +2 | |
| 19:18:55 | melwitt | dansmith: no, I think that's a create. it will return the one instance, or the first instance if it's a multi create | |
| 19:19:08 | melwitt | that's a POST | |
| 19:19:13 | dansmith | oh you're right | |
| 19:19:56 | openstackgerrit | Ed Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 19:20:03 | edleafe | jaypipes: cdent: ^^ please review | |
| 19:20:16 | cdent | edleafe: yessir | |
| 19:20:24 | mriedem | there is no "200 GET https://[::1]:8774/v2.1/servers" in that log | |
| 19:20:49 | dansmith | how is that possible | |
| 19:21:08 | melwitt | this is very odd | |
| 19:22:32 | dansmith | mriedem: so your race... | |
| 19:22:48 | dansmith | mriedem: we look up the map, if we fail, we look up the buildreq, if that fails we re-check for the map | |
| 19:23:25 | dansmith | by the time we've deleted the buildreq, we've finished up the instance map and saved it | |
| 19:23:31 | dansmith | so I don't see that there is any window there | |
| 19:23:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: debugging for instance not found in tripleo https://review.openstack.org/428360 | |
| 19:23:54 | dansmith | are you thinking of a different place? | |
| 19:23:55 | mriedem | dansmith: i was thinking it's like this | |
| 19:24:03 | mriedem | 1. https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2271 we don't have an instance mappingyet | |
| 19:24:23 | mriedem | well, shit that's it | |
| 19:24:46 | mriedem | if there is no instance mapping, that doesn't lookup the build req | |
| 19:24:48 | mriedem | at all, right? | |
| 19:25:09 | dansmith | but we create the instance map early | |
| 19:25:09 | mriedem | the conditionals in there are only caring if there is an instance mapping AND it has a cell mapping | |
| 19:25:12 | melwitt | uh oh, looks tha tway | |
| 19:25:16 | dansmith | lemme look | |
| 19:25:28 | mriedem | we create the instance mapping after the instance is created, but before the build request is deleted | |
| 19:25:43 | dansmith | no | |
| 19:25:45 | dansmith | we update it | |
| 19:25:53 | dansmith | we create the instance mapping at almost the same time as the buildreq | |
| 19:26:01 | dansmith | although the buildreq is first by a few python cycles | |
| 19:26:11 | dansmith | this would be nanoseconds of a window | |
| 19:26:15 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L997 | |
| 19:26:15 | mriedem | oh right | |
| 19:26:21 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1010 | |
| 19:26:27 | dansmith | picoseconds even | |
| 19:26:33 | dansmith | yeah | |
| 19:26:52 | mriedem | i thought you just started making up small time units now | |
| 19:26:59 | mriedem | danoseconds | |
| 19:27:00 | dansmith | femtoseconds | |
| 19:27:12 | melwitt | smithoseconds | |
| 19:27:19 | mriedem | only Vek can say femto-anything | |
| 19:27:21 | dansmith | mriedem: so I can swap the order of those in a test patch to run experimental on, but I'm highly dubious | |
| 19:27:35 | dansmith | mriedem: Vek gave me the femtotorch | |
| 19:27:49 | mriedem | ha | |
| 19:27:57 | dansmith | it was a small and quick handoff | |
| 19:28:06 | mriedem | how appropriate | |
| 19:28:07 | melwitt | /facepalm | |
| 19:28:19 | mriedem | rimshot? | |
| 19:30:03 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Reorder two electrons to see if it fixes a race https://review.openstack.org/428385 | |
| 19:30:24 | melwitt | haha, like that commit message | |
| 19:30:31 | dansmith | so this probably comes from the logic in get_instance needing to be defensive against pre-cellsv2 data, so we could also just rearrange that and probably simplify | |
| 19:54:32 | amoralej | mriedem, dansmith, i've been debugging by printing value of service_version right before https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2268 | |
| 19:54:38 | amoralej | and i'm getting allways 0 | |
| 19:55:06 | amoralej | in fact if y remove the "service_version < 15" part of the if, tempest test passes | |
| 19:55:24 | amoralej | how can i see the versions of each service? | |
| 19:55:37 | amoralej | EmilienM ^ | |
| 19:55:40 | dansmith | amoralej: orly | |
| 19:56:04 | dansmith | amoralej: maybe we're grabbing a cached version of zero before we start computes | |
| 19:56:24 | dansmith | not sure what would have changed around that specific thing recently though | |
| 19:56:38 | amoralej | i can try to remove the compute node and register it again | |
| 19:58:34 | dansmith | we really should never cache version=0, which I think we're probably doing right now | |
| 20:10:27 | amoralej | dansmith, in nova database all services have version 16 | |
| 20:10:35 | amoralej | but nova_cell0 has no services | |
| 20:10:47 | mriedem | there are no hosts for nova_cell0 | |
| 20:11:02 | mriedem | cell0 is just a place to dump dead instances that didn't get built | |
| 20:11:16 | cdent | edleafe: my testing suggests there's no urgency, but it does suggest that checking the allocations is not necessary if we change (and validate) the default min_unit when creating inventory | |
| 20:11:32 | cdent | if that's done, then there's no need to check the allocations manually | |
| 20:11:54 | dansmith | amoralej: the problem is it may have 16 at the end, but might've had zero early such that it cached zero and never saw the 16 | |
| 20:12:38 | amoralej | dansmith, in that case it should work if i restart nova api? | |
| 20:12:39 | amoralej | i can try | |
| 20:12:49 | dansmith | amoralej: yeah | |
| 20:13:56 | openstackgerrit | Ed Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 20:14:44 | mriedem | vdrok: want to test this? https://review.openstack.org/#/c/428375/ | |
| 20:14:59 | openstackgerrit | Dan Smith proposed openstack/nova master: Don't ever cache service versions at zero https://review.openstack.org/428404 | |
| 20:16:04 | amoralej | dansmith, reboot didn't work | |
| 20:16:06 | amoralej | but | |
| 20:16:17 | dansmith | amoralej: did you get all api workers on all nodes? | |
| 20:16:26 | amoralej | replacing nova-osapi_compute by nova-compute worked | |
| 20:16:37 | dansmith | oh jeez | |
| 20:16:38 | dansmith | where? | |
| 20:16:38 | amoralej | so | |
| 20:16:47 | amoralej | https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2268 | |
| 20:16:50 | dansmith | cripes I see | |
| 20:16:51 | dansmith | jeezus | |
| 20:17:02 | dansmith | ffs | |
| 20:17:02 | amoralej | i mean https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2264 | |
| 20:17:04 | dansmith | how. | |
| 20:17:05 | dansmith | many. | |