| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 19:23:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: debugging for instance not found in tripleo https://review.openstack.org/428360 | |
| 19:23:54 | dansmith | are you thinking of a different place? | |
| 19:23:55 | mriedem | dansmith: i was thinking it's like this | |
| 19:24:03 | mriedem | 1. https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2271 we don't have an instance mappingyet | |
| 19:24:23 | mriedem | well, shit that's it | |
| 19:24:46 | mriedem | if there is no instance mapping, that doesn't lookup the build req | |
| 19:24:48 | mriedem | at all, right? | |
| 19:25:09 | dansmith | but we create the instance map early | |
| 19:25:09 | mriedem | the conditionals in there are only caring if there is an instance mapping AND it has a cell mapping | |
| 19:25:12 | melwitt | uh oh, looks tha tway | |
| 19:25:16 | dansmith | lemme look | |
| 19:25:28 | mriedem | we create the instance mapping after the instance is created, but before the build request is deleted | |
| 19:25:43 | dansmith | no | |
| 19:25:45 | dansmith | we update it | |
| 19:25:53 | dansmith | we create the instance mapping at almost the same time as the buildreq | |
| 19:26:01 | dansmith | although the buildreq is first by a few python cycles | |
| 19:26:11 | dansmith | this would be nanoseconds of a window | |
| 19:26:15 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L997 | |
| 19:26:15 | mriedem | oh right | |
| 19:26:21 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1010 | |
| 19:26:27 | dansmith | picoseconds even | |
| 19:26:33 | dansmith | yeah | |
| 19:26:52 | mriedem | i thought you just started making up small time units now | |
| 19:26:59 | mriedem | danoseconds | |
| 19:27:00 | dansmith | femtoseconds | |
| 19:27:12 | melwitt | smithoseconds | |
| 19:27:19 | mriedem | only Vek can say femto-anything | |
| 19:27:21 | dansmith | mriedem: so I can swap the order of those in a test patch to run experimental on, but I'm highly dubious | |
| 19:27:35 | dansmith | mriedem: Vek gave me the femtotorch | |
| 19:27:49 | mriedem | ha | |
| 19:27:57 | dansmith | it was a small and quick handoff | |
| 19:28:06 | mriedem | how appropriate | |
| 19:28:07 | melwitt | /facepalm | |
| 19:28:19 | mriedem | rimshot? | |
| 19:30:03 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Reorder two electrons to see if it fixes a race https://review.openstack.org/428385 | |
| 19:30:24 | melwitt | haha, like that commit message | |
| 19:30:31 | dansmith | so this probably comes from the logic in get_instance needing to be defensive against pre-cellsv2 data, so we could also just rearrange that and probably simplify | |
| 19:54:32 | amoralej | mriedem, dansmith, i've been debugging by printing value of service_version right before https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2268 | |
| 19:54:38 | amoralej | and i'm getting allways 0 | |
| 19:55:06 | amoralej | in fact if y remove the "service_version < 15" part of the if, tempest test passes | |
| 19:55:24 | amoralej | how can i see the versions of each service? | |
| 19:55:37 | amoralej | EmilienM ^ | |
| 19:55:40 | dansmith | amoralej: orly | |
| 19:56:04 | dansmith | amoralej: maybe we're grabbing a cached version of zero before we start computes | |
| 19:56:24 | dansmith | not sure what would have changed around that specific thing recently though | |
| 19:56:38 | amoralej | i can try to remove the compute node and register it again | |
| 19:58:34 | dansmith | we really should never cache version=0, which I think we're probably doing right now | |
| 20:10:27 | amoralej | dansmith, in nova database all services have version 16 | |
| 20:10:35 | amoralej | but nova_cell0 has no services | |
| 20:10:47 | mriedem | there are no hosts for nova_cell0 | |
| 20:11:02 | mriedem | cell0 is just a place to dump dead instances that didn't get built | |
| 20:11:16 | cdent | edleafe: my testing suggests there's no urgency, but it does suggest that checking the allocations is not necessary if we change (and validate) the default min_unit when creating inventory | |
| 20:11:32 | cdent | if that's done, then there's no need to check the allocations manually | |
| 20:11:54 | dansmith | amoralej: the problem is it may have 16 at the end, but might've had zero early such that it cached zero and never saw the 16 | |
| 20:12:38 | amoralej | dansmith, in that case it should work if i restart nova api? | |
| 20:12:39 | amoralej | i can try | |
| 20:12:49 | dansmith | amoralej: yeah | |
| 20:13:56 | openstackgerrit | Ed Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 20:14:44 | mriedem | vdrok: want to test this? https://review.openstack.org/#/c/428375/ | |
| 20:14:59 | openstackgerrit | Dan Smith proposed openstack/nova master: Don't ever cache service versions at zero https://review.openstack.org/428404 | |
| 20:16:04 | amoralej | dansmith, reboot didn't work | |
| 20:16:06 | amoralej | but | |
| 20:16:17 | dansmith | amoralej: did you get all api workers on all nodes? | |
| 20:16:26 | amoralej | replacing nova-osapi_compute by nova-compute worked | |
| 20:16:37 | dansmith | oh jeez | |
| 20:16:38 | dansmith | where? | |
| 20:16:38 | amoralej | so | |
| 20:16:47 | amoralej | https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2268 | |
| 20:16:50 | dansmith | cripes I see | |
| 20:16:51 | dansmith | jeezus | |
| 20:17:02 | dansmith | ffs | |
| 20:17:02 | amoralej | i mean https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2264 | |
| 20:17:04 | dansmith | how. | |
| 20:17:05 | dansmith | many. | |
| 20:17:06 | dansmith | times. | |
| 20:17:30 | amoralej | is service-name configurable? | |
| 20:17:34 | dansmith | no | |
| 20:17:36 | dansmith | this is a bug | |
| 20:17:43 | dansmith | I dunno why we're not hitting it elsewhere | |
| 20:17:45 | amoralej | EmilienM ^ | |
| 20:18:19 | EmilienM | amoralej: nice catch! | |
| 20:18:37 | amoralej | dansmith, and https://github.com/openstack/nova/blame/master/nova/compute/api.py#L1707 | |
| 20:18:49 | dansmith | wait wait wai | |
| 20:19:01 | dansmith | I think that's actually looking for the api service not the compute service | |
| 20:19:04 | dansmith | so it should be right | |
| 20:19:26 | melwitt | yeah, that should be right. it used to be wrong as 'nova-api' in the past, somewhere in compute/api I recall | |
| 20:19:39 | cdent | edleafe: I left more about the min_unit stuff on https://review.openstack.org/#/c/416752/7 | |
| 20:19:41 | dansmith | right, and I was thinking it was looking for nova-compute, which is what the change was | |
| 20:19:45 | dansmith | so I jumped | |
| 20:19:46 | amoralej | and where api services are registered? | |
| 20:19:54 | dansmith | amoralej: you changing it just means you're looking up a different service | |
| 20:19:55 | amoralej | i don't see it in services table | |
| 20:20:05 | amoralej | we are running it as wsgi | |
| 20:20:22 | dansmith | oh, hmm, I bet that is a problem | |
| 20:20:50 | amoralej | dansmith, this is my services table http://paste.openstack.org/show/597426/ | |
| 20:20:50 | edleafe | cdent: I can get to those changes today | |
| 20:20:52 | dansmith | amoralej: you see no service records in the services table for nova-osapi_compute? | |
| 20:21:01 | dansmith | yeah | |
| 20:21:08 | dansmith | because you don't use our service infrastructure | |
| 20:21:22 | dansmith | mriedem: we've never said we're okay to run under wsgi did we? | |