| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 19:26:33 | dansmith | yeah | |
| 19:26:52 | mriedem | i thought you just started making up small time units now | |
| 19:26:59 | mriedem | danoseconds | |
| 19:27:00 | dansmith | femtoseconds | |
| 19:27:12 | melwitt | smithoseconds | |
| 19:27:19 | mriedem | only Vek can say femto-anything | |
| 19:27:21 | dansmith | mriedem: so I can swap the order of those in a test patch to run experimental on, but I'm highly dubious | |
| 19:27:35 | dansmith | mriedem: Vek gave me the femtotorch | |
| 19:27:49 | mriedem | ha | |
| 19:27:57 | dansmith | it was a small and quick handoff | |
| 19:28:06 | mriedem | how appropriate | |
| 19:28:07 | melwitt | /facepalm | |
| 19:28:19 | mriedem | rimshot? | |
| 19:30:03 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Reorder two electrons to see if it fixes a race https://review.openstack.org/428385 | |
| 19:30:24 | melwitt | haha, like that commit message | |
| 19:30:31 | dansmith | so this probably comes from the logic in get_instance needing to be defensive against pre-cellsv2 data, so we could also just rearrange that and probably simplify | |
| 19:54:32 | amoralej | mriedem, dansmith, i've been debugging by printing value of service_version right before https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2268 | |
| 19:54:38 | amoralej | and i'm getting allways 0 | |
| 19:55:06 | amoralej | in fact if y remove the "service_version < 15" part of the if, tempest test passes | |
| 19:55:24 | amoralej | how can i see the versions of each service? | |
| 19:55:37 | amoralej | EmilienM ^ | |
| 19:55:40 | dansmith | amoralej: orly | |
| 19:56:04 | dansmith | amoralej: maybe we're grabbing a cached version of zero before we start computes | |
| 19:56:24 | dansmith | not sure what would have changed around that specific thing recently though | |
| 19:56:38 | amoralej | i can try to remove the compute node and register it again | |
| 19:58:34 | dansmith | we really should never cache version=0, which I think we're probably doing right now | |
| 20:10:27 | amoralej | dansmith, in nova database all services have version 16 | |
| 20:10:35 | amoralej | but nova_cell0 has no services | |
| 20:10:47 | mriedem | there are no hosts for nova_cell0 | |
| 20:11:02 | mriedem | cell0 is just a place to dump dead instances that didn't get built | |
| 20:11:16 | cdent | edleafe: my testing suggests there's no urgency, but it does suggest that checking the allocations is not necessary if we change (and validate) the default min_unit when creating inventory | |
| 20:11:32 | cdent | if that's done, then there's no need to check the allocations manually | |
| 20:11:54 | dansmith | amoralej: the problem is it may have 16 at the end, but might've had zero early such that it cached zero and never saw the 16 | |
| 20:12:38 | amoralej | dansmith, in that case it should work if i restart nova api? | |
| 20:12:39 | amoralej | i can try | |
| 20:12:49 | dansmith | amoralej: yeah | |
| 20:13:56 | openstackgerrit | Ed Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 20:14:44 | mriedem | vdrok: want to test this? https://review.openstack.org/#/c/428375/ | |
| 20:14:59 | openstackgerrit | Dan Smith proposed openstack/nova master: Don't ever cache service versions at zero https://review.openstack.org/428404 | |
| 20:16:04 | amoralej | dansmith, reboot didn't work | |
| 20:16:06 | amoralej | but | |
| 20:16:17 | dansmith | amoralej: did you get all api workers on all nodes? | |
| 20:16:26 | amoralej | replacing nova-osapi_compute by nova-compute worked | |
| 20:16:37 | dansmith | oh jeez | |
| 20:16:38 | amoralej | so | |
| 20:16:38 | dansmith | where? | |
| 20:16:47 | amoralej | https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2268 | |
| 20:16:50 | dansmith | cripes I see | |
| 20:16:51 | dansmith | jeezus | |
| 20:17:02 | amoralej | i mean https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2264 | |
| 20:17:02 | dansmith | ffs | |
| 20:17:04 | dansmith | how. | |
| 20:17:05 | dansmith | many. | |
| 20:17:06 | dansmith | times. | |
| 20:17:30 | amoralej | is service-name configurable? | |
| 20:17:34 | dansmith | no | |
| 20:17:36 | dansmith | this is a bug | |
| 20:17:43 | dansmith | I dunno why we're not hitting it elsewhere | |
| 20:17:45 | amoralej | EmilienM ^ | |
| 20:18:19 | EmilienM | amoralej: nice catch! | |
| 20:18:37 | amoralej | dansmith, and https://github.com/openstack/nova/blame/master/nova/compute/api.py#L1707 | |
| 20:18:49 | dansmith | wait wait wai | |
| 20:19:01 | dansmith | I think that's actually looking for the api service not the compute service | |
| 20:19:04 | dansmith | so it should be right | |
| 20:19:26 | melwitt | yeah, that should be right. it used to be wrong as 'nova-api' in the past, somewhere in compute/api I recall | |
| 20:19:39 | cdent | edleafe: I left more about the min_unit stuff on https://review.openstack.org/#/c/416752/7 | |
| 20:19:41 | dansmith | right, and I was thinking it was looking for nova-compute, which is what the change was | |
| 20:19:45 | dansmith | so I jumped | |
| 20:19:46 | amoralej | and where api services are registered? | |
| 20:19:54 | dansmith | amoralej: you changing it just means you're looking up a different service | |
| 20:19:55 | amoralej | i don't see it in services table | |
| 20:20:05 | amoralej | we are running it as wsgi | |
| 20:20:22 | dansmith | oh, hmm, I bet that is a problem | |
| 20:20:50 | edleafe | cdent: I can get to those changes today | |
| 20:20:50 | amoralej | dansmith, this is my services table http://paste.openstack.org/show/597426/ | |
| 20:20:52 | dansmith | amoralej: you see no service records in the services table for nova-osapi_compute? | |
| 20:21:01 | dansmith | yeah | |
| 20:21:08 | dansmith | because you don't use our service infrastructure | |
| 20:21:22 | dansmith | mriedem: we've never said we're okay to run under wsgi did we? | |
| 20:21:37 | mriedem | dansmith: we don't have anything official | |
| 20:21:37 | amoralej | EmilienM ^ | |
| 20:21:39 | mriedem | but i know people run it | |
| 20:21:45 | mriedem | i think Fuel did too | |
| 20:21:48 | cdent | edleafe: feel free, but I don't think there's any rush: because the RT sets min_unit correctly, any "proper" use of the placement api will work okay and a bad admin can't send 0 allocations to the inventory that's been written (because min_unit is 1) | |
| 20:22:18 | cdent | (if you do do it, add me to the review so it is on my radar so I know not to do it) | |
| 20:22:23 | dansmith | the thing is, I'm not sure what in the last day would have broken | |
| 20:22:39 | edleafe | cdent: Understood, but I hate building those checks into things external to placement | |
| 20:23:07 | mriedem | dansmith: even with your change, we would have hit this i think https://review.openstack.org/#/c/427782/3/nova/compute/api.py@2269 | |
| 20:23:10 | cdent | edleafe: oh yeah, totally agree | |
| 20:23:14 | EmilienM | nova gate is not running api in wsgi with apache? | |
| 20:23:16 | edleafe | cdent: because the next time some service tries to use placement, we have to copy/paste all that external stuff there, too | |
| 20:23:22 | mriedem | EmilienM: no | |
| 20:23:27 | dansmith | mriedem: on wsgi, yeah | |
| 20:23:37 | EmilienM | why? | |
| 20:23:47 | mriedem | EmilienM: no one has made that happen | |
| 20:24:04 | dansmith | and we have things like this that hook into our service framework | |
| 20:24:27 | mriedem | apache support was hacked in several releases ago as a kind of 'i want to see if i can do this' but it wasn't like a supported new feature or anything | |
| 20:24:32 | amoralej | i'm trying to test it with nova-api | |
| 20:24:32 | mriedem | it was a little toy patch | |
| 20:25:42 | EmilienM | tripleo has been running it in productince since Newton | |