| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 19:56:38 | amoralej | i can try to remove the compute node and register it again | |
| 19:58:34 | dansmith | we really should never cache version=0, which I think we're probably doing right now | |
| 20:10:27 | amoralej | dansmith, in nova database all services have version 16 | |
| 20:10:35 | amoralej | but nova_cell0 has no services | |
| 20:10:47 | mriedem | there are no hosts for nova_cell0 | |
| 20:11:02 | mriedem | cell0 is just a place to dump dead instances that didn't get built | |
| 20:11:16 | cdent | edleafe: my testing suggests there's no urgency, but it does suggest that checking the allocations is not necessary if we change (and validate) the default min_unit when creating inventory | |
| 20:11:32 | cdent | if that's done, then there's no need to check the allocations manually | |
| 20:11:54 | dansmith | amoralej: the problem is it may have 16 at the end, but might've had zero early such that it cached zero and never saw the 16 | |
| 20:12:38 | amoralej | dansmith, in that case it should work if i restart nova api? | |
| 20:12:39 | amoralej | i can try | |
| 20:12:49 | dansmith | amoralej: yeah | |
| 20:13:56 | openstackgerrit | Ed Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 20:14:44 | mriedem | vdrok: want to test this? https://review.openstack.org/#/c/428375/ | |
| 20:14:59 | openstackgerrit | Dan Smith proposed openstack/nova master: Don't ever cache service versions at zero https://review.openstack.org/428404 | |
| 20:16:04 | amoralej | dansmith, reboot didn't work | |
| 20:16:06 | amoralej | but | |
| 20:16:17 | dansmith | amoralej: did you get all api workers on all nodes? | |
| 20:16:26 | amoralej | replacing nova-osapi_compute by nova-compute worked | |
| 20:16:37 | dansmith | oh jeez | |
| 20:16:38 | amoralej | so | |
| 20:16:38 | dansmith | where? | |
| 20:16:47 | amoralej | https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2268 | |
| 20:16:50 | dansmith | cripes I see | |
| 20:16:51 | dansmith | jeezus | |
| 20:17:02 | amoralej | i mean https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2264 | |
| 20:17:02 | dansmith | ffs | |
| 20:17:04 | dansmith | how. | |
| 20:17:05 | dansmith | many. | |
| 20:17:06 | dansmith | times. | |
| 20:17:30 | amoralej | is service-name configurable? | |
| 20:17:34 | dansmith | no | |
| 20:17:36 | dansmith | this is a bug | |
| 20:17:43 | dansmith | I dunno why we're not hitting it elsewhere | |
| 20:17:45 | amoralej | EmilienM ^ | |
| 20:18:19 | EmilienM | amoralej: nice catch! | |
| 20:18:37 | amoralej | dansmith, and https://github.com/openstack/nova/blame/master/nova/compute/api.py#L1707 | |
| 20:18:49 | dansmith | wait wait wai | |
| 20:19:01 | dansmith | I think that's actually looking for the api service not the compute service | |
| 20:19:04 | dansmith | so it should be right | |
| 20:19:26 | melwitt | yeah, that should be right. it used to be wrong as 'nova-api' in the past, somewhere in compute/api I recall | |
| 20:19:39 | cdent | edleafe: I left more about the min_unit stuff on https://review.openstack.org/#/c/416752/7 | |
| 20:19:41 | dansmith | right, and I was thinking it was looking for nova-compute, which is what the change was | |
| 20:19:45 | dansmith | so I jumped | |
| 20:19:46 | amoralej | and where api services are registered? | |
| 20:19:54 | dansmith | amoralej: you changing it just means you're looking up a different service | |
| 20:19:55 | amoralej | i don't see it in services table | |
| 20:20:05 | amoralej | we are running it as wsgi | |
| 20:20:22 | dansmith | oh, hmm, I bet that is a problem | |
| 20:20:50 | edleafe | cdent: I can get to those changes today | |
| 20:20:50 | amoralej | dansmith, this is my services table http://paste.openstack.org/show/597426/ | |
| 20:20:52 | dansmith | amoralej: you see no service records in the services table for nova-osapi_compute? | |
| 20:21:01 | dansmith | yeah | |
| 20:21:08 | dansmith | because you don't use our service infrastructure | |
| 20:21:22 | dansmith | mriedem: we've never said we're okay to run under wsgi did we? | |
| 20:21:37 | mriedem | dansmith: we don't have anything official | |
| 20:21:37 | amoralej | EmilienM ^ | |
| 20:21:39 | mriedem | but i know people run it | |
| 20:21:45 | mriedem | i think Fuel did too | |
| 20:21:48 | cdent | edleafe: feel free, but I don't think there's any rush: because the RT sets min_unit correctly, any "proper" use of the placement api will work okay and a bad admin can't send 0 allocations to the inventory that's been written (because min_unit is 1) | |
| 20:22:18 | cdent | (if you do do it, add me to the review so it is on my radar so I know not to do it) | |
| 20:22:23 | dansmith | the thing is, I'm not sure what in the last day would have broken | |
| 20:22:39 | edleafe | cdent: Understood, but I hate building those checks into things external to placement | |
| 20:23:07 | mriedem | dansmith: even with your change, we would have hit this i think https://review.openstack.org/#/c/427782/3/nova/compute/api.py@2269 | |
| 20:23:10 | cdent | edleafe: oh yeah, totally agree | |
| 20:23:14 | EmilienM | nova gate is not running api in wsgi with apache? | |
| 20:23:16 | edleafe | cdent: because the next time some service tries to use placement, we have to copy/paste all that external stuff there, too | |
| 20:23:22 | mriedem | EmilienM: no | |
| 20:23:27 | dansmith | mriedem: on wsgi, yeah | |
| 20:23:37 | EmilienM | why? | |
| 20:23:47 | mriedem | EmilienM: no one has made that happen | |
| 20:24:04 | dansmith | and we have things like this that hook into our service framework | |
| 20:24:27 | mriedem | apache support was hacked in several releases ago as a kind of 'i want to see if i can do this' but it wasn't like a supported new feature or anything | |
| 20:24:32 | amoralej | i'm trying to test it with nova-api | |
| 20:24:32 | mriedem | it was a little toy patch | |
| 20:25:42 | EmilienM | tripleo has been running it in productince since Newton | |
| 20:25:50 | EmilienM | production* | |
| 20:26:29 | dansmith | so few real-world users at this point I guess | |
| 20:26:52 | dansmith | dims_: since you were originally involved, any idea how many real people running fuel might be running nova api under apache? | |
| 20:27:15 | EmilienM | dansmith: few real-world users? It's in OSP10 | |
| 20:27:56 | dansmith | EmilienM: yeah, I dunno what the footprint of that looks like | |
| 20:28:34 | mriedem | jaypipes: "any idea how many real people running fuel might be running nova api under apache?" | |
| 20:29:01 | mwhahaha | mriedem: a bunch, it's in 9 believe | |
| 20:30:31 | mriedem | https://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/wsgi/nova-api.py#L15 | |
| 20:30:35 | mriedem | EXPERIMENTAL support script for running Nova API under Apache2 etc. | |
| 20:30:37 | mriedem | anywho | |
| 20:30:56 | jroll | experimental means battle-tested production grade software, right | |
| 20:31:01 | jroll | like nova-bm | |
| 20:31:05 | mriedem | carrier grade | |
| 20:31:09 | jroll | ++ | |
| 20:31:48 | cdent | I'm unclear why exporting deploy() as application is so fraught with risk? | |
| 20:32:03 | EmilienM | mriedem: yeah a lot of operators are doing that | |
| 20:32:13 | EmilienM | mriedem: I don't have numbers but it's something common | |
| 20:32:49 | mriedem | cdent: like i said earlier, it was just a 'lets see if this can happen' thing long ago that we never built CI around | |
| 20:32:50 | jroll | cdent: I think the current theory is that it doesn't get any of this code: https://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/service.py#L139 | |
| 20:33:00 | melwitt | yeah, we didn't do it but thought we should have since things like the api rate limiting can't work as-is | |
| 20:33:01 | jroll | specifically line 154 | |
| 20:33:25 | mriedem | so i think we provide a workaround hack in ocata | |
| 20:33:28 | cdent | thank jroll that's what I was after | |
| 20:33:33 | mriedem | and work on running nova-api under apache in one of the gate jobs in pike | |