| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 20:13:56 | openstackgerrit | Ed Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 20:14:44 | mriedem | vdrok: want to test this? https://review.openstack.org/#/c/428375/ | |
| 20:14:59 | openstackgerrit | Dan Smith proposed openstack/nova master: Don't ever cache service versions at zero https://review.openstack.org/428404 | |
| 20:16:04 | amoralej | dansmith, reboot didn't work | |
| 20:16:06 | amoralej | but | |
| 20:16:17 | dansmith | amoralej: did you get all api workers on all nodes? | |
| 20:16:26 | amoralej | replacing nova-osapi_compute by nova-compute worked | |
| 20:16:37 | dansmith | oh jeez | |
| 20:16:38 | dansmith | where? | |
| 20:16:38 | amoralej | so | |
| 20:16:47 | amoralej | https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2268 | |
| 20:16:50 | dansmith | cripes I see | |
| 20:16:51 | dansmith | jeezus | |
| 20:17:02 | dansmith | ffs | |
| 20:17:02 | amoralej | i mean https://github.com/openstack/nova/blame/master/nova/compute/api.py#L2264 | |
| 20:17:04 | dansmith | how. | |
| 20:17:05 | dansmith | many. | |
| 20:17:06 | dansmith | times. | |
| 20:17:30 | amoralej | is service-name configurable? | |
| 20:17:34 | dansmith | no | |
| 20:17:36 | dansmith | this is a bug | |
| 20:17:43 | dansmith | I dunno why we're not hitting it elsewhere | |
| 20:17:45 | amoralej | EmilienM ^ | |
| 20:18:19 | EmilienM | amoralej: nice catch! | |
| 20:18:37 | amoralej | dansmith, and https://github.com/openstack/nova/blame/master/nova/compute/api.py#L1707 | |
| 20:18:49 | dansmith | wait wait wai | |
| 20:19:01 | dansmith | I think that's actually looking for the api service not the compute service | |
| 20:19:04 | dansmith | so it should be right | |
| 20:19:26 | melwitt | yeah, that should be right. it used to be wrong as 'nova-api' in the past, somewhere in compute/api I recall | |
| 20:19:39 | cdent | edleafe: I left more about the min_unit stuff on https://review.openstack.org/#/c/416752/7 | |
| 20:19:41 | dansmith | right, and I was thinking it was looking for nova-compute, which is what the change was | |
| 20:19:45 | dansmith | so I jumped | |
| 20:19:46 | amoralej | and where api services are registered? | |
| 20:19:54 | dansmith | amoralej: you changing it just means you're looking up a different service | |
| 20:19:55 | amoralej | i don't see it in services table | |
| 20:20:05 | amoralej | we are running it as wsgi | |
| 20:20:22 | dansmith | oh, hmm, I bet that is a problem | |
| 20:20:50 | amoralej | dansmith, this is my services table http://paste.openstack.org/show/597426/ | |
| 20:20:50 | edleafe | cdent: I can get to those changes today | |
| 20:20:52 | dansmith | amoralej: you see no service records in the services table for nova-osapi_compute? | |
| 20:21:01 | dansmith | yeah | |
| 20:21:08 | dansmith | because you don't use our service infrastructure | |
| 20:21:22 | dansmith | mriedem: we've never said we're okay to run under wsgi did we? | |
| 20:21:37 | amoralej | EmilienM ^ | |
| 20:21:37 | mriedem | dansmith: we don't have anything official | |
| 20:21:39 | mriedem | but i know people run it | |
| 20:21:45 | mriedem | i think Fuel did too | |
| 20:21:48 | cdent | edleafe: feel free, but I don't think there's any rush: because the RT sets min_unit correctly, any "proper" use of the placement api will work okay and a bad admin can't send 0 allocations to the inventory that's been written (because min_unit is 1) | |
| 20:22:18 | cdent | (if you do do it, add me to the review so it is on my radar so I know not to do it) | |
| 20:22:23 | dansmith | the thing is, I'm not sure what in the last day would have broken | |
| 20:22:39 | edleafe | cdent: Understood, but I hate building those checks into things external to placement | |
| 20:23:07 | mriedem | dansmith: even with your change, we would have hit this i think https://review.openstack.org/#/c/427782/3/nova/compute/api.py@2269 | |
| 20:23:10 | cdent | edleafe: oh yeah, totally agree | |
| 20:23:14 | EmilienM | nova gate is not running api in wsgi with apache? | |
| 20:23:16 | edleafe | cdent: because the next time some service tries to use placement, we have to copy/paste all that external stuff there, too | |
| 20:23:22 | mriedem | EmilienM: no | |
| 20:23:27 | dansmith | mriedem: on wsgi, yeah | |
| 20:23:37 | EmilienM | why? | |
| 20:23:47 | mriedem | EmilienM: no one has made that happen | |
| 20:24:04 | dansmith | and we have things like this that hook into our service framework | |
| 20:24:27 | mriedem | apache support was hacked in several releases ago as a kind of 'i want to see if i can do this' but it wasn't like a supported new feature or anything | |
| 20:24:32 | mriedem | it was a little toy patch | |
| 20:24:32 | amoralej | i'm trying to test it with nova-api | |
| 20:25:42 | EmilienM | tripleo has been running it in productince since Newton | |
| 20:25:50 | EmilienM | production* | |
| 20:26:29 | dansmith | so few real-world users at this point I guess | |
| 20:26:52 | dansmith | dims_: since you were originally involved, any idea how many real people running fuel might be running nova api under apache? | |
| 20:27:15 | EmilienM | dansmith: few real-world users? It's in OSP10 | |
| 20:27:56 | dansmith | EmilienM: yeah, I dunno what the footprint of that looks like | |
| 20:28:34 | mriedem | jaypipes: "any idea how many real people running fuel might be running nova api under apache?" | |
| 20:29:01 | mwhahaha | mriedem: a bunch, it's in 9 believe | |
| 20:30:31 | mriedem | https://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/wsgi/nova-api.py#L15 | |
| 20:30:35 | mriedem | EXPERIMENTAL support script for running Nova API under Apache2 etc. | |
| 20:30:37 | mriedem | anywho | |
| 20:30:56 | jroll | experimental means battle-tested production grade software, right | |
| 20:31:01 | jroll | like nova-bm | |
| 20:31:05 | mriedem | carrier grade | |
| 20:31:09 | jroll | ++ | |
| 20:31:48 | cdent | I'm unclear why exporting deploy() as application is so fraught with risk? | |
| 20:32:03 | EmilienM | mriedem: yeah a lot of operators are doing that | |
| 20:32:13 | EmilienM | mriedem: I don't have numbers but it's something common | |
| 20:32:49 | mriedem | cdent: like i said earlier, it was just a 'lets see if this can happen' thing long ago that we never built CI around | |
| 20:32:50 | jroll | cdent: I think the current theory is that it doesn't get any of this code: https://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/service.py#L139 | |
| 20:33:00 | melwitt | yeah, we didn't do it but thought we should have since things like the api rate limiting can't work as-is | |
| 20:33:01 | jroll | specifically line 154 | |
| 20:33:25 | mriedem | so i think we provide a workaround hack in ocata | |
| 20:33:28 | cdent | thank jroll that's what I was after | |
| 20:33:33 | mriedem | and work on running nova-api under apache in one of the gate jobs in pike | |
| 20:33:45 | jroll | cdent: it's overridden for the wsgi thing, I guess, but same idea: https://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/service.py#L376 | |
| 20:33:49 | jroll | no problem | |
| 20:35:00 | mwhahaha | so to answer, fuel doesn't use apache for the nova api. it's haproxy fronting the standalone service | |
| 20:35:09 | EmilienM | I think Nova team should consider moving the experimental tripleo job to the gate. We have seen over the last weeks (more probably) that it brings very useful feedback on how it works outside devstack | |
| 20:35:36 | dansmith | mwhahaha: ah, good to know | |
| 20:37:28 | EmilienM | mriedem: I think testing api service with apache is part of the goal for Pike | |
| 20:37:38 | mriedem | which goal? | |
| 20:37:42 | mriedem | how many goals are we up to now? | |
| 20:37:59 | mriedem | i said we can start running it in one gating job for nova in pike as a start | |
| 20:38:01 | EmilienM | in https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html#completion-criteria | |
| 20:38:09 | mriedem | because we have a unicorn snowflake job for stuff like that already | |
| 20:38:11 | EmilienM | mriedem: 2 goals | |