| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-21 | |||
| 18:04:00 | melwitt | I wonder if this is relevant https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1436-L1438 | |
| 18:04:32 | mriedem | probably | |
| 18:04:47 | mriedem | the spike in failures started when placement fixture was turned on globally in the IntegratedHelpers mixin | |
| 18:05:35 | melwitt | yeah, that lines up with the fact that "The current placement NoAuthMiddleware returns a 401 in case a token is not provided" | |
| 18:05:40 | melwitt | I just don't know what that means | |
| 18:06:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L32 | |
| 18:06:26 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L41 | |
| 18:07:13 | cdent | if that’s playing a part, then it is likely that the problem is when compute requests land on the placement api (which seems to be the core problem here) | |
| 18:07:14 | melwitt | so does that imply that a request is being made that does send the x-auth-token header? is there anything other than GET/PUT/DELETE/POST? | |
| 18:07:22 | melwitt | *does not | |
| 18:07:38 | melwitt | oh. compute requests landing on placement api | |
| 18:07:53 | mriedem | i think there is an eventlet switch that goofs things up | |
| 18:07:55 | mriedem | or that's the theory | |
| 18:08:17 | melwitt | I guess I don't understand that | |
| 18:08:30 | mriedem | this is what compute does https://github.com/openstack/nova/blob/master/nova/api/openstack/auth.py#L32 | |
| 18:09:13 | cdent | placement does what it does to behave like a normal auth middleware and not fake more than it should, it basically stripped that middleware back to the basics | |
| 18:09:24 | cdent | changing it would not fix the real problem here | |
| 18:09:28 | cdent | it would mask it | |
| 18:09:31 | cdent | and we don’t want to do that do we? | |
| 18:09:39 | mriedem | right we do'nt send a fake token for compute requests https://github.com/openstack/nova/blob/master/nova/tests/functional/api/client.py#L142 | |
| 18:10:00 | melwitt | I mean, how does something switch to the wrong api, how does a compute request end up going to the placement api | |
| 18:10:37 | superdan | bad threading | |
| 18:10:39 | cdent | melwitt: the theory is that something is causing eventlet sockets to get confused | |
| 18:10:51 | superdan | yeah | |
| 18:11:23 | melwitt | \:| okay | |
| 18:11:44 | superdan | melwitt: your hair is messed up? | |
| 18:11:59 | melwitt | that's my raised unibrow | |
| 18:12:01 | superdan | eyebrows? | |
| 18:12:02 | superdan | okay | |
| 18:12:03 | superdan | heh | |
| 18:12:09 | cdent | we could run one of the apis (presunably placement) on wsgi intercept instead of a separate server thread, and then it wouldn’t be on threads? | |
| 18:12:31 | cdent | (or rather not in the same way) | |
| 18:16:31 | melwitt | so that means we have two wsgi services total now? it seems like this intercept thing would allow us to set up each one separately (with intercept) right? (because of different host/port combos) | |
| 18:16:49 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: retry on authentication failure in api_client https://review.openstack.org/486190 | |
| 18:17:06 | mriedem | as far as i can tell these are started on the same host and port | |
| 18:17:11 | mriedem | 127.0.0.1:0 | |
| 18:17:14 | melwitt | just thinking if it would still work if someday we had a third wsgi service | |
| 18:17:14 | cdent | melwitt: yes, that’s right, they would | |
| 18:17:18 | cdent | 0 means choose a port | |
| 18:17:44 | cdent | melwitt: yues | |
| 18:17:58 | melwitt | because we do need a real way to isolate these from each other. cool | |
| 18:18:06 | mriedem | [nova.placement.wsgi.server] (4697) wsgi starting up on http://127.0.0.1:45989' | |
| 18:18:15 | mriedem | [nova.osapi_compute.wsgi.server] (4697) wsgi starting up on http://127.0.0.1:33219' | |
| 18:18:16 | mriedem | ok | |
| 18:18:37 | melwitt | so I guess we could do the retry for now and then replace it with intercept whenever one of us gets it working | |
| 18:18:53 | melwitt | assuming that getting intercept to work might be not easy | |
| 18:19:42 | cdent | melwitt: it may require unwinding some of the fixture’s pieces, because the deal wsgi intercept is it takes away the need for fakes | |
| 18:19:57 | cdent | fake requests I mean, you use a real http client to make real http requests to a fake socket | |
| 18:19:57 | mriedem | i think i've got an easier workaround for now | |
| 18:20:13 | melwitt | cdent: ah, okay | |
| 18:20:45 | cdent | melwitt: I think it is probably worth doing regardless of the outcome here, but I’m probably a bit too biased to be the decider on such thing | |
| 18:21:08 | cdent | mriedem: i’m starting to die from lack of air, halp | |
| 18:21:54 | melwitt | cdent: I agree we should do it. just wanted to be clear that I wasn't suggesting we hold off on a workaround because of it | |
| 18:22:27 | openstackgerrit | Merged openstack/nova master: Don't cast cinderclient microversions to float https://review.openstack.org/486096 | |
| 18:22:46 | mriedem | sec | |
| 18:22:47 | mriedem | pushing it pu | |
| 18:22:49 | mriedem | *up | |
| 18:22:51 | melwitt | mriedem is trying to kill cdent | |
| 18:22:59 | cdent | I knew it | |
| 18:26:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass X-Auth-Token in TestOpenStackClient._authenticate https://review.openstack.org/486193 | |
| 18:28:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass X-Auth-Token in TestOpenStackClient._authenticate https://review.openstack.org/486193 | |
| 18:28:33 | mriedem | logstash is back up | |
| 18:28:52 | melwitt | f yes | |
| 18:29:30 | mriedem | language | |
| 18:32:34 | mriedem | doesn't seem to be finding anything thoguh | |
| 18:32:36 | mriedem | *though | |
| 18:32:39 | cdent | mriedem: interesting. I’m not sure your solution will work. It’s trying to prevent the 401 response from placement happening (which it will) but placement’s no auth will not response with the response.headers that the _authenticate method is supposed to provide to its callers | |
| 18:33:14 | cdent | placement no auth middleware doesn’t not set response headers | |
| 18:33:45 | cdent | so the retry thing that I did is more likely to get the desired outcome, isn’t it? (I’m not entirely sure, the gears withing gears isn’t clear) | |
| 18:34:01 | cdent | (double negative above not intentional) | |
| 18:34:06 | mriedem | yo'ure probably right, because the compute api noauth returns a request context, | |
| 18:34:10 | mriedem | that the compute api code requires | |
| 18:34:27 | mriedem | so even though we avoid the 401, the request won't have a context in it | |
| 18:34:50 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285 | |
| 18:34:51 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Translate the return value of attachment_create and _update https://review.openstack.org/486194 | |
| 18:35:20 | ildikov | mriedem: refactor + fixed old flow tests ^^ | |
| 18:36:42 | cdent | whatever the outcome, I can investigate the wsgi intercept stuff on monday, at least to spike it to see if it has any potential | |
| 18:36:45 | mriedem | cdent: let's un-DNM yours | |
| 18:36:49 | mriedem | and just push it in | |
| 18:36:55 | cdent | k, on it | |
| 18:39:43 | openstackgerrit | Chris Dent proposed openstack/nova master: retry on authentication failure in api_client https://review.openstack.org/486190 | |
| 18:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: retry on authentication failure in api_client https://review.openstack.org/486190 | |
| 18:41:12 | cdent | pushed that before I saw your coments melwitt and mriedem, got the NOTE anyway by chance, but efect typo still there | |
| 18:41:23 | mriedem | it is?!!?!??!?! | |
| 18:41:42 | melwitt | lol | |
| 18:41:58 | cdent | dood, I’m just going to let you do everything henceforth | |
| 18:42:06 | cdent | you are fleet of foot | |
| 18:42:20 | mriedem | i also have a light step | |
| 18:42:24 | mriedem | you'll never see it coming | |
| 18:42:56 | melwitt | well, that was fun | |
| 18:43:05 | mriedem | back to business | |
| 18:43:51 | cdent | this is me when you come for me mriedem https://www.youtube.com/watch?v=vZvbhy5lDgY | |
| 18:44:01 | melwitt | not getting any results in logstash, I see they're talking about some missing entries in #openstack-infra | |
| 18:44:15 | mriedem | i don't want to be the guy from blue velvet | |
| 18:44:22 | melwitt | haha | |
| 18:44:23 | mriedem | the 02 guy | |
| 18:44:54 | openstackgerrit | Merged openstack/nova master: Updated from global requirements https://review.openstack.org/485634 | |
| 18:45:42 | cdent | I’ve always felt a kinship for that guy in the oil, waiting to blow up. I think there ought to be a DSM-5 entry for that specific mental illness. | |
| 18:46:52 | melwitt | hah | |
| 18:48:15 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Translate the return value of attachment_create and _update https://review.openstack.org/486194 | |