| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-21 | |||
| 17:50:45 | cdent | which this? | |
| 17:50:53 | mriedem | the random auth failures in functional tests | |
| 17:51:23 | mriedem | i don't see one | |
| 17:51:34 | melwitt | mriedem: oh, sorry. I think I approved that. I thought bc the jobs were passing it wasn't needed anymore, I didn't know it was a thing to get more info | |
| 17:51:45 | mriedem | yeah it's for debug | |
| 17:52:25 | melwitt | I guess put it back, with a comment that says what it's for | |
| 17:52:36 | mriedem | i assume the "BaseException.message has been deprecated as of Python 2.6" was for something else | |
| 17:52:41 | mriedem | b/c i've seen that before too | |
| 17:52:50 | mriedem | it wouldn't actually help in what we're seeing here | |
| 17:52:54 | mriedem | so i'm going to push something else for that | |
| 17:52:55 | mriedem | it | |
| 17:52:55 | mriedem | push | |
| 17:52:56 | mriedem | real | |
| 17:53:10 | melwitt | I thought that's what was causing those warnings was setting of the message attribute | |
| 17:57:26 | mriedem | i don't remember anymore, i added it here https://github.com/openstack/nova/commit/01dd1a05a213c0cbd0097188418cabe915291c8d | |
| 17:57:33 | mriedem | anywho, not the issue here | |
| 17:58:06 | leakypipes | mriedem: this tempest.api.identity.admin.v3.test_users.UsersV3TestJSON.test_password_history_not_enforced_in_admin_reset failure... grrr. | |
| 17:59:12 | mriedem | i think we have a signature for that one | |
| 17:59:22 | mriedem | http://status.openstack.org/elastic-recheck/#1702211 ? | |
| 17:59:42 | mriedem | cdent: melwitt: https://bugs.launchpad.net/nova/+bug/1705753 | |
| 17:59:43 | openstack | Launchpad bug 1705753 in OpenStack Compute (nova) "Random OpenStackApiAuthenticationException: Authentication error in nova functional tests" [Undecided,New] | |
| 18:01:08 | cdent | thanks, mriedem | |
| 18:01:28 | mriedem | i'll push a debug patch | |
| 18:02:38 | cdent | mriedem: do you want me to try the retry on auth fail thingie? | |
| 18:03:40 | mriedem | sure | |
| 18:03:53 | cdent | it seems like it might of some use but doesn’t really get at whatever the issue is, sadly | |
| 18:03:57 | cdent | but yeah, i’ll make one go | |
| 18:04:00 | melwitt | I wonder if this is relevant https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1436-L1438 | |
| 18:04:32 | mriedem | probably | |
| 18:04:47 | mriedem | the spike in failures started when placement fixture was turned on globally in the IntegratedHelpers mixin | |
| 18:05:35 | melwitt | yeah, that lines up with the fact that "The current placement NoAuthMiddleware returns a 401 in case a token is not provided" | |
| 18:05:40 | melwitt | I just don't know what that means | |
| 18:06:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L32 | |
| 18:06:26 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L41 | |
| 18:07:13 | cdent | if that’s playing a part, then it is likely that the problem is when compute requests land on the placement api (which seems to be the core problem here) | |
| 18:07:14 | melwitt | so does that imply that a request is being made that does send the x-auth-token header? is there anything other than GET/PUT/DELETE/POST? | |
| 18:07:22 | melwitt | *does not | |
| 18:07:38 | melwitt | oh. compute requests landing on placement api | |
| 18:07:53 | mriedem | i think there is an eventlet switch that goofs things up | |
| 18:07:55 | mriedem | or that's the theory | |
| 18:08:17 | melwitt | I guess I don't understand that | |
| 18:08:30 | mriedem | this is what compute does https://github.com/openstack/nova/blob/master/nova/api/openstack/auth.py#L32 | |
| 18:09:13 | cdent | placement does what it does to behave like a normal auth middleware and not fake more than it should, it basically stripped that middleware back to the basics | |
| 18:09:24 | cdent | changing it would not fix the real problem here | |
| 18:09:28 | cdent | it would mask it | |
| 18:09:31 | cdent | and we don’t want to do that do we? | |
| 18:09:39 | mriedem | right we do'nt send a fake token for compute requests https://github.com/openstack/nova/blob/master/nova/tests/functional/api/client.py#L142 | |
| 18:10:00 | melwitt | I mean, how does something switch to the wrong api, how does a compute request end up going to the placement api | |
| 18:10:37 | superdan | bad threading | |
| 18:10:39 | cdent | melwitt: the theory is that something is causing eventlet sockets to get confused | |
| 18:10:51 | superdan | yeah | |
| 18:11:23 | melwitt | \:| okay | |
| 18:11:44 | superdan | melwitt: your hair is messed up? | |
| 18:11:59 | melwitt | that's my raised unibrow | |
| 18:12:01 | superdan | eyebrows? | |
| 18:12:02 | superdan | okay | |
| 18:12:03 | superdan | heh | |
| 18:12:09 | cdent | we could run one of the apis (presunably placement) on wsgi intercept instead of a separate server thread, and then it wouldn’t be on threads? | |
| 18:12:31 | cdent | (or rather not in the same way) | |
| 18:16:31 | melwitt | so that means we have two wsgi services total now? it seems like this intercept thing would allow us to set up each one separately (with intercept) right? (because of different host/port combos) | |
| 18:16:49 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: retry on authentication failure in api_client https://review.openstack.org/486190 | |
| 18:17:06 | mriedem | as far as i can tell these are started on the same host and port | |
| 18:17:11 | mriedem | 127.0.0.1:0 | |
| 18:17:14 | cdent | melwitt: yes, that’s right, they would | |
| 18:17:14 | melwitt | just thinking if it would still work if someday we had a third wsgi service | |
| 18:17:18 | cdent | 0 means choose a port | |
| 18:17:44 | cdent | melwitt: yues | |
| 18:17:58 | melwitt | because we do need a real way to isolate these from each other. cool | |
| 18:18:06 | mriedem | [nova.placement.wsgi.server] (4697) wsgi starting up on http://127.0.0.1:45989' | |
| 18:18:15 | mriedem | [nova.osapi_compute.wsgi.server] (4697) wsgi starting up on http://127.0.0.1:33219' | |
| 18:18:16 | mriedem | ok | |
| 18:18:37 | melwitt | so I guess we could do the retry for now and then replace it with intercept whenever one of us gets it working | |
| 18:18:53 | melwitt | assuming that getting intercept to work might be not easy | |
| 18:19:42 | cdent | melwitt: it may require unwinding some of the fixture’s pieces, because the deal wsgi intercept is it takes away the need for fakes | |
| 18:19:57 | mriedem | i think i've got an easier workaround for now | |
| 18:19:57 | cdent | fake requests I mean, you use a real http client to make real http requests to a fake socket | |
| 18:20:13 | melwitt | cdent: ah, okay | |
| 18:20:45 | cdent | melwitt: I think it is probably worth doing regardless of the outcome here, but I’m probably a bit too biased to be the decider on such thing | |
| 18:21:08 | cdent | mriedem: i’m starting to die from lack of air, halp | |
| 18:21:54 | melwitt | cdent: I agree we should do it. just wanted to be clear that I wasn't suggesting we hold off on a workaround because of it | |
| 18:22:27 | openstackgerrit | Merged openstack/nova master: Don't cast cinderclient microversions to float https://review.openstack.org/486096 | |
| 18:22:46 | mriedem | sec | |
| 18:22:47 | mriedem | pushing it pu | |
| 18:22:49 | mriedem | *up | |
| 18:22:51 | melwitt | mriedem is trying to kill cdent | |
| 18:22:59 | cdent | I knew it | |
| 18:26:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass X-Auth-Token in TestOpenStackClient._authenticate https://review.openstack.org/486193 | |
| 18:28:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass X-Auth-Token in TestOpenStackClient._authenticate https://review.openstack.org/486193 | |
| 18:28:33 | mriedem | logstash is back up | |
| 18:28:52 | melwitt | f yes | |
| 18:29:30 | mriedem | language | |
| 18:32:34 | mriedem | doesn't seem to be finding anything thoguh | |
| 18:32:36 | mriedem | *though | |
| 18:32:39 | cdent | mriedem: interesting. I’m not sure your solution will work. It’s trying to prevent the 401 response from placement happening (which it will) but placement’s no auth will not response with the response.headers that the _authenticate method is supposed to provide to its callers | |
| 18:33:14 | cdent | placement no auth middleware doesn’t not set response headers | |
| 18:33:45 | cdent | so the retry thing that I did is more likely to get the desired outcome, isn’t it? (I’m not entirely sure, the gears withing gears isn’t clear) | |
| 18:34:01 | cdent | (double negative above not intentional) | |
| 18:34:06 | mriedem | yo'ure probably right, because the compute api noauth returns a request context, | |
| 18:34:10 | mriedem | that the compute api code requires | |
| 18:34:27 | mriedem | so even though we avoid the 401, the request won't have a context in it | |