| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-21 | |||
| 16:37:53 | leakypipes | cdent: sure, please do if you'd like | |
| 16:38:00 | cdent | ✔ | |
| 16:38:35 | cdent | am I slow or is gerrit slow? | |
| 16:39:01 | figleaf | cdent: both are possible | |
| 16:39:56 | figleaf | leakypipes: making progress on the alternates, and then lost connectivity to my dev env. Will continue after lunch | |
| 16:40:07 | leakypipes | figleaf: sounds good, thanks Ed | |
| 16:47:59 | melwitt | I wonder what the func test job fail rate is, it seems like it's got to be near 50% | |
| 16:51:50 | leakypipes | melwitt: yes, likely. I'm hoping the bug fix for that join order will help some. | |
| 16:53:14 | melwitt | yeah, true. most of the fails I see are "authentication error" and versioned notifications stuff | |
| 16:56:09 | cdent | melwitt: I think that auth error is part of the compute and placement fixtures not working well together all the time | |
| 16:57:04 | melwitt | cdent: yeah, I've heard that. recently mriedem_afk tried adding the keepalive=False globally but it didn't seem to have helped | |
| 16:57:05 | cdent | leakypipes: as far as I can tell the functionality for shared providers on /resource_providers was never implemented, only talked about: it’s talked about in the commit message of this: https://review.openstack.org/#/c/460798/ but I can’t find where it happened | |
| 16:57:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: provide an RFB security proxy implementation https://review.openstack.org/345399 | |
| 16:57:29 | leakypipes | cdent: heh, yes, that's correct. :) | |
| 16:57:51 | leakypipes | cdent: I'd forgotten about that | |
| 16:58:57 | cdent | so I reckon we’ve got at least two options: a) implement it b) decide that actually we never wanted that on resource providers anyway, that shared thing is sort of magical any way for “listing” resource providers, so doesn’t belong there | |
| 16:59:48 | cdent | leakypipes: I think I actually prefer option b for that url. you? | |
| 17:01:40 | leakypipes | cdent: not sure I follow you... GET /resource_providers is intended to return a list of the resource providers that can accomodate the requested resources, and in this case, if one of the requested resources is a shared resource, we naturally need to account for sharing providers (we just don't need to return information that indicates a sharing provider would be used) | |
| 17:04:21 | cdent | What I’m saying is that maybe (now that we have /allocation_candidates) listing /resource_providers should just do that: list resource providers that match queries, where resources is just another type of query. the concept of a shared resource isn’t necessarily a concept that is “natural” when request a listing of resource provider for no specific purpose | |
| 17:04:42 | cdent | (i’m not sold on this, just throwing it out there for thinking) | |
| 17:06:42 | leakypipes | cdent: I'm not actually sure what you're advocating :) | |
| 17:07:04 | cdent | i’m advocating leaving /resource_providers as it is right now | |
| 17:08:12 | cdent | leakypipes: or rather not advocating, wondering if it makes sense to do so | |
| 17:08:33 | leakypipes | cdent: I see. I don't suppose it would hurt to leave it as-is for a little while as we decide on that question. | |
| 17:11:04 | cdent | figleaf: you have any thoughts on the last few lines? | |
| 17:27:13 | cdent | have fun stephenfin | |
| 17:35:11 | mriedem | maybe we should put a dumb retry in the functional osapi client fixture if we get a 401 or 403 response for now? | |
| 17:35:20 | mriedem | as a hack until we're to RC1? | |
| 17:38:18 | melwitt | yeah, I forgot about that idea | |
| 17:41:27 | melwitt | I don't understand what the right way to fix this would be | |
| 17:42:14 | mriedem | rm -rf nova/tests/functional | |
| 17:42:25 | melwitt | hah | |
| 17:42:38 | cdent | is the auth error the main offender at this point? | |
| 17:42:54 | mriedem | it's what i've been noticing after the global keepalive=False change | |
| 17:43:14 | cdent | common url, or multiple urls? | |
| 17:43:15 | mriedem | which is odd since we use the noauth middleware i thought in all of the osapi fixture tests | |
| 17:43:27 | melwitt | yeah. I don't get it | |
| 17:43:35 | mriedem | i'll find the last one i just rechecked | |
| 17:44:03 | mriedem | http://logs.openstack.org/11/485011/4/gate/gate-nova-tox-functional-py35-ubuntu-xenial/42d69de/console.html#_2017-07-21_14_55_31_532907 | |
| 17:45:25 | mriedem | i've been seeing these in the functional jobs too | |
| 17:45:26 | mriedem | sys:1: ResourceWarning: unclosed file <_io.FileIO name=1 mode='wb' closefd=True> | |
| 17:46:41 | cdent | mriedem: is that on both py27 and py35 or or just py35? | |
| 17:47:15 | mriedem | only seeing it on py3 | |
| 17:47:29 | mriedem | py3 jobs are also way chattier about warnings | |
| 17:48:16 | melwitt | gdi looks like logstash.o.o is busted again | |
| 17:48:26 | melwitt | I wanted to check which jobs have "OpenStackApiAuthenticationException: Authentication error" | |
| 17:48:30 | cdent | I get unclosed file errors all over the place in lots of other things besides nova in py35 | |
| 17:48:35 | cdent | s/errors/warnings/ | |
| 17:48:57 | mriedem | so when we get that auth error we aren't providing the original response text | |
| 17:49:10 | mriedem | i had that working in the functional tests but leakypipes removed it | |
| 17:49:24 | mriedem | and then i raged | |
| 17:49:32 | leakypipes | hmm? | |
| 17:49:38 | mriedem | you know what you did | |
| 17:49:42 | mriedem | sec | |
| 17:49:53 | mriedem | https://github.com/openstack/nova/commit/de8096a59d80d10ff1ccf14e0b345be641ba4f07 | |
| 17:50:16 | mriedem | this is actually a bit different | |
| 17:50:23 | mriedem | do we have a bug for this anywhere? | |
| 17:50:45 | cdent | which this? | |
| 17:50:53 | mriedem | the random auth failures in functional tests | |
| 17:51:23 | mriedem | i don't see one | |
| 17:51:34 | melwitt | mriedem: oh, sorry. I think I approved that. I thought bc the jobs were passing it wasn't needed anymore, I didn't know it was a thing to get more info | |
| 17:51:45 | mriedem | yeah it's for debug | |
| 17:52:25 | melwitt | I guess put it back, with a comment that says what it's for | |
| 17:52:36 | mriedem | i assume the "BaseException.message has been deprecated as of Python 2.6" was for something else | |
| 17:52:41 | mriedem | b/c i've seen that before too | |
| 17:52:50 | mriedem | it wouldn't actually help in what we're seeing here | |
| 17:52:54 | mriedem | so i'm going to push something else for that | |
| 17:52:55 | mriedem | it | |
| 17:52:55 | mriedem | push | |
| 17:52:56 | mriedem | real | |
| 17:53:10 | melwitt | I thought that's what was causing those warnings was setting of the message attribute | |
| 17:57:26 | mriedem | i don't remember anymore, i added it here https://github.com/openstack/nova/commit/01dd1a05a213c0cbd0097188418cabe915291c8d | |
| 17:57:33 | mriedem | anywho, not the issue here | |
| 17:58:06 | leakypipes | mriedem: this tempest.api.identity.admin.v3.test_users.UsersV3TestJSON.test_password_history_not_enforced_in_admin_reset failure... grrr. | |
| 17:59:12 | mriedem | i think we have a signature for that one | |
| 17:59:22 | mriedem | http://status.openstack.org/elastic-recheck/#1702211 ? | |
| 17:59:42 | mriedem | cdent: melwitt: https://bugs.launchpad.net/nova/+bug/1705753 | |
| 17:59:43 | openstack | Launchpad bug 1705753 in OpenStack Compute (nova) "Random OpenStackApiAuthenticationException: Authentication error in nova functional tests" [Undecided,New] | |
| 18:01:08 | cdent | thanks, mriedem | |
| 18:01:28 | mriedem | i'll push a debug patch | |
| 18:02:38 | cdent | mriedem: do you want me to try the retry on auth fail thingie? | |
| 18:03:40 | mriedem | sure | |
| 18:03:53 | cdent | it seems like it might of some use but doesn’t really get at whatever the issue is, sadly | |
| 18:03:57 | cdent | but yeah, i’ll make one go | |
| 18:04:00 | melwitt | I wonder if this is relevant https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1436-L1438 | |
| 18:04:32 | mriedem | probably | |
| 18:04:47 | mriedem | the spike in failures started when placement fixture was turned on globally in the IntegratedHelpers mixin | |
| 18:05:35 | melwitt | yeah, that lines up with the fact that "The current placement NoAuthMiddleware returns a 401 in case a token is not provided" | |
| 18:05:40 | melwitt | I just don't know what that means | |
| 18:06:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L32 | |
| 18:06:26 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L41 | |
| 18:07:13 | cdent | if that’s playing a part, then it is likely that the problem is when compute requests land on the placement api (which seems to be the core problem here) | |
| 18:07:14 | melwitt | so does that imply that a request is being made that does send the x-auth-token header? is there anything other than GET/PUT/DELETE/POST? | |
| 18:07:22 | melwitt | *does not | |
| 18:07:38 | melwitt | oh. compute requests landing on placement api | |
| 18:07:53 | mriedem | i think there is an eventlet switch that goofs things up | |
| 18:07:55 | mriedem | or that's the theory | |
| 18:08:17 | melwitt | I guess I don't understand that | |
| 18:08:30 | mriedem | this is what compute does https://github.com/openstack/nova/blob/master/nova/api/openstack/auth.py#L32 | |
| 18:09:13 | cdent | placement does what it does to behave like a normal auth middleware and not fake more than it should, it basically stripped that middleware back to the basics | |