| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-04 | |||
| 18:04:58 | mriedem | gah, plus i screwed up the topic branch | |
| 18:05:32 | efried | mriedem mordred jaypipes [service catalog bp] I'd like to settle this issue of determining interface if not otherwise specified. | |
| 18:05:43 | mriedem | pistols at dawn it is | |
| 18:05:56 | efried | At some point in the early (at least in my involvement) discussions, there was talk of trying the known ones in order. | |
| 18:06:02 | efried | This is a dict lookup, so that's cheap. | |
| 18:06:14 | mriedem | maybe the idea with starting using internal first is to avoid defaulting to public in ksa? | |
| 18:06:16 | efried | But on the other side, there's "use the default from ksa" | |
| 18:06:30 | efried | ...which is "public", for whatever that's worth. | |
| 18:06:44 | efried | Can any of you remember the rationale behind the "loop through 'em" suggestion? | |
| 18:07:04 | mriedem | efried: that was mordred or jaypipes' idea i thought, but presumably to hit the internal/admin urls before public | |
| 18:07:09 | efried | mriedem Yeah, starting with "internal" seemed like a more... uh... "secure" idea. | |
| 18:07:15 | mriedem | since today ops are probably using internal/admin | |
| 18:08:08 | efried | Okay, right now I've got internal->public->admin; that's an easy swap. But question remains: do we want to do the loop thing at all? | |
| 18:09:10 | mriedem | i don't really feel strongly either way, looping over a hard-coded list which might not match what's actually in the service catalog seems weird to me | |
| 18:09:52 | efried | mriedem Yeah, that makes me a tad uncomfortable too. Maybe mordred can point us to a way to get that list from ksa. | |
| 18:10:51 | mriedem | https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identity/base.py#L173 | |
| 18:11:01 | mriedem | the docs say one thing but i'm pretty sure interface is open ended | |
| 18:11:23 | mriedem | although at one point i thought i found json schema in the keystone api for v3 that restricted the interfaces | |
| 18:11:24 | mriedem | to a known set | |
| 18:13:06 | efried | One approach is to say, we try these three, and if it ain't one of those, they better specify it. | |
| 18:13:10 | efried | They can always specify it. | |
| 18:14:14 | mriedem | efried: https://github.com/openstack/keystone/blob/master/keystone/catalog/schema.py#L94 | |
| 18:14:47 | efried | noyce. | |
| 18:14:47 | mriedem | so the interface types are validated via jsonschema in the keystone api, at least for the v3 api i think | |
| 18:15:02 | mriedem | so iterating through admin/internal/public probably seems fine | |
| 18:15:45 | efried | What order would you like? | |
| 18:16:06 | mriedem | don't ask me :) | |
| 18:16:08 | mriedem | ask operators | |
| 18:16:13 | mriedem | #openstack-operators | |
| 18:17:01 | abhi89 | dansmith, jaypipes: please review https://review.openstack.org/#/c/462561/.. its for stable/ocata merge.. | |
| 18:43:48 | mriedem | melwitt: looks like we have a KeyError when the tests run now because of ^ https://review.openstack.org/#/c/460305/6/nova/tests/unit/compute/test_compute.py | |
| 18:47:07 | mriedem | https://bugs.launchpad.net/nova/+bug/1688366 | |
| 18:47:07 | openstack | Launchpad bug 1688366 in OpenStack Compute (nova) "KeyError: u'overs' in unit test runs" [Low,Triaged] | |
| 18:47:28 | melwitt | mriedem: sigh ... I miss that config option that used to make format errors fatal for tests | |
| 18:47:42 | mriedem | i don't know why that was removed | |
| 18:47:50 | mriedem | maybe it was too useful | |
| 18:48:38 | melwitt | yeah, really. it was removed as part of the centralize-config-options work, and I tried to recreate the behavior with wonky monkey patching and wasn't successful https://review.openstack.org/#/c/411615 | |
| 18:52:44 | dansmith | 184489 | |
| 18:52:50 | dansmith | heh sorry | |
| 18:53:10 | melwitt | there's been something in the unit tests spewing garbled text into the output for months now, I assume that's another KeyError some place in a test exception | |
| 18:56:37 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Updates for [glance]api_servers and ksa https://review.openstack.org/461481 | |
| 18:56:47 | efried | mriedem ^^ | |
| 19:05:29 | openstackgerrit | Steve Noyes proposed openstack/nova master: Add Cinder v3 detach to shutdown_instance https://review.openstack.org/456877 | |
| 19:08:20 | dansmith | melwitt: I wonder if I'm failing this test because of legit lack of cells support in consoleauth? http://logs.openstack.org/37/458537/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fe108a8/logs/testr_results.html.gz | |
| 19:12:45 | melwitt | dansmith: I would guess so. looking to see if we do a target cell for getting the vnc console | |
| 19:13:19 | melwitt | I can't remember what the state of the compute rpcapi mq switching is | |
| 19:13:44 | dansmith | yeah, I think it might actually be in the proxy thing | |
| 19:16:40 | melwitt | in nova/api/openstack/compute/remote_consoles.py it gets an instance and passes it down to get_vnc_console, so if the context has already been targeted by the get_instance, it should work | |
| 19:17:52 | dansmith | melwitt: right, but that's to get the url, right? then we actually connect through the novncproxy to get to the rfb data, | |
| 19:18:01 | dansmith | and it has to know where to proxy to, which is what is failing I think | |
| 19:18:20 | dansmith | it makes a consoleauth rpc call, which may be going to the wrong place, I'm guessing | |
| 19:18:21 | melwitt | oh, yeah. must be | |
| 19:18:44 | dansmith | although I don't see anything barfing about an rpc timeout or anything | |
| 19:19:29 | melwitt | yeah, dtp's change made the consoleauth calls stay at the API and that call shouldn't try to mq switch | |
| 19:20:58 | melwitt | I can never remember how the consoles work. get the url first, now looking for where it does the consoleauth part | |
| 19:21:08 | dansmith | yeah I have to re-learn each time | |
| 19:22:18 | melwitt | okay, the consoleauth is in compute/api, in get_vnc_console, so that should be fine I think | |
| 19:23:04 | dansmith | that's the thing that authorizes, not what checks right? | |
| 19:23:22 | dansmith | yeah | |
| 19:23:31 | dansmith | so we go there, create a token, return a url | |
| 19:23:42 | dansmith | then we connect to the proxy, do a websocket and expect rfb data | |
| 19:23:54 | dansmith | but we have to check the token in there again, and we do a rpc call to do it | |
| 19:24:12 | dansmith | https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L105-L106 | |
| 19:25:00 | melwitt | okay, proxy is running at the API, so is consoleauth, seems like it should be okay | |
| 19:25:20 | dansmith | but the tokens are in the cell db, no? | |
| 19:25:57 | melwitt | tokens are in memcached | |
| 19:26:49 | melwitt | why are there no errors in the logs anywhere :| | |
| 19:27:01 | dansmith | oh, heh, and so the consoleauth service is just there to ask memcache for us for checking tokens? | |
| 19:27:04 | dansmith | yeah, that's the weird thing | |
| 19:27:49 | melwitt | that's my understanding, yeah | |
| 19:28:19 | dansmith | yeah, looks like it | |
| 19:28:19 | dansmith | wtf | |
| 19:28:25 | dansmith | oh no, | |
| 19:28:28 | dansmith | it also calls to compute | |
| 19:28:55 | dansmith | to validate a port | |
| 19:29:06 | dansmith | but it does the mapping lookup and the target | |
| 19:29:16 | dansmith | so yeah seems like maybe no logs because it's mostly working | |
| 19:29:47 | melwitt | I dunno if this is relevant but I don't see a log in consoleauth for the test you're failing http://logs.openstack.org/37/458537/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fe108a8/logs/screen-n-cauth.txt.gz?#_May_04_18_01_45_091782 | |
| 19:30:44 | dansmith | I do see an authorize console report though | |
| 19:31:06 | dansmith | not for that test though yeah | |
| 19:31:25 | melwitt | yeah, based on the project/user names from the context | |
| 19:33:04 | dansmith | so, I see the corresponding things in n-api.log for that server actions test, | |
| 19:33:09 | dansmith | but nothing for the vnc test | |
| 19:33:15 | dansmith | dub tee eff | |
| 19:34:03 | melwitt | I can't even find that server uuid in n-api log | |
| 19:34:09 | dansmith | Body: {"console": {"url": "http://198.72.124.114:6080/vnc_auto.html?token=8bfe6807-a8be-44de-bca7-d9087fb2da4a", "type": "novnc"}} | |
| 19:34:15 | dansmith | sure looks like it was called by tempest though | |
| 19:34:51 | dansmith | hmm, yeah that's super bizarre | |
| 19:34:57 | melwitt | yeah, like, wtf | |
| 19:34:59 | dansmith | that server uuid is nowhere in the logs | |
| 19:35:31 | dansmith | this finished at noon, so it has to be without the journal usage | |
| 19:35:41 | melwitt | I found it a few times in n-cpu, but nothing about it spawning | |
| 19:36:16 | melwitt | just accounting stuff | |
| 19:36:44 | dansmith | o.O | |
| 19:36:54 | dansmith | I also noticed what looked like a few restarts of the api service at the top of the log | |
| 19:37:08 | dansmith | like in the tripleo logs, but that's because of puppet I think | |
| 19:37:19 | dansmith | so I wonder if we have an api worker that isn't actually logging | |
| 19:37:42 | bauzas | dansmith: edleafe: jaypipes: FWIW, I was writing some UTs for the scheduler-claims | |
| 19:38:11 | dansmith | mriedem: maybe those are the two api worker threads starting actually | |
| 19:38:27 | jaypipes | bauzas: cool. will review promptly when you push. | |