| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-01 | |||
| 18:12:23 | melwitt | what do you mean by "cleanup" of remaining things? | |
| 18:14:32 | mnaser | melwitt: yeah I meant that if you didn't want things to be smooth and consoles not be invalidated | |
| 18:14:46 | mnaser | now curiously I have this other issue where the calculated token hash doesn't seem to match the one in the db.. | |
| 18:16:10 | mnaser | I get token id `dc574410-2c5c-4762-ae0d-f1689372337f` .. the sha256 is b7cc4998dea5dd0a3b663fcbe8df7c487246b3d3e52ec3361530b091477e5b28 which matches the db entry | |
| 18:16:28 | mnaser | but when I do a print(token_hash) in the novncproxy codebase.. I get e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
| 18:17:29 | melwitt | hm, that's strange. I'm not aware of anything that could have changed there | |
| 18:17:43 | mnaser | ok, looks like the function is being called with token == '' | |
| 18:19:20 | melwitt | hm... empty string should mean that the GET url didn't have the token in it | |
| 18:19:55 | mnaser | yeah url im hitting has a token, im following it up the stack, im up to _get_connect_info_database() and that gets a token == none | |
| 18:20:00 | melwitt | that's where the token comes from (GET request param or cookie) in the novncproxy | |
| 18:23:04 | melwitt | mnaser: https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L195 | |
| 18:23:26 | mnaser | melwitt: that gets called when we call /websockify which doesn't seem to have a token | |
| 18:23:34 | mnaser | print('path', self.path) in here shows | |
| 18:23:41 | mnaser | ('path', '/websockify') | |
| 18:25:10 | mnaser | oh crap, I have a dr appointment in 5 minutes gotta run, ill dig into this after if you don't keep looking | |
| 18:25:19 | mnaser | but looks like unless its not sending cookies with a token.. we can't do anything | |
| 18:25:34 | melwitt | ok. I'll keep looking. I'm not familiar with the /websockify path so I need to find what that is | |
| 18:26:47 | melwitt | because usually the path should be like this http://ip:port/?token=xyz | |
| 18:33:36 | mnaser | melwitt: indeed and xyz is there but it looks like the token lookup seems to happen in the websockify part | |
| 18:34:49 | mnaser | Weird thing is that code has been there for 5 years so.. | |
| 18:35:13 | mnaser | Unless the behaviour to forward token from cookie was removed with more recent novnc versions | |
| 18:35:29 | melwitt | yeah, I know. I'm puzzled as to what's going on here | |
| 18:35:55 | mnaser | We pull master novnc in OSA so mahbe its recent behaviour change | |
| 18:36:27 | melwitt | I'll take a look in novnc | |
| 18:37:03 | mnaser | Okay | |
| 18:37:11 | melwitt | I thought the cookie thing was a backup, if the token is not in the GET request params, but forwarding via cookie does sound really familiar | |
| 18:37:16 | mnaser | melwitt: we use vnc_lite.html in OSA. https://github.com/novnc/noVNC/commit/51f9f0098d306bbc67cc8e02ae547921b6f6585c | |
| 18:38:14 | melwitt | I had dug into this stuff back when I worked on the db backend for console auths, back when we were thinking we could pass the instance uuid in the GET request too (turns out bc of novnc limitations, more than one request param would not work) | |
| 18:38:50 | melwitt | (it only forwards the first request param) | |
| 18:39:05 | mnaser | So it always probably used a cookie but that seems to have been pulled out from vnc_lite.html which is for some historic reason what’s being used in OSA | |
| 18:40:14 | melwitt | huh, interesting | |
| 18:40:30 | mnaser | The worrying thing is I can’t find any token references in novnc | |
| 18:40:32 | melwitt | / If a token variable is passed in, set the parameter in a cookie. | |
| 18:40:33 | melwitt | / This is used by nova-novncproxy. | |
| 18:40:49 | mnaser | Where do you see this melwitt ? | |
| 18:40:51 | melwitt | seeing that all in red being removed. "used by nova-novncproxy" :P | |
| 18:41:03 | melwitt | mnaser: in the commit you linked https://github.com/novnc/noVNC/commit/51f9f0098d306bbc67cc8e02ae547921b6f6585c#diff-1d6838e3812778e95699b90d530543a1L173 | |
| 18:41:20 | mnaser | ugh | |
| 18:41:42 | mnaser | What does devstack use | |
| 18:42:04 | mnaser | I wonder how it hasn’t been affected | |
| 18:42:15 | melwitt | must not be using lite? | |
| 18:43:07 | melwitt | bc this was removed from the vnc_lite.html file, unless I'm missing something | |
| 18:43:45 | mnaser | Of course | |
| 18:43:58 | mnaser | NOVNC_FROM_PACKAGE defined in jobs | |
| 18:44:43 | melwitt | http://logs.openstack.org/65/627765/12/check/tempest-full-py3/b00b2ab/controller/logs/devstacklog.txt.gz#_2019-04-01_08_01_25_051 | |
| 18:45:17 | mnaser | Yikes well i guess we can ask novnc to revert | |
| 18:49:53 | melwitt | mnaser: I don't understand, how do you use lite vs not? | |
| 18:50:23 | mnaser | melwitt: vnc_lite seems to be what vnc_auto was a while ago | |
| 18:50:36 | mnaser | Searching the code doesn’t give any results for token | |
| 18:50:40 | melwitt | mnaser: and which one you use depends on the url you use? | |
| 18:51:05 | melwitt | yeah, ok. looks like we have to ask for a revert of at least that part | |
| 18:51:22 | mnaser | melwitt: I assume we’re using the lite which handled the token stuff and did the cookie handoff | |
| 18:51:45 | melwitt | mnaser: yeah, I'm just saying, how do you choose whether to use lite or not | |
| 18:51:56 | mnaser | dunno :P | |
| 18:52:01 | melwitt | haha ok | |
| 18:52:13 | mnaser | It seems like there’s something called token plugin | |
| 18:52:24 | melwitt | at first I was thinking, can we use non-lite but then you pointed out only lite was handling token | |
| 18:52:30 | melwitt | so my thought was moo | |
| 18:52:49 | mnaser | Don’t quote me on this stuff I’m going over it on my phone heh | |
| 18:53:30 | melwitt | :) | |
| 18:54:43 | melwitt | I do wonder why we have to rely on the cookie forwarding. I wonder why it doesn't just show up as a GET request param by the time we get it | |
| 18:54:59 | melwitt | from the way our code is written, it looks like the expectation was that the cookie was a fall back | |
| 18:55:14 | melwitt | but in practice, we were always falling back | |
| 18:56:41 | melwitt | going to see if I can find out | |
| 19:15:57 | openstackgerrit | Eric Fried proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | |
| 19:16:34 | openstackgerrit | Eric Fried proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | |
| 19:17:38 | openstackgerrit | Eric Fried proposed openstack/nova master: Mock time.sleep() in unit tests https://review.openstack.org/648762 | |
| 19:25:36 | mnaser | melwitt: so I think the problem here is the /websockify HTTP request comes afterwards | |
| 19:26:01 | mnaser | so its GET /vnc_lite.html?token=foobar .. and then CONNECT /websockify | |
| 19:26:12 | mnaser | so you can't really reliably tie that GET request and the CONNECT request | |
| 19:26:28 | mnaser | UNLESS we sent a Set-Cookie: token when you visit vnc_lite.html | |
| 19:26:36 | mnaser | that would workaround it | |
| 19:27:18 | melwitt | mnaser: I see. well, from some googling it looks like something like this might work? GET /vnc_lite.html?path=/websockify?token=foobar | |
| 19:28:19 | openstackgerrit | Eric Fried proposed openstack/nova master: Add placement as required project to functional py36 and 37 https://review.openstack.org/649068 | |
| 19:28:29 | melwitt | second ? should probably be a & | |
| 19:28:45 | mnaser | I think it would be cleaner to set-cookie | |
| 19:28:59 | melwitt | or maybe not | |
| 19:30:02 | melwitt | mnaser: ok. I can see if I can figure out how to do that, unless you want to | |
| 19:32:32 | mnaser | melwitt: im looking around right now, just need to figure out where the HTTP requests are handled in nova-novncconsole | |
| 19:32:35 | mnaser | ill welcome all help :P | |
| 19:32:44 | mnaser | *novncproxy | |
| 19:34:15 | mnaser | think it relies on upstream this https://github.com/novnc/websockify/blob/master/websockify/websocketproxy.py#L267 | |
| 19:34:26 | mnaser | self.token_plugin = kwargs.pop('token_plugin', None) | |
| 19:34:28 | mnaser | oh I wonder what that is | |
| 19:35:39 | melwitt | mnaser: I looked at that and it doesn't seem like something that would help us. but I could be wrong https://github.com/novnc/websockify/wiki/Token-based-target-selection | |
| 19:37:42 | mnaser | https://github.com/novnc/websockify/blob/master/websockify/websocketserver.py#L35-L52 | |
| 19:37:42 | mnaser | maybe we somehow override the html portion | |
| 19:37:48 | melwitt | mnaser: I think handling of HTTP (GET) requests might be this https://github.com/novnc/websockify/blob/6e09ec25486f431e2acc463f994c4eb4bebc6ddd/websockify/websockifyserver.py#L268 | |
| 19:37:57 | mnaser | oh that's even better | |
| 19:38:26 | melwitt | one of those, I don't really understand it yet | |
| 19:39:08 | efried | This one came across my desk again just now, can I please get a couple of pairs of core eyeballs on it? https://review.openstack.org/#/c/635533/ | |
| 19:41:10 | mnaser | I'm thinking https://github.com/novnc/websockify/wiki/Token-based-target-selection#custom-plugins | |
| 19:43:08 | melwitt | mnaser: hm. so instead of validating the token in new_websocket_client we'd do it in a new lookup method? | |
| 19:43:24 | melwitt | 'lookup' | |
| 19:43:29 | mnaser | melwitt: yeah, if we write a custom plugin, we can validate the token in it and then give it back to novnc | |
| 19:43:33 | efried | mriedem: This looks like a deadlock starting n-api? http://logs.openstack.org/36/649036/1/check/tempest-full-py3/a8c7223/controller/logs/screen-n-api.txt.gz | |
| 19:43:53 | melwitt | mnaser: it says lookup has to return a host:port tuple? | |
| 19:44:31 | mnaser | or none, none (if we don't validate I guess) | |
| 19:44:43 | efried | (maybe not, I guess the locks are being released quickly) | |
| 19:44:55 | mnaser | and the host:port tuple business can be grabbed from the db | |