| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-09 | |||
| 18:35:09 | noonedeadpunk | but probably possible to come up with some regex | |
| 18:35:47 | sean-k-mooney | you shoudl be able to have the websoceket be wss://<domain>/novnc?token=... | |
| 18:36:54 | noonedeadpunk | so I just got `https://domain.com/console/vnc_lite.html` loading but still wss was just wss:// |
|
| 18:37:17 | noonedeadpunk | so it ended up with "something went wrong" | |
| 18:37:35 | sean-k-mooney | right but the websocket is not beeing writhe properly | |
| 18:38:07 | sean-k-mooney | you said the url looked correct form nova right | |
| 18:38:23 | sean-k-mooney | do you have an exmaple of the consol url returned | |
| 18:38:29 | noonedeadpunk | yes, but what does return wss URL? | |
| 18:39:02 | sean-k-mooney | at least for the serial console oepnstack server console show returns it | |
| 18:39:22 | noonedeadpunk | openstack console url show returns `https://domain.com/console/vnc_lite.html?path=%3Ftoken%3D76a1303f-c40c-4169-912d-19c05432dbc5` is what is returned | |
| 18:39:25 | sean-k-mooney | but i think we get https back by defualt for novnc | |
| 18:39:30 | noonedeadpunk | it loads, vnc console renders | |
| 18:40:00 | sean-k-mooney | so with devstack the dash board load at "http://192.168.16.127/dashboard/auth/login/?next=/dashboard/" | |
| 18:40:10 | noonedeadpunk | but then html5 tries to connect to wss on `wss://domain.com/?token=76a1303f-c40c-4169-912d-19c05432dbc5` | |
| 18:41:05 | noonedeadpunk | ok, I gues I'm jsut confused where this wss path is coming from | |
| 18:41:14 | noonedeadpunk | as indeed it's probably not from nova at all | |
| 18:41:30 | sean-k-mooney | its part of the html content provided to the javascript app | |
| 18:43:16 | sean-k-mooney | so in my case it returns http://192.168.16.127:6080/vnc_lite.html?path=%3Ftoken%3Dd8a68169-dbd6-4333-8b4a-f57bc3beb53c by edfault | |
| 18:43:44 | sean-k-mooney | if i open that direclty that is what nova is serving and what ironic is embding in an iframe | |
| 18:43:55 | noonedeadpunk | yup, true, and that's working as well (if clients don't have 6080 firewalled) | |
| 18:44:10 | noonedeadpunk | byt heir isp | |
| 18:44:26 | noonedeadpunk | ok, I'll try to do some rewrites indeed. | |
| 18:44:53 | noonedeadpunk | and look one more time what does construct wss uri | |
| 18:45:15 | sean-k-mooney | tehre are two path to how htis works. we use websoxkify to wrap th raw tcp connectoin to the vnc console create by qemu | |
| 18:45:41 | sean-k-mooney | and we use novnc ot provide the webpage that conenct to that and provde the vnc clinet | |
| 18:45:49 | noonedeadpunk | as htmlk5 console does have `const path = readQueryVariable('path', 'websockify');` | |
| 18:46:36 | noonedeadpunk | and it does `url += '/' + path;` - that's why I kind of complained that smth not passing html5 all details for it to perform | |
| 18:47:27 | sean-k-mooney | ok but that novnc that doing that | |
| 18:47:32 | sean-k-mooney | not nova | |
| 18:47:55 | sean-k-mooney | there might eb some header you can set like x_forwared_for or similar | |
| 18:48:35 | sean-k-mooney | in my case the path query arge is empty | |
| 18:48:45 | noonedeadpunk | I think it's just because path being used to pass token | |
| 18:48:49 | sean-k-mooney | well technialy path=%3F | |
| 18:48:58 | noonedeadpunk | ah | |
| 18:49:00 | noonedeadpunk | true | |
| 18:49:20 | noonedeadpunk | but it's what nova replies then | |
| 18:49:54 | sean-k-mooney | you may be able to add it to novncproxy_base_url | |
| 18:50:31 | sean-k-mooney | so instead of http://127.0.0.1:6080/vnc_auto.html | |
| 18:50:59 | sean-k-mooney | http://domain/subpath/vnc_auto.html?path=/subpath | |
| 18:51:55 | sean-k-mooney | can you try setting that in your broser and see if setting path=/subpath is enouch | |
| 18:51:58 | noonedeadpunk | hm, I somehow don't think that will work? | |
| 18:52:09 | noonedeadpunk | yeah, I'm trying now | |
| 18:52:24 | noonedeadpunk | in terms won't work - regarding defining novncproxy_base_url that way | |
| 18:52:28 | sean-k-mooney | well its being used by vnc_lite.html | |
| 18:52:32 | sean-k-mooney | whic is part of novnc | |
| 18:52:57 | sean-k-mooney | nova might not allow you pas qurey sting arge in baseurl | |
| 18:53:03 | sean-k-mooney | but it could be added by the proxy | |
| 18:53:16 | sean-k-mooney | *reverse_proxy when rewriteing | |
| 18:53:31 | sean-k-mooney | we coudl maybe add a cofnig option for that if it works | |
| 18:54:24 | sean-k-mooney | based on https://github.com/novnc/noVNC/blob/master/vnc_lite.html#L134-L153 | |
| 18:54:28 | sean-k-mooney | i dont see why it woudl not | |
| 18:54:51 | noonedeadpunk | yeah, adding manually shiould work | |
| 18:55:03 | noonedeadpunk | what probably wont - adjustin novncproxy_base_url as you said | |
| 18:55:29 | sean-k-mooney | ya we likely dont have logic to parse the query arges and append them | |
| 18:55:31 | noonedeadpunk | I don't think that nova gonna respect already defined path and wont'd add default path=%3F | |
| 18:55:34 | sean-k-mooney | but should be doable exterally | |
| 18:56:05 | noonedeadpunk | yeah, this should be not hard to do indeed | |
| 18:56:21 | noonedeadpunk | (I'd say would make sense for config option though) | |
| 18:56:51 | noonedeadpunk | or indeed parse for the option | |
| 19:00:29 | sean-k-mooney | the other thing you said is there isp blockes 6080 | |
| 19:00:47 | sean-k-mooney | we do hwave supprot for using alternitive ports alredy | |
| 19:00:58 | sean-k-mooney | so that might be an option in that case | |
| 19:01:46 | sean-k-mooney | it would be a pretty minor feature to add novncproxy_extra_query_args | |
| 19:05:45 | noonedeadpunk | ok, so `https://domain.com/console/vnc_lite.html?path=console%3Ftoken%3Dd7bf7a47-f081-4500-a961-05ddb7994e13` | |
| 19:05:48 | noonedeadpunk | does work indeed | |
| 19:07:05 | noonedeadpunk | so thanks a ton for your time | |
| 19:07:45 | noonedeadpunk | I know I was annoying but quite happy now as you said - it should be trivial both to rewrite and implement a "native" way | |
| 19:08:13 | sean-k-mooney | your bringing a pain point to our attention that not annoying | |
| 19:08:24 | sean-k-mooney | i still think subdomains are preferable | |
| 19:08:43 | sean-k-mooney | but if someone wants to add this minor feature im not agaisnt it | |
| 19:08:53 | sean-k-mooney | i just rpaobly wont go implemtnet it in my spare time | |
| 19:10:01 | sean-k-mooney | noonedeadpunk: you may have hit https://github.com/haproxy/haproxy/issues/829 by the way | |
| 19:12:11 | sean-k-mooney | there is a set-pathq to set items in teh query string now | |
| 19:14:13 | sean-k-mooney | i guess that migh now be https://www.haproxy.com/documentation/haproxy-configuration-tutorials/http-rewrites/#set-the-query-string | |
| 19:15:37 | sean-k-mooney | so i think you can jsut add "http-request set-query path=/subpath" | |
| 19:16:31 | sean-k-mooney | noonedeadpunk: im going to call it a day. if that works you could porpsoe a docs update if you were so inclidned or a devstack path | |
| 19:16:44 | sean-k-mooney | *patch | |
| 19:20:01 | noonedeadpunk | ++ thanks for the help as usual! | |
| 19:30:15 | noonedeadpunk | yeah, so here it goes: https://opendev.org/openstack/nova/src/branch/master/nova/objects/console_auth_token.py#L78 | |
| 19:49:53 | samcat116 | Hi all, whats a good place to start debugging the GET /servers api being slow? Running `openstack server list --all-projects --no-name-lookup` on a stack with about 1500 instances takes over 48s to return, with the GET https://example.com:8774/v2.1/servers/detail?deleted=False&all_tenants=True call taking over 30s itself. CPU usage across our 3 controllers is around 50% and I'm running with 10 api_workers, so don't think its a raw | |
| 19:49:54 | samcat116 | horsepower issue | |
| 22:10:54 | sean-k-mooney | samcat116: the server detail list need to hit neutron for each server to lookup secuirty groups, but openstack client is doing more then jsut /v2.1/servers/detail?deleted=False&all_tenants=Tr | |
| 22:10:57 | sean-k-mooney | | ue | |
| 22:12:40 | sean-k-mooney | that specific query however does require conenctin to all cell db and pulling a lot of info so its one of the more expensive api calls you can make to nova | |
| 22:13:05 | sean-k-mooney | its not as expensieve as simple tenant usage but it s more expensive then just about any other read only call | |
| 22:19:20 | samcat116 | Ok I guess I’ll look on the neutron side as we are pretty port and network heavy as well | |
| 22:19:37 | samcat116 | I just don’t know why it would be that slow | |
| 22:20:08 | sean-k-mooney | it might not be related to that but its on eof tthe peices of data we lookup form other services | |
| 22:20:23 | sean-k-mooney | we cache alotof the info reatled to other services but not all | |
| 22:20:56 | sean-k-mooney | tehre are some profiling tools liek os-profiler | |
| 22:21:06 | sean-k-mooney | but im not famialr with who to properly use them | |
| 22:21:45 | samcat116 | Ok. I do have memcache all setup so I would assume this could be heavily cached | |
| 22:22:01 | sean-k-mooney | no | |
| 22:22:17 | sean-k-mooney | i think we have almsot no caching for this | |
| 22:22:38 | sean-k-mooney | we use memcache very sparingly | |
| 22:23:31 | sean-k-mooney | its main use in nova is for the metadata api | |
| 22:24:15 | sean-k-mooney | we generally dont cache api responces | |
| 22:25:30 | sean-k-mooney | the other main use of memcafhe is for keytone auth | |
| 22:27:47 | samcat116 | Good to know | |
| 22:30:08 | samcat116 | How far off of an estimate could this be. Just trying to figure out if this is an order of magnitude slower than it should be or not | |