| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-09 | |||
| 18:44:10 | noonedeadpunk | byt heir isp | |
| 18:44:26 | noonedeadpunk | ok, I'll try to do some rewrites indeed. | |
| 18:44:53 | noonedeadpunk | and look one more time what does construct wss uri | |
| 18:45:15 | sean-k-mooney | tehre are two path to how htis works. we use websoxkify to wrap th raw tcp connectoin to the vnc console create by qemu | |
| 18:45:41 | sean-k-mooney | and we use novnc ot provide the webpage that conenct to that and provde the vnc clinet | |
| 18:45:49 | noonedeadpunk | as htmlk5 console does have `const path = readQueryVariable('path', 'websockify');` | |
| 18:46:36 | noonedeadpunk | and it does `url += '/' + path;` - that's why I kind of complained that smth not passing html5 all details for it to perform | |
| 18:47:27 | sean-k-mooney | ok but that novnc that doing that | |
| 18:47:32 | sean-k-mooney | not nova | |
| 18:47:55 | sean-k-mooney | there might eb some header you can set like x_forwared_for or similar | |
| 18:48:35 | sean-k-mooney | in my case the path query arge is empty | |
| 18:48:45 | noonedeadpunk | I think it's just because path being used to pass token | |
| 18:48:49 | sean-k-mooney | well technialy path=%3F | |
| 18:48:58 | noonedeadpunk | ah | |
| 18:49:00 | noonedeadpunk | true | |
| 18:49:20 | noonedeadpunk | but it's what nova replies then | |
| 18:49:54 | sean-k-mooney | you may be able to add it to novncproxy_base_url | |
| 18:50:31 | sean-k-mooney | so instead of http://127.0.0.1:6080/vnc_auto.html | |
| 18:50:59 | sean-k-mooney | http://domain/subpath/vnc_auto.html?path=/subpath | |
| 18:51:55 | sean-k-mooney | can you try setting that in your broser and see if setting path=/subpath is enouch | |
| 18:51:58 | noonedeadpunk | hm, I somehow don't think that will work? | |
| 18:52:09 | noonedeadpunk | yeah, I'm trying now | |
| 18:52:24 | noonedeadpunk | in terms won't work - regarding defining novncproxy_base_url that way | |
| 18:52:28 | sean-k-mooney | well its being used by vnc_lite.html | |
| 18:52:32 | sean-k-mooney | whic is part of novnc | |
| 18:52:57 | sean-k-mooney | nova might not allow you pas qurey sting arge in baseurl | |
| 18:53:03 | sean-k-mooney | but it could be added by the proxy | |
| 18:53:16 | sean-k-mooney | *reverse_proxy when rewriteing | |
| 18:53:31 | sean-k-mooney | we coudl maybe add a cofnig option for that if it works | |
| 18:54:24 | sean-k-mooney | based on https://github.com/novnc/noVNC/blob/master/vnc_lite.html#L134-L153 | |
| 18:54:28 | sean-k-mooney | i dont see why it woudl not | |
| 18:54:51 | noonedeadpunk | yeah, adding manually shiould work | |
| 18:55:03 | noonedeadpunk | what probably wont - adjustin novncproxy_base_url as you said | |
| 18:55:29 | sean-k-mooney | ya we likely dont have logic to parse the query arges and append them | |
| 18:55:31 | noonedeadpunk | I don't think that nova gonna respect already defined path and wont'd add default path=%3F | |
| 18:55:34 | sean-k-mooney | but should be doable exterally | |
| 18:56:05 | noonedeadpunk | yeah, this should be not hard to do indeed | |
| 18:56:21 | noonedeadpunk | (I'd say would make sense for config option though) | |
| 18:56:51 | noonedeadpunk | or indeed parse for the option | |
| 19:00:29 | sean-k-mooney | the other thing you said is there isp blockes 6080 | |
| 19:00:47 | sean-k-mooney | we do hwave supprot for using alternitive ports alredy | |
| 19:00:58 | sean-k-mooney | so that might be an option in that case | |
| 19:01:46 | sean-k-mooney | it would be a pretty minor feature to add novncproxy_extra_query_args | |
| 19:05:45 | noonedeadpunk | ok, so `https://domain.com/console/vnc_lite.html?path=console%3Ftoken%3Dd7bf7a47-f081-4500-a961-05ddb7994e13` | |
| 19:05:48 | noonedeadpunk | does work indeed | |
| 19:07:05 | noonedeadpunk | so thanks a ton for your time | |
| 19:07:45 | noonedeadpunk | I know I was annoying but quite happy now as you said - it should be trivial both to rewrite and implement a "native" way | |
| 19:08:13 | sean-k-mooney | your bringing a pain point to our attention that not annoying | |
| 19:08:24 | sean-k-mooney | i still think subdomains are preferable | |
| 19:08:43 | sean-k-mooney | but if someone wants to add this minor feature im not agaisnt it | |
| 19:08:53 | sean-k-mooney | i just rpaobly wont go implemtnet it in my spare time | |
| 19:10:01 | sean-k-mooney | noonedeadpunk: you may have hit https://github.com/haproxy/haproxy/issues/829 by the way | |
| 19:12:11 | sean-k-mooney | there is a set-pathq to set items in teh query string now | |
| 19:14:13 | sean-k-mooney | i guess that migh now be https://www.haproxy.com/documentation/haproxy-configuration-tutorials/http-rewrites/#set-the-query-string | |
| 19:15:37 | sean-k-mooney | so i think you can jsut add "http-request set-query path=/subpath" | |
| 19:16:31 | sean-k-mooney | noonedeadpunk: im going to call it a day. if that works you could porpsoe a docs update if you were so inclidned or a devstack path | |
| 19:16:44 | sean-k-mooney | *patch | |
| 19:20:01 | noonedeadpunk | ++ thanks for the help as usual! | |
| 19:30:15 | noonedeadpunk | yeah, so here it goes: https://opendev.org/openstack/nova/src/branch/master/nova/objects/console_auth_token.py#L78 | |
| 19:49:53 | samcat116 | Hi all, whats a good place to start debugging the GET /servers api being slow? Running `openstack server list --all-projects --no-name-lookup` on a stack with about 1500 instances takes over 48s to return, with the GET https://example.com:8774/v2.1/servers/detail?deleted=False&all_tenants=True call taking over 30s itself. CPU usage across our 3 controllers is around 50% and I'm running with 10 api_workers, so don't think its a raw | |
| 19:49:54 | samcat116 | horsepower issue | |
| 22:10:54 | sean-k-mooney | samcat116: the server detail list need to hit neutron for each server to lookup secuirty groups, but openstack client is doing more then jsut /v2.1/servers/detail?deleted=False&all_tenants=Tr | |
| 22:10:57 | sean-k-mooney | | ue | |
| 22:12:40 | sean-k-mooney | that specific query however does require conenctin to all cell db and pulling a lot of info so its one of the more expensive api calls you can make to nova | |
| 22:13:05 | sean-k-mooney | its not as expensieve as simple tenant usage but it s more expensive then just about any other read only call | |
| 22:19:20 | samcat116 | Ok I guess I’ll look on the neutron side as we are pretty port and network heavy as well | |
| 22:19:37 | samcat116 | I just don’t know why it would be that slow | |
| 22:20:08 | sean-k-mooney | it might not be related to that but its on eof tthe peices of data we lookup form other services | |
| 22:20:23 | sean-k-mooney | we cache alotof the info reatled to other services but not all | |
| 22:20:56 | sean-k-mooney | tehre are some profiling tools liek os-profiler | |
| 22:21:06 | sean-k-mooney | but im not famialr with who to properly use them | |
| 22:21:45 | samcat116 | Ok. I do have memcache all setup so I would assume this could be heavily cached | |
| 22:22:01 | sean-k-mooney | no | |
| 22:22:17 | sean-k-mooney | i think we have almsot no caching for this | |
| 22:22:38 | sean-k-mooney | we use memcache very sparingly | |
| 22:23:31 | sean-k-mooney | its main use in nova is for the metadata api | |
| 22:24:15 | sean-k-mooney | we generally dont cache api responces | |
| 22:25:30 | sean-k-mooney | the other main use of memcafhe is for keytone auth | |
| 22:27:47 | samcat116 | Good to know | |
| 22:30:08 | samcat116 | How far off of an estimate could this be. Just trying to figure out if this is an order of magnitude slower than it should be or not | |
| 22:30:41 | sean-k-mooney | that kind of depend on other factors like how many cells you have | |
| 22:30:53 | sean-k-mooney | you not the first person to ask about this | |
| 22:31:13 | sean-k-mooney | samcat116: can you try making the same query to /server instead of /server/detials | |
| 22:31:47 | sean-k-mooney | the delta is /server just hits the api db and list all the server we know about without going to the cell dbs or calling other services | |
| 22:32:17 | sean-k-mooney | that should be very fast as it basicaly a select instance_uuid from istance_mappings | |
| 22:35:01 | sean-k-mooney | if /servers/ is fast and /servers/detail is slow then one thing to check is if any of the cells are slow to respond | |
| 22:35:31 | sean-k-mooney | internally the api when repondign ot /servers/detail is calling into each cell db and geting the data form the instances table | |
| 22:38:06 | sean-k-mooney | the security group info i thin need a call to neutron for every port | |
| 22:38:41 | samcat116 | Ok I’ll try that. We only have one cell | |
| 22:39:09 | sean-k-mooney | that should simplfy thing then as the api is not making parralel requests | |
| 22:41:07 | sean-k-mooney | i think the security gorups are the only thing not cached in that responce. i.e. that is not just a direct db lookup | |
| 22:42:39 | sean-k-mooney | this has been know for a long time https://bugs.launchpad.net/nova/+bug/1923560 and a fix was prospoed here https://review.opendev.org/c/openstack/nova/+/786348 | |
| 22:43:39 | sean-k-mooney | i was personaly on the side of removing the security groups form the instance obejct entirly | |
| 22:43:46 | sean-k-mooney | btu if we dont then we shoudl cache them | |
| #openstack-nova - 2024-09-10 | |||
| 00:04:27 | samcat116 | So /server?all_tenants=True vs /servers/detail?all_tenants=True was 1.3s vs 26s | |
| 00:04:33 | samcat116 | s/server/servers/, s/all_tenants/all\_tenants/, s/all_tenants/all\_tenants/ | |
| 00:18:06 | sean-k-mooney | we woudl expect there to be a delta but not over an order of magnitde. | |
| 07:04:49 | bauzas | because we don't call the cells DB for the first call :) | |
| 07:08:37 | opendevreview | Tobias Urdin proposed openstack/nova master: api-ref: Add new lines not allowed for server description https://review.opendev.org/c/openstack/nova/+/928758 | |
| 09:06:50 | bauzas | well, looking at the relnotes, the cycle highlights will be quite simple to write... | |