| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-20 | |||
| 12:52:21 | belmoreira | Hi, I need your help to understand if I'm missing something in the new vnc configuration | |
| 12:52:29 | belmoreira | I can finally move nova to "train" release and I'm digging again into the vncproxy changes that were introduced by this time. The vncproxy does now the token validation from the cell DB. In stein I'm running with "workarounds/enable_consoleauth" | |
| 12:53:02 | belmoreira | Currently, I have the same vncproxy for all cells. Means that the user gets the same console_url and I only need to open 1 port in the firewall. Also, haproxy configuration is trivial | |
| 12:53:16 | belmoreira | With the new arch, the console_url needs to be redirected to the vncproxy of the cell for the token validation | |
| 12:53:42 | belmoreira | This means that deployments with a large number of cells need to be creative in the way they expose the different console_urls (per cell) | |
| 12:53:53 | belmoreira | maybe I'm missing something something here... | |
| 15:11:26 | bauzas | belmoreira: sorry, I saw your pings but I don't know how to help you | |
| 15:18:02 | belmoreira | Hi bauzas. Thanks, to me this new approach seems really heavy for deployments with a lot cells. For now I'm hacking something similar to [1] to not have a vncproxy per cell. | |
| 15:18:10 | belmoreira | [1] https://github.com/openstack/nova/blob/0bd61915ee1d96ca339f342a190e395a39afbcf9/nova/api/openstack/compute/console_auth_tokens.py#L42 | |
| 15:18:30 | belmoreira | maybe we can discuss this in the PTG | |
| 15:26:51 | dansmith | it seems strange to me that someone with lots of cells would want to *not* shard that service across cells | |
| 15:27:04 | dansmith | especially with geo-distributed cells | |
| 15:31:35 | bauzas | agreed with dansmith | |
| 15:31:45 | dansmith | (he dropped) | |
| 15:32:04 | bauzas | hah, my internal meeting trampled this discussion | |
| 15:32:25 | bauzas | -ETOOMANYMEETINHS | |
| 15:32:29 | bauzas | :) | |
| 15:57:20 | kashyap | bauzas: Drop the needless ones on the floor like hot potatoes. And embrace JOMO (joy of missing out) | |
| 15:59:06 | bauzas | hah | |
| 15:59:20 | bauzas | nah, I'm still digesting my Friday-late meeting :p | |
| 16:06:51 | sean-k-mooney | if the central site has direct connectivity to the edge site then you could just centralise the novnc proxy instnace | |
| 16:07:03 | sean-k-mooney | but ya i would have assumed you would want them at each edge site too | |
| 16:07:11 | sean-k-mooney | well each cell | |
| 16:07:15 | sean-k-mooney | not nessisarly edge | |
| 16:08:04 | sean-k-mooney | i was assuming you would run the novnc proxy on the same host as the cell conductor | |
| 16:09:19 | dansmith | sean-k-mooney: I think the change he's referring to was one to make the service only look in one cell, which means you can centralize services, but not unify them (i.e. you need multiple ports and endpoints, regardless of where they are) | |
| 16:13:53 | sean-k-mooney | ah i see | |
| 16:14:51 | sean-k-mooney | unless we moved this to the api db, or allowed the proxy to connect to multiple cell dbs im not sure how we would adress that | |
| 16:15:06 | dansmith | it used to I think, that's the point | |
| 16:15:24 | dansmith | IIRC we removed that ability when we eliminated the consoleauth service | |
| 16:15:55 | melwitt | it (nova-consoleauth) used to use memcache (one instance) to store token auths for the entire deployment | |
| 16:16:08 | dansmith | I imagine that we could add back in just the api db lookup part (like metadata) but I think the expectation was was to make it shard, which I think is a better design, personally | |
| 16:16:12 | dansmith | ah right | |
| 16:18:02 | sean-k-mooney | so really without some way to pass the cell mapping info to a web server there is really no way to use a reverse proxy to expose it over one port/endpoint now | |
| 16:18:12 | melwitt | but yeah, adding a console_auth_token_mappings table would be one way to make it so you only need one console proxy | |
| 16:18:46 | sean-k-mooney | how i would proably try and set it up personaly is have it use a subdomain per cell in the url and have a reverse proxy bind to the single port | |
| 16:18:50 | dansmith | oh, do we not get the instance id as well? | |
| 16:18:53 | sean-k-mooney | then have it delegate to the correct backedn | |
| 16:18:58 | dansmith | that would suck to have to add another mapping :/ | |
| 16:18:59 | melwitt | no we don't, token only | |
| 16:19:10 | dansmith | well then I'm pretty -1 on that plan | |
| 16:19:19 | sean-k-mooney | we dont need to add anything in nova | |
| 16:19:29 | dansmith | you could scatter/gather to find it | |
| 16:19:47 | sean-k-mooney | to find the inial url | |
| 16:19:53 | sean-k-mooney | ya you could | |
| 16:19:54 | dansmith | sean-k-mooney: that's not necessary, because we given them the url from the proxy anyway | |
| 16:20:04 | dansmith | sean-k-mooney: belmiro just doesn't want that | |
| 16:20:26 | dansmith | presumably because he doesn't want to run multiple services and have multiple firewall rules | |
| 16:20:29 | melwitt | oh yeah, I guess he said as much already (scatter gather) | |
| 16:21:03 | sean-k-mooney | ya if he does not want to run multiple proxy instances | |
| 16:21:10 | dansmith | melwitt: ah, I hadn't even clicked the link, but yeah | |
| 16:21:52 | melwitt | we could add another config option! for choosing whether you want a central console auth | |
| 16:21:55 | dansmith | melwitt: I think we'd want that as a toggle | |
| 16:21:57 | dansmith | yeah | |
| 16:22:11 | dansmith | because if you want most efficient and least-shared, you don't want it doing that | |
| 16:22:18 | melwitt | yeah | |
| 18:47:16 | belmoreira | bauzas dansmith melwitt sean-k-mooney I see that you discussed the vncproxy topic. Sorry I needed to leave the office (end of the working day here). | |
| 18:48:03 | sean-k-mooney | no worries. did you add it to the ptg adgenda | |
| 18:48:47 | belmoreira | not yet | |
| 18:48:55 | sean-k-mooney | was dansmith correct when ne assumed you did not want to run multiple novnc proxy instance (1 per cell) | |
| 18:49:22 | belmoreira | let me explain my concern. | |
| 18:49:27 | sean-k-mooney | or are you just concerned about how many port you need to open in the fire wall | |
| 18:49:29 | sean-k-mooney | sure | |
| 18:49:42 | belmoreira | having the vncproxy per cell in theory is good, because we are sharding the service per cell. But it depends in the deployment... For deployments that only expose the console_url in the internal network is ok. | |
| 18:50:05 | belmoreira | However, in my case I need to expose the vncproxy externally. Having only one set os vncproxies allow me to open only one port in the external firewall and have only one console_url address masked by the load balancer. | |
| 18:50:27 | belmoreira | The current approach of having a vncproxy per cell, means that I will have a different console_url per cell. Mapping this with the LB I will need at least to have a different frontend per cell. If I do it per port is a lot of open ports... | |
| 18:50:38 | sean-k-mooney | well you could | |
| 18:50:49 | sean-k-mooney | you can use a reverse proxy instead of a loadblance | |
| 18:51:06 | sean-k-mooney | and expose only one port and have it route the reuest to the backend based on a partil path match | |
| 18:51:17 | sean-k-mooney | or using a subdomain per cell | |
| 18:51:53 | sean-k-mooney | so the reverse proxy is the only thing you open the firewall too and have it dispatch internally to the per cell proxy based on a part of the url | |
| 18:51:56 | belmoreira | true, but in those cases we are also exposing the cell architecture to the user | |
| 18:52:21 | sean-k-mooney | yes at least to the extend needed to match on the url | |
| 18:52:30 | belmoreira | the console_url will be different per cell | |
| 18:52:39 | sean-k-mooney | ya it would be | |
| 18:53:02 | sean-k-mooney | so if we allowed a singel vnc proxy to connect to any of the cell dbs that would be your preference | |
| 18:54:27 | sean-k-mooney | belmoreira: dansmith and melwitt can correct me if i get this wrong but i think what they were suggesting was add a config option to denote if the novnc proxy should connect to on celldb or multiple and having it do a scater gater request to each cell db in the case of multi cell mode | |
| 18:55:07 | belmoreira | having that possibility would be great. I already patch it and have it working in my test infrastructure | |
| 18:55:19 | sean-k-mooney | belmoreira: that would assume that the novnc proxy can actully connect to all the hyperviors in any cell but i belive that is the cause for your env right | |
| 18:56:06 | belmoreira | yes | |
| 18:58:02 | belmoreira | my ideal setup is to have a set of vncproxies that can connect to any hypervisor in the region | |
| 19:00:10 | sean-k-mooney | ya which this would give you | |
| 19:00:51 | sean-k-mooney | i think the main issue is from the consol url we dont know which instance it for without looking up the token and to do that we need to check the cell db | |
| 19:01:10 | sean-k-mooney | so in this case we would need to check multiple cell dbs which increase the load on the db since only one will have the token | |
| 19:01:30 | sean-k-mooney | so we would not want to do that by default but we could allow you to opt into it | |
| 19:02:31 | belmoreira | basically is something similar to the code that I pointed earlier | |
| 19:03:01 | belmoreira | I agree that this shouldn't be the default. Small deployments would not benefit from it | |
| 19:04:59 | dansmith | yeah that's what I meant.. scatter/gather to find the cell that a token is in is not very efficient, but I think the alternative is a lot more work for the few people that might want it | |
| 19:05:19 | dansmith | I'd definitely prefer solving that at the load balancer level with a url suffix or something like that, | |
| 19:05:44 | dansmith | so I think adding a config to scatter/gather is okay and let's not add a new mapping table just for this until/unless performance becomes an issue | |
| 19:08:42 | belmoreira | I think it would be ok. Consoles are not a popular api call. I can report after in terms of performance. | |
| 19:11:47 | belmoreira | thank you all. I can add this into the ptg agenda we need to discuss it more | |
| 19:13:46 | belmoreira | I need to leave now. thank you again | |
| 21:15:36 | zigo | What's the problem with https://review.opendev.org/c/openstack/nova/+/809759 ? (ie what's happening with this nova-tox-validate-backport check?) | |
| 21:57:22 | artom | zigo, it checks that the "source" hash is merged in an upstream branch | |
| 21:57:45 | artom | zigo, so if you cherry pick from wallaby to victoria, the victoria check will fail until the wallaby one merged | |
| 21:59:07 | artom | zigo, ah, no, I had the completely wrong idea without even opening the link | |
| 21:59:11 | artom | "Stable branch requires either cherry-pick -x headers or [stable-only] tag!" is what explains it | |
| 21:59:15 | artom | In the job output | |
| 22:28:08 | clarkb | sounds like someone needs to update the bot or fix the job. Cherrypicking the gitreview change doesn't make sense as it is different than masters | |