| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-30 | |||
| 21:17:18 | dansmith | which is fine in the trivial case, but not so much when we're making multiple calls as part of a complex operation, like against cinder or neutron | |
| 21:17:39 | dansmith | mriedem: might be what, arbitrary? | |
| 21:17:45 | dansmith | or might be constrained? | |
| 21:18:32 | dansmith | based on the description in that spec, though, it tries them in order, so you'd hammer internal1 until it fails, and then try internal2, it sounds like | |
| 21:18:37 | dansmith | which wouldn't be what you'd want really | |
| 21:21:26 | openstackgerrit | Eric Fried proposed openstack/nova master: Deprecate [glance]api_servers https://review.opendev.org/692227 | |
| 21:21:28 | efried | dansmith: ^ | |
| 21:21:28 | mriedem | dansmith: i think valid_interfaces is arbitrary but i've had to look this up multiple times in the ksa code | |
| 21:21:38 | dansmith | efried: thanks | |
| 21:21:50 | efried | dansmith: no, valid_interfaces is public/private/internal only | |
| 21:21:59 | efried | I think they can have 'URL' suffixes for legacy reasons | |
| 21:22:01 | mriedem | also, on that note in the spec for api_servers, i want to say the godaddy or smorrison made a point that they still really needed that when the spec was being reviewed | |
| 21:22:48 | dansmith | mriedem: yeah, I *think* people are going to be quite unhappy about it, but we'll see | |
| 21:22:56 | mriedem | might have been in the ML, i don't see it in review comments | |
| 21:22:59 | efried | "really needed" meaning it was easier for us to continue to support it than it was for them to stop relying on our shoddy load balancing hack | |
| 21:24:27 | efried | I'm having trouble finding sympathy for "wah, I don't want to set up a load balancer in my cloud" from someone who has enough resources to have multiple glance API endpoints in the first place. | |
| 21:24:35 | efried | Isn't there an openstack project that does load balancing for you? | |
| 21:24:50 | dansmith | efried: a six-node mini cloud would still have three controllers in our setups | |
| 21:25:01 | mriedem | efried: are you thinking of octavia? | |
| 21:25:10 | dansmith | efried: we don't deploy less than three controllers (and thus glance apis) no matter how big or small your cloud is, which means it's *always* an issue | |
| 21:25:25 | efried | ...yes, it looks like I may be thinking of octavia | |
| 21:25:39 | mriedem | i don't know much about octavia, but doesn't that spawn VMs to run the load balancer? | |
| 21:25:46 | mriedem | and thus kind of a chicken and egg issue? | |
| 21:25:49 | efried | I dunno how it works | |
| 21:25:50 | dansmith | yeah, octavia is not a solution to this problem | |
| 21:26:02 | mriedem | johnsom would know | |
| 21:26:04 | dansmith | it's a thing you use on top of a functioning cloud to LB an application | |
| 21:26:07 | mriedem | right | |
| 21:26:08 | dansmith | (afaik) | |
| 21:27:14 | efried | it's not a chicken/egg when you're using it as a *load balancer*. It would be chicken/egg if you were using it for HA. | |
| 21:27:15 | mriedem | e.g. we don't use trove instances to run the nova db... | |
| 21:27:21 | efried | which, again, this is not and never has been. | |
| 21:28:03 | dansmith | efried: it's totally chicken and egg for nova to use it between it and glance, for whatever reason :D | |
| 21:28:20 | efried | but at this point I need to stfu and get back to unwedging osc. I'll look for discussion on that patch I guess. | |
| 21:28:33 | dansmith | yep, thanks pointing people at that soonly | |
| 21:28:36 | efried | and probably expect dansmith to start a ML thread for his people | |
| 21:28:37 | efried | yeah | |
| 21:29:01 | dansmith | yeah, octavia requires a functioning nova to work against | |
| 21:29:09 | dansmith | so nova using octavia to talk to glance is not a thing :) | |
| 21:29:26 | mriedem | here is the thread btw http://lists.openstack.org/pipermail/openstack-dev/2017-April/116028.html | |
| 21:29:45 | mriedem | which also bled into may | |
| 21:30:00 | mriedem | http://lists.openstack.org/pipermail/openstack-dev/2017-May/116574.html is godaddy | |
| 21:32:10 | dansmith | yeah | |
| 21:32:59 | dansmith | I know tripleo and similar things HA certain endpoints, but AFAIK, glance and rabbit have always been multiple servers | |
| 21:33:38 | dansmith | especially for fundamental things like that, it's far easier to set that kind of thing in config vs. something you need an api to do, | |
| 21:33:45 | dansmith | for the same reason that cells in the database is problematic | |
| 21:33:50 | dansmith | anyway | |
| 21:34:05 | johnsom | o/ Not all of the Octavia drivers use nova, but I still would not recommend using Octavia load balancers for internal API traffic. There are many chickens and eggs to fight with for that. I.e. keystone, neutron, potentially glance/nova, etc. | |
| 21:34:35 | mriedem | note that a lot of that ML thread is about removing the endpoint_override options, which we haven't done i don't think | |
| 21:34:40 | mriedem | ksa has an override option | |
| 21:35:12 | mriedem | http://lists.openstack.org/pipermail/openstack-dev/2017-May/116149.html sums up what dan was saying | |
| 21:35:17 | dansmith | well, if I'm understanding this right, once it's down in ksa then we lose the ability to retry smartly right? | |
| 21:35:33 | mriedem | i'm just saying the thread diverged a bit, | |
| 21:35:34 | dansmith | yup | |
| 21:35:47 | mriedem | between overrding the endpoint (don't use the catalog) and multiple api servers for load balancing | |
| 21:37:12 | mriedem | something something openstack ironic > k8s ingress > nova|glance > profit!!! | |
| 21:37:13 | dansmith | and godaddy and blair there actually mention failover more than load-balancing | |
| 21:37:35 | dansmith | heh, well, k8s actually handles this as part of the infra, AFAIK, so ... yeah ;) | |
| 21:38:05 | mriedem | yeah, that's blair's "I think it'd be useful if all OpenStack APIs and their clients actively handled this poor-man's HA without having to resort to haproxy" | |
| 21:38:20 | mriedem | i.e. would be nice if openstack just handled this natively | |
| 21:40:01 | dansmith | right | |
| 21:40:17 | sean-k-mooney | honestly k8s with the nginx ingress control + cert manger + lets encrypt is a nice way to get free tls and do load balancing or at least revers proxying | |
| 21:40:24 | dansmith | because it's kinda dumb for us to reject all kinds of pet features and then not be elastic and failure-handly ourselves | |
| 21:41:22 | sean-k-mooney | johnsom: does octavia have any support for metallb? | |
| 21:42:06 | johnsom | sean-k-mooney: not at the moment | |
| 21:42:28 | johnsom | You can use Octavia for you ingress in k8s though | |
| 21:42:52 | sean-k-mooney | yes | |
| 21:43:10 | sean-k-mooney | but that does seam well like more effort | |
| 21:43:45 | johnsom | The other thing to think about when replacing an LB tier is rate limiting, WAF, and TLS offload | |
| 21:44:18 | sean-k-mooney | you do not nessisarly want your load balancer to have to do those things | |
| 21:44:23 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#glance.api_servers isn't an LB tier :) | |
| 21:44:25 | sean-k-mooney | but yes | |
| 21:45:45 | johnsom | I guess that is the context I missed. That seems duplicate to keystone, why not just deprecate it? | |
| 21:45:59 | mriedem | keystone doesn't have a round robin concept like what this thing is | |
| 21:46:06 | mriedem | it's just a very dumb, oh endpoint 1 failed, try endpoint 2 | |
| 21:46:11 | johnsom | DNS provides round robin | |
| 21:47:25 | dansmith | DNS doesn't provide smart round robin unless you resolve the name to all the addresses and then try them yourself | |
| 21:47:39 | dansmith | smart meaning "know when to stop retrying" | |
| 21:47:56 | dansmith | dumb use of DNS RR gets you poor-man's LB, which is not what we're talking about | |
| 21:48:26 | dansmith | since we're off on a tangent, I'm gonna drop off | |
| 21:52:48 | sean-k-mooney | dansmith: so i have scolled back to the point where you have strong opions on how this should work. what was the orginial question? | |
| 21:53:12 | openstackgerrit | Merged openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963 | |
| 21:53:18 | openstackgerrit | Merged openstack/nova master: Move Destination object tests to their own test class https://review.opendev.org/683017 | |
| 21:53:25 | openstackgerrit | Merged openstack/nova master: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/683961 | |
| 21:54:45 | sean-k-mooney | dansmith: was it solely how to support glance active active | |
| 21:56:24 | sean-k-mooney | and then the load balancer converstation was a tangent form [glance]/api_servers not being a loadblancing teir or even really a ha solution | |
| 21:56:49 | eandersson | Is there a way to make nginx retry when uwsgi (nova-metadata-api) gets overwhelmed? | |
| 21:57:22 | eandersson | Seeing nova-metadata-api falling behind and causing nginx to throw 502s | |
| 21:58:10 | sean-k-mooney | am can you just increase the process count on the uwsgi config | |
| 21:58:17 | eandersson | We have 5x 24 nova-metadata-plus processes and feels like it should be high enough | |
| 21:58:35 | sean-k-mooney | ah ok | |
| 21:58:36 | eandersson | Kinda feel like 120 should be enough. | |
| 21:58:58 | sean-k-mooney | have you truned on memcache caching for those too | |
| 21:59:00 | eandersson | I'll probably end up bumping it, but feel like I am missing something | |
| 21:59:25 | eandersson | [cache] has memcached enabled, and keystone auth too | |
| 21:59:43 | donnyd | melwitt: thank you :) | |
| 22:00:04 | sean-k-mooney | ok because if you have lots of worker but use the normal in memeory caceh you end up rebuilding stuff in each worker over and over again | |
| 22:03:01 | eandersson | Gonna bump the worker count to like 200, but will dig deeper | |
| 22:03:24 | eandersson | Might do something like https://stackoverflow.com/questions/44581719/resource-temporarily-unavailable-using-uwsgi-nginx as well to see if that helps a bit | |
| 22:03:45 | eandersson | since it looks like a ddos, but kinda unavoidable when you have... many many thousands vms :p | |