| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-30 | |||
| 21:34:35 | mriedem | note that a lot of that ML thread is about removing the endpoint_override options, which we haven't done i don't think | |
| 21:34:40 | mriedem | ksa has an override option | |
| 21:35:12 | mriedem | http://lists.openstack.org/pipermail/openstack-dev/2017-May/116149.html sums up what dan was saying | |
| 21:35:17 | dansmith | well, if I'm understanding this right, once it's down in ksa then we lose the ability to retry smartly right? | |
| 21:35:33 | mriedem | i'm just saying the thread diverged a bit, | |
| 21:35:34 | dansmith | yup | |
| 21:35:47 | mriedem | between overrding the endpoint (don't use the catalog) and multiple api servers for load balancing | |
| 21:37:12 | mriedem | something something openstack ironic > k8s ingress > nova|glance > profit!!! | |
| 21:37:13 | dansmith | and godaddy and blair there actually mention failover more than load-balancing | |
| 21:37:35 | dansmith | heh, well, k8s actually handles this as part of the infra, AFAIK, so ... yeah ;) | |
| 21:38:05 | mriedem | yeah, that's blair's "I think it'd be useful if all OpenStack APIs and their clients actively handled this poor-man's HA without having to resort to haproxy" | |
| 21:38:20 | mriedem | i.e. would be nice if openstack just handled this natively | |
| 21:40:01 | dansmith | right | |
| 21:40:17 | sean-k-mooney | honestly k8s with the nginx ingress control + cert manger + lets encrypt is a nice way to get free tls and do load balancing or at least revers proxying | |
| 21:40:24 | dansmith | because it's kinda dumb for us to reject all kinds of pet features and then not be elastic and failure-handly ourselves | |
| 21:41:22 | sean-k-mooney | johnsom: does octavia have any support for metallb? | |
| 21:42:06 | johnsom | sean-k-mooney: not at the moment | |
| 21:42:28 | johnsom | You can use Octavia for you ingress in k8s though | |
| 21:42:52 | sean-k-mooney | yes | |
| 21:43:10 | sean-k-mooney | but that does seam well like more effort | |
| 21:43:45 | johnsom | The other thing to think about when replacing an LB tier is rate limiting, WAF, and TLS offload | |
| 21:44:18 | sean-k-mooney | you do not nessisarly want your load balancer to have to do those things | |
| 21:44:23 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#glance.api_servers isn't an LB tier :) | |
| 21:44:25 | sean-k-mooney | but yes | |
| 21:45:45 | johnsom | I guess that is the context I missed. That seems duplicate to keystone, why not just deprecate it? | |
| 21:45:59 | mriedem | keystone doesn't have a round robin concept like what this thing is | |
| 21:46:06 | mriedem | it's just a very dumb, oh endpoint 1 failed, try endpoint 2 | |
| 21:46:11 | johnsom | DNS provides round robin | |
| 21:47:25 | dansmith | DNS doesn't provide smart round robin unless you resolve the name to all the addresses and then try them yourself | |
| 21:47:39 | dansmith | smart meaning "know when to stop retrying" | |
| 21:47:56 | dansmith | dumb use of DNS RR gets you poor-man's LB, which is not what we're talking about | |
| 21:48:26 | dansmith | since we're off on a tangent, I'm gonna drop off | |
| 21:52:48 | sean-k-mooney | dansmith: so i have scolled back to the point where you have strong opions on how this should work. what was the orginial question? | |
| 21:53:12 | openstackgerrit | Merged openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963 | |
| 21:53:18 | openstackgerrit | Merged openstack/nova master: Move Destination object tests to their own test class https://review.opendev.org/683017 | |
| 21:53:25 | openstackgerrit | Merged openstack/nova master: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/683961 | |
| 21:54:45 | sean-k-mooney | dansmith: was it solely how to support glance active active | |
| 21:56:24 | sean-k-mooney | and then the load balancer converstation was a tangent form [glance]/api_servers not being a loadblancing teir or even really a ha solution | |
| 21:56:49 | eandersson | Is there a way to make nginx retry when uwsgi (nova-metadata-api) gets overwhelmed? | |
| 21:57:22 | eandersson | Seeing nova-metadata-api falling behind and causing nginx to throw 502s | |
| 21:58:10 | sean-k-mooney | am can you just increase the process count on the uwsgi config | |
| 21:58:17 | eandersson | We have 5x 24 nova-metadata-plus processes and feels like it should be high enough | |
| 21:58:35 | sean-k-mooney | ah ok | |
| 21:58:36 | eandersson | Kinda feel like 120 should be enough. | |
| 21:58:58 | sean-k-mooney | have you truned on memcache caching for those too | |
| 21:59:00 | eandersson | I'll probably end up bumping it, but feel like I am missing something | |
| 21:59:25 | eandersson | [cache] has memcached enabled, and keystone auth too | |
| 21:59:43 | donnyd | melwitt: thank you :) | |
| 22:00:04 | sean-k-mooney | ok because if you have lots of worker but use the normal in memeory caceh you end up rebuilding stuff in each worker over and over again | |
| 22:03:01 | eandersson | Gonna bump the worker count to like 200, but will dig deeper | |
| 22:03:24 | eandersson | Might do something like https://stackoverflow.com/questions/44581719/resource-temporarily-unavailable-using-uwsgi-nginx as well to see if that helps a bit | |
| 22:03:45 | eandersson | since it looks like a ddos, but kinda unavoidable when you have... many many thousands vms :p | |
| 22:04:53 | sean-k-mooney | ya its possible your hitting connection limits. how is memory looking on those nodes. you dont see OOM event or any other red flags | |
| 22:10:34 | eandersson | We had to beef them up due to neutron. | |
| 22:10:53 | eandersson | So got plenty of memory :D | |
| 22:11:11 | eandersson | Might be stacking too many services. Could be hitting limits on context switching. | |
| 22:11:32 | sean-k-mooney | im glad but also sad. "whats using all the ram? our cloud software :(" | |
| 22:12:04 | eandersson | Yea I think I mentioned this already, but each neutron worker peaks out at 8.2gb memory per process when under heavy load :'( | |
| 22:12:28 | sean-k-mooney | ya that is much more hten i was expecting | |
| 22:13:23 | eandersson | I do wonder what kind of hardware people use for their larget clusters | |
| 22:13:28 | eandersson | for the "control plane" | |
| 22:13:48 | mriedem | are you sure it's using the cache properly? do you see "Using cached metadata for" in the metadata API debug logs? | |
| 22:14:11 | eandersson | I'll check in our lab setup sec | |
| 22:15:12 | sean-k-mooney | enabling memcache instead of the dict cache makes a huge difference so if its not useing it properly you could definetly reduce teh amount of workers as a resutl of fixing it | |
| 22:16:04 | mriedem | fwiw cern and huawei public cloud run a metadata api per cell with direct access to the cell db rather than a huge since metadata api that spans the entire deployment | |
| 22:16:16 | mriedem | s/since/single/ | |
| 22:16:42 | mriedem | https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#nova-metadata-api-service | |
| 22:16:43 | sean-k-mooney | mriedem: ya that seams like a better way to approch scaling | |
| 22:16:56 | mriedem | blizzard is only 1 cell though right? | |
| 22:17:51 | sean-k-mooney | dont you have to modify neutorn to proxy to the local instance in that case too | |
| 22:17:56 | eandersson | Yea seeing | |
| 22:17:57 | eandersson | > Using cached metadata for X get_metadata_by_remote_address | |
| 22:18:06 | eandersson | Yea one cell many regions | |
| 22:19:12 | mriedem | sean-k-mooney: yes that's in that doc | |
| 22:20:05 | sean-k-mooney | woudl it make sense even in a singel cell deployment to configure different neutron metadata proxies to hit different apis directly rather then loadbalnce? | |
| 22:21:27 | sean-k-mooney | eandersson: out of interest do you knwo how long the resonce form the metadata api take when you dont get teh 502s | |
| 22:22:12 | eandersson | ~0.004 | |
| 22:23:06 | sean-k-mooney | ok that is normal. we have seen up to 10s of seconds in the gate randomly before we enabeld the memcache caching | |
| 22:23:44 | eandersson | I might have overtuned nginx tbh combined with the nature of metadata | |
| 22:24:25 | eandersson | I set worker count to auto in nginx which ends up creating processors based on the vcpu count | |
| 22:24:38 | eandersson | but uwsgi worker is set to 2/vcpu | |
| 22:25:13 | sean-k-mooney | so i honestly dont know how much load you have but i would try a smaller value like 8 | |
| 22:25:20 | sean-k-mooney | instead of 24 | |
| 22:25:32 | sean-k-mooney | i certely would not have more then 1 per hyper tread | |
| 22:25:48 | eandersson | Yea exactly got one per core now (48/2) | |
| 22:26:10 | eandersson | for nova-api at least | |
| 22:27:08 | sean-k-mooney | ya if your resonce time is in the order of 4ms i would bring that down as i siad to like 8-16 and see if that help rahter then go from 24 to 30 | |
| 22:27:23 | eandersson | We did a lot of benchmarking, but it's difficult to tell how the system performs in real enviornments | |
| 23:14:11 | sean-k-mooney | artom: i left a comment for you here https://review.opendev.org/#/c/692206/3/nova/api/openstack/compute/shelve.py | |
| 23:14:24 | sean-k-mooney | artom: i think that would also fix the issue more generally | |
| 23:14:39 | sean-k-mooney | but i have not tested it | |
| 23:15:25 | sean-k-mooney | anyway night all o/ | |
| 23:17:01 | artom | sean-k-mooney, nice idea | |
| 23:17:13 | artom | I'd kinda be afraid of any side effects | |
| 23:17:30 | artom | It'd a wide-reaching change | |
| 23:43:44 | openstackgerrit | Merged openstack/nova master: fixtures: Add support for security groups https://review.opendev.org/686802 | |
| 23:47:56 | openstackgerrit | Merged openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344 | |
| #openstack-nova - 2019-10-31 | |||
| 01:25:57 | openstackgerrit | Merged openstack/nova master: nova-net: Use deepcopy on value returned by NeutronFixture https://review.opendev.org/688139 | |
| 02:04:58 | openstackgerrit | melanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests https://review.opendev.org/692252 | |
| 02:07:43 | openstackgerrit | melanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests https://review.opendev.org/692252 | |