| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-30 | |||
| 20:57:31 | efried | but | |
| 20:57:39 | efried | we don't want people to use api_servers. | |
| 20:57:47 | dansmith | and that is because why? | |
| 20:57:49 | efried | They should use a single endpoint that's a front for a load balancer | |
| 20:58:02 | efried | and use standard ksa options like every other service | |
| 20:58:25 | dansmith | yeah, so the problem there is that you have to have a HA LB for that | |
| 20:59:04 | dansmith | cloud client software needs to handle failures like this to be robust, I'm not sure why nova shouldn't | |
| 20:59:11 | efried | I thought there was a project that did that for you | |
| 20:59:26 | efried | do we try to HA other things? cinder, keystone, neutron...? | |
| 20:59:33 | efried | (trick question) | |
| 20:59:52 | dansmith | well, glance and keystone would be much easier to retry than the others | |
| 20:59:59 | smcginnis | Not the API, but cinder services can be run active/active. | |
| 21:00:22 | smcginnis | And I've seen people use the active/passive pacemaker setup for things too. | |
| 21:00:29 | dansmith | yeah | |
| 21:00:31 | efried | smcginnis: behind a single catalog endpoint tho? | |
| 21:00:36 | smcginnis | Yeah | |
| 21:00:39 | efried | melwitt: would you mind giving this another look? https://review.opendev.org/#/c/615704/ | |
| 21:00:52 | dansmith | efried: well, point is, if you're going to deprecate it, I'd appreciate you getting that patch up so I can point the people I'm talking to about it | |
| 21:01:13 | dansmith | efried: I remember you talking about this a while ago, had kinda assumed it had already happened | |
| 21:01:32 | efried | dansmith: are you playing devil's advocate or do you really think nova should be responsible for load balancing glance? | |
| 21:01:34 | dansmith | tripleo is using api_servers, so they're going to need some notice | |
| 21:02:05 | dansmith | efried: not load balancing, but being able to have multiple endpoints and find one that works, yeah | |
| 21:02:33 | efried | yeah, "everyone" is probably still using api_servers, because we only added the standard ksa stuff in, I think, queens. | |
| 21:02:41 | dansmith | having to have a floating VIP and HA'ing services behind that VIP for everything really sucks | |
| 21:02:59 | efried | is tripleo using multiple endpoints to api_servers? | |
| 21:03:12 | dansmith | efried: just to be clear, "the ksa stuff" doesn't imply anything about multiple endpoints right? | |
| 21:03:15 | dansmith | efried: they are | |
| 21:03:19 | melwitt | efried: whoa, that is old. I can look at it again once I figure out what it's about again | |
| 21:03:37 | dansmith | efried: for glance, but we don't do the failover stuff, so not really for much gain | |
| 21:03:38 | efried | melwitt: thanks, I got a nudge from a downstream (not $employer) who needs it. | |
| 21:03:47 | melwitt | ack | |
| 21:04:02 | efried | dansmith: right, the way nova handles it really isn't HA-ish, it's more load-balance-ish. | |
| 21:04:29 | dansmith | efried: currently, right | |
| 21:04:51 | efried | dansmith: and no, "the ksa stuff" doesn't have anything for mulitiple endpoints... I don't think. (Can you put multiple endpoints in the service catalog?) | |
| 21:05:25 | dansmith | gah | |
| 21:05:30 | efried | dansmith: I can tell you this, cutting over to sdk to talk to glance, api_servers will have to be gone first. | |
| 21:05:46 | dansmith | brb | |
| 21:06:34 | dansmith | efried: I dunno about the service catalog, probably not | |
| 21:06:43 | dansmith | not saying that's a good thing, but.. :) | |
| 21:07:10 | dansmith | efried: okay so, can you put up a patch to deprecate it real quick so I can point people to it? | |
| 21:07:17 | efried | I'm going to have to nail down mordred for the more philosophical aspects of this. | |
| 21:07:20 | dansmith | efried: -W it if you want for completeness and comments or something | |
| 21:07:37 | efried | I can do that, yeah, though now you've got me doubting if we *should*. | |
| 21:07:38 | dansmith | efried: ack, we can blame lots of stuff on him like this | |
| 21:07:47 | dansmith | efried: that's fine, a place to collect feedback | |
| 21:07:50 | efried | ... | |
| 21:08:09 | dansmith | efried: the tripleo people can go voice their hate there if they want, or they may say "meh, okay, we can VIP this too" I honestly don't know | |
| 21:08:20 | efried | cool | |
| 21:08:39 | dansmith | efried: can you do that, like, uh, quickly so I can reply to someone quickly? :D | |
| 21:08:47 | efried | on it rn | |
| 21:08:52 | dansmith | spanks. | |
| 21:08:56 | efried | if people would stop bugging me in IRC. | |
| 21:12:57 | efried | dansmith: while you're waiting, I found this (search for api_servers) http://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/use-ksa-adapter-for-endpoints.html#proposed-change | |
| 21:13:47 | dansmith | "The exception is [glance]api_servers, which will continue to be supported." | |
| 21:14:51 | dansmith | I wonder if "valid_interfaces" is an arbitrary list, such that you could have internal1, internal2, internal3? | |
| 21:15:48 | openstackgerrit | Joshua Cornutt proposed openstack/nova master: Moving get_hash_str() from md5 to sha-256 https://review.opendev.org/615704 | |
| 21:16:20 | dansmith | I also don't know how LBs (or HAproxy) handle a failed http session post-request.. meaning if I do a GET against glance and then the connection is reset after I've sent the GET, does the LB retry and re-play that against the next server? | |
| 21:16:24 | mriedem | dansmith: i think it might be, we've had to dig into the ksa code a few times to answer that | |
| 21:16:44 | dansmith | I'd kindof doubt it.. figure it'd fail the client, mark the server, next request doesn't include the bad server | |
| 21:16:56 | dansmith | which means if we don't retry we propagate the failure up | |
| 21:17:18 | dansmith | which is fine in the trivial case, but not so much when we're making multiple calls as part of a complex operation, like against cinder or neutron | |
| 21:17:39 | dansmith | mriedem: might be what, arbitrary? | |
| 21:17:45 | dansmith | or might be constrained? | |
| 21:18:32 | dansmith | based on the description in that spec, though, it tries them in order, so you'd hammer internal1 until it fails, and then try internal2, it sounds like | |
| 21:18:37 | dansmith | which wouldn't be what you'd want really | |
| 21:21:26 | openstackgerrit | Eric Fried proposed openstack/nova master: Deprecate [glance]api_servers https://review.opendev.org/692227 | |
| 21:21:28 | efried | dansmith: ^ | |
| 21:21:28 | mriedem | dansmith: i think valid_interfaces is arbitrary but i've had to look this up multiple times in the ksa code | |
| 21:21:38 | dansmith | efried: thanks | |
| 21:21:50 | efried | dansmith: no, valid_interfaces is public/private/internal only | |
| 21:21:59 | efried | I think they can have 'URL' suffixes for legacy reasons | |
| 21:22:01 | mriedem | also, on that note in the spec for api_servers, i want to say the godaddy or smorrison made a point that they still really needed that when the spec was being reviewed | |
| 21:22:48 | dansmith | mriedem: yeah, I *think* people are going to be quite unhappy about it, but we'll see | |
| 21:22:56 | mriedem | might have been in the ML, i don't see it in review comments | |
| 21:22:59 | efried | "really needed" meaning it was easier for us to continue to support it than it was for them to stop relying on our shoddy load balancing hack | |
| 21:24:27 | efried | I'm having trouble finding sympathy for "wah, I don't want to set up a load balancer in my cloud" from someone who has enough resources to have multiple glance API endpoints in the first place. | |
| 21:24:35 | efried | Isn't there an openstack project that does load balancing for you? | |
| 21:24:50 | dansmith | efried: a six-node mini cloud would still have three controllers in our setups | |
| 21:25:01 | mriedem | efried: are you thinking of octavia? | |
| 21:25:10 | dansmith | efried: we don't deploy less than three controllers (and thus glance apis) no matter how big or small your cloud is, which means it's *always* an issue | |
| 21:25:25 | efried | ...yes, it looks like I may be thinking of octavia | |
| 21:25:39 | mriedem | i don't know much about octavia, but doesn't that spawn VMs to run the load balancer? | |
| 21:25:46 | mriedem | and thus kind of a chicken and egg issue? | |
| 21:25:49 | efried | I dunno how it works | |
| 21:25:50 | dansmith | yeah, octavia is not a solution to this problem | |
| 21:26:02 | mriedem | johnsom would know | |
| 21:26:04 | dansmith | it's a thing you use on top of a functioning cloud to LB an application | |
| 21:26:07 | mriedem | right | |
| 21:26:08 | dansmith | (afaik) | |
| 21:27:14 | efried | it's not a chicken/egg when you're using it as a *load balancer*. It would be chicken/egg if you were using it for HA. | |
| 21:27:15 | mriedem | e.g. we don't use trove instances to run the nova db... | |
| 21:27:21 | efried | which, again, this is not and never has been. | |
| 21:28:03 | dansmith | efried: it's totally chicken and egg for nova to use it between it and glance, for whatever reason :D | |
| 21:28:20 | efried | but at this point I need to stfu and get back to unwedging osc. I'll look for discussion on that patch I guess. | |
| 21:28:33 | dansmith | yep, thanks pointing people at that soonly | |
| 21:28:36 | efried | and probably expect dansmith to start a ML thread for his people | |
| 21:28:37 | efried | yeah | |
| 21:29:01 | dansmith | yeah, octavia requires a functioning nova to work against | |
| 21:29:09 | dansmith | so nova using octavia to talk to glance is not a thing :) | |
| 21:29:26 | mriedem | here is the thread btw http://lists.openstack.org/pipermail/openstack-dev/2017-April/116028.html | |