| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-30 | |||
| 18:45:00 | mriedem | we could also just have a generic decorator in the api methods that handles this exception and returns HTTPConflict so we don't have to do the dance in the wsgi controller route handler code | |
| 18:45:32 | artom | Yep | |
| 18:45:48 | artom | I wasn't aware of the widespreaded-ness of this | |
| 18:45:59 | artom | Such a decorator would also remain backportable | |
| 18:46:52 | mriedem | yeah and no maybe, | |
| 18:46:59 | mriedem | not all route handlers have 409 in their @wsgi.expected_errors((404, 409)) decorator | |
| 18:47:13 | mriedem | i don't know how much that matters honestly - i wouldn't be surprised if we leak things through in some cases | |
| 18:47:44 | artom | We could always do the wait and see thing | |
| 18:47:55 | artom | Wack this mole, and if another mole pops up, think about a more generic appraoch | |
| 18:48:23 | mriedem | that's what i'd probably do | |
| 18:48:28 | mriedem | no point overengineering this now | |
| 18:48:35 | artom | Ack | |
| 19:00:13 | umbSublime | Is there any reference documentation to help fill up a new blueprint (to avoid wasting peoples time) | |
| 19:03:03 | mriedem | umbSublime: https://docs.openstack.org/nova/latest/contributor/blueprints.html | |
| 19:14:17 | efried | mriedem: added comment as requested to https://review.opendev.org/#/c/682486/ and gibi is +2, wanna send it? | |
| 19:15:16 | mriedem | hmmm, maybe | |
| 19:15:37 | dansmith | cah-ripes | |
| 19:15:57 | dansmith | change failed in the gate and rechecked 4.5h ago, still hasn't started a single job. | |
| 19:16:07 | mriedem | the gate is not happy | |
| 19:16:15 | mriedem | buttloads of ssh fails in guests for different reasons, | |
| 19:16:23 | mriedem | the timed out talking to cell db thing, | |
| 19:16:28 | mriedem | https read timeout errors | |
| 19:16:46 | artom | b'Expected: save(expected_tast_state=[None])' | |
| 19:16:46 | artom | b'Actual: save(expected_task_state=[None])' | |
| 19:16:48 | artom | *sigh* | |
| 19:16:56 | artom | Oh! | |
| 19:19:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Avoid error 500 on shelve task_state race https://review.opendev.org/692206 | |
| 19:28:26 | openstackgerrit | Merged openstack/nova master: api-ref: remove mention about os-migrations no longer being extended https://review.opendev.org/682102 | |
| 19:32:46 | efried | mriedem: are there other projects that are running tempest-slow that could stop doing that, similar to https://review.opendev.org/683988 ? | |
| 19:36:59 | mriedem | efried: don't know | |
| 19:43:30 | mriedem | efried: i'd like to remove that part of the comment that gibi pointed out and then approve, is that ok with you? | |
| 19:43:47 | efried | sure | |
| 19:44:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Only allow one scheduler service in tests https://review.opendev.org/682486 | |
| 19:54:59 | openstackgerrit | Merged openstack/nova master: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/682741 | |
| 19:55:05 | openstackgerrit | Merged openstack/nova master: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/683597 | |
| 19:57:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Avoid error 500 on shelve task_state race https://review.opendev.org/692206 | |
| 20:52:21 | dansmith | efried: asking because I think I've seen some notes from you around this code in the past, | |
| 20:52:41 | dansmith | but do you have strong opinions on what we should do if we have, say, three glance endpoints and we get an error from one trying to do something? | |
| 20:52:57 | efried | dansmith: we should not have three glance endpoints | |
| 20:52:57 | dansmith | looks like we round-robin the available endpoints, but don't move onto the next one after a failure | |
| 20:53:09 | efried | unless you mean three interfaces | |
| 20:53:20 | dansmith | efried: three urls | |
| 20:53:23 | efried | f, did we not get rid of that api_servers thing yet? | |
| 20:53:25 | efried | stand by | |
| 20:53:35 | dansmith | no, the round-robin-ing you mean? that's still there | |
| 20:54:10 | dansmith | I honestly thought that we had pushed responsibility for that down into the glance client, but ... the round-robin stuff is there | |
| 20:54:12 | efried | omg we haven't even deprecated it yet! | |
| 20:54:22 | efried | we should do that. | |
| 20:54:37 | dansmith | efried: use more words.. what are the people currently using that supposed to do? | |
| 20:54:55 | efried | sorry | |
| 20:54:56 | efried | okay | |
| 20:55:27 | efried | people using api_servers were theoretically stuffing multiple endpoints in there, and then they were responsible for making sure all of those endpoints pointed to the same images. | |
| 20:55:37 | dansmith | right | |
| 20:55:41 | efried | nova would, on a given call, pick "the next one" and use it. | |
| 20:56:00 | efried | afaik nova never had any logic to, say, "try the next one" if one failed. | |
| 20:56:13 | dansmith | right, it doesn't, but if we can have multiples we should, but.. go on | |
| 20:56:18 | efried | Because nova doesn't keep track of how many there are. It just loads them up and cycles over them. | |
| 20:56:40 | efried | so if we had only one, we still "cycle" over just the one. | |
| 20:56:50 | efried | and if we tried to add a "try next" thing, we would just end up trying... the same one. | |
| 20:57:28 | dansmith | well, all we'd have to do is retry N-1 times for an endpoint count of N and then we'd hit them all and not dupe one or more | |
| 20:57:29 | efried | if api_servers was a thing we still wanted people to use, I could see enhancing our logic to do that properly - keep track of how many there are and, on certain classes of error, try the next and so on until we succeed or come full circle | |
| 20:57:31 | efried | but | |
| 20:57:39 | efried | we don't want people to use api_servers. | |
| 20:57:47 | dansmith | and that is because why? | |
| 20:57:49 | efried | They should use a single endpoint that's a front for a load balancer | |
| 20:58:02 | efried | and use standard ksa options like every other service | |
| 20:58:25 | dansmith | yeah, so the problem there is that you have to have a HA LB for that | |
| 20:59:04 | dansmith | cloud client software needs to handle failures like this to be robust, I'm not sure why nova shouldn't | |
| 20:59:11 | efried | I thought there was a project that did that for you | |
| 20:59:26 | efried | do we try to HA other things? cinder, keystone, neutron...? | |
| 20:59:33 | efried | (trick question) | |
| 20:59:52 | dansmith | well, glance and keystone would be much easier to retry than the others | |
| 20:59:59 | smcginnis | Not the API, but cinder services can be run active/active. | |
| 21:00:22 | smcginnis | And I've seen people use the active/passive pacemaker setup for things too. | |
| 21:00:29 | dansmith | yeah | |
| 21:00:31 | efried | smcginnis: behind a single catalog endpoint tho? | |
| 21:00:36 | smcginnis | Yeah | |
| 21:00:39 | efried | melwitt: would you mind giving this another look? https://review.opendev.org/#/c/615704/ | |
| 21:00:52 | dansmith | efried: well, point is, if you're going to deprecate it, I'd appreciate you getting that patch up so I can point the people I'm talking to about it | |
| 21:01:13 | dansmith | efried: I remember you talking about this a while ago, had kinda assumed it had already happened | |
| 21:01:32 | efried | dansmith: are you playing devil's advocate or do you really think nova should be responsible for load balancing glance? | |
| 21:01:34 | dansmith | tripleo is using api_servers, so they're going to need some notice | |
| 21:02:05 | dansmith | efried: not load balancing, but being able to have multiple endpoints and find one that works, yeah | |
| 21:02:33 | efried | yeah, "everyone" is probably still using api_servers, because we only added the standard ksa stuff in, I think, queens. | |
| 21:02:41 | dansmith | having to have a floating VIP and HA'ing services behind that VIP for everything really sucks | |
| 21:02:59 | efried | is tripleo using multiple endpoints to api_servers? | |
| 21:03:12 | dansmith | efried: just to be clear, "the ksa stuff" doesn't imply anything about multiple endpoints right? | |
| 21:03:15 | dansmith | efried: they are | |
| 21:03:19 | melwitt | efried: whoa, that is old. I can look at it again once I figure out what it's about again | |
| 21:03:37 | dansmith | efried: for glance, but we don't do the failover stuff, so not really for much gain | |
| 21:03:38 | efried | melwitt: thanks, I got a nudge from a downstream (not $employer) who needs it. | |
| 21:03:47 | melwitt | ack | |
| 21:04:02 | efried | dansmith: right, the way nova handles it really isn't HA-ish, it's more load-balance-ish. | |
| 21:04:29 | dansmith | efried: currently, right | |
| 21:04:51 | efried | dansmith: and no, "the ksa stuff" doesn't have anything for mulitiple endpoints... I don't think. (Can you put multiple endpoints in the service catalog?) | |
| 21:05:25 | dansmith | gah | |
| 21:05:30 | efried | dansmith: I can tell you this, cutting over to sdk to talk to glance, api_servers will have to be gone first. | |
| 21:05:46 | dansmith | brb | |
| 21:06:34 | dansmith | efried: I dunno about the service catalog, probably not | |
| 21:06:43 | dansmith | not saying that's a good thing, but.. :) | |