| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 16:03:19 | mriedem | stephenfin: saying "because the only people that can solve this are in this channel/on the review already" is not true imo - if you've got a hard upgrade thing coming for operators, you likely should get some feedback from them before pushing forward | |
| 16:03:28 | alex_xu | sean-k-mooney: thanks a lot | |
| 16:04:07 | sean-k-mooney | mriedem: the upgrade will be signifcantly harder if we also have to deal with numa in placment in the same release | |
| 16:04:26 | mriedem | sean-k-mooney: i don't know what that has to do with this at all | |
| 16:04:34 | sean-k-mooney | mriedem: if we defer pcpus in placment to U we will have to deal with both in one go | |
| 16:04:55 | mriedem | i didn't say anything about deferring | |
| 16:05:33 | mriedem | i said, does anyone outside of the 3 people reviewing this that will actually have to deal with the upgrade know what the plan is | |
| 16:05:36 | mriedem | and are they ok with it | |
| 16:05:39 | sean-k-mooney | right but the current upgrade approch is the best we could come up with and we went to the MLs and asked if the toggel was ok | |
| 16:05:54 | mriedem | and no operators even saw that thread, | |
| 16:06:06 | mriedem | which is why i asked (again) if it could be posed with a [nova][ops] tag | |
| 16:06:08 | mriedem | to get visibility | |
| 16:06:18 | mriedem | lack of feedback from operators is not agreement | |
| 16:06:50 | sean-k-mooney | well we did ask cern in irc | |
| 16:06:56 | dansmith | I think it's probably good to get feedback not just from ops,. | |
| 16:07:00 | sean-k-mooney | but i would have liked other to comment too | |
| 16:07:03 | dansmith | but from people that have to do this in the deployment tools | |
| 16:07:08 | mriedem | it would be a lot better to know before releasing train that "this sucks but it's not terrible" rather than "this is a no-go for me" | |
| 16:07:14 | dansmith | as this adds at least one more atomic reconfigure/restart of the deployment | |
| 16:07:49 | mriedem | sure, i lump mnaser into the ops and tooling (OSA) camps | |
| 16:08:00 | dansmith | yup | |
| 16:08:04 | stephenfin | I don't see what the actual issue is though | |
| 16:08:10 | mriedem | sean-k-mooney: and cern (surya? belmiro?) said what? | |
| 16:08:10 | sean-k-mooney | dansmith: for what its worth we talked about this internally with our tripleo folks that will be implementing and they were ok and actully prefered the seperate config flip step | |
| 16:08:21 | sean-k-mooney | mriedem: we ask belmiro | |
| 16:08:30 | dansmith | sean-k-mooney: preferred to what/ | |
| 16:08:32 | sean-k-mooney | and he was ok with the config | |
| 16:08:39 | stephenfin | You do your upgrade and nothing changes. At some point after the upgrade, you go tweak knobs on the compute nodes followed by a knob on the scheduler | |
| 16:08:42 | stephenfin | and you're done | |
| 16:08:51 | sean-k-mooney | ill see if i can find the irc logs | |
| 16:08:53 | dansmith | stephenfin: and restart the whole deployment atomically :) | |
| 16:09:03 | stephenfin | no, you don't need to do that | |
| 16:09:09 | dansmith | no? | |
| 16:09:15 | dansmith | you say "immediately" in your comment | |
| 16:09:38 | stephenfin | I said we'd have to do that immediately if I wasn't doing the things I was doing to prevent that | |
| 16:10:00 | sean-k-mooney | dansmith: the alternitve was to do the doble report of resouces as both vcpu and pcpu by the way. and that was not done for a spciric reason i cant rememebr | |
| 16:10:49 | dansmith | sean-k-mooney: yeah, that's a terrible alternative, agreed :) | |
| 16:10:57 | dansmith | "Would you prefer an extra config step or a kick in the nuts?" | |
| 16:11:08 | dansmith | I could get most people to agree to the first | |
| 16:11:19 | sean-k-mooney | it did not require the config and it was self healing but ok | |
| 16:11:58 | dansmith | seems to me that with the current plan, | |
| 16:12:00 | dansmith | after they've upgraded, | |
| 16:12:01 | sean-k-mooney | i think we did not do it because of an issue with reshapes | |
| 16:12:13 | stephenfin | sean-k-mooney: Not reshapes, no | |
| 16:12:17 | stephenfin | http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008501.html | |
| 16:12:20 | dansmith | they have to change all (or some fraction) of their computes to the new config to expose the new resources, | |
| 16:12:21 | dansmith | restart them, | |
| 16:12:36 | dansmith | then tweak the scheduler config to ask for the new thing, then restart those, | |
| 16:12:42 | dansmith | then fix the rest of the computes before running out of capacity, and then restart those | |
| 16:12:43 | dansmith | right? | |
| 16:12:54 | sean-k-mooney | dansmith more or less | |
| 16:12:58 | dansmith | that's the most graceful thing | |
| 16:13:05 | stephenfin | dansmith: exactly, yeah | |
| 16:13:10 | dansmith | which sounds like (a) hard to automate and (b) laborious | |
| 16:13:22 | dansmith | otherwise you're looking for full atomic downtime while you do all that in one go | |
| 16:13:53 | dansmith | I can't imagine OSA is going to decide what the sufficient fraction for conversion is, | |
| 16:13:57 | sean-k-mooney | well for FFU we take down the whole cloud contol plain so that not unprecidented | |
| 16:14:04 | dansmith | convert that set, reconfig/restart control services, etc | |
| 16:14:12 | stephenfin | the scheduler option exists to prevent the need for that atomic upgrade | |
| 16:14:15 | dansmith | sean-k-mooney: this is for rolling one release | |
| 16:14:17 | stephenfin | *exists solely | |
| 16:14:33 | mriedem | sean-k-mooney: vexxhost doesn't need to FFU because they actually don't suck at CD | |
| 16:14:34 | dansmith | stephenfin: which likely only works for the case where the humans decide when to throw that switch | |
| 16:15:10 | sean-k-mooney | mriedem: :) yes but telco dont upgrade untill the last second. | |
| 16:15:21 | stephenfin | dansmith: Yeah, I've told mschuppert et al internally to not even try automating this in TripleO | |
| 16:15:27 | dansmith | stephenfin: exactly | |
| 16:15:34 | stephenfin | dansmith: But manual human intervention is going to be necessary anyway | |
| 16:15:38 | mriedem | oh right, openstack's only consumer, telco's | |
| 16:15:55 | dansmith | stephenfin: I don't think that's a given | |
| 16:15:59 | stephenfin | yeah, it is | |
| 16:16:07 | dansmith | alright, well, end of discussion then huh? | |
| 16:16:09 | sean-k-mooney | stephenfin: well they will be automatining it as a seperate step that you run but the full detail are tbd | |
| 16:16:28 | stephenfin | wait, I'm preparing my longer answer :) | |
| 16:16:50 | stephenfin | we can't tell if a host is intended for pinned workloads, unpinned workloads or (bad!) both | |
| 16:17:00 | dansmith | if converting from the current cpuset config to the new one is not something a computer can automate, then this is all unreasonable, IMHO | |
| 16:17:21 | stephenfin | so we can't therefore tell whether we should be mapping 'vcpu_pin_set' to '[compute] cpu_dedicated_set' or '[compute] cpu_shared_set' | |
| 16:17:30 | stephenfin | assuming 'vcpu_pin_set' is even set, which it doesn't have to be | |
| 16:17:35 | mriedem | stephenfin: couldn't we detect that if the compute was reporting a trait saying what it's configured for? | |
| 16:17:52 | dansmith | mriedem: he's saying the config doesn't currently include the intended behavior | |
| 16:18:10 | dansmith | which is fine, the operator may have to tell the tool what they're using their pinning for, | |
| 16:18:15 | stephenfin | mriedem: To paraphrase a kid with a spoon, there is no trait | |
| 16:18:21 | dansmith | but the actual conversion of the formats does not need to be hand-edited everywhere | |
| 16:18:37 | mriedem | stephenfin: i don't know what that means (the spoon kid thing) but sure there is never a trait unless we add one | |
| 16:18:50 | dansmith | matrix | |
| 16:18:52 | dansmith | can't believe you didn't get a 90s movie reference dude | |
| 16:18:55 | mriedem | my point was, if the control plane needs to know things about how the compute is configured/supported, then we use traits for that now | |
| 16:19:01 | mriedem | i'm not a matrix fanboy | |
| 16:19:27 | stephenfin | mriedem: We don't need a trait though - we have resources | |
| 16:19:34 | dansmith | right, that's not really the problem | |
| 16:20:32 | dansmith | the controller side of this seems easy to make flexible enough to handle the rolling config of computes to me | |
| 16:20:32 | stephenfin | the problem is that we're going from a world where two different types of resource have been munged together, and we're trying to unmunge them as cleanly as possible | |
| 16:21:06 | stephenfin | using service versions? | |
| 16:21:13 | dansmith | I would say that the scheduler should ask for the new format by default. If placement returns some options, then we filter and schedule to those if possible. Basically, prefer the upgraded machines | |
| 16:21:21 | dansmith | if we get back no candidates or filter them all out, | |
| 16:21:25 | sean-k-mooney | we are going form a world where teh VCPU reouse was the number of virtual cpus avialabel to it meaning the number of shared cpus | |
| 16:21:44 | dansmith | we check the service version to determine if there are old computes in the deployment. If so, we query again for the older format to see if there's any room that way | |
| 16:21:49 | artom | So... maybe we need entirely new resource names then? | |
| 16:22:00 | dansmith | potentially cache that determination for ten minutes or something, but... | |
| 16:22:15 | mriedem | artom: moving away from VCPU is a non starter to me | |
| 16:22:20 | dansmith | then you can upgrade and convert computes in one step, which is what OSA and other tools are going to want to do.. | |