| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 15:58:37 | mriedem | mnaser: there was one which no operators read :) | |
| 15:58:43 | mriedem | b/c it was'nt tagged for [nova] or [ops] | |
| 15:58:43 | sean-k-mooney | so the pmem code is tacking a shortcut by updating the flavor. on a hard reboot that instance would be pinned | |
| 15:59:07 | stephenfin | sean-k-mooney: I'm looking at that at the moment. I don't like it. | |
| 15:59:09 | stephenfin | Not at all | |
| 15:59:31 | stephenfin | Assuming you're referring to https://review.opendev.org/#/c/678455/25/nova/virt/libvirt/driver.py@5458 | |
| 15:59:33 | mriedem | stephenfin: for the sake of everyone's clarity, could you post a new ML thread with the proposed upgrade path for PCPU and tag with [nova] and [ops]? | |
| 15:59:36 | sean-k-mooney | stephenfin: if we need to create a numa toplogy we should move it to where we do it for hugepages right? | |
| 15:59:56 | sean-k-mooney | stephenfin: yes | |
| 16:00:03 | sean-k-mooney | stephenfin: that is the hack that i dont like | |
| 16:00:05 | stephenfin | sean-k-mooney: exactly what I'm writing in a comment as we speak | |
| 16:00:19 | stephenfin | what is it with people trying to hack flavors :D | |
| 16:00:22 | stephenfin | mriedem: sure | |
| 16:00:35 | stephenfin | though I really don't see the point | |
| 16:00:42 | dansmith | stephenfin: "intractable" is a bit of a silly characterization :) | |
| 16:00:43 | alex_xu | sean-k-mooney: yea, agree with you | |
| 16:00:50 | stephenfin | because the only people that can solve this are in this channel/on the review already | |
| 16:01:42 | sean-k-mooney | stephenfin: the issue is that we want a numa topology in the xml. but not in the numa toplogy filter | |
| 16:01:55 | stephenfin | dansmith: Possibly :) I have been thinking about this for quite some time though and we've gone through a lot of options, so it starts looking like that to me, heh | |
| 16:03:16 | sean-k-mooney | alex_xu: since stephenfin is looking at it im gong to review his cpu code then ill come back to this after i test artoms code | |
| 16:03:19 | mriedem | stephenfin: saying "because the only people that can solve this are in this channel/on the review already" is not true imo - if you've got a hard upgrade thing coming for operators, you likely should get some feedback from them before pushing forward | |
| 16:03:28 | alex_xu | sean-k-mooney: thanks a lot | |
| 16:04:07 | sean-k-mooney | mriedem: the upgrade will be signifcantly harder if we also have to deal with numa in placment in the same release | |
| 16:04:26 | mriedem | sean-k-mooney: i don't know what that has to do with this at all | |
| 16:04:34 | sean-k-mooney | mriedem: if we defer pcpus in placment to U we will have to deal with both in one go | |
| 16:04:55 | mriedem | i didn't say anything about deferring | |
| 16:05:33 | mriedem | i said, does anyone outside of the 3 people reviewing this that will actually have to deal with the upgrade know what the plan is | |
| 16:05:36 | mriedem | and are they ok with it | |
| 16:05:39 | sean-k-mooney | right but the current upgrade approch is the best we could come up with and we went to the MLs and asked if the toggel was ok | |
| 16:05:54 | mriedem | and no operators even saw that thread, | |
| 16:06:06 | mriedem | which is why i asked (again) if it could be posed with a [nova][ops] tag | |
| 16:06:08 | mriedem | to get visibility | |
| 16:06:18 | mriedem | lack of feedback from operators is not agreement | |
| 16:06:50 | sean-k-mooney | well we did ask cern in irc | |
| 16:06:56 | dansmith | I think it's probably good to get feedback not just from ops,. | |
| 16:07:00 | sean-k-mooney | but i would have liked other to comment too | |
| 16:07:03 | dansmith | but from people that have to do this in the deployment tools | |
| 16:07:08 | mriedem | it would be a lot better to know before releasing train that "this sucks but it's not terrible" rather than "this is a no-go for me" | |
| 16:07:14 | dansmith | as this adds at least one more atomic reconfigure/restart of the deployment | |
| 16:07:49 | mriedem | sure, i lump mnaser into the ops and tooling (OSA) camps | |
| 16:08:00 | dansmith | yup | |
| 16:08:04 | stephenfin | I don't see what the actual issue is though | |
| 16:08:10 | sean-k-mooney | dansmith: for what its worth we talked about this internally with our tripleo folks that will be implementing and they were ok and actully prefered the seperate config flip step | |
| 16:08:10 | mriedem | sean-k-mooney: and cern (surya? belmiro?) said what? | |
| 16:08:21 | sean-k-mooney | mriedem: we ask belmiro | |
| 16:08:30 | dansmith | sean-k-mooney: preferred to what/ | |
| 16:08:32 | sean-k-mooney | and he was ok with the config | |
| 16:08:39 | stephenfin | You do your upgrade and nothing changes. At some point after the upgrade, you go tweak knobs on the compute nodes followed by a knob on the scheduler | |
| 16:08:42 | stephenfin | and you're done | |
| 16:08:51 | sean-k-mooney | ill see if i can find the irc logs | |
| 16:08:53 | dansmith | stephenfin: and restart the whole deployment atomically :) | |
| 16:09:03 | stephenfin | no, you don't need to do that | |
| 16:09:09 | dansmith | no? | |
| 16:09:15 | dansmith | you say "immediately" in your comment | |
| 16:09:38 | stephenfin | I said we'd have to do that immediately if I wasn't doing the things I was doing to prevent that | |
| 16:10:00 | sean-k-mooney | dansmith: the alternitve was to do the doble report of resouces as both vcpu and pcpu by the way. and that was not done for a spciric reason i cant rememebr | |
| 16:10:49 | dansmith | sean-k-mooney: yeah, that's a terrible alternative, agreed :) | |
| 16:10:57 | dansmith | "Would you prefer an extra config step or a kick in the nuts?" | |
| 16:11:08 | dansmith | I could get most people to agree to the first | |
| 16:11:19 | sean-k-mooney | it did not require the config and it was self healing but ok | |
| 16:11:58 | dansmith | seems to me that with the current plan, | |
| 16:12:00 | dansmith | after they've upgraded, | |
| 16:12:01 | sean-k-mooney | i think we did not do it because of an issue with reshapes | |
| 16:12:13 | stephenfin | sean-k-mooney: Not reshapes, no | |
| 16:12:17 | stephenfin | http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008501.html | |
| 16:12:20 | dansmith | they have to change all (or some fraction) of their computes to the new config to expose the new resources, | |
| 16:12:21 | dansmith | restart them, | |
| 16:12:36 | dansmith | then tweak the scheduler config to ask for the new thing, then restart those, | |
| 16:12:42 | dansmith | then fix the rest of the computes before running out of capacity, and then restart those | |
| 16:12:43 | dansmith | right? | |
| 16:12:54 | sean-k-mooney | dansmith more or less | |
| 16:12:58 | dansmith | that's the most graceful thing | |
| 16:13:05 | stephenfin | dansmith: exactly, yeah | |
| 16:13:10 | dansmith | which sounds like (a) hard to automate and (b) laborious | |
| 16:13:22 | dansmith | otherwise you're looking for full atomic downtime while you do all that in one go | |
| 16:13:53 | dansmith | I can't imagine OSA is going to decide what the sufficient fraction for conversion is, | |
| 16:13:57 | sean-k-mooney | well for FFU we take down the whole cloud contol plain so that not unprecidented | |
| 16:14:04 | dansmith | convert that set, reconfig/restart control services, etc | |
| 16:14:12 | stephenfin | the scheduler option exists to prevent the need for that atomic upgrade | |
| 16:14:15 | dansmith | sean-k-mooney: this is for rolling one release | |
| 16:14:17 | stephenfin | *exists solely | |
| 16:14:33 | mriedem | sean-k-mooney: vexxhost doesn't need to FFU because they actually don't suck at CD | |
| 16:14:34 | dansmith | stephenfin: which likely only works for the case where the humans decide when to throw that switch | |
| 16:15:10 | sean-k-mooney | mriedem: :) yes but telco dont upgrade untill the last second. | |
| 16:15:21 | stephenfin | dansmith: Yeah, I've told mschuppert et al internally to not even try automating this in TripleO | |
| 16:15:27 | dansmith | stephenfin: exactly | |
| 16:15:34 | stephenfin | dansmith: But manual human intervention is going to be necessary anyway | |
| 16:15:38 | mriedem | oh right, openstack's only consumer, telco's | |
| 16:15:55 | dansmith | stephenfin: I don't think that's a given | |
| 16:15:59 | stephenfin | yeah, it is | |
| 16:16:07 | dansmith | alright, well, end of discussion then huh? | |
| 16:16:09 | sean-k-mooney | stephenfin: well they will be automatining it as a seperate step that you run but the full detail are tbd | |
| 16:16:28 | stephenfin | wait, I'm preparing my longer answer :) | |
| 16:16:50 | stephenfin | we can't tell if a host is intended for pinned workloads, unpinned workloads or (bad!) both | |
| 16:17:00 | dansmith | if converting from the current cpuset config to the new one is not something a computer can automate, then this is all unreasonable, IMHO | |
| 16:17:21 | stephenfin | so we can't therefore tell whether we should be mapping 'vcpu_pin_set' to '[compute] cpu_dedicated_set' or '[compute] cpu_shared_set' | |
| 16:17:30 | stephenfin | assuming 'vcpu_pin_set' is even set, which it doesn't have to be | |
| 16:17:35 | mriedem | stephenfin: couldn't we detect that if the compute was reporting a trait saying what it's configured for? | |
| 16:17:52 | dansmith | mriedem: he's saying the config doesn't currently include the intended behavior | |
| 16:18:10 | dansmith | which is fine, the operator may have to tell the tool what they're using their pinning for, | |