Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-10
16:13:05 stephenfin dansmith: exactly, yeah
16:13:10 dansmith which sounds like (a) hard to automate and (b) laborious
16:13:22 dansmith otherwise you're looking for full atomic downtime while you do all that in one go
16:13:53 dansmith I can't imagine OSA is going to decide what the sufficient fraction for conversion is,
16:13:57 sean-k-mooney well for FFU we take down the whole cloud contol plain so that not unprecidented
16:14:04 dansmith convert that set, reconfig/restart control services, etc
16:14:12 stephenfin the scheduler option exists to prevent the need for that atomic upgrade
16:14:15 dansmith sean-k-mooney: this is for rolling one release
16:14:17 stephenfin *exists solely
16:14:33 mriedem sean-k-mooney: vexxhost doesn't need to FFU because they actually don't suck at CD
16:14:34 dansmith stephenfin: which likely only works for the case where the humans decide when to throw that switch
16:15:10 sean-k-mooney mriedem: :) yes but telco dont upgrade untill the last second.
16:15:21 stephenfin dansmith: Yeah, I've told mschuppert et al internally to not even try automating this in TripleO
16:15:27 dansmith stephenfin: exactly
16:15:34 stephenfin dansmith: But manual human intervention is going to be necessary anyway
16:15:38 mriedem oh right, openstack's only consumer, telco's
16:15:55 dansmith stephenfin: I don't think that's a given
16:15:59 stephenfin yeah, it is
16:16:07 dansmith alright, well, end of discussion then huh?
16:16:09 sean-k-mooney stephenfin: well they will be automatining it as a seperate step that you run but the full detail are tbd
16:16:28 stephenfin wait, I'm preparing my longer answer :)
16:16:50 stephenfin we can't tell if a host is intended for pinned workloads, unpinned workloads or (bad!) both
16:17:00 dansmith if converting from the current cpuset config to the new one is not something a computer can automate, then this is all unreasonable, IMHO
16:17:21 stephenfin so we can't therefore tell whether we should be mapping 'vcpu_pin_set' to '[compute] cpu_dedicated_set' or '[compute] cpu_shared_set'
16:17:30 stephenfin assuming 'vcpu_pin_set' is even set, which it doesn't have to be
16:17:35 mriedem stephenfin: couldn't we detect that if the compute was reporting a trait saying what it's configured for?
16:17:52 dansmith mriedem: he's saying the config doesn't currently include the intended behavior
16:18:10 dansmith which is fine, the operator may have to tell the tool what they're using their pinning for,
16:18:15 stephenfin mriedem: To paraphrase a kid with a spoon, there is no trait
16:18:21 dansmith but the actual conversion of the formats does not need to be hand-edited everywhere
16:18:37 mriedem stephenfin: i don't know what that means (the spoon kid thing) but sure there is never a trait unless we add one
16:18:50 dansmith matrix
16:18:52 dansmith can't believe you didn't get a 90s movie reference dude
16:18:55 mriedem my point was, if the control plane needs to know things about how the compute is configured/supported, then we use traits for that now
16:19:01 mriedem i'm not a matrix fanboy
16:19:27 stephenfin mriedem: We don't need a trait though - we have resources
16:19:34 dansmith right, that's not really the problem
16:20:32 stephenfin the problem is that we're going from a world where two different types of resource have been munged together, and we're trying to unmunge them as cleanly as possible
16:20:32 dansmith the controller side of this seems easy to make flexible enough to handle the rolling config of computes to me
16:21:06 stephenfin using service versions?
16:21:13 dansmith I would say that the scheduler should ask for the new format by default. If placement returns some options, then we filter and schedule to those if possible. Basically, prefer the upgraded machines
16:21:21 dansmith if we get back no candidates or filter them all out,
16:21:25 sean-k-mooney we are going form a world where teh VCPU reouse was the number of virtual cpus avialabel to it meaning the number of shared cpus
16:21:44 dansmith we check the service version to determine if there are old computes in the deployment. If so, we query again for the older format to see if there's any room that way
16:21:49 artom So... maybe we need entirely new resource names then?
16:22:00 dansmith potentially cache that determination for ten minutes or something, but...
16:22:15 mriedem artom: moving away from VCPU is a non starter to me
16:22:20 dansmith then you can upgrade and convert computes in one step, which is what OSA and other tools are going to want to do..
16:22:22 mriedem it's baked into *everything*
16:22:29 dansmith they want to upgrade and fix the config as one step generally
16:23:11 artom mriedem, I'm not saying remove it, I'm saying leave it "legacy CPU resource thing", and come up with new resources to mean shared CPU and dedicated CPU
16:23:19 dansmith artom: -3
16:23:58 artom Anyways, I have 0 context and func tests to fix
16:24:29 sean-k-mooney artom: we talked about SCPU and PCPU in the past but we said no we want to keep VCPU resouces
16:24:37 artom dansmith, hey man, dinner first
16:24:42 sean-k-mooney so way to late to go back that route
16:24:42 stephenfin dansmith: so tl;dr: kill the static scheduler-only config option and instead do "give me PCPU, but if you can't give me PCPU then search for VCPU" instead
16:25:14 dansmith stephenfin: yes, but only the last part of there are old computes around. once you do that one time and find everything is upgraded, stop even doing that check
16:25:27 dansmith stephenfin: remove that compat step in U, no deprecation cycle needed for that
16:25:46 stephenfin What do you mean by old computes?
16:25:51 sean-k-mooney dansmith: if we did that we would need a trait to make the scond query safe
16:26:21 mriedem stephenfin: older than train computes
16:26:25 mriedem based on the nova-compute service rpc api version
16:26:26 dansmith stephenfin: service version will tell you if all computes have been upgraded.. so actually, maybe just always do that in T if no candidates, because they could do the upgrade and the config tweak separately
16:26:26 sean-k-mooney we would need a support_PCPU capablity trait and need to add it as a forbiden trait for the second query
16:26:43 stephenfin that won't work though
16:26:49 dansmith stephenfin: so yeah, what you said
16:26:51 stephenfin because by default we don't report PCPU on Train
16:27:08 stephenfin doing so would force people to set new config options as soon as they upgrade
16:27:14 stephenfin which we can't do
16:27:29 sean-k-mooney stephenfin: if you look at artom migratio ncode we ignore the "can live migrate with numa" config if everythin is upgraded
16:27:40 dansmith stephenfin: that's why I backed away from the version check there
16:28:30 sean-k-mooney that would still require everything to be restated however.
16:28:33 stephenfin sean-k-mooney: right, but that's because the "can live migrate with numa" check only needs to know that code is new enough. it doesn't need the operator to tweak some config first
16:28:38 stephenfin which this does
16:28:58 stephenfin dansmith: Ah, yeah, I missed the "so actually"
16:29:35 dansmith stephenfin: the reason I was heading in that direction, is because:
16:29:36 sean-k-mooney i think if we were to do the double query as i said we need the compute capableity trait to protect against lading on a new host when old hsot are available and we ask for vcpus
16:29:41 mriedem you'd only do that fallback for VCPU if the flavor in the request spec (in the scheduler) has resources:PCPU=x right?
16:30:12 stephenfin sean-k-mooney: not gonna happen - the NUMATopologyFilter or libvirt driver protect us
16:30:15 dansmith stephenfin: I was going to suggest we also always expose the inventory, even if we have to synthesize it from the older config, but I'm guessing you're going to say we'd potentially expose as shared or dedicated and be wrong for the intention of the operator right?
16:30:17 sean-k-mooney mriedem: no we are translating hw:cpu_policy=dedicated into PCPU requests
16:30:38 dansmith stephenfin: require them to convert their configs before U, but expose the new inventory right away
16:30:44 mriedem ok but my point is, just make sure we're not unconditionally doing that fallback re-query
16:30:50 stephenfin sean-k-mooney: because if you land on a Train compute node, that'll have explicitly set 'NUMATopology.pcpuset' to None
16:31:11 dansmith mriedem: yes, only do the fallback query for PCPU things
16:31:19 sean-k-mooney ok we could have issue with the limit paramater on placement
16:31:26 stephenfin and if that's None, we've got nothing to pin to
16:31:33 stephenfin so placement will pass but the filter will fail
16:31:40 sean-k-mooney but if we do the second query with out a limit then ya the numa toplogy filter would prevent that
16:31:49 sean-k-mooney so we dont need the trait
16:32:11 stephenfin dansmith: Yeah, exactly
16:32:17 dansmith I don't think the limit is a problem
16:32:31 dansmith stephenfin: so they express their intent right now how?
16:32:33 sean-k-mooney dansmith: doesnt cern set it to like 15
16:32:43 stephenfin dansmith: My previous solution had been to expose CPU inventory on hosts without the new configuration as both PCPU and VCPU
16:33:09 dansmith sean-k-mooney: we're asking placement for a query that will return things with PCPU resources.. if placement returns nothing then there's nothing that will fit
16:33:10 sean-k-mooney the default limit is 1000 so that should not be a proablem but if its really low then i expect it could be
16:33:13 dansmith sean-k-mooney: their problem is different
16:33:31 sean-k-mooney dansmith: im not talking baout the first query for pcpus

Earlier   Later