Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-13
17:03:23 bauzas efried: and I wanted to discuss this with you
17:03:52 bauzas because I'm very afraid of any potential issue we would have in Ussuri
17:03:59 bauzas if we flip to changing the world
17:03:59 efried stephenfin and dansmith need to be involved, but I think they're are on calls rn
17:04:22 bauzas my point is, I don't know the figure but not all clouds care about NUMA
17:04:37 bauzas for those clouds, I'd prefer us to not change their lifes
17:04:51 bauzas and pretending there will be no regressions
17:05:07 efried So dansmith's argument was "don't care about NUMA" does *not* mean "give me shitty performance".
17:05:24 bauzas on the other hand, if we allow a default to be "no-op", then we can work on the NUMA implementation seamlessly and iteratively like we did for Cells v2
17:05:27 efried The [workaround] and reversible reshape gives you the way to deal with regressions.
17:05:29 dansmith I don't really understand.. I thought the agreement was to default the new behavior off for U, not reshape by default, let people opt-in during U and then flip the default (or remove it) for V?
17:05:55 efried ugh, no, if we were doing that, there would be no good motivation to hack the splitting thing in.
17:06:02 efried And also no motivation for operators to opt in.
17:06:07 efried so it would be a waste of effort.
17:06:17 bauzas dansmith: that's what i wrote but looks like the outcome of tuesday's discussion between you, sean-k-mooney and efried was the other way
17:06:26 dansmith bauzas: not that I remember
17:06:37 dansmith but I'll admit to being completely exhausted by this conversation
17:06:41 bauzas efried: dansmith: floor is your
17:06:52 bauzas dansmith: and tbh, me too
17:07:04 efried you people need to do more cardio
17:07:11 bauzas I do gym twice a week
17:07:16 stephenfin lyarwood: comments on https://review.opendev.org/#/c/706880/
17:07:30 bauzas (and skiing, but that's irrelevant)
17:07:49 bauzas efried: anyway, my point is,
17:07:59 stephenfin dansmith, efried, bauzas: so, from the top
17:08:14 bauzas we can't ask operators to modify their configs *before* they upgrade or *before* they restart their clouds
17:08:20 stephenfin from what CERN are saying, they're already dividing their hosts into those for NUMA and those for not NUMA
17:08:23 efried nope. We're not asking that.
17:08:42 bauzas efried: I know, but what you propose will frighten them
17:08:52 stephenfin so I'm not sure why we can't do the same in placement
17:08:55 bauzas because, we flip to NUMA everywhere
17:09:18 bauzas for CERN, that would mean non-NUMA cells would be NUMA-speaking from Ussuri
17:09:26 efried which will happen at some point anyway.
17:09:37 bauzas and they would have to let them speak... what? after this
17:09:53 bauzas efried: I don't disagree with you, and I think this could be Victoria
17:09:59 efried if we default 'off', nobody is going to switch it on. Then in V (or whenever) we switch the default and have this issue.
17:10:01 stephenfin if you care about NUMA affinity, configure things so placement speaks NUMA, otherwise YAGNI
17:10:06 bauzas but not Ussuri
17:10:23 bauzas stephenfin: that's what I propose
17:10:31 stephenfin it seems so much simpler
17:11:13 bauzas honestly, my vision of the work to do is :
17:11:48 bauzas 'plumb, plumb, plumb things on one side, and mark this feature as opt-in'
17:12:33 bauzas in the eventuality of a very bad situation close to RC1, then we just add an 'EXPERIMENTAL' flag on the option
17:12:38 efried stephenfin: so in that scenario, you upgrade your control plane, and then any hw:numa*-havin flavors will simply refuse to land until you've upgraded *and* opted-in some hosts.
17:12:38 bauzas boom, problem solved.
17:12:52 stephenfin no
17:12:59 efried or that's what the 'fallback' query is for
17:13:12 stephenfin yeah, short term fallback query like we do for PCPU
17:13:24 efried so, still you're doing two queries and either merging the results or violating pack/spread and server affinity groups.
17:14:12 stephenfin with a big ass warning saying "you're using this host for NUMA instances - update configuration now or perish in a future release"
17:14:22 stephenfin yeah, but some hosts can choose to never opt-in
17:14:25 efried And you can land NUMA-aware flavors on either kind of host
17:14:25 stephenfin because they don't care
17:14:35 bauzas yeah, that's my thoughts
17:14:44 efried What about NUMA-agnostic flavors? Those can only land on un-upgraded or un-reshaped hosts, right?
17:14:47 efried So one-way segregation?
17:15:02 stephenfin they never boot pinned instances and their instance floats across all (enabled) host cores as before
17:15:07 bauzas efried: non-NUMA would stick with non-NUMA hosts
17:15:15 stephenfin non-NUMA or non-upgraded
17:15:19 bauzas right
17:15:22 stephenfin because we can't distinguish
17:15:38 bauzas correct, we just say "not those hosts"
17:15:45 efried well, we could distinguish if we wanted to.
17:15:49 bauzas thru a forbidden triat
17:16:05 stephenfin not without operator intervention
17:16:33 stephenfin the operator would have to do something to say "this host is intended to be a non-NUMA host"
17:16:39 efried we could make the segregation complete by simply adding a trait to (even unreshaped) U hosts.
17:17:06 stephenfin how do you tell the difference between unreshaped and intentionally non-NUMA hosts?
17:17:17 efried unreshaped U is intentionally non-NUMA.
17:17:36 stephenfin it can't be - you'd break upgrades
17:18:00 efried um
17:18:01 efried yes
17:18:05 efried that's what we're talking about doing.
17:18:12 stephenfin the query for NUMA-based instances in U would be "all NUMA hosts + all unreshaped hosts"
17:18:34 stephenfin the query for non-NUMA-based instances would be "all non-NUMA hosts + all unreshaped hosts"
17:18:59 efried or "all NUMA U hosts + all unreshaped pre-U hosts" and "all non-NUMA U hosts + all unreshaped pre-U hosts"
17:19:14 efried because I thought we were trying to segregate from U+
17:19:45 stephenfin again, you'll break upgrades
17:19:57 efried how so?
17:20:24 stephenfin you might not be reshaping
17:20:45 stephenfin if NUMA'ness if optional long term
17:21:08 stephenfin by V, a host will identify itself as either caring about NUMA or not caring
17:21:34 stephenfin but before then, we're in an uncertain state where the host _might_ be NUMA or might not
17:21:49 stephenfin and we'd need the operator to do something to tell us which one it is
17:21:54 bauzas stephenfin: if the operator doesn't reshape, then all hosts are non-NUMA
17:22:05 bauzas we don't need to distinguish them
17:22:42 lyarwood stephenfin: thanks, just sent some comments back. FWIW it's part of this bugfix series https://review.opendev.org/#/q/topic:bug/1861071
17:22:53 bauzas it's just that we gonna add a specific forbidden trait for ensuring either way that non-NUMA instances can't land on NUMA hosts
17:23:16 stephenfin bauzas: how will you ever kill the fallback query in that case?
17:23:21 bauzas if the operator starts definining NUMA hosts, then he will shard its cloud, but I'm cool with it
17:24:06 bauzas stephenfin: the failback query should only be for 'NUMA-aware' instances
17:24:18 bauzas .... aaaaand I probably messed this up
17:24:24 bauzas (in the last rev of the spec)
17:24:26 stephenfin we want to make sure a non-NUMA instance will not land on a NUMA host, but long term shouldn't we also make sure a NUMA instance won't land on a non-NUMA host?
17:24:54 bauzas stephenfin: yeah
17:25:16 stephenfin okay, then you need to find some way to indicate that yes, this *really* is a non-NUMA host
17:26:31 stephenfin that way your queries can be "give me all NUMA hosts and all unconfigured hosts, but *not* any non-NUMA hosts"
17:26:35 stephenfin and vice versa
17:26:36 stephenfin right?
17:26:51 bauzas sec, wrapping up things in my mind

Earlier   Later