| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 17:08:14 | bauzas | we can't ask operators to modify their configs *before* they upgrade or *before* they restart their clouds | |
| 17:08:20 | stephenfin | from what CERN are saying, they're already dividing their hosts into those for NUMA and those for not NUMA | |
| 17:08:23 | efried | nope. We're not asking that. | |
| 17:08:42 | bauzas | efried: I know, but what you propose will frighten them | |
| 17:08:52 | stephenfin | so I'm not sure why we can't do the same in placement | |
| 17:08:55 | bauzas | because, we flip to NUMA everywhere | |
| 17:09:18 | bauzas | for CERN, that would mean non-NUMA cells would be NUMA-speaking from Ussuri | |
| 17:09:26 | efried | which will happen at some point anyway. | |
| 17:09:37 | bauzas | and they would have to let them speak... what? after this | |
| 17:09:53 | bauzas | efried: I don't disagree with you, and I think this could be Victoria | |
| 17:09:59 | efried | if we default 'off', nobody is going to switch it on. Then in V (or whenever) we switch the default and have this issue. | |
| 17:10:01 | stephenfin | if you care about NUMA affinity, configure things so placement speaks NUMA, otherwise YAGNI | |
| 17:10:06 | bauzas | but not Ussuri | |
| 17:10:23 | bauzas | stephenfin: that's what I propose | |
| 17:10:31 | stephenfin | it seems so much simpler | |
| 17:11:13 | bauzas | honestly, my vision of the work to do is : | |
| 17:11:48 | bauzas | 'plumb, plumb, plumb things on one side, and mark this feature as opt-in' | |
| 17:12:33 | bauzas | in the eventuality of a very bad situation close to RC1, then we just add an 'EXPERIMENTAL' flag on the option | |
| 17:12:38 | bauzas | boom, problem solved. | |
| 17:12:38 | efried | stephenfin: so in that scenario, you upgrade your control plane, and then any hw:numa*-havin flavors will simply refuse to land until you've upgraded *and* opted-in some hosts. | |
| 17:12:52 | stephenfin | no | |
| 17:12:59 | efried | or that's what the 'fallback' query is for | |
| 17:13:12 | stephenfin | yeah, short term fallback query like we do for PCPU | |
| 17:13:24 | efried | so, still you're doing two queries and either merging the results or violating pack/spread and server affinity groups. | |
| 17:14:12 | stephenfin | with a big ass warning saying "you're using this host for NUMA instances - update configuration now or perish in a future release" | |
| 17:14:22 | stephenfin | yeah, but some hosts can choose to never opt-in | |
| 17:14:25 | stephenfin | because they don't care | |
| 17:14:25 | efried | And you can land NUMA-aware flavors on either kind of host | |
| 17:14:35 | bauzas | yeah, that's my thoughts | |
| 17:14:44 | efried | What about NUMA-agnostic flavors? Those can only land on un-upgraded or un-reshaped hosts, right? | |
| 17:14:47 | efried | So one-way segregation? | |
| 17:15:02 | stephenfin | they never boot pinned instances and their instance floats across all (enabled) host cores as before | |
| 17:15:07 | bauzas | efried: non-NUMA would stick with non-NUMA hosts | |
| 17:15:15 | stephenfin | non-NUMA or non-upgraded | |
| 17:15:19 | bauzas | right | |
| 17:15:22 | stephenfin | because we can't distinguish | |
| 17:15:38 | bauzas | correct, we just say "not those hosts" | |
| 17:15:45 | efried | well, we could distinguish if we wanted to. | |
| 17:15:49 | bauzas | thru a forbidden triat | |
| 17:16:05 | stephenfin | not without operator intervention | |
| 17:16:33 | stephenfin | the operator would have to do something to say "this host is intended to be a non-NUMA host" | |
| 17:16:39 | efried | we could make the segregation complete by simply adding a trait to (even unreshaped) U hosts. | |
| 17:17:06 | stephenfin | how do you tell the difference between unreshaped and intentionally non-NUMA hosts? | |
| 17:17:17 | efried | unreshaped U is intentionally non-NUMA. | |
| 17:17:36 | stephenfin | it can't be - you'd break upgrades | |
| 17:18:00 | efried | um | |
| 17:18:01 | efried | yes | |
| 17:18:05 | efried | that's what we're talking about doing. | |
| 17:18:12 | stephenfin | the query for NUMA-based instances in U would be "all NUMA hosts + all unreshaped hosts" | |
| 17:18:34 | stephenfin | the query for non-NUMA-based instances would be "all non-NUMA hosts + all unreshaped hosts" | |
| 17:18:59 | efried | or "all NUMA U hosts + all unreshaped pre-U hosts" and "all non-NUMA U hosts + all unreshaped pre-U hosts" | |
| 17:19:14 | efried | because I thought we were trying to segregate from U+ | |
| 17:19:45 | stephenfin | again, you'll break upgrades | |
| 17:19:57 | efried | how so? | |
| 17:20:24 | stephenfin | you might not be reshaping | |
| 17:20:45 | stephenfin | if NUMA'ness if optional long term | |
| 17:21:08 | stephenfin | by V, a host will identify itself as either caring about NUMA or not caring | |
| 17:21:34 | stephenfin | but before then, we're in an uncertain state where the host _might_ be NUMA or might not | |
| 17:21:49 | stephenfin | and we'd need the operator to do something to tell us which one it is | |
| 17:21:54 | bauzas | stephenfin: if the operator doesn't reshape, then all hosts are non-NUMA | |
| 17:22:05 | bauzas | we don't need to distinguish them | |
| 17:22:42 | lyarwood | stephenfin: thanks, just sent some comments back. FWIW it's part of this bugfix series https://review.opendev.org/#/q/topic:bug/1861071 | |
| 17:22:53 | bauzas | it's just that we gonna add a specific forbidden trait for ensuring either way that non-NUMA instances can't land on NUMA hosts | |
| 17:23:16 | stephenfin | bauzas: how will you ever kill the fallback query in that case? | |
| 17:23:21 | bauzas | if the operator starts definining NUMA hosts, then he will shard its cloud, but I'm cool with it | |
| 17:24:06 | bauzas | stephenfin: the failback query should only be for 'NUMA-aware' instances | |
| 17:24:18 | bauzas | .... aaaaand I probably messed this up | |
| 17:24:24 | bauzas | (in the last rev of the spec) | |
| 17:24:26 | stephenfin | we want to make sure a non-NUMA instance will not land on a NUMA host, but long term shouldn't we also make sure a NUMA instance won't land on a non-NUMA host? | |
| 17:24:54 | bauzas | stephenfin: yeah | |
| 17:25:16 | stephenfin | okay, then you need to find some way to indicate that yes, this *really* is a non-NUMA host | |
| 17:26:31 | stephenfin | that way your queries can be "give me all NUMA hosts and all unconfigured hosts, but *not* any non-NUMA hosts" | |
| 17:26:35 | stephenfin | and vice versa | |
| 17:26:36 | stephenfin | right? | |
| 17:26:51 | bauzas | sec, wrapping up things in my mind | |
| 17:27:05 | efried | so by your proposal, we actually need a three-way conf opt in U. | |
| 17:27:07 | bauzas | there are two timeframes in my mind | |
| 17:27:27 | bauzas | Ussuri where hosts can be unconfigured | |
| 17:27:36 | bauzas | (because default is no reshape) | |
| 17:27:38 | stephenfin | in a future release, those would simply become "give me all NUMA hosts" or "give me all non-NUMA hosts", depending on your instance type | |
| 17:27:49 | bauzas | Victoria where all hosts are configured | |
| 17:27:50 | stephenfin | efried: yeah, I was thinking a boolean that defaults to None | |
| 17:27:57 | stephenfin | I think we can do that | |
| 17:28:02 | stephenfin | none/unset | |
| 17:28:15 | efried | - "This host is NUMA" ==> reshape, only land hw:numa* flavors | |
| 17:28:15 | efried | - "This host is not NUMA" ==> no reshape, only land non-hw:numa* flavors | |
| 17:28:15 | efried | - None (default in U) ==> no reshape, looks just like a T host, land either type of flavor | |
| 17:28:27 | stephenfin | yup | |
| 17:28:32 | bauzas | I can write this | |
| 17:28:37 | efried | and then, what, make None illegal in V?? | |
| 17:28:47 | bauzas | efried: I'm cool with it | |
| 17:28:50 | efried | Thus breaking upgrades?? | |
| 17:28:54 | bauzas | nope | |
| 17:28:55 | stephenfin | V, W, X, ... at some point in the future | |
| 17:28:56 | bauzas | becaue | |
| 17:29:00 | bauzas | because, | |
| 17:29:04 | bauzas | we can test things | |
| 17:29:18 | bauzas | and see 'okay, look, this is harmless' | |
| 17:29:28 | bauzas | so, once we all agree, we remove the None value | |
| 17:29:29 | stephenfin | essentially this would become one of the things you have to configure | |