| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 17:12:59 | efried | or that's what the 'fallback' query is for | |
| 17:13:12 | stephenfin | yeah, short term fallback query like we do for PCPU | |
| 17:13:24 | efried | so, still you're doing two queries and either merging the results or violating pack/spread and server affinity groups. | |
| 17:14:12 | stephenfin | with a big ass warning saying "you're using this host for NUMA instances - update configuration now or perish in a future release" | |
| 17:14:22 | stephenfin | yeah, but some hosts can choose to never opt-in | |
| 17:14:25 | stephenfin | because they don't care | |
| 17:14:25 | efried | And you can land NUMA-aware flavors on either kind of host | |
| 17:14:35 | bauzas | yeah, that's my thoughts | |
| 17:14:44 | efried | What about NUMA-agnostic flavors? Those can only land on un-upgraded or un-reshaped hosts, right? | |
| 17:14:47 | efried | So one-way segregation? | |
| 17:15:02 | stephenfin | they never boot pinned instances and their instance floats across all (enabled) host cores as before | |
| 17:15:07 | bauzas | efried: non-NUMA would stick with non-NUMA hosts | |
| 17:15:15 | stephenfin | non-NUMA or non-upgraded | |
| 17:15:19 | bauzas | right | |
| 17:15:22 | stephenfin | because we can't distinguish | |
| 17:15:38 | bauzas | correct, we just say "not those hosts" | |
| 17:15:45 | efried | well, we could distinguish if we wanted to. | |
| 17:15:49 | bauzas | thru a forbidden triat | |
| 17:16:05 | stephenfin | not without operator intervention | |
| 17:16:33 | stephenfin | the operator would have to do something to say "this host is intended to be a non-NUMA host" | |
| 17:16:39 | efried | we could make the segregation complete by simply adding a trait to (even unreshaped) U hosts. | |
| 17:17:06 | stephenfin | how do you tell the difference between unreshaped and intentionally non-NUMA hosts? | |
| 17:17:17 | efried | unreshaped U is intentionally non-NUMA. | |
| 17:17:36 | stephenfin | it can't be - you'd break upgrades | |
| 17:18:00 | efried | um | |
| 17:18:01 | efried | yes | |
| 17:18:05 | efried | that's what we're talking about doing. | |
| 17:18:12 | stephenfin | the query for NUMA-based instances in U would be "all NUMA hosts + all unreshaped hosts" | |
| 17:18:34 | stephenfin | the query for non-NUMA-based instances would be "all non-NUMA hosts + all unreshaped hosts" | |
| 17:18:59 | efried | or "all NUMA U hosts + all unreshaped pre-U hosts" and "all non-NUMA U hosts + all unreshaped pre-U hosts" | |
| 17:19:14 | efried | because I thought we were trying to segregate from U+ | |
| 17:19:45 | stephenfin | again, you'll break upgrades | |
| 17:19:57 | efried | how so? | |
| 17:20:24 | stephenfin | you might not be reshaping | |
| 17:20:45 | stephenfin | if NUMA'ness if optional long term | |
| 17:21:08 | stephenfin | by V, a host will identify itself as either caring about NUMA or not caring | |
| 17:21:34 | stephenfin | but before then, we're in an uncertain state where the host _might_ be NUMA or might not | |
| 17:21:49 | stephenfin | and we'd need the operator to do something to tell us which one it is | |
| 17:21:54 | bauzas | stephenfin: if the operator doesn't reshape, then all hosts are non-NUMA | |
| 17:22:05 | bauzas | we don't need to distinguish them | |
| 17:22:42 | lyarwood | stephenfin: thanks, just sent some comments back. FWIW it's part of this bugfix series https://review.opendev.org/#/q/topic:bug/1861071 | |
| 17:22:53 | bauzas | it's just that we gonna add a specific forbidden trait for ensuring either way that non-NUMA instances can't land on NUMA hosts | |
| 17:23:16 | stephenfin | bauzas: how will you ever kill the fallback query in that case? | |
| 17:23:21 | bauzas | if the operator starts definining NUMA hosts, then he will shard its cloud, but I'm cool with it | |
| 17:24:06 | bauzas | stephenfin: the failback query should only be for 'NUMA-aware' instances | |
| 17:24:18 | bauzas | .... aaaaand I probably messed this up | |
| 17:24:24 | bauzas | (in the last rev of the spec) | |
| 17:24:26 | stephenfin | we want to make sure a non-NUMA instance will not land on a NUMA host, but long term shouldn't we also make sure a NUMA instance won't land on a non-NUMA host? | |
| 17:24:54 | bauzas | stephenfin: yeah | |
| 17:25:16 | stephenfin | okay, then you need to find some way to indicate that yes, this *really* is a non-NUMA host | |
| 17:26:31 | stephenfin | that way your queries can be "give me all NUMA hosts and all unconfigured hosts, but *not* any non-NUMA hosts" | |
| 17:26:35 | stephenfin | and vice versa | |
| 17:26:36 | stephenfin | right? | |
| 17:26:51 | bauzas | sec, wrapping up things in my mind | |
| 17:27:05 | efried | so by your proposal, we actually need a three-way conf opt in U. | |
| 17:27:07 | bauzas | there are two timeframes in my mind | |
| 17:27:27 | bauzas | Ussuri where hosts can be unconfigured | |
| 17:27:36 | bauzas | (because default is no reshape) | |
| 17:27:38 | stephenfin | in a future release, those would simply become "give me all NUMA hosts" or "give me all non-NUMA hosts", depending on your instance type | |
| 17:27:49 | bauzas | Victoria where all hosts are configured | |
| 17:27:50 | stephenfin | efried: yeah, I was thinking a boolean that defaults to None | |
| 17:27:57 | stephenfin | I think we can do that | |
| 17:28:02 | stephenfin | none/unset | |
| 17:28:15 | efried | - "This host is NUMA" ==> reshape, only land hw:numa* flavors | |
| 17:28:15 | efried | - "This host is not NUMA" ==> no reshape, only land non-hw:numa* flavors | |
| 17:28:15 | efried | - None (default in U) ==> no reshape, looks just like a T host, land either type of flavor | |
| 17:28:27 | stephenfin | yup | |
| 17:28:32 | bauzas | I can write this | |
| 17:28:37 | efried | and then, what, make None illegal in V?? | |
| 17:28:47 | bauzas | efried: I'm cool with it | |
| 17:28:50 | efried | Thus breaking upgrades?? | |
| 17:28:54 | bauzas | nope | |
| 17:28:55 | stephenfin | V, W, X, ... at some point in the future | |
| 17:28:56 | bauzas | becaue | |
| 17:29:00 | bauzas | because, | |
| 17:29:04 | bauzas | we can test things | |
| 17:29:18 | bauzas | and see 'okay, look, this is harmless' | |
| 17:29:28 | bauzas | so, once we all agree, we remove the None value | |
| 17:29:29 | stephenfin | essentially this would become one of the things you have to configure | |
| 17:29:43 | stephenfin | like 'compute_driver' | |
| 17:29:45 | bauzas | and de facto all instances act upon NUMA checking | |
| 17:29:45 | efried | I mean, if we're going to segregate eventually, then at some point we "break upgrades". | |
| 17:30:03 | efried | btw, dansmith specifically said he didn't want two modes long term. | |
| 17:30:36 | stephenfin | yeah, but by that point they'll have had a couple of cycles of warnings saying "yo, you *really* need to set this config option" | |
| 17:30:56 | stephenfin | efried: yeah, I don't understand why that's a bad thing | |
| 17:31:18 | dansmith | I officially give up, please proceed. | |
| 17:31:26 | efried | sigh | |
| 17:31:28 | stephenfin | I get that all instances should have some kind of NUMA awareness | |
| 17:31:30 | efried | okay, back to PS16 | |
| 17:32:51 | efried | stephenfin: tbc, if we go this route, we don't need can_split ever, right? | |
| 17:33:04 | stephenfin | but it's a nice-to-have and I don't imagine everyone really cares | |
| 17:33:26 | stephenfin | efried: correct | |
| 17:33:52 | stephenfin | if we're going with the "everything is mapped to NUMA", then I think we should move the ball forward on 'can_split' instead | |
| 17:34:07 | stephenfin | because if we don't, it won't ever happen :) | |
| 17:34:30 | stephenfin | implement that, then use it for NUMA in V | |
| 17:34:42 | bauzas | folks, you lost me | |
| 17:35:17 | stephenfin | but as cdent saw from the openstack-discuss thread, no one's really asking for their NUMA-based instance to coexist alongside their "I don't care about NUMA"-based instances | |
| 17:35:44 | stephenfin | bauzas: A boolean '[compute] enable_numa' option that default to unset (None) | |
| 17:35:56 | efried | bauzas: that ^, but otherwise PS16. | |
| 17:36:29 | stephenfin | when unset, we start flashing a warning saying "you need to decide if this host is meant for NUMA-based instances or not" | |