| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 19:42:36 | sean-k-mooney | and in that case the numa=2 case is likely to work | |
| 19:42:47 | sean-k-mooney | so i dont think it would spike much | |
| 19:43:03 | dansmith | small guests are less likely to care about numa, and thus more likely to fit into numa=1, and thus more likely to be easily movable | |
| 19:43:23 | dansmith | large guests are more likely to need numa for proper performance and have the moving restriction today | |
| 19:44:34 | sean-k-mooney | efried: dansmith im wondering if we could/should have a second spec or defer this to the implematnion | |
| 19:44:53 | efried | This spec needs to say whether we're going to try to do the progressive splitting thing. | |
| 19:45:03 | sean-k-mooney | e.g. if we agree withthe proposed placmenet modeling should we decied how to do the query splitting seperatly | |
| 19:45:11 | efried | But that's not the only factor in play. We also need to address the partially-upgraded cloud. | |
| 19:45:13 | sean-k-mooney | vvs fallbackj | |
| 19:45:50 | sean-k-mooney | so with the progress spilting i think we still need the fallback | |
| 19:45:54 | sean-k-mooney | to cover that case | |
| 19:46:13 | sean-k-mooney | the fall back is the only thing that can ever land on the non upgraded hosts | |
| 19:46:36 | efried | I'm not sure we should do the fallback thing at all. | |
| 19:46:48 | efried | Not if there's a way we can simply avoid doing the translation until all computes are upgraded. | |
| 19:47:03 | sean-k-mooney | we cant without a global config | |
| 19:47:11 | sean-k-mooney | which is why we have the fallback for pcpus | |
| 19:47:26 | efried | I thought the control plane was able to tell which computes were at what level? | |
| 19:47:27 | dansmith | efried: we have to do the fallback no? | |
| 19:47:29 | efried | By RPC something something? | |
| 19:47:33 | dansmith | efried: it can | |
| 19:47:34 | sean-k-mooney | it was a gloabl config untill we agreed to do the fallback at the end of train | |
| 19:48:01 | efried | dansmith: isn't that going to end up violating pack/spread weighing and server affinity groups, because it will always favor upgraded/reshaped computes? | |
| 19:48:38 | dansmith | yes? | |
| 19:49:01 | efried | I mean, I see your point, that you really can't upgrade a compute *and* reshape it unless the scheduler is going to do some translating. | |
| 19:49:04 | dansmith | or, do both queries, merge the two results and let the filters/weighers decide? | |
| 19:49:11 | efried | gross. But yeah. | |
| 19:49:33 | dansmith | doesn't seem more gross to me.. two queries, yes, but everything will start with two queries effectively | |
| 19:50:10 | sean-k-mooney | dansmith: do we merge them for PCPUs or is there a reason we dont? | |
| 19:50:17 | dansmith | idk | |
| 19:50:19 | efried | IIRC we don't | |
| 19:50:31 | efried | we do one, then if no results, we do the other | |
| 19:50:47 | sean-k-mooney | right which by the way we still do | |
| 19:51:05 | efried | as for a reason, probably just didn't think about the weighing etc. | |
| 19:51:25 | sean-k-mooney | no i remebere this coming up | |
| 19:51:42 | sean-k-mooney | i just dont recally why we chose to/not to merge them | |
| 19:52:35 | sean-k-mooney | so is the poposal always do both queries and merge them or to the progressive spliting | |
| 19:53:17 | efried | Both | |
| 19:53:22 | efried | because | |
| 19:53:40 | efried | we want upgraded hosts to cut over to NUMA-modeled by default. | |
| 19:53:52 | sean-k-mooney | ok so progressive for non numa and both qureies for numa | |
| 19:54:17 | efried | I think we need two queries for both. | |
| 19:54:41 | sean-k-mooney | i think it will be more like 2 for numa and 5 for non numa | |
| 19:54:46 | efried | yeah | |
| 19:55:03 | sean-k-mooney | 5 asumign we dint max_numa_nodes=5 | |
| 19:55:08 | efried | 4 | |
| 19:55:15 | sean-k-mooney | sorry 4 | |
| 19:55:25 | efried | I asked this earlier: what's the max number of NUMA nodes we know of on any system? Is it 4? | |
| 19:55:28 | sean-k-mooney | could we bail out if the first numa query passed | |
| 19:55:34 | efried | yes | |
| 19:56:35 | sean-k-mooney | on a 32-64 core 2 socket amd eypci host with a numa node per l3 region 16-32 numa node | |
| 19:56:56 | efried | ye gods | |
| 19:57:04 | sean-k-mooney | if you dont expose a numa node per l3 region then 8 is more realisitc | |
| 19:57:15 | efried | Can we tell from the db? | |
| 19:57:32 | efried | I guess we could tell by querying placement. | |
| 19:57:34 | sean-k-mooney | its in teh hohst numa toplogy blob in the cell db | |
| 19:57:53 | sean-k-mooney | so yes technically. | |
| 19:58:08 | efried | but I don't know that we really want to go discovering that every time we schedule. | |
| 19:58:24 | sean-k-mooney | every time defintly not | |
| 19:58:31 | sean-k-mooney | schduler config option? | |
| 19:58:33 | efried | configurable [scheduler]max_implicit_numa_nodes ? | |
| 19:58:36 | efried | yeah | |
| 19:58:51 | efried | because even if it's possible to have a 32-way split, that would almost never be a good idea probably. | |
| 19:58:57 | sean-k-mooney | set it to 0 to diable numa qureis entrily otherwise its the limit | |
| 19:59:17 | sean-k-mooney | efried: it give you almost a 35% performace boost | |
| 19:59:19 | efried | "disable" meaning what? | |
| 19:59:43 | sean-k-mooney | efried: disabel mean i have disable numa reporting in my entire cloud so dont even try | |
| 20:00:13 | efried | Need to grok how that query would be different from the non-upgraded query. | |
| 20:00:24 | sean-k-mooney | it would be the same | |
| 20:00:32 | sean-k-mooney | just want we do today in tran | |
| 20:00:33 | efried | Oh, because it would have the HW_NUMA_ROOT trait. | |
| 20:01:09 | sean-k-mooney | well require=!HW_NUMA_ROOT would be the detla form train i guess | |
| 20:01:22 | sean-k-mooney | but if no host reports numa thats a noop | |
| 20:01:55 | efried | In this new picture, I'm not sure we need/want to forbid that trait. | |
| 20:02:09 | sean-k-mooney | we dont | |
| 20:02:16 | sean-k-mooney | if we are allowed to invent numa toplogies | |
| 20:02:26 | sean-k-mooney | we could specificly to land on un upgraded hosts | |
| 20:02:30 | sean-k-mooney | but it has no other use | |
| 20:02:55 | efried | in this case we don't *want* to target un-upgraded hosts. | |
| 20:03:05 | efried | we want to *allow* landing there, but not *force* it ever. | |
| 20:03:18 | sean-k-mooney | sure | |
| 20:04:08 | efried | hum, but what we don't want is to land across numa nodes on an upgraded host. So yeah, I think the 'fallback' query in both cases should have !HW_NUMA_ROOT. | |
| 20:04:23 | sean-k-mooney | so [scheduler]max_implicit_numa_nodes=0 means dont add HW_NUMA_ROOT or granular groups, anyting above 0 is the amount of numa nodes to try for progressive spliting | |
| 20:04:51 | efried | yeah. But only for NUMA-agnostic flavors. For NUMA-aware flavors, we have an explicit number of nodes we're trying for. | |
| 20:05:03 | sean-k-mooney | yes | |
| 20:05:09 | efried | Do you have it in you to write this up, sean-k-mooney? | |
| 20:05:27 | efried | I made a start on it, but I need to step away for... possibly the rest of the day. | |
| 20:05:32 | sean-k-mooney | ill start a second etherpad. | |
| 20:05:45 | sean-k-mooney | or if you have a start put it in one and i can extend it. | |
| 20:06:01 | sean-k-mooney | efried: do we still need teh per host config option in this model | |
| 20:06:15 | sean-k-mooney | i think no if we do the progressive spliting | |
| 20:06:34 | sean-k-mooney | but we might want to for perfomcne reasin if we jsut want to turn it off | |
| 20:06:58 | efried | yes we do, because sometimes the progressive splitting won't get a result, and they want to force a host to behave like a Train host. | |
| 20:07:15 | sean-k-mooney | ok | |
| 20:07:17 | efried | but that's why the workaround is *off* by default. You have to really need it to turn it on. | |
| 20:07:35 | sean-k-mooney | ok that makes sense | |
| 20:07:59 | sean-k-mooney | dansmith: would you be oke with a [scheduler]/max_implicit_numa_nodes config option | |
| 20:08:13 | sean-k-mooney | to contol the progessive spliting | |
| 20:08:34 | efried | "config-driven API behavior" warning. Not sure I see a better alternative though. | |
| 20:09:26 | sean-k-mooney | efried: well the virt driver can today dowhatever the hell it like in this case anyway so im not sure its an observable thing | |
| 20:09:34 | sean-k-mooney | at least form the api perspctive | |