| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 19:38:00 | dansmith | efried: only on cold migration | |
| 19:38:02 | dansmith | sorry | |
| 19:38:05 | dansmith | efried: only on live migration | |
| 19:38:28 | dansmith | efried: on cold migration it could change because you're rebooting and there shouldn't be anything in the guest that persistently cares what the topology is.. | |
| 19:38:29 | efried | Okay, that would be a new limitation. | |
| 19:38:32 | spatel | sean-k-mooney: totally understand but i am going to loose 16 cpu in that case. but anyway i can live with that | |
| 19:38:45 | dansmith | efried: we have that limitation today and it's handled by the numa live migration stuff, AFAIK | |
| 19:39:34 | efried | dansmith: oh, my understanding was that, by lying to the guest and saying it only has one NUMA node, regardless of how many are under the covers, we can change the under-the-covers on live migration without "affecting" the guest. It would still have crappy performance on both sides, but it wouldn't notice that anything had changed. | |
| 19:39:38 | sean-k-mooney | efried: well the limitation would be dont change the view of hardwar the vm sees in live migration | |
| 19:39:46 | dansmith | efried: and selecting a host in that situation should be the same as selecting a host for a new instance boot where the flavor cares deeply about the topology | |
| 19:39:47 | sean-k-mooney | when framed that way its what we do today | |
| 19:40:20 | dansmith | efried: only insofar as it is unaware of how stupid it's being, regardless of what is underneath | |
| 19:40:30 | sean-k-mooney | if we lye to the vm so it only sees 1 numa node we can in some cases change the mapping underneth yes | |
| 19:41:03 | dansmith | efried: so yes, if all guests are numa-aware then migrating indifferent ones becomes a little more restrictive, but that's the same goal as not pretending this stuff doesn't exist at boot time, IMHO | |
| 19:41:04 | efried | dansmith: okay, I understand and agree with that; I'm just questioning whether that's going to effectively spike the chances of NVH trying to live migrate a NUMA-agnostic instance. | |
| 19:41:42 | efried | sounds like the answer is yes, and we're okay with that. | |
| 19:41:46 | dansmith | efried: it may increase the difficulty of moving things, yes | |
| 19:42:13 | sean-k-mooney | efried: well in a non full clould it very likely that the numa=1 case will just work | |
| 19:42:18 | sean-k-mooney | unless the vm is very large | |
| 19:42:23 | dansmith | efried: I refer to the documented goal of not trying to schedule the last byte of memory | |
| 19:42:32 | dansmith | yep | |
| 19:42:36 | sean-k-mooney | and in that case the numa=2 case is likely to work | |
| 19:42:47 | sean-k-mooney | so i dont think it would spike much | |
| 19:43:03 | dansmith | small guests are less likely to care about numa, and thus more likely to fit into numa=1, and thus more likely to be easily movable | |
| 19:43:23 | dansmith | large guests are more likely to need numa for proper performance and have the moving restriction today | |
| 19:44:34 | sean-k-mooney | efried: dansmith im wondering if we could/should have a second spec or defer this to the implematnion | |
| 19:44:53 | efried | This spec needs to say whether we're going to try to do the progressive splitting thing. | |
| 19:45:03 | sean-k-mooney | e.g. if we agree withthe proposed placmenet modeling should we decied how to do the query splitting seperatly | |
| 19:45:11 | efried | But that's not the only factor in play. We also need to address the partially-upgraded cloud. | |
| 19:45:13 | sean-k-mooney | vvs fallbackj | |
| 19:45:50 | sean-k-mooney | so with the progress spilting i think we still need the fallback | |
| 19:45:54 | sean-k-mooney | to cover that case | |
| 19:46:13 | sean-k-mooney | the fall back is the only thing that can ever land on the non upgraded hosts | |
| 19:46:36 | efried | I'm not sure we should do the fallback thing at all. | |
| 19:46:48 | efried | Not if there's a way we can simply avoid doing the translation until all computes are upgraded. | |
| 19:47:03 | sean-k-mooney | we cant without a global config | |
| 19:47:11 | sean-k-mooney | which is why we have the fallback for pcpus | |
| 19:47:26 | efried | I thought the control plane was able to tell which computes were at what level? | |
| 19:47:27 | dansmith | efried: we have to do the fallback no? | |
| 19:47:29 | efried | By RPC something something? | |
| 19:47:33 | dansmith | efried: it can | |
| 19:47:34 | sean-k-mooney | it was a gloabl config untill we agreed to do the fallback at the end of train | |
| 19:48:01 | efried | dansmith: isn't that going to end up violating pack/spread weighing and server affinity groups, because it will always favor upgraded/reshaped computes? | |
| 19:48:38 | dansmith | yes? | |
| 19:49:01 | efried | I mean, I see your point, that you really can't upgrade a compute *and* reshape it unless the scheduler is going to do some translating. | |
| 19:49:04 | dansmith | or, do both queries, merge the two results and let the filters/weighers decide? | |
| 19:49:11 | efried | gross. But yeah. | |
| 19:49:33 | dansmith | doesn't seem more gross to me.. two queries, yes, but everything will start with two queries effectively | |
| 19:50:10 | sean-k-mooney | dansmith: do we merge them for PCPUs or is there a reason we dont? | |
| 19:50:17 | dansmith | idk | |
| 19:50:19 | efried | IIRC we don't | |
| 19:50:31 | efried | we do one, then if no results, we do the other | |
| 19:50:47 | sean-k-mooney | right which by the way we still do | |
| 19:51:05 | efried | as for a reason, probably just didn't think about the weighing etc. | |
| 19:51:25 | sean-k-mooney | no i remebere this coming up | |
| 19:51:42 | sean-k-mooney | i just dont recally why we chose to/not to merge them | |
| 19:52:35 | sean-k-mooney | so is the poposal always do both queries and merge them or to the progressive spliting | |
| 19:53:17 | efried | Both | |
| 19:53:22 | efried | because | |
| 19:53:40 | efried | we want upgraded hosts to cut over to NUMA-modeled by default. | |
| 19:53:52 | sean-k-mooney | ok so progressive for non numa and both qureies for numa | |
| 19:54:17 | efried | I think we need two queries for both. | |
| 19:54:41 | sean-k-mooney | i think it will be more like 2 for numa and 5 for non numa | |
| 19:54:46 | efried | yeah | |
| 19:55:03 | sean-k-mooney | 5 asumign we dint max_numa_nodes=5 | |
| 19:55:08 | efried | 4 | |
| 19:55:15 | sean-k-mooney | sorry 4 | |
| 19:55:25 | efried | I asked this earlier: what's the max number of NUMA nodes we know of on any system? Is it 4? | |
| 19:55:28 | sean-k-mooney | could we bail out if the first numa query passed | |
| 19:55:34 | efried | yes | |
| 19:56:35 | sean-k-mooney | on a 32-64 core 2 socket amd eypci host with a numa node per l3 region 16-32 numa node | |
| 19:56:56 | efried | ye gods | |
| 19:57:04 | sean-k-mooney | if you dont expose a numa node per l3 region then 8 is more realisitc | |
| 19:57:15 | efried | Can we tell from the db? | |
| 19:57:32 | efried | I guess we could tell by querying placement. | |
| 19:57:34 | sean-k-mooney | its in teh hohst numa toplogy blob in the cell db | |
| 19:57:53 | sean-k-mooney | so yes technically. | |
| 19:58:08 | efried | but I don't know that we really want to go discovering that every time we schedule. | |
| 19:58:24 | sean-k-mooney | every time defintly not | |
| 19:58:31 | sean-k-mooney | schduler config option? | |
| 19:58:33 | efried | configurable [scheduler]max_implicit_numa_nodes ? | |
| 19:58:36 | efried | yeah | |
| 19:58:51 | efried | because even if it's possible to have a 32-way split, that would almost never be a good idea probably. | |
| 19:58:57 | sean-k-mooney | set it to 0 to diable numa qureis entrily otherwise its the limit | |
| 19:59:17 | sean-k-mooney | efried: it give you almost a 35% performace boost | |
| 19:59:19 | efried | "disable" meaning what? | |
| 19:59:43 | sean-k-mooney | efried: disabel mean i have disable numa reporting in my entire cloud so dont even try | |
| 20:00:13 | efried | Need to grok how that query would be different from the non-upgraded query. | |
| 20:00:24 | sean-k-mooney | it would be the same | |
| 20:00:32 | sean-k-mooney | just want we do today in tran | |
| 20:00:33 | efried | Oh, because it would have the HW_NUMA_ROOT trait. | |
| 20:01:09 | sean-k-mooney | well require=!HW_NUMA_ROOT would be the detla form train i guess | |
| 20:01:22 | sean-k-mooney | but if no host reports numa thats a noop | |
| 20:01:55 | efried | In this new picture, I'm not sure we need/want to forbid that trait. | |
| 20:02:09 | sean-k-mooney | we dont | |
| 20:02:16 | sean-k-mooney | if we are allowed to invent numa toplogies | |
| 20:02:26 | sean-k-mooney | we could specificly to land on un upgraded hosts | |
| 20:02:30 | sean-k-mooney | but it has no other use | |
| 20:02:55 | efried | in this case we don't *want* to target un-upgraded hosts. | |
| 20:03:05 | efried | we want to *allow* landing there, but not *force* it ever. | |