| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 19:37:01 | spatel | running vm on single numa0 give very high performance compare to running on both numa node (with cpu_socket=2 and cpu_threads=2 option) | |
| 19:37:32 | efried | dansmith: TL;DR: In the world where you say you don't care about NUMA, we give you an instance that's NUMA-ified, but with whatever split we were able to fit. | |
| 19:37:32 | efried | If you migrate that instance, you have to preserve that topo on the target; you can't just munge it to a new shape. | |
| 19:37:32 | efried | Which will then limit where you can fit it on migration. | |
| 19:37:36 | sean-k-mooney | spatel: i was away but just saw it now. fundemtally i think erlang is jut not optimising correctly. im not really sure how to help other then suggestin run more small vms if you can scale out the application horrizontally instead of vertically | |
| 19:37:44 | dansmith | sean-k-mooney: presumably if the flavor on the instance is not numa-aware then we can just find a new host during cold migration and give it a new topology when it moves | |
| 19:37:59 | sean-k-mooney | dansmith: yes we could | |
| 19:38:00 | dansmith | efried: only on cold migration | |
| 19:38:02 | dansmith | sorry | |
| 19:38:05 | dansmith | efried: only on live migration | |
| 19:38:28 | dansmith | efried: on cold migration it could change because you're rebooting and there shouldn't be anything in the guest that persistently cares what the topology is.. | |
| 19:38:29 | efried | Okay, that would be a new limitation. | |
| 19:38:32 | spatel | sean-k-mooney: totally understand but i am going to loose 16 cpu in that case. but anyway i can live with that | |
| 19:38:45 | dansmith | efried: we have that limitation today and it's handled by the numa live migration stuff, AFAIK | |
| 19:39:34 | efried | dansmith: oh, my understanding was that, by lying to the guest and saying it only has one NUMA node, regardless of how many are under the covers, we can change the under-the-covers on live migration without "affecting" the guest. It would still have crappy performance on both sides, but it wouldn't notice that anything had changed. | |
| 19:39:38 | sean-k-mooney | efried: well the limitation would be dont change the view of hardwar the vm sees in live migration | |
| 19:39:46 | dansmith | efried: and selecting a host in that situation should be the same as selecting a host for a new instance boot where the flavor cares deeply about the topology | |
| 19:39:47 | sean-k-mooney | when framed that way its what we do today | |
| 19:40:20 | dansmith | efried: only insofar as it is unaware of how stupid it's being, regardless of what is underneath | |
| 19:40:30 | sean-k-mooney | if we lye to the vm so it only sees 1 numa node we can in some cases change the mapping underneth yes | |
| 19:41:03 | dansmith | efried: so yes, if all guests are numa-aware then migrating indifferent ones becomes a little more restrictive, but that's the same goal as not pretending this stuff doesn't exist at boot time, IMHO | |
| 19:41:04 | efried | dansmith: okay, I understand and agree with that; I'm just questioning whether that's going to effectively spike the chances of NVH trying to live migrate a NUMA-agnostic instance. | |
| 19:41:42 | efried | sounds like the answer is yes, and we're okay with that. | |
| 19:41:46 | dansmith | efried: it may increase the difficulty of moving things, yes | |
| 19:42:13 | sean-k-mooney | efried: well in a non full clould it very likely that the numa=1 case will just work | |
| 19:42:18 | sean-k-mooney | unless the vm is very large | |
| 19:42:23 | dansmith | efried: I refer to the documented goal of not trying to schedule the last byte of memory | |
| 19:42:32 | dansmith | yep | |
| 19:42:36 | sean-k-mooney | and in that case the numa=2 case is likely to work | |
| 19:42:47 | sean-k-mooney | so i dont think it would spike much | |
| 19:43:03 | dansmith | small guests are less likely to care about numa, and thus more likely to fit into numa=1, and thus more likely to be easily movable | |
| 19:43:23 | dansmith | large guests are more likely to need numa for proper performance and have the moving restriction today | |
| 19:44:34 | sean-k-mooney | efried: dansmith im wondering if we could/should have a second spec or defer this to the implematnion | |
| 19:44:53 | efried | This spec needs to say whether we're going to try to do the progressive splitting thing. | |
| 19:45:03 | sean-k-mooney | e.g. if we agree withthe proposed placmenet modeling should we decied how to do the query splitting seperatly | |
| 19:45:11 | efried | But that's not the only factor in play. We also need to address the partially-upgraded cloud. | |
| 19:45:13 | sean-k-mooney | vvs fallbackj | |
| 19:45:50 | sean-k-mooney | so with the progress spilting i think we still need the fallback | |
| 19:45:54 | sean-k-mooney | to cover that case | |
| 19:46:13 | sean-k-mooney | the fall back is the only thing that can ever land on the non upgraded hosts | |
| 19:46:36 | efried | I'm not sure we should do the fallback thing at all. | |
| 19:46:48 | efried | Not if there's a way we can simply avoid doing the translation until all computes are upgraded. | |
| 19:47:03 | sean-k-mooney | we cant without a global config | |
| 19:47:11 | sean-k-mooney | which is why we have the fallback for pcpus | |
| 19:47:26 | efried | I thought the control plane was able to tell which computes were at what level? | |
| 19:47:27 | dansmith | efried: we have to do the fallback no? | |
| 19:47:29 | efried | By RPC something something? | |
| 19:47:33 | dansmith | efried: it can | |
| 19:47:34 | sean-k-mooney | it was a gloabl config untill we agreed to do the fallback at the end of train | |
| 19:48:01 | efried | dansmith: isn't that going to end up violating pack/spread weighing and server affinity groups, because it will always favor upgraded/reshaped computes? | |
| 19:48:38 | dansmith | yes? | |
| 19:49:01 | efried | I mean, I see your point, that you really can't upgrade a compute *and* reshape it unless the scheduler is going to do some translating. | |
| 19:49:04 | dansmith | or, do both queries, merge the two results and let the filters/weighers decide? | |
| 19:49:11 | efried | gross. But yeah. | |
| 19:49:33 | dansmith | doesn't seem more gross to me.. two queries, yes, but everything will start with two queries effectively | |
| 19:50:10 | sean-k-mooney | dansmith: do we merge them for PCPUs or is there a reason we dont? | |
| 19:50:17 | dansmith | idk | |
| 19:50:19 | efried | IIRC we don't | |
| 19:50:31 | efried | we do one, then if no results, we do the other | |
| 19:50:47 | sean-k-mooney | right which by the way we still do | |
| 19:51:05 | efried | as for a reason, probably just didn't think about the weighing etc. | |
| 19:51:25 | sean-k-mooney | no i remebere this coming up | |
| 19:51:42 | sean-k-mooney | i just dont recally why we chose to/not to merge them | |
| 19:52:35 | sean-k-mooney | so is the poposal always do both queries and merge them or to the progressive spliting | |
| 19:53:17 | efried | Both | |
| 19:53:22 | efried | because | |
| 19:53:40 | efried | we want upgraded hosts to cut over to NUMA-modeled by default. | |
| 19:53:52 | sean-k-mooney | ok so progressive for non numa and both qureies for numa | |
| 19:54:17 | efried | I think we need two queries for both. | |
| 19:54:41 | sean-k-mooney | i think it will be more like 2 for numa and 5 for non numa | |
| 19:54:46 | efried | yeah | |
| 19:55:03 | sean-k-mooney | 5 asumign we dint max_numa_nodes=5 | |
| 19:55:08 | efried | 4 | |
| 19:55:15 | sean-k-mooney | sorry 4 | |
| 19:55:25 | efried | I asked this earlier: what's the max number of NUMA nodes we know of on any system? Is it 4? | |
| 19:55:28 | sean-k-mooney | could we bail out if the first numa query passed | |
| 19:55:34 | efried | yes | |
| 19:56:35 | sean-k-mooney | on a 32-64 core 2 socket amd eypci host with a numa node per l3 region 16-32 numa node | |
| 19:56:56 | efried | ye gods | |
| 19:57:04 | sean-k-mooney | if you dont expose a numa node per l3 region then 8 is more realisitc | |
| 19:57:15 | efried | Can we tell from the db? | |
| 19:57:32 | efried | I guess we could tell by querying placement. | |
| 19:57:34 | sean-k-mooney | its in teh hohst numa toplogy blob in the cell db | |
| 19:57:53 | sean-k-mooney | so yes technically. | |
| 19:58:08 | efried | but I don't know that we really want to go discovering that every time we schedule. | |
| 19:58:24 | sean-k-mooney | every time defintly not | |
| 19:58:31 | sean-k-mooney | schduler config option? | |
| 19:58:33 | efried | configurable [scheduler]max_implicit_numa_nodes ? | |
| 19:58:36 | efried | yeah | |
| 19:58:51 | efried | because even if it's possible to have a 32-way split, that would almost never be a good idea probably. | |
| 19:58:57 | sean-k-mooney | set it to 0 to diable numa qureis entrily otherwise its the limit | |
| 19:59:17 | sean-k-mooney | efried: it give you almost a 35% performace boost | |
| 19:59:19 | efried | "disable" meaning what? | |
| 19:59:43 | sean-k-mooney | efried: disabel mean i have disable numa reporting in my entire cloud so dont even try | |
| 20:00:13 | efried | Need to grok how that query would be different from the non-upgraded query. | |
| 20:00:24 | sean-k-mooney | it would be the same | |
| 20:00:32 | sean-k-mooney | just want we do today in tran | |
| 20:00:33 | efried | Oh, because it would have the HW_NUMA_ROOT trait. | |
| 20:01:09 | sean-k-mooney | well require=!HW_NUMA_ROOT would be the detla form train i guess | |
| 20:01:22 | sean-k-mooney | but if no host reports numa thats a noop | |