| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 19:22:20 | sean-k-mooney | the other way imacpts the perfomce of all non numa instnaces | |
| 19:22:37 | efried | sean-k-mooney: I don't think it has to take all that stuff into account at all. | |
| 19:23:07 | efried | sean-k-mooney: It should behave *exactly* as if you said hw:numa_nodes=$n with no other hw:numa*-isms. | |
| 19:23:40 | efried | (except I think we said we would bounce if we couldn't split evenly; that restriction would have to be lifted for this case.) | |
| 19:24:12 | sean-k-mooney | so we would create a fake flaovr where we overrid that and pass it to the current get numa constratis funct | |
| 19:24:19 | sean-k-mooney | *function | |
| 19:24:25 | efried | if you like | |
| 19:24:33 | efried | that would be the spirit, anyway. | |
| 19:24:36 | sean-k-mooney | i gues that makes it simpler | |
| 19:25:14 | sean-k-mooney | so the get_numa_constraits function will reject any invalid toplogy with regard to even spliting with an excetion | |
| 19:25:32 | sean-k-mooney | so we would just loop and contiue if an excption is raised up to the limit | |
| 19:26:09 | efried | or we relax the constraint to split as close to evenly as possible. Or do that split first. | |
| 19:26:33 | efried | Implementation detail. Point is, it shouldn't be super hard to figure out. | |
| 19:26:49 | sean-k-mooney | we could use the asemetric numa modeling support that is there yes | |
| 19:27:19 | sean-k-mooney | e.g. if you had a 9 core vm and we are on numa=2 do 4cpus+5cpus | |
| 19:28:01 | sean-k-mooney | instead of going to 3 numa nodes with 3 cpus | |
| 19:28:14 | efried | right | |
| 19:28:36 | sean-k-mooney | im not sure which is more likely to cause fragmentation of the top of my head | |
| 19:28:50 | sean-k-mooney | we should tell people to just use powers of 2 | |
| 19:28:51 | efried | example I gave above was with 10 VCPUs, we would try 10, then 5/5, then 3/3/4, then 2/3/2/3. | |
| 19:28:55 | efried | no | |
| 19:29:01 | sean-k-mooney | i was jokeing | |
| 19:29:10 | efried | we should tell people to use real numa specs if they care. | |
| 19:29:19 | sean-k-mooney | it does make life easier when they do but ya i can see that working | |
| 19:30:51 | sean-k-mooney | ok if we caluate the split in the tempory flavor we pass to the numa constratis function then it would populate the instance numa toplogy object as if the user had set it manuyaly in teh falvor | |
| 19:31:07 | sean-k-mooney | then if we save that in the instnace we could ensure we dont break live migration by changing it | |
| 19:31:08 | efried | just so. | |
| 19:31:32 | efried | well, I would expect we shouldn't save it in the flavor, because we want the instance to be able to morph to fit somewhere else if it needs to. | |
| 19:31:46 | sean-k-mooney | right | |
| 19:31:57 | efried | but I don't know how that works, do you break an instance if you "change" its topo from under it? | |
| 19:31:58 | sean-k-mooney | i ment save the instance_numa_toplogy object | |
| 19:32:01 | sean-k-mooney | not the falavor | |
| 19:32:21 | sean-k-mooney | efried: during live migration you would | |
| 19:32:38 | sean-k-mooney | cold migration it might mess up some manula config but it should not break it in general | |
| 19:32:58 | efried | hm, well that's a bummer. So how do we migrate numa-agnostic instances today? | |
| 19:32:58 | sean-k-mooney | i was suggesting once we select a toplogy we store it in the request_spec and instnace | |
| 19:33:11 | sean-k-mooney | so that it stays the same for the liftime of the instance unless you resize | |
| 19:33:14 | sean-k-mooney | or rebuild | |
| 19:33:34 | sean-k-mooney | am today | |
| 19:33:44 | sean-k-mooney | non numa isntace are alwasy exposed as 1 numa node | |
| 19:33:54 | sean-k-mooney | so it never chagnes form the gest point of view | |
| 19:34:02 | sean-k-mooney | so we just lie to the guest | |
| 19:34:10 | efried | couldn't we continue lying to the guest? | |
| 19:34:21 | sean-k-mooney | we can yes | |
| 19:34:39 | efried | does the guest do things differently if it knows CPU x is affined to memory y? | |
| 19:34:51 | sean-k-mooney | so we would do the progessive spliting to select the host resouces and present it as 1 numa node to the guest | |
| 19:35:04 | sean-k-mooney | but that will have worse performace then telling it its actull toplogy | |
| 19:35:10 | sean-k-mooney | yes | |
| 19:35:22 | sean-k-mooney | the kernel will take that into account wehn allocation memroy for a proces | |
| 19:35:36 | sean-k-mooney | trying to use numa local memory ahead of remote numa memory | |
| 19:35:44 | spatel | sean-k-mooney: hey | |
| 19:35:50 | efried | well, I guess this is a problem we would have eventually anyway, right dansmith? | |
| 19:35:52 | sean-k-mooney | if we dotn expose the toplogy to the vm the vm kernel wont know how to optimise | |
| 19:36:01 | spatel | did you see my last mesg? | |
| 19:36:43 | dansmith | efried: what? needing everything to support guests with numa? | |
| 19:37:01 | spatel | running vm on single numa0 give very high performance compare to running on both numa node (with cpu_socket=2 and cpu_threads=2 option) | |
| 19:37:32 | efried | dansmith: TL;DR: In the world where you say you don't care about NUMA, we give you an instance that's NUMA-ified, but with whatever split we were able to fit. | |
| 19:37:32 | efried | If you migrate that instance, you have to preserve that topo on the target; you can't just munge it to a new shape. | |
| 19:37:32 | efried | Which will then limit where you can fit it on migration. | |
| 19:37:36 | sean-k-mooney | spatel: i was away but just saw it now. fundemtally i think erlang is jut not optimising correctly. im not really sure how to help other then suggestin run more small vms if you can scale out the application horrizontally instead of vertically | |
| 19:37:44 | dansmith | sean-k-mooney: presumably if the flavor on the instance is not numa-aware then we can just find a new host during cold migration and give it a new topology when it moves | |
| 19:37:59 | sean-k-mooney | dansmith: yes we could | |
| 19:38:00 | dansmith | efried: only on cold migration | |
| 19:38:02 | dansmith | sorry | |
| 19:38:05 | dansmith | efried: only on live migration | |
| 19:38:28 | dansmith | efried: on cold migration it could change because you're rebooting and there shouldn't be anything in the guest that persistently cares what the topology is.. | |
| 19:38:29 | efried | Okay, that would be a new limitation. | |
| 19:38:32 | spatel | sean-k-mooney: totally understand but i am going to loose 16 cpu in that case. but anyway i can live with that | |
| 19:38:45 | dansmith | efried: we have that limitation today and it's handled by the numa live migration stuff, AFAIK | |
| 19:39:34 | efried | dansmith: oh, my understanding was that, by lying to the guest and saying it only has one NUMA node, regardless of how many are under the covers, we can change the under-the-covers on live migration without "affecting" the guest. It would still have crappy performance on both sides, but it wouldn't notice that anything had changed. | |
| 19:39:38 | sean-k-mooney | efried: well the limitation would be dont change the view of hardwar the vm sees in live migration | |
| 19:39:46 | dansmith | efried: and selecting a host in that situation should be the same as selecting a host for a new instance boot where the flavor cares deeply about the topology | |
| 19:39:47 | sean-k-mooney | when framed that way its what we do today | |
| 19:40:20 | dansmith | efried: only insofar as it is unaware of how stupid it's being, regardless of what is underneath | |
| 19:40:30 | sean-k-mooney | if we lye to the vm so it only sees 1 numa node we can in some cases change the mapping underneth yes | |
| 19:41:03 | dansmith | efried: so yes, if all guests are numa-aware then migrating indifferent ones becomes a little more restrictive, but that's the same goal as not pretending this stuff doesn't exist at boot time, IMHO | |
| 19:41:04 | efried | dansmith: okay, I understand and agree with that; I'm just questioning whether that's going to effectively spike the chances of NVH trying to live migrate a NUMA-agnostic instance. | |
| 19:41:42 | efried | sounds like the answer is yes, and we're okay with that. | |
| 19:41:46 | dansmith | efried: it may increase the difficulty of moving things, yes | |
| 19:42:13 | sean-k-mooney | efried: well in a non full clould it very likely that the numa=1 case will just work | |
| 19:42:18 | sean-k-mooney | unless the vm is very large | |
| 19:42:23 | dansmith | efried: I refer to the documented goal of not trying to schedule the last byte of memory | |
| 19:42:32 | dansmith | yep | |
| 19:42:36 | sean-k-mooney | and in that case the numa=2 case is likely to work | |
| 19:42:47 | sean-k-mooney | so i dont think it would spike much | |
| 19:43:03 | dansmith | small guests are less likely to care about numa, and thus more likely to fit into numa=1, and thus more likely to be easily movable | |
| 19:43:23 | dansmith | large guests are more likely to need numa for proper performance and have the moving restriction today | |
| 19:44:34 | sean-k-mooney | efried: dansmith im wondering if we could/should have a second spec or defer this to the implematnion | |
| 19:44:53 | efried | This spec needs to say whether we're going to try to do the progressive splitting thing. | |
| 19:45:03 | sean-k-mooney | e.g. if we agree withthe proposed placmenet modeling should we decied how to do the query splitting seperatly | |
| 19:45:11 | efried | But that's not the only factor in play. We also need to address the partially-upgraded cloud. | |
| 19:45:13 | sean-k-mooney | vvs fallbackj | |
| 19:45:50 | sean-k-mooney | so with the progress spilting i think we still need the fallback | |
| 19:45:54 | sean-k-mooney | to cover that case | |
| 19:46:13 | sean-k-mooney | the fall back is the only thing that can ever land on the non upgraded hosts | |
| 19:46:36 | efried | I'm not sure we should do the fallback thing at all. | |
| 19:46:48 | efried | Not if there's a way we can simply avoid doing the translation until all computes are upgraded. | |
| 19:47:03 | sean-k-mooney | we cant without a global config | |
| 19:47:11 | sean-k-mooney | which is why we have the fallback for pcpus | |
| 19:47:26 | efried | I thought the control plane was able to tell which computes were at what level? | |