| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-04 | |||
| 15:18:16 | stephenfin | i.e. give me a host that can fit all N in 1 NUMA node, else give me one that fit them in 2 nodes, ... | |
| 15:18:33 | stephenfin | *all N instance cores | |
| 15:18:40 | efried | stephenfin: but if in 2 nodes, we can split evenly or asymmetrically... | |
| 15:18:42 | bauzas | D/ I state in the alternatives section that this whole plan sucks because we miss placement expressivity | |
| 15:18:49 | bauzas | WFY folks ? | |
| 15:19:34 | dansmith | efried: right, and we'd want some "minimum split is 70/30" type expression too.. I get that it's not something we can do today, and will be harder to land that across now two separate projects | |
| 15:19:51 | dansmith | I'm just saying, the user sees this as a rock and a hard place, with no real-life justification for it | |
| 15:20:29 | bauzas | ttyl later folks and will scroll back | |
| 15:20:30 | dansmith | I guess if you don't care about a specific topology, you want the same percentage of memory on each node as cpus | |
| 15:20:33 | gibi | efried: thanks the summary about group_policy, it works fo me | |
| 15:20:33 | sean-k-mooney | ok back | |
| 15:20:40 | stephenfin | efried: yeah, correct, otherwise you force people to use those awful 'hw:numa.cpu{N}=<cpumap>' extra specs | |
| 15:21:08 | stephenfin | dansmith: sure, though it's a bit of weird one, implementation wise | |
| 15:22:05 | stephenfin | because placement will give us e.g. a 70/30 split on cores, but we wouldn't really be reflecting this in the pinning of the instance to the host | |
| 15:22:33 | stephenfin | so we'll have to be careful not to do strict NUMA memory affinity in that case | |
| 15:22:38 | dansmith | yeah | |
| 15:23:04 | dansmith | presumably there's a middle ground between fully constrained and unconstrained, | |
| 15:23:20 | dansmith | where your vcpus are constrained to only run on cores that are on the numa node they represent, right? | |
| 15:23:40 | stephenfin | Correct. That's what happens when you turn on a NUMA topology without pinning at the moment | |
| 15:23:51 | dansmith | you balance between cores looser than pinning, but... right okay | |
| 15:24:01 | stephenfin | i.e. use 'hw:numa_nodes' or 'hw:mem_page_size' | |
| 15:24:22 | dansmith | that seems fine to me then | |
| 15:24:25 | stephenfin | We "pin" to the whole range of enabled cores from N NUMA nodes | |
| 15:24:55 | stephenfin | rn there's no way to say give me N $resource from adjacent/child providers, right? That's what the 'can_split' thing was supposed to do? | |
| 15:25:08 | dansmith | if placement were able to cough up a topology for 1-2 numa nodes (i.e. i don't care) and then I get vcpus loosely pinned to cores on the right numa node according to the memory split... | |
| 15:25:24 | stephenfin | yeah, ideal | |
| 15:25:35 | dansmith | aye | |
| 15:26:02 | stephenfin | I guess to retain the current behavior, you'd cough up a topology for *all* NUMA nodes on the host | |
| 15:26:27 | stephenfin | but we can't do that since it would break e.g. a 1 core instance on a 2 node host | |
| 15:26:41 | dansmith | that's why it has to be a range I think | |
| 15:26:50 | stephenfin | yup | |
| 15:27:28 | sean-k-mooney | you know we can totally allcoate memroy from multiple host numa nodes and expose it as one gues numa node with qemu right | |
| 15:27:33 | sean-k-mooney | same with cores | |
| 15:27:44 | dansmith | sure, but that's not helpful | |
| 15:27:47 | efried | isn't that what we do today for a don't-care-about-NUMA guest? | |
| 15:28:09 | efried | and is the exact flexibility we're talking about getting rid of? | |
| 15:28:21 | dansmith | nobody is asking to be lied to :) | |
| 15:28:44 | sean-k-mooney | dansmith: well im jsut saying we can always use the resources that correspond to the placmenet allcaotion and expose a different virtual numa toplogy if we were willing to not require the 1:1 mapping unless you said you cared about numa | |
| 15:28:54 | dansmith | efried: no, that's not flexibility | |
| 15:29:10 | dansmith | efried: nobody is asking for "show me one numa node even though that's not the truth" | |
| 15:29:11 | sean-k-mooney | efried: lie to it yes | |
| 15:29:19 | mriedem | ignorance is bliss | |
| 15:29:31 | efried | If I ask for a kosher sausage, I'm going to be upset if it's pork. | |
| 15:29:31 | efried | If I ask for a sausage, I'm going to be fine if the sausage is beef or pork. | |
| 15:29:33 | dansmith | sean-k-mooney: gotcha | |
| 15:29:56 | efried | It's not about being lied to. It's about not caring. | |
| 15:30:05 | dansmith | sigh | |
| 15:30:08 | efried | I'm not convinced that everyone cares. | |
| 15:30:34 | dansmith | if we lie to the guest, then the guest *os* is *going* to make bad decisions that don't represent what is actually being offered | |
| 15:30:38 | dansmith | nobody wants that, | |
| 15:30:48 | dansmith | they're opting into that over the more painful "care about this in extreme detail" | |
| 15:31:35 | sean-k-mooney | efried: if you add hw:numa_nodes=1 to a random flavor we normaly expepct about a 20-30% performance improvment | |
| 15:31:39 | efried | then isn't it the responsibility of the libvirt driver (not the scheduler) to take a generic simple request and make a real numa topo out of it? | |
| 15:31:51 | sean-k-mooney | even thoughg it is still floating over cores and using 4k small pages | |
| 15:32:09 | sean-k-mooney | just becuase the memroy and cpus all come form a single numa node | |
| 15:33:19 | efried | Now it's the job of the driver (via the overloaded NTF, presumably?) to carve those N VCPUs and M MEMORY_MBs out of whatever NUMA nodes they're available in, and create the appropriate topo for the guest, no matter how many nodes that happens to be? | |
| 15:33:19 | efried | iow if my host is configured monolithically in placement and my VM requests simply VCPU=N,MEMORY_MB=M, we'll place the VM even if (and without knowing) the resources have to be spread across NUMA nodes. | |
| 15:34:21 | dansmith | no? the virt driver doesn't have visibility into enough of the (nova) system to make those kinds of decisions I don't think | |
| 15:34:37 | sean-k-mooney | if we model numa in placment then the rps the allcoation come form force the dirver to allcoate the resouces form spefici host numa nodes | |
| 15:34:54 | efried | yes, but then we *must* frame the request accordingly. | |
| 15:35:01 | sean-k-mooney | yes | |
| 15:35:02 | efried | that's the whole problem we're trying to avoid | |
| 15:35:26 | sean-k-mooney | the only way to avoid that is to not repor tnuma in placment | |
| 15:35:27 | efried | because, once again, the VM didn't care about the specifics of the NUMA topology. By making it a real one, of whatever shape, we're still conferring the perf advantages to the VM. But we would do that at the host, having decided there are enough resources in total. | |
| 15:35:42 | dansmith | efried: the request is the important part here because we're talking about multiple computers.. the scheduler is looking for something that fits best amongst the options, not "well, we're on this host how do we best cram this into the hole we have" | |
| 15:36:09 | efried | right, I'm saying from the perspective of the scheduler, any number of fits can be considered "best". | |
| 15:36:27 | sean-k-mooney | right we use weighers to determin what best is | |
| 15:36:45 | sean-k-mooney | we have disused the idea of have a weigher based on the allcoation candiate in the past | |
| 15:36:47 | dansmith | which is why the scheduler doesn't pick actual resources, it picks hosts, and why before placement, we got that wrong a *lot* | |
| 15:36:53 | dansmith | which means we reschedule, which is super expensive | |
| 15:37:00 | sean-k-mooney | but that still does not change the fact that the placment query is the import thing to get right | |
| 15:37:06 | dansmith | sean-k-mooney: agreed | |
| 15:37:47 | sean-k-mooney | in the non numa case if we had a weigher and we had 1 allcoation candiate with 1 numa node and another with 2 we could chosse the singel numa node | |
| 15:38:00 | sean-k-mooney | but i dont know how to allow that today with the /ac api | |
| 15:38:38 | sean-k-mooney | that is kind fo what can_split was ment to solve but that is not a thing currnly | |
| 15:39:17 | sean-k-mooney | in the non numa case we would jsut lump everything in the un numberd group and say you can split the vcpus and ram | |
| 15:39:44 | sean-k-mooney | then weigh by the least number of numa nodes | |
| 15:40:00 | sean-k-mooney | but i dont see that happening anytime soon | |
| 15:45:07 | efried | Agreed. | |
| 15:45:16 | efried | So barring the ideal | |
| 15:45:28 | efried | we agreed on this 80/20 approach | |
| 15:45:44 | sean-k-mooney | with the partioning of the cloud | |
| 15:45:50 | efried | yes | |
| 15:46:42 | sean-k-mooney | ya so you know way way way back before numa and pinning was merged they was a counter propoal to make them host wide config options | |
| 15:46:50 | sean-k-mooney | we are slowly getting back to that | |
| 15:47:11 | efried | and almost never have to reschedule in either case. | |
| 15:47:11 | efried | and one complex strict placement query for NUMA | |
| 15:47:11 | efried | so that we can have one simple placement query for non-NUMA | |
| 15:47:39 | sean-k-mooney | yes although i think at some point we will want to have 1 code path | |
| 15:47:48 | efried | That's the 20 | |
| 15:48:10 | sean-k-mooney | yes but it could also be a refinment of scope | |
| 15:48:20 | efried | a refinement of scope for Ussuri? | |
| 15:48:37 | efried | or adding restrictions in future releases? | |
| 15:48:46 | sean-k-mooney | no. if we say all vms are numa vms in the future we reduce fucntionality as we did with cpu pinning | |
| 15:48:49 | sean-k-mooney | but make this all simpler | |
| 15:49:14 | sean-k-mooney | efried: to model PCPUs in placment we reduced the funcatiolity of the thread policies | |
| 15:49:30 | sean-k-mooney | we could in a future relase consider the same here | |
| 15:49:42 | stephenfin | ralonsoh: One more question regarding this comment -> "The same patch also introduced an error when retrieving the network ID. The network ID is stored in a key named 'floating_network_id'" | |
| 15:50:05 | sean-k-mooney | every time we try to do that we end up makeing no progress at all | |
| 15:50:20 | sean-k-mooney | so im saying defer to V+ and take the incremental improvment in U | |