| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-23 | |||
| 13:54:46 | fried_rice | bauwser: Unless we start special-casing by resource class, no can do. Because of the DISK_GB example. | |
| 13:54:53 | bauwser | fried_rice: I'm talking of a user using a flavor that doesn't ask for NUMA resources, just vCPUs, yeah | |
| 13:55:07 | fried_rice | ...which would be an issue *today* (whereas NUMA is a tomorrow thing) | |
| 13:55:08 | sean-k-mooney[m] | bauwser: fried_rice this will be a behaviour change form today. Today we only numa affites if you ask for it | |
| 13:55:19 | bauwser | what sean-k-mooney[m] wrote | |
| 13:55:28 | bauwser | that would be a terrible upgrade impact | |
| 13:55:30 | mriedem | sahid: np, i left some comments in the 2nd patch in the code series | |
| 13:55:56 | bauwser | hence why I originally modeled my spec with vCPU resources being on the root RP | |
| 13:56:09 | bauwser | and only NUMA specific resources being on the children | |
| 13:56:16 | bauwser | to keep existing behaviour compatible | |
| 13:56:33 | fried_rice | bauwser: Okay, but then we would need some way to decrement the NUMA resources when the top-level resources are requested/consumed. | |
| 13:56:37 | sahid | mriedem: ok i will address them, i also have noticed your comment on the first patch | |
| 13:56:59 | bauwser | fried_rice: that's a separate NUMA resource class, see my spec | |
| 13:57:05 | bauwser | the terms are maybe not good | |
| 13:57:24 | leakypipes | sahid: np. thanks for your patience. | |
| 13:57:35 | leakypipes | sahid: I should have the cpu-resources next revision up shortly for your review. | |
| 13:57:50 | bauwser | but if I'm asking for resources:VCPU=2,NUMA_VCPU=2 then it'll decrement both | |
| 13:58:06 | sahid | leakypipes: ok, sure i will | |
| 13:58:13 | sahid | review it | |
| 13:58:17 | sean-k-mooney[m] | Well i think we need to have vcpus under numa but i think we will need something likek a grouping level attribute that defaults to host. E. G. Host, numa, rp grouping | |
| 13:58:21 | fried_rice | bauwser: yup. But how is that better than asking for resources1:VCPU:1&resources2=VCPU:1 | |
| 13:59:00 | bauwser | fried_rice: because existing flavors will still work for non-NUMA queries | |
| 13:59:03 | bauwser | that's all the difference | |
| 13:59:25 | sean-k-mooney[m] | Fried_rice how do you know how to create the groups | |
| 13:59:29 | bauwser | here, you're asking to change all the operator's flavor to explicitly tell whether they care about NUMA sharding or not | |
| 13:59:31 | fried_rice | bauwser: Existing flavors have resources:VCPU=2,NUMA_VCPU=2 ? | |
| 13:59:46 | bauwser | existing flavors just have VCPU resources | |
| 13:59:50 | fried_rice | right | |
| 14:00:11 | figleaf | fried_rice: how could you possibly know which NUMA nodes to decrement resources from? | |
| 14:00:15 | bauwser | like I wrote, we can have Nova to ask for both resource classes if it sees that | |
| 14:00:23 | fried_rice | So look, there are ways to handle this with and without changing the placement API | |
| 14:00:33 | fried_rice | yeah, that'd be one way. | |
| 14:00:47 | sean-k-mooney[m] | Existing flavours without numa_nodes set dont have numa affinity for cpus | |
| 14:00:53 | fried_rice | The other way is to have nova see that the flavor doesn't care about NUMA affinity and *nova* splits up the resource requests | |
| 14:00:56 | bauwser | I mean, if a flavor tells only about vCPUs, then query Placement only for that, but make allocations for both VCPU and NUMA_SOMETHING_VCPU | |
| 14:01:11 | bauwser | fried_rice: that's what I wrote in the spec | |
| 14:01:16 | bauwser | nova could make the translation | |
| 14:01:38 | bauwser | leakypipes: ^ I hope you follow | |
| 14:01:46 | fried_rice | bauwser: Cool; either way, we're having nova massage the request. But I prefer the path where we're actually representing true inventory numbers. | |
| 14:01:48 | sean-k-mooney[m] | Fried_rice how does nova know how to split it up | |
| 14:02:12 | fried_rice | sean-k-mooney[m]: For VCPU, it's one per request group, plain and simple. | |
| 14:02:13 | bauwser | sean-k-mooney[m]: there are two different things, the placement query and the allocation | |
| 14:02:39 | bauwser | fried_rice: also, note I'd like to not depend on request groups for NUMA | |
| 14:02:48 | fried_rice | bauwser: Whyzat? | |
| 14:03:09 | fried_rice | I don't think you can get around it, especially when you start talking about e.g. NICs. | |
| 14:03:12 | sean-k-mooney[m] | Flavor has 16 vcpus and dose not care about numa. All hosts have 2 10 cpus. How does nova know it needs to split into two groups | |
| 14:03:27 | bauwser | again, that requires operators opiniating about which specific placement they should do per flavor for non-NUMA specific instances | |
| 14:03:36 | fried_rice | sean-k-mooney[m]: it doesn't split into two groups. It splits into 16 | |
| 14:03:51 | bauwser | sean-k-mooney[m]: my vision was that we would ask placement only for VCPUs that are on the root RP | |
| 14:03:55 | fried_rice | Sorry guys, got to run for a bit. | |
| 14:04:07 | bauwser | sean-k-mooney[m]: but once we get a RP, we would make the allocation against a child | |
| 14:04:12 | bauwser | well, erm | |
| 14:04:16 | bauwser | nevermind | |
| 14:04:19 | bauwser | it's stupid | |
| 14:04:34 | sean-k-mooney[m] | So 16 request of 1 cpu | |
| 14:04:34 | bauwser | what we need is to get all the children that match the query | |
| 14:04:45 | bauwser | and only consume one of those | |
| 14:04:59 | bauwser | fuuu, I'm getting headaches | |
| 14:05:59 | sean-k-mooney[m] | leakypipes: is this something we should cover in your spec | |
| 14:06:19 | figleaf | bauwser: "only in root" - that was one of the decisions we made back in Atlanta. A machine would be configured by the operator ahead of time. If a VCPU was assigned to a NUMA node, it could not be considered also to be part of the root compute node | |
| 14:06:51 | bauwser | sean-k-mooney[m]: NUMA related things should be covered in https://review.openstack.org/#/c/552924/3/specs/rocky/approved/numa-topology-with-rps.rst | |
| 14:07:38 | bauwser | figleaf: I don't disagree with that statement | |
| 14:07:46 | sean-k-mooney[m] | figleaf: almost all server systems are numa system even systems with one socket | |
| 14:08:05 | bauwser | figleaf: I'm just trying to evaluate the operator impact of a NoValidHost if we begin counting NUMA nodes for non-NUMA specific instances | |
| 14:08:25 | figleaf | sean-k-mooney[m]: sure, but the choice on how to consume them (respecting NUMA, or ignoring) is an up-front decision by the operator. | |
| 14:08:38 | sean-k-mooney[m] | bauwser: i think it also needs to be in jays cpu resources spec | |
| 14:09:10 | bauwser | figleaf: so you agree with the fact that a classic flavor just asking for VCPUs shouldn't be accounting NUMA nodes, right? | |
| 14:09:28 | sean-k-mooney[m] | figleaf: no its a choice of the virt driver | |
| 14:09:36 | bauwser | how the inventory is modeled is one thing | |
| 14:09:53 | bauwser | and I'm fine with having VCPU resources on a NUMA node if that helps | |
| 14:09:57 | figleaf | bauwser: of course, but it would only select a host that was configured *not* to use NUMA. IOW, the ram, vcpu, etc, was inventory of the compute | |
| 14:10:15 | bauwser | but my point is, I don't want to get a NoValidHost if that restricts my existing flavor to specific NUMA nodes | |
| 14:10:51 | openstackgerrit | Merged openstack/nova master: Change compute mgr placement check to region_name https://review.openstack.org/554759 | |
| 14:10:54 | bauwser | figleaf: that's super costly then | |
| 14:11:06 | openstackgerrit | Merged openstack/nova master: Remove translate and a TODO https://review.openstack.org/554827 | |
| 14:11:07 | figleaf | bauwser: if you configure all your machines not to use NUMA, and request a NUMA-specific flavor, you will get NoValidHost. The reverse would also be true | |
| 14:11:20 | bauwser | figleaf: because I want to have both compute workloads that can do NUMA affinity and regular workloads that don't care | |
| 14:11:42 | bauwser | figleaf: again, that's a huuuuuge behavioural change | |
| 14:11:50 | figleaf | bauwser: sure, then configure some of your machines for NUMA, and others for not-NUMA | |
| 14:11:58 | figleaf | The balance is up to the operator | |
| 14:11:58 | bauwser | it's suboptimal | |
| 14:12:09 | sean-k-mooney[m] | figleaf: we don't have a compute node option for ignore numa | |
| 14:12:27 | cdent | making the optimal solution is introducing an enormous amount of complexity that is making hard to finish _any_ solution | |
| 14:12:28 | bauwser | sean-k-mooney[m]: I guess figleaf wants it | |
| 14:12:46 | sean-k-mooney[m] | Figleaf but that is not something you can configure normally | |
| 14:12:53 | bauwser | if you opt for non-NUMA, then you get a flat inventoty | |
| 14:13:14 | bauwser | if you opt for NUMA topology, then you're stuck with NUMA-only workloads on that single node | |
| 14:13:34 | bauwser | :/ | |
| 14:14:04 | figleaf | sean-k-mooney[m]: it would be how resources are reported. If the virt driver reports that a NUMA node owns some inventory, then that NUMA node has to satisfy the request. There is no concept in placement for the kind of sharing you're talking about | |
| 14:14:05 | bauwser | cdent: that's the whole purpose of the spec | |
| 14:14:28 | cdent | bauwser: which "that's"? | |
| 14:14:30 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 14:14:34 | bauwser | cdent: to see if we can model the NUMA inventories in any other ways that would help to solve the problem without requiring too much engineering | |
| 14:15:06 | leakypipes | cfriesen, bauwser, sahid, sean-k-mooney[m], finucannot, fried_rice: ^^ ok, new revision pushed. lemme know if that looks a bit better. | |
| 14:15:11 | sean-k-mooney[m] | figleaf: again how would yo do that. Numa is a feature of the hardware its not a software option so your talking about ordering hardware with out multiple memory controllers or configure mirrors dimms in the bios | |
| 14:15:18 | bauwser | leakypipes: you missed a NUMA conversation | |
| 14:15:31 | cdent | bauwser: okay. My statement was more along the lines of "trying to satisfy all this NUMA and other enhanced platform stuff, in general, is breaking the 80/20 rule" | |
| 14:15:55 | leakypipes | bauwser: not really. see the spec. :) | |
| 14:16:00 | bauwser | cdent: the problem is that like sean-k-mooney[m] said, modern architectures are NUMA-based, but 80% of our workloads don't care | |
| 14:16:13 | sean-k-mooney[m] | figleaf: not in placement but that is how nova works today. | |