| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 10:13:48 | bauzas | sudipto: so, yeah, this one is targeted to be implemented thanks to a nested-resource-provided BP | |
| 10:14:05 | bauzas | basically, we'd say that that compute UUID == RP UUID | |
| 10:14:05 | sudipto | bauzas, yeah i read through this: https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/nested-resource-providers.html | |
| 10:14:34 | bauzas | and it would be parent of nested resource providers giving details on NICs or CPUs | |
| 10:14:51 | bauzas | sudipto: exactly, so FWIW it will be a pike thing | |
| 10:15:06 | sudipto | However, i could visualise how it would work for a numa_node's cpus or numa_node's memory (which would be qutitiative) - but if there's a set of cpus for a numa node, how do we represent it was my question... | |
| 10:15:35 | sudipto | since we have defined a "total" field in the inventory table. | |
| 10:18:45 | johnthetubaguy | tdurakov: sorry, late getting onto IRC this morning, hows things? | |
| 10:20:02 | johnthetubaguy | bauzas: good point on updating the allocations, that change just fixes the bleeding | |
| 10:20:26 | bauzas | sudipto: FWIW, I don't think cpu sets are quantitative amounts | |
| 10:20:35 | johnthetubaguy | bauzas: I have a feeling if you restart nova-compute, we end up writing out updated allocations, but I am not 100% sure on that, the code scares me lots | |
| 10:20:43 | bauzas | johnthetubaguy: so I asked yesterday | |
| 10:20:59 | bauzas | johnthetubaguy: and I've been told that periodic updates rewrite the whole allocations | |
| 10:21:05 | bauzas | so we should be good | |
| 10:21:19 | johnthetubaguy | bauzas: hmm, honestly, that sounds terrible, but it fixes the bug | |
| 10:21:21 | bauzas | but I can look at code | |
| 10:21:44 | johnthetubaguy | bauzas: I want to do that for the ironic change anyways, I don't get how that works right now | |
| 10:22:32 | sudipto | bauzas, but i am wondering that has to be accounted for in case of pinning cpusets | |
| 10:24:17 | bauzas | sudipto: I think a CPU set a list of differnet RPs with an inventory of a VCPU resource class with amount of 1 | |
| 10:24:38 | bauzas | sudipto: and all those RPs are nested children of a parent RP which is compute node | |
| 10:25:01 | sudipto | bauzas, ok so basically we will have to create RPs for each individual cpuset then? | |
| 10:25:25 | bauzas | sudipto: that's what I'd think but I'm not the mastermind :) | |
| 10:25:35 | sudipto | and then map that to a parent RP of numa_node and root RP of the compute node | |
| 10:25:43 | sudipto | ok, that sounds like a lot of queries to me. | |
| 10:25:44 | johnthetubaguy | I think the resource providers end up modelling the NUMA node topology, as nested resource providers | |
| 10:25:58 | bauzas | johnthetubaguy: that's what I said to sudipto | |
| 10:26:12 | bauzas | johnthetubaguy: the related question was rather how we model that in a nested RP world | |
| 10:26:19 | sudipto | yeah | |
| 10:26:32 | johnthetubaguy | did that go in the spec, i remember it being nasty | |
| 10:26:44 | bauzas | but like I said, I just think each cpu pin is a RP with an inventory of 1 | |
| 10:26:51 | sudipto | johnthetubaguy, didn't read that in the spec. I may have missed if it was there. | |
| 10:27:00 | bauzas | a NUMA cell would be a parent RP | |
| 10:27:05 | bauzas | of that CPU pin | |
| 10:27:17 | johnthetubaguy | The NUMA node child providers will have inventory records populated for the NUMA_CORE, NUMA_THREAD and NUMA_MEMORY_MB resource classes. | |
| 10:27:19 | bauzas | and a compute RP would be the parent of all NUMA cell RPs | |
| 10:27:24 | johnthetubaguy | http://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/nested-resource-providers.html#proposed-change | |
| 10:27:35 | bauzas | johnthetubaguy: right, but that doesn't mention the CPU pins | |
| 10:27:40 | sudipto | yeah | |
| 10:27:52 | johnthetubaguy | ah, gotcha | |
| 10:28:09 | johnthetubaguy | thats a qualitative problem, which I guess is what bauzas said earlier | |
| 10:28:17 | bauzas | johnthetubaguy: not really | |
| 10:28:24 | bauzas | johnthetubaguy: you need to *consume* a pin | |
| 10:28:40 | sudipto | yeah consume the pin quantitatively. | |
| 10:28:51 | bauzas | johnthetubaguy: hence my thinking of having a RP being the CPU pin itself | |
| 10:29:04 | bauzas | with an inventory of VCPU with amount of 1 | |
| 10:29:18 | johnthetubaguy | so there are a few sorts of pins right now, which one is this? | |
| 10:29:33 | bauzas | and pinning the CPU would mean writing an allocation against that inventory | |
| 10:29:41 | johnthetubaguy | really you care about getting a dedicated pCPU on the "correct" numa node, I thought? | |
| 10:29:58 | johnthetubaguy | or is this something else | |
| 10:30:04 | bauzas | johnthetubaguy: I'm considering the CPU-pinning feature we implemented in Nova | |
| 10:30:18 | bauzas | there are many ways to schedule the pinning | |
| 10:30:23 | bauzas | but AFAICT, we fully pin | |
| 10:30:47 | sudipto | we could choose to pin it to a NUMA node or choose to just pin a cpuset (older nova behaviour) | |
| 10:31:18 | bauzas | whatever | |
| 10:31:21 | sudipto | My problem is with the representation of the inventory data for cpusets itself with the current inventory field called 'total' | |
| 10:31:51 | bauzas | and ? | |
| 10:31:55 | johnthetubaguy | so the current inventroy doesn't work right, hence the need for nested, and the new resources of NUMA_CORE, etc | |
| 10:32:07 | bauzas | if you consider a single pCPU being a RP | |
| 10:32:25 | sudipto | so we start considering every single pCPU as a RP and then do what bauzas suggested. | |
| 10:32:27 | bauzas | with an VCPU inventory attached to it | |
| 10:32:28 | johnthetubaguy | well, no, I think a NUMA_CORE is a colleciton of pCPU, and you consume one of them | |
| 10:32:47 | johnthetubaguy | but I may have this upside down | |
| 10:32:51 | bauzas | johnthetubaguy: you mean a NUMA cell | |
| 10:33:04 | johnthetubaguy | well, no | |
| 10:33:16 | bauzas | I'm confused then | |
| 10:33:21 | johnthetubaguy | a NUMA cell, reports as a provider of NUMA_CORE | |
| 10:33:27 | johnthetubaguy | contains lots of pCPUs you can claim | |
| 10:33:28 | johnthetubaguy | I think | |
| 10:33:31 | bauzas | righrt | |
| 10:33:42 | bauzas | that's just another indirection | |
| 10:33:56 | johnthetubaguy | the pCPUs are just the NUMA_CORE resources | |
| 10:33:57 | johnthetubaguy | I think | |
| 10:33:58 | bauzas | I just feel for CPU pinning that we need to have an inventory of 1 | |
| 10:34:12 | sudipto | ok | |
| 10:34:17 | bauzas | so you "take" that pCPU | |
| 10:34:21 | johnthetubaguy | you don't care about the particular pCPU though | |
| 10:34:31 | johnthetubaguy | you only care about the NUMA node its on relative to the others you have | |
| 10:34:32 | bauzas | that's a scheduling decision | |
| 10:34:40 | bauzas | the fact is | |
| 10:34:43 | johnthetubaguy | doesn't seem to need the ironic 1:1 treatment | |
| 10:34:52 | bauzas | the current scheduler doesn't choose the ping | |
| 10:34:53 | bauzas | pin | |
| 10:35:00 | bauzas | it leaves the compute picking one pin | |
| 10:35:13 | johnthetubaguy | sure, the new one can pick the NUMA node, doesn't have to pick the pCPU | |
| 10:35:21 | bauzas | it just verifies that the the cell has at least one pin left | |
| 10:35:23 | bauzas | but | |
| 10:35:34 | bauzas | we have a shit ton of corner cases in the current model | |
| 10:35:43 | bauzas | and libvirt is pretty laxist on that | |
| 10:36:12 | bauzas | I just discovered that libvirt is totally okay having two VMs asking the same full excluse access to a pCPU | |
| 10:36:22 | bauzas | exclusive* | |
| 10:36:24 | johnthetubaguy | yep, the key thing is what do you care about, you don't care its pin 7, you care its in the same NUMA_NODE as your other resources | |
| 10:36:39 | bauzas | yeah, but there are situations where we fail | |
| 10:36:57 | bauzas | I'm not a NUMA specialist but I swear having seen problems with that | |
| 10:37:06 | johnthetubaguy | we do today, because we don't lock on the NUMA cell, only on the compute node | |
| 10:37:11 | bauzas | mostly because we don't claim that in the scheduler | |
| 10:37:12 | johnthetubaguy | at least thats my understanding of the issue | |
| 10:37:24 | bauzas | right | |
| 10:37:31 | johnthetubaguy | hence the change above | |
| 10:37:37 | lyarwood | johnthetubaguy / bauzas ; apologies if it's too late to ask but would https://review.openstack.org/#/c/420146/ be something we can squeeze in ahead of the RC1 tag? ./me lost track of it this week with everything else flying around. | |
| 10:37:55 | bauzas | anyway, I think sorting that problem is not trival and could be discussed at PTH | |