| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 20:10:06 | jaypipes | cfriesen: that is technical debt I do not wish to continue carrying. | |
| 20:11:09 | jaypipes | cfriesen: I do not wish to get into the business of treating some special snowflake workloads differently -- with regards to the allocation behaviour -- from every other instance. | |
| 20:11:11 | mriedem | melwitt: small things for https://review.openstack.org/#/c/555092/ | |
| 20:11:18 | sean-k-mooney | cfriesen: technically you could proably do it with an out of tree weigher if you really wanted too. sort by host without ht and if there are none left increase allocation... | |
| 20:11:35 | cfriesen | fair enough. we have a decent number of clients with all-in-one single-node installs that will be unhappy if we tell them that HT is going to be an all-or-nothing kind of thing. | |
| 20:11:41 | melwitt | mriedem: thx | |
| 20:12:08 | jaypipes | cfriesen: all-in-one single-node installs of a "cloud". hmm... | |
| 20:12:22 | mriedem | sean-k-mooney: cfriesen: which of you are going to summarize the last 4 hours talking about this in the ML tonight? | |
| 20:12:26 | mriedem | per the PTG retrospective? | |
| 20:12:44 | cfriesen | https://etherpad.openstack.org/p/cpu-resource-accounting | |
| 20:12:46 | sean-k-mooney | mriedem: we have a partial summary here | |
| 20:13:40 | sean-k-mooney | :) we need to add the last case though e.g. deprecate cpu_thread_policy | |
| 20:13:40 | cfriesen | jaypipes: I suspect a number of distrib-cloud folks would be not thrilled with HT becoming a per-compute-node thing. | |
| 20:14:01 | jaypipes | cfriesen: technically, with the HW_CPU_HYPERTHREADING trait solution, one could feasibly set the trait against only one NUMA node provider on the single compute host and effectively carve out one NUMA node for HT-tolerant workloads and the other NUMA node for non HT-tolerant workloads... | |
| 20:14:13 | sean-k-mooney | cfriesen: well there is one other way around this | |
| 20:14:15 | cfriesen | yeah, that's true | |
| 20:14:23 | cfriesen | better than per compute node | |
| 20:14:29 | sean-k-mooney | you can exclude the HT form the new shared and dedicate pin sets | |
| 20:14:49 | mriedem | 555314 has been promoted to top of gate now btw | |
| 20:14:55 | cfriesen | sean-k-mooney: doesn't buy you anything because then nothing can run on the ones you excluded | |
| 20:15:32 | sean-k-mooney | cfriesen: it means you can have HT for you shared cores and no HT for your dedicated cores | |
| 20:16:19 | jaypipes | I still think the "apply the HW_CPU_HYPERTHREADING trait to one of the NUMA nodes" is a better, simpler solution. | |
| 20:16:52 | sean-k-mooney | jaypipes: oh had not read that when i typed that am yes that would work | |
| 20:17:05 | jaypipes | and have the flavors that cannot tolerate their pCPUs being on hyperthread siblings add a forbidden=HW_CPU_HYPERTHREADING to the request to placement. | |
| 20:17:06 | sean-k-mooney | jaypipes: you would not even have to do it at the numa level | |
| 20:17:15 | cfriesen | jaypipes: something else...looking at line 80 in the etherpad. if you have a multi-numa-node guest with some of the vcpus being "shared" and some being "dedicated", it actually matters which are which because it affects how many of each you need for the numa-specific RPs | |
| 20:17:30 | sean-k-mooney | you could have 2 RP of VCPU under the same numa node with different tratis | |
| 20:18:04 | jaypipes | cfriesen: sure, but doesn't the CONF.cpu_dedicated_set and CONF.cpu_shared_set pinning strings allow you to specify that properly? | |
| 20:18:09 | sean-k-mooney | cfriesen: the numa toplogy filter will fix that | |
| 20:18:24 | jaypipes | sean-k-mooney: keep it simple... | |
| 20:19:11 | sean-k-mooney | jaypipes: well i was assuming ... actully no i was going to say the operator created teh RPs but the virtdriver/compute agent does | |
| 20:19:25 | jaypipes | sean-k-mooney: right. | |
| 20:19:45 | cfriesen | jaypipes: no. suppose I ask for two numa nodes, with 3 "shared" and 1 "dedicated" in virtual numa node 1, and 1 "shared" and 3 "dedicated" in virtual numa node 2. | |
| 20:19:58 | cfriesen | jaypipes: each virtual numa node must map to a physical numa node | |
| 20:20:34 | sean-k-mooney | cfriesen: yes following so far. what part of that will the numa topology filter not validate | |
| 20:20:48 | jaypipes | cfriesen: no, that's not how things currently work, according to sahid and sean-k-mooney at least... multiple virtual NUMA nodes do *not* necessarily map to multiple *host* NUMA nodes. | |
| 20:21:11 | cfriesen | jaypipes: that's true, but each virtual numa node must not cross physical numa node boundaries | |
| 20:21:21 | jaypipes | cfriesen: ah, yes, for sure. | |
| 20:21:28 | sean-k-mooney | jaypipes: in the libvirt driver they do but its not required too. its a result of a icehose bug | |
| 20:21:34 | jaypipes | cfriesen: but, as sean-k-mooney says, the NUMATopologyFilter will catch that stuff. | |
| 20:21:50 | jaypipes | cfriesen: none of that will impact the *amount* of PCPU or VCPU resources that are requested. | |
| 20:22:15 | cfriesen | jaypipes: sure it does, because it affects how many of each I need to ask for from the RP that represents a single numa node | |
| 20:23:30 | sean-k-mooney | cfriesen: but jay is suggesting we dont ask placement for x pcpus in numa node 1 and y on node 2 we just ask for x+y pcpus | |
| 20:24:01 | jaypipes | sean-k-mooney: we *could* ask for 1 PCPU on one provider and 2 PCPU on another provider using granular request groups if we wanted. | |
| 20:24:15 | cfriesen | jaypipes: suppose I have vcpus 0-3 on virtual numa node 0, and vcpus 4-7 on virtual numa node 1. If I specify that I want vpus 0-3 to be shared, that's fundamentally a different request than if I say I want 0,4 to be shared | |
| 20:24:19 | openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | |
| 20:24:43 | sean-k-mooney | jaypipes: yes we could. and as a step 2 we might want to for the numa reasons but what your proposing today would not block us doing that in the future | |
| 20:24:58 | jaypipes | cfriesen: could we do granular request groups for that? i.e. resources1:VCPU=1, resources2:VCPU:1 | |
| 20:25:23 | jaypipes | cfriesen: would say "find my a provider tree that has providers that can serve 1 VCPU each | |
| 20:25:51 | cfriesen | is that how we're representing virtual numa nodes? | |
| 20:26:01 | sean-k-mooney | jaypipes: that kind of what i was thinking of whith the numa stuff on line 203 of https://etherpad.openstack.org/p/cpu-resource-accounting | |
| 20:26:10 | jaypipes | k | |
| 20:26:38 | jaypipes | sean-k-mooney: ack | |
| 20:26:41 | cfriesen | because we're going to need to be able to ask for "X PCPU, Y VCPU, Z memory pages of size A" all on one host NUMA node | |
| 20:27:12 | sean-k-mooney | cfriesen: that should be fine with granular requrest | |
| 20:27:22 | sean-k-mooney | i think | |
| 20:27:23 | jaypipes | cfriesen: we haven't considered memory pages yet, but yeah, that's what granular request groups are for. | |
| 20:27:55 | jaypipes | btw, I proposed standardizing memory pages as resource classes a year ago: https://review.openstack.org/#/c/442718/ | |
| 20:28:00 | sean-k-mooney | jaypipes: the mem_page_size can just be a trait | |
| 20:28:15 | jaypipes | please no. | |
| 20:28:18 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional regression test for bug 1746509 https://review.openstack.org/555092 | |
| 20:28:19 | openstack | bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] https://launchpad.net/bugs/1746509 - Assigned to melanie witt (melwitt) | |
| 20:28:19 | openstackgerrit | melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093 | |
| 20:28:26 | jaypipes | sean-k-mooney: are they consumable things? | |
| 20:28:35 | sean-k-mooney | jaypipes: the mempages are | |
| 20:28:45 | jaypipes | right, so they should be resource classes. | |
| 20:28:47 | cfriesen | okay. so I think the end-user would find it convenient to specify *which* guest CPUs are shared/dedicated. If we specify only "how many" then they have to discover it at boot time and have sufficiently flexible software to handle setting up dynamic affinity based on what they discovered. | |
| 20:29:00 | jaypipes | sean-k-mooney: but, meh, another day for that discussion on memory pages. | |
| 20:29:07 | sean-k-mooney | so we have a rp with an inventory of mempages with a 2MB trait as a child of the numa node | |
| 20:29:52 | jaypipes | cfriesen: no disagreement from me, but again.... not related to placement/resource accounting. | |
| 20:30:01 | sean-k-mooney | jaypipes: oh i was agreeing the mempage should be a resouce class. i just dont know if we want mempage2MB and mempage1G | |
| 20:30:24 | jaypipes | sean-k-mooney: let's discuss the mempages stuff another day, eh? :) | |
| 20:30:34 | sean-k-mooney | jaypipes: sure | |
| 20:31:17 | sean-k-mooney | can we put this in https://etherpad.openstack.org/p/cpu-resource-accounting then copy it into the spec | |
| 20:31:17 | cfriesen | jaypipes: so if we allow the end user to specify something like "hw:shared_vcpus=0,1,4,5", it seems to me that nova should internally map that to the desired granular request rather than needing to specify it explicitly in the flavor extra spec. | |
| 20:32:20 | cfriesen | jaypipes: given this, it is implied that we want 4 VCPU resources, with the remainder being PCPU | |
| 20:33:34 | cfriesen | jaypipes: alternately, if we explicitly specify the VCPU and PCPU count, and make them discover the mapping at boot time, then we wouldn't need the "hw:shared_vcpus" extra spec | |
| 20:33:34 | jaypipes | cfriesen: lemme make sure I understand you... | |
| 20:34:58 | jaypipes | cfriesen: so you are saying that, just for VCPU and PCPU resource classes, that if nova sees the magic hw:shared_vcpus extra spec, that nova should figure out how many VCPU and how many PCPU it should ask for from placement instead of requiring the admin to put a "resources:VCPU=X" and "resources:PCPU=X" extra spec in the flavor? | |
| 20:35:33 | sean-k-mooney | cfriesen: so flavor.vcpus=8, hw:shared_vcpus=0,1,4,5 resouce[vcpu]=4 resouces[pcpu]=4 meaning all odd guest cpus are shared and all even cores(those not in that list) are dedicate cores. | |
| 20:36:14 | cfriesen | jaypipes: I think that would be the most convenient option for the end user | |
| 20:37:00 | sean-k-mooney | cfriesen: the hw:shared_vcpus=0,1,4,5 dose not change the placement request however right its just for the virt diriver/numa topology filter | |
| 20:37:19 | jaypipes | cfriesen: I prefer to have them discover the mapping at boot time and just specify VCPU and PCPU counts. | |
| 20:37:46 | cfriesen | jaypipes: that would be more generic, yes. | |
| 20:38:26 | jaypipes | cfriesen: that gives the virt driver the freedom to map the guest CPUs whatever way it needs, and once the virt driver makes that mapping decision, it could just write it to the instance metadata for the guest to read on boot. | |
| 20:38:58 | cfriesen | jaypipes: in that case I'd suggest that we make the lower-numbered vCPUs in a virtual numa node be "shared", and the higher-numbered ones "dedicated" | |
| 20:39:15 | cfriesen | but yeah, up to the virt driver | |
| 20:39:59 | rybridges | mlavalle: {"versions": [{"min_version": "1.0", "max_version": "1.4", "id": "v1.0"} | |
| 20:40:02 | jaypipes | cfriesen, sean-k-mooney: ok, I'm going to update my spec and attempt as best as possible to recap the above decisions. | |
| 20:40:10 | cfriesen | we'd have to persist the mapping somewhere to preserve it over live migration. and it'd be nice to keep it over cold migration/evacuate too | |
| 20:40:24 | jaypipes | cfriesen: instance metadata... | |
| 20:40:45 | jaypipes | cfriesen: just like how we save device metadata/tags today, right? | |
| 20:41:00 | cfriesen | should work, I think | |
| 20:42:06 | sean-k-mooney | cfriesen: you say per numa node but you realsie libvirt does not map guest cores to virtual numa nodes right | |
| 20:42:53 | sean-k-mooney | actully you kind of can. | |
| 20:42:55 | sean-k-mooney | <numa> | |
| 20:42:57 | sean-k-mooney | <cell id='0' cpus='0-3' memory='512000' unit='KiB'/> | |
| 20:43:04 | sean-k-mooney | </numa> | |
| 20:43:38 | sean-k-mooney | cpus in the cell id is the guest logical cpu | |
| 20:45:48 | sean-k-mooney | jaypipes: the gotcha with the metadata preseting the mapping is that on live migration we would want to make sure the mapping did not change as the running workload would not likely call the metadata api again | |