| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 20:01:18 | jaypipes | sean-k-mooney: if by several people you do accurately mean 2-3 people in the world, I'm willing to live with that. | |
| 20:01:35 | mriedem | doesn't stephenfin have a tattoo for that extra spec? | |
| 20:01:47 | sean-k-mooney | mriedem: i dont think so | |
| 20:01:52 | cfriesen | jaypipes: can we update the allocation? | |
| 20:02:03 | jaypipes | mriedem: no, but dansmith told me he just got an NFV tattoo. | |
| 20:02:04 | sean-k-mooney | mriedem: if he does that a life lesson in its self | |
| 20:02:07 | jaypipes | on his butt cheek. | |
| 20:02:28 | mriedem | oh i was thinking of https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/share-pci-between-numa-nodes.html | |
| 20:02:29 | dansmith | do not do no | |
| 20:02:46 | sean-k-mooney | cfriesen: we could but this would be the only place we update teh allocation from the compute node in the future | |
| 20:02:54 | sean-k-mooney | cfriesen: hence why we should not | |
| 20:03:06 | cfriesen | sean-k-mooney: I was thinking update it from the scheduler after picking the destination | |
| 20:03:48 | sean-k-mooney | cfriesen: but then again we need to down call to the compute node to know if we need to update it and then have to go to placement and make another allocation_candiates request | |
| 20:04:10 | cfriesen | sean-k-mooney: no, I'm pretty sure that the scheduler has information on number of ht siblings | |
| 20:04:29 | sean-k-mooney | cfriesen: in the host numa topology blob yes | |
| 20:05:08 | cfriesen | sean-k-mooney: I'm looking at the PUT /allocations/{consumer_uuid} operation, which seems to allow updating an existing allocation | |
| 20:05:14 | sean-k-mooney | but im not sure we still have the host_state object at the point we are selecting the node | |
| 20:05:23 | mlavalle | rybridges: what microversion of Placement you are using? | |
| 20:05:53 | cfriesen | sean-k-mooney: that's easy to solve | |
| 20:06:10 | sean-k-mooney | jaypipes: oh an when i say several people i mainly mean my management chain so meh | |
| 20:06:10 | jaypipes | cfriesen: no, we really can't update the allocation... | |
| 20:06:20 | cfriesen | jaypipes: what is that PUT operation then? | |
| 20:06:48 | jaypipes | cfriesen: are you referring to within the scheduler or somewhere else? | |
| 20:06:59 | cfriesen | https://developer.openstack.org/api-ref/placement/#update-allocations | |
| 20:07:49 | sean-k-mooney | jaypipes: yes he means update the claim form the schduler by checking the numa toplogy blob from the host state object for the selected host at the very end before we down call to the compute node | |
| 20:08:10 | jaypipes | cfriesen: that call overwrites the allocations that were made for an instance against one or more providers. | |
| 20:08:53 | sean-k-mooney | cfriesen: its really for things like interface or volumen attach in the future | |
| 20:09:23 | cfriesen | jaypipes: right, so we do the initial allocation, then if HT is enabled on the selecte host and the flavor has cpu_thread_policy=ISOLATE we update the PCPU resource allocation to use twice as many | |
| 20:09:24 | jaypipes | sean-k-mooney: well... not really. it exists to fully replace the set of allocations for a consumer. | |
| 20:09:59 | sean-k-mooney | jaypipes: yes but we do an atomic update using the generation ides so the caller is respocible for doing the merge | |
| 20:10:06 | jaypipes | cfriesen: that is technical debt I do not wish to continue carrying. | |
| 20:11:09 | jaypipes | cfriesen: I do not wish to get into the business of treating some special snowflake workloads differently -- with regards to the allocation behaviour -- from every other instance. | |
| 20:11:11 | mriedem | melwitt: small things for https://review.openstack.org/#/c/555092/ | |
| 20:11:18 | sean-k-mooney | cfriesen: technically you could proably do it with an out of tree weigher if you really wanted too. sort by host without ht and if there are none left increase allocation... | |
| 20:11:35 | cfriesen | fair enough. we have a decent number of clients with all-in-one single-node installs that will be unhappy if we tell them that HT is going to be an all-or-nothing kind of thing. | |
| 20:11:41 | melwitt | mriedem: thx | |
| 20:12:08 | jaypipes | cfriesen: all-in-one single-node installs of a "cloud". hmm... | |
| 20:12:22 | mriedem | sean-k-mooney: cfriesen: which of you are going to summarize the last 4 hours talking about this in the ML tonight? | |
| 20:12:26 | mriedem | per the PTG retrospective? | |
| 20:12:44 | cfriesen | https://etherpad.openstack.org/p/cpu-resource-accounting | |
| 20:12:46 | sean-k-mooney | mriedem: we have a partial summary here | |
| 20:13:40 | sean-k-mooney | :) we need to add the last case though e.g. deprecate cpu_thread_policy | |
| 20:13:40 | cfriesen | jaypipes: I suspect a number of distrib-cloud folks would be not thrilled with HT becoming a per-compute-node thing. | |
| 20:14:01 | jaypipes | cfriesen: technically, with the HW_CPU_HYPERTHREADING trait solution, one could feasibly set the trait against only one NUMA node provider on the single compute host and effectively carve out one NUMA node for HT-tolerant workloads and the other NUMA node for non HT-tolerant workloads... | |
| 20:14:13 | sean-k-mooney | cfriesen: well there is one other way around this | |
| 20:14:15 | cfriesen | yeah, that's true | |
| 20:14:23 | cfriesen | better than per compute node | |
| 20:14:29 | sean-k-mooney | you can exclude the HT form the new shared and dedicate pin sets | |
| 20:14:49 | mriedem | 555314 has been promoted to top of gate now btw | |
| 20:14:55 | cfriesen | sean-k-mooney: doesn't buy you anything because then nothing can run on the ones you excluded | |
| 20:15:32 | sean-k-mooney | cfriesen: it means you can have HT for you shared cores and no HT for your dedicated cores | |
| 20:16:19 | jaypipes | I still think the "apply the HW_CPU_HYPERTHREADING trait to one of the NUMA nodes" is a better, simpler solution. | |
| 20:16:52 | sean-k-mooney | jaypipes: oh had not read that when i typed that am yes that would work | |
| 20:17:05 | jaypipes | and have the flavors that cannot tolerate their pCPUs being on hyperthread siblings add a forbidden=HW_CPU_HYPERTHREADING to the request to placement. | |
| 20:17:06 | sean-k-mooney | jaypipes: you would not even have to do it at the numa level | |
| 20:17:15 | cfriesen | jaypipes: something else...looking at line 80 in the etherpad. if you have a multi-numa-node guest with some of the vcpus being "shared" and some being "dedicated", it actually matters which are which because it affects how many of each you need for the numa-specific RPs | |
| 20:17:30 | sean-k-mooney | you could have 2 RP of VCPU under the same numa node with different tratis | |
| 20:18:04 | jaypipes | cfriesen: sure, but doesn't the CONF.cpu_dedicated_set and CONF.cpu_shared_set pinning strings allow you to specify that properly? | |
| 20:18:09 | sean-k-mooney | cfriesen: the numa toplogy filter will fix that | |
| 20:18:24 | jaypipes | sean-k-mooney: keep it simple... | |
| 20:19:11 | sean-k-mooney | jaypipes: well i was assuming ... actully no i was going to say the operator created teh RPs but the virtdriver/compute agent does | |
| 20:19:25 | jaypipes | sean-k-mooney: right. | |
| 20:19:45 | cfriesen | jaypipes: no. suppose I ask for two numa nodes, with 3 "shared" and 1 "dedicated" in virtual numa node 1, and 1 "shared" and 3 "dedicated" in virtual numa node 2. | |
| 20:19:58 | cfriesen | jaypipes: each virtual numa node must map to a physical numa node | |
| 20:20:34 | sean-k-mooney | cfriesen: yes following so far. what part of that will the numa topology filter not validate | |
| 20:20:48 | jaypipes | cfriesen: no, that's not how things currently work, according to sahid and sean-k-mooney at least... multiple virtual NUMA nodes do *not* necessarily map to multiple *host* NUMA nodes. | |
| 20:21:11 | cfriesen | jaypipes: that's true, but each virtual numa node must not cross physical numa node boundaries | |
| 20:21:21 | jaypipes | cfriesen: ah, yes, for sure. | |
| 20:21:28 | sean-k-mooney | jaypipes: in the libvirt driver they do but its not required too. its a result of a icehose bug | |
| 20:21:34 | jaypipes | cfriesen: but, as sean-k-mooney says, the NUMATopologyFilter will catch that stuff. | |
| 20:21:50 | jaypipes | cfriesen: none of that will impact the *amount* of PCPU or VCPU resources that are requested. | |
| 20:22:15 | cfriesen | jaypipes: sure it does, because it affects how many of each I need to ask for from the RP that represents a single numa node | |
| 20:23:30 | sean-k-mooney | cfriesen: but jay is suggesting we dont ask placement for x pcpus in numa node 1 and y on node 2 we just ask for x+y pcpus | |
| 20:24:01 | jaypipes | sean-k-mooney: we *could* ask for 1 PCPU on one provider and 2 PCPU on another provider using granular request groups if we wanted. | |
| 20:24:15 | cfriesen | jaypipes: suppose I have vcpus 0-3 on virtual numa node 0, and vcpus 4-7 on virtual numa node 1. If I specify that I want vpus 0-3 to be shared, that's fundamentally a different request than if I say I want 0,4 to be shared | |
| 20:24:19 | openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | |
| 20:24:43 | sean-k-mooney | jaypipes: yes we could. and as a step 2 we might want to for the numa reasons but what your proposing today would not block us doing that in the future | |
| 20:24:58 | jaypipes | cfriesen: could we do granular request groups for that? i.e. resources1:VCPU=1, resources2:VCPU:1 | |
| 20:25:23 | jaypipes | cfriesen: would say "find my a provider tree that has providers that can serve 1 VCPU each | |
| 20:25:51 | cfriesen | is that how we're representing virtual numa nodes? | |
| 20:26:01 | sean-k-mooney | jaypipes: that kind of what i was thinking of whith the numa stuff on line 203 of https://etherpad.openstack.org/p/cpu-resource-accounting | |
| 20:26:10 | jaypipes | k | |
| 20:26:38 | jaypipes | sean-k-mooney: ack | |
| 20:26:41 | cfriesen | because we're going to need to be able to ask for "X PCPU, Y VCPU, Z memory pages of size A" all on one host NUMA node | |
| 20:27:12 | sean-k-mooney | cfriesen: that should be fine with granular requrest | |
| 20:27:22 | sean-k-mooney | i think | |
| 20:27:23 | jaypipes | cfriesen: we haven't considered memory pages yet, but yeah, that's what granular request groups are for. | |
| 20:27:55 | jaypipes | btw, I proposed standardizing memory pages as resource classes a year ago: https://review.openstack.org/#/c/442718/ | |
| 20:28:00 | sean-k-mooney | jaypipes: the mem_page_size can just be a trait | |
| 20:28:15 | jaypipes | please no. | |
| 20:28:18 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional regression test for bug 1746509 https://review.openstack.org/555092 | |
| 20:28:19 | openstack | bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] https://launchpad.net/bugs/1746509 - Assigned to melanie witt (melwitt) | |
| 20:28:19 | openstackgerrit | melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093 | |
| 20:28:26 | jaypipes | sean-k-mooney: are they consumable things? | |
| 20:28:35 | sean-k-mooney | jaypipes: the mempages are | |
| 20:28:45 | jaypipes | right, so they should be resource classes. | |
| 20:28:47 | cfriesen | okay. so I think the end-user would find it convenient to specify *which* guest CPUs are shared/dedicated. If we specify only "how many" then they have to discover it at boot time and have sufficiently flexible software to handle setting up dynamic affinity based on what they discovered. | |
| 20:29:00 | jaypipes | sean-k-mooney: but, meh, another day for that discussion on memory pages. | |
| 20:29:07 | sean-k-mooney | so we have a rp with an inventory of mempages with a 2MB trait as a child of the numa node | |
| 20:29:52 | jaypipes | cfriesen: no disagreement from me, but again.... not related to placement/resource accounting. | |