Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-22
20:14:15 cfriesen yeah, that's true
20:14:23 cfriesen better than per compute node
20:14:29 sean-k-mooney you can exclude the HT form the new shared and dedicate pin sets
20:14:49 mriedem 555314 has been promoted to top of gate now btw
20:14:55 cfriesen sean-k-mooney: doesn't buy you anything because then nothing can run on the ones you excluded
20:15:32 sean-k-mooney cfriesen: it means you can have HT for you shared cores and no HT for your dedicated cores
20:16:19 jaypipes I still think the "apply the HW_CPU_HYPERTHREADING trait to one of the NUMA nodes" is a better, simpler solution.
20:16:52 sean-k-mooney jaypipes: oh had not read that when i typed that am yes that would work
20:17:05 jaypipes and have the flavors that cannot tolerate their pCPUs being on hyperthread siblings add a forbidden=HW_CPU_HYPERTHREADING to the request to placement.
20:17:06 sean-k-mooney jaypipes: you would not even have to do it at the numa level
20:17:15 cfriesen jaypipes: something else...looking at line 80 in the etherpad. if you have a multi-numa-node guest with some of the vcpus being "shared" and some being "dedicated", it actually matters which are which because it affects how many of each you need for the numa-specific RPs
20:17:30 sean-k-mooney you could have 2 RP of VCPU under the same numa node with different tratis
20:18:04 jaypipes cfriesen: sure, but doesn't the CONF.cpu_dedicated_set and CONF.cpu_shared_set pinning strings allow you to specify that properly?
20:18:09 sean-k-mooney cfriesen: the numa toplogy filter will fix that
20:18:24 jaypipes sean-k-mooney: keep it simple...
20:19:11 sean-k-mooney jaypipes: well i was assuming ... actully no i was going to say the operator created teh RPs but the virtdriver/compute agent does
20:19:25 jaypipes sean-k-mooney: right.
20:19:45 cfriesen jaypipes: no. suppose I ask for two numa nodes, with 3 "shared" and 1 "dedicated" in virtual numa node 1, and 1 "shared" and 3 "dedicated" in virtual numa node 2.
20:19:58 cfriesen jaypipes: each virtual numa node must map to a physical numa node
20:20:34 sean-k-mooney cfriesen: yes following so far. what part of that will the numa topology filter not validate
20:20:48 jaypipes cfriesen: no, that's not how things currently work, according to sahid and sean-k-mooney at least... multiple virtual NUMA nodes do *not* necessarily map to multiple *host* NUMA nodes.
20:21:11 cfriesen jaypipes: that's true, but each virtual numa node must not cross physical numa node boundaries
20:21:21 jaypipes cfriesen: ah, yes, for sure.
20:21:28 sean-k-mooney jaypipes: in the libvirt driver they do but its not required too. its a result of a icehose bug
20:21:34 jaypipes cfriesen: but, as sean-k-mooney says, the NUMATopologyFilter will catch that stuff.
20:21:50 jaypipes cfriesen: none of that will impact the *amount* of PCPU or VCPU resources that are requested.
20:22:15 cfriesen jaypipes: sure it does, because it affects how many of each I need to ask for from the RP that represents a single numa node
20:23:30 sean-k-mooney cfriesen: but jay is suggesting we dont ask placement for x pcpus in numa node 1 and y on node 2 we just ask for x+y pcpus
20:24:01 jaypipes sean-k-mooney: we *could* ask for 1 PCPU on one provider and 2 PCPU on another provider using granular request groups if we wanted.
20:24:15 cfriesen jaypipes: suppose I have vcpus 0-3 on virtual numa node 0, and vcpus 4-7 on virtual numa node 1. If I specify that I want vpus 0-3 to be shared, that's fundamentally a different request than if I say I want 0,4 to be shared
20:24:19 openstackgerrit Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762
20:24:43 sean-k-mooney jaypipes: yes we could. and as a step 2 we might want to for the numa reasons but what your proposing today would not block us doing that in the future
20:24:58 jaypipes cfriesen: could we do granular request groups for that? i.e. resources1:VCPU=1, resources2:VCPU:1
20:25:23 jaypipes cfriesen: would say "find my a provider tree that has providers that can serve 1 VCPU each
20:25:51 cfriesen is that how we're representing virtual numa nodes?
20:26:01 sean-k-mooney jaypipes: that kind of what i was thinking of whith the numa stuff on line 203 of https://etherpad.openstack.org/p/cpu-resource-accounting
20:26:10 jaypipes k
20:26:38 jaypipes sean-k-mooney: ack
20:26:41 cfriesen because we're going to need to be able to ask for "X PCPU, Y VCPU, Z memory pages of size A" all on one host NUMA node
20:27:12 sean-k-mooney cfriesen: that should be fine with granular requrest
20:27:22 sean-k-mooney i think
20:27:23 jaypipes cfriesen: we haven't considered memory pages yet, but yeah, that's what granular request groups are for.
20:27:55 jaypipes btw, I proposed standardizing memory pages as resource classes a year ago: https://review.openstack.org/#/c/442718/
20:28:00 sean-k-mooney jaypipes: the mem_page_size can just be a trait
20:28:15 jaypipes please no.
20:28:18 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1746509 https://review.openstack.org/555092
20:28:19 openstack bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] https://launchpad.net/bugs/1746509 - Assigned to melanie witt (melwitt)
20:28:19 openstackgerrit melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093
20:28:26 jaypipes sean-k-mooney: are they consumable things?
20:28:35 sean-k-mooney jaypipes: the mempages are
20:28:45 jaypipes right, so they should be resource classes.
20:28:47 cfriesen okay. so I think the end-user would find it convenient to specify *which* guest CPUs are shared/dedicated. If we specify only "how many" then they have to discover it at boot time and have sufficiently flexible software to handle setting up dynamic affinity based on what they discovered.
20:29:00 jaypipes sean-k-mooney: but, meh, another day for that discussion on memory pages.
20:29:07 sean-k-mooney so we have a rp with an inventory of mempages with a 2MB trait as a child of the numa node
20:29:52 jaypipes cfriesen: no disagreement from me, but again.... not related to placement/resource accounting.
20:30:01 sean-k-mooney jaypipes: oh i was agreeing the mempage should be a resouce class. i just dont know if we want mempage2MB and mempage1G
20:30:24 jaypipes sean-k-mooney: let's discuss the mempages stuff another day, eh? :)
20:30:34 sean-k-mooney jaypipes: sure
20:31:17 sean-k-mooney can we put this in https://etherpad.openstack.org/p/cpu-resource-accounting then copy it into the spec
20:31:17 cfriesen jaypipes: so if we allow the end user to specify something like "hw:shared_vcpus=0,1,4,5", it seems to me that nova should internally map that to the desired granular request rather than needing to specify it explicitly in the flavor extra spec.
20:32:20 cfriesen jaypipes: given this, it is implied that we want 4 VCPU resources, with the remainder being PCPU
20:33:34 cfriesen jaypipes: alternately, if we explicitly specify the VCPU and PCPU count, and make them discover the mapping at boot time, then we wouldn't need the "hw:shared_vcpus" extra spec
20:33:34 jaypipes cfriesen: lemme make sure I understand you...
20:34:58 jaypipes cfriesen: so you are saying that, just for VCPU and PCPU resource classes, that if nova sees the magic hw:shared_vcpus extra spec, that nova should figure out how many VCPU and how many PCPU it should ask for from placement instead of requiring the admin to put a "resources:VCPU=X" and "resources:PCPU=X" extra spec in the flavor?
20:35:33 sean-k-mooney cfriesen: so flavor.vcpus=8, hw:shared_vcpus=0,1,4,5 resouce[vcpu]=4 resouces[pcpu]=4 meaning all odd guest cpus are shared and all even cores(those not in that list) are dedicate cores.
20:36:14 cfriesen jaypipes: I think that would be the most convenient option for the end user
20:37:00 sean-k-mooney cfriesen: the hw:shared_vcpus=0,1,4,5 dose not change the placement request however right its just for the virt diriver/numa topology filter
20:37:19 jaypipes cfriesen: I prefer to have them discover the mapping at boot time and just specify VCPU and PCPU counts.
20:37:46 cfriesen jaypipes: that would be more generic, yes.
20:38:26 jaypipes cfriesen: that gives the virt driver the freedom to map the guest CPUs whatever way it needs, and once the virt driver makes that mapping decision, it could just write it to the instance metadata for the guest to read on boot.
20:38:58 cfriesen jaypipes: in that case I'd suggest that we make the lower-numbered vCPUs in a virtual numa node be "shared", and the higher-numbered ones "dedicated"
20:39:15 cfriesen but yeah, up to the virt driver
20:39:59 rybridges mlavalle: {"versions": [{"min_version": "1.0", "max_version": "1.4", "id": "v1.0"}
20:40:02 jaypipes cfriesen, sean-k-mooney: ok, I'm going to update my spec and attempt as best as possible to recap the above decisions.
20:40:10 cfriesen we'd have to persist the mapping somewhere to preserve it over live migration. and it'd be nice to keep it over cold migration/evacuate too
20:40:24 jaypipes cfriesen: instance metadata...
20:40:45 jaypipes cfriesen: just like how we save device metadata/tags today, right?
20:41:00 cfriesen should work, I think
20:42:06 sean-k-mooney cfriesen: you say per numa node but you realsie libvirt does not map guest cores to virtual numa nodes right
20:42:53 sean-k-mooney actully you kind of can.
20:42:55 sean-k-mooney <numa>
20:42:57 sean-k-mooney <cell id='0' cpus='0-3' memory='512000' unit='KiB'/>
20:43:04 sean-k-mooney </numa>
20:43:38 sean-k-mooney cpus in the cell id is the guest logical cpu
20:45:48 sean-k-mooney jaypipes: the gotcha with the metadata preseting the mapping is that on live migration we would want to make sure the mapping did not change as the running workload would not likely call the metadata api again
20:46:16 cfriesen sean-k-mooney: yeah, that's doable
20:46:41 sean-k-mooney that said NFV + live migration does not mix no matter how much telcos want it to
20:47:00 cfriesen sean-k-mooney: it just means that NUMATopologyFilter needs to check for that case
20:47:19 jaypipes sean-k-mooney: we'd still want to update the instance metadata to reflect the pinning on the destination host, though, so when the workload rebooted, it was able to configure itself.
20:47:26 sean-k-mooney cfriesen: meaning the numa toplogy filter would have to query the metatdata service
20:47:55 cfriesen jaypipes: no, we'd need to keep the "which vcpus are dedicated" mapping the same over the live migration
20:47:59 sean-k-mooney cfriesen: maybe put that in its own filter.
20:48:22 cfriesen jaypipes: we don't need to store the virtual-to-physical mapping in the metadata
20:49:08 sean-k-mooney cfriesen: ok i should have left 2 hours ago. letse leave the edgecase to live migration of snowflake vnf to another spec?
20:49:24 cfriesen sean-k-mooney: lol...me too. got an issue to debug
20:49:55 sean-k-mooney cfriesen: it has no effect on placement just the numatoplogy+everything_else_super filter
20:50:26 sean-k-mooney jaypipes: are you ok writing this up in the spec?
20:50:31 cfriesen sean-k-mooney: correct
20:52:02 sean-k-mooney ok im off tomorow but if you ping me ill proably check irc at some point
20:57:01 cfriesen if we report the "which vcpus are dedicated" mapping via metadata, what happens if the virtual routing is such that the guest has no access to the metadata server? will the config drive get suitably updated?

Earlier   Later