Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-22
19:51:03 sean-k-mooney cfriesen: well you can have lavor.vcpu=4,resouce[vcpu]=8:traits:ht_count=require thread_policy=isolate and flavor.vcpu=4,resouce[vcpu]=4:traits:HT_count_2=forbid thread_policy=isolate
19:51:47 sean-k-mooney cfriesen: it should have been HT_count_2=require when resouce[vcpu]=8 not forbid originally
19:52:08 jaypipes sean-k-mooney: that just doesn't seem right to me.
19:52:10 cfriesen I really don't want to have multiple extra-spec keys that depend on the value of other extra-spec keys
19:52:22 jaypipes and certainly isn't very understandable to me.
19:52:28 cfriesen agreed, that's a mess. :)
19:53:13 mriedem we don't go through the scheduler when a host is forced, but conductor does the resource allocation 'claim'
19:53:22 mriedem for live migrate and evacuate
19:53:41 sean-k-mooney so we keep coming back to we deprecate and remove tread_policy=isolate or extend the allocation on the compute host which breaks the workflow
19:54:29 sean-k-mooney or we change isolate to mean "host with HT off"
19:55:35 cfriesen mriedem: for "cold migrate to this compute node" we need to be going through the scheduler to do cpu pinning, pci, etc.
19:57:29 mriedem cfriesen: we do go through the scheduler for cold migrate
19:57:34 mriedem there is no force for cold migrate
19:57:35 mriedem remember
19:57:37 mriedem -5
19:57:41 cfriesen jaypipes: is there an actual API spec somewhere for placement? https://docs.openstack.org/nova/latest/user/placement.html doesn't seem to document the HTTP calls.
19:57:56 mriedem cfriesen: https://developer.openstack.org/api-ref/placement/
19:58:17 mriedem https://docs.openstack.org/nova/latest/user/placement.html#rest-api goes to ^
19:58:19 cfriesen mriedem: we allow an optional host, which is not a "force" but is a suggested destination
19:58:28 mriedem cfriesen: yeah, and
19:58:29 mriedem ?
19:58:32 mriedem it goes through the scheduler filters
19:58:33 mriedem and placement
19:58:58 cfriesen mriedem: right, but for that placement call do we ask for all the possible allocation candidates or do we specify a particular host?
19:59:15 jaypipes sean-k-mooney: at this point, I'd much prefer a single trait called HW_CPU_HYPERTHREADING whose absence indicates that hyperthreads are not enabled on the host.
19:59:30 mriedem cfriesen: placement doesn't know about 'hosts'
19:59:41 jaypipes sean-k-mooney: and using the forbidden traits stuff to find compute hosts that don't have hyperthreading enabled.
19:59:42 mriedem cfriesen: we say, 'hey placement, here is my request, give me your shit'
19:59:51 mriedem and then we restrict to just that requested host for the filtering
20:00:00 jaypipes mriedem: and placement goes 'sorry, I don't speak jive'.
20:00:14 mriedem is that a 418?
20:00:19 jaypipes correct.
20:00:19 cfriesen mriedem: okay, so the call to placement still filters all the RPs, then the scheduler filters narrow it down to the requested host?
20:00:29 mriedem cfriesen: yes
20:00:31 jaypipes cfriesen: yes
20:00:41 sean-k-mooney jaypipes: ya im leaning that way too. just get rid of hw:cpu_thread_policy as technical debt but i know that will piss of several people
20:01:18 jaypipes sean-k-mooney: if by several people you do accurately mean 2-3 people in the world, I'm willing to live with that.
20:01:35 mriedem doesn't stephenfin have a tattoo for that extra spec?
20:01:47 sean-k-mooney mriedem: i dont think so
20:01:52 cfriesen jaypipes: can we update the allocation?
20:02:03 jaypipes mriedem: no, but dansmith told me he just got an NFV tattoo.
20:02:04 sean-k-mooney mriedem: if he does that a life lesson in its self
20:02:07 jaypipes on his butt cheek.
20:02:28 mriedem oh i was thinking of https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/share-pci-between-numa-nodes.html
20:02:29 dansmith do not do no
20:02:46 sean-k-mooney cfriesen: we could but this would be the only place we update teh allocation from the compute node in the future
20:02:54 sean-k-mooney cfriesen: hence why we should not
20:03:06 cfriesen sean-k-mooney: I was thinking update it from the scheduler after picking the destination
20:03:48 sean-k-mooney cfriesen: but then again we need to down call to the compute node to know if we need to update it and then have to go to placement and make another allocation_candiates request
20:04:10 cfriesen sean-k-mooney: no, I'm pretty sure that the scheduler has information on number of ht siblings
20:04:29 sean-k-mooney cfriesen: in the host numa topology blob yes
20:05:08 cfriesen sean-k-mooney: I'm looking at the PUT /allocations/{consumer_uuid} operation, which seems to allow updating an existing allocation
20:05:14 sean-k-mooney but im not sure we still have the host_state object at the point we are selecting the node
20:05:23 mlavalle rybridges: what microversion of Placement you are using?
20:05:53 cfriesen sean-k-mooney: that's easy to solve
20:06:10 jaypipes cfriesen: no, we really can't update the allocation...
20:06:10 sean-k-mooney jaypipes: oh an when i say several people i mainly mean my management chain so meh
20:06:20 cfriesen jaypipes: what is that PUT operation then?
20:06:48 jaypipes cfriesen: are you referring to within the scheduler or somewhere else?
20:06:59 cfriesen https://developer.openstack.org/api-ref/placement/#update-allocations
20:07:49 sean-k-mooney jaypipes: yes he means update the claim form the schduler by checking the numa toplogy blob from the host state object for the selected host at the very end before we down call to the compute node
20:08:10 jaypipes cfriesen: that call overwrites the allocations that were made for an instance against one or more providers.
20:08:53 sean-k-mooney cfriesen: its really for things like interface or volumen attach in the future
20:09:23 cfriesen jaypipes: right, so we do the initial allocation, then if HT is enabled on the selecte host and the flavor has cpu_thread_policy=ISOLATE we update the PCPU resource allocation to use twice as many
20:09:24 jaypipes sean-k-mooney: well... not really. it exists to fully replace the set of allocations for a consumer.
20:09:59 sean-k-mooney jaypipes: yes but we do an atomic update using the generation ides so the caller is respocible for doing the merge
20:10:06 jaypipes cfriesen: that is technical debt I do not wish to continue carrying.
20:11:09 jaypipes cfriesen: I do not wish to get into the business of treating some special snowflake workloads differently -- with regards to the allocation behaviour -- from every other instance.
20:11:11 mriedem melwitt: small things for https://review.openstack.org/#/c/555092/
20:11:18 sean-k-mooney cfriesen: technically you could proably do it with an out of tree weigher if you really wanted too. sort by host without ht and if there are none left increase allocation...
20:11:35 cfriesen fair enough. we have a decent number of clients with all-in-one single-node installs that will be unhappy if we tell them that HT is going to be an all-or-nothing kind of thing.
20:11:41 melwitt mriedem: thx
20:12:08 jaypipes cfriesen: all-in-one single-node installs of a "cloud". hmm...
20:12:22 mriedem sean-k-mooney: cfriesen: which of you are going to summarize the last 4 hours talking about this in the ML tonight?
20:12:26 mriedem per the PTG retrospective?
20:12:44 cfriesen https://etherpad.openstack.org/p/cpu-resource-accounting
20:12:46 sean-k-mooney mriedem: we have a partial summary here
20:13:40 cfriesen jaypipes: I suspect a number of distrib-cloud folks would be not thrilled with HT becoming a per-compute-node thing.
20:13:40 sean-k-mooney :) we need to add the last case though e.g. deprecate cpu_thread_policy
20:14:01 jaypipes cfriesen: technically, with the HW_CPU_HYPERTHREADING trait solution, one could feasibly set the trait against only one NUMA node provider on the single compute host and effectively carve out one NUMA node for HT-tolerant workloads and the other NUMA node for non HT-tolerant workloads...
20:14:13 sean-k-mooney cfriesen: well there is one other way around this
20:14:15 cfriesen yeah, that's true
20:14:23 cfriesen better than per compute node
20:14:29 sean-k-mooney you can exclude the HT form the new shared and dedicate pin sets
20:14:49 mriedem 555314 has been promoted to top of gate now btw
20:14:55 cfriesen sean-k-mooney: doesn't buy you anything because then nothing can run on the ones you excluded
20:15:32 sean-k-mooney cfriesen: it means you can have HT for you shared cores and no HT for your dedicated cores
20:16:19 jaypipes I still think the "apply the HW_CPU_HYPERTHREADING trait to one of the NUMA nodes" is a better, simpler solution.
20:16:52 sean-k-mooney jaypipes: oh had not read that when i typed that am yes that would work
20:17:05 jaypipes and have the flavors that cannot tolerate their pCPUs being on hyperthread siblings add a forbidden=HW_CPU_HYPERTHREADING to the request to placement.
20:17:06 sean-k-mooney jaypipes: you would not even have to do it at the numa level
20:17:15 cfriesen jaypipes: something else...looking at line 80 in the etherpad. if you have a multi-numa-node guest with some of the vcpus being "shared" and some being "dedicated", it actually matters which are which because it affects how many of each you need for the numa-specific RPs
20:17:30 sean-k-mooney you could have 2 RP of VCPU under the same numa node with different tratis
20:18:04 jaypipes cfriesen: sure, but doesn't the CONF.cpu_dedicated_set and CONF.cpu_shared_set pinning strings allow you to specify that properly?
20:18:09 sean-k-mooney cfriesen: the numa toplogy filter will fix that
20:18:24 jaypipes sean-k-mooney: keep it simple...
20:19:11 sean-k-mooney jaypipes: well i was assuming ... actully no i was going to say the operator created teh RPs but the virtdriver/compute agent does
20:19:25 jaypipes sean-k-mooney: right.
20:19:45 cfriesen jaypipes: no. suppose I ask for two numa nodes, with 3 "shared" and 1 "dedicated" in virtual numa node 1, and 1 "shared" and 3 "dedicated" in virtual numa node 2.
20:19:58 cfriesen jaypipes: each virtual numa node must map to a physical numa node

Earlier   Later