Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-27
15:39:29 TheJulia who is making a room?
15:39:42 leakypipes TheJulia, jgwentworth: https://hangouts.google.com/call/v-fWz1j1qrJCQ4mjKR-cAAEE
15:43:15 openstackgerrit Lance Bragstad proposed openstack/nova master: Simplify logic in get_enforcer https://review.openstack.org/531008
15:54:42 openstackgerrit Lance Bragstad proposed openstack/nova master: Deprecate rule:context_is_admin https://review.openstack.org/564349
16:01:57 mriedem finucannot: off the top of your head,
16:02:31 mriedem does hw:cpu_policy=shared just mean vcpus can float across pcpus in any number? like, do we have a way to say that the vcpus must be evenly distributed?
16:03:00 finucannot what do you mean, in any number?
16:03:13 mriedem i think i might be looking for
16:03:14 mriedem FLAVOR-CORES: (comma-separated list of integers) A list of instance vCPUs to map to instance NUMA node N. If not specified, vCPUs are evenly divided among available NUMA nodes.
16:03:17 mriedem from https://docs.openstack.org/nova/latest/user/flavors.html
16:03:51 finucannot Unless you're specifying guest NUMA topologies, I don't think that's what you're looking for
16:04:01 finucannot *requesting
16:04:27 mriedem Kevin_Zheng: around?
16:04:35 finucannot That lets you say "I want 2 cores to go to _guest_ NUMA node 0 and 6 cores to _guest_ NUMA node 1"
16:04:56 finucannot (as an example)
16:05:03 Kevin_Zheng Yeah
16:05:09 mriedem i was thinking if i have a flavor with 8 vcpu and i want those divided evently across numa node 0 and 1
16:05:16 mriedem *evenly
16:05:38 finucannot mriedem: hw:numa_nodes=2
16:05:51 finucannot nova won't split guest NUMA nodes across host NUMA nodes
16:06:23 finucannot (nor will it allow two guest NUMA nodes be placed on the same host NUMA node. I think that's a mistake, but that's neither here nor there)
16:06:45 sean-k-mooney finucannot: that is a limitation of the libvirt dirver
16:07:26 finucannot sean-k-mooney: What is? Splitting guest NUMA nodes or placing two guest NUMA nodes on the same host node?
16:07:41 sean-k-mooney finucannot: the api allowas multiple guest numa nodes to be on the same host numa node but the recent placement discustion regarding resource groups changes that
16:07:50 Kevin_Zheng If I specify numa-nodes=2 and cpu_policy=strict do I got some CPI
16:08:07 sean-k-mooney finucannot: having a guest numa node span host numa nodes would violate that api
16:08:24 Kevin_Zheng Some vcpu on host numa0 and others in host numa1?
16:08:28 sean-k-mooney finucannot: having two guest numanodes mapped to a single host numa node would not
16:08:35 finucannot sean-k-mooney: to what API do you refer?
16:08:56 finucannot Kevin_Zheng: I assume you mean hw:cpu_policy=dedicated
16:08:57 sean-k-mooney the meaning of the flavor extra specs
16:09:08 Kevin_Zheng Ah yes
16:09:31 finucannot In which case, yes, the 1/N guest cores go on each host NUMA node, where N is the number of guest NUMA nodes
16:09:58 Kevin_Zheng OK THANKS that’s what I’m asking
16:10:07 sean-k-mooney finucannot: yes that is the default bevahoir if hw:numa_nodes=n
16:10:42 finucannot sean-k-mooney: Yeah, I'm in agreement with you there. There's no reason we shouldn't be able to squash 2 or more guest NUMA nodes on the same host NUMA node
16:11:06 finucannot So long as we don't split a guest NUMA node across two or more host nodes
16:11:24 Kevin_Zheng one more thing, it is available in mitaka? Correct?
16:11:28 finucannot sean-k-mooney: bauzas, fried_rice, sahid and I discussed that a while back
16:11:35 mriedem Kevin_Zheng: yes hw:cpu_policy is in mitaka
16:11:38 sean-k-mooney finucannot: the current behavior was due to a bug in the icehose implementation where we acidetally use the virtual numa node id as the host numa node it was mapped too
16:11:41 mriedem Kevin_Zheng: using NUMATopologyFilter
16:11:51 finucannot Kevin_Zheng: cpu_policy is. cpu_thread_policy is not, iirc
16:12:05 mriedem that is also
16:12:28 finucannot Oh, wow, I've been hanging around here longer than I thought
16:12:38 sahid finucannot: i explained to you the reason why we should not
16:12:51 mriedem finucannot: i think a lot of this stuff has been around since juno
16:12:58 mriedem juno was the big nfv push
16:13:26 sean-k-mooney finucannot: cpu_policy is dedicate/shared cpu_thread_policy is isolate/perfer... neither have anything to do with numa
16:13:31 mriedem https://review.openstack.org/#/c/140408/ kilo
16:13:46 finucannot sahid: IIRC, we agreed to park the discussion because we'd reached an impasse :)
16:14:11 sean-k-mooney mriedem: yes the inital patches were for icehouse but it landed in juno after we set up the intel nfv ci to test it
16:14:35 sahid we were not in a impasse, the only way for an user to ensure that he can pin it's application on two differents cpu is to use guest numa topology
16:15:01 openstackgerrit Merged openstack/nova master: Remove explicit instance.info_cache.delete() https://review.openstack.org/563689
16:15:17 finucannot mriedem: Aye, and https://review.openstack.org/#/c/202655/ is Mitaka
16:15:20 mriedem https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/virt-driver-cpu-thread-pinning.html
16:15:21 mriedem yeah
16:16:25 finucannot sahid: Two different physical CPUs?
16:16:28 sean-k-mooney mriedem: there were patch before either of those specs for icehose and juno
16:17:48 finucannot sahid: Even then, I still don't see why you'd care about that. The only reason to place them on separate nodes is because there are resources (PCI, GPU, vSwitches) associated with those nodes that we want some affinity to
16:18:52 sean-k-mooney sahid: you can use cpu_policy=dedicated + cpu_thread_policy=isolate to gurarteee that
16:18:53 sahid finucannot: no only, some use cases like realtime or zero drop packets want to have best effort vcpu running on different numa nodes
16:19:50 sahid sean-k-mooney: what you say does not exactly what i mean
16:20:00 sahid you can be on the same socket
16:20:19 sean-k-mooney you stated you wanted different cpus. not different sockets
16:20:32 sean-k-mooney and 2 numa nodes will not guartee different sockets
16:20:49 leakypipes jroll, TheJulia: k, reviewed.
16:21:07 jroll thanks leakypipes
16:21:21 finucannot sean-k-mooney: I assume you're referring to Cluster on Die?
16:21:27 sean-k-mooney finucannot: yep
16:22:58 sean-k-mooney finucannot: we covered this in quete a lot of detail in leakypipes cpu spec actully.
16:23:29 sean-k-mooney finucannot: also here https://etherpad.openstack.org/p/cpu-resource-accounting
16:24:24 finucannot sahid: I'm not sure what using cores from a different NUMA socket would get us that can't be achieved by disabling SMT/HyperThreading
16:25:06 sean-k-mooney finucannot: well isolates has the same effect as disabling HT
16:25:17 finucannot If we do that, the only resources the cores will be sharing is cache and a couple of buses
16:25:43 sean-k-mooney finucannot: when you say cache you mean l3 cache
16:26:05 finucannot and unless you're planning on dedicating every core on the NUMA node to the guest, you're still at risk of other guests on that node screwing with said cache
16:26:12 finucannot sean-k-mooney: yes on both counts
16:26:59 sean-k-mooney l1 cache is per HT l2 cache is shared between HT on same physical core and l3 cache is shared across socket(with cluster on die ist across socket but 1 regoin per numa node)
16:27:30 sean-k-mooney * /across socket/within socket/
16:28:03 finucannot Yup, that was my understanding of things too
16:28:46 finucannot lstopo is a nifty tool to that end
16:28:56 finucannot Aaaaaanyway, it's Friday evening and thisaguy is going to watch Thanos get his ass kicked (hopefully)
16:30:31 openstack bugzilla.redhat.com bug 1468217 in kernel-rt "KVM: paravirt raw_spinlock priority bump for housekeeping vcpus" [Unspecified,Closed: wontfix] - Assigned to mtosatti
16:30:31 sahid that is one example: https://bugzilla.redhat.com/show_bug.cgi?id=1468217, it's really easy to increase latency on realtime application. my thinking is we should not fake guest numa topology, our current behavior is good and we should keep it.
16:31:19 sahid but probably suer have workaround by using some other isolation so, i'm not going to argue more of that :)
16:35:06 sean-k-mooney sahid: our current behavior is ok but suboptimal in several cases. the main one being multi numa node guest do not create multi numa pcie root trees meaning we can effectly force cross numa traffic event though we have cores of the guess on the same numa node pci device.
16:35:41 sahid sean-k-mooney: what about to have an option?
16:38:18 sahid i'm just scary about to break some specific use cases
16:38:29 sahid if we try to be simple, why user would ask for 2 nodes?
16:38:40 sahid they can just ask for 1
16:38:51 sean-k-mooney sahid: the user does not the admin does
16:39:13 sahid yeah... it's the user who is going to select the flavor, no?
16:39:18 sean-k-mooney one reason was to allow guest with hugepages to have more cores then fit on a single socket
16:40:28 sahid sean-k-mooney: you do not reply for what they could ask for 2
16:40:38 sahid s/wht/why
16:41:10 sean-k-mooney that was an example of why they would ask for 2 numa nodes
16:41:37 sean-k-mooney if you ask for one numa nodes with 16 cpus and your only have 10 cpus per numa node that vm cannont boot
16:42:22 sean-k-mooney so when operators want to allow vms with more cores tehn will fit in one host numa node the set hw:numa_nodes=2+
16:44:52 sahid sean-k-mooney: no no... guest can have 16 vCPUs, 1 numa node - the vCPUs are going to be pinned on different host numa node

Earlier   Later