Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-27
16:09:31 finucannot In which case, yes, the 1/N guest cores go on each host NUMA node, where N is the number of guest NUMA nodes
16:09:58 Kevin_Zheng OK THANKS that’s what I’m asking
16:10:07 sean-k-mooney finucannot: yes that is the default bevahoir if hw:numa_nodes=n
16:10:42 finucannot sean-k-mooney: Yeah, I'm in agreement with you there. There's no reason we shouldn't be able to squash 2 or more guest NUMA nodes on the same host NUMA node
16:11:06 finucannot So long as we don't split a guest NUMA node across two or more host nodes
16:11:24 Kevin_Zheng one more thing, it is available in mitaka? Correct?
16:11:28 finucannot sean-k-mooney: bauzas, fried_rice, sahid and I discussed that a while back
16:11:35 mriedem Kevin_Zheng: yes hw:cpu_policy is in mitaka
16:11:38 sean-k-mooney finucannot: the current behavior was due to a bug in the icehose implementation where we acidetally use the virtual numa node id as the host numa node it was mapped too
16:11:41 mriedem Kevin_Zheng: using NUMATopologyFilter
16:11:51 finucannot Kevin_Zheng: cpu_policy is. cpu_thread_policy is not, iirc
16:12:05 mriedem that is also
16:12:28 finucannot Oh, wow, I've been hanging around here longer than I thought
16:12:38 sahid finucannot: i explained to you the reason why we should not
16:12:51 mriedem finucannot: i think a lot of this stuff has been around since juno
16:12:58 mriedem juno was the big nfv push
16:13:26 sean-k-mooney finucannot: cpu_policy is dedicate/shared cpu_thread_policy is isolate/perfer... neither have anything to do with numa
16:13:31 mriedem https://review.openstack.org/#/c/140408/ kilo
16:13:46 finucannot sahid: IIRC, we agreed to park the discussion because we'd reached an impasse :)
16:14:11 sean-k-mooney mriedem: yes the inital patches were for icehouse but it landed in juno after we set up the intel nfv ci to test it
16:14:35 sahid we were not in a impasse, the only way for an user to ensure that he can pin it's application on two differents cpu is to use guest numa topology
16:15:01 openstackgerrit Merged openstack/nova master: Remove explicit instance.info_cache.delete() https://review.openstack.org/563689
16:15:17 finucannot mriedem: Aye, and https://review.openstack.org/#/c/202655/ is Mitaka
16:15:20 mriedem https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/virt-driver-cpu-thread-pinning.html
16:15:21 mriedem yeah
16:16:25 finucannot sahid: Two different physical CPUs?
16:16:28 sean-k-mooney mriedem: there were patch before either of those specs for icehose and juno
16:17:48 finucannot sahid: Even then, I still don't see why you'd care about that. The only reason to place them on separate nodes is because there are resources (PCI, GPU, vSwitches) associated with those nodes that we want some affinity to
16:18:52 sean-k-mooney sahid: you can use cpu_policy=dedicated + cpu_thread_policy=isolate to gurarteee that
16:18:53 sahid finucannot: no only, some use cases like realtime or zero drop packets want to have best effort vcpu running on different numa nodes
16:19:50 sahid sean-k-mooney: what you say does not exactly what i mean
16:20:00 sahid you can be on the same socket
16:20:19 sean-k-mooney you stated you wanted different cpus. not different sockets
16:20:32 sean-k-mooney and 2 numa nodes will not guartee different sockets
16:20:49 leakypipes jroll, TheJulia: k, reviewed.
16:21:07 jroll thanks leakypipes
16:21:21 finucannot sean-k-mooney: I assume you're referring to Cluster on Die?
16:21:27 sean-k-mooney finucannot: yep
16:22:58 sean-k-mooney finucannot: we covered this in quete a lot of detail in leakypipes cpu spec actully.
16:23:29 sean-k-mooney finucannot: also here https://etherpad.openstack.org/p/cpu-resource-accounting
16:24:24 finucannot sahid: I'm not sure what using cores from a different NUMA socket would get us that can't be achieved by disabling SMT/HyperThreading
16:25:06 sean-k-mooney finucannot: well isolates has the same effect as disabling HT
16:25:17 finucannot If we do that, the only resources the cores will be sharing is cache and a couple of buses
16:25:43 sean-k-mooney finucannot: when you say cache you mean l3 cache
16:26:05 finucannot and unless you're planning on dedicating every core on the NUMA node to the guest, you're still at risk of other guests on that node screwing with said cache
16:26:12 finucannot sean-k-mooney: yes on both counts
16:26:59 sean-k-mooney l1 cache is per HT l2 cache is shared between HT on same physical core and l3 cache is shared across socket(with cluster on die ist across socket but 1 regoin per numa node)
16:27:30 sean-k-mooney * /across socket/within socket/
16:28:03 finucannot Yup, that was my understanding of things too
16:28:46 finucannot lstopo is a nifty tool to that end
16:28:56 finucannot Aaaaaanyway, it's Friday evening and thisaguy is going to watch Thanos get his ass kicked (hopefully)
16:30:31 openstack bugzilla.redhat.com bug 1468217 in kernel-rt "KVM: paravirt raw_spinlock priority bump for housekeeping vcpus" [Unspecified,Closed: wontfix] - Assigned to mtosatti
16:30:31 sahid that is one example: https://bugzilla.redhat.com/show_bug.cgi?id=1468217, it's really easy to increase latency on realtime application. my thinking is we should not fake guest numa topology, our current behavior is good and we should keep it.
16:31:19 sahid but probably suer have workaround by using some other isolation so, i'm not going to argue more of that :)
16:35:06 sean-k-mooney sahid: our current behavior is ok but suboptimal in several cases. the main one being multi numa node guest do not create multi numa pcie root trees meaning we can effectly force cross numa traffic event though we have cores of the guess on the same numa node pci device.
16:35:41 sahid sean-k-mooney: what about to have an option?
16:38:18 sahid i'm just scary about to break some specific use cases
16:38:29 sahid if we try to be simple, why user would ask for 2 nodes?
16:38:40 sahid they can just ask for 1
16:38:51 sean-k-mooney sahid: the user does not the admin does
16:39:13 sahid yeah... it's the user who is going to select the flavor, no?
16:39:18 sean-k-mooney one reason was to allow guest with hugepages to have more cores then fit on a single socket
16:40:28 sahid sean-k-mooney: you do not reply for what they could ask for 2
16:40:38 sahid s/wht/why
16:41:10 sean-k-mooney that was an example of why they would ask for 2 numa nodes
16:41:37 sean-k-mooney if you ask for one numa nodes with 16 cpus and your only have 10 cpus per numa node that vm cannont boot
16:42:22 sean-k-mooney so when operators want to allow vms with more cores tehn will fit in one host numa node the set hw:numa_nodes=2+
16:44:52 sahid sean-k-mooney: no no... guest can have 16 vCPUs, 1 numa node - the vCPUs are going to be pinned on different host numa node
16:44:59 sean-k-mooney sahid: finucannot mriedem if you want to know the full simantics of the numa stuff this is the oldes spec i can still find that covers it http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/implemented/virt-driver-numa-placement.rst there were 2 before that that did not merge in icehose but
16:45:19 sahid they is no problem with that
16:45:21 sean-k-mooney sahid: no that is not allowed if hw:numa_nodes=1 is set
16:45:44 sean-k-mooney sahid: that is only allowed if hw:numa_nodes is not set at all
16:46:07 sean-k-mooney sahid: in whcich case the bevavior is virt driver defied
16:46:39 openstackgerrit Vladyslav Drok proposed openstack/nova master: Placement: allow to set reserved value equal to total for inventory https://review.openstack.org/564838
16:46:44 sahid yes, if you defined 1 it's not allowed, and that looks good, no?
16:46:59 sahid so basically you want to change that, right?
16:47:06 sean-k-mooney sahid: no
16:47:37 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Make sure microversion < 2.62 does not show host(Id) for instance actions https://review.openstack.org/564782
16:47:56 sean-k-mooney sahid: what we said is that the virt dirver today is allowed to have 2 virtual numa nodes mapped to the same host numa node
16:48:56 sean-k-mooney sahid: a virtual numa node is never allowed to span a phyical one.
16:51:12 sahid sean-k-mooney: so it means that for an user asking for 1 or 2 virtual numa node mean the same behavior?
16:52:04 sahid but he will be happy to see 2 NUMA nodes when doing "lscpu", right? :)
16:52:25 sean-k-mooney sahid: asking for hw:numa_node=x just means you will see x numa nodes within the guest and you are guarenteed that a numa node from the guest view point will not span host numa nodes,
16:53:07 sahid not today
16:53:32 sean-k-mooney sahid: today that is what the api of the hw:numa_nodes extra spec says
16:53:46 sahid the guest vCPUs are really running on different X different numa nodes
16:54:31 sahid well the virt driver as well
16:54:34 sean-k-mooney sahid: yes but that is a libvirt dirive implentation detail. hyperv support numa tology too and at least initally it did not have the same implmentation
16:56:04 sean-k-mooney sahid: this is the most importat section of the spec in this regard http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/implemented/virt-driver-numa-placement.rst#n94
16:56:59 sahid this section is just saying that, you can ask for a specific host numa node
16:57:11 sahid can't
16:58:14 sean-k-mooney that part of it but that is not the only intent of that section.
17:00:16 sean-k-mooney there is also this wiki on the topic https://wiki.openstack.org/wiki/VirtDriverGuestCPUMemoryPlacement
17:01:18 sahid ok sean-k-mooney i will try to discuss with some people iternally to ensure that is not going to break anything
17:01:53 sean-k-mooney do you have a specifc change you are proposing ot make to numa topology
17:01:56 sahid but my point is to do not let user think it's guest run 2 numa nodes where it's not
17:02:31 sean-k-mooney sahid: the guest has never had any guarteee that 2 virtual numa node ment 2 host numa nodes.
17:02:31 sahid sean-k-mooney: i'm not sure i understand?
17:02:57 sahid sean-k-mooney: please it's the current behavior and how the spec has been defined and implemented
17:03:57 sean-k-mooney sahid: i helped write that spec and that how it was impemented in the libvirt driver. anyway we are not currently proposing changing it but future virtdirver are not bound to do the same thing as libvirt

Earlier   Later