| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 13:03:05 | bauzas | efried: that said, if we need to shard, what kind of query param are you thinking ? | |
| 13:03:33 | openstackgerrit | sahid proposed openstack/nova master: libvirt: move version to string in utils https://review.openstack.org/560455 | |
| 13:03:34 | openstackgerrit | sahid proposed openstack/nova master: libvirt: refactor get_base_config to accept host arg https://review.openstack.org/560456 | |
| 13:03:35 | openstackgerrit | sahid proposed openstack/nova master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/484997 | |
| 13:03:36 | bauzas | that's the problem with numbered request groups | |
| 13:03:41 | efried | bauzas: For explicitly separating request groups to separate resource providers? I was thinking something like ?separate_providers=resources1,resources2,... | |
| 13:03:55 | efried | ...if placement is where we want to handle it. | |
| 13:04:26 | efried | bauzas: But the other option is to handle it in the NUMATopologyFilter. Placement would give us back all the candidates, which would include the ones that are sharded and the ones that are combined. And the filter would just pick the ones that are appropriately sharded. | |
| 13:04:35 | bauzas | so, ?separate_providers=resources1,resources2&resources1:VCPU=1&resources2:VCPU=1 ? | |
| 13:04:44 | efried | bauzas: yes | |
| 13:05:14 | efried | Implementing that will be a bear, but yes. | |
| 13:05:20 | bauzas | if placement folks are accepting that, then I'd prefer to do that by Placement instead of the filter | |
| 13:05:34 | bauzas | because the less we have in the filter, the better it will be | |
| 13:05:42 | efried | I don't see it making Rocky, tbh | |
| 13:05:46 | bauzas | longer term of course | |
| 13:05:50 | bauzas | efried: yeah, I know | |
| 13:06:00 | bauzas | efried: what I'd love is some consensus on that spec for Rocky | |
| 13:06:03 | efried | But yes, eventually we recognize we're going to need that functionality in placement. | |
| 13:06:20 | efried | bauzas: which spec? | |
| 13:06:21 | bauzas | but then, earlyj working on Rocky-3 if I'm lucky so we can land Stein-1 | |
| 13:06:29 | bauzas | efried: the NUMA one | |
| 13:06:53 | efried | bauzas: Sure, agreed; but we need to crisp up what exactly you're trying to address, and how. | |
| 13:07:00 | efried | That's not yet clear IMO | |
| 13:07:19 | bauzas | efried: for Rocky, I'm only planning to implement nested RPs for vGPUs and fix the vGPU caveats | |
| 13:07:25 | bauzas | plus that spec | |
| 13:07:34 | bauzas | but implementing that spec for Rocky-3 or later | |
| 13:07:49 | bauzas | unless someone picks the ball | |
| 13:07:54 | efried | Okay, gotcha. | |
| 13:08:28 | efried | So you're not thinking to land the code related to the NUMA spec until "later". But you want the spec baked by Rocky-3 | |
| 13:10:03 | stephenfin | bauzas: The latter | |
| 13:10:32 | efried | johnthetubaguy_: You around? I approved https://review.openstack.org/#/c/553605/ on the basis that your -1 has now been addressed, but there's still time to pull it out if you disagree. | |
| 13:10:39 | stephenfin | bauzas: typically those guest nodes would be scheduled to different NUMA nodes, but if that's not possible they'll be squeezed onto the same one | |
| 13:11:00 | bauzas | efried: that's correct | |
| 13:11:00 | stephenfin | bauzas: At least that's the case to the best of my recollection | |
| 13:11:09 | efried | ahjeez. "anti-affinity preferred" is a use case we haven't even thought about yet. | |
| 13:11:14 | bauzas | efried: trying to get a consensus for Rocky-1 or Rocky-2 | |
| 13:11:25 | bauzas | efried: while I'm implementing other things | |
| 13:11:35 | efried | bauzas: roger that | |
| 13:11:42 | bauzas | efried: but once I'm done with the other things, going back to NUMA for implementing | |
| 13:12:04 | bauzas | honestly, if we don't need new Placement version, I think it can be done very quickly | |
| 13:12:07 | bauzas | efried: ^ | |
| 13:12:21 | efried | yes | |
| 13:12:25 | bauzas | efried: it's just about translating specs into Placement queries | |
| 13:12:32 | efried | yes | |
| 13:12:48 | bauzas | so, at least the NUMAFilter would still check the hosts, but it would only check the accepted ones | |
| 13:13:10 | efried | And I actually like the idea of translating the existing numa-related flavor specs into placement queries, rather than asking folks to rewrite their flavors with placement-y syntax. | |
| 13:13:16 | bauzas | and given the default flag value for NUMA will be None, nothing should change actually | |
| 13:13:40 | bauzas | unless someone wants to test that magic bullet to restrict hosts passed to the scheduler | |
| 13:13:43 | efried | Because that way we can make placement queries that would be... unreasonable for humans to come up with. | |
| 13:14:04 | bauzas | it's more about an upgrade question for me | |
| 13:14:19 | bauzas | efried: modifying flavors could be a problem for upgrading | |
| 13:14:29 | bauzas | if we support the existing flavors, that's better | |
| 13:14:36 | efried | ++ | |
| 13:15:04 | bauzas | stephenfin: so, to clarify, you mean that we don't explicitely shard between NUMA nodes ? | |
| 13:15:43 | stephenfin | bauzas: We do but it's best effort | |
| 13:16:09 | bauzas | stephenfin: tbc, if you're asking for hw:numa_nodes=2 but you only have one NUMA node (or even a UMA topology), then we accept the host, right?N | |
| 13:16:19 | stephenfin | bauzas: yup | |
| 13:16:25 | bauzas | perfect, efried ^ | |
| 13:17:00 | efried | Cool beans. One down, three to go. | |
| 13:17:07 | bauzas | stephenfin: so, it's more about NUMA "affinity" of multiple resources, rather than NUMA 'anti-affinity' of different CPUs | |
| 13:17:19 | bauzas | still right? | |
| 13:17:35 | bauzas | if so, numbered request groups is the perfect expression | |
| 13:17:49 | stephenfin | It's not really anything to do with NUMA affinity, tbh. It's purely to do with the guest topology | |
| 13:17:54 | bauzas | perfect | |
| 13:18:16 | bauzas | it doesn't guaranttee you'll land all your CPUs on specific NUMA nodes | |
| 13:18:23 | stephenfin | The fact that we don't split a guest NUMA node across a host NUMA node is a performance improvement but not essential by any means | |
| 13:18:26 | bauzas | you could have all your CPUs on the same | |
| 13:18:33 | stephenfin | correct | |
| 13:18:41 | bauzas | excellent, good news | |
| 13:18:49 | bauzas | and tbh, I understand the reasoning | |
| 13:19:01 | stephenfin | so booting an instance with hw:numa_nodes=4 on a dual socket system is a valid thing to do | |
| 13:19:02 | bauzas | you could discover the host topology as an end-user if we were restricting | |
| 13:19:12 | stephenfin | assuming total vCPUs < total host CPUs | |
| 13:19:20 | stephenfin | 'zactly | |
| 13:19:24 | bauzas | just by trying to boot a couple of different topologies | |
| 13:19:27 | stephenfin | There's also zero reason to allow it | |
| 13:19:53 | bauzas | efried: so you agree the numbered request groups feature exactly matches the above ? ^ | |
| 13:20:57 | sahid | bauzas, stephenfin, if user is asking for hw:numa_node=2 and host does not have at least 2 NUMA node we don't accept the host | |
| 13:21:08 | sahid | or it's a bug and we should fix it | |
| 13:21:27 | stephenfin | sahid: Why? | |
| 13:21:36 | bauzas | sahid: that's very different from what stephenfin says | |
| 13:21:48 | bauzas | http://www.quickmeme.com/meme/361gwd | |
| 13:22:00 | sahid | because it's all performance related | |
| 13:22:22 | sahid | it's all about distance between pci devices, memory channel an cpus | |
| 13:24:07 | sahid | stephenfin: we should have in hardware.py constrainsts something which check that | |
| 13:24:14 | stephenfin | bauzas: Ah, crap. sahid's correct there | |
| 13:24:42 | stephenfin | bauzas, sahid: https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1548-L1553 | |
| 13:25:35 | bauzas | sahid: so, like I said, say I'm an evil folk named Eve, I could guess the host topology by booting a couple of test instances having various NUMA guest topologies, right? | |
| 13:25:58 | stephenfin | bauzas: and https://github.com/openstack/nova/blob/master/nova/objects/instance_numa_topology.py#L217-L219 | |
| 13:26:23 | bauzas | I understand the performance reason for affinitizing the guest, but not for anti-affinitizing it | |
| 13:26:27 | stephenfin | I thought that was comparing the length of cell.cpuset, a la https://github.com/openstack/nova/blob/master/nova/objects/instance_numa_topology.py#L82-L83 | |
| 13:26:39 | stephenfin | sahid: Yeah, why do we do that? It seems unnecessary | |
| 13:27:34 | stephenfin | Like I said above, I get the reason for not splitting guest NUMA nodes across host NUMA nodes | |
| 13:27:37 | stephenfin | and for placing at least one guest NUMA node's CPUs on the host NUMA node associated with a PCI device | |
| 13:27:52 | bauzas | stephenfin: checking whether the host has the same number of NUMA nodes that the guest is one thing | |
| 13:28:10 | bauzas | stephenfin: the other thing being that we would restrict that to different nodes | |
| 13:28:18 | sahid | what is the use-cases? I mean why you want to have differente NUMA nodes if the CPU is using a memory channel which is on an different NUMA node? as I said it's all performance related | |
| 13:28:59 | stephenfin | sahid: Right, but I have two NUMA nodes and I boot the instance with four NUMA nodes. Why can't two go on one host node and two on the other? | |
| 13:29:03 | bauzas | sahid: performance is affinity, and I don't disagree with you | |
| 13:29:24 | stephenfin | Sounds like you'd get the same performance as with an instance with two NUMA nodes that's split across the two host nodes | |