| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 12:32:41 | efried | Good UGT morning bhagyashris | |
| 12:33:33 | efried | bhagyashris: Are you going to take over https://review.openstack.org/560444 and https://review.openstack.org/560459 ? | |
| 12:34:55 | bhagyashris | efried: Good morning :) | |
| 12:42:22 | bhagyashris | efried: ok i will take. | |
| 12:43:34 | efried | bhagyashris: Great. Can you put your name down on the etherpad https://etherpad.openstack.org/p/rocky-nova-priorities-tracking see L115 | |
| 12:43:54 | efried | jaypipes: FYI ^ | |
| 12:47:39 | bauzas | stephenfin: quick question, if I'm providing a extra spec saying --property hw:numa_nodes=N, does it mean that the instance will use 2 physical NUMA nodes, or just that the instance XML will have 2 nodes ? | |
| 12:48:13 | bauzas | if the latter, are we supporting to not have sharded the NUMA nodes ? | |
| 12:48:37 | efried | sean-k-mooney, cfriesen: ^ | |
| 12:48:40 | bauzas | stephenfin: context is https://review.openstack.org/#/c/552924/7/specs/rocky/approved/numa-topology-with-rps.rst@185 | |
| 12:52:02 | bauzas | efried: as you can see, I wonder if we would still need a specific sharding param | |
| 12:52:47 | bauzas | efried: because when we discussed yesterday, I was only thinking of vGPUs affinited to CPUs, not about vCPUs being anti-affinited by NUMA nodes | |
| 12:53:05 | bauzas | so the latter would need something missing now | |
| 12:53:21 | bauzas | and then, in that case, that would be in my spec : | |
| 12:57:33 | efried | bauzas: Yup; and I totally can't answer that question (whether we need explicit anti-affinity/sharding) | |
| 12:57:47 | efried | But your spec definitely must. | |
| 12:57:47 | bauzas | efried: when looking at the doc, see " FLAVOR-NODES: (integer) The number of host NUMA nodes to restrict execution of instance vCPU threads to. If not specified, the vCPU threads can run on any number of the host NUMA nodes available." | |
| 12:57:51 | bauzas | https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs-numa-topology | |
| 12:58:17 | efried | bauzas: Oh, that's interesting. I read that as a *maximum*. | |
| 12:58:41 | efried | In which case, we're covered. | |
| 12:58:50 | efried | sort of | |
| 12:58:59 | efried | We need to split into N numbered request groups | |
| 12:59:24 | efried | The only issue is that we can't specify that in a flexible way - it would have to be specific numbers of procs in each group. | |
| 12:59:36 | efried | and the only reasonable way to do that is to divide them evenly. | |
| 13:00:03 | efried | That still means we get *at most* N separate NUMA nodes, but because granular doesn't guarantee separation, we could get anywhere from 1..N | |
| 13:00:13 | bauzas | that said, in https://docs.openstack.org/nova/latest/admin/cpu-topologies.html | |
| 13:00:38 | bauzas | it says " Inadequate per-node resources will result in scheduling failures. Resources that are specific to a node include not only CPUs and memory, but also PCI and SR-IOV resources. It is not possible to use multiple resources from different nodes without requesting a multi-node layout. As such, it may be necessary to ensure PCI or SR-IOV resources are associated with the same NUMA node or force a multi-node layout." | |
| 13:01:02 | efried | That's fair. | |
| 13:01:22 | bauzas | what I understand from the above is that if I'm asking for 2 nodes but then only have 1 node, then NoValidHost | |
| 13:01:33 | bauzas | that's honeslty confusing | |
| 13:01:35 | efried | no, that's not how I read it. | |
| 13:01:52 | openstack | Launchpad bug 1466780 in OpenStack Compute (nova) "nova libvirt pinning not reflected in VirtCPUTopology" [Undecided,Invalid] - Assigned to Stephen Finucane (stephenfinucane) | |
| 13:01:52 | bauzas | also given https://bugs.launchpad.net/nova/+bug/1466780 and " There is no correlation required between the NUMA topology exposed in the instance and how the instance is actually pinned on the host. This is by design. See this invalid bug for more information." | |
| 13:02:06 | bauzas | but I take that only for CPU pinning | |
| 13:02:15 | efried | I read it to mean, "if you're restricting to one NUMA node, but no single node has all the resources available, NoValidHost" Which is totally legit. | |
| 13:02:35 | bauzas | let's wait for the others, then | |
| 13:02:37 | kholkina | the spec for user-data update need your review https://review.openstack.org/#/c/547964/ | |
| 13:02:43 | efried | But it brings us to a different issue, which is whether your spec claims to handle device affinity as well as proc/mem | |
| 13:03:05 | bauzas | efried: that said, if we need to shard, what kind of query param are you thinking ? | |
| 13:03:33 | openstackgerrit | sahid proposed openstack/nova master: libvirt: move version to string in utils https://review.openstack.org/560455 | |
| 13:03:34 | openstackgerrit | sahid proposed openstack/nova master: libvirt: refactor get_base_config to accept host arg https://review.openstack.org/560456 | |
| 13:03:35 | openstackgerrit | sahid proposed openstack/nova master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/484997 | |
| 13:03:36 | bauzas | that's the problem with numbered request groups | |
| 13:03:41 | efried | bauzas: For explicitly separating request groups to separate resource providers? I was thinking something like ?separate_providers=resources1,resources2,... | |
| 13:03:55 | efried | ...if placement is where we want to handle it. | |
| 13:04:26 | efried | bauzas: But the other option is to handle it in the NUMATopologyFilter. Placement would give us back all the candidates, which would include the ones that are sharded and the ones that are combined. And the filter would just pick the ones that are appropriately sharded. | |
| 13:04:35 | bauzas | so, ?separate_providers=resources1,resources2&resources1:VCPU=1&resources2:VCPU=1 ? | |
| 13:04:44 | efried | bauzas: yes | |
| 13:05:14 | efried | Implementing that will be a bear, but yes. | |
| 13:05:20 | bauzas | if placement folks are accepting that, then I'd prefer to do that by Placement instead of the filter | |
| 13:05:34 | bauzas | because the less we have in the filter, the better it will be | |
| 13:05:42 | efried | I don't see it making Rocky, tbh | |
| 13:05:46 | bauzas | longer term of course | |
| 13:05:50 | bauzas | efried: yeah, I know | |
| 13:06:00 | bauzas | efried: what I'd love is some consensus on that spec for Rocky | |
| 13:06:03 | efried | But yes, eventually we recognize we're going to need that functionality in placement. | |
| 13:06:20 | efried | bauzas: which spec? | |
| 13:06:21 | bauzas | but then, earlyj working on Rocky-3 if I'm lucky so we can land Stein-1 | |
| 13:06:29 | bauzas | efried: the NUMA one | |
| 13:06:53 | efried | bauzas: Sure, agreed; but we need to crisp up what exactly you're trying to address, and how. | |
| 13:07:00 | efried | That's not yet clear IMO | |
| 13:07:19 | bauzas | efried: for Rocky, I'm only planning to implement nested RPs for vGPUs and fix the vGPU caveats | |
| 13:07:25 | bauzas | plus that spec | |
| 13:07:34 | bauzas | but implementing that spec for Rocky-3 or later | |
| 13:07:49 | bauzas | unless someone picks the ball | |
| 13:07:54 | efried | Okay, gotcha. | |
| 13:08:28 | efried | So you're not thinking to land the code related to the NUMA spec until "later". But you want the spec baked by Rocky-3 | |
| 13:10:03 | stephenfin | bauzas: The latter | |
| 13:10:32 | efried | johnthetubaguy_: You around? I approved https://review.openstack.org/#/c/553605/ on the basis that your -1 has now been addressed, but there's still time to pull it out if you disagree. | |
| 13:10:39 | stephenfin | bauzas: typically those guest nodes would be scheduled to different NUMA nodes, but if that's not possible they'll be squeezed onto the same one | |
| 13:11:00 | stephenfin | bauzas: At least that's the case to the best of my recollection | |
| 13:11:00 | bauzas | efried: that's correct | |
| 13:11:09 | efried | ahjeez. "anti-affinity preferred" is a use case we haven't even thought about yet. | |
| 13:11:14 | bauzas | efried: trying to get a consensus for Rocky-1 or Rocky-2 | |
| 13:11:25 | bauzas | efried: while I'm implementing other things | |
| 13:11:35 | efried | bauzas: roger that | |
| 13:11:42 | bauzas | efried: but once I'm done with the other things, going back to NUMA for implementing | |
| 13:12:04 | bauzas | honestly, if we don't need new Placement version, I think it can be done very quickly | |
| 13:12:07 | bauzas | efried: ^ | |
| 13:12:21 | efried | yes | |
| 13:12:25 | bauzas | efried: it's just about translating specs into Placement queries | |
| 13:12:32 | efried | yes | |
| 13:12:48 | bauzas | so, at least the NUMAFilter would still check the hosts, but it would only check the accepted ones | |
| 13:13:10 | efried | And I actually like the idea of translating the existing numa-related flavor specs into placement queries, rather than asking folks to rewrite their flavors with placement-y syntax. | |
| 13:13:16 | bauzas | and given the default flag value for NUMA will be None, nothing should change actually | |
| 13:13:40 | bauzas | unless someone wants to test that magic bullet to restrict hosts passed to the scheduler | |
| 13:13:43 | efried | Because that way we can make placement queries that would be... unreasonable for humans to come up with. | |
| 13:14:04 | bauzas | it's more about an upgrade question for me | |
| 13:14:19 | bauzas | efried: modifying flavors could be a problem for upgrading | |
| 13:14:29 | bauzas | if we support the existing flavors, that's better | |
| 13:14:36 | efried | ++ | |
| 13:15:04 | bauzas | stephenfin: so, to clarify, you mean that we don't explicitely shard between NUMA nodes ? | |
| 13:15:43 | stephenfin | bauzas: We do but it's best effort | |
| 13:16:09 | bauzas | stephenfin: tbc, if you're asking for hw:numa_nodes=2 but you only have one NUMA node (or even a UMA topology), then we accept the host, right?N | |
| 13:16:19 | stephenfin | bauzas: yup | |
| 13:16:25 | bauzas | perfect, efried ^ | |
| 13:17:00 | efried | Cool beans. One down, three to go. | |
| 13:17:07 | bauzas | stephenfin: so, it's more about NUMA "affinity" of multiple resources, rather than NUMA 'anti-affinity' of different CPUs | |
| 13:17:19 | bauzas | still right? | |
| 13:17:35 | bauzas | if so, numbered request groups is the perfect expression | |