| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-07 | |||
| 15:09:10 | efried | including things like accelerators? | |
| 15:09:15 | efried | and VGPUs? | |
| 15:09:21 | efried | And bandwidth? | |
| 15:09:31 | efried | esp. that last one | |
| 15:09:38 | efried | because we need to be able to do affinity by subtree matching | |
| 15:09:39 | sean-k-mooney | if we also support mixed cpus in instance we will jsut combien the mask with the hw:numa_cpu.X=Y status | |
| 15:09:46 | sean-k-mooney | *spec | |
| 15:09:54 | noonedeadpunk | mnaser: I guess writing a patch might be quicker that filing in a bug... | |
| 15:09:56 | efried | ...when the requests come from different thingies | |
| 15:10:07 | huaqiang | sean-k-mooney: morning | |
| 15:10:47 | stephenfin | efried: 'hw:numa_affinity:$RESOURCE_CLASS:$POLICY' or something similar? | |
| 15:11:26 | stephenfin | e.g. 'hw:numa_affinity:VCPU:prefer' | |
| 15:11:30 | stephenfin | *VGPU | |
| 15:11:52 | gibi | I don't think we ever want to request bandwidth via flavor exta_spec | |
| 15:11:59 | efried | exactly gibi | |
| 15:12:12 | efried | but we still need to be able to express (NUMA) affinity | |
| 15:12:31 | stephenfin | We're not. We're just saying what the affinity of whatever the bandwidth resource class is | |
| 15:12:40 | stephenfin | Assuming we can know that ahead of time (idk) | |
| 15:12:48 | efried | by tying together the port bw request group suffix and the NUMA request group suffix | |
| 15:13:32 | efried | there's going to be a coupling problem for sure | |
| 15:13:38 | efried | like what we have for group_policy, only much worse. | |
| 15:17:24 | gibi | if nova start using the port uuid as the port suffix for the bandwidth request then a single use flavor can be created where the port group can be referenced for affinity. But this sounds pretty horrific | |
| 15:17:37 | gibi | s/port suffix/group suffix/ | |
| 15:18:09 | efried | agree, I don't think we want the suffix to be a port UUID, do we? | |
| 15:18:36 | gibi | efried: currently each port's resource request is understood as a separate group | |
| 15:18:52 | gibi | efried: so a natural suffix for the group could be the port uuid | |
| 15:19:12 | sean-k-mooney | stephenfin: why woudl we need hw:numa_affinity:VCPU:prefer | |
| 15:19:26 | stephenfin | sean-k-mooney: *VGPU | |
| 15:19:43 | sean-k-mooney | oh for graphics ya | |
| 15:20:12 | stephenfin | Just some way to define NUMA affinity for individual resource class types | |
| 15:20:24 | sean-k-mooney | cpus and memory are are affinites by adding hw:numa_node or one of the option that imples it so i was confused | |
| 15:20:27 | stephenfin | It doesn't allow you to define the same for individual instances, but that's slightly insane | |
| 15:20:34 | stephenfin | Yup, my mistake | |
| 15:21:19 | sean-k-mooney | well you would be surprised what we have approved in the past and never implmeneted. | |
| 15:22:31 | sean-k-mooney | stephenfin: https://github.com/openstack/nova-specs/blob/master/specs/newton/approved/virt-image-props-boot-override.rst | |
| 15:23:34 | efried | gibi: I guess it's not really relevant until we can tie in whatever piece of hardware is associated with the NUMA node. RP for a NIC PF, kind of thing. Cause today you don't request network VF resources or whatever, right? | |
| 15:24:20 | gibi | efried: right. we don't request VF resource from neutron via the resource request. | |
| 15:25:07 | gibi | efried: you are also correct that when VF will be requested from neturon along with the bandwidth then expressing NUMA affinity will be an issue to be solved | |
| 15:25:59 | efried | gibi: Won't the majority of affinity use cases involve only one NUMA node? | |
| 15:26:18 | sean-k-mooney | we request vf internally in nova currently based on the vif type | |
| 15:26:32 | efried | sean-k-mooney: yes but VF not tracked by placement yet | |
| 15:26:36 | efried | is my point | |
| 15:26:36 | sean-k-mooney | but sriov vf will be owned and created by nova in placmenet in the future | |
| 15:26:39 | sean-k-mooney | not by neutron | |
| 15:26:40 | efried | right | |
| 15:26:47 | efried | oh | |
| 15:26:54 | efried | oh? | |
| 15:27:06 | efried | That will make it tougher | |
| 15:27:12 | sean-k-mooney | yes all pci decice handeling is currently done by nova | |
| 15:27:14 | efried | to affine the bw resources with the vf resources | |
| 15:27:26 | sean-k-mooney | neutron does not have a whitelist of device that can be used | |
| 15:27:28 | efried | what if it was a VIF resource? | |
| 15:28:03 | efried | Anyway, if we had a one-NUMA-node use case, we could handle this fairly well. Even multiple NUMA nodes if they're symmetrical. | |
| 15:28:05 | sean-k-mooney | we discussed adding vif ressource that would be owned by the neutron backend too | |
| 15:28:51 | efried | A case where I want $this specific VF/bw associated with $big_numa and $that specific VF/bw associated with $small_numa that things get really hairy. | |
| 15:29:13 | sean-k-mooney | well nomally its not symetric | |
| 15:29:25 | sean-k-mooney | idealy it would be | |
| 15:29:37 | sean-k-mooney | but its quite common to only have nics attached to numa 0 | |
| 15:29:39 | efried | It's not? I thought that was the point of saying hw:numa_nodes=2 | |
| 15:29:51 | sean-k-mooney | no | |
| 15:29:54 | efried | Sorry, I mean if the CPU/mem resources in the NUMA nodes are symmetric | |
| 15:30:05 | efried | I don't mean "same number of VFs per NUMA node" | |
| 15:30:05 | sean-k-mooney | hw:numa_nodes=2 is refering to the guest numa toplogy | |
| 15:30:19 | sean-k-mooney | oh by default yest | |
| 15:30:22 | sean-k-mooney | *yes | |
| 15:30:28 | efried | when we do that we're dividing evenly, yah? | |
| 15:31:00 | sean-k-mooney | you can change that with hw:numa_mem.0=512 hw:numa_mem.1=1536 | |
| 15:31:02 | huaqiang | sean-k-mooney: hi | |
| 15:31:20 | efried | sean-k-mooney: Yes, exactly, that's my point: those cases are rarer, right? | |
| 15:31:23 | sean-k-mooney | efried: yes if you jsut do hw:numa_nodes it devices cpus and ram evenly | |
| 15:31:36 | sean-k-mooney | we also check its evenly deivisable | |
| 15:31:50 | efried | sean-k-mooney: Do we currently support VF NUMA affinity? | |
| 15:31:56 | sean-k-mooney | efried: yes | |
| 15:32:06 | sean-k-mooney | for 4+ years | |
| 15:32:29 | sean-k-mooney | we require it in the case of neutron sriov which is what im trying to chagne | |
| 15:32:51 | sean-k-mooney | with https://review.opendev.org/#/c/674072/ | |
| 15:33:05 | sean-k-mooney | gibi: ^ i would like your input on that by the way | |
| 15:33:26 | sean-k-mooney | huaqiang: hi | |
| 15:33:42 | efried | sean-k-mooney: even in the case of lopsided NUMA nodes? You can say which one you want your VF associated with? | |
| 15:33:52 | sean-k-mooney | efried: no | |
| 15:33:58 | sean-k-mooney | you cannot specify that at all | |
| 15:34:00 | efried | okay, then we're fine. | |
| 15:34:08 | efried | we continue to say you can't specify that. | |
| 15:34:16 | sean-k-mooney | we guarentee it will come form one of the numa ndoes the vm is mapped too | |
| 15:34:37 | sean-k-mooney | but we give no garuntee it on a specific guest numa node | |
| 15:34:48 | gibi | efried: if nova keeps handling the VFs then nova can do a VF-NUMA affinity. Bandwidth is already tied to VF today in case of SRIOV. | |
| 15:35:02 | sean-k-mooney | currently its always preseneted as if its attache to guest numa node 0 regradelss of what host numa node it came from | |
| 15:35:05 | efried | and we can give the requests for NUMA nodes and VFs generic known suffixes, and use same_subtree to tie them together. | |
| 15:35:16 | efried | perfect | |
| 15:35:20 | sean-k-mooney | i also want to chagne that but not in a way tha will impact placmenet or shcudleing in any way | |
| 15:35:40 | efried | Yeah, let's not give any more granular control over that | |
| 15:35:47 | efried | cause then we would have to support it via placement :) | |
| 15:36:06 | efried | Okay, it's the middle of the night for huaqiang, let's wrap up and yield the floor... | |
| 15:36:17 | sean-k-mooney | efried: what i want to add i the use of a livirt feature that allows mupliple pci roots in the guest | |
| 15:36:34 | huaqiang | sean-k-mooney: got the comments for the 'using PCPU VCPU in one instance', thanks | |
| 15:36:52 | sean-k-mooney | that will allwo us to attach the pci device to the virtual numa node that maps to the host numa node the deive is on | |
| 15:37:16 | huaqiang | but I checked the existing logic for 'hw:cpu_policy=dedicated' and 'hw_cpu_policy=shared', the final policy is 'dedicated' | |
| 15:38:06 | sean-k-mooney | can you show me that in code | |
| 15:38:18 | huaqiang | sure | |
| 15:39:27 | sean-k-mooney | its this https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1327-L1338 | |