| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 13:11:54 | stephenfin | The better model, which is what jaypipes, gibi and I discussed, was to use placement for this and collaboratively build up this model between nova and neutron. However, placement isn't there yet | |
| 13:12:28 | stephenfin | sahid: So this is making the best of a bad situation. If I've missed something though, definitely let me know. | |
| 13:12:35 | stephenfin | I'll put all the above in the review | |
| 13:16:13 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.openstack.org/552924 | |
| 13:21:38 | sahid | stephenfin: we don'-t need to ask that question "give me the NUMA affinity for this PCI device I've attached to my bridge" to OVS | |
| 13:22:04 | sahid | the operator is going to configure OVS and actually DPDK based on where the device is located | |
| 13:22:45 | sahid | so basically what we need is just to retrieve where the vhu ports are created | |
| 13:22:55 | sahid | this can be done by asking OVS | |
| 13:23:16 | sahid | so the neutron agent can return such information to nova by the binding detail od the port | |
| 13:23:20 | stephenfin | So just query the PMD pinning information for a bridge? | |
| 13:23:24 | sahid | do i have mised something? | |
| 13:24:38 | sahid | hum... no sure i underatand the PMD pinning for a bridge? | |
| 13:25:07 | stephenfin | It seemed like a big assumption to make (that PMD threads would be affined with the NIC), especially given that they don't have to be with recent releases | |
| 13:25:56 | sahid | well if operators want best perofrmance they have to do that | |
| 13:26:09 | stephenfin | Sorry, not the bridge. I'm referring to this https://developers.redhat.com/blog/2017/06/28/ovs-dpdk-parameters-dealing-with-multi-numa/ | |
| 13:26:14 | sahid | it's not our responsability (i don't think so) | |
| 13:26:27 | stephenfin | e.g. 'ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xF0' | |
| 13:26:45 | sahid | so that is configurd by operator | |
| 13:27:08 | sahid | we are expecting that the pmd to run on the same NUMA node where the phy NIC is, right? | |
| 13:27:41 | stephenfin | Yes. There can be multiple NICs | |
| 13:27:45 | stephenfin | ...too | |
| 13:28:11 | stephenfin | So NIC X is on NUMA node 0, while NIC Y is on NUMA node 1 | |
| 13:28:35 | sahid | but NIC X and NIC Y have different network | |
| 13:28:37 | sahid | right? | |
| 13:28:51 | stephenfin | and NIC X is connected to/tagged with physnet_x, and NIC Y to physnet_y | |
| 13:28:52 | stephenfin | right | |
| 13:29:10 | sahid | so when neutron is asking to create a port for X | |
| 13:29:38 | sahid | the agent can query OVS to know where that vhu is located? | |
| 13:29:54 | stephenfin | Can they? | |
| 13:30:02 | sahid | yep | |
| 13:30:08 | sahid | let a sec to find the command | |
| 13:30:27 | stephenfin | Right, I didn't know that :D | |
| 13:31:06 | alex_xu_ | efried: I missed one thing. If the resource provider X is the compute node. RP X provides VCPU and memory also. Each instance will consume the VCPU and memory. In that case, RP X will be returned | |
| 13:31:14 | sahid | stephenfin: https://software.intel.com/en-us/articles/vhost-user-numa-awareness-in-open-vswitch-with-dpdk | |
| 13:31:28 | sahid | if we can know here a port is located | |
| 13:31:31 | stephenfin | mikal: Just the man I'm looking for. Fancy jotting down your thoughts on https://review.openstack.org/554195 | |
| 13:31:38 | sahid | the agent can return this information to nova, no? | |
| 13:32:19 | stephenfin | (random aside: the OVS (-DPDK) documentation is hands-down awful. It sucks that we have to resort to random blogs for this critical information) | |
| 13:32:38 | stephenfin | sahid: If it's what we want then I don't see why not. Lemme check | |
| 13:33:42 | stephenfin | ooh, so 'pmd-rxq-show' does seem to be exactly what I wanted | |
| 13:35:06 | stephenfin | ...and we'd just assume that OVS was configured correctly so that there are PMD threads on all NUMA nodes, which is Red Hat's guidance at least | |
| 13:35:18 | stephenfin | sean-k-mooney[m]: If you're about, any thoughts on ^ | |
| 13:36:17 | efried | alex_xu_: Ah, then that's a mistake in modeling. | |
| 13:37:20 | efried | alex_xu_: The RP tree should not be set up such that the compute host provides inventory of FPGA. | |
| 13:37:45 | alex_xu_ | efried: yes, I realized that also, that isn't the FPGA case now | |
| 13:37:49 | bauzas | alex_xu_: efried: which specific spec are you discussing ? | |
| 13:37:59 | stephenfin | efried: See what you made me do? https://review.openstack.org/556850 | |
| 13:38:19 | alex_xu_ | bauzas: here https://review.openstack.org/#/c/554305/ | |
| 13:38:46 | alex_xu_ | efried: basically, it only can happened in the compute node RP, | |
| 13:39:04 | bauzas | alex_xu_: ack, thanks | |
| 13:39:15 | efried | alex_xu_: Well, really any RP in the tree which provides inventory in multiple RCs that are "unrelated" to each other. | |
| 13:39:25 | efried | alex_xu_: This is actually what I wrote a bug about last year. | |
| 13:39:54 | alex_xu_ | efried: ha, currently we have VGPU, but will change soon :) | |
| 13:40:00 | kashyap | "the operator is going to configure OVS and actually DPDK based on where the device is located" --> Hope the Operator has the necessary PhDs to configure DPDK et al | |
| 13:40:33 | alex_xu_ | efried: but I agree with that | |
| 13:40:47 | alex_xu_ | I feel it will be rare case in the future | |
| 13:41:00 | bauzas | alex_xu_: efried: FWIW, I discussed this morning with jianghuaw_ about some possible quirks with VGPUs | |
| 13:41:10 | bauzas | and I'd love your thoughts | |
| 13:41:16 | bauzas | because it would need to spec up | |
| 13:41:36 | bauzas | the fact is that physical GPUs support multiple types | |
| 13:41:51 | bauzas | depending on the type, you can have different number of vGPUs | |
| 13:41:58 | bauzas | so, guess what? | |
| 13:42:12 | efried | alex_xu_: I can't find the bug right offhand, but I think the scenario I described was: compute RP has HDD disk, sharing provider (or in fact another RP in the tree) has SSD disk. I ask for disk & VCPU, I ask for the HDD trait. I will get back candidates which include the SSD, because the compute RP is satisfying the HDD trait, even though it's not providing the disk resource. | |
| 13:42:18 | alex_xu_ | we need to update inventory based the user request? | |
| 13:42:26 | bauzas | alex_xu_: efried: in theory, a nested RP that would be a pGPU could have different inventories for the same resource class | |
| 13:43:12 | bauzas | alex_xu_: efried: but the trick is, once a vGPU is created, then all the other types but the one picked for that vGPU will have total=0 for their inventories | |
| 13:43:30 | efried | bauzas: I think we've discussed similar issues in the past. Trying to recall what we came up with as a solution... | |
| 13:43:35 | bauzas | alex_xu_: efried: so there are 2 ways to consider that | |
| 13:43:53 | bauzas | either we ask the operator to define which type he wants to support per-GPU | |
| 13:44:06 | alex_xu_ | efried: ah, I see, but I guess that bug is killed by the reality that we don't support two different storage on the host | |
| 13:44:08 | bauzas | and in that case, the enabled_vgpu_types option needs to be change | |
| 13:44:11 | bauzas | changed* | |
| 13:44:14 | efried | alex_xu_: Oh, but we do :) | |
| 13:44:53 | bauzas | or, we magically provide a 2nd level of the tree with nested RPs, meaning that a pGPU will have multiple children, each one being something like pGPU_thistype | |
| 13:45:30 | efried | bauzas: That only helps if you want to lock it down (predefine and preallocate) | |
| 13:45:36 | bauzas | the second solution makes the thing less impactful for operators, but it adds a second level to the tree just for VGPUs | |
| 13:45:47 | bauzas | efried: not really | |
| 13:46:03 | bauzas | efried: because at start, each inventory will provide all the possible vGPUs | |
| 13:46:24 | efried | bauzas: But you'd still be counting on an allocation from *here* affecting the inventory over *there* | |
| 13:46:30 | efried | which we can't do. | |
| 13:46:40 | bauzas | efried: in theory the allocation would be against the pGPU_type | |
| 13:46:46 | bauzas | not the pGPU itself | |
| 13:47:02 | efried | Yeah, I get that, but you would have had to pre-designate how many of each type you start with. | |
| 13:47:21 | bauzas | efried: that's the config option | |
| 13:47:23 | efried | You can't show "full" inventory for all types. | |
| 13:47:35 | efried | Okay, right, so swhat I'm saying, it's locked down ahead of time. | |
| 13:47:40 | efried | via the config option | |
| 13:47:47 | bauzas | for the moment, the config option is a ListOpt supporting types | |
| 13:48:15 | bauzas | but honestly, I think we should do something less crazy and just have an option that would do like pci passthrough | |
| 13:48:28 | bauzas | ie. "for that PCI device, here is the type" | |
| 13:48:51 | bauzas | jaypipes: thoughts on that ? ^ | |
| 13:48:58 | efried | bauzas: If you want ultimate flexibility, I think you need to provide inventory of some RC that lets you represent "units of VGPU-ness", where different types consume a different number of that resource. And then your request would have to be well-behaved and ask for resources=VGPU:1,VGPU_UNITY_THINGY:4 | |
| 13:49:26 | efried | ...where the request for VGPU_UNITY_THINGY has to be correct for the type you're requesting. | |
| 13:49:53 | openstackgerrit | Merged openstack/os-traits master: Updated from global requirements https://review.openstack.org/551599 | |
| 13:50:04 | bauzas | efried: that's where I think we're over engineering | |
| 13:50:22 | efried | That would be clearer as | |
| 13:50:22 | efried | resources=VGPU_TYPE_X:1,VGPU_UNITY_THINGY:4 | |
| 13:50:22 | efried | or | |
| 13:50:22 | efried | resources=VGPU:1,VGPU_UNITY_THINGY:4&required=VGPU_TYPE_X | |
| 13:50:25 | bauzas | efried: operators want flavors like VGPU=2&trait=MY_TYPE | |