| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 14:48:20 | dmitriis | sean-k-mooney: i.e. having a DPU and a regular SR-IOV card | |
| 14:48:37 | sean-k-mooney | well dpu and another nic that can be used with ovs or linux bridge | |
| 14:48:41 | dmitriis | right | |
| 14:49:03 | dmitriis | I haven't tested that explicitly but just trying to think what would that entail | |
| 14:49:07 | sean-k-mooney | basically it was raised that by using the dpu you are limiting the numer or port that can be created to the number of vfs | |
| 14:49:22 | sean-k-mooney | dmitriis: i think it should just work honestly | |
| 14:49:32 | sean-k-mooney | i was just wondering if you had considerd it | |
| 14:49:50 | dmitriis | sean-k-mooney: yes, the limitation on the number of ports have definitely come up in our conversations | |
| 14:50:11 | sean-k-mooney | dmitriis: all it really requires is the ml2 dirver changes to not require you to designate a host as only dpu enabled | |
| 14:50:18 | dmitriis | and nothing immediately jumps into mind regarding OVS on the host being a problem | |
| 14:50:23 | sean-k-mooney | e.g. make the binding desiion per port based on vnic_type | |
| 14:50:54 | sean-k-mooney | ack that is what my assement was too | |
| 14:51:04 | dmitriis | sean-k-mooney: yeah and IIRC it's just based on the vnic_type, we just added handling for vnic_type smartnic to the OVN mechanism driver | |
| 14:51:14 | dmitriis | so it should just bind a normal port regularly | |
| 14:51:20 | sean-k-mooney | yep | |
| 14:51:23 | sean-k-mooney | cool | |
| 14:51:44 | sean-k-mooney | we are still trying to assess what woudl be requried to eventually support this in our product | |
| 14:52:04 | dmitriis | sean-k-mooney: yeah, plus hardware is hard to come by | |
| 14:52:12 | sean-k-mooney | that too | |
| 14:52:45 | dmitriis | I think this feature will be useful for FPGA-based SmartNICs as well. I've seen the ones with a separate CPU but with an FPGA instead of an ASIC | |
| 14:53:24 | dmitriis | considering the DPDK dataplane usage at the SmartNIC's CPU side for that with rte_flow for offload | |
| 14:53:26 | sean-k-mooney | i do potentially have access to a BF2 that i coudl use but only 1 so i obviouly cant test multi node and in general an all in one deployment while useful is not fully reflective of how it would be deployed in a datacenter | |
| 14:53:34 | dmitriis | the Nova and Neutron bits would work the same though | |
| 14:53:46 | sean-k-mooney | dmitriis: yes in princial it could be | |
| 14:53:54 | dmitriis | sean-k-mooney: I have an idea on how to test multi-node with just one BF2 | |
| 14:54:15 | sean-k-mooney | the only fpga based smartnic i have use used armstong os with only 2 arm cores and i think 4GB of ram | |
| 14:54:44 | sean-k-mooney | dmitriis: run two copies of nova-compute with difernt host values in teh conf on the same host | |
| 14:54:53 | dmitriis | sean-k-mooney: 2 VMs with nested virt a the hypervisor, plus 2 VMs at the DPU side. Each VM uses 1 PF. VPD can be faked by bind mounting a file in the right sysfs location | |
| 14:54:53 | sean-k-mooney | and use differnt ports on the BF2? | |
| 14:55:08 | dmitriis | BF2's ARM CPU is virt-capable btw xD | |
| 14:55:43 | sean-k-mooney | dmitriis: that proably wotn work becaue wehn the PF is passed to the qemu instacle the pcie capablityies are not in the pci config space in teh guest | |
| 14:56:10 | sean-k-mooney | lspci in the guest will only print the pci capablities not the extened pcie cabalities like sriov | |
| 14:56:15 | sean-k-mooney | in my previous experince | |
| 14:56:20 | sean-k-mooney | even when using q35 | |
| 14:56:28 | dmitriis | sean-k-mooney: I was about to ask about q35 :^) | |
| 14:56:30 | sean-k-mooney | if it does work let me know :) | |
| 14:56:37 | dmitriis | sean-k-mooney: there's another option | |
| 14:56:49 | dmitriis | BF2 on one host + ConnectX on another | |
| 14:57:16 | dmitriis | just use the remote_managed feature with ovs/ovn-controller running locally on the connectx node | |
| 14:57:47 | sean-k-mooney | ya i tought about that | |
| 14:58:00 | sean-k-mooney | jsut addign the serial to the chasis table for the hosts ovn | |
| 14:58:09 | dmitriis | yes | |
| 14:58:15 | sean-k-mooney | in prinicapal that would owrk without BF2 | |
| 14:58:43 | dmitriis | yes, technically, we just made Nova and Neutron realize that networking agents may run remotely but the trivial case is running locally | |
| 14:58:47 | sean-k-mooney | i did not really want to say that to our qe however :) | |
| 14:59:00 | dmitriis | yeah, fair enough :^) | |
| 14:59:25 | sean-k-mooney | but ya in principal we can test most of the integration that way for any nic that supprot vpd | |
| 15:00:01 | dmitriis | or even bind mount a file to the right sysfs location | |
| 15:00:03 | sean-k-mooney | i.e. if we can get the serial,mac and vf number we can test the end to end integration | |
| 15:00:20 | dmitriis | I got used to crafting VPD blobs since I needed to unit test the libvirt change | |
| 15:00:33 | dmitriis | basically, I crafted them byte-by-byte | |
| 15:00:51 | dmitriis | sean-k-mooney: ^ yes | |
| 15:01:02 | sean-k-mooney | ya thats doable | |
| 15:01:20 | sean-k-mooney | although i dont think it woudl be hard to have a python class that modeled it and just serialise it as a byte string | |
| 15:01:35 | sean-k-mooney | rather then do it by hand | |
| 15:01:47 | sean-k-mooney | unfortunetly i dont think we can use that to test in the upstream ci | |
| 15:02:03 | dmitriis | sean-k-mooney: yeah, it's not too hard. The only tricky part there is checksum calculation but it's a simple algorithm | |
| 15:02:24 | sean-k-mooney | crc32 or similar i assume | |
| 15:02:42 | sean-k-mooney | it would not be hard to look at the c code and ectra | |
| 15:02:47 | dmitriis | sean-k-mooney: just one's complement | |
| 15:02:56 | sean-k-mooney | that said im not sure libvirt cares about the crc | |
| 15:03:11 | sean-k-mooney | ack | |
| 15:03:30 | dmitriis | sean-k-mooney: IIRC I validate the checksum for the read-only portion there | |
| 15:03:49 | sean-k-mooney | oh in the nova code? | |
| 15:03:57 | sean-k-mooney | i dont think i recall seeing that | |
| 15:04:06 | dmitriis | sean-k-mooney: no-no, in the Libvirt code that parses VPD | |
| 15:04:12 | sean-k-mooney | ah | |
| 15:04:14 | sean-k-mooney | ok | |
| 15:04:29 | dmitriis | sean-k-mooney: there is a chance that something may be added to the mellanox CI. I know CX5 hw is there based on the logs but no BF2 (yet) | |
| 15:04:47 | dmitriis | maybe I'll have some info later this week about that | |
| 15:05:10 | dmitriis | there are automation challenges: i.e. we need to bring up devstack + also program the DPU | |
| 15:05:19 | dmitriis | but devices availability first I guess | |
| 15:05:26 | sean-k-mooney | https://github.com/SeanMooney/arbiterd/tree/master/arbiterd_tests/test_data/sys | |
| 15:05:26 | sean-k-mooney | i actully have been doing some experiment on the side lately and as part of my test infra i have a fake copy of part of /sys | |
| 15:05:58 | sean-k-mooney | the idea being that in my func test it would use the fake copy of the file system | |
| 15:06:23 | sean-k-mooney | its too bad we cant create fake pf devices using kernel modules for testing | |
| 15:06:50 | sean-k-mooney | i have looked into it in the past and we can create the fake netdevs for sriov | |
| 15:06:52 | dmitriis | sean-k-mooney: there's netdevsim but it doesn't create PCI devices | |
| 15:06:54 | sean-k-mooney | but not the pci toploty | |
| 15:07:00 | sean-k-mooney | yep | |
| 15:07:01 | sean-k-mooney | exactly | |
| 15:07:09 | dmitriis | can't pass them to a VM, yep | |
| 15:07:13 | sean-k-mooney | i tried to use that for sriov testing in the upstream ci in the past | |
| 15:07:34 | sean-k-mooney | i also was lookign at the mdev equivlent | |
| 15:07:46 | dmitriis | we utilize netdevsim for ovn-vif testing | |
| 15:07:47 | sean-k-mooney | same issue no pci endpoints | |
| 15:08:09 | dmitriis | and Frode is working on some code to extend it in the kernel but it's only usable for the ovn-vif testing part | |
| 15:08:22 | sean-k-mooney | ya that kind fo makes sense since it woudl not need the pci adress just the netdevs | |
| 15:09:00 | sean-k-mooney | if it ever gets extened to implement pci vfs/pfs | |
| 15:09:18 | sean-k-mooney | we would be able to use it for basic pci pasthorugh testing | |
| 15:09:28 | sean-k-mooney | we woudl not need the networking to actully work | |
| 15:09:35 | sean-k-mooney | just enough to pass it to qemu | |
| 15:09:50 | sean-k-mooney | that is a lot of work but would be super useful for us | |
| 15:09:51 | dmitriis | sean-k-mooney: I also looked at a possibility of using the rocker switch in QEMU but it's useful for switchdev testing but doesn't have SR-IOV emulation | |
| 15:10:10 | sean-k-mooney | ya | |
| 15:10:33 | dmitriis | sean-k-mooney: yes, also Libvirt's testing of VF-related stuff is quite minimal. They have the same problem with emulating VFs presumably | |
| 15:10:42 | sean-k-mooney | so at some point we likely will want to supprot vdpa with subfunction rather then vfs as the parent | |
| 15:10:59 | sean-k-mooney | the rocker switch might be useful to test that eventually | |
| 15:12:12 | dmitriis | yeah, SFs are something we also started looking into but there's more work to do at the ovn-vif side. Even for VFs we ran into a race at the BF2 side https://github.com/ovn-org/ovn-vif/pull/1 and SFs are even more dynamic | |
| 15:13:23 | dmitriis | TL;DR: SR-IOV is enabled at the hypervisor and then BF2 realizes that it needs to create representors and the DPU kernel starts creating udev events for them and rename them | |