| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 14:56:30 | sean-k-mooney | if it does work let me know :) | |
| 14:56:37 | dmitriis | sean-k-mooney: there's another option | |
| 14:56:49 | dmitriis | BF2 on one host + ConnectX on another | |
| 14:57:16 | dmitriis | just use the remote_managed feature with ovs/ovn-controller running locally on the connectx node | |
| 14:57:47 | sean-k-mooney | ya i tought about that | |
| 14:58:00 | sean-k-mooney | jsut addign the serial to the chasis table for the hosts ovn | |
| 14:58:09 | dmitriis | yes | |
| 14:58:15 | sean-k-mooney | in prinicapal that would owrk without BF2 | |
| 14:58:43 | dmitriis | yes, technically, we just made Nova and Neutron realize that networking agents may run remotely but the trivial case is running locally | |
| 14:58:47 | sean-k-mooney | i did not really want to say that to our qe however :) | |
| 14:59:00 | dmitriis | yeah, fair enough :^) | |
| 14:59:25 | sean-k-mooney | but ya in principal we can test most of the integration that way for any nic that supprot vpd | |
| 15:00:01 | dmitriis | or even bind mount a file to the right sysfs location | |
| 15:00:03 | sean-k-mooney | i.e. if we can get the serial,mac and vf number we can test the end to end integration | |
| 15:00:20 | dmitriis | I got used to crafting VPD blobs since I needed to unit test the libvirt change | |
| 15:00:33 | dmitriis | basically, I crafted them byte-by-byte | |
| 15:00:51 | dmitriis | sean-k-mooney: ^ yes | |
| 15:01:02 | sean-k-mooney | ya thats doable | |
| 15:01:20 | sean-k-mooney | although i dont think it woudl be hard to have a python class that modeled it and just serialise it as a byte string | |
| 15:01:35 | sean-k-mooney | rather then do it by hand | |
| 15:01:47 | sean-k-mooney | unfortunetly i dont think we can use that to test in the upstream ci | |
| 15:02:03 | dmitriis | sean-k-mooney: yeah, it's not too hard. The only tricky part there is checksum calculation but it's a simple algorithm | |
| 15:02:24 | sean-k-mooney | crc32 or similar i assume | |
| 15:02:42 | sean-k-mooney | it would not be hard to look at the c code and ectra | |
| 15:02:47 | dmitriis | sean-k-mooney: just one's complement | |
| 15:02:56 | sean-k-mooney | that said im not sure libvirt cares about the crc | |
| 15:03:11 | sean-k-mooney | ack | |
| 15:03:30 | dmitriis | sean-k-mooney: IIRC I validate the checksum for the read-only portion there | |
| 15:03:49 | sean-k-mooney | oh in the nova code? | |
| 15:03:57 | sean-k-mooney | i dont think i recall seeing that | |
| 15:04:06 | dmitriis | sean-k-mooney: no-no, in the Libvirt code that parses VPD | |
| 15:04:12 | sean-k-mooney | ah | |
| 15:04:14 | sean-k-mooney | ok | |
| 15:04:29 | dmitriis | sean-k-mooney: there is a chance that something may be added to the mellanox CI. I know CX5 hw is there based on the logs but no BF2 (yet) | |
| 15:04:47 | dmitriis | maybe I'll have some info later this week about that | |
| 15:05:10 | dmitriis | there are automation challenges: i.e. we need to bring up devstack + also program the DPU | |
| 15:05:19 | dmitriis | but devices availability first I guess | |
| 15:05:26 | sean-k-mooney | https://github.com/SeanMooney/arbiterd/tree/master/arbiterd_tests/test_data/sys | |
| 15:05:26 | sean-k-mooney | i actully have been doing some experiment on the side lately and as part of my test infra i have a fake copy of part of /sys | |
| 15:05:58 | sean-k-mooney | the idea being that in my func test it would use the fake copy of the file system | |
| 15:06:23 | sean-k-mooney | its too bad we cant create fake pf devices using kernel modules for testing | |
| 15:06:50 | sean-k-mooney | i have looked into it in the past and we can create the fake netdevs for sriov | |
| 15:06:52 | dmitriis | sean-k-mooney: there's netdevsim but it doesn't create PCI devices | |
| 15:06:54 | sean-k-mooney | but not the pci toploty | |
| 15:07:00 | sean-k-mooney | yep | |
| 15:07:01 | sean-k-mooney | exactly | |
| 15:07:09 | dmitriis | can't pass them to a VM, yep | |
| 15:07:13 | sean-k-mooney | i tried to use that for sriov testing in the upstream ci in the past | |
| 15:07:34 | sean-k-mooney | i also was lookign at the mdev equivlent | |
| 15:07:46 | dmitriis | we utilize netdevsim for ovn-vif testing | |
| 15:07:47 | sean-k-mooney | same issue no pci endpoints | |
| 15:08:09 | dmitriis | and Frode is working on some code to extend it in the kernel but it's only usable for the ovn-vif testing part | |
| 15:08:22 | sean-k-mooney | ya that kind fo makes sense since it woudl not need the pci adress just the netdevs | |
| 15:09:00 | sean-k-mooney | if it ever gets extened to implement pci vfs/pfs | |
| 15:09:18 | sean-k-mooney | we would be able to use it for basic pci pasthorugh testing | |
| 15:09:28 | sean-k-mooney | we woudl not need the networking to actully work | |
| 15:09:35 | sean-k-mooney | just enough to pass it to qemu | |
| 15:09:50 | sean-k-mooney | that is a lot of work but would be super useful for us | |
| 15:09:51 | dmitriis | sean-k-mooney: I also looked at a possibility of using the rocker switch in QEMU but it's useful for switchdev testing but doesn't have SR-IOV emulation | |
| 15:10:10 | sean-k-mooney | ya | |
| 15:10:33 | dmitriis | sean-k-mooney: yes, also Libvirt's testing of VF-related stuff is quite minimal. They have the same problem with emulating VFs presumably | |
| 15:10:42 | sean-k-mooney | so at some point we likely will want to supprot vdpa with subfunction rather then vfs as the parent | |
| 15:10:59 | sean-k-mooney | the rocker switch might be useful to test that eventually | |
| 15:12:12 | dmitriis | yeah, SFs are something we also started looking into but there's more work to do at the ovn-vif side. Even for VFs we ran into a race at the BF2 side https://github.com/ovn-org/ovn-vif/pull/1 and SFs are even more dynamic | |
| 15:13:23 | dmitriis | TL;DR: SR-IOV is enabled at the hypervisor and then BF2 realizes that it needs to create representors and the DPU kernel starts creating udev events for them and rename them | |
| 15:13:31 | sean-k-mooney | ack in a similar vain i was hoping to eventualy start using https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mtty.c and https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mdpy.c or | |
| 15:13:33 | sean-k-mooney | https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mbochs.c to test our generic mdev support | |
| 15:14:40 | dmitriis | mtty, interesting | |
| 15:14:57 | sean-k-mooney | i belive it just echos back what you send to it | |
| 15:15:05 | sean-k-mooney | but that woudl be super simple to test in tempest | |
| 15:15:15 | dmitriis | yes, quite useful | |
| 15:15:41 | sean-k-mooney | our current mdev code i belive assumes each mdev has a pci device as a parent | |
| 15:15:48 | sean-k-mooney | so we would need to relax that | |
| 15:15:58 | sean-k-mooney | to use the driver but if we did we could get full ci coverage for it | |
| 15:16:22 | sean-k-mooney | when people start using mdev for subfuction or non VF usecase we might need to do that anyway | |
| 15:16:32 | sean-k-mooney | at which point we can test it properly in ci | |
| 15:17:18 | dmitriis | I heard that SFs were introduced to support container use-cases because of scalability issues, less so for VMs. Doesn't mean VMs cannot use them but still. | |
| 15:17:52 | dmitriis | scalability ~ small number of ports | |
| 15:18:03 | sean-k-mooney | depend on who you ask. i know intel were interestin in there version for contaienr but also saw it as a way to adress the vm case in terms of number of ports | |
| 15:18:37 | sean-k-mooney | dmitriis: really the use case is the same allow more then max_vf ports | |
| 15:19:15 | sean-k-mooney | that said i think recent connectx/bf2 cards have started to get into the 1000+ vf range | |
| 15:19:24 | dmitriis | sean-k-mooney: yeah, there are some parts of the SR-IOV spec that can be used to overcome the 256 limit | |
| 15:19:33 | dmitriis | some neat tricks with PCI address allocation | |
| 15:20:04 | sean-k-mooney | yes you multiple buses to card to work around that | |
| 15:20:09 | dmitriis | yep | |
| 15:21:15 | sean-k-mooney | contaienr and vm cloud system really prefer to pertend that the port limit is infinity | |
| 15:21:40 | sean-k-mooney | like k8s and opnestack really dont treat ports as a finite reqouce outside fo sriov | |
| 15:22:27 | dmitriis | sean-k-mooney: there's another problem: accounting and quotas on certain hw limits. There's a finite number of flows that can be programmed and not a lot of data on hw limits per VF/SF | |
| 15:22:37 | dmitriis | likewise, no APIs to query that AFAIK | |
| 15:22:38 | sean-k-mooney | in reality most customer proably wont hit that limit but they are woried fi they layer things like openshift running on top of openstack they might | |
| 15:23:25 | dmitriis | sean-k-mooney: yes, plus with layers on top of OpenStack people start running overlays on top of overlays | |
| 15:23:32 | sean-k-mooney | ya there are both limits on the number of tables and the table row count and then seperat lmits on but the hardwar flows offload | |
| 15:23:34 | dmitriis | and doing that using guest CPU | |
| 15:24:30 | sean-k-mooney | ok i better get back to reviewing your changes before my next meeting :) | |
| 15:24:43 | dmitriis | sean-k-mooney: right, I'll get to re-submitting too :^) | |
| 15:49:24 | sean-k-mooney | dmitriis: is there anyting in the port beyond the vnic_type that tells nova that we need a vf | |
| 15:49:56 | sean-k-mooney | dmitriis: gibi be might not be able to reuse vnic_type smartnic | |
| 15:50:11 | opendevreview | Tobias Urdin proposed openstack/nova master: Cleanup old resize instances dir before resize https://review.opendev.org/c/openstack/nova/+/827865 | |
| 15:50:34 | sean-k-mooney | if we assume that all port that have vnic_type smartnic require a vf that would break ironics use of it yes? | |
| 15:50:55 | opendevreview | Balazs Gibizer proposed openstack/placement master: Fix perfload jobs after consumer_types https://review.opendev.org/c/openstack/placement/+/828167 | |