| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 15:04:29 | dmitriis | sean-k-mooney: there is a chance that something may be added to the mellanox CI. I know CX5 hw is there based on the logs but no BF2 (yet) | |
| 15:04:47 | dmitriis | maybe I'll have some info later this week about that | |
| 15:05:10 | dmitriis | there are automation challenges: i.e. we need to bring up devstack + also program the DPU | |
| 15:05:19 | dmitriis | but devices availability first I guess | |
| 15:05:26 | sean-k-mooney | i actully have been doing some experiment on the side lately and as part of my test infra i have a fake copy of part of /sys | |
| 15:05:26 | sean-k-mooney | https://github.com/SeanMooney/arbiterd/tree/master/arbiterd_tests/test_data/sys | |
| 15:05:58 | sean-k-mooney | the idea being that in my func test it would use the fake copy of the file system | |
| 15:06:23 | sean-k-mooney | its too bad we cant create fake pf devices using kernel modules for testing | |
| 15:06:50 | sean-k-mooney | i have looked into it in the past and we can create the fake netdevs for sriov | |
| 15:06:52 | dmitriis | sean-k-mooney: there's netdevsim but it doesn't create PCI devices | |
| 15:06:54 | sean-k-mooney | but not the pci toploty | |
| 15:07:00 | sean-k-mooney | yep | |
| 15:07:01 | sean-k-mooney | exactly | |
| 15:07:09 | dmitriis | can't pass them to a VM, yep | |
| 15:07:13 | sean-k-mooney | i tried to use that for sriov testing in the upstream ci in the past | |
| 15:07:34 | sean-k-mooney | i also was lookign at the mdev equivlent | |
| 15:07:46 | dmitriis | we utilize netdevsim for ovn-vif testing | |
| 15:07:47 | sean-k-mooney | same issue no pci endpoints | |
| 15:08:09 | dmitriis | and Frode is working on some code to extend it in the kernel but it's only usable for the ovn-vif testing part | |
| 15:08:22 | sean-k-mooney | ya that kind fo makes sense since it woudl not need the pci adress just the netdevs | |
| 15:09:00 | sean-k-mooney | if it ever gets extened to implement pci vfs/pfs | |
| 15:09:18 | sean-k-mooney | we would be able to use it for basic pci pasthorugh testing | |
| 15:09:28 | sean-k-mooney | we woudl not need the networking to actully work | |
| 15:09:35 | sean-k-mooney | just enough to pass it to qemu | |
| 15:09:50 | sean-k-mooney | that is a lot of work but would be super useful for us | |
| 15:09:51 | dmitriis | sean-k-mooney: I also looked at a possibility of using the rocker switch in QEMU but it's useful for switchdev testing but doesn't have SR-IOV emulation | |
| 15:10:10 | sean-k-mooney | ya | |
| 15:10:33 | dmitriis | sean-k-mooney: yes, also Libvirt's testing of VF-related stuff is quite minimal. They have the same problem with emulating VFs presumably | |
| 15:10:42 | sean-k-mooney | so at some point we likely will want to supprot vdpa with subfunction rather then vfs as the parent | |
| 15:10:59 | sean-k-mooney | the rocker switch might be useful to test that eventually | |
| 15:12:12 | dmitriis | yeah, SFs are something we also started looking into but there's more work to do at the ovn-vif side. Even for VFs we ran into a race at the BF2 side https://github.com/ovn-org/ovn-vif/pull/1 and SFs are even more dynamic | |
| 15:13:23 | dmitriis | TL;DR: SR-IOV is enabled at the hypervisor and then BF2 realizes that it needs to create representors and the DPU kernel starts creating udev events for them and rename them | |
| 15:13:31 | sean-k-mooney | ack in a similar vain i was hoping to eventualy start using https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mtty.c and https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mdpy.c or | |
| 15:13:33 | sean-k-mooney | https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mbochs.c to test our generic mdev support | |
| 15:14:40 | dmitriis | mtty, interesting | |
| 15:14:57 | sean-k-mooney | i belive it just echos back what you send to it | |
| 15:15:05 | sean-k-mooney | but that woudl be super simple to test in tempest | |
| 15:15:15 | dmitriis | yes, quite useful | |
| 15:15:41 | sean-k-mooney | our current mdev code i belive assumes each mdev has a pci device as a parent | |
| 15:15:48 | sean-k-mooney | so we would need to relax that | |
| 15:15:58 | sean-k-mooney | to use the driver but if we did we could get full ci coverage for it | |
| 15:16:22 | sean-k-mooney | when people start using mdev for subfuction or non VF usecase we might need to do that anyway | |
| 15:16:32 | sean-k-mooney | at which point we can test it properly in ci | |
| 15:17:18 | dmitriis | I heard that SFs were introduced to support container use-cases because of scalability issues, less so for VMs. Doesn't mean VMs cannot use them but still. | |
| 15:17:52 | dmitriis | scalability ~ small number of ports | |
| 15:18:03 | sean-k-mooney | depend on who you ask. i know intel were interestin in there version for contaienr but also saw it as a way to adress the vm case in terms of number of ports | |
| 15:18:37 | sean-k-mooney | dmitriis: really the use case is the same allow more then max_vf ports | |
| 15:19:15 | sean-k-mooney | that said i think recent connectx/bf2 cards have started to get into the 1000+ vf range | |
| 15:19:24 | dmitriis | sean-k-mooney: yeah, there are some parts of the SR-IOV spec that can be used to overcome the 256 limit | |
| 15:19:33 | dmitriis | some neat tricks with PCI address allocation | |
| 15:20:04 | sean-k-mooney | yes you multiple buses to card to work around that | |
| 15:20:09 | dmitriis | yep | |
| 15:21:15 | sean-k-mooney | contaienr and vm cloud system really prefer to pertend that the port limit is infinity | |
| 15:21:40 | sean-k-mooney | like k8s and opnestack really dont treat ports as a finite reqouce outside fo sriov | |
| 15:22:27 | dmitriis | sean-k-mooney: there's another problem: accounting and quotas on certain hw limits. There's a finite number of flows that can be programmed and not a lot of data on hw limits per VF/SF | |
| 15:22:37 | dmitriis | likewise, no APIs to query that AFAIK | |
| 15:22:38 | sean-k-mooney | in reality most customer proably wont hit that limit but they are woried fi they layer things like openshift running on top of openstack they might | |
| 15:23:25 | dmitriis | sean-k-mooney: yes, plus with layers on top of OpenStack people start running overlays on top of overlays | |
| 15:23:32 | sean-k-mooney | ya there are both limits on the number of tables and the table row count and then seperat lmits on but the hardwar flows offload | |
| 15:23:34 | dmitriis | and doing that using guest CPU | |
| 15:24:30 | sean-k-mooney | ok i better get back to reviewing your changes before my next meeting :) | |
| 15:24:43 | dmitriis | sean-k-mooney: right, I'll get to re-submitting too :^) | |
| 15:49:24 | sean-k-mooney | dmitriis: is there anyting in the port beyond the vnic_type that tells nova that we need a vf | |
| 15:49:56 | sean-k-mooney | dmitriis: gibi be might not be able to reuse vnic_type smartnic | |
| 15:50:11 | opendevreview | Tobias Urdin proposed openstack/nova master: Cleanup old resize instances dir before resize https://review.opendev.org/c/openstack/nova/+/827865 | |
| 15:50:34 | sean-k-mooney | if we assume that all port that have vnic_type smartnic require a vf that would break ironics use of it yes? | |
| 15:50:55 | opendevreview | Balazs Gibizer proposed openstack/placement master: Fix perfload jobs after consumer_types https://review.opendev.org/c/openstack/placement/+/828167 | |
| 15:51:22 | gibi | sean-k-mooney: I don't know how ironic uses the smartnic vnic_type | |
| 15:51:38 | gibi | melwitt: I think the perfload job is easy to fix https://review.opendev.org/c/openstack/placement/+/828167 | |
| 15:51:49 | dmitriis | sean-k-mooney: (thinking) | |
| 15:51:55 | sean-k-mooney | https://specs.openstack.org/openstack/ironic-specs/specs/12.1/support-smart-nic.html | |
| 15:54:47 | sean-k-mooney | gibi: they bascially wanted to supprot ovs running on the smartnic also | |
| 15:54:54 | sean-k-mooney | but using ml2/ovs | |
| 15:55:12 | sean-k-mooney | with the neutron agetn deployed on the smartnic | |
| 15:55:14 | dmitriis | sean-k-mooney: they have a special config option for the neutron-openvswitch-agent as well | |
| 15:55:24 | sean-k-mooney | yes | |
| 15:55:55 | sean-k-mooney | dmitriis: the possible problem i am seing is when we go to scedule the vm we have not bound the port | |
| 15:56:08 | sean-k-mooney | since we have not selected a host yet | |
| 15:56:23 | sean-k-mooney | so i dont think we will be able to tell the difference between the ironic usage and the new usage | |
| 15:56:33 | sean-k-mooney | and we wont know if we shoudl request a VF or not | |
| 15:57:01 | sean-k-mooney | so we might need to use a differnt off-path vnic type | |
| 15:57:04 | sean-k-mooney | instead | |
| 15:57:29 | dmitriis | sean-k-mooney: in Nova we decide if we want to request a remote_managed port or not | |
| 15:57:36 | dmitriis | and add that to a InstancePCIRequest | |
| 15:57:48 | sean-k-mooney | dmitriis: yes based on the vnic_type right | |
| 15:57:51 | dmitriis | yes | |
| 15:58:01 | sean-k-mooney | right so if we only look at that we need a new vnic type | |
| 15:58:27 | dmitriis | but that's in Nova, trying to figure out how that affects ironic | |
| 15:58:27 | sean-k-mooney | since that decision need to be made before we schdule and therefor cannot depend on the host or driver | |
| 15:58:43 | sean-k-mooney | booting ironic service via nova api | |
| 15:58:47 | sean-k-mooney | *servers | |
| 15:59:05 | sean-k-mooney | in both cases the vnic_type would be the same | |
| 15:59:20 | sean-k-mooney | the falvor woudl be differnt but we would not know its an ironic flavor | |
| 16:00:21 | sean-k-mooney | dmitriis: so i think we need to quickly add a new vnic in neutron-lib and update the nova and neutron code to use that | |
| 16:01:13 | opendevreview | Takashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action https://review.opendev.org/c/openstack/nova/+/828168 | |
| 16:01:39 | opendevreview | Takashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action https://review.opendev.org/c/openstack/nova/+/828168 | |
| 16:03:00 | dmitriis | sean-k-mooney: we could, trying to think if there's anything we've missed that could allow us to avoid that. We originally wanted to add a new VNIC type but then decided to reuse VNIC_TYPE_SMARTNIC after some reviews. | |
| 16:03:36 | sean-k-mooney | yep i proably suggested the reuse :) but i think its proably required | |
| 16:03:46 | sean-k-mooney | *the new vnic | |
| 16:03:59 | sean-k-mooney | i dont think there is anything else on the port we can use to differenciate | |