| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 14:39:05 | sean-k-mooney | dmitriis: im trying to avoid failure in nova when thing like that ar changed | |
| 14:39:54 | dmitriis | sean-k-mooney: yeah, probably worth one VM failing to boot than the whole daemon dying because of the exception | |
| 14:39:56 | dmitriis | totally agree | |
| 14:40:50 | sean-k-mooney | yep plus as i said in the neutron ml2 driver you can fail the port binding an log an explit error message if the required info is not present | |
| 14:41:18 | dmitriis | sean-k-mooney: also very true | |
| 14:41:25 | sean-k-mooney | you could proably also add a check in nova at a differnt point if you wanted before that but i woudl just log the error form neutron personlaly | |
| 14:42:18 | dmitriis | sean-k-mooney: I'll log a warning in Nova in this case and simply return no data in the profile and let Neutron fail to bind | |
| 14:42:38 | sean-k-mooney | well you shoudl not return an empty profile | |
| 14:42:52 | sean-k-mooney | this code will be used for other VFs that may not be remote managed | |
| 14:43:04 | sean-k-mooney | and tehy do not need the vf number | |
| 14:43:11 | sean-k-mooney | we pass the vf pci adress | |
| 14:43:20 | sean-k-mooney | that is all they need in the general case | |
| 14:43:38 | dmitriis | rephrasing: 1. if PF MAC cannot be retrieved, do not include vf_num or card_serial_number either | |
| 14:43:56 | dmitriis | 2. if vf_num cannot be retrieved, do not include pf_mac and card_serial_number | |
| 14:44:24 | sean-k-mooney | ah well ya perhaps | |
| 14:44:39 | sean-k-mooney | i was thinking just include all the info you can always but i guess that works too | |
| 14:45:35 | dmitriis | ok, just need to fix the tests a little and I'll re-upload | |
| 14:45:45 | sean-k-mooney | ack | |
| 14:46:57 | sean-k-mooney | dmitriis: by the way have you tested this feature with a second ovs on the compute host | |
| 14:47:27 | sean-k-mooney | we had some internal question regarding can we have both offpath port and on path port on the same host | |
| 14:48:09 | sean-k-mooney | the reason for that was to use vm management port with vnic_type=normal and datapalne prots with vnic_type=smartnic | |
| 14:48:20 | dmitriis | sean-k-mooney: i.e. having a DPU and a regular SR-IOV card | |
| 14:48:37 | sean-k-mooney | well dpu and another nic that can be used with ovs or linux bridge | |
| 14:48:41 | dmitriis | right | |
| 14:49:03 | dmitriis | I haven't tested that explicitly but just trying to think what would that entail | |
| 14:49:07 | sean-k-mooney | basically it was raised that by using the dpu you are limiting the numer or port that can be created to the number of vfs | |
| 14:49:22 | sean-k-mooney | dmitriis: i think it should just work honestly | |
| 14:49:32 | sean-k-mooney | i was just wondering if you had considerd it | |
| 14:49:50 | dmitriis | sean-k-mooney: yes, the limitation on the number of ports have definitely come up in our conversations | |
| 14:50:11 | sean-k-mooney | dmitriis: all it really requires is the ml2 dirver changes to not require you to designate a host as only dpu enabled | |
| 14:50:18 | dmitriis | and nothing immediately jumps into mind regarding OVS on the host being a problem | |
| 14:50:23 | sean-k-mooney | e.g. make the binding desiion per port based on vnic_type | |
| 14:50:54 | sean-k-mooney | ack that is what my assement was too | |
| 14:51:04 | dmitriis | sean-k-mooney: yeah and IIRC it's just based on the vnic_type, we just added handling for vnic_type smartnic to the OVN mechanism driver | |
| 14:51:14 | dmitriis | so it should just bind a normal port regularly | |
| 14:51:20 | sean-k-mooney | yep | |
| 14:51:23 | sean-k-mooney | cool | |
| 14:51:44 | sean-k-mooney | we are still trying to assess what woudl be requried to eventually support this in our product | |
| 14:52:04 | dmitriis | sean-k-mooney: yeah, plus hardware is hard to come by | |
| 14:52:12 | sean-k-mooney | that too | |
| 14:52:45 | dmitriis | I think this feature will be useful for FPGA-based SmartNICs as well. I've seen the ones with a separate CPU but with an FPGA instead of an ASIC | |
| 14:53:24 | dmitriis | considering the DPDK dataplane usage at the SmartNIC's CPU side for that with rte_flow for offload | |
| 14:53:26 | sean-k-mooney | i do potentially have access to a BF2 that i coudl use but only 1 so i obviouly cant test multi node and in general an all in one deployment while useful is not fully reflective of how it would be deployed in a datacenter | |
| 14:53:34 | dmitriis | the Nova and Neutron bits would work the same though | |
| 14:53:46 | sean-k-mooney | dmitriis: yes in princial it could be | |
| 14:53:54 | dmitriis | sean-k-mooney: I have an idea on how to test multi-node with just one BF2 | |
| 14:54:15 | sean-k-mooney | the only fpga based smartnic i have use used armstong os with only 2 arm cores and i think 4GB of ram | |
| 14:54:44 | sean-k-mooney | dmitriis: run two copies of nova-compute with difernt host values in teh conf on the same host | |
| 14:54:53 | dmitriis | sean-k-mooney: 2 VMs with nested virt a the hypervisor, plus 2 VMs at the DPU side. Each VM uses 1 PF. VPD can be faked by bind mounting a file in the right sysfs location | |
| 14:54:53 | sean-k-mooney | and use differnt ports on the BF2? | |
| 14:55:08 | dmitriis | BF2's ARM CPU is virt-capable btw xD | |
| 14:55:43 | sean-k-mooney | dmitriis: that proably wotn work becaue wehn the PF is passed to the qemu instacle the pcie capablityies are not in the pci config space in teh guest | |
| 14:56:10 | sean-k-mooney | lspci in the guest will only print the pci capablities not the extened pcie cabalities like sriov | |
| 14:56:15 | sean-k-mooney | in my previous experince | |
| 14:56:20 | sean-k-mooney | even when using q35 | |
| 14:56:28 | dmitriis | sean-k-mooney: I was about to ask about q35 :^) | |
| 14:56:30 | sean-k-mooney | if it does work let me know :) | |
| 14:56:37 | dmitriis | sean-k-mooney: there's another option | |
| 14:56:49 | dmitriis | BF2 on one host + ConnectX on another | |
| 14:57:16 | dmitriis | just use the remote_managed feature with ovs/ovn-controller running locally on the connectx node | |
| 14:57:47 | sean-k-mooney | ya i tought about that | |
| 14:58:00 | sean-k-mooney | jsut addign the serial to the chasis table for the hosts ovn | |
| 14:58:09 | dmitriis | yes | |
| 14:58:15 | sean-k-mooney | in prinicapal that would owrk without BF2 | |
| 14:58:43 | dmitriis | yes, technically, we just made Nova and Neutron realize that networking agents may run remotely but the trivial case is running locally | |
| 14:58:47 | sean-k-mooney | i did not really want to say that to our qe however :) | |
| 14:59:00 | dmitriis | yeah, fair enough :^) | |
| 14:59:25 | sean-k-mooney | but ya in principal we can test most of the integration that way for any nic that supprot vpd | |
| 15:00:01 | dmitriis | or even bind mount a file to the right sysfs location | |
| 15:00:03 | sean-k-mooney | i.e. if we can get the serial,mac and vf number we can test the end to end integration | |
| 15:00:20 | dmitriis | I got used to crafting VPD blobs since I needed to unit test the libvirt change | |
| 15:00:33 | dmitriis | basically, I crafted them byte-by-byte | |
| 15:00:51 | dmitriis | sean-k-mooney: ^ yes | |
| 15:01:02 | sean-k-mooney | ya thats doable | |
| 15:01:20 | sean-k-mooney | although i dont think it woudl be hard to have a python class that modeled it and just serialise it as a byte string | |
| 15:01:35 | sean-k-mooney | rather then do it by hand | |
| 15:01:47 | sean-k-mooney | unfortunetly i dont think we can use that to test in the upstream ci | |
| 15:02:03 | dmitriis | sean-k-mooney: yeah, it's not too hard. The only tricky part there is checksum calculation but it's a simple algorithm | |
| 15:02:24 | sean-k-mooney | crc32 or similar i assume | |
| 15:02:42 | sean-k-mooney | it would not be hard to look at the c code and ectra | |
| 15:02:47 | dmitriis | sean-k-mooney: just one's complement | |
| 15:02:56 | sean-k-mooney | that said im not sure libvirt cares about the crc | |
| 15:03:11 | sean-k-mooney | ack | |
| 15:03:30 | dmitriis | sean-k-mooney: IIRC I validate the checksum for the read-only portion there | |
| 15:03:49 | sean-k-mooney | oh in the nova code? | |
| 15:03:57 | sean-k-mooney | i dont think i recall seeing that | |
| 15:04:06 | dmitriis | sean-k-mooney: no-no, in the Libvirt code that parses VPD | |
| 15:04:12 | sean-k-mooney | ah | |
| 15:04:14 | sean-k-mooney | ok | |
| 15:04:29 | dmitriis | sean-k-mooney: there is a chance that something may be added to the mellanox CI. I know CX5 hw is there based on the logs but no BF2 (yet) | |
| 15:04:47 | dmitriis | maybe I'll have some info later this week about that | |
| 15:05:10 | dmitriis | there are automation challenges: i.e. we need to bring up devstack + also program the DPU | |
| 15:05:19 | dmitriis | but devices availability first I guess | |
| 15:05:26 | sean-k-mooney | https://github.com/SeanMooney/arbiterd/tree/master/arbiterd_tests/test_data/sys | |
| 15:05:26 | sean-k-mooney | i actully have been doing some experiment on the side lately and as part of my test infra i have a fake copy of part of /sys | |
| 15:05:58 | sean-k-mooney | the idea being that in my func test it would use the fake copy of the file system | |
| 15:06:23 | sean-k-mooney | its too bad we cant create fake pf devices using kernel modules for testing | |
| 15:06:50 | sean-k-mooney | i have looked into it in the past and we can create the fake netdevs for sriov | |
| 15:06:52 | dmitriis | sean-k-mooney: there's netdevsim but it doesn't create PCI devices | |
| 15:06:54 | sean-k-mooney | but not the pci toploty | |