| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 14:26:41 | sean-k-mooney | as a general pattern | |
| 14:28:28 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: workarounds: Add libvirt_disable_apic https://review.opendev.org/c/openstack/nova/+/805628 | |
| 14:28:45 | sean-k-mooney | dmitriis: i say perfer as im open to being conviced otherwise but in general i woudl prefer not to reate retiving info as an error. its true it might mean that device cannot be remote managed but in that case the operator has incorrectly tagged it. on the neutron side the ml2/driver can check the profile and fail the binding if the info is not present tha it needs | |
| 14:32:36 | dmitriis | sean-k-mooney: I think it's the generic SR-IOV handling code in the kernel that creates the sysfs entry | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L144-L147 | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L295 | |
| 14:32:36 | dmitriis | In this case, the case where the info won't be retrievable will likely never happen so I have no issue in making it optional. Just trying to reason about what to put into a comment there. | |
| 14:34:04 | sean-k-mooney | just say somthign like. this should be created by default on all modeern kernels but we make it optional to cater for expotic hardware or older kernel where this may not be ture | |
| 14:34:40 | dmitriis | sean-k-mooney: ack, will do | |
| 14:35:55 | sean-k-mooney | dmitriis looking at 4.14 for example im not sure it does the same https://github.com/torvalds/linux/blob/v4.14/drivers/pci/iov.c#L165-L176 | |
| 14:35:59 | sean-k-mooney | well | |
| 14:36:14 | sean-k-mooney | it has the physfn symlink | |
| 14:36:25 | sean-k-mooney | i guess it has the virtfn symlink too | |
| 14:36:46 | sean-k-mooney | so i guess it would not break on centos 8 | |
| 14:37:17 | sean-k-mooney | dmitriis: i know mellonox/nvida changed where the representor netdevs were create for example in a relitivly recent kernel | |
| 14:37:46 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b37de19c58c877f5174d76d0a4ba5ab519f464e8#diff-087288ddacb7516a4762ef38c4361754c2d6bb07a52952b06aa6f607dc3811fe | |
| 14:38:02 | sean-k-mooney | 5.8 https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1 | |
| 14:39:05 | sean-k-mooney | dmitriis: im trying to avoid failure in nova when thing like that ar changed | |
| 14:39:54 | dmitriis | sean-k-mooney: yeah, probably worth one VM failing to boot than the whole daemon dying because of the exception | |
| 14:39:56 | dmitriis | totally agree | |
| 14:40:50 | sean-k-mooney | yep plus as i said in the neutron ml2 driver you can fail the port binding an log an explit error message if the required info is not present | |
| 14:41:18 | dmitriis | sean-k-mooney: also very true | |
| 14:41:25 | sean-k-mooney | you could proably also add a check in nova at a differnt point if you wanted before that but i woudl just log the error form neutron personlaly | |
| 14:42:18 | dmitriis | sean-k-mooney: I'll log a warning in Nova in this case and simply return no data in the profile and let Neutron fail to bind | |
| 14:42:38 | sean-k-mooney | well you shoudl not return an empty profile | |
| 14:42:52 | sean-k-mooney | this code will be used for other VFs that may not be remote managed | |
| 14:43:04 | sean-k-mooney | and tehy do not need the vf number | |
| 14:43:11 | sean-k-mooney | we pass the vf pci adress | |
| 14:43:20 | sean-k-mooney | that is all they need in the general case | |
| 14:43:38 | dmitriis | rephrasing: 1. if PF MAC cannot be retrieved, do not include vf_num or card_serial_number either | |
| 14:43:56 | dmitriis | 2. if vf_num cannot be retrieved, do not include pf_mac and card_serial_number | |
| 14:44:24 | sean-k-mooney | ah well ya perhaps | |
| 14:44:39 | sean-k-mooney | i was thinking just include all the info you can always but i guess that works too | |
| 14:45:35 | dmitriis | ok, just need to fix the tests a little and I'll re-upload | |
| 14:45:45 | sean-k-mooney | ack | |
| 14:46:57 | sean-k-mooney | dmitriis: by the way have you tested this feature with a second ovs on the compute host | |
| 14:47:27 | sean-k-mooney | we had some internal question regarding can we have both offpath port and on path port on the same host | |
| 14:48:09 | sean-k-mooney | the reason for that was to use vm management port with vnic_type=normal and datapalne prots with vnic_type=smartnic | |
| 14:48:20 | dmitriis | sean-k-mooney: i.e. having a DPU and a regular SR-IOV card | |
| 14:48:37 | sean-k-mooney | well dpu and another nic that can be used with ovs or linux bridge | |
| 14:48:41 | dmitriis | right | |
| 14:49:03 | dmitriis | I haven't tested that explicitly but just trying to think what would that entail | |
| 14:49:07 | sean-k-mooney | basically it was raised that by using the dpu you are limiting the numer or port that can be created to the number of vfs | |
| 14:49:22 | sean-k-mooney | dmitriis: i think it should just work honestly | |
| 14:49:32 | sean-k-mooney | i was just wondering if you had considerd it | |
| 14:49:50 | dmitriis | sean-k-mooney: yes, the limitation on the number of ports have definitely come up in our conversations | |
| 14:50:11 | sean-k-mooney | dmitriis: all it really requires is the ml2 dirver changes to not require you to designate a host as only dpu enabled | |
| 14:50:18 | dmitriis | and nothing immediately jumps into mind regarding OVS on the host being a problem | |
| 14:50:23 | sean-k-mooney | e.g. make the binding desiion per port based on vnic_type | |
| 14:50:54 | sean-k-mooney | ack that is what my assement was too | |
| 14:51:04 | dmitriis | sean-k-mooney: yeah and IIRC it's just based on the vnic_type, we just added handling for vnic_type smartnic to the OVN mechanism driver | |
| 14:51:14 | dmitriis | so it should just bind a normal port regularly | |
| 14:51:20 | sean-k-mooney | yep | |
| 14:51:23 | sean-k-mooney | cool | |
| 14:51:44 | sean-k-mooney | we are still trying to assess what woudl be requried to eventually support this in our product | |
| 14:52:04 | dmitriis | sean-k-mooney: yeah, plus hardware is hard to come by | |
| 14:52:12 | sean-k-mooney | that too | |
| 14:52:45 | dmitriis | I think this feature will be useful for FPGA-based SmartNICs as well. I've seen the ones with a separate CPU but with an FPGA instead of an ASIC | |
| 14:53:24 | dmitriis | considering the DPDK dataplane usage at the SmartNIC's CPU side for that with rte_flow for offload | |
| 14:53:26 | sean-k-mooney | i do potentially have access to a BF2 that i coudl use but only 1 so i obviouly cant test multi node and in general an all in one deployment while useful is not fully reflective of how it would be deployed in a datacenter | |
| 14:53:34 | dmitriis | the Nova and Neutron bits would work the same though | |
| 14:53:46 | sean-k-mooney | dmitriis: yes in princial it could be | |
| 14:53:54 | dmitriis | sean-k-mooney: I have an idea on how to test multi-node with just one BF2 | |
| 14:54:15 | sean-k-mooney | the only fpga based smartnic i have use used armstong os with only 2 arm cores and i think 4GB of ram | |
| 14:54:44 | sean-k-mooney | dmitriis: run two copies of nova-compute with difernt host values in teh conf on the same host | |
| 14:54:53 | sean-k-mooney | and use differnt ports on the BF2? | |
| 14:54:53 | dmitriis | sean-k-mooney: 2 VMs with nested virt a the hypervisor, plus 2 VMs at the DPU side. Each VM uses 1 PF. VPD can be faked by bind mounting a file in the right sysfs location | |
| 14:55:08 | dmitriis | BF2's ARM CPU is virt-capable btw xD | |
| 14:55:43 | sean-k-mooney | dmitriis: that proably wotn work becaue wehn the PF is passed to the qemu instacle the pcie capablityies are not in the pci config space in teh guest | |
| 14:56:10 | sean-k-mooney | lspci in the guest will only print the pci capablities not the extened pcie cabalities like sriov | |
| 14:56:15 | sean-k-mooney | in my previous experince | |
| 14:56:20 | sean-k-mooney | even when using q35 | |
| 14:56:28 | dmitriis | sean-k-mooney: I was about to ask about q35 :^) | |
| 14:56:30 | sean-k-mooney | if it does work let me know :) | |
| 14:56:37 | dmitriis | sean-k-mooney: there's another option | |
| 14:56:49 | dmitriis | BF2 on one host + ConnectX on another | |
| 14:57:16 | dmitriis | just use the remote_managed feature with ovs/ovn-controller running locally on the connectx node | |
| 14:57:47 | sean-k-mooney | ya i tought about that | |
| 14:58:00 | sean-k-mooney | jsut addign the serial to the chasis table for the hosts ovn | |
| 14:58:09 | dmitriis | yes | |
| 14:58:15 | sean-k-mooney | in prinicapal that would owrk without BF2 | |
| 14:58:43 | dmitriis | yes, technically, we just made Nova and Neutron realize that networking agents may run remotely but the trivial case is running locally | |
| 14:58:47 | sean-k-mooney | i did not really want to say that to our qe however :) | |
| 14:59:00 | dmitriis | yeah, fair enough :^) | |
| 14:59:25 | sean-k-mooney | but ya in principal we can test most of the integration that way for any nic that supprot vpd | |
| 15:00:01 | dmitriis | or even bind mount a file to the right sysfs location | |
| 15:00:03 | sean-k-mooney | i.e. if we can get the serial,mac and vf number we can test the end to end integration | |
| 15:00:20 | dmitriis | I got used to crafting VPD blobs since I needed to unit test the libvirt change | |
| 15:00:33 | dmitriis | basically, I crafted them byte-by-byte | |
| 15:00:51 | dmitriis | sean-k-mooney: ^ yes | |
| 15:01:02 | sean-k-mooney | ya thats doable | |
| 15:01:20 | sean-k-mooney | although i dont think it woudl be hard to have a python class that modeled it and just serialise it as a byte string | |
| 15:01:35 | sean-k-mooney | rather then do it by hand | |
| 15:01:47 | sean-k-mooney | unfortunetly i dont think we can use that to test in the upstream ci | |
| 15:02:03 | dmitriis | sean-k-mooney: yeah, it's not too hard. The only tricky part there is checksum calculation but it's a simple algorithm | |
| 15:02:24 | sean-k-mooney | crc32 or similar i assume | |
| 15:02:42 | sean-k-mooney | it would not be hard to look at the c code and ectra | |
| 15:02:47 | dmitriis | sean-k-mooney: just one's complement | |
| 15:02:56 | sean-k-mooney | that said im not sure libvirt cares about the crc | |
| 15:03:11 | sean-k-mooney | ack | |