| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-08-28 | |||
| 12:52:53 | sean-k-mooney | can you express what exactly is required to match | |
| 12:54:13 | sean-k-mooney | are you useing the vf number to identigfy the dpdk port or somehting like that | |
| 12:54:33 | sean-k-mooney | i.e. stdvio10 maps to dpdk10 | |
| 12:54:39 | dvo-plv | in our case to create correct socket, we need socket name base ( stdvio ) and vf number according to the formula "domain bus slot" * 8 + "func number" | |
| 12:55:09 | dvo-plv | I added comment here https://review.opendev.org/c/openstack/os-vif/+/859574/8/vif_plug_ovs/ovs.py#352 | |
| 12:55:57 | sean-k-mooney | that does not expalin why tha tis need and the algorhtim you implemted does nto match that | |
| 12:57:11 | sean-k-mooney | what the code does is take 0000:2b:01.2 then reduce it to 01.2 then split that to 01 and 2 | |
| 12:57:16 | dvo-plv | we reserved first 4 pci for pf needs | |
| 12:57:27 | sean-k-mooney | and do 8*1+2 | |
| 12:58:36 | sean-k-mooney | so your are discarding the domain and bus numbers then lookig only at the slot and function | |
| 12:59:41 | sean-k-mooney | so 0000:2b:01.2 and 0000:3b:01.2 will compute the same socket | |
| 12:59:42 | dvo-plv | yes, we use this approach to work with single nic support | |
| 13:00:07 | sean-k-mooney | right which i i dont think is suffent to move forward with this | |
| 13:00:46 | dvo-plv | multi nic support is in the development, so I would like to provide method, which can calculate correct vf number and socket for all vendors by passing formula to the binding profile | |
| 13:01:15 | sean-k-mooney | that feels like a hack | |
| 13:01:41 | sean-k-mooney | i also dont like the idea of effectlivly embedding a dsl to expres the formula which is then interpreted at run time | |
| 13:04:29 | sean-k-mooney | dvo-plv: lookign tat the ovs and dpdk docs | |
| 13:04:54 | sean-k-mooney | the way we add port representors is by pasing the PCI address of the PF + the represnetor vf offest | |
| 13:04:56 | sean-k-mooney | https://docs.openvswitch.org/en/latest/topics/dpdk/phy/?highlight=dpdk#representors | |
| 13:05:04 | sean-k-mooney | i.e. ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \ | |
| 13:05:06 | sean-k-mooney | options:dpdk-devargs=0000:08:00.0,representor=vf3 | |
| 13:06:02 | dvo-plv | yes, we do the same, but representor identifier is not fixed, so it depends on vendor dpdk driver | |
| 13:06:36 | sean-k-mooney | well thats kind of a problem | |
| 13:06:52 | sean-k-mooney | since the code you want to modify is mento be vendor neutral | |
| 13:07:16 | sean-k-mooney | meaning your are not allowewd to make vendor speciifc chagnes to it | |
| 13:07:42 | dvo-plv | this why, I would like to provide config via binding profile | |
| 13:09:15 | sean-k-mooney | no | |
| 13:09:45 | sean-k-mooney | thats not really an option | |
| 13:10:10 | sean-k-mooney | at least not without alot of other changes | |
| 13:10:25 | sean-k-mooney | so here https://review.opendev.org/c/openstack/os-vif/+/859574/8/vif_plug_ovs/tests/unit/test_plugin.py#440 | |
| 13:10:36 | sean-k-mooney | you are creatign the port doing effectivly | |
| 13:10:55 | sean-k-mooney | ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \ | |
| 13:10:56 | sean-k-mooney | options:dpdk-devargs=0000:08:00.0,representor=vf3 | |
| 13:14:58 | sean-k-mooney | when you say the representor identifyier are you referign to vf3 | |
| 13:15:05 | sean-k-mooney | or the vhost-user socket file name | |
| 13:15:17 | sean-k-mooney | dvo-plv: or both? | |
| 13:15:35 | dvo-plv | both | |
| 13:15:53 | sean-k-mooney | so the convention in ovs is the socket name should match the port name | |
| 13:16:23 | sean-k-mooney | at least for interfaces of type vhost-user and vhost-user-client | |
| 13:16:38 | sean-k-mooney | or whatever the ovs constats for those are | |
| 13:17:09 | dvo-plv | I believe it can be correct for dpdk port type too | |
| 13:19:04 | dvo-plv | but as far as there is no convention regarding name formulation, we have to hardcore socket name on the dpdk initialization step | |
| 13:19:59 | dvo-plv | we create vf and socket at the same time and then pass it to the some application to use | |
| 13:20:13 | sean-k-mooney | that not how thign will work in nova/openstack | |
| 13:20:30 | sean-k-mooney | the VFs woudl have to be staticaly allocated effectivly at boot time | |
| 13:20:58 | sean-k-mooney | what driver is the PF bound too? | |
| 13:21:32 | sean-k-mooney | its bound to a kernel dirver not DPDK yes? | |
| 13:22:08 | dvo-plv | we use default vfio | |
| 13:22:40 | sean-k-mooney | for the VFs or the PF | |
| 13:23:59 | sean-k-mooney | vfio-pci is ok for the vf but not the PF | |
| 13:28:27 | sean-k-mooney | based on https://doc.dpdk.org/guides-22.11/prog_guide/switch_representation.html#vf-representors i think the PF are bound to a kernel dirver and the VFs are boud to vfio-pci then managed by the dpdk userspace driver | |
| 13:31:44 | sean-k-mooney | althogh im not sure that is correct if i look at https://doc.dpdk.org/guides-22.11/prog_guide/switch_representation.html#basic-sr-iov | |
| 13:32:07 | sean-k-mooney | "A DPDK application running on the hypervisor owns the PF device, which is arbitrarily assigned port index 3" | |
| 13:35:27 | sean-k-mooney | honestly i think upstream ovs need to be modifed to take the vhost-user socket path as a pramter when we do the port add | |
| 13:35:42 | dvo-plv | At the beginning we about that here https://review.opendev.org/c/openstack/nova-specs/+/859290/4..18//COMMIT_MSG#b9 | |
| 13:35:55 | dvo-plv | so this is why we firstly created our mech driver | |
| 13:36:51 | dvo-plv | so, maybe correct way be better to get back our mech driver ? | |
| 13:37:33 | sean-k-mooney | if it needs to be vendor specific it woudl need to be out of tree both on the ml2 side and os-vif side | |
| 13:38:11 | sean-k-mooney | the reason im askign about the PF dirver by the way is to ensure libvirt can enumebrate the VFs | |
| 13:38:40 | sean-k-mooney | it cant do that as far as i am aware if the PF is added to dpdk | |
| 13:40:19 | dvo-plv | we use vfio-pci for pf and vf fucntions | |
| 13:40:37 | dvo-plv | we can observer our interfaces only via dpdk driver | |
| 13:40:45 | dvo-plv | dpdk ntnic-pmd | |
| 13:40:52 | sean-k-mooney | then this wont work with libvirt | |
| 13:41:27 | sean-k-mooney | since we will have no nodedevs for the VFs | |
| 13:41:49 | sean-k-mooney | so we will have not entrieds in the pci_deivce table | |
| 13:42:10 | sean-k-mooney | unless your driver is implemeting a SYSfs interface for them? | |
| 13:45:20 | dvo-plv | yes ,https://paste.opendev.org/show/b8UDrLC9e0iKrEJR6z6j/ | |
| 13:45:47 | sean-k-mooney | if you have that interface then we can use sysfs to lookup the vf number | |
| 13:46:18 | sean-k-mooney | which os-vif already has code for. | |
| 13:48:57 | sean-k-mooney | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L360-L378 | |
| 13:49:49 | dvo-plv | yes, I remember that I tried to use this algo, but it does not correct for us as far as I remmember for example we have this pci 000:2b:01.1/, so according to the fomrula vf number shoulde be 9, but sysfs calc it like virtfn5 | |
| 13:50:17 | sean-k-mooney | that sounds like a bug in your dpdk drvier then | |
| 13:52:52 | sean-k-mooney | lookign at that output it is the 6th VF so 5 is correct if 0 indexed | |
| 13:56:48 | sean-k-mooney | i think the best way to make this portabel and upstreamable would be to add the socket name as a dpdkarg wehn adding the port and use the data for sysfs | |
| 13:57:29 | sean-k-mooney | taht would require 2 changes to the dpdk driver but it would mena we would not have to calulate the socket path in a special way | |
| 13:57:56 | sean-k-mooney | so we can keep the neutron ml2 driver unmodifed at least in terms of the vhost-user path generation | |
| 14:05:54 | dvo-plv | this is not a bug, because we reserved first 4 pci for physical interfaces | |
| 14:10:14 | dvo-plv | So what about to get back to the separate ml2 plugin ? like Agilio ? | |
| 14:15:09 | dvo-plv | sorry, but i did not get your option with dpdk args | |
| 14:15:54 | dvo-plv | socket creates by dpdk pmd driver on the init moment | |
| 14:16:13 | dvo-plv | ...-a 0000:2b:00.0,representor=[4-6],portqueues=[4:1,5:1,6:1] -a 0000:2b:00.4 -a 0000:2b:00.5 -a 0000:2b:00.6" | |
| 14:16:35 | dvo-plv | we pass next other config to the dpdk driver | |
| 14:16:52 | dvo-plv | after that it create sockets with names stdvio+vfnumber | |
| 14:18:46 | sean-k-mooney | shoudnt the socket be created by qemu with dpdk as the clinet | |
| 14:19:15 | sean-k-mooney | when dpdk is the server then if the vswitch is restated it breaks network connectivty for all vms | |
| 14:19:39 | sean-k-mooney | this is why we moved to qemu server dpdk client mode many years ago | |
| 14:22:18 | dvo-plv | this is not an issue for us, after ovs restart and socket recreating, connectivity get back | |
| 14:22:58 | sean-k-mooney | qemu will not reconnect to the vhos-user socket if its recreated by dpdk | |
| 14:23:04 | sean-k-mooney | the socket FD will change | |
| 14:23:27 | sean-k-mooney | there is a way to make qemu do that i belive htat was added signifcinaly later but we dont configre that | |
| 14:24:18 | sean-k-mooney | we do not supprot genreating " <reconnect enabled='yes' timeout='10'/>" | |
| 14:24:52 | sean-k-mooney | https://libvirt.org/formatdomain.html#vhost-user-interface | |
| 14:26:30 | sean-k-mooney | so if qemu is runing in client mode and you restart ovs to do a package upgrae it will break network connectivnty for all guests until you hard reboot them | |
| 14:26:38 | sean-k-mooney | have you tested this in an openstack envionment? | |
| 14:26:46 | dvo-plv | this is qemu command, -chardev socket,id=char0,path=/usr/local/var/run/stdvio6,server we set the server on the qemu side | |
| 14:27:16 | sean-k-mooney | ok then qemu is runing in server mode whch means qemu is creating the socket not dpdk | |
| 14:27:29 | sean-k-mooney | dpdk is runing in client mode and connecting to the socekt create by qemu | |
| 14:27:44 | sean-k-mooney | that is the way we recommend you use vhost user | |
| 14:28:39 | sean-k-mooney | what im suggetign is we add the path to the dpdk args when os-vif creates the interface in ovs | |