| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-08-28 | |||
| 13:15:53 | sean-k-mooney | so the convention in ovs is the socket name should match the port name | |
| 13:16:23 | sean-k-mooney | at least for interfaces of type vhost-user and vhost-user-client | |
| 13:16:38 | sean-k-mooney | or whatever the ovs constats for those are | |
| 13:17:09 | dvo-plv | I believe it can be correct for dpdk port type too | |
| 13:19:04 | dvo-plv | but as far as there is no convention regarding name formulation, we have to hardcore socket name on the dpdk initialization step | |
| 13:19:59 | dvo-plv | we create vf and socket at the same time and then pass it to the some application to use | |
| 13:20:13 | sean-k-mooney | that not how thign will work in nova/openstack | |
| 13:20:30 | sean-k-mooney | the VFs woudl have to be staticaly allocated effectivly at boot time | |
| 13:20:58 | sean-k-mooney | what driver is the PF bound too? | |
| 13:21:32 | sean-k-mooney | its bound to a kernel dirver not DPDK yes? | |
| 13:22:08 | dvo-plv | we use default vfio | |
| 13:22:40 | sean-k-mooney | for the VFs or the PF | |
| 13:23:59 | sean-k-mooney | vfio-pci is ok for the vf but not the PF | |
| 13:28:27 | sean-k-mooney | based on https://doc.dpdk.org/guides-22.11/prog_guide/switch_representation.html#vf-representors i think the PF are bound to a kernel dirver and the VFs are boud to vfio-pci then managed by the dpdk userspace driver | |
| 13:31:44 | sean-k-mooney | althogh im not sure that is correct if i look at https://doc.dpdk.org/guides-22.11/prog_guide/switch_representation.html#basic-sr-iov | |
| 13:32:07 | sean-k-mooney | "A DPDK application running on the hypervisor owns the PF device, which is arbitrarily assigned port index 3" | |
| 13:35:27 | sean-k-mooney | honestly i think upstream ovs need to be modifed to take the vhost-user socket path as a pramter when we do the port add | |
| 13:35:42 | dvo-plv | At the beginning we about that here https://review.opendev.org/c/openstack/nova-specs/+/859290/4..18//COMMIT_MSG#b9 | |
| 13:35:55 | dvo-plv | so this is why we firstly created our mech driver | |
| 13:36:51 | dvo-plv | so, maybe correct way be better to get back our mech driver ? | |
| 13:37:33 | sean-k-mooney | if it needs to be vendor specific it woudl need to be out of tree both on the ml2 side and os-vif side | |
| 13:38:11 | sean-k-mooney | the reason im askign about the PF dirver by the way is to ensure libvirt can enumebrate the VFs | |
| 13:38:40 | sean-k-mooney | it cant do that as far as i am aware if the PF is added to dpdk | |
| 13:40:19 | dvo-plv | we use vfio-pci for pf and vf fucntions | |
| 13:40:37 | dvo-plv | we can observer our interfaces only via dpdk driver | |
| 13:40:45 | dvo-plv | dpdk ntnic-pmd | |
| 13:40:52 | sean-k-mooney | then this wont work with libvirt | |
| 13:41:27 | sean-k-mooney | since we will have no nodedevs for the VFs | |
| 13:41:49 | sean-k-mooney | so we will have not entrieds in the pci_deivce table | |
| 13:42:10 | sean-k-mooney | unless your driver is implemeting a SYSfs interface for them? | |
| 13:45:20 | dvo-plv | yes ,https://paste.opendev.org/show/b8UDrLC9e0iKrEJR6z6j/ | |
| 13:45:47 | sean-k-mooney | if you have that interface then we can use sysfs to lookup the vf number | |
| 13:46:18 | sean-k-mooney | which os-vif already has code for. | |
| 13:48:57 | sean-k-mooney | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L360-L378 | |
| 13:49:49 | dvo-plv | yes, I remember that I tried to use this algo, but it does not correct for us as far as I remmember for example we have this pci 000:2b:01.1/, so according to the fomrula vf number shoulde be 9, but sysfs calc it like virtfn5 | |
| 13:50:17 | sean-k-mooney | that sounds like a bug in your dpdk drvier then | |
| 13:52:52 | sean-k-mooney | lookign at that output it is the 6th VF so 5 is correct if 0 indexed | |
| 13:56:48 | sean-k-mooney | i think the best way to make this portabel and upstreamable would be to add the socket name as a dpdkarg wehn adding the port and use the data for sysfs | |
| 13:57:29 | sean-k-mooney | taht would require 2 changes to the dpdk driver but it would mena we would not have to calulate the socket path in a special way | |
| 13:57:56 | sean-k-mooney | so we can keep the neutron ml2 driver unmodifed at least in terms of the vhost-user path generation | |
| 14:05:54 | dvo-plv | this is not a bug, because we reserved first 4 pci for physical interfaces | |
| 14:10:14 | dvo-plv | So what about to get back to the separate ml2 plugin ? like Agilio ? | |
| 14:15:09 | dvo-plv | sorry, but i did not get your option with dpdk args | |
| 14:15:54 | dvo-plv | socket creates by dpdk pmd driver on the init moment | |
| 14:16:13 | dvo-plv | ...-a 0000:2b:00.0,representor=[4-6],portqueues=[4:1,5:1,6:1] -a 0000:2b:00.4 -a 0000:2b:00.5 -a 0000:2b:00.6" | |
| 14:16:35 | dvo-plv | we pass next other config to the dpdk driver | |
| 14:16:52 | dvo-plv | after that it create sockets with names stdvio+vfnumber | |
| 14:18:46 | sean-k-mooney | shoudnt the socket be created by qemu with dpdk as the clinet | |
| 14:19:15 | sean-k-mooney | when dpdk is the server then if the vswitch is restated it breaks network connectivty for all vms | |
| 14:19:39 | sean-k-mooney | this is why we moved to qemu server dpdk client mode many years ago | |
| 14:22:18 | dvo-plv | this is not an issue for us, after ovs restart and socket recreating, connectivity get back | |
| 14:22:58 | sean-k-mooney | qemu will not reconnect to the vhos-user socket if its recreated by dpdk | |
| 14:23:04 | sean-k-mooney | the socket FD will change | |
| 14:23:27 | sean-k-mooney | there is a way to make qemu do that i belive htat was added signifcinaly later but we dont configre that | |
| 14:24:18 | sean-k-mooney | we do not supprot genreating " <reconnect enabled='yes' timeout='10'/>" | |
| 14:24:52 | sean-k-mooney | https://libvirt.org/formatdomain.html#vhost-user-interface | |
| 14:26:30 | sean-k-mooney | so if qemu is runing in client mode and you restart ovs to do a package upgrae it will break network connectivnty for all guests until you hard reboot them | |
| 14:26:38 | sean-k-mooney | have you tested this in an openstack envionment? | |
| 14:26:46 | dvo-plv | this is qemu command, -chardev socket,id=char0,path=/usr/local/var/run/stdvio6,server we set the server on the qemu side | |
| 14:27:16 | sean-k-mooney | ok then qemu is runing in server mode whch means qemu is creating the socket not dpdk | |
| 14:27:29 | sean-k-mooney | dpdk is runing in client mode and connecting to the socekt create by qemu | |
| 14:27:44 | sean-k-mooney | that is the way we recommend you use vhost user | |
| 14:28:39 | sean-k-mooney | what im suggetign is we add the path to the dpdk args when os-vif creates the interface in ovs | |
| 14:28:52 | sean-k-mooney | and the driver shoudl use that instead of trying to calulate it | |
| 14:29:36 | dvo-plv | yes, we formulate socket name by pci address according to the formula, because we reserve 4 first pcis for physical ports | |
| 14:30:24 | sean-k-mooney | yep but your not actully passing the vf number to dpdk in teh representor arg | |
| 14:30:40 | sean-k-mooney | what you are pasing is the intger offset to add to the pf adress | |
| 14:31:58 | sean-k-mooney | that is why your expect virtfn5 -> ../0000:2b:01.1/ to be 9 | |
| 14:32:40 | dvo-plv | yes, and this offset can not give us ability to work with sysfs | |
| 14:32:51 | dvo-plv | + socket name like stdvio + vf_num | |
| 14:33:05 | sean-k-mooney | that not the vf_num | |
| 14:33:13 | sean-k-mooney | its the pci endpoint number | |
| 14:33:21 | sean-k-mooney | its a diffent thing | |
| 14:33:29 | dvo-plv | sure | |
| 14:34:04 | sean-k-mooney | really the dpdk driver should jsut ascp representor=vf5 + the pf address | |
| 14:34:08 | sean-k-mooney | and compute that internaly | |
| 14:34:30 | dvo-plv | we can get this offset from here in some way root@server23:~# cat /sys/bus/pci/devices/0000\:2b\:00.0/sriov_offset | |
| 14:34:31 | dvo-plv | 4 | |
| 14:36:17 | sean-k-mooney | if we can read that form sysfs | |
| 14:36:39 | sean-k-mooney | then we can drop the algother in os-vif and use the exisitng fucntion and add the ofset | |
| 14:36:55 | sean-k-mooney | that still does not really help for the ml2 driver | |
| 14:37:42 | sean-k-mooney | do you have a document that describes the step by step process of adding the representor netdevs to ovs-dpdk manually | |
| 14:38:30 | dvo-plv | one moment | |
| 14:39:56 | dvo-plv | https://docs.napatech.com/r/Getting-Started-with-Napatech-Link-VirtualizationTM-Software/Create-the-OVS-Provider-Bridge-and-Start-2-VMs | |
| 14:41:48 | sean-k-mooney | that is incompelte. it does not have the port add comands for the represtor prots | |
| 14:42:45 | dvo-plv | item 4 Add the dpdkvp0 virtual port to the br-int bridge: | |
| 15:12:31 | dvo-plv | maybe we can create logic like with vhostuser_socket_dir, if vhostuser_socket_name is set, get this name from config + pci from sysfs? | |
| 16:59:28 | noonedeadpunk | hey folks. I wanna double-check one thing with you, that I'm not missing anything. So for volume to be attached to a VM with scsi bus - volume *must* be created from the image? | |
| 16:59:58 | noonedeadpunk | As I've found a spec to allow volume define that as well, but it was abandoned | |
| 17:11:56 | sean-k-mooney | the volume no | |
| 17:12:07 | sean-k-mooney | but the vm root disk must have hw_disk_bus=scsi | |
| 17:12:43 | sean-k-mooney | we only look at the metadata on the root disk be that a local disk or cinder volume root disk | |
| 17:12:57 | sean-k-mooney | and all other cinder volume must use the same disk bus | |
| 17:13:18 | sean-k-mooney | noonedeadpunk: so right now we do not support attching block devices with diffent busses | |
| 17:13:37 | sean-k-mooney | there is a way to ocationlaly make that work in a speicific edgecase but its not supported upstream | |
| 17:14:14 | noonedeadpunk | aha, ok, I see then | |
| 17:14:25 | sean-k-mooney | noonedeadpunk: we discussed in the last ptg addign a way to supprot per volume disk bueses | |
| 17:14:33 | sean-k-mooney | but no one actully worked on it | |
| 17:14:38 | noonedeadpunk | So if you found yourself in situation when 25 volumes are not enough - you should start from scratch kinda? | |
| 17:15:09 | sean-k-mooney | well. it depends | |