| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-08-28 | |||
| 10:35:26 | bauzas | this morning, paperwork + email scrubbing :( | |
| 10:35:53 | sean-k-mooney | i feel like configing that to morning is probably ambious | |
| 10:39:06 | bauzas | tbh, I need way more coffee before looking at the gate :) | |
| 10:41:17 | sean-k-mooney | its relitivly fine at the moment | |
| 10:41:44 | sean-k-mooney | bar the normaly volume detach issues which i think will be a ptg topic | |
| 11:34:39 | gibi | bauzas: o/ welcome back. we did not pass the bug triage baton around during your PTO, I was lazy to find a candidate during the meeting. Otherwise the meetings were held and producitve as much as summer period allows. | |
| 11:34:55 | bauzas | ++ thanks for the offer | |
| 12:42:48 | dvo-plv | sean-k-mooney, What do you think regarding multi nic support in the our patch by adding additional configuration via config file or even better using binding profile | |
| 12:43:11 | sean-k-mooney | hard no on config file | |
| 12:43:32 | sean-k-mooney | or user specifying data in the binding profile | |
| 12:44:00 | sean-k-mooney | well it depends on where the config file would be used, whats in it and how that is confiugred | |
| 12:44:12 | sean-k-mooney | but i dont think that likely to work. | |
| 12:44:18 | dvo-plv | get configuration here to generate correct socket name https://review.opendev.org/c/openstack/neutron/+/869510/13/neutron/plugins/ml2/drivers/openvswitch/mech_driver/mech_openvswitch.py#221 | |
| 12:44:30 | dvo-plv | and here to create correct vf number https://review.opendev.org/c/openstack/os-vif/+/859574/8/vif_plug_ovs/ovs.py#349 | |
| 12:44:31 | sean-k-mooney | note that binding_profile is expclivily for nova to pass data to the network backend | |
| 12:45:29 | sean-k-mooney | as in you woudl have someihng in the neurton l2 agents config file | |
| 12:45:44 | sean-k-mooney | which gets pass as part fo agent['configurations'] | |
| 12:45:58 | sean-k-mooney | that you need use to calulate the vf number | |
| 12:47:45 | sean-k-mooney | i dont really think that is the correct approch honestly | |
| 12:49:00 | dvo-plv | but we need to correlate somehow formula for socket and vf number calculating | |
| 12:49:59 | sean-k-mooney | how i think this shoudl work is likely nova shoudl gather the info required adn store it in the binidng_profile and then neutron shoudl use that to calualte the path | |
| 12:50:25 | dvo-plv | we use simple number for vf and socket name stdvio+vf number. Mellanox uses something like pf0vf0 formulation | |
| 12:50:49 | dvo-plv | sorry, what info required, where what entity we should parse to get this info | |
| 12:51:35 | sean-k-mooney | the vf number is only meainifn if you know the parent PF pci device | |
| 12:52:53 | sean-k-mooney | can you express what exactly is required to match | |
| 12:54:13 | sean-k-mooney | are you useing the vf number to identigfy the dpdk port or somehting like that | |
| 12:54:33 | sean-k-mooney | i.e. stdvio10 maps to dpdk10 | |
| 12:54:39 | dvo-plv | in our case to create correct socket, we need socket name base ( stdvio ) and vf number according to the formula "domain bus slot" * 8 + "func number" | |
| 12:55:09 | dvo-plv | I added comment here https://review.opendev.org/c/openstack/os-vif/+/859574/8/vif_plug_ovs/ovs.py#352 | |
| 12:55:57 | sean-k-mooney | that does not expalin why tha tis need and the algorhtim you implemted does nto match that | |
| 12:57:11 | sean-k-mooney | what the code does is take 0000:2b:01.2 then reduce it to 01.2 then split that to 01 and 2 | |
| 12:57:16 | dvo-plv | we reserved first 4 pci for pf needs | |
| 12:57:27 | sean-k-mooney | and do 8*1+2 | |
| 12:58:36 | sean-k-mooney | so your are discarding the domain and bus numbers then lookig only at the slot and function | |
| 12:59:41 | sean-k-mooney | so 0000:2b:01.2 and 0000:3b:01.2 will compute the same socket | |
| 12:59:42 | dvo-plv | yes, we use this approach to work with single nic support | |
| 13:00:07 | sean-k-mooney | right which i i dont think is suffent to move forward with this | |
| 13:00:46 | dvo-plv | multi nic support is in the development, so I would like to provide method, which can calculate correct vf number and socket for all vendors by passing formula to the binding profile | |
| 13:01:15 | sean-k-mooney | that feels like a hack | |
| 13:01:41 | sean-k-mooney | i also dont like the idea of effectlivly embedding a dsl to expres the formula which is then interpreted at run time | |
| 13:04:29 | sean-k-mooney | dvo-plv: lookign tat the ovs and dpdk docs | |
| 13:04:54 | sean-k-mooney | the way we add port representors is by pasing the PCI address of the PF + the represnetor vf offest | |
| 13:04:56 | sean-k-mooney | https://docs.openvswitch.org/en/latest/topics/dpdk/phy/?highlight=dpdk#representors | |
| 13:05:04 | sean-k-mooney | i.e. ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \ | |
| 13:05:06 | sean-k-mooney | options:dpdk-devargs=0000:08:00.0,representor=vf3 | |
| 13:06:02 | dvo-plv | yes, we do the same, but representor identifier is not fixed, so it depends on vendor dpdk driver | |
| 13:06:36 | sean-k-mooney | well thats kind of a problem | |
| 13:06:52 | sean-k-mooney | since the code you want to modify is mento be vendor neutral | |
| 13:07:16 | sean-k-mooney | meaning your are not allowewd to make vendor speciifc chagnes to it | |
| 13:07:42 | dvo-plv | this why, I would like to provide config via binding profile | |
| 13:09:15 | sean-k-mooney | no | |
| 13:09:45 | sean-k-mooney | thats not really an option | |
| 13:10:10 | sean-k-mooney | at least not without alot of other changes | |
| 13:10:25 | sean-k-mooney | so here https://review.opendev.org/c/openstack/os-vif/+/859574/8/vif_plug_ovs/tests/unit/test_plugin.py#440 | |
| 13:10:36 | sean-k-mooney | you are creatign the port doing effectivly | |
| 13:10:55 | sean-k-mooney | ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk \ | |
| 13:10:56 | sean-k-mooney | options:dpdk-devargs=0000:08:00.0,representor=vf3 | |
| 13:14:58 | sean-k-mooney | when you say the representor identifyier are you referign to vf3 | |
| 13:15:05 | sean-k-mooney | or the vhost-user socket file name | |
| 13:15:17 | sean-k-mooney | dvo-plv: or both? | |
| 13:15:35 | dvo-plv | both | |
| 13:15:53 | sean-k-mooney | so the convention in ovs is the socket name should match the port name | |
| 13:16:23 | sean-k-mooney | at least for interfaces of type vhost-user and vhost-user-client | |
| 13:16:38 | sean-k-mooney | or whatever the ovs constats for those are | |
| 13:17:09 | dvo-plv | I believe it can be correct for dpdk port type too | |
| 13:19:04 | dvo-plv | but as far as there is no convention regarding name formulation, we have to hardcore socket name on the dpdk initialization step | |
| 13:19:59 | dvo-plv | we create vf and socket at the same time and then pass it to the some application to use | |
| 13:20:13 | sean-k-mooney | that not how thign will work in nova/openstack | |
| 13:20:30 | sean-k-mooney | the VFs woudl have to be staticaly allocated effectivly at boot time | |
| 13:20:58 | sean-k-mooney | what driver is the PF bound too? | |
| 13:21:32 | sean-k-mooney | its bound to a kernel dirver not DPDK yes? | |
| 13:22:08 | dvo-plv | we use default vfio | |
| 13:22:40 | sean-k-mooney | for the VFs or the PF | |
| 13:23:59 | sean-k-mooney | vfio-pci is ok for the vf but not the PF | |
| 13:28:27 | sean-k-mooney | based on https://doc.dpdk.org/guides-22.11/prog_guide/switch_representation.html#vf-representors i think the PF are bound to a kernel dirver and the VFs are boud to vfio-pci then managed by the dpdk userspace driver | |
| 13:31:44 | sean-k-mooney | althogh im not sure that is correct if i look at https://doc.dpdk.org/guides-22.11/prog_guide/switch_representation.html#basic-sr-iov | |
| 13:32:07 | sean-k-mooney | "A DPDK application running on the hypervisor owns the PF device, which is arbitrarily assigned port index 3" | |
| 13:35:27 | sean-k-mooney | honestly i think upstream ovs need to be modifed to take the vhost-user socket path as a pramter when we do the port add | |
| 13:35:42 | dvo-plv | At the beginning we about that here https://review.opendev.org/c/openstack/nova-specs/+/859290/4..18//COMMIT_MSG#b9 | |
| 13:35:55 | dvo-plv | so this is why we firstly created our mech driver | |
| 13:36:51 | dvo-plv | so, maybe correct way be better to get back our mech driver ? | |
| 13:37:33 | sean-k-mooney | if it needs to be vendor specific it woudl need to be out of tree both on the ml2 side and os-vif side | |
| 13:38:11 | sean-k-mooney | the reason im askign about the PF dirver by the way is to ensure libvirt can enumebrate the VFs | |
| 13:38:40 | sean-k-mooney | it cant do that as far as i am aware if the PF is added to dpdk | |
| 13:40:19 | dvo-plv | we use vfio-pci for pf and vf fucntions | |
| 13:40:37 | dvo-plv | we can observer our interfaces only via dpdk driver | |
| 13:40:45 | dvo-plv | dpdk ntnic-pmd | |
| 13:40:52 | sean-k-mooney | then this wont work with libvirt | |
| 13:41:27 | sean-k-mooney | since we will have no nodedevs for the VFs | |
| 13:41:49 | sean-k-mooney | so we will have not entrieds in the pci_deivce table | |
| 13:42:10 | sean-k-mooney | unless your driver is implemeting a SYSfs interface for them? | |
| 13:45:20 | dvo-plv | yes ,https://paste.opendev.org/show/b8UDrLC9e0iKrEJR6z6j/ | |
| 13:45:47 | sean-k-mooney | if you have that interface then we can use sysfs to lookup the vf number | |
| 13:46:18 | sean-k-mooney | which os-vif already has code for. | |
| 13:48:57 | sean-k-mooney | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L360-L378 | |
| 13:49:49 | dvo-plv | yes, I remember that I tried to use this algo, but it does not correct for us as far as I remmember for example we have this pci 000:2b:01.1/, so according to the fomrula vf number shoulde be 9, but sysfs calc it like virtfn5 | |
| 13:50:17 | sean-k-mooney | that sounds like a bug in your dpdk drvier then | |
| 13:52:52 | sean-k-mooney | lookign at that output it is the 6th VF so 5 is correct if 0 indexed | |
| 13:56:48 | sean-k-mooney | i think the best way to make this portabel and upstreamable would be to add the socket name as a dpdkarg wehn adding the port and use the data for sysfs | |
| 13:57:29 | sean-k-mooney | taht would require 2 changes to the dpdk driver but it would mena we would not have to calulate the socket path in a special way | |