| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-22 | |||
| 12:09:06 | Dmitrii-Sh | sean-k-mooney: there's a possibility that there will be 2 SmartNICs in one physical host | |
| 12:09:06 | Dmitrii-Sh | sean-k-mooney: there's a possibility that there will be 2 SmartNICs in one physical host | |
| 12:09:18 | sean-k-mooney | so assuming the agent/ovn would be runnign on the smart nic you could use the servers hostname for those values | |
| 12:09:19 | sean-k-mooney | so assuming the agent/ovn would be runnign on the smart nic you could use the servers hostname for those values | |
| 12:09:20 | Dmitrii-Sh | which creates an issue with this - I considered it as an initial hack | |
| 12:09:20 | Dmitrii-Sh | which creates an issue with this - I considered it as an initial hack | |
| 12:09:34 | sean-k-mooney | ah right | |
| 12:09:42 | sean-k-mooney | ya that does make things harder | |
| 12:09:42 | sean-k-mooney | ya that does make things harder | |
| 12:09:52 | Dmitrii-Sh | sean-k-mooney: hence the dances with PCI VPD serial numbers | |
| 12:09:52 | Dmitrii-Sh | sean-k-mooney: hence the dances with PCI VPD serial numbers | |
| 12:10:08 | Dmitrii-Sh | as we don't have another way of tracking which functions belong where on both sides | |
| 12:10:08 | Dmitrii-Sh | as we don't have another way of tracking which functions belong where on both sides | |
| 12:10:14 | sean-k-mooney | still i think we coudl encode some info in ovn or the agent config such that the ml2 driver could maintain the mapping | |
| 12:10:14 | sean-k-mooney | still i think we coudl encode some info in ovn or the agent config such that the ml2 driver could maintain the mapping | |
| 12:10:38 | sean-k-mooney | it just wont be as simple | |
| 12:10:38 | sean-k-mooney | it just wont be as simple | |
| 12:10:54 | sean-k-mooney | i see one other issue potentaillyu | |
| 12:10:54 | sean-k-mooney | i see one other issue potentaillyu | |
| 12:11:17 | sean-k-mooney | the pci address will it always be the same form the host vs smartnic perspecitve | |
| 12:11:17 | sean-k-mooney | the pci address will it always be the same form the host vs smartnic perspecitve | |
| 12:12:40 | Dmitrii-Sh | Didn't quite catch that. The SmartNIC host sees a different PCIe topology from the hypervisor host | |
| 12:12:40 | Dmitrii-Sh | Didn't quite catch that. The SmartNIC host sees a different PCIe topology from the hypervisor host | |
| 12:12:47 | Dmitrii-Sh | different addresses too | |
| 12:12:47 | Dmitrii-Sh | different addresses too | |
| 12:12:58 | Dmitrii-Sh | basically each host does its own PCIe topology enumeration | |
| 12:12:58 | Dmitrii-Sh | basically each host does its own PCIe topology enumeration | |
| 12:13:34 | Dmitrii-Sh | it's kind of like with blades and non-transparent bridging or managed PCIe switches that hide certain parts of the overall topology | |
| 12:13:34 | Dmitrii-Sh | it's kind of like with blades and non-transparent bridging or managed PCIe switches that hide certain parts of the overall topology | |
| 12:14:17 | sean-k-mooney | right | |
| 12:14:41 | sean-k-mooney | so the only thing that we currently pass to neturon to tell it which device we selected is the pci address | |
| 12:14:41 | sean-k-mooney | so the only thing that we currently pass to neturon to tell it which device we selected is the pci address | |
| 12:14:58 | sean-k-mooney | from the nova-compute agents point of view | |
| 12:14:58 | sean-k-mooney | from the nova-compute agents point of view | |
| 12:15:05 | Dmitrii-Sh | yes, and that's the problem in this case | |
| 12:15:05 | Dmitrii-Sh | yes, and that's the problem in this case | |
| 12:15:23 | Dmitrii-Sh | the ARM host doens't know how the hypervisor host enumerated its topology | |
| 12:15:23 | Dmitrii-Sh | the ARM host doens't know how the hypervisor host enumerated its topology | |
| 12:15:24 | sean-k-mooney | we cannot use macs and other info like vendor id and project id are not applicable | |
| 12:15:24 | sean-k-mooney | we cannot use macs and other info like vendor id and project id are not applicable | |
| 12:15:31 | Dmitrii-Sh | doesn't* | |
| 12:15:31 | Dmitrii-Sh | doesn't* | |
| 12:15:42 | Dmitrii-Sh | sean-k-mooney: yes | |
| 12:15:42 | Dmitrii-Sh | sean-k-mooney: yes | |
| 12:15:45 | sean-k-mooney | yep so we would need the neuron ml2 driver to translate | |
| 12:15:45 | sean-k-mooney | yep so we would need the neuron ml2 driver to translate | |
| 12:16:09 | sean-k-mooney | we could optionally add some addtional info to the binding profile which i assuem is what you were suggesting | |
| 12:16:09 | sean-k-mooney | we could optionally add some addtional info to the binding profile which i assuem is what you were suggesting | |
| 12:16:33 | Dmitrii-Sh | sean-k-mooney: yes, the PCIe VPD serial number could be it. Additionally, we need a way to send a logical number of a VF | |
| 12:16:33 | Dmitrii-Sh | sean-k-mooney: yes, the PCIe VPD serial number could be it. Additionally, we need a way to send a logical number of a VF | |
| 12:16:43 | Dmitrii-Sh | to pick the right representor on the other side | |
| 12:16:43 | Dmitrii-Sh | to pick the right representor on the other side | |
| 12:16:57 | Dmitrii-Sh | the logical numbers are maintained in the NIC firmware, so presumably they are stable | |
| 12:16:57 | Dmitrii-Sh | the logical numbers are maintained in the NIC firmware, so presumably they are stable | |
| 12:17:11 | sean-k-mooney | i think we can discuover both of those statically and put them in the extra_info column | |
| 12:17:11 | sean-k-mooney | i think we can discuover both of those statically and put them in the extra_info column | |
| 12:17:21 | sean-k-mooney | then we can include that in the port update | |
| 12:17:21 | sean-k-mooney | then we can include that in the port update | |
| 12:17:56 | sean-k-mooney | Dmitrii-Sh: i think we can avoid the need for any db schema changes on the nova side | |
| 12:17:56 | sean-k-mooney | Dmitrii-Sh: i think we can avoid the need for any db schema changes on the nova side | |
| 12:18:45 | sean-k-mooney | currently we only use the extra_info column to store nic features. i was going to store vdpa device paths in them but we removed it | |
| 12:18:45 | Dmitrii-Sh | sean-k-mooney: that's one way to do it. I was planning to do the translation based on the relationship of "transport node" RPs (name == VPD serial) and OVN chassis RPs (name == smartnic_hostname). | |
| 12:18:45 | sean-k-mooney | currently we only use the extra_info column to store nic features. i was going to store vdpa device paths in them but we removed it | |
| 12:18:45 | Dmitrii-Sh | sean-k-mooney: that's one way to do it. I was planning to do the translation based on the relationship of "transport node" RPs (name == VPD serial) and OVN chassis RPs (name == smartnic_hostname). | |
| 12:19:22 | Dmitrii-Sh | So, in essence, the Nova schema changes are there to build the Placement state ("transport node" RPs) | |
| 12:19:22 | Dmitrii-Sh | So, in essence, the Nova schema changes are there to build the Placement state ("transport node" RPs) | |
| 12:19:50 | Dmitrii-Sh | I could build this info as devices are discovered and push it to the Placement DB right away | |
| 12:19:50 | Dmitrii-Sh | I could build this info as devices are discovered and push it to the Placement DB right away | |
| 12:20:13 | sean-k-mooney | well what we are going to discsus later is create rp for each PCI device as a chiled of the compute node rp with the pci address as the RP name | |
| 12:20:13 | sean-k-mooney | well what we are going to discsus later is create rp for each PCI device as a chiled of the compute node rp with the pci address as the RP name | |
| 12:20:31 | sean-k-mooney | thoguh we were not going to suggest havign an rp per vf just per PF | |
| 12:20:31 | sean-k-mooney | thoguh we were not going to suggest havign an rp per vf just per PF | |
| 12:21:03 | sean-k-mooney | so the pci device would be in the sub tree of the compute node RP not under the neutron RPs | |
| 12:21:03 | sean-k-mooney | so the pci device would be in the sub tree of the compute node RP not under the neutron RPs | |
| 12:22:29 | gibi | do I understand correctly from above that one VF from this smartnic can be made available to more than one compute node / hypervisor? | |
| 12:22:29 | gibi | do I understand correctly from above that one VF from this smartnic can be made available to more than one compute node / hypervisor? | |
| 12:22:39 | Dmitrii-Sh | sean-k-mooney: *thinking* | |
| 12:22:39 | Dmitrii-Sh | sean-k-mooney: *thinking* | |
| 12:23:10 | sean-k-mooney | gibi: ish basiably ovs and either ovn/neutron-l2 agent woudl be running on a smartnic isntalled in a server | |
| 12:23:10 | sean-k-mooney | gibi: ish basiably ovs and either ovn/neutron-l2 agent woudl be running on a smartnic isntalled in a server | |
| 12:23:21 | sean-k-mooney | gibi: on the arm cores runnign linux | |
| 12:23:21 | sean-k-mooney | gibi: on the arm cores runnign linux | |
| 12:24:02 | gibi | so from other perspective do we need to represent this smartnic as a sharing RP in placement or can it be under a given compute RP? | |
| 12:24:02 | gibi | so from other perspective do we need to represent this smartnic as a sharing RP in placement or can it be under a given compute RP? | |
| 12:24:03 | Dmitrii-Sh | gibi: there is some IO virtualization sharing tech to allow VFs of one device to be selectively shared across different compute nodes (e.g. in a blade). I have some links in the spec to people describing it. | |
| 12:24:03 | Dmitrii-Sh | gibi: there is some IO virtualization sharing tech to allow VFs of one device to be selectively shared across different compute nodes (e.g. in a blade). I have some links in the spec to people describing it. | |
| 12:24:29 | gibi | Dmitrii-Sh: OK, so it is possible | |
| 12:24:29 | gibi | Dmitrii-Sh: OK, so it is possible | |
| 12:24:54 | Dmitrii-Sh | hence I am going for shared resource providers when it comes to representing SmartNICS | |
| 12:24:54 | Dmitrii-Sh | hence I am going for shared resource providers when it comes to representing SmartNICS | |
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |