| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-22 | |||
| 12:12:47 | Dmitrii-Sh | different addresses too | |
| 12:12:58 | Dmitrii-Sh | basically each host does its own PCIe topology enumeration | |
| 12:12:58 | Dmitrii-Sh | basically each host does its own PCIe topology enumeration | |
| 12:13:34 | Dmitrii-Sh | it's kind of like with blades and non-transparent bridging or managed PCIe switches that hide certain parts of the overall topology | |
| 12:13:34 | Dmitrii-Sh | it's kind of like with blades and non-transparent bridging or managed PCIe switches that hide certain parts of the overall topology | |
| 12:14:17 | sean-k-mooney | right | |
| 12:14:41 | sean-k-mooney | so the only thing that we currently pass to neturon to tell it which device we selected is the pci address | |
| 12:14:41 | sean-k-mooney | so the only thing that we currently pass to neturon to tell it which device we selected is the pci address | |
| 12:14:58 | sean-k-mooney | from the nova-compute agents point of view | |
| 12:14:58 | sean-k-mooney | from the nova-compute agents point of view | |
| 12:15:05 | Dmitrii-Sh | yes, and that's the problem in this case | |
| 12:15:05 | Dmitrii-Sh | yes, and that's the problem in this case | |
| 12:15:23 | Dmitrii-Sh | the ARM host doens't know how the hypervisor host enumerated its topology | |
| 12:15:23 | Dmitrii-Sh | the ARM host doens't know how the hypervisor host enumerated its topology | |
| 12:15:24 | sean-k-mooney | we cannot use macs and other info like vendor id and project id are not applicable | |
| 12:15:24 | sean-k-mooney | we cannot use macs and other info like vendor id and project id are not applicable | |
| 12:15:31 | Dmitrii-Sh | doesn't* | |
| 12:15:31 | Dmitrii-Sh | doesn't* | |
| 12:15:42 | Dmitrii-Sh | sean-k-mooney: yes | |
| 12:15:42 | Dmitrii-Sh | sean-k-mooney: yes | |
| 12:15:45 | sean-k-mooney | yep so we would need the neuron ml2 driver to translate | |
| 12:15:45 | sean-k-mooney | yep so we would need the neuron ml2 driver to translate | |
| 12:16:09 | sean-k-mooney | we could optionally add some addtional info to the binding profile which i assuem is what you were suggesting | |
| 12:16:09 | sean-k-mooney | we could optionally add some addtional info to the binding profile which i assuem is what you were suggesting | |
| 12:16:33 | Dmitrii-Sh | sean-k-mooney: yes, the PCIe VPD serial number could be it. Additionally, we need a way to send a logical number of a VF | |
| 12:16:33 | Dmitrii-Sh | sean-k-mooney: yes, the PCIe VPD serial number could be it. Additionally, we need a way to send a logical number of a VF | |
| 12:16:43 | Dmitrii-Sh | to pick the right representor on the other side | |
| 12:16:43 | Dmitrii-Sh | to pick the right representor on the other side | |
| 12:16:57 | Dmitrii-Sh | the logical numbers are maintained in the NIC firmware, so presumably they are stable | |
| 12:16:57 | Dmitrii-Sh | the logical numbers are maintained in the NIC firmware, so presumably they are stable | |
| 12:17:11 | sean-k-mooney | i think we can discuover both of those statically and put them in the extra_info column | |
| 12:17:11 | sean-k-mooney | i think we can discuover both of those statically and put them in the extra_info column | |
| 12:17:21 | sean-k-mooney | then we can include that in the port update | |
| 12:17:21 | sean-k-mooney | then we can include that in the port update | |
| 12:17:56 | sean-k-mooney | Dmitrii-Sh: i think we can avoid the need for any db schema changes on the nova side | |
| 12:17:56 | sean-k-mooney | Dmitrii-Sh: i think we can avoid the need for any db schema changes on the nova side | |
| 12:18:45 | sean-k-mooney | currently we only use the extra_info column to store nic features. i was going to store vdpa device paths in them but we removed it | |
| 12:18:45 | Dmitrii-Sh | sean-k-mooney: that's one way to do it. I was planning to do the translation based on the relationship of "transport node" RPs (name == VPD serial) and OVN chassis RPs (name == smartnic_hostname). | |
| 12:18:45 | sean-k-mooney | currently we only use the extra_info column to store nic features. i was going to store vdpa device paths in them but we removed it | |
| 12:18:45 | Dmitrii-Sh | sean-k-mooney: that's one way to do it. I was planning to do the translation based on the relationship of "transport node" RPs (name == VPD serial) and OVN chassis RPs (name == smartnic_hostname). | |
| 12:19:22 | Dmitrii-Sh | So, in essence, the Nova schema changes are there to build the Placement state ("transport node" RPs) | |
| 12:19:22 | Dmitrii-Sh | So, in essence, the Nova schema changes are there to build the Placement state ("transport node" RPs) | |
| 12:19:50 | Dmitrii-Sh | I could build this info as devices are discovered and push it to the Placement DB right away | |
| 12:19:50 | Dmitrii-Sh | I could build this info as devices are discovered and push it to the Placement DB right away | |
| 12:20:13 | sean-k-mooney | well what we are going to discsus later is create rp for each PCI device as a chiled of the compute node rp with the pci address as the RP name | |
| 12:20:13 | sean-k-mooney | well what we are going to discsus later is create rp for each PCI device as a chiled of the compute node rp with the pci address as the RP name | |
| 12:20:31 | sean-k-mooney | thoguh we were not going to suggest havign an rp per vf just per PF | |
| 12:20:31 | sean-k-mooney | thoguh we were not going to suggest havign an rp per vf just per PF | |
| 12:21:03 | sean-k-mooney | so the pci device would be in the sub tree of the compute node RP not under the neutron RPs | |
| 12:21:03 | sean-k-mooney | so the pci device would be in the sub tree of the compute node RP not under the neutron RPs | |
| 12:22:29 | gibi | do I understand correctly from above that one VF from this smartnic can be made available to more than one compute node / hypervisor? | |
| 12:22:29 | gibi | do I understand correctly from above that one VF from this smartnic can be made available to more than one compute node / hypervisor? | |
| 12:22:39 | Dmitrii-Sh | sean-k-mooney: *thinking* | |
| 12:22:39 | Dmitrii-Sh | sean-k-mooney: *thinking* | |
| 12:23:10 | sean-k-mooney | gibi: ish basiably ovs and either ovn/neutron-l2 agent woudl be running on a smartnic isntalled in a server | |
| 12:23:10 | sean-k-mooney | gibi: ish basiably ovs and either ovn/neutron-l2 agent woudl be running on a smartnic isntalled in a server | |
| 12:23:21 | sean-k-mooney | gibi: on the arm cores runnign linux | |
| 12:23:21 | sean-k-mooney | gibi: on the arm cores runnign linux | |
| 12:24:02 | gibi | so from other perspective do we need to represent this smartnic as a sharing RP in placement or can it be under a given compute RP? | |
| 12:24:02 | gibi | so from other perspective do we need to represent this smartnic as a sharing RP in placement or can it be under a given compute RP? | |
| 12:24:03 | Dmitrii-Sh | gibi: there is some IO virtualization sharing tech to allow VFs of one device to be selectively shared across different compute nodes (e.g. in a blade). I have some links in the spec to people describing it. | |
| 12:24:03 | Dmitrii-Sh | gibi: there is some IO virtualization sharing tech to allow VFs of one device to be selectively shared across different compute nodes (e.g. in a blade). I have some links in the spec to people describing it. | |
| 12:24:29 | gibi | Dmitrii-Sh: OK, so it is possible | |
| 12:24:29 | gibi | Dmitrii-Sh: OK, so it is possible | |
| 12:24:54 | Dmitrii-Sh | hence I am going for shared resource providers when it comes to representing SmartNICS | |
| 12:24:54 | Dmitrii-Sh | hence I am going for shared resource providers when it comes to representing SmartNICS | |
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |
| 12:27:16 | gibi | this is crazy talk :D | |
| 12:27:16 | gibi | this is crazy talk :D | |
| 12:27:32 | gibi | sorry, my brain failed | |
| 12:27:32 | gibi | sorry, my brain failed | |
| 12:27:53 | sean-k-mooney | idea bing you would jsut grab a stanarcd server and install libvirt on it then you would just install an OpenstackPowerd smarntic that rann all the openstack servics in the server and its an instance compute node after some config updates | |
| 12:27:53 | sean-k-mooney | idea bing you would jsut grab a stanarcd server and install libvirt on it then you would just install an OpenstackPowerd smarntic that rann all the openstack servics in the server and its an instance compute node after some config updates | |
| 12:28:24 | sean-k-mooney | gibi: in this case they are jsut trying to move the networkign contol plane to the smarnic and not all the contol plane | |
| 12:28:24 | sean-k-mooney | gibi: in this case they are jsut trying to move the networkign contol plane to the smarnic and not all the contol plane | |
| 12:28:34 | gibi | ... and then at some point that smartnic is connected to two hypervisors and we are back to the one compute host multiple compute node situation :) | |
| 12:28:34 | gibi | ... and then at some point that smartnic is connected to two hypervisors and we are back to the one compute host multiple compute node situation :) | |
| 12:28:37 | sean-k-mooney | same basic idea however just run less stuff on the hypervior host | |
| 12:28:37 | sean-k-mooney | same basic idea however just run less stuff on the hypervior host | |
| 12:29:09 | sean-k-mooney | gibi: hehe lol lets hope this and rack scale design never have a baby | |
| 12:29:09 | sean-k-mooney | gibi: hehe lol lets hope this and rack scale design never have a baby | |
| 12:30:03 | sean-k-mooney | Dmitrii-Sh: by the way os-vif can connect to ovs over tcp to do the pluggin fo the interface into the ovs bridge | |
| 12:30:03 | sean-k-mooney | Dmitrii-Sh: by the way os-vif can connect to ovs over tcp to do the pluggin fo the interface into the ovs bridge | |
| 12:30:04 | Dmitrii-Sh | sean-k-mooney: I see. (1) ok, there will be some more info stored per pci_device but that's not too bad (2) ok, this can be done during the port update (3) yes, Neutron would need to look up a hostname of a chassis based on the serial number. | |
| 12:30:04 | Dmitrii-Sh | sean-k-mooney: I see. (1) ok, there will be some more info stored per pci_device but that's not too bad (2) ok, this can be done during the port update (3) yes, Neutron would need to look up a hostname of a chassis based on the serial number. | |
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |