| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-22 | |||
| 12:17:56 | sean-k-mooney | Dmitrii-Sh: i think we can avoid the need for any db schema changes on the nova side | |
| 12:17:56 | sean-k-mooney | Dmitrii-Sh: i think we can avoid the need for any db schema changes on the nova side | |
| 12:18:45 | Dmitrii-Sh | sean-k-mooney: that's one way to do it. I was planning to do the translation based on the relationship of "transport node" RPs (name == VPD serial) and OVN chassis RPs (name == smartnic_hostname). | |
| 12:18:45 | sean-k-mooney | currently we only use the extra_info column to store nic features. i was going to store vdpa device paths in them but we removed it | |
| 12:18:45 | Dmitrii-Sh | sean-k-mooney: that's one way to do it. I was planning to do the translation based on the relationship of "transport node" RPs (name == VPD serial) and OVN chassis RPs (name == smartnic_hostname). | |
| 12:18:45 | sean-k-mooney | currently we only use the extra_info column to store nic features. i was going to store vdpa device paths in them but we removed it | |
| 12:19:22 | Dmitrii-Sh | So, in essence, the Nova schema changes are there to build the Placement state ("transport node" RPs) | |
| 12:19:22 | Dmitrii-Sh | So, in essence, the Nova schema changes are there to build the Placement state ("transport node" RPs) | |
| 12:19:50 | Dmitrii-Sh | I could build this info as devices are discovered and push it to the Placement DB right away | |
| 12:19:50 | Dmitrii-Sh | I could build this info as devices are discovered and push it to the Placement DB right away | |
| 12:20:13 | sean-k-mooney | well what we are going to discsus later is create rp for each PCI device as a chiled of the compute node rp with the pci address as the RP name | |
| 12:20:13 | sean-k-mooney | well what we are going to discsus later is create rp for each PCI device as a chiled of the compute node rp with the pci address as the RP name | |
| 12:20:31 | sean-k-mooney | thoguh we were not going to suggest havign an rp per vf just per PF | |
| 12:20:31 | sean-k-mooney | thoguh we were not going to suggest havign an rp per vf just per PF | |
| 12:21:03 | sean-k-mooney | so the pci device would be in the sub tree of the compute node RP not under the neutron RPs | |
| 12:21:03 | sean-k-mooney | so the pci device would be in the sub tree of the compute node RP not under the neutron RPs | |
| 12:22:29 | gibi | do I understand correctly from above that one VF from this smartnic can be made available to more than one compute node / hypervisor? | |
| 12:22:29 | gibi | do I understand correctly from above that one VF from this smartnic can be made available to more than one compute node / hypervisor? | |
| 12:22:39 | Dmitrii-Sh | sean-k-mooney: *thinking* | |
| 12:22:39 | Dmitrii-Sh | sean-k-mooney: *thinking* | |
| 12:23:10 | sean-k-mooney | gibi: ish basiably ovs and either ovn/neutron-l2 agent woudl be running on a smartnic isntalled in a server | |
| 12:23:10 | sean-k-mooney | gibi: ish basiably ovs and either ovn/neutron-l2 agent woudl be running on a smartnic isntalled in a server | |
| 12:23:21 | sean-k-mooney | gibi: on the arm cores runnign linux | |
| 12:23:21 | sean-k-mooney | gibi: on the arm cores runnign linux | |
| 12:24:02 | gibi | so from other perspective do we need to represent this smartnic as a sharing RP in placement or can it be under a given compute RP? | |
| 12:24:02 | gibi | so from other perspective do we need to represent this smartnic as a sharing RP in placement or can it be under a given compute RP? | |
| 12:24:03 | Dmitrii-Sh | gibi: there is some IO virtualization sharing tech to allow VFs of one device to be selectively shared across different compute nodes (e.g. in a blade). I have some links in the spec to people describing it. | |
| 12:24:03 | Dmitrii-Sh | gibi: there is some IO virtualization sharing tech to allow VFs of one device to be selectively shared across different compute nodes (e.g. in a blade). I have some links in the spec to people describing it. | |
| 12:24:29 | gibi | Dmitrii-Sh: OK, so it is possible | |
| 12:24:29 | gibi | Dmitrii-Sh: OK, so it is possible | |
| 12:24:54 | Dmitrii-Sh | hence I am going for shared resource providers when it comes to representing SmartNICS | |
| 12:24:54 | Dmitrii-Sh | hence I am going for shared resource providers when it comes to representing SmartNICS | |
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |
| 12:27:16 | gibi | this is crazy talk :D | |
| 12:27:16 | gibi | this is crazy talk :D | |
| 12:27:32 | gibi | sorry, my brain failed | |
| 12:27:32 | gibi | sorry, my brain failed | |
| 12:27:53 | sean-k-mooney | idea bing you would jsut grab a stanarcd server and install libvirt on it then you would just install an OpenstackPowerd smarntic that rann all the openstack servics in the server and its an instance compute node after some config updates | |
| 12:27:53 | sean-k-mooney | idea bing you would jsut grab a stanarcd server and install libvirt on it then you would just install an OpenstackPowerd smarntic that rann all the openstack servics in the server and its an instance compute node after some config updates | |
| 12:28:24 | sean-k-mooney | gibi: in this case they are jsut trying to move the networkign contol plane to the smarnic and not all the contol plane | |
| 12:28:24 | sean-k-mooney | gibi: in this case they are jsut trying to move the networkign contol plane to the smarnic and not all the contol plane | |
| 12:28:34 | gibi | ... and then at some point that smartnic is connected to two hypervisors and we are back to the one compute host multiple compute node situation :) | |
| 12:28:34 | gibi | ... and then at some point that smartnic is connected to two hypervisors and we are back to the one compute host multiple compute node situation :) | |
| 12:28:37 | sean-k-mooney | same basic idea however just run less stuff on the hypervior host | |
| 12:28:37 | sean-k-mooney | same basic idea however just run less stuff on the hypervior host | |
| 12:29:09 | sean-k-mooney | gibi: hehe lol lets hope this and rack scale design never have a baby | |
| 12:29:09 | sean-k-mooney | gibi: hehe lol lets hope this and rack scale design never have a baby | |
| 12:30:03 | sean-k-mooney | Dmitrii-Sh: by the way os-vif can connect to ovs over tcp to do the pluggin fo the interface into the ovs bridge | |
| 12:30:03 | sean-k-mooney | Dmitrii-Sh: by the way os-vif can connect to ovs over tcp to do the pluggin fo the interface into the ovs bridge | |
| 12:30:04 | Dmitrii-Sh | sean-k-mooney: I see. (1) ok, there will be some more info stored per pci_device but that's not too bad (2) ok, this can be done during the port update (3) yes, Neutron would need to look up a hostname of a chassis based on the serial number. | |
| 12:30:04 | Dmitrii-Sh | sean-k-mooney: I see. (1) ok, there will be some more info stored per pci_device but that's not too bad (2) ok, this can be done during the port update (3) yes, Neutron would need to look up a hostname of a chassis based on the serial number. | |
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |
| 12:31:06 | sean-k-mooney | Dmitrii-Sh: corrrect today. but you could tell use what endpoitn to connecto to in the binding:detail from the ml2 driver | |
| 12:31:06 | sean-k-mooney | Dmitrii-Sh: corrrect today. but you could tell use what endpoitn to connecto to in the binding:detail from the ml2 driver | |
| 12:31:13 | sean-k-mooney | if we need that | |
| 12:31:13 | sean-k-mooney | if we need that | |
| 12:31:32 | sean-k-mooney | e.g. make the config value the defualt and allow it to change per port if needed | |
| 12:31:32 | sean-k-mooney | e.g. make the config value the defualt and allow it to change per port if needed | |
| 12:32:28 | sean-k-mooney | Dmitrii-Sh: yep so if we take this approch we can still track pci device in placment as a seperat effort without having to take account of the network backend in use | |
| 12:32:28 | sean-k-mooney | Dmitrii-Sh: yep so if we take this approch we can still track pci device in placment as a seperat effort without having to take account of the network backend in use | |
| 12:32:54 | gibi | hm does this whole thing also suggest that a single neutron-ovs-agent will handle more than on OVS instance (one per smartnic)? | |
| 12:32:54 | gibi | hm does this whole thing also suggest that a single neutron-ovs-agent will handle more than on OVS instance (one per smartnic)? | |
| 12:33:19 | gibi | s/on OVS/one OVS/ | |
| 12:33:19 | gibi | s/on OVS/one OVS/ | |
| 12:33:30 | sean-k-mooney | no but we might have multipel ovs agents on the same physical server one running on each smart nic | |
| 12:33:30 | sean-k-mooney | no but we might have multipel ovs agents on the same physical server one running on each smart nic | |
| 12:33:52 | gibi | wait, not just the OVS runs on the smartnic but also the neutron agent? | |
| 12:33:52 | gibi | wait, not just the OVS runs on the smartnic but also the neutron agent? | |
| 12:33:56 | sean-k-mooney | gibi: unless we revisit the scaleable ovs agent spec whic intended to od that | |
| 12:33:56 | sean-k-mooney | gibi: unless we revisit the scaleable ovs agent spec whic intended to od that | |
| 12:34:28 | sean-k-mooney | gibi: well in teh case of ovn theere would be a seperate ovn contoler per smartnic form what i understand | |
| 12:34:28 | sean-k-mooney | gibi: well in teh case of ovn theere would be a seperate ovn contoler per smartnic form what i understand | |
| 12:34:49 | sean-k-mooney | so if we are talking about ml2/ovs i would assume we would have 1 agent per smartnic Dmitrii-Sh is that correct? | |
| 12:34:49 | sean-k-mooney | so if we are talking about ml2/ovs i would assume we would have 1 agent per smartnic Dmitrii-Sh is that correct? | |
| 12:35:10 | Dmitrii-Sh | sean-k-mooney: ok, I'll have a look at whether something falls over with that approach. One challenge is NIC resource allocation when multiple compute nodes are involved but let's see. | |
| 12:35:10 | Dmitrii-Sh | sean-k-mooney: ok, I'll have a look at whether something falls over with that approach. One challenge is NIC resource allocation when multiple compute nodes are involved but let's see. | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |