| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-22 | |||
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |
| 12:31:06 | sean-k-mooney | Dmitrii-Sh: corrrect today. but you could tell use what endpoitn to connecto to in the binding:detail from the ml2 driver | |
| 12:31:06 | sean-k-mooney | Dmitrii-Sh: corrrect today. but you could tell use what endpoitn to connecto to in the binding:detail from the ml2 driver | |
| 12:31:13 | sean-k-mooney | if we need that | |
| 12:31:13 | sean-k-mooney | if we need that | |
| 12:31:32 | sean-k-mooney | e.g. make the config value the defualt and allow it to change per port if needed | |
| 12:31:32 | sean-k-mooney | e.g. make the config value the defualt and allow it to change per port if needed | |
| 12:32:28 | sean-k-mooney | Dmitrii-Sh: yep so if we take this approch we can still track pci device in placment as a seperat effort without having to take account of the network backend in use | |
| 12:32:28 | sean-k-mooney | Dmitrii-Sh: yep so if we take this approch we can still track pci device in placment as a seperat effort without having to take account of the network backend in use | |
| 12:32:54 | gibi | hm does this whole thing also suggest that a single neutron-ovs-agent will handle more than on OVS instance (one per smartnic)? | |
| 12:32:54 | gibi | hm does this whole thing also suggest that a single neutron-ovs-agent will handle more than on OVS instance (one per smartnic)? | |
| 12:33:19 | gibi | s/on OVS/one OVS/ | |
| 12:33:19 | gibi | s/on OVS/one OVS/ | |
| 12:33:30 | sean-k-mooney | no but we might have multipel ovs agents on the same physical server one running on each smart nic | |
| 12:33:30 | sean-k-mooney | no but we might have multipel ovs agents on the same physical server one running on each smart nic | |
| 12:33:52 | gibi | wait, not just the OVS runs on the smartnic but also the neutron agent? | |
| 12:33:52 | gibi | wait, not just the OVS runs on the smartnic but also the neutron agent? | |
| 12:33:56 | sean-k-mooney | gibi: unless we revisit the scaleable ovs agent spec whic intended to od that | |
| 12:33:56 | sean-k-mooney | gibi: unless we revisit the scaleable ovs agent spec whic intended to od that | |
| 12:34:28 | sean-k-mooney | gibi: well in teh case of ovn theere would be a seperate ovn contoler per smartnic form what i understand | |
| 12:34:28 | sean-k-mooney | gibi: well in teh case of ovn theere would be a seperate ovn contoler per smartnic form what i understand | |
| 12:34:49 | sean-k-mooney | so if we are talking about ml2/ovs i would assume we would have 1 agent per smartnic Dmitrii-Sh is that correct? | |
| 12:34:49 | sean-k-mooney | so if we are talking about ml2/ovs i would assume we would have 1 agent per smartnic Dmitrii-Sh is that correct? | |
| 12:35:10 | Dmitrii-Sh | sean-k-mooney: ok, I'll have a look at whether something falls over with that approach. One challenge is NIC resource allocation when multiple compute nodes are involved but let's see. | |
| 12:35:10 | Dmitrii-Sh | sean-k-mooney: ok, I'll have a look at whether something falls over with that approach. One challenge is NIC resource allocation when multiple compute nodes are involved but let's see. | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |
| 12:36:11 | sean-k-mooney | its not crazy i have seen hardware that was designed to work that way | |
| 12:36:11 | sean-k-mooney | its not crazy i have seen hardware that was designed to work that way | |
| 12:36:29 | sean-k-mooney | it was a top of rack swtich that had a pcie card that plugged into your servers | |
| 12:36:29 | sean-k-mooney | it was a top of rack swtich that had a pcie card that plugged into your servers | |
| 12:36:51 | sean-k-mooney | each server tought it was getting a nic but actully it all the nic hardware was on the switch | |
| 12:36:51 | sean-k-mooney | each server tought it was getting a nic but actully it all the nic hardware was on the switch | |
| 12:37:37 | Dmitrii-Sh | I guess that would be similar to 1 SmartNIC with PCIe bifurcation? Some lanes go to one CPU, some to another? | |
| 12:37:37 | Dmitrii-Sh | I guess that would be similar to 1 SmartNIC with PCIe bifurcation? Some lanes go to one CPU, some to another? | |
| 12:37:54 | sean-k-mooney | kind of yes | |
| 12:37:54 | sean-k-mooney | kind of yes | |
| 12:38:08 | Dmitrii-Sh | so, in this case, there would be 2 PFs exposed to the hypervisor host under different root complexes | |
| 12:38:08 | Dmitrii-Sh | so, in this case, there would be 2 PFs exposed to the hypervisor host under different root complexes | |
| 12:38:28 | Dmitrii-Sh | and the NUMA node association of PFs (and their VFs) would be based on that too | |
| 12:38:28 | Dmitrii-Sh | and the NUMA node association of PFs (and their VFs) would be based on that too | |
| 12:38:33 | sean-k-mooney | basically it was a ginant pcie switch on one end that connectd to multiple hosts and then a ether net switch on the other | |
| 12:38:33 | sean-k-mooney | basically it was a ginant pcie switch on one end that connectd to multiple hosts and then a ether net switch on the other | |
| 12:39:11 | sean-k-mooney | Dmitrii-Sh: ya you can do the pci biforcation with mellonox connetx5 today | |
| 12:39:11 | sean-k-mooney | Dmitrii-Sh: ya you can do the pci biforcation with mellonox connetx5 today | |
| 12:39:40 | sean-k-mooney | they have a 200G sku that has 2x 16 connectors that can connect to 2 different cpus | |
| 12:39:40 | sean-k-mooney | they have a 200G sku that has 2x 16 connectors that can connect to 2 different cpus | |
| 12:39:55 | sean-k-mooney | it get 2 PFs and the VF can float between them | |
| 12:39:55 | sean-k-mooney | it get 2 PFs and the VF can float between them | |
| 12:40:26 | Dmitrii-Sh | sean-k-mooney: yes, there's a special connector on the NIC and an add-in card that plugs to a different topology | |
| 12:40:26 | Dmitrii-Sh | sean-k-mooney: yes, there's a special connector on the NIC and an add-in card that plugs to a different topology | |
| 12:40:40 | Dmitrii-Sh | and the NIC is smart enough to use 2 sets of lanes differently | |
| 12:40:40 | Dmitrii-Sh | and the NIC is smart enough to use 2 sets of lanes differently | |
| 12:40:53 | sean-k-mooney | yep hardware vendors make software enginers look sane | |
| 12:40:53 | sean-k-mooney | yep hardware vendors make software enginers look sane | |
| 12:41:06 | Dmitrii-Sh | sean-k-mooney: :^D | |
| 12:41:06 | Dmitrii-Sh | sean-k-mooney: :^D | |
| 12:41:29 | gibi | lol | |
| 12:42:00 | Dmitrii-Sh | sean-k-mooney: ok, I am aiming to make the design extensible such that those things can be supported but I will avoid focusing on it for the initial proposal then | |
| 12:42:00 | Dmitrii-Sh | sean-k-mooney: ok, I am aiming to make the design extensible such that those things can be supported but I will avoid focusing on it for the initial proposal then | |
| 12:42:11 | Dmitrii-Sh | it's just very difficult to extend it later as I see it | |
| 12:42:11 | Dmitrii-Sh | it's just very difficult to extend it later as I see it | |
| 12:42:28 | sean-k-mooney | i spent quite a lot of my time at intel dealing with topic like this which is why this seam like a realitvly normal request to me | |
| 12:42:28 | sean-k-mooney | i spent quite a lot of my time at intel dealing with topic like this which is why this seam like a realitvly normal request to me | |
| 12:43:06 | sean-k-mooney | Dmitrii-Sh: ya i think that multi host nics proably need cyborg to manage well | |
| 12:43:06 | sean-k-mooney | Dmitrii-Sh: ya i think that multi host nics proably need cyborg to manage well | |
| 12:43:09 | Dmitrii-Sh | sean-k-mooney: good thing you did, otherwise my requests would look insane | |
| 12:43:09 | Dmitrii-Sh | sean-k-mooney: good thing you did, otherwise my requests would look insane | |
| 12:43:32 | sean-k-mooney | Dmitrii-Sh: same as remove nic over a pcie fabric | |
| 12:43:32 | sean-k-mooney | Dmitrii-Sh: same as remove nic over a pcie fabric | |
| 12:44:13 | sean-k-mooney | Dmitrii-Sh: we were looking at this problem space when we were looking at how to integrate a rackscale design composeable server with nova | |
| 12:44:13 | sean-k-mooney | Dmitrii-Sh: we were looking at this problem space when we were looking at how to integrate a rackscale design composeable server with nova | |
| 12:45:11 | sean-k-mooney | Dmitrii-Sh: basicaly i wanted to allow create a composable physical server form disaggreated devices (nics, storage, gpus, fpga) via a new nova virt driver | |
| 12:45:11 | sean-k-mooney | Dmitrii-Sh: basicaly i wanted to allow create a composable physical server form disaggreated devices (nics, storage, gpus, fpga) via a new nova virt driver | |
| 12:45:49 | sean-k-mooney | so it would be like creating a vm but with realy hardware using a pcie/rdma based frabric | |
| 12:45:49 | sean-k-mooney | so it would be like creating a vm but with realy hardware using a pcie/rdma based frabric | |
| 12:46:46 | Dmitrii-Sh | sean-k-mooney: right, I've seen some hw like this with 1U PCIe switches and composable topologies | |
| 12:46:46 | Dmitrii-Sh | sean-k-mooney: right, I've seen some hw like this with 1U PCIe switches and composable topologies | |
| 12:48:43 | sean-k-mooney | :( we never pushed it publically. i thought it was on https://github.com/intel-orchestration-software but we did boot a node with a poc virt driver internally | |
| 12:48:43 | sean-k-mooney | :( we never pushed it publically. i thought it was on https://github.com/intel-orchestration-software but we did boot a node with a poc virt driver internally | |
| 12:50:45 | Dmitrii-Sh | sean-k-mooney: I think I came across this (but haven't had this in a lab): https://gigaio.com/products/ | |
| 12:50:45 | Dmitrii-Sh | sean-k-mooney: I think I came across this (but haven't had this in a lab): https://gigaio.com/products/ | |
| 12:51:18 | sean-k-mooney | Dmitrii-Sh: i think we both have a lot to digest. can you add the topic to the end of https://etherpad.opendev.org/p/nova-xena-ptg | |
| 12:51:18 | sean-k-mooney | Dmitrii-Sh: i think we both have a lot to digest. can you add the topic to the end of https://etherpad.opendev.org/p/nova-xena-ptg | |
| 12:51:55 | sean-k-mooney | as i said if we dont get to it im happy to review and we could set up a call or somehting in a week or too if needed | |
| 12:51:55 | sean-k-mooney | as i said if we dont get to it im happy to review and we could set up a call or somehting in a week or too if needed | |
| 12:52:07 | sean-k-mooney | oh its already there | |