| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-22 | |||
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:00 | Dmitrii-Sh | much like with shared storage | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:26 | sean-k-mooney | Dmitrii-Sh: i think the simpelst thing to do is as follows. 0 do not modify placment for this specifically. 1 add the extra identifiesr to the pci_device extra info column, 2. add that info the the prot profile. 3 modify the ml2 drivers to sotre the coallation beween the ovn chassi name and the device and have it use that value instead | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:25:51 | gibi | Dmitrii-Sh: thanks, I get it | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:25 | sean-k-mooney | gibi: yep you can do it with connectx-6 cards i think intel were working on it at one point too but no idea if its still a thing they are planning on | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |
| 12:26:50 | sean-k-mooney | gibi: one of the things i was investigating in the past was could we move nova-compute to the smart nic too | |
| 12:27:16 | gibi | this is crazy talk :D | |
| 12:27:16 | gibi | this is crazy talk :D | |
| 12:27:32 | gibi | sorry, my brain failed | |
| 12:27:32 | gibi | sorry, my brain failed | |
| 12:27:53 | sean-k-mooney | idea bing you would jsut grab a stanarcd server and install libvirt on it then you would just install an OpenstackPowerd smarntic that rann all the openstack servics in the server and its an instance compute node after some config updates | |
| 12:27:53 | sean-k-mooney | idea bing you would jsut grab a stanarcd server and install libvirt on it then you would just install an OpenstackPowerd smarntic that rann all the openstack servics in the server and its an instance compute node after some config updates | |
| 12:28:24 | sean-k-mooney | gibi: in this case they are jsut trying to move the networkign contol plane to the smarnic and not all the contol plane | |
| 12:28:24 | sean-k-mooney | gibi: in this case they are jsut trying to move the networkign contol plane to the smarnic and not all the contol plane | |
| 12:28:34 | gibi | ... and then at some point that smartnic is connected to two hypervisors and we are back to the one compute host multiple compute node situation :) | |
| 12:28:34 | gibi | ... and then at some point that smartnic is connected to two hypervisors and we are back to the one compute host multiple compute node situation :) | |
| 12:28:37 | sean-k-mooney | same basic idea however just run less stuff on the hypervior host | |
| 12:28:37 | sean-k-mooney | same basic idea however just run less stuff on the hypervior host | |
| 12:29:09 | sean-k-mooney | gibi: hehe lol lets hope this and rack scale design never have a baby | |
| 12:29:09 | sean-k-mooney | gibi: hehe lol lets hope this and rack scale design never have a baby | |
| 12:30:03 | sean-k-mooney | Dmitrii-Sh: by the way os-vif can connect to ovs over tcp to do the pluggin fo the interface into the ovs bridge | |
| 12:30:03 | sean-k-mooney | Dmitrii-Sh: by the way os-vif can connect to ovs over tcp to do the pluggin fo the interface into the ovs bridge | |
| 12:30:04 | Dmitrii-Sh | sean-k-mooney: I see. (1) ok, there will be some more info stored per pci_device but that's not too bad (2) ok, this can be done during the port update (3) yes, Neutron would need to look up a hostname of a chassis based on the serial number. | |
| 12:30:04 | Dmitrii-Sh | sean-k-mooney: I see. (1) ok, there will be some more info stored per pci_device but that's not too bad (2) ok, this can be done during the port update (3) yes, Neutron would need to look up a hostname of a chassis based on the serial number. | |
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:27 | Dmitrii-Sh | sean-k-mooney: I think os-vif can only use one OVS endpoint | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:28 | sean-k-mooney | Dmitrii-Sh: so we can have os-vif running in the compute agent talk to ovs on the smart nic if it has an ip | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |
| 12:30:36 | Dmitrii-Sh | sean-k-mooney: comes back to the dual-NIC case | |
| 12:31:06 | sean-k-mooney | Dmitrii-Sh: corrrect today. but you could tell use what endpoitn to connecto to in the binding:detail from the ml2 driver | |
| 12:31:06 | sean-k-mooney | Dmitrii-Sh: corrrect today. but you could tell use what endpoitn to connecto to in the binding:detail from the ml2 driver | |
| 12:31:13 | sean-k-mooney | if we need that | |
| 12:31:13 | sean-k-mooney | if we need that | |
| 12:31:32 | sean-k-mooney | e.g. make the config value the defualt and allow it to change per port if needed | |
| 12:31:32 | sean-k-mooney | e.g. make the config value the defualt and allow it to change per port if needed | |
| 12:32:28 | sean-k-mooney | Dmitrii-Sh: yep so if we take this approch we can still track pci device in placment as a seperat effort without having to take account of the network backend in use | |
| 12:32:28 | sean-k-mooney | Dmitrii-Sh: yep so if we take this approch we can still track pci device in placment as a seperat effort without having to take account of the network backend in use | |
| 12:32:54 | gibi | hm does this whole thing also suggest that a single neutron-ovs-agent will handle more than on OVS instance (one per smartnic)? | |
| 12:32:54 | gibi | hm does this whole thing also suggest that a single neutron-ovs-agent will handle more than on OVS instance (one per smartnic)? | |
| 12:33:19 | gibi | s/on OVS/one OVS/ | |
| 12:33:19 | gibi | s/on OVS/one OVS/ | |
| 12:33:30 | sean-k-mooney | no but we might have multipel ovs agents on the same physical server one running on each smart nic | |
| 12:33:30 | sean-k-mooney | no but we might have multipel ovs agents on the same physical server one running on each smart nic | |
| 12:33:52 | gibi | wait, not just the OVS runs on the smartnic but also the neutron agent? | |
| 12:33:52 | gibi | wait, not just the OVS runs on the smartnic but also the neutron agent? | |
| 12:33:56 | sean-k-mooney | gibi: unless we revisit the scaleable ovs agent spec whic intended to od that | |
| 12:33:56 | sean-k-mooney | gibi: unless we revisit the scaleable ovs agent spec whic intended to od that | |
| 12:34:28 | sean-k-mooney | gibi: well in teh case of ovn theere would be a seperate ovn contoler per smartnic form what i understand | |
| 12:34:28 | sean-k-mooney | gibi: well in teh case of ovn theere would be a seperate ovn contoler per smartnic form what i understand | |
| 12:34:49 | sean-k-mooney | so if we are talking about ml2/ovs i would assume we would have 1 agent per smartnic Dmitrii-Sh is that correct? | |
| 12:34:49 | sean-k-mooney | so if we are talking about ml2/ovs i would assume we would have 1 agent per smartnic Dmitrii-Sh is that correct? | |
| 12:35:10 | Dmitrii-Sh | sean-k-mooney: ok, I'll have a look at whether something falls over with that approach. One challenge is NIC resource allocation when multiple compute nodes are involved but let's see. | |
| 12:35:10 | Dmitrii-Sh | sean-k-mooney: ok, I'll have a look at whether something falls over with that approach. One challenge is NIC resource allocation when multiple compute nodes are involved but let's see. | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |
| 12:36:11 | sean-k-mooney | its not crazy i have seen hardware that was designed to work that way | |
| 12:36:11 | sean-k-mooney | its not crazy i have seen hardware that was designed to work that way | |
| 12:36:29 | sean-k-mooney | it was a top of rack swtich that had a pcie card that plugged into your servers | |
| 12:36:29 | sean-k-mooney | it was a top of rack swtich that had a pcie card that plugged into your servers | |
| 12:36:51 | sean-k-mooney | each server tought it was getting a nic but actully it all the nic hardware was on the switch | |
| 12:36:51 | sean-k-mooney | each server tought it was getting a nic but actully it all the nic hardware was on the switch | |
| 12:37:37 | Dmitrii-Sh | I guess that would be similar to 1 SmartNIC with PCIe bifurcation? Some lanes go to one CPU, some to another? | |
| 12:37:37 | Dmitrii-Sh | I guess that would be similar to 1 SmartNIC with PCIe bifurcation? Some lanes go to one CPU, some to another? | |
| 12:37:54 | sean-k-mooney | kind of yes | |
| 12:37:54 | sean-k-mooney | kind of yes | |
| 12:38:08 | Dmitrii-Sh | so, in this case, there would be 2 PFs exposed to the hypervisor host under different root complexes | |
| 12:38:08 | Dmitrii-Sh | so, in this case, there would be 2 PFs exposed to the hypervisor host under different root complexes | |
| 12:38:28 | Dmitrii-Sh | and the NUMA node association of PFs (and their VFs) would be based on that too | |
| 12:38:28 | Dmitrii-Sh | and the NUMA node association of PFs (and their VFs) would be based on that too | |
| 12:38:33 | sean-k-mooney | basically it was a ginant pcie switch on one end that connectd to multiple hosts and then a ether net switch on the other | |
| 12:38:33 | sean-k-mooney | basically it was a ginant pcie switch on one end that connectd to multiple hosts and then a ether net switch on the other | |
| 12:39:11 | sean-k-mooney | Dmitrii-Sh: ya you can do the pci biforcation with mellonox connetx5 today | |
| 12:39:11 | sean-k-mooney | Dmitrii-Sh: ya you can do the pci biforcation with mellonox connetx5 today | |
| 12:39:40 | sean-k-mooney | they have a 200G sku that has 2x 16 connectors that can connect to 2 different cpus | |
| 12:39:40 | sean-k-mooney | they have a 200G sku that has 2x 16 connectors that can connect to 2 different cpus | |
| 12:39:55 | sean-k-mooney | it get 2 PFs and the VF can float between them | |
| 12:39:55 | sean-k-mooney | it get 2 PFs and the VF can float between them | |
| 12:40:26 | Dmitrii-Sh | sean-k-mooney: yes, there's a special connector on the NIC and an add-in card that plugs to a different topology | |
| 12:40:26 | Dmitrii-Sh | sean-k-mooney: yes, there's a special connector on the NIC and an add-in card that plugs to a different topology | |
| 12:40:40 | Dmitrii-Sh | and the NIC is smart enough to use 2 sets of lanes differently | |
| 12:40:40 | Dmitrii-Sh | and the NIC is smart enough to use 2 sets of lanes differently | |
| 12:40:53 | sean-k-mooney | yep hardware vendors make software enginers look sane | |
| 12:40:53 | sean-k-mooney | yep hardware vendors make software enginers look sane | |
| 12:41:06 | Dmitrii-Sh | sean-k-mooney: :^D | |
| 12:41:06 | Dmitrii-Sh | sean-k-mooney: :^D | |
| 12:41:29 | gibi | lol | |
| 12:42:00 | Dmitrii-Sh | sean-k-mooney: ok, I am aiming to make the design extensible such that those things can be supported but I will avoid focusing on it for the initial proposal then | |