| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-22 | |||
| 12:35:10 | Dmitrii-Sh | sean-k-mooney: ok, I'll have a look at whether something falls over with that approach. One challenge is NIC resource allocation when multiple compute nodes are involved but let's see. | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:12 | sean-k-mooney | gibi: the neturon l2 agent and ovn southd are basically equvlent | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:15 | Dmitrii-Sh | sean-k-mooney: yes, 1 per nic | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:40 | Dmitrii-Sh | sean-k-mooney: I considered having something crazy that had 1 ARM host but 2 NICs | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:43 | sean-k-mooney | Dmitrii-Sh: can we declare multi host nics out of scope of the inital proposal | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |
| 12:35:45 | Dmitrii-Sh | but I haven't seen hw like that | |
| 12:36:11 | sean-k-mooney | its not crazy i have seen hardware that was designed to work that way | |
| 12:36:11 | sean-k-mooney | its not crazy i have seen hardware that was designed to work that way | |
| 12:36:29 | sean-k-mooney | it was a top of rack swtich that had a pcie card that plugged into your servers | |
| 12:36:29 | sean-k-mooney | it was a top of rack swtich that had a pcie card that plugged into your servers | |
| 12:36:51 | sean-k-mooney | each server tought it was getting a nic but actully it all the nic hardware was on the switch | |
| 12:36:51 | sean-k-mooney | each server tought it was getting a nic but actully it all the nic hardware was on the switch | |
| 12:37:37 | Dmitrii-Sh | I guess that would be similar to 1 SmartNIC with PCIe bifurcation? Some lanes go to one CPU, some to another? | |
| 12:37:37 | Dmitrii-Sh | I guess that would be similar to 1 SmartNIC with PCIe bifurcation? Some lanes go to one CPU, some to another? | |
| 12:37:54 | sean-k-mooney | kind of yes | |
| 12:37:54 | sean-k-mooney | kind of yes | |
| 12:38:08 | Dmitrii-Sh | so, in this case, there would be 2 PFs exposed to the hypervisor host under different root complexes | |
| 12:38:08 | Dmitrii-Sh | so, in this case, there would be 2 PFs exposed to the hypervisor host under different root complexes | |
| 12:38:28 | Dmitrii-Sh | and the NUMA node association of PFs (and their VFs) would be based on that too | |
| 12:38:28 | Dmitrii-Sh | and the NUMA node association of PFs (and their VFs) would be based on that too | |
| 12:38:33 | sean-k-mooney | basically it was a ginant pcie switch on one end that connectd to multiple hosts and then a ether net switch on the other | |
| 12:38:33 | sean-k-mooney | basically it was a ginant pcie switch on one end that connectd to multiple hosts and then a ether net switch on the other | |
| 12:39:11 | sean-k-mooney | Dmitrii-Sh: ya you can do the pci biforcation with mellonox connetx5 today | |
| 12:39:11 | sean-k-mooney | Dmitrii-Sh: ya you can do the pci biforcation with mellonox connetx5 today | |
| 12:39:40 | sean-k-mooney | they have a 200G sku that has 2x 16 connectors that can connect to 2 different cpus | |
| 12:39:40 | sean-k-mooney | they have a 200G sku that has 2x 16 connectors that can connect to 2 different cpus | |
| 12:39:55 | sean-k-mooney | it get 2 PFs and the VF can float between them | |
| 12:39:55 | sean-k-mooney | it get 2 PFs and the VF can float between them | |
| 12:40:26 | Dmitrii-Sh | sean-k-mooney: yes, there's a special connector on the NIC and an add-in card that plugs to a different topology | |
| 12:40:26 | Dmitrii-Sh | sean-k-mooney: yes, there's a special connector on the NIC and an add-in card that plugs to a different topology | |
| 12:40:40 | Dmitrii-Sh | and the NIC is smart enough to use 2 sets of lanes differently | |
| 12:40:40 | Dmitrii-Sh | and the NIC is smart enough to use 2 sets of lanes differently | |
| 12:40:53 | sean-k-mooney | yep hardware vendors make software enginers look sane | |
| 12:40:53 | sean-k-mooney | yep hardware vendors make software enginers look sane | |
| 12:41:06 | Dmitrii-Sh | sean-k-mooney: :^D | |
| 12:41:06 | Dmitrii-Sh | sean-k-mooney: :^D | |
| 12:41:29 | gibi | lol | |
| 12:42:00 | Dmitrii-Sh | sean-k-mooney: ok, I am aiming to make the design extensible such that those things can be supported but I will avoid focusing on it for the initial proposal then | |
| 12:42:00 | Dmitrii-Sh | sean-k-mooney: ok, I am aiming to make the design extensible such that those things can be supported but I will avoid focusing on it for the initial proposal then | |
| 12:42:11 | Dmitrii-Sh | it's just very difficult to extend it later as I see it | |
| 12:42:11 | Dmitrii-Sh | it's just very difficult to extend it later as I see it | |
| 12:42:28 | sean-k-mooney | i spent quite a lot of my time at intel dealing with topic like this which is why this seam like a realitvly normal request to me | |
| 12:42:28 | sean-k-mooney | i spent quite a lot of my time at intel dealing with topic like this which is why this seam like a realitvly normal request to me | |
| 12:43:06 | sean-k-mooney | Dmitrii-Sh: ya i think that multi host nics proably need cyborg to manage well | |
| 12:43:06 | sean-k-mooney | Dmitrii-Sh: ya i think that multi host nics proably need cyborg to manage well | |
| 12:43:09 | Dmitrii-Sh | sean-k-mooney: good thing you did, otherwise my requests would look insane | |
| 12:43:09 | Dmitrii-Sh | sean-k-mooney: good thing you did, otherwise my requests would look insane | |
| 12:43:32 | sean-k-mooney | Dmitrii-Sh: same as remove nic over a pcie fabric | |
| 12:43:32 | sean-k-mooney | Dmitrii-Sh: same as remove nic over a pcie fabric | |
| 12:44:13 | sean-k-mooney | Dmitrii-Sh: we were looking at this problem space when we were looking at how to integrate a rackscale design composeable server with nova | |
| 12:44:13 | sean-k-mooney | Dmitrii-Sh: we were looking at this problem space when we were looking at how to integrate a rackscale design composeable server with nova | |
| 12:45:11 | sean-k-mooney | Dmitrii-Sh: basicaly i wanted to allow create a composable physical server form disaggreated devices (nics, storage, gpus, fpga) via a new nova virt driver | |
| 12:45:11 | sean-k-mooney | Dmitrii-Sh: basicaly i wanted to allow create a composable physical server form disaggreated devices (nics, storage, gpus, fpga) via a new nova virt driver | |
| 12:45:49 | sean-k-mooney | so it would be like creating a vm but with realy hardware using a pcie/rdma based frabric | |
| 12:45:49 | sean-k-mooney | so it would be like creating a vm but with realy hardware using a pcie/rdma based frabric | |
| 12:46:46 | Dmitrii-Sh | sean-k-mooney: right, I've seen some hw like this with 1U PCIe switches and composable topologies | |
| 12:46:46 | Dmitrii-Sh | sean-k-mooney: right, I've seen some hw like this with 1U PCIe switches and composable topologies | |
| 12:48:43 | sean-k-mooney | :( we never pushed it publically. i thought it was on https://github.com/intel-orchestration-software but we did boot a node with a poc virt driver internally | |
| 12:48:43 | sean-k-mooney | :( we never pushed it publically. i thought it was on https://github.com/intel-orchestration-software but we did boot a node with a poc virt driver internally | |
| 12:50:45 | Dmitrii-Sh | sean-k-mooney: I think I came across this (but haven't had this in a lab): https://gigaio.com/products/ | |
| 12:50:45 | Dmitrii-Sh | sean-k-mooney: I think I came across this (but haven't had this in a lab): https://gigaio.com/products/ | |
| 12:51:18 | sean-k-mooney | Dmitrii-Sh: i think we both have a lot to digest. can you add the topic to the end of https://etherpad.opendev.org/p/nova-xena-ptg | |
| 12:51:18 | sean-k-mooney | Dmitrii-Sh: i think we both have a lot to digest. can you add the topic to the end of https://etherpad.opendev.org/p/nova-xena-ptg | |
| 12:51:55 | sean-k-mooney | as i said if we dont get to it im happy to review and we could set up a call or somehting in a week or too if needed | |
| 12:51:55 | sean-k-mooney | as i said if we dont get to it im happy to review and we could set up a call or somehting in a week or too if needed | |
| 12:52:07 | sean-k-mooney | oh its already there | |
| 12:52:07 | sean-k-mooney | oh its already there | |
| 12:52:25 | sean-k-mooney | Transport Node bp/spec RFC and this topic dont really seam realted to me | |
| 12:52:25 | sean-k-mooney | Transport Node bp/spec RFC and this topic dont really seam realted to me | |
| 12:52:34 | sean-k-mooney | but i see where you were comign form | |
| 12:52:34 | sean-k-mooney | but i see where you were comign form | |
| 12:56:33 | gibi | Dmitrii-Sh: thanks for the well written spec, I forwarded it to my downstream counterparts to see if my company is also interested in solving these use case. I agree with Sean about trying to get to this on the PTG but it might not fit into the timeframe. | |
| 12:56:33 | gibi | Dmitrii-Sh: thanks for the well written spec, I forwarded it to my downstream counterparts to see if my company is also interested in solving these use case. I agree with Sean about trying to get to this on the PTG but it might not fit into the timeframe. | |
| 13:01:31 | Dmitrii-Sh | sean-k-mooney: the gigaio stuff is unrelated, yes, just sharing what I've seen :^) | |
| 13:01:31 | Dmitrii-Sh | sean-k-mooney: yes, added it a little earlier to the etherpad | |
| 13:01:31 | Dmitrii-Sh | sean-k-mooney: the gigaio stuff is unrelated, yes, just sharing what I've seen :^) | |
| 13:01:31 | Dmitrii-Sh | sean-k-mooney: yes, added it a little earlier to the etherpad | |
| 13:02:06 | Dmitrii-Sh | gibi: thanks a lot, much appreciated | |
| 13:02:06 | Dmitrii-Sh | gibi: thanks a lot, much appreciated | |
| 13:02:45 | sean-k-mooney | Dmitrii-Sh: im adding a summery of what i think the problem statemnt is and a sumamry of what i was propsing as an alternitve desing. | |
| 13:02:45 | sean-k-mooney | Dmitrii-Sh: im adding a summery of what i think the problem statemnt is and a sumamry of what i was propsing as an alternitve desing. | |
| 13:02:57 | sean-k-mooney | feel free to extend of correct eitehr | |
| 13:02:57 | sean-k-mooney | feel free to extend of correct eitehr | |
| 13:03:14 | sean-k-mooney | im also addig a link to the irc logs of this conversation if peopel want to review it | |
| 13:03:14 | sean-k-mooney | im also addig a link to the irc logs of this conversation if peopel want to review it | |
| 13:06:20 | Dmitrii-Sh | sean-k-mooney: hmm, all the colors are gone from the etherpad | |
| 13:06:20 | Dmitrii-Sh | sean-k-mooney: hmm, all the colors are gone from the etherpad | |
| 13:06:25 | Dmitrii-Sh | at least for me | |
| 13:06:25 | Dmitrii-Sh | at least for me | |
| 13:06:34 | sean-k-mooney | dont press contol shift c | |
| 13:06:34 | sean-k-mooney | dont press contol shift c | |
| 13:07:12 | sean-k-mooney | yep i had the wrong window slected and ctrl-shift-c is apparently clear all colors | |
| 13:07:12 | sean-k-mooney | yep i had the wrong window slected and ctrl-shift-c is apparently clear all colors | |
| 13:07:23 | sean-k-mooney | i contol z it and it dissconnected me | |
| 13:07:23 | sean-k-mooney | i contol z it and it dissconnected me | |