| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2017-11-15 | |||
| 15:18:10 | masuberu | well intel is quite big | |
| 15:18:12 | masuberu | I guess | |
| 15:18:16 | shaohe_feng__ | masuberu: hi | |
| 15:18:22 | zhipeng | haha yes | |
| 15:18:33 | Guest69871 | Hi Masuberu, This is Li from Huawei, our team has a lot of experience using FPGAs in Openstack | |
| 15:18:36 | masuberu | the person coming tomorrow is from either China or Singapore I think | |
| 15:18:58 | zhipeng | masuberu could you describe a possible setup ? | |
| 15:19:20 | zhipeng | For example if you are gonna have a VM connected to the FPGA | |
| 15:19:38 | zhipeng | or just feeding data directly to GTAK4 on that FPGA ? | |
| 15:20:04 | masuberu | what ever is easier, I was thinking of giving a go and see if could be possible to integrate FPGAs with nova and see if the vms could see it | |
| 15:20:13 | shaohe_feng__ | masuberu: I'm also working on FPGA. And some of us has started to support FPGA on k8s | |
| 15:20:14 | masuberu | otherwise I am happy of using different methods | |
| 15:20:27 | masuberu | ok | |
| 15:20:36 | masuberu | I am totally open to suggestions | |
| 15:20:45 | masuberu | the setup is | |
| 15:21:13 | masuberu | 4 compute nodes with AMD cpus (56 cores) 512 RAM and ~4.5TB spinning disks 10k | |
| 15:21:17 | Guest69871 | Do you have the developed IPs to use for the FPGA? | |
| 15:21:26 | zhipeng | #info masuberu from Garvan Institute of Medical Research | |
| 15:21:35 | masuberu | and 3 nodes with intel cpus (numa 2 sockets 14 cpus each) 512 GB RAM and 20 TB nvmes | |
| 15:21:56 | zhipeng | #link https://www.youtube.com/watch?v=HSAJpAxORAw | |
| 15:21:57 | masuberu | all connected through 2x25GB bond network | |
| 15:22:04 | masuberu | network is mellanox ethernet | |
| 15:22:11 | masuberu | mellanos switch 100GB | |
| 15:22:44 | masuberu | it is not big setup but I think it is a good starting point | |
| 15:23:39 | LiLiu_ | #info Li | |
| 15:23:41 | masuberu | I don't know how many FPGAs I will get, I will know tomorrow | |
| 15:23:53 | zhipeng | shaohe_feng_ any tips on nova integrates with FPGA ? | |
| 15:24:11 | masuberu | so I am not sure if that explains most of the questions? please feel free to ask | |
| 15:24:32 | zhipeng | masuberu will GTAK4 run as an IP for the FPGA ? | |
| 15:25:06 | masuberu | tomorrow I have a meeting with Albert from intel and he will brief me about how it works | |
| 15:25:16 | LiLiu_ | Question: is there multiple IPs to be used in the GATK4 | |
| 15:25:42 | masuberu | LiLiu_, what do you mean by multiple IPs? | |
| 15:26:06 | LiLiu_ | I mean different bitstreams | |
| 15:26:50 | masuberu | so GATK4 is a bunch of tools bioinformatitians use to create pipelines | |
| 15:27:39 | masuberu | you can run a tool at a time or multiple ones at the same time, depends on what you are doing | |
| 15:27:54 | zhipeng | #link https://github.com/broadinstitute/gatk | |
| 15:28:33 | LiLiu_ | OK, I assume you will have multiple types of IPs in your system. Do you chain the FPGAs together or each FPGA only talk to the tied CPU ? | |
| 15:29:00 | masuberu | ok, I will ask that question to the intel guy tomorrow | |
| 15:29:09 | masuberu | I honestly never used a FPGA before | |
| 15:29:45 | zhipeng | shaohe_feng__ any tips on nova integrates with FPGA ? | |
| 15:31:07 | LiLiu_ | I see, my team works a lot on FPGA heterogeneous system in Openstack, we may talk more offline if you are interested, masuberu. | |
| 15:31:28 | masuberu | sure | |
| 15:31:38 | LiLiu_ | liliu1@huawei.com | |
| 15:33:04 | zhipeng | did some googling | |
| 15:33:05 | LiLiu_ | I wanna know more about the scope/size of the system you guys are targeting | |
| 15:33:28 | masuberu | it will depends on the performance of this tests | |
| 15:33:29 | zhipeng | seems like the pairHMM will be the part that need acceleration from FPGA or GPU ? | |
| 15:34:09 | masuberu | for instance we have capacity to sequence 1200 genomes per month so it all depends how much hardware we need for that | |
| 15:34:31 | zhipeng | #link https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/wp/wp-accelerating-genomics-opencl-fpgas.pdf | |
| 15:34:47 | shaohe_feng__ | zhipeng: actually, every accelerator in Intel's FPGA is a sriov device. | |
| 15:35:02 | masuberu | zhipeng, yep I got that document yesterday from Albert | |
| 15:35:22 | zhipeng | shaohe_feng__ how about the altera ones ? | |
| 15:35:26 | zhipeng | the same ? | |
| 15:35:37 | shaohe_feng__ | zhipeng: altera means intel :) | |
| 15:35:54 | zhipeng | haha | |
| 15:36:16 | masuberu | nallatech 385A is what I am getting | |
| 15:36:16 | shaohe_feng__ | zhipeng: there maybe 3 method to support FPGA. | |
| 15:36:30 | masuberu | getting for testing | |
| 15:36:54 | shaohe_feng__ | zhipeng: 1. pre-programming FPGA on the host. | |
| 15:38:02 | shaohe_feng__ | zhipeng: 2. the orchestration program FPGA | |
| 15:38:15 | shaohe_feng__ | zhipeng: 3. VM program FPGA. | |
| 15:38:36 | shaohe_feng__ | zhipeng: I have got a FPGA server, and I'm trying to do some test on it | |
| 15:38:54 | masuberu | shaohe_feng__, the person coming tomorrow is called AlbertZQ Wang, not sure if you know him... | |
| 15:39:25 | shaohe_feng__ | zhipeng: AlbertZQ Wang from intel? | |
| 15:39:36 | shaohe_feng__ | ^ masuberu: | |
| 15:39:52 | masuberu | yes | |
| 15:40:02 | shaohe_feng__ | which site? China or US | |
| 15:40:17 | masuberu | I guess China but not 100% sure | |
| 15:40:22 | masuberu | maybe either of them | |
| 15:41:48 | masuberu | any questions? | |
| 15:41:53 | shaohe_feng__ | masuberu: haha, we can connect each other convenience in intel's internal connection tool. | |
| 15:42:23 | masuberu | yeah they showed me that tool last time they came to my office | |
| 15:42:38 | masuberu | any plan? | |
| 15:42:59 | zhipeng | i think this is definitely a very interesting use case for us | |
| 15:43:22 | zhipeng | it would be great if you could come back with more details from the Intel guys tomorrow | |
| 15:43:34 | zhipeng | and we could hammer out details for future work | |
| 15:43:49 | zhipeng | i think getting Nova connect with FPGA is easy | |
| 15:44:15 | zhipeng | but how to sched and manage your FPGAs is what we will be working on | |
| 15:44:21 | masuberu | zhipeng, you mean because of SRIOV? | |
| 15:44:49 | masuberu | it can be nova or k8s | |
| 15:44:55 | masuberu | or another method | |
| 15:45:05 | zhipeng | yes because the current support for Nova is good enough | |
| 15:45:08 | zhipeng | to set it up | |
| 15:45:11 | masuberu | ok | |
| 15:45:18 | masuberu | what about overhead? | |
| 15:45:36 | zhipeng | what type of overhead you are thinking about ? | |
| 15:45:38 | masuberu | is it going to have much overhead due to virtualization? | |
| 15:45:50 | zhipeng | i think that depends | |
| 15:46:05 | zhipeng | how you setup the workload between VM and FPGA | |
| 15:46:37 | masuberu | ok, is there anything I can read about that? because I don't know much to be honest | |
| 15:46:46 | zhipeng | for example if you gonna have FPGA do all the GATK heavy lifting and VM just for running a data pipe application | |
| 15:46:53 | masuberu | and I want to get the most of this project | |
| 15:47:04 | zhipeng | i don't think there will be much overhead | |
| 15:47:05 | masuberu | I don't know for how long I can have the FPGAs | |
| 15:47:54 | masuberu | lets say we decide to run on the numa nodes, then I can even enable cpu afinity | |
| 15:48:10 | masuberu | to make sure process use local memory | |
| 15:48:16 | masuberu | but that is a different story | |
| 15:48:31 | zhipeng | yes | |
| 15:48:48 | masuberu | I am more worry about the lack of documentation about openstack+FPGAs | |
| 15:49:16 | zhipeng | you can just go through the nova pci support docs | |