| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-05-30 | |||
| 14:22:40 | sum12 | any recurring comments ? | |
| 14:23:23 | Li_Liu | 1. Managing GPUs is still not a killer app for Cyborg. Prob showing other capabilities on top of this might draw more attentions | |
| 14:25:12 | Li_Liu | 2. In order to become a strong piece of OpenStack or even addition to k8s in the future, we need to push the development further and faster | |
| 14:25:36 | Li_Liu | as we are actually facing competitions from the Nova vGPU management. | |
| 14:26:09 | Li_Liu | currently we are at pretty much the same stage as them. | |
| 14:27:28 | Li_Liu | some people are looking at us since they think currently vGPU management in Nova should just be temperate. | |
| 14:28:14 | Li_Liu | They believe once Cyborg becomes more mature, it should be the one actually solve the general problem | |
| 14:28:32 | Li_Liu | This is from me. | |
| 14:28:41 | Li_Liu | anyone else has anything to share? | |
| 14:29:24 | Sundar | I moderated a session on Cyborg/FPGA in Cloud/NFV. | |
| 14:29:56 | shaohe_feng | Sundar, can you give a summary about it? | |
| 14:30:04 | Sundar | It was meant for discussion with operators and end users. We had Verizon, SKA and CERN there, apart from many developers, incl. Nova developers | |
| 14:30:35 | Sundar | The discusison is here: #link https://etherpad.openstack.org/p/Cyborg-FPGA-Support-for-Cloud-NFV | |
| 14:31:14 | Sundar | One of the main conclusions is that the model where VM issues programming requests (what we called run-time programming) is important | |
| 14:31:27 | Sundar | However, we do not have a good end-to-end flow for that today. | |
| 14:31:44 | Sundar | This is an open question that we should address. | |
| 14:32:11 | Li_Liu | The main concern is who to trigger the programming right? | |
| 14:32:57 | Sundar | That is one of the main concerns, the other being the validity of the bitstream itself | |
| 14:33:11 | Li_Liu | ah, that one | |
| 14:34:15 | Sundar | For example, if a developer is developing new RTL inside the VM, the resulting bitstream may have bugs. Can that compromise the shell logic? Can it result in local hot spots or other power issues that affect the FPGA chip itself? | |
| 14:35:06 | Sundar | I was hoping to get feedback from operators on how they plan to validate bitstreams. Has Huawei considered that question? | |
| 14:35:45 | Li_Liu | in terms of operation, shall we 1. let this rtl be synthesis in a wel controlled environment. 2. Let this bitstream load on a clean room enviroment first | |
| 14:36:07 | Li_Liu | actually, internally, we did have some discussion | |
| 14:36:57 | Li_Liu | for now, we might not care too much about it as most of the IPs are in house developed | |
| 14:37:18 | Li_Liu | but in the case of letting 3rd party do the development | |
| 14:37:41 | Li_Liu | I think at lease 1. let this rtl be synthesis in a well controlled environment. is needed | |
| 14:38:49 | Sundar | If the environment is well-controlled, the bitstream may still have bugs. So, it is necessary but not sufficient. | |
| 14:40:17 | Sundar | Re. "Let this bitstream load on a clean room enviroment first", that is a good idea. So, do we confine bitstream development to bare-metal instances, so that a tenant can affect only himself or the local FPGA? | |
| 14:40:27 | Li_Liu | yes, that's why the 2nd point is also needed | |
| 14:41:46 | Li_Liu | it depends, it might just be a VM with a entire bitstream. The cloud provider should provide a CI/CD framework for such dev work | |
| 14:41:57 | Li_Liu | entire FPGA sorry :( | |
| 14:42:38 | Sundar | Sure. By clean room environment, do you mean bare-metal instances for development? | |
| 14:44:20 | shaohe_feng | Li_Liu, do you means is a VM just own a region, it does not be allowed to do program except it own the whole FPGA? | |
| 14:44:33 | Li_Liu | hmm, IMO developer/user might not even care/sense this. All they care is I submit my rtl and it passed all the sanity test | |
| 14:45:01 | Li_Liu | shaohe_feng we are not there yet | |
| 14:46:24 | Li_Liu | after the sanity test, the synthesized bitstream is allowed to be loaded on the PR/FPGA | |
| 14:47:58 | Sundar | What is the sanity test? Checking for power issues depends on workloads and dynamic conditions | |
| 14:50:03 | Li_Liu | Sundar, as least the sanity test should check the rlt for security concerns | |
| 14:50:46 | shaohe_feng | Sundar, Li_Liu, the scenario, a tenant by a VM from cloud provider, he want this VM as his development evn, which means he will do his RTL iterative development in this VM | |
| 14:50:51 | Li_Liu | to identify any malicious behavior | |
| 14:52:29 | shaohe_feng | so there will be error during his development. | |
| 14:53:19 | Li_Liu | shaohe_feng, I agree. | |
| 14:54:57 | Li_Liu | Cloud might also want to provide Synthesis as a Service/Sanity Test as a Service | |
| 14:55:36 | Li_Liu | this we all the bitstream that are being loaded on the FPGA are gated by these services | |
| 14:56:23 | shaohe_feng | yes. | |
| 14:59:14 | Li_Liu | I am not sure if Synthesis as a Service/Sanity Test as a Service will be in the Cyborg Scope. But I think all the cloud operator should definitely consider them | |
| 14:59:56 | shaohe_feng | Sundar, what's your opinion? | |
| 15:01:22 | Sundar | Yes, cloud operators need to do 'something'. That often runs into conflict with two things: (a) bitstream developers often want confidentiality, (b) it is tough to take a bitstream (or even a post-fit netlist) and check whether it is 'good' | |
| 15:01:56 | Sundar | I suspect there is no silver bullet, and the answer will be some combination of technology and business practices | |
| 15:02:57 | Sundar | Anyways, the other big question was: how does the VM initiate programming? | |
| 15:03:54 | Li_Liu | I agree, point b is rather complicated. | |
| 15:04:12 | shaohe_feng | Sundar, does the user need to push his image to glance, let cyborg do the program or he do the program in his local VM? | |
| 15:04:20 | Li_Liu | currently, our in house implementation is using REST calls | |
| 15:04:35 | Li_Liu | to call the reprogramming service api | |
| 15:04:44 | Sundar | One way is to let the VM have access to programming the FPGA directly. Another way is to have the VM generate the request and let it come to Cyborg. There are 2 ways to do that, both with pros and cons. | |
| 15:05:51 | Sundar | Shaohe: both models are possible. The main question is, is the developer/tenant ok with sharing his bitstreams with the cloud provider? If so, we can upload it to Glance first | |
| 15:05:53 | Li_Liu | yes, we can provide both capabilities depends on senario | |
| 15:06:16 | Li_Liu | use the 1st way when user owns the BM and entire FPGA | |
| 15:06:27 | Li_Liu | use 2nd way elsewhere | |
| 15:07:14 | Sundar | Li_Liu: Sure. One way is for the VM to invoke Cyborg REST API and target the region assigned to it. Another way is for the VM to have a connection to host e.g. virtio-serial. That way, the bitstream from the VM is not going through many network paths before coming back to the host | |
| 15:08:00 | Li_Liu | We did consider the virtio-serial way | |
| 15:08:07 | Sundar | It also requires less knowledge on the VM's part: it only sends the bitstream ID or the bitstream itself, and Cyborg will target the region assigned to the VM. The VM need not know which region is assigned to it | |
| 15:08:34 | Sundar | Li_Liu: Cool. What is your evaluation of virtio-serial? | |
| 15:08:43 | Li_Liu | but the problem is in order to control the access on who can program the FPGA, the host still need to contact Cyborg anyways | |
| 15:09:07 | Sundar | Well, the Cyborg agent on the host can listen to virtio-serial | |
| 15:09:24 | Li_Liu | so the flow becomes virtio-serial --> Host --> Rest Call --> Cyborg --> access check --> Host --> Program | |
| 15:09:26 | Sundar | May be it is multi-threaded, or has helper daemons, for scalability | |
| 15:10:01 | Sundar | We don;t need REST API in this 2nd flow. The request via virtio-serial is acted upon by the Cyborg agent | |
| 15:10:11 | Li_Liu | it does not save too much network access but make the flow rather complecated | |
| 15:10:52 | Li_Liu | But the Agent needs to download the bitstream from glance | |
| 15:11:07 | Sundar | Are you assuming that REST API is involved, even with virtio-serial? If not, it is a simple 1-hop without TCP/IP networking | |
| 15:11:13 | Li_Liu | and glance needs to check if this user can use this specific bitstream | |
| 15:11:44 | Li_Liu | I think is real use case, you can not avoid REST calls | |
| 15:11:56 | Li_Liu | This is a cloud environment after all | |
| 15:12:28 | Sundar | Li_Liu: true. Either the VM provides the bitstream, which is validated for authenticity/encryption in some way. Or, the VM request refers to a bitstream id in glance | |
| 15:13:12 | Sundar | Li_Liu: "you can not avoid REST calls" Why is that? For checking whether the tenant has privileges? | |
| 15:14:37 | Li_Liu | One of the methodologies is not to allow use send a bitstream to load directly | |
| 15:15:39 | Li_Liu | this is case, the bitstreams are either coming from Glance, or from a synthesis server | |
| 15:16:32 | Li_Liu | we don't want to allow the use send a unkown bitstream through virtio-serial and blindly A | |
| 15:16:40 | Li_Liu | load it to the FPGA | |
| 15:17:02 | Sundar | Oops, running late for my next meeting. Good discussion. :) I'll check back here as much as I can. | |
| 15:17:27 | Li_Liu | actully, I gotta run as well... I am still on vacation... | |
| 15:18:05 | Li_Liu | shaohe_feng, could you terminate the meeting and log the conversation, we can pick it up later | |
| 15:20:40 | shaohe_feng | OK. we can discuss more about it next time. | |
| 15:21:00 | shaohe_feng | #topic docs | |
| 15:21:13 | shaohe_feng | we need to improve cyborg docs. | |
| 15:21:44 | shaohe_feng | I have update the wiki https://wiki.openstack.org/wiki/Cyborg#How_to_contribute | |
| 15:21:48 | shaohe_feng | add DevStack Quick Start link to the wiki | |
| 15:21:55 | shaohe_feng | so New contributors can follow it setup their cyborg env. | |
| 15:22:00 | shaohe_feng | Any issue during the setup, they can fix it and feedback to community. | |
| 15:22:04 | Li_Liu | I will try to work on it with couple other folks in doc tesam | |
| 15:22:23 | shaohe_feng | Li_Liu, good, thanks. | |
| 15:22:41 | shaohe_feng | AoB? | |
| 15:23:48 | shaohe_feng | OK, let's end the meeting. | |
| 15:23:50 | shaohe_feng | #endmeeting | |
| 15:24:17 | shaohe_feng | #endmeeting | |
| 15:24:31 | shaohe_feng | bye. | |
| 15:26:38 | sum12 | i think start meeting was not considered as there was space before startmeeting :) | |
| 15:27:24 | Li_Liu | lol | |
| 15:45:25 | shaohe_feng | #endmeeting | |