| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-04-25 | |||
| 14:15:11 | Sundar | Howard, yes, I think vGPUs, esp.of different types, will also require further consideration | |
| 14:15:52 | Sundar | CRDs are eseentially custom resource classes: we can instantiate resources for a CRD. | |
| 14:16:33 | Sundar | There are also two ongoing proposals for including resource classes: | |
| 14:16:51 | Sundar | The first one is: https://docs.google.com/document/d/1qKiIVs9AMh2Ua5thhtvWqOqW0MSle_RV3lfriO1Aj6U/edit# | |
| 14:17:28 | Sundar | An alternative proposal for resource classes is at: https://docs.google.com/document/d/1666PPUs4Lz56TqKygcy6mXkNazde-vwA7q4e5H92sUc/edit# | |
| 14:18:04 | Sundar | Thei stated goals and non-goals are not exactly the same. | |
| 14:18:05 | Li_Liu | We don't have access two those 2 google docs.. | |
| 14:18:38 | Sundar | Li_Liu, these are supposed to be public dos -- please ask for access | |
| 14:18:41 | Sundar | *docs | |
| 14:18:50 | Li_Liu | ok, just did | |
| 14:19:26 | zhipeng | Sundar there is a PR on Resource API | |
| 14:19:45 | zhipeng | this is correlates to jiaying's or vish's ? | |
| 14:19:45 | Sundar | IMHO, lot of the discussion comes from GPU background. For FPGAs, we are trying to get alignment within Intel first | |
| 14:20:37 | Sundar | Zhipeng, Jiaying's proposal came first -- but I need to look at the PR before confirming | |
| 14:21:34 | zhipeng | okey :) | |
| 14:21:54 | Sundar | So, in short, our task is to find a way of handling FPGAs, without having the benefit of a tree-structured data model, but handling the same FPGA structures and usage models | |
| 14:21:57 | NokMikeR | is the vGPU is vendor specific? ie tied to a particular driver implementation? | |
| 14:23:03 | Li_Liu | Sundar, does the k8s community have any plans to add the tree-structure data model in the near future? | |
| 14:23:05 | Sundar | NokMikeR, the discussions I have seen have been centered on Nvidia's vGPU types, though not necessarily phrased in vendor-specific terms | |
| 14:23:15 | NokMikeR | in other words how do you differentie the features on one vGPU vs another if the underlying features in the real gpu are different - or are they abstracted somehow? | |
| 14:23:20 | Sundar | Li_Liu, None that I am aware of. | |
| 14:24:14 | NokMikeR | Sundar: ok thought so re: nvid gpus. | |
| 14:24:35 | Sundar | NokMikeR, in OpenStack, the answer is clearer: the device itself exposes different vGPU types as traits, and their capacities as units of a generic accelerator RC | |
| 14:25:44 | Sundar | Cyborg needs to handle GPUs and FPGAs of course. But, IMHO, there is enough attention on GPUs :) It is FPGAs that need further thought :) | |
| 14:25:57 | zhipeng | Sundar Jiaying's proposal, as far as I understand, still tries to modify the k8s core functionality ? | |
| 14:26:37 | shaohe_feng_ | Sundar: do we decide to only support one vendor GPU or FPGA in this release without nest Provider? | |
| 14:26:42 | Sundar | Zhipeng, yes, it requires changes on controller side and kubelet changes | |
| 14:28:15 | Sundar | Shaohe: for Rocky, I was proposing to include only device of a particualr type: one GPU or one FPGA. But, based on feedback, w ehave to relax it to multiple devices of the same type, i.e., | |
| 14:28:29 | Sundar | you could have 2 GPUs of the same type, 2 FPGAs of the same type etc. | |
| 14:28:51 | zhipeng | Sundar if we say, propose a CRD type of kube-cyborg thing, will it make sense to the res mgmt wg people ? | |
| 14:29:00 | zhipeng | meaning that similar to OpenStack | |
| 14:29:10 | zhipeng | we view accelerator not part of the general compute infra | |
| 14:29:46 | zhipeng | and have its own model and scheduling process if needed | |
| 14:30:25 | Sundar | ZHipeng, we can propose CRDs, but the exact workflows will matter. | |
| 14:30:49 | Li_Liu | zhipeng, you are saying, similar to what we did to Cyborg, we cut a piece out from K8S? | |
| 14:30:51 | zhipeng | I think the main pain point is still at scheduler extention | |
| 14:31:07 | zhipeng | which Derek also mentioned KubeCon last Dec | |
| 14:31:31 | zhipeng | Li_Liu essentially a out-of-band controller for accelerators | |
| 14:31:43 | shaohe_feng_ | Sundar: will cyborg support nest provider in Rocky release? | |
| 14:32:10 | zhipeng | shaohe_feng_ I think Placement won't support it | |
| 14:32:29 | shaohe_feng_ | zhipeng: Got it. | |
| 14:32:40 | zhipeng | but the way we are modeling it is very close to nrp, correct me if i'm wrong Li_Liu | |
| 14:32:43 | Sundar | Zhipeng, I was also advocating a scheduler extension. But apparently it is not popular within the community. There is a proposal to revamp the scheduler itself: https://docs.google.com/document/d/1NskpTHpOBWtIa5XsgB4bwPHz4IdRxL1RNvdjI7RVGio/edit# | |
| 14:32:58 | Sundar | So, the scheduler, as well as its extension APIs, may change | |
| 14:34:22 | zhipeng | well CRDs are generally great for API aggregation, but complex for resource related functionalities | |
| 14:34:32 | Sundar | Here is a possible way to get to a few basic cases without anything fancy (this is not fully agreed upon, please take this as an option, not a plan): | |
| 14:34:37 | zhipeng | like binding the resource to the pod | |
| 14:34:46 | zhipeng | since the process is external via CRD | |
| 14:34:46 | Sundar | •Publish each region type as a resource. E.g. intel.com/fpga-dcp, intel.com/fpga-vg. •The pod spec asks for a region type as a resource, and also specifies a bitstream ID. That could be a label. •An admission controller inserts an init container on seeing a FPGA resource. •The scheduler picks a node based on the requested region type (and ignores the bitstream ID). •The init container pulls the bitstream wi | |
| 14:35:07 | Sundar | Ah, that didn't come out well in IRC -- let me re-type | |
| 14:35:24 | Sundar | •Publish each region type as a resource. E.g. intel.com/fpga-dcp, intel.com/fpga-vg | |
| 14:35:34 | Sundar | •The pod spec asks for a region type as a resource, and also specifies a bitstream ID. That could be a label | |
| 14:35:45 | Sundar | •An admission controller inserts an init container on seeing a FPGA resource. | |
| 14:35:59 | Sundar | •The scheduler picks a node based on the requested region type (and ignores the bitstream ID). | |
| 14:36:10 | Sundar | •The init container pulls the bitstream with that ID from a bitstream repository (mechanism TBD) and programs the selected device. | |
| 14:37:20 | Sundar | I have heard that, if we give a higher security context to the init container for programming, it may affect other containers in the same pod. I am still trying to find evidence for that | |
| 14:37:46 | zhipeng | lol this is just too complicated | |
| 14:38:07 | zhipeng | thx Sundar I think we have a good understanding of the status qup | |
| 14:38:10 | zhipeng | quo | |
| 14:38:18 | Sundar | Zhipeng, more complicated than other proposals out there? ;) | |
| 14:38:25 | zhipeng | anyone else got questions regarding k8s ? | |
| 14:38:54 | zhipeng | if you are attending KubeCon we could meet f2f, and give them hell XD | |
| 14:39:09 | Sundar | lol | |
| 14:41:04 | Li_Liu | if you guys have any dial-in-able meeting during kubecon, please loop us in | |
| 14:41:36 | shaohe_feng_ | yes, loop us in | |
| 14:42:02 | zhipengh[m] | Okey I will give a howler if a bridge is available | |
| 14:42:33 | zhipengh[m] | Seems like my PC irc client just died | |
| 14:43:05 | zhipengh[m] | #topic Sub team arrangements | |
| 14:46:08 | zhipeng | phew | |
| 14:46:13 | Sundar | :) | |
| 14:46:41 | zhipeng | cell phone irc bouncer crashed just now | |
| 14:46:57 | zhipeng | moving on | |
| 14:47:05 | zhipeng | #topic subteam arrangments | |
| 14:47:35 | zhipeng | okey so given recent events, I think it is necessary to reorg the subteams | |
| 14:48:23 | zhipeng | and also encourage subteam to organize their specific meetings | |
| 14:48:30 | zhipeng | for specific topics | |
| 14:48:45 | zhipeng | so I would suggest shaohe to help lead the driver subteam | |
| 14:49:10 | zhipeng | work with our Xilinx and lenovo colleagues on FPGA and GPU driver in Rocky | |
| 14:49:29 | shaohe_feng_ | Ok. | |
| 14:49:40 | zhipeng | Li Liu help lead the doc team, to work with our CMCC member and others to make documentation as good as your spec :) | |
| 14:49:49 | Li_Liu | sure | |
| 14:49:59 | zhipeng | I will keep on the release mgmt side | |
| 14:50:35 | zhipeng | shaohe_feng_ you can sync up with Chuck_ on a meeting time more suited for US west coast | |
| 14:50:55 | zhipeng | mainly China morning times I guess | |
| 14:51:13 | shaohe_feng_ | zhipeng: what is Chuck_? | |
| 14:51:28 | Chuck_ | I am Chuck_ :-) | |
| 14:51:37 | Chuck_ | Hi Shaohe, this is Chuck from Xilinx | |
| 14:51:44 | Sundar | lol | |
| 14:51:46 | shaohe_feng_ | Chuck_: hello | |
| 14:51:46 | Chuck_ | I work from US west time | |
| 14:51:54 | zhipeng | I will add Chuck_ into our wechat group as well | |
| 14:52:13 | zhipeng | talk in Chinese :) | |
| 14:52:47 | Li_Liu | count me in for those driver meeting shaohe_feng_ | |
| 14:52:52 | Li_Liu | :) | |
| 14:52:53 | Chuck_ | yes, look forward to working with you. | |
| 14:53:20 | shaohe_feng_ | Li_Liu: OK. | |
| 14:53:21 | zhipeng | and subteam plz send report to the mailing list, you can decide whether it is bi-weekly or weekly | |
| 14:53:26 | zhipeng | or monthly even | |
| 14:53:28 | zhipeng | up to you | |
| 14:53:56 | Sundar | Is it all WeChat in Chinse then? ;) I can join if that helps | |