| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-04-19 | |||
| 18:00:35 | openstackgerrit | Merged openstack/cyborg master: Implemented the Objects and APIs for vf/pf https://review.openstack.org/552734 | |
| #openstack-cyborg - 2018-04-20 | |||
| 07:03:02 | openstackgerrit | jiapei proposed openstack/cyborg master: Fix package import error https://review.openstack.org/562954 | |
| #openstack-cyborg - 2018-04-21 | |||
| 00:47:31 | openstackgerrit | melissaml proposed openstack/python-cyborgclient master: Trivial: Update pypi url to new url https://review.openstack.org/563352 | |
| #openstack-cyborg - 2018-04-23 | |||
| 00:34:54 | openstackgerrit | Sundar Nadathur proposed openstack/cyborg master: Specification for Cyborg Agent-Driver API. https://review.openstack.org/561849 | |
| 00:54:50 | openstackgerrit | Sundar Nadathur proposed openstack/cyborg master: Specification for Cyborg/Nova interaction for scheduling. https://review.openstack.org/554717 | |
| #openstack-cyborg - 2018-04-24 | |||
| 04:39:50 | openstackgerrit | Li Liu proposed openstack/cyborg master: Added cyborg fpga programming spec https://review.openstack.org/559395 | |
| #openstack-cyborg - 2018-04-25 | |||
| 02:25:15 | openstackgerrit | Li Liu proposed openstack/cyborg master: Added bitstream metadata standardization spec https://review.openstack.org/558265 | |
| 13:59:19 | Li_Liu | #info Li Liu | |
| 13:59:21 | zhipeng | #startmeeting openstack-cyborg | |
| 13:59:22 | openstack | Meeting started Wed Apr 25 13:59:21 2018 UTC and is due to finish in 60 minutes. The chair is zhipeng. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 13:59:23 | openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 13:59:25 | openstack | The meeting name has been set to 'openstack_cyborg' | |
| 13:59:34 | zhipeng | #topic Roll Call | |
| 13:59:41 | zhipeng | #info Howard | |
| 13:59:45 | NokMikeR | #info Mike | |
| 13:59:48 | Sundar | #info Sundar | |
| 14:00:16 | Sundar | Hi Howard and Mike | |
| 14:00:27 | NokMikeR | Hello Sundar, Howard et all. | |
| 14:00:37 | circ-user-c3hdH | #info Helloway | |
| 14:00:40 | zhipeng | Hello everyone :) | |
| 14:00:49 | Li_Liu | #info Li Liu | |
| 14:01:10 | Li_Liu | Hi guys | |
| 14:03:15 | shaohe_feng_ | #info shaohe | |
| 14:03:27 | shaohe_feng_ | hi all | |
| 14:03:48 | zhipeng | hi | |
| 14:04:45 | zhipeng | let's start then | |
| 14:05:01 | zhipeng | #topic KubeCon EU ResMgmt WG preparation | |
| 14:05:09 | zhipeng | #link https://docs.google.com/document/d/1j3vrG6BgE0hUDs2e-1ZUegKN4W4Adb1B6oJ6j-4kyPU/edit?usp=sharing | |
| 14:05:29 | xinran_ | Hi all | |
| 14:05:36 | zhipeng | so i think I mentioned that for this year's planning I want to be able to align what we have done here | |
| 14:05:41 | zhipeng | with the k8s community | |
| 14:06:09 | zhipeng | KubeCon EU is around the corner next week, and it would be a great place to start participating | |
| 14:06:27 | edleafe | #info edleafe | |
| 14:06:28 | zhipeng | Sundar could you help share some status about the resource mgmt wg in k8s ? | |
| 14:06:36 | Sundar | Sure | |
| 14:06:49 | Li_Liu | that's great | |
| 14:07:05 | Sundar | We started participating last year, with a document describing the FPGA structure and use cases | |
| 14:07:42 | Sundar | The main thing to note is that the FPGA structural model -- with regions, accelerators, local memory etc. -- is the same independent of orch framework -- openStack, K8s etc | |
| 14:08:21 | Sundar | Also, the use cases defined in the Cyborg/Nova spec stay the same -- FPGA as a Service, Accelerated Function as a Service, etc. | |
| 14:08:25 | Sundar | The main difference is in the set of mechanisms available | |
| 14:09:05 | Sundar | In OpenSTack, we have the notion of nested Resource Providers (nRPs), which provides a natural tree structure that matches many device topologies | |
| 14:09:44 | Sundar | The data models and resource handling in K8s is still evolving | |
| 14:10:46 | Sundar | What we have now is the device plugin mechanism: there is standard API by which kubelet can invoke a plugin for a category of devices | |
| 14:11:41 | Sundar | The plugin advertises a resource name, e.g. intel.com/fpga-a10, and lists the devices corresponding to that. There is also a provision to update that list over time, and to report the health of each device | |
| 14:12:09 | Sundar | Based on this information, when a pod spec asks fora resource, the standard K8s scheduler picks a node and informs the kubelet on that node | |
| 14:12:35 | Sundar | The kubelet then invokes another API on the device plugin to allocate a device of the requested type and prepare it | |
| 14:13:22 | Sundar | After that, the kubelet invokes a container runtime (e.g. Docker) through the CRI with an OCI runtime spec | |
| 14:14:05 | Sundar | This basic mechanism does not include the nested structure of FPGAs, and we have been discussing how to fit that in | |
| 14:14:35 | Sundar | However, there are many options: we can use Custom Resource Definitions (CRDs) https://kubernetes.io/docs/concepts/api-extension/custom-resources/#customresourcedefinitions | |
| 14:14:36 | zhipeng | and vGPU as well I suppose ? | |
| 14:15:11 | Sundar | Howard, yes, I think vGPUs, esp.of different types, will also require further consideration | |
| 14:15:52 | Sundar | CRDs are eseentially custom resource classes: we can instantiate resources for a CRD. | |
| 14:16:33 | Sundar | There are also two ongoing proposals for including resource classes: | |
| 14:16:51 | Sundar | The first one is: https://docs.google.com/document/d/1qKiIVs9AMh2Ua5thhtvWqOqW0MSle_RV3lfriO1Aj6U/edit# | |
| 14:17:28 | Sundar | An alternative proposal for resource classes is at: https://docs.google.com/document/d/1666PPUs4Lz56TqKygcy6mXkNazde-vwA7q4e5H92sUc/edit# | |
| 14:18:04 | Sundar | Thei stated goals and non-goals are not exactly the same. | |
| 14:18:05 | Li_Liu | We don't have access two those 2 google docs.. | |
| 14:18:38 | Sundar | Li_Liu, these are supposed to be public dos -- please ask for access | |
| 14:18:41 | Sundar | *docs | |
| 14:18:50 | Li_Liu | ok, just did | |
| 14:19:26 | zhipeng | Sundar there is a PR on Resource API | |
| 14:19:45 | Sundar | IMHO, lot of the discussion comes from GPU background. For FPGAs, we are trying to get alignment within Intel first | |
| 14:19:45 | zhipeng | this is correlates to jiaying's or vish's ? | |
| 14:20:37 | Sundar | Zhipeng, Jiaying's proposal came first -- but I need to look at the PR before confirming | |
| 14:21:34 | zhipeng | okey :) | |
| 14:21:54 | Sundar | So, in short, our task is to find a way of handling FPGAs, without having the benefit of a tree-structured data model, but handling the same FPGA structures and usage models | |
| 14:21:57 | NokMikeR | is the vGPU is vendor specific? ie tied to a particular driver implementation? | |
| 14:23:03 | Li_Liu | Sundar, does the k8s community have any plans to add the tree-structure data model in the near future? | |
| 14:23:05 | Sundar | NokMikeR, the discussions I have seen have been centered on Nvidia's vGPU types, though not necessarily phrased in vendor-specific terms | |
| 14:23:15 | NokMikeR | in other words how do you differentie the features on one vGPU vs another if the underlying features in the real gpu are different - or are they abstracted somehow? | |
| 14:23:20 | Sundar | Li_Liu, None that I am aware of. | |
| 14:24:14 | NokMikeR | Sundar: ok thought so re: nvid gpus. | |
| 14:24:35 | Sundar | NokMikeR, in OpenStack, the answer is clearer: the device itself exposes different vGPU types as traits, and their capacities as units of a generic accelerator RC | |
| 14:25:44 | Sundar | Cyborg needs to handle GPUs and FPGAs of course. But, IMHO, there is enough attention on GPUs :) It is FPGAs that need further thought :) | |
| 14:25:57 | zhipeng | Sundar Jiaying's proposal, as far as I understand, still tries to modify the k8s core functionality ? | |
| 14:26:37 | shaohe_feng_ | Sundar: do we decide to only support one vendor GPU or FPGA in this release without nest Provider? | |
| 14:26:42 | Sundar | Zhipeng, yes, it requires changes on controller side and kubelet changes | |
| 14:28:15 | Sundar | Shaohe: for Rocky, I was proposing to include only device of a particualr type: one GPU or one FPGA. But, based on feedback, w ehave to relax it to multiple devices of the same type, i.e., | |
| 14:28:29 | Sundar | you could have 2 GPUs of the same type, 2 FPGAs of the same type etc. | |
| 14:28:51 | zhipeng | Sundar if we say, propose a CRD type of kube-cyborg thing, will it make sense to the res mgmt wg people ? | |
| 14:29:00 | zhipeng | meaning that similar to OpenStack | |
| 14:29:10 | zhipeng | we view accelerator not part of the general compute infra | |
| 14:29:46 | zhipeng | and have its own model and scheduling process if needed | |
| 14:30:25 | Sundar | ZHipeng, we can propose CRDs, but the exact workflows will matter. | |
| 14:30:49 | Li_Liu | zhipeng, you are saying, similar to what we did to Cyborg, we cut a piece out from K8S? | |
| 14:30:51 | zhipeng | I think the main pain point is still at scheduler extention | |
| 14:31:07 | zhipeng | which Derek also mentioned KubeCon last Dec | |
| 14:31:31 | zhipeng | Li_Liu essentially a out-of-band controller for accelerators | |
| 14:31:43 | shaohe_feng_ | Sundar: will cyborg support nest provider in Rocky release? | |
| 14:32:10 | zhipeng | shaohe_feng_ I think Placement won't support it | |
| 14:32:29 | shaohe_feng_ | zhipeng: Got it. | |
| 14:32:40 | zhipeng | but the way we are modeling it is very close to nrp, correct me if i'm wrong Li_Liu | |
| 14:32:43 | Sundar | Zhipeng, I was also advocating a scheduler extension. But apparently it is not popular within the community. There is a proposal to revamp the scheduler itself: https://docs.google.com/document/d/1NskpTHpOBWtIa5XsgB4bwPHz4IdRxL1RNvdjI7RVGio/edit# | |
| 14:32:58 | Sundar | So, the scheduler, as well as its extension APIs, may change | |
| 14:34:22 | zhipeng | well CRDs are generally great for API aggregation, but complex for resource related functionalities | |
| 14:34:32 | Sundar | Here is a possible way to get to a few basic cases without anything fancy (this is not fully agreed upon, please take this as an option, not a plan): | |
| 14:34:37 | zhipeng | like binding the resource to the pod | |
| 14:34:46 | Sundar | •Publish each region type as a resource. E.g. intel.com/fpga-dcp, intel.com/fpga-vg. •The pod spec asks for a region type as a resource, and also specifies a bitstream ID. That could be a label. •An admission controller inserts an init container on seeing a FPGA resource. •The scheduler picks a node based on the requested region type (and ignores the bitstream ID). •The init container pulls the bitstream wi | |
| 14:34:46 | zhipeng | since the process is external via CRD | |
| 14:35:07 | Sundar | Ah, that didn't come out well in IRC -- let me re-type | |