| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-05-09 | |||
| 14:12:00 | sundar | Yes, Shaohe. I clicked on the doodle now. Sunday eve or Mon 7 am PST are both good | |
| 14:13:31 | zhipeng | ok let's do the Mon UTC 2:00pm | |
| 14:13:57 | zhipeng | which will be 10:00pm at China, 10:00 am EST, 7:00am PST | |
| 14:15:19 | zhipeng | it's the most popular time | |
| 14:15:22 | zhipeng | okey ? | |
| 14:15:32 | NokMikeR | ok | |
| 14:15:32 | sundar | Sounds good to me! | |
| 14:16:02 | zhipeng | #agreed driver subteam meeting time every Mon UTC1400 | |
| 14:16:04 | Li_Liu | Monday? | |
| 14:16:15 | zhipeng | Monday | |
| 14:16:17 | Li_Liu | ok | |
| 14:16:21 | Li_Liu | sounds good | |
| 14:16:42 | zhipeng | #topic new core reviewer promotion | |
| 14:17:06 | zhipeng | in order to increase our review bandwidth, i hereby promote Sundar to be the new core reviewer | |
| 14:17:24 | zhipeng | Sundar has been very active and taking charge of several important specs | |
| 14:17:57 | zhipeng | so as usual, we will have one week time for any feedback, and acknowledgment of the promotion next Wed :) | |
| 14:18:26 | kosamara | #info kosamara | |
| 14:18:29 | sundar | Thanks, zhipeng | |
| 14:18:41 | Li_Liu | gratz :) | |
| 14:19:03 | shaohe | gratz :) | |
| 14:19:19 | zhipeng | well not yet :) | |
| 14:19:39 | zhipeng | let's wait for a week for feedback | |
| 14:20:00 | sundar | Thanks, Li_Liu and shaohe :). As zhipeng says, one week from now, we will know. | |
| 14:20:06 | zhipeng | but would like to thanks for Sundar's great effort so far | |
| 14:20:12 | zhipeng | :) | |
| 14:20:16 | zhipeng | okey moving on | |
| 14:20:41 | zhipeng | #topic KubeCon feedback | |
| 14:20:58 | zhipeng | okey last week i attend KubeCon and the resource mgmt wg deep dive session | |
| 14:21:34 | zhipeng | k8s res mgmt wg is the center in k8s which deals with general and acceleration resources | |
| 14:22:10 | zhipeng | My takeaway is that the support for a general accelerator mgmt, is still not in any shape in k8s | |
| 14:22:23 | zhipeng | Google is interested in GPU passthrough support for ML, mainly | |
| 14:23:32 | zhipeng | so if anyone wants to introduce any feature in-tree | |
| 14:24:26 | zhipeng | that would require a PoC up-front | |
| 14:24:59 | zhipeng | many things we discussed here, like vGPU types, general accelerator support including FPGA and others | |
| 14:25:05 | zhipeng | are viewed non-priority | |
| 14:25:22 | zhipeng | the resource class/resource api PRs are also long shot | |
| 14:25:28 | zhipeng | according to vishnu | |
| 14:27:29 | sundar | zhipeng: Agreed with your assessments. If we can get just one feature in -- passing annotations to the device plugin API -- that will help us meet most basic FPGA use cases, IMHO | |
| 14:27:55 | zhipeng | Sasha mentioned Intel team has finished a FPGA DPI PoC | |
| 14:28:04 | zhipeng | but also just pre-programmed FPGAs | |
| 14:28:39 | sundar | We can couple that with a scheduler extension. However, scheduler extensions are not viewed favorably because the scheduler fra,ework itself may change and the APIs may change along with it. | |
| 14:29:00 | zhipeng | yes, and also DPI is designed at the node level | |
| 14:29:09 | sundar | However, we can do a POC base don it, including programming support, and revamp it when the APIs change. Just my thought. :) | |
| 14:29:15 | zhipeng | and a mostly "reschedule" focused mechanism | |
| 14:29:36 | sundar | Yes, we do have a POC that does only pre-programmed use case. That does not show the strength of FPGAs, which is reprogramming | |
| 14:30:01 | zhipeng | so DPI is designed to mostly work for hot-plug use case, not scheduling upfront | |
| 14:30:22 | zhipeng | the scheduling will be retriggered once the node discover the DPI Plugin | |
| 14:30:35 | zhipeng | anyways it is the current lay of land | |
| 14:31:04 | zhipeng | so my thinking is, maybe it is reasonable to introduce a CRD framework for cyborg into k8s community | |
| 14:31:35 | zhipeng | so that we could have all of our data model preserved, has leeway on the api and scheduling design | |
| 14:31:46 | zhipeng | maintain a k8s-ish API interface | |
| 14:32:12 | zhipeng | and a out-of-band general accelerator mgmt functionality, not bound to DPI development | |
| 14:32:42 | zhipeng | i don't know what other team member's thought on this matter ? | |
| 14:32:44 | sundar | The CRD framework does not allow AFAIK for a nested topology, which OpenStack supports. | |
| 14:33:02 | zhipeng | CRD is just an API mechinism right ? | |
| 14:33:06 | NokMikeR | needs more blinking lights | |
| 14:33:12 | zhipeng | not implementation specific | |
| 14:33:56 | sundar | Yes. How do we model regions inside FPGAs, accelerators inside regions, local memory inside either ... | |
| 14:34:10 | zhipeng | that could all be done in Cyborg | |
| 14:35:18 | zhipeng | for example if you look at kubernetes service catalog | |
| 14:35:25 | Li_Liu | k8s will do the scheduling for Cyborg then? | |
| 14:35:43 | zhipeng | we could use scheduling extention maybe for that | |
| 14:36:31 | zhipeng | but I doubt Google wants to have the k8s core doing scheduling that taking accelerators into consideration | |
| 14:37:46 | sundar | The Cyborg implementation could relate different resources. Agreed. The CRD discussions also seem to get into resource classes etc., which seem to be a long shot, as you said. Yes, agreed that scheduling core cannot be changed | |
| 14:37:46 | Li_Liu | but the scheduling extension still require some change in k8s main tree right? | |
| 14:38:10 | sundar | Li_Liu: scheduler extension is a standard mechanism in K8s today | |
| 14:38:21 | zhipeng | yep what sundar said | |
| 14:38:33 | sundar | However, the scheduler framework itself may evolve, and the extension APIs may change along with it | |
| 14:39:12 | sundar | Link to proposed K8s scheduler framework: https://docs.google.com/document/d/1NskpTHpOBWtIa5XsgB4bwPHz4IdRxL1RNvdjI7RVGio/edit# | |
| 14:40:45 | zhipeng | #link https://medium.com/@trstringer/create-kubernetes-controllers-for-core-and-custom-resources-62fc35ad64a3 | |
| 14:40:52 | zhipeng | some crd fundamentals | |
| 14:41:40 | shaohe | zhipeng: can not open it. | |
| 14:41:49 | zhipeng | so as I understand, crd is basically a way that we write a non-core k8s-ish controller | |
| 14:41:53 | zhipeng | shaohe you need vpn | |
| 14:42:07 | zhipeng | it listens upon the api-server | |
| 14:42:28 | zhipeng | and the keyword will trigger the request going to the crd controller, instead of the core k8s controller | |
| 14:42:40 | zhipeng | basically a hat on cyborg, if you will | |
| 14:44:11 | Li_Liu | so it's a subscribe/notify model right? | |
| 14:44:41 | zhipeng | in essence, as I understand yes | |
| 14:46:14 | NokMikeR | in go land | |
| 14:47:56 | zhipeng | yep :) | |
| 14:48:02 | NokMikeR | is it trival to use python from go or how does the cyborg api interaction work to k8 and go? | |
| 14:48:28 | zhipeng | we could have gRPC clients that abstract away the lang difference | |
| 14:48:45 | NokMikeR | ok | |
| 14:49:17 | NokMikeR | I need that for English to Finnish to English also :) | |
| 14:50:09 | zhipeng | haha | |
| 14:50:22 | zhipeng | you need google duplex for that :P | |
| 14:51:10 | NokMikeR | :) | |
| 14:51:17 | sundar | Exactly, gRPC -- as Zhipeng said. The controller is a separate daemon. Also, kubelet and Cyborg DP will also be separate processes. | |
| 14:52:34 | zhipeng | let's keep discussion alive offline :) | |
| 14:52:49 | zhipeng | #topic bugs and issues | |
| 14:53:26 | zhipeng | shaohe a colleague of mine report that when devstack starts, he could not find cyborg services | |
| 14:53:37 | zhipeng | have you encountered similar problem ? | |
| 14:55:28 | zhipeng | shaohe dropped i think | |
| 14:55:39 | zhipeng | okey let's move on to the next topic then | |
| 14:55:43 | NokMikeR | I reported the same problem many moons ago, but have not tried lately to install it | |
| 14:56:31 | zhipeng | NokMikeR i think during some of the past fixes it turns out ok | |
| 14:56:42 | zhipeng | i'm not sure if some of the recent patches breaks it | |
| 14:57:45 | NokMikeR | the mutable config page shows a failure? https://review.openstack.org/#/c/559303/ | |
| 14:58:08 | shaohe_ | ping | |