| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-05-09 | |||
| 14:19:39 | zhipeng | let's wait for a week for feedback | |
| 14:20:00 | sundar | Thanks, Li_Liu and shaohe :). As zhipeng says, one week from now, we will know. | |
| 14:20:06 | zhipeng | but would like to thanks for Sundar's great effort so far | |
| 14:20:12 | zhipeng | :) | |
| 14:20:16 | zhipeng | okey moving on | |
| 14:20:41 | zhipeng | #topic KubeCon feedback | |
| 14:20:58 | zhipeng | okey last week i attend KubeCon and the resource mgmt wg deep dive session | |
| 14:21:34 | zhipeng | k8s res mgmt wg is the center in k8s which deals with general and acceleration resources | |
| 14:22:10 | zhipeng | My takeaway is that the support for a general accelerator mgmt, is still not in any shape in k8s | |
| 14:22:23 | zhipeng | Google is interested in GPU passthrough support for ML, mainly | |
| 14:23:32 | zhipeng | so if anyone wants to introduce any feature in-tree | |
| 14:24:26 | zhipeng | that would require a PoC up-front | |
| 14:24:59 | zhipeng | many things we discussed here, like vGPU types, general accelerator support including FPGA and others | |
| 14:25:05 | zhipeng | are viewed non-priority | |
| 14:25:22 | zhipeng | the resource class/resource api PRs are also long shot | |
| 14:25:28 | zhipeng | according to vishnu | |
| 14:27:29 | sundar | zhipeng: Agreed with your assessments. If we can get just one feature in -- passing annotations to the device plugin API -- that will help us meet most basic FPGA use cases, IMHO | |
| 14:27:55 | zhipeng | Sasha mentioned Intel team has finished a FPGA DPI PoC | |
| 14:28:04 | zhipeng | but also just pre-programmed FPGAs | |
| 14:28:39 | sundar | We can couple that with a scheduler extension. However, scheduler extensions are not viewed favorably because the scheduler fra,ework itself may change and the APIs may change along with it. | |
| 14:29:00 | zhipeng | yes, and also DPI is designed at the node level | |
| 14:29:09 | sundar | However, we can do a POC base don it, including programming support, and revamp it when the APIs change. Just my thought. :) | |
| 14:29:15 | zhipeng | and a mostly "reschedule" focused mechanism | |
| 14:29:36 | sundar | Yes, we do have a POC that does only pre-programmed use case. That does not show the strength of FPGAs, which is reprogramming | |
| 14:30:01 | zhipeng | so DPI is designed to mostly work for hot-plug use case, not scheduling upfront | |
| 14:30:22 | zhipeng | the scheduling will be retriggered once the node discover the DPI Plugin | |
| 14:30:35 | zhipeng | anyways it is the current lay of land | |
| 14:31:04 | zhipeng | so my thinking is, maybe it is reasonable to introduce a CRD framework for cyborg into k8s community | |
| 14:31:35 | zhipeng | so that we could have all of our data model preserved, has leeway on the api and scheduling design | |
| 14:31:46 | zhipeng | maintain a k8s-ish API interface | |
| 14:32:12 | zhipeng | and a out-of-band general accelerator mgmt functionality, not bound to DPI development | |
| 14:32:42 | zhipeng | i don't know what other team member's thought on this matter ? | |
| 14:32:44 | sundar | The CRD framework does not allow AFAIK for a nested topology, which OpenStack supports. | |
| 14:33:02 | zhipeng | CRD is just an API mechinism right ? | |
| 14:33:06 | NokMikeR | needs more blinking lights | |
| 14:33:12 | zhipeng | not implementation specific | |
| 14:33:56 | sundar | Yes. How do we model regions inside FPGAs, accelerators inside regions, local memory inside either ... | |
| 14:34:10 | zhipeng | that could all be done in Cyborg | |
| 14:35:18 | zhipeng | for example if you look at kubernetes service catalog | |
| 14:35:25 | Li_Liu | k8s will do the scheduling for Cyborg then? | |
| 14:35:43 | zhipeng | we could use scheduling extention maybe for that | |
| 14:36:31 | zhipeng | but I doubt Google wants to have the k8s core doing scheduling that taking accelerators into consideration | |
| 14:37:46 | sundar | The Cyborg implementation could relate different resources. Agreed. The CRD discussions also seem to get into resource classes etc., which seem to be a long shot, as you said. Yes, agreed that scheduling core cannot be changed | |
| 14:37:46 | Li_Liu | but the scheduling extension still require some change in k8s main tree right? | |
| 14:38:10 | sundar | Li_Liu: scheduler extension is a standard mechanism in K8s today | |
| 14:38:21 | zhipeng | yep what sundar said | |
| 14:38:33 | sundar | However, the scheduler framework itself may evolve, and the extension APIs may change along with it | |
| 14:39:12 | sundar | Link to proposed K8s scheduler framework: https://docs.google.com/document/d/1NskpTHpOBWtIa5XsgB4bwPHz4IdRxL1RNvdjI7RVGio/edit# | |
| 14:40:45 | zhipeng | #link https://medium.com/@trstringer/create-kubernetes-controllers-for-core-and-custom-resources-62fc35ad64a3 | |
| 14:40:52 | zhipeng | some crd fundamentals | |
| 14:41:40 | shaohe | zhipeng: can not open it. | |
| 14:41:49 | zhipeng | so as I understand, crd is basically a way that we write a non-core k8s-ish controller | |
| 14:41:53 | zhipeng | shaohe you need vpn | |
| 14:42:07 | zhipeng | it listens upon the api-server | |
| 14:42:28 | zhipeng | and the keyword will trigger the request going to the crd controller, instead of the core k8s controller | |
| 14:42:40 | zhipeng | basically a hat on cyborg, if you will | |
| 14:44:11 | Li_Liu | so it's a subscribe/notify model right? | |
| 14:44:41 | zhipeng | in essence, as I understand yes | |
| 14:46:14 | NokMikeR | in go land | |
| 14:47:56 | zhipeng | yep :) | |
| 14:48:02 | NokMikeR | is it trival to use python from go or how does the cyborg api interaction work to k8 and go? | |
| 14:48:28 | zhipeng | we could have gRPC clients that abstract away the lang difference | |
| 14:48:45 | NokMikeR | ok | |
| 14:49:17 | NokMikeR | I need that for English to Finnish to English also :) | |
| 14:50:09 | zhipeng | haha | |
| 14:50:22 | zhipeng | you need google duplex for that :P | |
| 14:51:10 | NokMikeR | :) | |
| 14:51:17 | sundar | Exactly, gRPC -- as Zhipeng said. The controller is a separate daemon. Also, kubelet and Cyborg DP will also be separate processes. | |
| 14:52:34 | zhipeng | let's keep discussion alive offline :) | |
| 14:52:49 | zhipeng | #topic bugs and issues | |
| 14:53:26 | zhipeng | shaohe a colleague of mine report that when devstack starts, he could not find cyborg services | |
| 14:53:37 | zhipeng | have you encountered similar problem ? | |
| 14:55:28 | zhipeng | shaohe dropped i think | |
| 14:55:39 | zhipeng | okey let's move on to the next topic then | |
| 14:55:43 | NokMikeR | I reported the same problem many moons ago, but have not tried lately to install it | |
| 14:56:31 | zhipeng | NokMikeR i think during some of the past fixes it turns out ok | |
| 14:56:42 | zhipeng | i'm not sure if some of the recent patches breaks it | |
| 14:57:45 | NokMikeR | the mutable config page shows a failure? https://review.openstack.org/#/c/559303/ | |
| 14:58:08 | shaohe_ | ping | |
| 14:58:23 | zhipeng | hey welcome back | |
| 14:58:25 | NokMikeR | pong | |
| 14:58:32 | zhipeng | NokMikeR that should be already fixed | |
| 14:58:38 | NokMikeR | ok | |
| 15:00:09 | zhipeng | so the specific problem is c-cond and c-agent are not running | |
| 15:00:24 | zhipeng | shaohe_ that is not normal right / | |
| 15:00:55 | shaohe_ | yes. | |
| 15:01:23 | zhipeng | if devstack succeed | |
| 15:01:38 | zhipeng | c-api, c-cond, c-agent should all be running right ? | |
| 15:01:38 | shaohe_ | devstack should report error, if c-cond and c-agent are not running | |
| 15:02:24 | zhipeng | ok i will contact the author | |
| 15:02:28 | zhipeng | okey moving on | |
| 15:03:18 | zhipeng | #topic spec review day | |
| 15:03:28 | shaohe_ | it should be cyborg-agent, cyborg-api, cyborg-cond | |
| 15:03:38 | zhipeng | nova and other team all have this custom of spec sprint, or runways | |
| 15:03:43 | zhipeng | let's have one as well | |
| 15:03:51 | zhipeng | shaohe_ ok I will let him know | |
| 15:04:08 | zhipeng | #link https://etherpad.openstack.org/p/cyborg-rocky-spec-day | |
| 15:04:12 | shaohe_ | is c-xxx cinder process? | |
| 15:04:25 | zhipeng | maybe he just referred wrong | |
| 15:04:31 | zhipeng | I should double check with him | |