Posted	Nick	Remark
#openstack-cyborg - 2018-04-25
14:23:05	Sundar	NokMikeR, the discussions I have seen have been centered on Nvidia's vGPU types, though not necessarily phrased in vendor-specific terms
14:23:15	NokMikeR	in other words how do you differentie the features on one vGPU vs another if the underlying features in the real gpu are different - or are they abstracted somehow?
14:23:20	Sundar	Li_Liu, None that I am aware of.
14:24:14	NokMikeR	Sundar: ok thought so re: nvid gpus.
14:24:35	Sundar	NokMikeR, in OpenStack, the answer is clearer: the device itself exposes different vGPU types as traits, and their capacities as units of a generic accelerator RC
14:25:44	Sundar	Cyborg needs to handle GPUs and FPGAs of course. But, IMHO, there is enough attention on GPUs :) It is FPGAs that need further thought :)
14:25:57	zhipeng	Sundar Jiaying's proposal, as far as I understand, still tries to modify the k8s core functionality ?
14:26:37	shaohe_feng_	Sundar: do we decide to only support one vendor GPU or FPGA in this release without nest Provider?
14:26:42	Sundar	Zhipeng, yes, it requires changes on controller side and kubelet changes
14:28:15	Sundar	Shaohe: for Rocky, I was proposing to include only device of a particualr type: one GPU or one FPGA. But, based on feedback, w ehave to relax it to multiple devices of the same type, i.e.,
14:28:29	Sundar	you could have 2 GPUs of the same type, 2 FPGAs of the same type etc.
14:28:51	zhipeng	Sundar if we say, propose a CRD type of kube-cyborg thing, will it make sense to the res mgmt wg people ?
14:29:00	zhipeng	meaning that similar to OpenStack
14:29:10	zhipeng	we view accelerator not part of the general compute infra
14:29:46	zhipeng	and have its own model and scheduling process if needed
14:30:25	Sundar	ZHipeng, we can propose CRDs, but the exact workflows will matter.
14:30:49	Li_Liu	zhipeng, you are saying, similar to what we did to Cyborg, we cut a piece out from K8S?
14:30:51	zhipeng	I think the main pain point is still at scheduler extention
14:31:07	zhipeng	which Derek also mentioned KubeCon last Dec
14:31:31	zhipeng	Li_Liu essentially a out-of-band controller for accelerators
14:31:43	shaohe_feng_	Sundar: will cyborg support nest provider in Rocky release?
14:32:10	zhipeng	shaohe_feng_ I think Placement won't support it
14:32:29	shaohe_feng_	zhipeng: Got it.
14:32:40	zhipeng	but the way we are modeling it is very close to nrp, correct me if i'm wrong Li_Liu
14:32:43	Sundar	Zhipeng, I was also advocating a scheduler extension. But apparently it is not popular within the community. There is a proposal to revamp the scheduler itself: https://docs.google.com/document/d/1NskpTHpOBWtIa5XsgB4bwPHz4IdRxL1RNvdjI7RVGio/edit#
14:32:58	Sundar	So, the scheduler, as well as its extension APIs, may change
14:34:22	zhipeng	well CRDs are generally great for API aggregation, but complex for resource related functionalities
14:34:32	Sundar	Here is a possible way to get to a few basic cases without anything fancy (this is not fully agreed upon, please take this as an option, not a plan):
14:34:37	zhipeng	like binding the resource to the pod
14:34:46	zhipeng	since the process is external via CRD
14:34:46	Sundar	•Publish each region type as a resource. E.g. intel.com/fpga-dcp, intel.com/fpga-vg. •The pod spec asks for a region type as a resource, and also specifies a bitstream ID. That could be a label. •An admission controller inserts an init container on seeing a FPGA resource. •The scheduler picks a node based on the requested region type (and ignores the bitstream ID). •The init container pulls the bitstream wi
14:35:07	Sundar	Ah, that didn't come out well in IRC -- let me re-type
14:35:24	Sundar	•Publish each region type as a resource. E.g. intel.com/fpga-dcp, intel.com/fpga-vg
14:35:34	Sundar	•The pod spec asks for a region type as a resource, and also specifies a bitstream ID. That could be a label
14:35:45	Sundar	•An admission controller inserts an init container on seeing a FPGA resource.
14:35:59	Sundar	•The scheduler picks a node based on the requested region type (and ignores the bitstream ID).
14:36:10	Sundar	•The init container pulls the bitstream with that ID from a bitstream repository (mechanism TBD) and programs the selected device.
14:37:20	Sundar	I have heard that, if we give a higher security context to the init container for programming, it may affect other containers in the same pod. I am still trying to find evidence for that
14:37:46	zhipeng	lol this is just too complicated
14:38:07	zhipeng	thx Sundar I think we have a good understanding of the status qup
14:38:10	zhipeng	quo
14:38:18	Sundar	Zhipeng, more complicated than other proposals out there? ;)
14:38:25	zhipeng	anyone else got questions regarding k8s ?
14:38:54	zhipeng	if you are attending KubeCon we could meet f2f, and give them hell XD
14:39:09	Sundar	lol
14:41:04	Li_Liu	if you guys have any dial-in-able meeting during kubecon, please loop us in
14:41:36	shaohe_feng_	yes, loop us in
14:42:02	zhipengh[m]	Okey I will give a howler if a bridge is available
14:42:33	zhipengh[m]	Seems like my PC irc client just died
14:43:05	zhipengh[m]	#topic Sub team arrangements
14:46:08	zhipeng	phew
14:46:13	Sundar	:)
14:46:41	zhipeng	cell phone irc bouncer crashed just now
14:46:57	zhipeng	moving on
14:47:05	zhipeng	#topic subteam arrangments
14:47:35	zhipeng	okey so given recent events, I think it is necessary to reorg the subteams
14:48:23	zhipeng	and also encourage subteam to organize their specific meetings
14:48:30	zhipeng	for specific topics
14:48:45	zhipeng	so I would suggest shaohe to help lead the driver subteam
14:49:10	zhipeng	work with our Xilinx and lenovo colleagues on FPGA and GPU driver in Rocky
14:49:29	shaohe_feng_	Ok.
14:49:40	zhipeng	Li Liu help lead the doc team, to work with our CMCC member and others to make documentation as good as your spec :)
14:49:49	Li_Liu	sure
14:49:59	zhipeng	I will keep on the release mgmt side
14:50:35	zhipeng	shaohe_feng_ you can sync up with Chuck_ on a meeting time more suited for US west coast
14:50:55	zhipeng	mainly China morning times I guess
14:51:13	shaohe_feng_	zhipeng: what is Chuck_?
14:51:28	Chuck_	I am Chuck_ :-)
14:51:37	Chuck_	Hi Shaohe, this is Chuck from Xilinx
14:51:44	Sundar	lol
14:51:46	shaohe_feng_	Chuck_: hello
14:51:46	Chuck_	I work from US west time
14:51:54	zhipeng	I will add Chuck_ into our wechat group as well
14:52:13	zhipeng	talk in Chinese :)
14:52:47	Li_Liu	count me in for those driver meeting shaohe_feng_
14:52:52	Li_Liu	:)
14:52:53	Chuck_	yes, look forward to working with you.
14:53:20	shaohe_feng_	Li_Liu: OK.
14:53:21	zhipeng	and subteam plz send report to the mailing list, you can decide whether it is bi-weekly or weekly
14:53:26	zhipeng	or monthly even
14:53:28	zhipeng	up to you
14:53:56	Sundar	Is it all WeChat in Chinse then? ;) I can join if that helps
14:54:10	zhipeng	it is for the China region devs :P
14:54:20	zhipeng	all in Chinese and crazy emoticons :P
14:54:24	Li_Liu	WeChat has a translation feature tho :)
14:54:39	shaohe_feng_	Sundar: we speak Chinese there. :)
14:54:53	shaohe_feng_	Sundar: you can learn Chinese.
14:55:24	Sundar	OK :) My daughters learnt Mandarin. I should have joined them
14:55:55	zhipeng	haha will learn a lot
14:56:34	Sundar	:) Do we have alignment on what use cases we will deliver in Rocky?
14:57:11	zhipeng	CERN HPC PoC could be the one for GPU
14:57:12	Sundar	Can we say we will deliver AFaaS pre-programmed, and FPGA aaS with request time programming? These are the simplest ones, and many customers want that
14:57:36	Sundar	Zhipeng, yes, GPU POC too
14:57:41	zhipeng	Sundar that is something we should be able to deliver
14:57:47	zhipeng	for FPGA
14:57:47	shaohe_feng_	Sundar: oh, do we will have same RC name for VGPU and FPGA, and other accelerators in Rocky release?
14:58:49	Sundar	Shaohe, yes our agreemnt with Nova is to use a generic RC for all accelerators
14:59:04	Sundar	as in the spec
14:59:21	Sundar	DO we have kosamara here?
14:59:34	Sundar	Any input on the spec from GPU perspective?

Earlier   Later