Posted	Nick	Remark
#openstack-cyborg - 2018-09-26
20:40:09	efried	...and if you want to reserve/restrict one, you have to explicitly blacklist it.
20:40:42	edmondsw	yeah, that's probably not a great idea
20:41:03	Sundar	efried, edmondsw: If some of those devices come pre-installed in the physical servers, like built-in NICs, I can understand the need to blacklist by default. That is not the case for accelerators in the data center.
20:41:57	efried	I mean, I'm having a tough time coming up with a practical scenario where it could cause you a real problem, it just feels wrong.
20:42:43	edmondsw	do we think there will never be a case where it makes sense for an accelerator to be owned by the hypervisor itself?
20:43:24	Sundar	edmondsw: Yes. That could be an explicit Cyborg configuration too.
20:45:18	Sundar	Isn't one of the pain points of PCI whitelisting that there is so much operator labor required? That may be unavoidable for common elements like NICs, but hopefully can be avoided in most cases for accelerators.
20:45:59	edmondsw	but accelerators are going to become more and more common
20:46:54	edmondsw	could we compromise with a conf setting that allows the user to specify whether they want whitelist-by-default behavior or not?
20:47:59	Sundar	edmondsw: That seems reasonable. That is a single setting for an entire deployment.
20:49:13	Sundar	We still need that to coexist with Nova's PCI whitelists. The accelerator's PCI ID may be in te white list, due to operator error, or because an upgrade moved the device from Nova to Cyborg, or whatever
20:49:59	Sundar	We can help that by providing a tool that the operator runs, wherein he feeds the Nova's PCI whitelist file and the output says which PCI IDs are in conflict
20:50:23	Sundar	edmondsw, efried: What do you think?
20:51:45	efried	Sundar: That sounds like an ambitious idea, but I'm sure it would be welcomed if you could pull it off :)
20:52:25	efried	TBC, coexisting with the nova PCI whitelist in the sense that the set of devices managed by each are mutually exclusive.
20:52:30	edmondsw	I'm not sure every conflict would be an error... but they could flick that conf setting to *not* whitelist-by-default in cyborg if that's the case
20:53:18	edmondsw	I put that badly... I meant it wouldn't necessarily be a nova error... could be that they need to blacklist it in cyborg
20:54:07	Sundar	edmondsw: Sure.
20:55:08	Sundar	efried: Do you see practical issues with such a tool? It seems to me that the tool is merely scanning CYborg's db for PCI IDs and the operator-provided PCI whitelist file for conflicts. I suppose wild cards need careful handling
20:55:36	efried	heh, yeah, good luck duplicating the logic nova uses to process that whitelist.
20:57:29	Sundar	efried: I'm probably being naive. How does Nova handle conflicts within the whitelist? Say one entry enables a device and another blacklists it.
20:57:54	efried	Sundar: Don't know off the top of my head.
20:58:23	Sundar	efried: NP. I nly intend this to be a helper tool, not some official way that is fool-proof
20:58:49	efried	Here are the notes I scribbled down a couple years ago when I was pawing through the code trying to understand how it's "working": https://etherpad.openstack.org/p/powervm-pci-passthrough-notes
20:59:40	efried	gosh, maybe it was only a year ago
21:02:43	Sundar	efried: Thanks for the notes.
21:02:47	Sundar	efried, edmondsw: Thanks for the good feedback. :)
#openstack-cyborg - 2018-09-27
03:14:01	openstackgerrit	Xinran WANG proposed openstack/cyborg master: Allocation/Deallocation API Specification https://review.openstack.org/597991
19:11:04	Sundar	efried: Please LMK when you are here.
19:11:18	efried	Sundar: ō/
19:11:55	Sundar	efried: On the Nova spec, you correctly noted that I am not asking to include acceN in Placement queries
19:12:21	Sundar	These additional properties are needed to specify bitstreams, functions and things that Nova doesn't care about
19:12:42	efried	I understand
19:12:49	Sundar	So, I have proposed that they be bundled with the request groups in the same way as resources or traots
19:13:29	Sundar	So, if we stick to that idea, do you expect pushback from other Nova developers? What does it take to get this through?
19:14:06	efried	I could see some people objecting to the fact that it *looks* like they're tied together
19:14:10	efried	which, of course, they are
19:15:19	efried	I think you're going to have the same problem gibi is trying to solve via https://review.openstack.org/#/c/597601/ -- there's no way to map a request group to a piece of an allocation.
19:15:45	efried	But that's orthogonal to the design decision of using accN to denote such information.
19:17:13	Sundar	The alternative is to split the device profile entries in each VAR and store it in Cyborg. The problem then is that the information gets duplicated. That can lead to issues with consistency.
19:17:49	efried	Which information gets duplicated?
19:18:20	efried	you mean include stuff like bitstream info in the device profile?
19:18:29	efried	that seems... reasonable.
19:18:34	efried	at least for phase 1
19:19:29	Sundar	For e.g., say the device profile says: { resources:ACCEL_GPU=2; trait:GPU_FOO=required}. Then we would create 2 separate VARs. Each one could contain a copy like: {resources: ACCEL_GPU=1, trait:GPU_FOO=required}
19:19:39	Sundar	So, there are 2 VARs, each requesting 1 resource
19:20:09	efried	yes, that's as it should be. One VAR, one accelerator.
19:20:17	Sundar	That means, the resources/traits go duplicated. Also, the semantics that both resources need to come from the same resource provider is lost from Cyborg's perspective
19:20:24	Sundar	*got
19:20:36	efried	So
19:20:53	efried	I was thinking a device profile would be for *one* device.
19:20:54	efried	But
19:21:17	efried	then you can't, as you say, demand that two devices come from the same provider
19:21:28	efried	or more generally, take advantage of granular syntax.
19:21:34	efried	unless
19:22:02	Sundar	Yes. Nova would have that info, and presumably can colocate them. But, if VCyborg needs that info to connect the accelerators together, or whatever, that is ruled out
19:22:03	efried	we number the extra spec key using the same pattern as we do for the other resources
19:22:12	efried	which is actually probably a better, more composable design.
19:22:18	Sundar	Yes!
19:22:24	Sundar	+1
19:22:38	Sundar	That is the proposal in the spec
19:22:48	efried	it is?
19:23:02	efried	I got the impression you had set up device profiles so they could contain more than one accelerator
19:23:29	Sundar	Yes, I propose to take the device profile fields, cobvert them to extra spec granular syntax and fold them in
19:23:51	Sundar	Yes, in the above example, we would convert that to:
19:24:12	Sundar	resources2:ACCEL_GPU=2; trait2:GPU_FOO=required
19:24:49	Sundar	The only sticking point is who does that numbering. I thought Cyborg or os-acc could do that. But the comments point out that there could be request groups in the flavor unrelated to device profiles or Cybirg
19:24:59	Sundar	So, Nova should do the numbering. I am fine with that
19:25:31	efried	Yeah, you should ask gibi how he did that for the network bandwidth PoC. Pretty sure he would have done it from the nova thing that parses the extra specs.
19:26:17	Sundar	OK. I'll take a look at the spec you cited above, and ask him.
19:26:21	Sundar	Thanks!
19:27:56	Sundar	efried: This spec seems orthogonal to our needs, Do you have pointers to other specs from him on this topic?
19:28:06	efried	Sundar: Here's where I suspect that is getting done: https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/scheduler/utils.py#L433
19:29:35	efried	Oh, the spec I cited above has nothing to do with renumbering request groups. That was just acking the fact that you're (probably) going to need a way to map a numbered group in the *request* to some piece of the allocation in the *response*. And placement has no way right now to help you with that - you basically have to reverse-engineer it yourself.
19:31:28	efried	Let's say for example that your user wants two accelerators of the same type (say VGPU), doesn't care if they're from the same provider or not, and also wants to associate some other resource with each - let's say VRAM.
19:31:58	efried	So the user specifies one dev profile with VGPU=1,VRAM=1024 and another with VGPU=1,VRAM=2048
19:32:21	efried	placement, in its infinite wisdom, decides to satisfy both accelerators from the same provider
19:32:41	efried	so you get back an allocation with: VGPU=2,VRAM=3072
19:33:17	efried	And now you have to reverse-engineer that to figure out that you actually wanted one VGPU=1,VRAM=1024 and one VGPU=1,VRAM=2048
19:33:46	efried	Simple example, but add in more device profiles at the same time and/or try to generalize that reverse engineering, and it quickly becomes problematic.
19:34:03	efried	That's what gibi proposed his spec for.
19:34:14	Sundar	Yea, I got that
19:34:30	efried	Because in placement, *while* we're calculating allocation candidates (but not after), we know which request group corresponds to which bit of each allocation request.
19:34:57	efried	We just don't carry that information forward to the response, or preserve it internally in any way.
19:34:58	Sundar	A related issue is that, if VRAM were not a standard resource, you may say accel:VRAM=2048 or something
19:35:33	efried	um, rather not. If it's not a standard resource, it can be CUSTOM_VRAM, but it should still be tracked in placement.
19:35:51	Sundar	Now, the accel extra spec has to go with the rest of the request group, for the device to be suitably configured during bind
19:36:01	efried	If not, you're setting yourself up for resource contention problems that placement was expressly design to obviate.
19:37:26	Sundar	OK, that was not a good example. What if I wnat to assign the GPU to host software like DPDK for cryptography, rather than to a VM? Then the dev prof may say accel:assign-to-host=true
19:38:43	Sundar	This is not a trait
19:38:52	Sundar	because it is not meant for scheduling,
19:39:02	Sundar	only to control how the assignment is done
19:39:20	efried	I don't understand that use case. But yeah, traits are another useful way to expose the problem.
19:40:52	efried	If I ask for resource1=VGPU:1&required1=CUSTOM_FOO&resource2=VGPU:1, and I get back two allocations from different providers, I need to go figure out (based on traits? or what I already know about the providers?) which VGPU I'm supposed to apply to CUSTOM_FOO.
19:41:30	efried	again, it's not sooper hard, but once you generalize it out, you end up essentially having to duplicate a ton of the placement logic.
19:42:03	Sundar	Yes
19:43:48	Sundar	OK, it still comes down to how gibi solved this problem, I suppose. I'll dig around.
19:44:04	efried	*this* problem I don't think gibi solved.
19:44:08	efried	He solved the renumbering problem.
19:44:50	efried	I believe his PoC behaved quite similarly to what you're proposing - the resources/required gizmos live in the profile (the binding profile? or port? in his case)

Earlier   Later