Posted	Nick	Remark
#openstack-nova - 2018-06-18
22:08:02	efried	oh, right, the main issue is "cyborg discovers automatically etc."
22:08:09	efried	This is missing the point, I think.
22:08:40	efried	When we do `openstack accelerator create` we're not getting a specific device. We're creating an ethereal meta-thingy that basically just has an ID for now.
22:09:15	efried	It's not until after placement has done its thang that you can actually (with the help of the plugin) associate that with an actual (possibly-virtual) device.
22:10:02	Sundar	efried: Sure, I am saying that Cyborg discovers PCI functions and such, apart from devices per se. So, all the information needed for an attach are already there.
22:10:17	efried	Except it's not.
22:10:44	Sundar	Could you clarify? What is missing?
22:10:53	efried	The main purpose of `openstack accelerator create <config details>` is to give <config details> a handle so we don't have to sling a random dict around all the services.
22:11:16	efried	It's the same principle as the port in neutron. You do a `port create` and get a UUID, but it's not associated with anything *real* yet.
22:11:36	efried	It's only much later, once you've landed on a real host, that you actually get a port on a NIC with an IP or whatever.
22:11:42	efried	Same thing here.
22:11:44	Sundar	That can be done from Nova compute -> os-acc, as opposed t doing at the controller between n-sched/placement and Cyborg
22:12:28	Sundar	The analogy with Neutron is ok, but do we need to copy it exactly?
22:12:41	Sundar	We need to see what makes sense in this context, right?
22:12:52	efried	heh, yeah, we should copy it insofar as it makes sense, and no further.
22:13:28	Sundar	What goes wrong if we create an attach object when n-cpu calls os-acc?
22:13:48	efried	well, what's an "attach object"?
22:13:58	efried	But let me answer that anyway
22:14:13	jaypipes	efried: it's pretty much the same as a "volume attachment". it's an ephemeral connection_info thing.
22:14:20	efried	The problem is that n-cond needed to know things about that accelerator long before we ever get to n-cpu.
22:14:28	efried	jaypipes: It was a semi-rhetorical question.
22:14:34	jaypipes	efried: yes, I know :)
22:14:43	Sundar	It is an object that encapsulates what it takes to attach an accelerator to a VM
22:14:46	jaypipes	efried: as mine was a semi-rhetorical answer.
22:15:03	jaypipes	efried: it's a non-consumable thing. it's just configuration info.
22:15:12	jaypipes	efried: i.e. it isn't a placement concern.
22:15:20	efried	Sundar: Example: I want my instance to have a VGPU with 1 display heads and resolution of 1024x768 (ambitious, I know).
22:15:48	efried	Sundar: I need a way to have that information passed from the very start of my `server create` request.
22:16:17	efried	So when I do `openstack server create --flavor foo --image bar --accelerator <what goes here?>`
22:16:29	efried	And the answer could be <random dict of values>
22:16:33	efried	but that sucks.
22:16:38	jaypipes	efried: like the neutron port profile..
22:16:45	efried	jaypipes: precisely
22:16:53	jaypipes	efried: which is used to created the instance pci device request
22:17:10	Sundar	I am not an expert on GPUs by a long shot :) but can't you apply traits on your GPU indicating what resolutions it can support, and use those traits in placement to pick a device?
22:17:12	efried	So instead we first do `openstack accelerator create --type vgpu --specs display_heads=1,resolution=1024x768`, or similar.
22:17:41	Sundar	"--specs display_heads=1,resolution=1024x768`" --> why can't this be a trait?
22:17:45	efried	And we get back an "accelerator UUID" Or maybe it's called an "attachment UUID". Whatever term, not important right now. Point is, it's just a handle for that config data.
22:17:52	efried	Sundar: It *is*. That's the point.
22:17:59	jaypipes	efried: it's >1 trait, to be clear.
22:18:15	efried	Sundar: We're not *creating* anything in placement at this point in the game. That was already done by cyborg when you did your discovery and whatnot.
22:18:49	efried	So now we got back some $acc_uuid and we can say `openstack server create --flavor foo --image bar --accelerator $acc_uuid`
22:19:31	Sundar	efried, jaypipes: Since it is a trait, placement can choose a GPU device in a node on on that basis, and have n-cpu call os-acc to 'configure' the gpu as needed?
22:19:55	efried	yes, Sundar, that's exactly what's going to happen. I'm talking through how placement is going to get that information.
22:19:58	mriedem	SpamapS: jroll: efried: n-api also logs requests at INFO http://logs.openstack.org/66/362766/109/gate/tempest-full/52a4e60/controller/logs/screen-n-api.txt.gz#_Jun_15_15_26_18_719528
22:20:55	SpamapS	That makes sense. It just doesn't get as many requests. :)
22:20:56	efried	Sundar: ...So now when the conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?) "hey, give me the details on this accelerator: $acc_uuid". Cyborg returns the AccGPU object, populated with an AccGPUProfile, which in turn is populated with information about display_heads=1 and resolution=1024x768
22:21:32	Sundar	efried: May be I am being dense. The placement knows enough to choose a RP (GPU in a node). What else does it need? The rest can be left to entities in the compute node, right?
22:21:33	efried	conductor parlays that into querystring content for GET /allocation_candidates and sends it down to placement.
22:22:11	efried	Sundar: Nope, placement knows what it's told. I need to tell it about more than just "I need one VGPU". Otherwise I have to do all my filtering later, which defeats the purpose of having used placement.
22:22:30	Sundar	"conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?)" This is not needed for selecting a node
22:22:43	efried	yes it is.
22:22:54	Sundar	You can do GET /a-c without talking to Cyboirg
22:22:55	efried	unless, as I say, you want to do all the filtering later, which defeats the purpose.
22:23:35	efried	Sundar: n-cond had to get the config details of the accelerator from *somewhere*.
22:23:44	efried	Sundar: Where did it get that from?
22:23:48	Sundar	E.g. resources:ACCELERATOR_GPU=1; traits:RESOLUTION-1024-768=required (or whatever)
22:24:12	efried	right, where does traits:RESOLUTION_1024X768=required come from?
22:24:27	Sundar	CYborg (or whatever) published it
22:24:28	efried	Sundar: Oh, this may be the source of confusion: are you thinking that comes from the flavor?
22:24:34	Sundar	Yes
22:25:04	efried	jaypipes, dansmith: Remind me why we're not putting accelerator stuff in flavors
22:25:22	efried	(which I suppose may be the same reason we don't put neutron port stuff in flavors??)
22:25:28	jaypipes	efried: we are putting that stuff in flavors and image metadata (at least, we are for *v*GPUs...)
22:26:25	Sundar	jaypipes: +1 Same for FPGAs
22:26:39	efried	well then
22:26:40	jaypipes	efried: that, of course, is what is being *requested* by the user. something (Cyborg I was assuming) needs to decorate resource providers representing the physical accelerators with those traits.
22:26:57	Sundar	Totally agreed
22:26:58	efried	yeah, got that
22:27:01	jaypipes	k
22:27:17	Sundar	Great. I need to drop now for a 3:30 PM call
22:27:31	Sundar	I am open to coming back tomorrow (or even later today) to close this
22:27:33	efried	We need to get a consensus with dansmith.
22:27:46	Sundar	OK, what's the best time for that?
22:27:58	efried	Sundar: You don't need to be around for that, necessarily.
22:27:58	jroll	jaypipes: I don't have specific advice, I was just curious what clint was referring to and then tried to help clarify :)
22:28:44	Sundar	OK. Thanks, efried and jaypipes. Take care!
22:28:46	efried	Sundar: I'll try to catch up with dansmith and see if it was just a misunderstanding or what.
22:29:49	efried	jaypipes: FYI, this is the flow that dansmith and I talked through the other day: http://logs.openstack.org/45/575545/1/check/build-openstack-sphinx-docs/77f4d6a/html/specs/rocky/approved/nova-cyborg-flow.html
22:30:03	efried	jaypipes: It closely follows the neutron port model with os-vif.
22:30:54	jaypipes	jroll: k, no prob. was just curious if we should make a specific change.
22:32:49	jaypipes	efried: I don't see *end users* ever doing `openstack acc create --type GPU <specs>`. Maybe some admin... maybe. though that said, I don't really know what <specs> would be.
22:33:11	jaypipes	efried: end users (at least that I'm aware of) don't have the ability to create "accelerators".
22:33:12	efried	jaypipes: same thing as whatever it would be in the flavor, more or less.
22:33:31	jaypipes	efried: I guess I don't see why we need a separate service for that, but ok.
22:33:33	efried	Okay, if this is all driven by the flavor, so be it.
22:33:58	efried	jaypipes: If we don't, we don't. I've never been a huge fan of the `port create` pre-step.
22:34:34	efried	I guess the difference is that every port needs to have specific *unique* config info (IP addr).
22:34:46	efried	...which doesn't make sense to be in a flavor.
22:36:25	jaypipes	efried: we're talking about different things entirely.
22:37:42	jaypipes	efried: I'll just leave this conversation like this... I don't see why Cyborg isn't the generic device manager...
22:37:56	efried	I fully hope it will be
22:38:03	jaypipes	in other words, I don't see the point of having `openstack acc $ANYTHING`.
22:38:20	efried	I'm on board with that, if dansmith is.
22:38:46	jaypipes	there should be a YAML inventory file describing devices (that can't for some reason be auto-discovered on a node) and that's that...
22:39:23	efried	The way the conversation with Sundar started was that his specs talk about os-acc parallelling os-vif, but then doesn't actually do that.
22:39:50	jaypipes	note that I didn't say that os-acc shouldn't exist.
22:40:40	efried	I wanted os-acc to have Acc* objects (corresponding to VIF*) and Acc*Profile (=~ VIF*PortProfile); and to define plug/unplug methods which call to plugins registered/loaded stevedore-wise.
22:40:49	efried	they didn't have any of that in their specs.
22:41:48	jaypipes	efried: but I don't see the point of a Cyborg REST API. sorry if that offends anyone. I just really only see the point of a utility library (os-acc) that can be called to configure/init a particular device slot.

Earlier   Later