Posted	Nick	Remark
#openstack-nova - 2018-06-18
21:37:12	efried	SpamapS: Because I reckon you do want to know if something goes wrong. So you don't just want to blow away the logs.
21:37:45	SpamapS	Yeah thats what I meant by dump them.. we'll not send the INFO+placement through to elastic.
21:38:11	efried	SpamapS: For ffdc, being able to correlate an ERROR in the nova logs to the corresponding placement calls - that's useful. And INFO-worthy IMO.
21:38:47	efried	again because placement has no way to distinguish what's ERROR-worthy from the client's perspective.
21:39:22	efried	Also finding it a little hard to believe that the placement log is noisier than the combined compute logs. Keep in mind that there's only one placement service.
21:39:54	SpamapS	the computes don't spray as much on INFO
21:40:06	SpamapS	unless they're super busy with churn
21:40:43	SpamapS	meanwhile they're constantly hammering on placement
21:40:48	efried	yup
21:52:01	jaypipes	SpamapS, jroll: do you have specific advice on what to down-level from INFO?
22:07:02	Sundar	efried: Please ping me when you can
22:07:17	efried	Sundar: I'm here.
22:07:35	efried	Sundar: I skimmed your responses, but didn't dig deep yet.
22:08:02	efried	oh, right, the main issue is "cyborg discovers automatically etc."
22:08:09	efried	This is missing the point, I think.
22:08:40	efried	When we do `openstack accelerator create` we're not getting a specific device. We're creating an ethereal meta-thingy that basically just has an ID for now.
22:09:15	efried	It's not until after placement has done its thang that you can actually (with the help of the plugin) associate that with an actual (possibly-virtual) device.
22:10:02	Sundar	efried: Sure, I am saying that Cyborg discovers PCI functions and such, apart from devices per se. So, all the information needed for an attach are already there.
22:10:17	efried	Except it's not.
22:10:44	Sundar	Could you clarify? What is missing?
22:10:53	efried	The main purpose of `openstack accelerator create <config details>` is to give <config details> a handle so we don't have to sling a random dict around all the services.
22:11:16	efried	It's the same principle as the port in neutron. You do a `port create` and get a UUID, but it's not associated with anything *real* yet.
22:11:36	efried	It's only much later, once you've landed on a real host, that you actually get a port on a NIC with an IP or whatever.
22:11:42	efried	Same thing here.
22:11:44	Sundar	That can be done from Nova compute -> os-acc, as opposed t doing at the controller between n-sched/placement and Cyborg
22:12:28	Sundar	The analogy with Neutron is ok, but do we need to copy it exactly?
22:12:41	Sundar	We need to see what makes sense in this context, right?
22:12:52	efried	heh, yeah, we should copy it insofar as it makes sense, and no further.
22:13:28	Sundar	What goes wrong if we create an attach object when n-cpu calls os-acc?
22:13:48	efried	well, what's an "attach object"?
22:13:58	efried	But let me answer that anyway
22:14:13	jaypipes	efried: it's pretty much the same as a "volume attachment". it's an ephemeral connection_info thing.
22:14:20	efried	The problem is that n-cond needed to know things about that accelerator long before we ever get to n-cpu.
22:14:28	efried	jaypipes: It was a semi-rhetorical question.
22:14:34	jaypipes	efried: yes, I know :)
22:14:43	Sundar	It is an object that encapsulates what it takes to attach an accelerator to a VM
22:14:46	jaypipes	efried: as mine was a semi-rhetorical answer.
22:15:03	jaypipes	efried: it's a non-consumable thing. it's just configuration info.
22:15:12	jaypipes	efried: i.e. it isn't a placement concern.
22:15:20	efried	Sundar: Example: I want my instance to have a VGPU with 1 display heads and resolution of 1024x768 (ambitious, I know).
22:15:48	efried	Sundar: I need a way to have that information passed from the very start of my `server create` request.
22:16:17	efried	So when I do `openstack server create --flavor foo --image bar --accelerator <what goes here?>`
22:16:29	efried	And the answer could be <random dict of values>
22:16:33	efried	but that sucks.
22:16:38	jaypipes	efried: like the neutron port profile..
22:16:45	efried	jaypipes: precisely
22:16:53	jaypipes	efried: which is used to created the instance pci device request
22:17:10	Sundar	I am not an expert on GPUs by a long shot :) but can't you apply traits on your GPU indicating what resolutions it can support, and use those traits in placement to pick a device?
22:17:12	efried	So instead we first do `openstack accelerator create --type vgpu --specs display_heads=1,resolution=1024x768`, or similar.
22:17:41	Sundar	"--specs display_heads=1,resolution=1024x768`" --> why can't this be a trait?
22:17:45	efried	And we get back an "accelerator UUID" Or maybe it's called an "attachment UUID". Whatever term, not important right now. Point is, it's just a handle for that config data.
22:17:52	efried	Sundar: It *is*. That's the point.
22:17:59	jaypipes	efried: it's >1 trait, to be clear.
22:18:15	efried	Sundar: We're not *creating* anything in placement at this point in the game. That was already done by cyborg when you did your discovery and whatnot.
22:18:49	efried	So now we got back some $acc_uuid and we can say `openstack server create --flavor foo --image bar --accelerator $acc_uuid`
22:19:31	Sundar	efried, jaypipes: Since it is a trait, placement can choose a GPU device in a node on on that basis, and have n-cpu call os-acc to 'configure' the gpu as needed?
22:19:55	efried	yes, Sundar, that's exactly what's going to happen. I'm talking through how placement is going to get that information.
22:19:58	mriedem	SpamapS: jroll: efried: n-api also logs requests at INFO http://logs.openstack.org/66/362766/109/gate/tempest-full/52a4e60/controller/logs/screen-n-api.txt.gz#_Jun_15_15_26_18_719528
22:20:55	SpamapS	That makes sense. It just doesn't get as many requests. :)
22:20:56	efried	Sundar: ...So now when the conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?) "hey, give me the details on this accelerator: $acc_uuid". Cyborg returns the AccGPU object, populated with an AccGPUProfile, which in turn is populated with information about display_heads=1 and resolution=1024x768
22:21:32	Sundar	efried: May be I am being dense. The placement knows enough to choose a RP (GPU in a node). What else does it need? The rest can be left to entities in the compute node, right?
22:21:33	efried	conductor parlays that into querystring content for GET /allocation_candidates and sends it down to placement.
22:22:11	efried	Sundar: Nope, placement knows what it's told. I need to tell it about more than just "I need one VGPU". Otherwise I have to do all my filtering later, which defeats the purpose of having used placement.
22:22:30	Sundar	"conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?)" This is not needed for selecting a node
22:22:43	efried	yes it is.
22:22:54	Sundar	You can do GET /a-c without talking to Cyboirg
22:22:55	efried	unless, as I say, you want to do all the filtering later, which defeats the purpose.
22:23:35	efried	Sundar: n-cond had to get the config details of the accelerator from *somewhere*.
22:23:44	efried	Sundar: Where did it get that from?
22:23:48	Sundar	E.g. resources:ACCELERATOR_GPU=1; traits:RESOLUTION-1024-768=required (or whatever)
22:24:12	efried	right, where does traits:RESOLUTION_1024X768=required come from?
22:24:27	Sundar	CYborg (or whatever) published it
22:24:28	efried	Sundar: Oh, this may be the source of confusion: are you thinking that comes from the flavor?
22:24:34	Sundar	Yes
22:25:04	efried	jaypipes, dansmith: Remind me why we're not putting accelerator stuff in flavors
22:25:22	efried	(which I suppose may be the same reason we don't put neutron port stuff in flavors??)
22:25:28	jaypipes	efried: we are putting that stuff in flavors and image metadata (at least, we are for *v*GPUs...)
22:26:25	Sundar	jaypipes: +1 Same for FPGAs
22:26:39	efried	well then
22:26:40	jaypipes	efried: that, of course, is what is being *requested* by the user. something (Cyborg I was assuming) needs to decorate resource providers representing the physical accelerators with those traits.
22:26:57	Sundar	Totally agreed
22:26:58	efried	yeah, got that
22:27:01	jaypipes	k
22:27:17	Sundar	Great. I need to drop now for a 3:30 PM call
22:27:31	Sundar	I am open to coming back tomorrow (or even later today) to close this
22:27:33	efried	We need to get a consensus with dansmith.
22:27:46	Sundar	OK, what's the best time for that?
22:27:58	jroll	jaypipes: I don't have specific advice, I was just curious what clint was referring to and then tried to help clarify :)
22:27:58	efried	Sundar: You don't need to be around for that, necessarily.
22:28:44	Sundar	OK. Thanks, efried and jaypipes. Take care!
22:28:46	efried	Sundar: I'll try to catch up with dansmith and see if it was just a misunderstanding or what.
22:29:49	efried	jaypipes: FYI, this is the flow that dansmith and I talked through the other day: http://logs.openstack.org/45/575545/1/check/build-openstack-sphinx-docs/77f4d6a/html/specs/rocky/approved/nova-cyborg-flow.html
22:30:03	efried	jaypipes: It closely follows the neutron port model with os-vif.
22:30:54	jaypipes	jroll: k, no prob. was just curious if we should make a specific change.
22:32:49	jaypipes	efried: I don't see *end users* ever doing `openstack acc create --type GPU <specs>`. Maybe some admin... maybe. though that said, I don't really know what <specs> would be.
22:33:11	jaypipes	efried: end users (at least that I'm aware of) don't have the ability to create "accelerators".
22:33:12	efried	jaypipes: same thing as whatever it would be in the flavor, more or less.
22:33:31	jaypipes	efried: I guess I don't see why we need a separate service for that, but ok.
22:33:33	efried	Okay, if this is all driven by the flavor, so be it.
22:33:58	efried	jaypipes: If we don't, we don't. I've never been a huge fan of the `port create` pre-step.

Earlier   Later