| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-18 | |||
| 21:37:10 | SpamapS | Ya, just wondering why people would think these would be useful enough for INFO. | |
| 21:37:12 | efried | SpamapS: Because I reckon you do want to know if something goes wrong. So you don't just want to blow away the logs. | |
| 21:37:45 | SpamapS | Yeah thats what I meant by dump them.. we'll not send the INFO+placement through to elastic. | |
| 21:38:11 | efried | SpamapS: For ffdc, being able to correlate an ERROR in the nova logs to the corresponding placement calls - that's useful. And INFO-worthy IMO. | |
| 21:38:47 | efried | again because placement has no way to distinguish what's ERROR-worthy from the client's perspective. | |
| 21:39:22 | efried | Also finding it a little hard to believe that the placement log is noisier than the combined compute logs. Keep in mind that there's only one placement service. | |
| 21:39:54 | SpamapS | the computes don't spray as much on INFO | |
| 21:40:06 | SpamapS | unless they're super busy with churn | |
| 21:40:43 | SpamapS | meanwhile they're constantly hammering on placement | |
| 21:40:48 | efried | yup | |
| 21:52:01 | jaypipes | SpamapS, jroll: do you have specific advice on what to down-level from INFO? | |
| 22:07:02 | Sundar | efried: Please ping me when you can | |
| 22:07:17 | efried | Sundar: I'm here. | |
| 22:07:35 | efried | Sundar: I skimmed your responses, but didn't dig deep yet. | |
| 22:08:02 | efried | oh, right, the main issue is "cyborg discovers automatically etc." | |
| 22:08:09 | efried | This is missing the point, I think. | |
| 22:08:40 | efried | When we do `openstack accelerator create` we're not getting a specific device. We're creating an ethereal meta-thingy that basically just has an ID for now. | |
| 22:09:15 | efried | It's not until after placement has done its thang that you can actually (with the help of the plugin) associate that with an actual (possibly-virtual) device. | |
| 22:10:02 | Sundar | efried: Sure, I am saying that Cyborg discovers PCI functions and such, apart from devices per se. So, all the information needed for an attach are already there. | |
| 22:10:17 | efried | Except it's not. | |
| 22:10:44 | Sundar | Could you clarify? What is missing? | |
| 22:10:53 | efried | The main purpose of `openstack accelerator create <config details>` is to give <config details> a handle so we don't have to sling a random dict around all the services. | |
| 22:11:16 | efried | It's the same principle as the port in neutron. You do a `port create` and get a UUID, but it's not associated with anything *real* yet. | |
| 22:11:36 | efried | It's only much later, once you've landed on a real host, that you actually get a port on a NIC with an IP or whatever. | |
| 22:11:42 | efried | Same thing here. | |
| 22:11:44 | Sundar | That can be done from Nova compute -> os-acc, as opposed t doing at the controller between n-sched/placement and Cyborg | |
| 22:12:28 | Sundar | The analogy with Neutron is ok, but do we need to copy it exactly? | |
| 22:12:41 | Sundar | We need to see what makes sense in this context, right? | |
| 22:12:52 | efried | heh, yeah, we should copy it insofar as it makes sense, and no further. | |
| 22:13:28 | Sundar | What goes wrong if we create an attach object when n-cpu calls os-acc? | |
| 22:13:48 | efried | well, what's an "attach object"? | |
| 22:13:58 | efried | But let me answer that anyway | |
| 22:14:13 | jaypipes | efried: it's pretty much the same as a "volume attachment". it's an ephemeral connection_info thing. | |
| 22:14:20 | efried | The problem is that n-cond needed to know things about that accelerator long before we ever get to n-cpu. | |
| 22:14:28 | efried | jaypipes: It was a semi-rhetorical question. | |
| 22:14:34 | jaypipes | efried: yes, I know :) | |
| 22:14:43 | Sundar | It is an object that encapsulates what it takes to attach an accelerator to a VM | |
| 22:14:46 | jaypipes | efried: as mine was a semi-rhetorical answer. | |
| 22:15:03 | jaypipes | efried: it's a non-consumable thing. it's just configuration info. | |
| 22:15:12 | jaypipes | efried: i.e. it isn't a placement concern. | |
| 22:15:20 | efried | Sundar: Example: I want my instance to have a VGPU with 1 display heads and resolution of 1024x768 (ambitious, I know). | |
| 22:15:48 | efried | Sundar: I need a way to have that information passed from the very start of my `server create` request. | |
| 22:16:17 | efried | So when I do `openstack server create --flavor foo --image bar --accelerator <what goes here?>` | |
| 22:16:29 | efried | And the answer could be <random dict of values> | |
| 22:16:33 | efried | but that sucks. | |
| 22:16:38 | jaypipes | efried: like the neutron port profile.. | |
| 22:16:45 | efried | jaypipes: precisely | |
| 22:16:53 | jaypipes | efried: which is used to created the instance pci device request | |
| 22:17:10 | Sundar | I am not an expert on GPUs by a long shot :) but can't you apply traits on your GPU indicating what resolutions it can support, and use those traits in placement to pick a device? | |
| 22:17:12 | efried | So instead we first do `openstack accelerator create --type vgpu --specs display_heads=1,resolution=1024x768`, or similar. | |
| 22:17:41 | Sundar | "--specs display_heads=1,resolution=1024x768`" --> why can't this be a trait? | |
| 22:17:45 | efried | And we get back an "accelerator UUID" Or maybe it's called an "attachment UUID". Whatever term, not important right now. Point is, it's just a handle for that config data. | |
| 22:17:52 | efried | Sundar: It *is*. That's the point. | |
| 22:17:59 | jaypipes | efried: it's >1 trait, to be clear. | |
| 22:18:15 | efried | Sundar: We're not *creating* anything in placement at this point in the game. That was already done by cyborg when you did your discovery and whatnot. | |
| 22:18:49 | efried | So now we got back some $acc_uuid and we can say `openstack server create --flavor foo --image bar --accelerator $acc_uuid` | |
| 22:19:31 | Sundar | efried, jaypipes: Since it is a trait, placement can choose a GPU device in a node on on that basis, and have n-cpu call os-acc to 'configure' the gpu as needed? | |
| 22:19:55 | efried | yes, Sundar, that's exactly what's going to happen. I'm talking through how placement is going to get that information. | |
| 22:19:58 | mriedem | SpamapS: jroll: efried: n-api also logs requests at INFO http://logs.openstack.org/66/362766/109/gate/tempest-full/52a4e60/controller/logs/screen-n-api.txt.gz#_Jun_15_15_26_18_719528 | |
| 22:20:55 | SpamapS | That makes sense. It just doesn't get as many requests. :) | |
| 22:20:56 | efried | Sundar: ...So now when the conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?) "hey, give me the details on this accelerator: $acc_uuid". Cyborg returns the AccGPU object, populated with an AccGPUProfile, which in turn is populated with information about display_heads=1 and resolution=1024x768 | |
| 22:21:32 | Sundar | efried: May be I am being dense. The placement knows enough to choose a RP (GPU in a node). What else does it need? The rest can be left to entities in the compute node, right? | |
| 22:21:33 | efried | conductor parlays that into querystring content for GET /allocation_candidates and sends it down to placement. | |
| 22:22:11 | efried | Sundar: Nope, placement knows what it's told. I need to tell it about more than just "I need one VGPU". Otherwise I have to do all my filtering later, which defeats the purpose of having used placement. | |
| 22:22:30 | Sundar | "conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?)" This is not needed for selecting a node | |
| 22:22:43 | efried | yes it is. | |
| 22:22:54 | Sundar | You can do GET /a-c without talking to Cyboirg | |
| 22:22:55 | efried | unless, as I say, you want to do all the filtering later, which defeats the purpose. | |
| 22:23:35 | efried | Sundar: n-cond had to get the config details of the accelerator from *somewhere*. | |
| 22:23:44 | efried | Sundar: Where did it get that from? | |
| 22:23:48 | Sundar | E.g. resources:ACCELERATOR_GPU=1; traits:RESOLUTION-1024-768=required (or whatever) | |
| 22:24:12 | efried | right, where does traits:RESOLUTION_1024X768=required come from? | |
| 22:24:27 | Sundar | CYborg (or whatever) published it | |
| 22:24:28 | efried | Sundar: Oh, this may be the source of confusion: are you thinking that comes from the flavor? | |
| 22:24:34 | Sundar | Yes | |
| 22:25:04 | efried | jaypipes, dansmith: Remind me why we're not putting accelerator stuff in flavors | |
| 22:25:22 | efried | (which I suppose may be the same reason we don't put neutron port stuff in flavors??) | |
| 22:25:28 | jaypipes | efried: we are putting that stuff in flavors and image metadata (at least, we are for *v*GPUs...) | |
| 22:26:25 | Sundar | jaypipes: +1 Same for FPGAs | |
| 22:26:39 | efried | well then | |
| 22:26:40 | jaypipes | efried: that, of course, is what is being *requested* by the user. something (Cyborg I was assuming) needs to decorate resource providers representing the physical accelerators with those traits. | |
| 22:26:57 | Sundar | Totally agreed | |
| 22:26:58 | efried | yeah, got that | |
| 22:27:01 | jaypipes | k | |
| 22:27:17 | Sundar | Great. I need to drop now for a 3:30 PM call | |
| 22:27:31 | Sundar | I am open to coming back tomorrow (or even later today) to close this | |
| 22:27:33 | efried | We need to get a consensus with dansmith. | |
| 22:27:46 | Sundar | OK, what's the best time for that? | |
| 22:27:58 | jroll | jaypipes: I don't have specific advice, I was just curious what clint was referring to and then tried to help clarify :) | |
| 22:27:58 | efried | Sundar: You don't need to be around for that, necessarily. | |
| 22:28:44 | Sundar | OK. Thanks, efried and jaypipes. Take care! | |
| 22:28:46 | efried | Sundar: I'll try to catch up with dansmith and see if it was just a misunderstanding or what. | |
| 22:29:49 | efried | jaypipes: FYI, this is the flow that dansmith and I talked through the other day: http://logs.openstack.org/45/575545/1/check/build-openstack-sphinx-docs/77f4d6a/html/specs/rocky/approved/nova-cyborg-flow.html | |
| 22:30:03 | efried | jaypipes: It closely follows the neutron port model with os-vif. | |
| 22:30:54 | jaypipes | jroll: k, no prob. was just curious if we should make a specific change. | |
| 22:32:49 | jaypipes | efried: I don't see *end users* ever doing `openstack acc create --type GPU <specs>`. Maybe some admin... maybe. though that said, I don't really know what <specs> would be. | |
| 22:33:11 | jaypipes | efried: end users (at least that I'm aware of) don't have the ability to create "accelerators". | |
| 22:33:12 | efried | jaypipes: same thing as whatever it would be in the flavor, more or less. | |
| 22:33:31 | jaypipes | efried: I guess I don't see why we need a separate service for that, but ok. | |
| 22:33:33 | efried | Okay, if this is all driven by the flavor, so be it. | |