| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-18 | |||
| 22:14:34 | jaypipes | efried: yes, I know :) | |
| 22:14:43 | Sundar | It is an object that encapsulates what it takes to attach an accelerator to a VM | |
| 22:14:46 | jaypipes | efried: as mine was a semi-rhetorical answer. | |
| 22:15:03 | jaypipes | efried: it's a non-consumable thing. it's just configuration info. | |
| 22:15:12 | jaypipes | efried: i.e. it isn't a placement concern. | |
| 22:15:20 | efried | Sundar: Example: I want my instance to have a VGPU with 1 display heads and resolution of 1024x768 (ambitious, I know). | |
| 22:15:48 | efried | Sundar: I need a way to have that information passed from the very start of my `server create` request. | |
| 22:16:17 | efried | So when I do `openstack server create --flavor foo --image bar --accelerator <what goes here?>` | |
| 22:16:29 | efried | And the answer could be <random dict of values> | |
| 22:16:33 | efried | but that sucks. | |
| 22:16:38 | jaypipes | efried: like the neutron port profile.. | |
| 22:16:45 | efried | jaypipes: precisely | |
| 22:16:53 | jaypipes | efried: which is used to created the instance pci device request | |
| 22:17:10 | Sundar | I am not an expert on GPUs by a long shot :) but can't you apply traits on your GPU indicating what resolutions it can support, and use those traits in placement to pick a device? | |
| 22:17:12 | efried | So instead we first do `openstack accelerator create --type vgpu --specs display_heads=1,resolution=1024x768`, or similar. | |
| 22:17:41 | Sundar | "--specs display_heads=1,resolution=1024x768`" --> why can't this be a trait? | |
| 22:17:45 | efried | And we get back an "accelerator UUID" Or maybe it's called an "attachment UUID". Whatever term, not important right now. Point is, it's just a handle for that config data. | |
| 22:17:52 | efried | Sundar: It *is*. That's the point. | |
| 22:17:59 | jaypipes | efried: it's >1 trait, to be clear. | |
| 22:18:15 | efried | Sundar: We're not *creating* anything in placement at this point in the game. That was already done by cyborg when you did your discovery and whatnot. | |
| 22:18:49 | efried | So now we got back some $acc_uuid and we can say `openstack server create --flavor foo --image bar --accelerator $acc_uuid` | |
| 22:19:31 | Sundar | efried, jaypipes: Since it is a trait, placement can choose a GPU device in a node on on that basis, and have n-cpu call os-acc to 'configure' the gpu as needed? | |
| 22:19:55 | efried | yes, Sundar, that's exactly what's going to happen. I'm talking through how placement is going to get that information. | |
| 22:19:58 | mriedem | SpamapS: jroll: efried: n-api also logs requests at INFO http://logs.openstack.org/66/362766/109/gate/tempest-full/52a4e60/controller/logs/screen-n-api.txt.gz#_Jun_15_15_26_18_719528 | |
| 22:20:55 | SpamapS | That makes sense. It just doesn't get as many requests. :) | |
| 22:20:56 | efried | Sundar: ...So now when the conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?) "hey, give me the details on this accelerator: $acc_uuid". Cyborg returns the AccGPU object, populated with an AccGPUProfile, which in turn is populated with information about display_heads=1 and resolution=1024x768 | |
| 22:21:32 | Sundar | efried: May be I am being dense. The placement knows enough to choose a RP (GPU in a node). What else does it need? The rest can be left to entities in the compute node, right? | |
| 22:21:33 | efried | conductor parlays that into querystring content for GET /allocation_candidates and sends it down to placement. | |
| 22:22:11 | efried | Sundar: Nope, placement knows what it's told. I need to tell it about more than just "I need one VGPU". Otherwise I have to do all my filtering later, which defeats the purpose of having used placement. | |
| 22:22:30 | Sundar | "conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?)" This is not needed for selecting a node | |
| 22:22:43 | efried | yes it is. | |
| 22:22:54 | Sundar | You can do GET /a-c without talking to Cyboirg | |
| 22:22:55 | efried | unless, as I say, you want to do all the filtering later, which defeats the purpose. | |
| 22:23:35 | efried | Sundar: n-cond had to get the config details of the accelerator from *somewhere*. | |
| 22:23:44 | efried | Sundar: Where did it get that from? | |
| 22:23:48 | Sundar | E.g. resources:ACCELERATOR_GPU=1; traits:RESOLUTION-1024-768=required (or whatever) | |
| 22:24:12 | efried | right, where does traits:RESOLUTION_1024X768=required come from? | |
| 22:24:27 | Sundar | CYborg (or whatever) published it | |
| 22:24:28 | efried | Sundar: Oh, this may be the source of confusion: are you thinking that comes from the flavor? | |
| 22:24:34 | Sundar | Yes | |
| 22:25:04 | efried | jaypipes, dansmith: Remind me why we're not putting accelerator stuff in flavors | |
| 22:25:22 | efried | (which I suppose may be the same reason we don't put neutron port stuff in flavors??) | |
| 22:25:28 | jaypipes | efried: we are putting that stuff in flavors and image metadata (at least, we are for *v*GPUs...) | |
| 22:26:25 | Sundar | jaypipes: +1 Same for FPGAs | |
| 22:26:39 | efried | well then | |
| 22:26:40 | jaypipes | efried: that, of course, is what is being *requested* by the user. something (Cyborg I was assuming) needs to decorate resource providers representing the physical accelerators with those traits. | |
| 22:26:57 | Sundar | Totally agreed | |
| 22:26:58 | efried | yeah, got that | |
| 22:27:01 | jaypipes | k | |
| 22:27:17 | Sundar | Great. I need to drop now for a 3:30 PM call | |
| 22:27:31 | Sundar | I am open to coming back tomorrow (or even later today) to close this | |
| 22:27:33 | efried | We need to get a consensus with dansmith. | |
| 22:27:46 | Sundar | OK, what's the best time for that? | |
| 22:27:58 | jroll | jaypipes: I don't have specific advice, I was just curious what clint was referring to and then tried to help clarify :) | |
| 22:27:58 | efried | Sundar: You don't need to be around for that, necessarily. | |
| 22:28:44 | Sundar | OK. Thanks, efried and jaypipes. Take care! | |
| 22:28:46 | efried | Sundar: I'll try to catch up with dansmith and see if it was just a misunderstanding or what. | |
| 22:29:49 | efried | jaypipes: FYI, this is the flow that dansmith and I talked through the other day: http://logs.openstack.org/45/575545/1/check/build-openstack-sphinx-docs/77f4d6a/html/specs/rocky/approved/nova-cyborg-flow.html | |
| 22:30:03 | efried | jaypipes: It closely follows the neutron port model with os-vif. | |
| 22:30:54 | jaypipes | jroll: k, no prob. was just curious if we should make a specific change. | |
| 22:32:49 | jaypipes | efried: I don't see *end users* ever doing `openstack acc create --type GPU <specs>`. Maybe some admin... maybe. though that said, I don't really know what <specs> would be. | |
| 22:33:11 | jaypipes | efried: end users (at least that I'm aware of) don't have the ability to create "accelerators". | |
| 22:33:12 | efried | jaypipes: same thing as whatever it would be in the flavor, more or less. | |
| 22:33:31 | jaypipes | efried: I guess I don't see why we need a separate service for that, but ok. | |
| 22:33:33 | efried | Okay, if this is all driven by the flavor, so be it. | |
| 22:33:58 | efried | jaypipes: If we don't, we don't. I've never been a huge fan of the `port create` pre-step. | |
| 22:34:34 | efried | I guess the difference is that every port needs to have specific *unique* config info (IP addr). | |
| 22:34:46 | efried | ...which doesn't make sense to be in a flavor. | |
| 22:36:25 | jaypipes | efried: we're talking about different things entirely. | |
| 22:37:42 | jaypipes | efried: I'll just leave this conversation like this... I don't see why Cyborg isn't the generic device manager... | |
| 22:37:56 | efried | I fully hope it will be | |
| 22:38:03 | jaypipes | in other words, I don't see the point of having `openstack acc $ANYTHING`. | |
| 22:38:20 | efried | I'm on board with that, if dansmith is. | |
| 22:38:46 | jaypipes | there should be a YAML inventory file describing devices (that can't for some reason be auto-discovered on a node) and that's that... | |
| 22:39:23 | efried | The way the conversation with Sundar started was that his specs talk about os-acc parallelling os-vif, but then doesn't actually do that. | |
| 22:39:50 | jaypipes | note that I didn't say that os-acc shouldn't exist. | |
| 22:40:40 | efried | I wanted os-acc to have Acc* objects (corresponding to VIF*) and Acc*Profile (=~ VIF*PortProfile); and to define plug/unplug methods which call to plugins registered/loaded stevedore-wise. | |
| 22:40:49 | efried | they didn't have any of that in their specs. | |
| 22:41:48 | jaypipes | efried: but I don't see the point of a Cyborg REST API. sorry if that offends anyone. I just really only see the point of a utility library (os-acc) that can be called to configure/init a particular device slot. | |
| 22:42:20 | efried | ...where os-acc has a plugin mechanism that in turn lets it call out to platform-specific code to do plugging etc. | |
| 22:42:31 | jaypipes | sure, that's fine with me. | |
| 22:42:55 | efried | I imagine the REST API may be useful for things like programming FPGAs or twiddling details in their database. | |
| 22:43:14 | efried | but Nova isn't going to touch it, as you say, because all Nova needs to interact with is os-acc. | |
| 22:43:46 | efried | now it's possible that os-acc will talk to the cyborg API. | |
| 22:43:48 | efried | isn't it? | |
| 22:43:50 | jaypipes | efried: nobody is going to be "prgramming FPGAs" via a REST API. :) | |
| 22:44:25 | jaypipes | efried: sure, I guess I'm fine with this sort of thing being in the Cyborg API: https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/deployables.py | |
| 22:44:46 | jaypipes | efried: what I *don't* think should be in the Cyborg API is this: https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/accelerators.py | |
| 22:44:55 | jaypipes | efried: because all that is is inventory and resource providers information. | |
| 22:45:05 | efried | jaypipes: Just meaning in the same sense that we have a REST API to create a server by identifying the UUID of an image to load up; cyborg would have an API to identify the UUID of a bitstream image to program onto the thing. | |
| 22:45:06 | jaypipes | efried: and that belongs in placement, IMHO. | |
| 22:45:08 | efried | is that off base? | |
| 22:45:37 | jaypipes | efried: no. https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/accelerators.py <-- that is the equivalent of compute_nodes in Nova, not instances table. | |
| 22:46:28 | efried | I was still talking about "programming FPGAs via a REST API". | |
| 22:46:34 | jaypipes | oh. | |
| 22:47:05 | jaypipes | efried: sure, that's fine. that's the /deployables REST API endpoint in Cyborg. | |
| 22:52:28 | efried | jaypipes: The other thing dansmith talked about was this need for a "bind" step. This is after allocate but before plug, and (I guess) lets cyborg know the actual RP we're grabbing. I think the purpose is so that cyborg can do their on-the-fly inventory/trait adjustment or perhaps final checking stuff? Not sure why this couldn't happen at the front of plug() before the plugin is invoked. | |
| 22:53:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Wait for network-vif-plugged before starting live migration https://review.openstack.org/558001 | |
| 22:53:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add check if neutron "binding-extended" extension is available https://review.openstack.org/523548 | |
| 22:53:36 | dansmith | efried: the point of bind would be to get the inventory adjustment step and start programming the thing as early as we know where it's going to be | |