Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-18
22:16:53 jaypipes efried: which is used to created the instance pci device request
22:17:10 Sundar I am not an expert on GPUs by a long shot :) but can't you apply traits on your GPU indicating what resolutions it can support, and use those traits in placement to pick a device?
22:17:12 efried So instead we first do `openstack accelerator create --type vgpu --specs display_heads=1,resolution=1024x768`, or similar.
22:17:41 Sundar "--specs display_heads=1,resolution=1024x768`" --> why can't this be a trait?
22:17:45 efried And we get back an "accelerator UUID" Or maybe it's called an "attachment UUID". Whatever term, not important right now. Point is, it's just a handle for that config data.
22:17:52 efried Sundar: It *is*. That's the point.
22:17:59 jaypipes efried: it's >1 trait, to be clear.
22:18:15 efried Sundar: We're not *creating* anything in placement at this point in the game. That was already done by cyborg when you did your discovery and whatnot.
22:18:49 efried So now we got back some $acc_uuid and we can say `openstack server create --flavor foo --image bar --accelerator $acc_uuid`
22:19:31 Sundar efried, jaypipes: Since it is a trait, placement can choose a GPU device in a node on on that basis, and have n-cpu call os-acc to 'configure' the gpu as needed?
22:19:55 efried yes, Sundar, that's exactly what's going to happen. I'm talking through how placement is going to get that information.
22:19:58 mriedem SpamapS: jroll: efried: n-api also logs requests at INFO http://logs.openstack.org/66/362766/109/gate/tempest-full/52a4e60/controller/logs/screen-n-api.txt.gz#_Jun_15_15_26_18_719528
22:20:55 SpamapS That makes sense. It just doesn't get as many requests. :)
22:20:56 efried Sundar: ...So now when the conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?) "hey, give me the details on this accelerator: $acc_uuid". Cyborg returns the AccGPU object, populated with an AccGPUProfile, which in turn is populated with information about display_heads=1 and resolution=1024x768
22:21:32 Sundar efried: May be I am being dense. The placement knows enough to choose a RP (GPU in a node). What else does it need? The rest can be left to entities in the compute node, right?
22:21:33 efried conductor parlays that into querystring content for GET /allocation_candidates and sends it down to placement.
22:22:11 efried Sundar: Nope, placement knows what it's told. I need to tell it about more than just "I need one VGPU". Otherwise I have to do all my filtering later, which defeats the purpose of having used placement.
22:22:30 Sundar "conductor needs to draw up a placement query for allocation candidates, it asks cyborg (via os-acc?)" This is not needed for selecting a node
22:22:43 efried yes it is.
22:22:54 Sundar You can do GET /a-c without talking to Cyboirg
22:22:55 efried unless, as I say, you want to do all the filtering later, which defeats the purpose.
22:23:35 efried Sundar: n-cond had to get the config details of the accelerator from *somewhere*.
22:23:44 efried Sundar: Where did it get that from?
22:23:48 Sundar E.g. resources:ACCELERATOR_GPU=1; traits:RESOLUTION-1024-768=required (or whatever)
22:24:12 efried right, where does traits:RESOLUTION_1024X768=required come from?
22:24:27 Sundar CYborg (or whatever) published it
22:24:28 efried Sundar: Oh, this may be the source of confusion: are you thinking that comes from the flavor?
22:24:34 Sundar Yes
22:25:04 efried jaypipes, dansmith: Remind me why we're not putting accelerator stuff in flavors
22:25:22 efried (which I suppose may be the same reason we don't put neutron port stuff in flavors??)
22:25:28 jaypipes efried: we are putting that stuff in flavors and image metadata (at least, we are for *v*GPUs...)
22:26:25 Sundar jaypipes: +1 Same for FPGAs
22:26:39 efried well then
22:26:40 jaypipes efried: that, of course, is what is being *requested* by the user. something (Cyborg I was assuming) needs to decorate resource providers representing the physical accelerators with those traits.
22:26:57 Sundar Totally agreed
22:26:58 efried yeah, got that
22:27:01 jaypipes k
22:27:17 Sundar Great. I need to drop now for a 3:30 PM call
22:27:31 Sundar I am open to coming back tomorrow (or even later today) to close this
22:27:33 efried We need to get a consensus with dansmith.
22:27:46 Sundar OK, what's the best time for that?
22:27:58 jroll jaypipes: I don't have specific advice, I was just curious what clint was referring to and then tried to help clarify :)
22:27:58 efried Sundar: You don't need to be around for that, necessarily.
22:28:44 Sundar OK. Thanks, efried and jaypipes. Take care!
22:28:46 efried Sundar: I'll try to catch up with dansmith and see if it was just a misunderstanding or what.
22:29:49 efried jaypipes: FYI, this is the flow that dansmith and I talked through the other day: http://logs.openstack.org/45/575545/1/check/build-openstack-sphinx-docs/77f4d6a/html/specs/rocky/approved/nova-cyborg-flow.html
22:30:03 efried jaypipes: It closely follows the neutron port model with os-vif.
22:30:54 jaypipes jroll: k, no prob. was just curious if we should make a specific change.
22:32:49 jaypipes efried: I don't see *end users* ever doing `openstack acc create --type GPU <specs>`. Maybe some admin... maybe. though that said, I don't really know what <specs> would be.
22:33:11 jaypipes efried: end users (at least that I'm aware of) don't have the ability to create "accelerators".
22:33:12 efried jaypipes: same thing as whatever it would be in the flavor, more or less.
22:33:31 jaypipes efried: I guess I don't see why we need a separate service for that, but ok.
22:33:33 efried Okay, if this is all driven by the flavor, so be it.
22:33:58 efried jaypipes: If we don't, we don't. I've never been a huge fan of the `port create` pre-step.
22:34:34 efried I guess the difference is that every port needs to have specific *unique* config info (IP addr).
22:34:46 efried ...which doesn't make sense to be in a flavor.
22:36:25 jaypipes efried: we're talking about different things entirely.
22:37:42 jaypipes efried: I'll just leave this conversation like this... I don't see why Cyborg isn't the generic device manager...
22:37:56 efried I fully hope it will be
22:38:03 jaypipes in other words, I don't see the point of having `openstack acc $ANYTHING`.
22:38:20 efried I'm on board with that, if dansmith is.
22:38:46 jaypipes there should be a YAML inventory file describing devices (that can't for some reason be auto-discovered on a node) and that's that...
22:39:23 efried The way the conversation with Sundar started was that his specs talk about os-acc parallelling os-vif, but then doesn't actually do that.
22:39:50 jaypipes note that I didn't say that os-acc shouldn't exist.
22:40:40 efried I wanted os-acc to have Acc* objects (corresponding to VIF*) and Acc*Profile (=~ VIF*PortProfile); and to define plug/unplug methods which call to plugins registered/loaded stevedore-wise.
22:40:49 efried they didn't have any of that in their specs.
22:41:48 jaypipes efried: but I don't see the point of a Cyborg REST API. sorry if that offends anyone. I just really only see the point of a utility library (os-acc) that can be called to configure/init a particular device slot.
22:42:20 efried ...where os-acc has a plugin mechanism that in turn lets it call out to platform-specific code to do plugging etc.
22:42:31 jaypipes sure, that's fine with me.
22:42:55 efried I imagine the REST API may be useful for things like programming FPGAs or twiddling details in their database.
22:43:14 efried but Nova isn't going to touch it, as you say, because all Nova needs to interact with is os-acc.
22:43:46 efried now it's possible that os-acc will talk to the cyborg API.
22:43:48 efried isn't it?
22:43:50 jaypipes efried: nobody is going to be "prgramming FPGAs" via a REST API. :)
22:44:25 jaypipes efried: sure, I guess I'm fine with this sort of thing being in the Cyborg API: https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/deployables.py
22:44:46 jaypipes efried: what I *don't* think should be in the Cyborg API is this: https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/accelerators.py
22:44:55 jaypipes efried: because all that is is inventory and resource providers information.
22:45:05 efried jaypipes: Just meaning in the same sense that we have a REST API to create a server by identifying the UUID of an image to load up; cyborg would have an API to identify the UUID of a bitstream image to program onto the thing.
22:45:06 jaypipes efried: and that belongs in placement, IMHO.
22:45:08 efried is that off base?
22:45:37 jaypipes efried: no. https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/accelerators.py <-- that is the equivalent of compute_nodes in Nova, not instances table.
22:46:28 efried I was still talking about "programming FPGAs via a REST API".
22:46:34 jaypipes oh.
22:47:05 jaypipes efried: sure, that's fine. that's the /deployables REST API endpoint in Cyborg.
22:52:28 efried jaypipes: The other thing dansmith talked about was this need for a "bind" step. This is after allocate but before plug, and (I guess) lets cyborg know the actual RP we're grabbing. I think the purpose is so that cyborg can do their on-the-fly inventory/trait adjustment or perhaps final checking stuff? Not sure why this couldn't happen at the front of plug() before the plugin is invoked.
22:53:35 openstackgerrit Matt Riedemann proposed openstack/nova master: Wait for network-vif-plugged before starting live migration https://review.openstack.org/558001
22:53:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Add check if neutron "binding-extended" extension is available https://review.openstack.org/523548
22:53:36 dansmith efried: the point of bind would be to get the inventory adjustment step and start programming the thing as early as we know where it's going to be
22:53:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Add "bind_ports_to_host" neutron API method https://review.openstack.org/523604
22:53:37 mriedem watch out
22:53:38 openstackgerrit Matt Riedemann proposed openstack/nova master: Add "delete_port_binding" network API method https://review.openstack.org/552170
22:53:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add "activate_port_binding" neutron API method https://review.openstack.org/555947
22:53:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Delete port bindings in setup_networks_on_host if teardown=True https://review.openstack.org/556333
22:53:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement migrate_instance_start method for neutron https://review.openstack.org/556334
22:53:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Add VIFMigrateData object for live migration https://review.openstack.org/515423
22:53:43 openstackgerrit Matt Riedemann proposed openstack/nova master: Add VIFMigrateData.get_dest_vif https://review.openstack.org/566931
22:53:44 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: factor out pre_live_migration plug_vifs call https://review.openstack.org/566932
22:53:45 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: use dest host port bindings during pre_live_migration https://review.openstack.org/566933
22:53:46 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: use dest host vif migrate details for live migration https://review.openstack.org/551370
22:53:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Annotate flows and handle PortBindingDeletionFailed in ComputeManager https://review.openstack.org/551371

Earlier   Later