| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-02 | |||
| 18:56:34 | efried | So of course the GPU example is going to be very simple. | |
| 18:56:43 | efried | I don't know what "GPU with SR-IOV support" means. | |
| 18:57:15 | Sundar | OK, that was hypothetical. I'll let you give your example | |
| 18:57:17 | efried | I thought SR-IOV had I/O (i.e. wires out the back) and GPU didn't. | |
| 18:58:01 | Sundar | Any PCI device can support SR-IOV. Many GPU devices don't, in practice | |
| 18:58:32 | Sundar | Anyway, please describe the full-card attach for a GPU | |
| 18:58:41 | efried | Again, I don't understand how a GPU does I/O; and for this example there's no V(irtualization). | |
| 18:59:08 | efried | Okay, so for GPU, we start with discovery. For the immediate future all of this is going to happen in nova, facilitated by the powervm virt driver. | |
| 18:59:38 | efried | Discovery consists of intersecting two sets: 1) What the operator wants to allow (whitelist) and 2) What's on the system. | |
| 18:59:53 | efried | Leaving aside the details of #1 (it's a file we parse) | |
| 19:00:37 | efried | We discover what's on the system by doing a GET of /rest/api/uom/ManagedSystem, which returns an XML payload for a <ManagedSystem/> object, in which is embedded a list of <IOSlot/> objects. | |
| 19:01:05 | efried | The <IOSlot/> objects have data fields like device ID, vendor ID, class, feature codes... like you would associate with a PCI device. | |
| 19:01:27 | efried | it doesn't have a PCI address. Instead it has Power-specific addressing fields DRC name and DRC index. | |
| 19:01:53 | efried | (the latter is 64 bits, which is one of the things that makes 32-bit PCI addresses suck for us - there's no reliable two-way mapping) | |
| 19:02:08 | efried | So now we intersect #1 and #2 and come up with a list of the devices for which we're going to allow passthrough. | |
| 19:02:25 | efried | We stuff those into child providers of the compute node. | |
| 19:03:28 | efried | The design we're currently working toward is to have one RP per device. Since we're not virtualizing yet, each will have an inventory of 1. The resource class will be CUSTOM_IOSLOT unless overridden in the whitelist file; for GPUs it'll likely be CUSTOM_GPU (unless we can swing a standard GPU resource class in Rocky). | |
| 19:03:37 | efried | (or even ACCELERATOR_GPU :) | |
| 19:04:29 | efried | We also tag that RP with some custom traits, including some that are made up out of the vendor ID, device ID, etc. | |
| 19:05:11 | efried | Now the operator comes along and builds a flavor with resources:GPU=1 and maybe trait:CUSTOM_VENDOR_ID_ABCD=required etc. | |
| 19:05:17 | efried | And does a spawn | |
| 19:05:49 | efried | Most of nova doesn't notice the GPU resource in the extra specs, just ignores it. But once we get into the powervm virt driver, we see it and act on it. | |
| 19:06:09 | efried | We see it in the allocation, to be clear. We don't care so much about the flavor at this point. | |
| 19:06:44 | efried | In the allocation we can see which provider UUID the resource came from. And we know which actual device that corresponds to on the system. (Because we cleverly named the RP after the DRC index, or something.) | |
| 19:07:08 | efried | We create the VM as normal, but before boot, we have to attach the device. | |
| 19:07:52 | efried | We do that by calling another REST API, probably POST /rest/api/uom/LogicalPartition/{uuid} with a payload including the <IOSlot/> entry corresponding to the device corresponding to the resource provider we peeled out of the allocation. | |
| 19:08:18 | efried | That REST API does magic on the platform to attach the device to the VM. Then we boot, and we're done. | |
| 19:08:21 | efried | So | |
| 19:08:24 | efried | In cyborg-land | |
| 19:08:49 | efried | that POST /rest/api/uom/LogicalPartition/{uuid} thing would be the guts of the os-acc plugin's plug() operation. | |
| 19:08:55 | efried | easy peasy. | |
| 19:09:00 | efried | But what about the discovery part? | |
| 19:10:04 | efried | This is where Cyborg shouldn't have to know anything about GET /rest/api/uom/ManagedSystem and how to parse the XML payload that comes in response. | |
| 19:10:16 | efried | or even the much easier pypowervm equivalent of ^ | |
| 19:10:35 | Sundar | Cool. The core of this flow -- representation in terms of RPs/RCs/traits and making Nova agnostic about the GPU resource class -- are the same as Cyborg ... | |
| 19:11:21 | efried | There's a reason for that :) | |
| 19:11:21 | Sundar | A few questions: who runs the discovery API (GET of /rest/api/uom/ManagedSystem)? Is it something on the compute node? | |
| 19:12:08 | efried | Yes, in pre-cyborg-land, the discovery happens in the powervm virt driver code, during get_available_resource / update_provider_tree | |
| 19:13:33 | efried | Note that this flow is in the early stages of development right now. We expect to have it working in our out-of-tree driver in Rocky. What we have currently implemented is for SR-IOV and is very different. | |
| 19:13:59 | Sundar | OK, for the other REST API (POST /rest/api/uom/LogicalPartition/{uuid}), something about the implementation has to come back to the compute node and poke around the device, right? If the pypowervm is doing that, why do you need the REST API? The pypowervm can just do the poking around. | |
| 19:14:06 | efried | The discovery and attach are similarly themed, in the sense that they involve PowerVM REST API calls. But we're not doing anything with resource providers yet. | |
| 19:14:32 | efried | "come back to the compute node and poke around the device"... | |
| 19:14:44 | efried | This is the thing about the power platform. | |
| 19:15:00 | efried | The management partition is not the hypervisor. | |
| 19:16:25 | efried | Power has true virtualization, in the sense that the host owns the hardware, and the "hypervisor" is a firmware thingy that I don't understand, and the management partition can only do its work by talking to that firmware gizmo. | |
| 19:17:04 | Sundar | So, the powervm virt driver is running in the mgmt partition and does not know about the hypervisor. Does the hypervisor run device drivers? | |
| 19:17:27 | efried | not in the sense that you think of device drivers. | |
| 19:18:24 | efried | One of the main points to note is that the devices themselves aren't "visible" to the management partition, unless you actually *attach* them to it. | |
| 19:18:53 | Sundar | How would a GPU device vendor support Power? Does he have to write a plugin/whatever for the hypervisor? | |
| 19:18:55 | efried | This is why pypowervm can't just "poke around the device" directly. | |
| 19:19:22 | efried | Somebody has to write drivers, yeah. | |
| 19:19:36 | efried | You're way out of my area of expertise here, though. | |
| 19:20:18 | Sundar | Hmmm, I need to think about this. | |
| 19:20:37 | Sundar | Would SR-IOV device flow differ from the above in any significant way? | |
| 19:20:38 | efried | I have no idea how e.g. FPGA programming is going to work. I'm only guessing it'll work like I mentioned above, where we'll have to attach the device to the management partition, where we'll have the right device driver installed, and we'll do the programming there, and then detach from the mgmt partition and attach to the VM. | |
| 19:21:12 | efried | So SR-IOV we actually have working today, just without placement in the picture; it's all hacked into the old PCI passthrough subsystem. | |
| 19:21:35 | efried | But SR-IOV is special (even more so) on power. | |
| 19:22:28 | efried | We virtualize the VF through the virtial I/O servers. (That's three virtuals in that sentence.) This allows us to do redundancy at the platform level (rather than aggregation at the VM) and it also enables live migration of SR-IOV VMs. | |
| 19:23:07 | efried | But - the discovery and plug steps are much the same in the sense that we do them by making REST calls. | |
| 19:23:34 | efried | It's just the nova flow that's totally different at the moment. I'm actually not sure how we're going to do SR-IOV with placement. Maybe just not, yet. | |
| 19:23:45 | efried | edmondsw: Have we considered ^ ? | |
| 19:24:22 | Sundar | Power's mgmt vs. hypervisor split sounds a bit like Xen's Dom 0 vs hypervisor split, except Xen drivers in dom0 can access devices directly (no REST API needed). Does that sound about right? | |
| 19:26:14 | efried | I'm afraid I don't know. | |
| 19:27:59 | Sundar | OK, let me process that a bit. Thanks, efried. I didn't think I would get a crash course in Power as part of Cyborg development. ;) | |
| 19:28:15 | efried | Sundar: Nobody expects the Spanish Inquisition. | |
| 19:28:35 | efried | And I'll thank you not to use "crash" and "Power" in the same sentence again. | |
| 19:28:41 | efried | Never happens. | |
| 19:28:54 | Sundar | haha, sure. | |
| 19:29:23 | efried | Thanks for the chat, Sundar. Keep up the good work, sir. | |
| 19:29:54 | Sundar | NP. Take care, efried. | |
| 19:56:10 | edmondsw | efried that's long... what were you asking? | |
| 19:56:59 | efried | edmondsw: What, if anything, our plan is for porting our SR-IOV support over to placement-land | |
| 19:57:55 | edmondsw | efried I thought you were considering that, but I have not spent time on it | |
| 19:59:02 | efried | edmondsw: I haven't really considered it yet, no. I think for the forseeable future if we continue to expose SR-IOV devices via PCI passthrough it will *probably* just work with the GPU-via-placement stuff. | |
| 19:59:13 | efried | But eventually we'll want to get it all into placement-land. | |
| 19:59:22 | efried | I just don't know what that's going to look like. | |
| 20:00:00 | edmondsw | I must be missing something. Hasn't SR-IOV been one of the example use cases in several conversations? | |
| 20:00:20 | efried | example use case of what, in conversations about what? | |
| 20:00:38 | edmondsw | placement, traits, resource providers, etc. | |
| 20:00:44 | edmondsw | nrp | |
| 20:00:49 | efried | I've talked a lot about SR-IOV in general, hypothetical terms when discussing those things, yes. | |
| 20:01:05 | efried | But we haven't ever gone into the specifics. | |
| 20:04:08 | openstackgerrit | Merged openstack/nova master: Request only instance_uuid in ironic node list https://review.openstack.org/539509 | |
| 20:05:53 | openstackgerrit | Matt Rabe proposed openstack/nova master: Add destination MSP IP address to PowerVM migrate data https://review.openstack.org/579676 | |
| 21:27:08 | openstackgerrit | Matt Rabe proposed openstack/nova master: Add destination MSP IP address to PowerVM migrate data https://review.openstack.org/579676 | |
| 21:28:45 | openstack | bug 1779635 in OpenStack Compute (nova) "placement allows RP parent loop in PUT resource_providers/{uuid}" [Medium,In progress] https://launchpad.net/bugs/1779635 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 21:28:45 | openstackgerrit | Merged openstack/nova master: Regression test for bug 1779635 https://review.openstack.org/579508 | |
| 23:11:10 | openstackgerrit | Merged openstack/nova master: Remove mox in unit/network/test_neutronv2.py (1) https://review.openstack.org/572018 | |
| #openstack-nova - 2018-07-03 | |||
| 01:08:11 | naichuans_ | efried: Thank you, will wait until reshape finished :) | |
| 01:31:28 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Add support specify volume type when boot instance https://review.openstack.org/579520 | |
| 01:49:49 | gmann | deepak_mourya: i think natsume replied on bug. 'id' is in ignored sort_keys - https://github.com/openstack/nova/blob/9382b7854286bba21235c4bfe39c585a6b894419/nova/api/openstack/compute/schemas/servers.py#L429 | |
| 02:38:27 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add policy to InstanceGroup object and api models. https://review.openstack.org/563375 | |
| 03:06:19 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add policy field to ServerGroup notification object https://review.openstack.org/563401 | |
| 03:12:57 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add fault info when unshelve instance failed https://review.openstack.org/579747 | |
| 03:31:55 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Change the anti-affinity Filter to adapt to new policy https://review.openstack.org/571166 | |
| 03:32:06 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Change the anti-affinity Filter to adapt to new policy https://review.openstack.org/571166 | |
| 03:32:38 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Adapt _validate_instance_group_policy to new policy model https://review.openstack.org/571465 | |
| 03:33:04 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Microversion 2.64 - Use new format policy in server group https://review.openstack.org/567534 | |
| 03:56:45 | openstackgerrit | Merged openstack/osc-placement master: Add nested resource providers (v1.14) https://review.openstack.org/546675 | |
| 03:56:46 | openstackgerrit | Merged openstack/osc-placement master: Limit allocation candidates (v1.15, v1.16) https://review.openstack.org/548043 | |