| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-02 | |||
| 18:32:19 | Sundar | The Cyborg driver goes on top of that | |
| 18:32:21 | efried | I imagine there's two levels of "driver" we'll need. | |
| 18:32:59 | efried | One that the platform can use to talk to the device, and one per guest OS that the VM needs to actually *use* the thing. | |
| 18:33:27 | efried | See, on kvm that's basically one thing, because the hypervisor and the guests are running the same OS. | |
| 18:33:43 | efried | ...and the device is "attached" to the hypervisor to begin with. | |
| 18:34:14 | Sundar | Yes, the first is the PF driver and the latter is the VF driver for SR-IOV-based devices. They need not be the same OS -- you could run Centos on the host and Ubuntu (or maybe Windows) on the VM | |
| 18:34:35 | Sundar | Does Power use SR-IOV? | |
| 18:34:38 | efried | Yes | |
| 18:34:47 | efried | That's a good example. | |
| 18:35:03 | Sundar | That's what I'd expect :-P but good to confirm! | |
| 18:35:16 | Sundar | OK, so you could have a PF driver on the host for mgmt ,.. | |
| 18:35:26 | Sundar | and a VF driver on the guest for FPGA access | |
| 18:35:29 | efried | In order to carve off a VF and set up its QoS etc. you have to make REST calls to the platform. | |
| 18:35:40 | efried | And to attach it to a VM, you have to call REST as well. | |
| 18:35:52 | efried | But the VM itself has to have the right drivers installed in order to see that VF as a network device. | |
| 18:36:13 | Sundar | Why do you assume a specific implementation, with REST API etc.? Let's keep it generic | |
| 18:37:09 | Sundar | Yes, you need to 'get a VF' -- either grab one that already exists (and was discovered), or create one on the fly ... | |
| 18:37:27 | Sundar | Either way, it is a device-specific op in my view. | |
| 18:38:06 | Sundar | The specific calls you may make would probably depend on the OS, but that is abstracted by the Cyborg driver and/or vendor-specific driver for that OS | |
| 18:38:40 | Sundar | For example, with Linux ... | |
| 18:39:06 | Sundar | you can run pci_sriov_enable() as soon as Cybprg driver discovers the device (or even earlier), and you get the list of VFs | |
| 18:39:22 | efried | See, this is exactly my point. | |
| 18:39:31 | Sundar | The Cyborg db stores them, and they can later be attached to a VAN | |
| 18:40:25 | efried | "Why do you assume a specific implementation, with REST API etc.? Let's keep it generic" ==> I'm mentioning specifically how this needs to be done on a Power system, because it's fundamentally different than what happens on kvm. | |
| 18:40:48 | efried | "as soon as Cyborg driver discovers the device" ==> *How* does the cyborg driver discover the device? | |
| 18:40:50 | Sundar | What REST API do you need for Power? | |
| 18:41:30 | efried | Assuming you're writing in python, the REST API is available via https://github.com/powervm/pypowervm | |
| 18:41:46 | efried | It's a totally Power-specific API. | |
| 18:42:16 | efried | And, once again, I would not expect cyborg to be calling into pypowervm. I would expect anything at that level to be relegated to the plugin. | |
| 18:43:54 | efried | pypowervm is awesome, but still requires some pretty esoteric domain knowledge of power systems. Knowledge that I wouldn't really expect the cyborg team to want to come up to speed on. | |
| 18:44:21 | efried | Though I would be delighted to support you if you wanted to get into that business :) | |
| 18:47:12 | dansmith | efried: the hypervisor and guests are not running the same OS in kvm | |
| 18:47:18 | dansmith | they might be, but that would be coincidence | |
| 18:47:35 | efried | yeah, my bad. | |
| 18:48:00 | Sundar | :) We do want to support all architectures. IMHO, it would be simplest if there are Cyborg drivers for Power devices as well. There is very little that core Cyborg assumes about devices, esp. ... | |
| 18:48:37 | Sundar | with the recent changes to make the scheduling independent of PCI-centric things like whitelists | |
| 18:49:26 | Sundar | You may have concerns about whether Cyborg can support such widely different architectures, ... | |
| 18:49:41 | Sundar | but Cyborg should support them | |
| 18:49:55 | efried | I do. Or rather, whether Cyborg will be motivated to support the... "less popular" ones. | |
| 18:50:23 | efried | wouldn't be the first time Power got marginalized because it's not kvm. | |
| 18:50:30 | Sundar | As I tell everybody :), open source is democratic -- popularity does not (or should not) count! | |
| 18:51:28 | efried | whereas if we architect it such that platform specific pieces can be written by those with skin in that platform (i.e. me and my team) then we're covered either way. | |
| 18:51:40 | Sundar | Got it -- based on your input, I have worked to make the scheduling PCI-independent. We can carry that forward into the compute node/os-acc as well | |
| 18:51:46 | efried | If you want to write it, great; but if you don't (don't want to, don't have time/funding, don't get around to it, whatever) then we can. | |
| 18:52:59 | Sundar | You (or anybody else) can write Cyborg drivers that wrap around the vendor-specific ones (like Intel OPAE) | |
| 18:53:26 | efried | Just so we're clear, "PCI address" is only one element of making sure things are generic. Kind of been using it as the flagship, but there's more to it than that. | |
| 18:53:43 | efried | But your efforts to make things non-PCI-centric have been noted and much appreciated. | |
| 18:54:39 | Sundar | Can you give me an end-to-end workflow for deploying a device on Power, so i can understand what all things are different? | |
| 18:55:01 | efried | Well | |
| 18:55:19 | efried | I can give you an example for SRIOV. | |
| 18:55:29 | efried | I can't speak to FPGA or anything "programmable". | |
| 18:55:45 | Sundar | OK, let's take a GPU | |
| 18:56:06 | Sundar | say with SR-IOV support? | |
| 18:56:07 | efried | So for GPU, I can tell you how we're going to do full-card attach. We don't have support for vGPU yet. | |
| 18:56:34 | efried | So of course the GPU example is going to be very simple. | |
| 18:56:43 | efried | I don't know what "GPU with SR-IOV support" means. | |
| 18:57:15 | Sundar | OK, that was hypothetical. I'll let you give your example | |
| 18:57:17 | efried | I thought SR-IOV had I/O (i.e. wires out the back) and GPU didn't. | |
| 18:58:01 | Sundar | Any PCI device can support SR-IOV. Many GPU devices don't, in practice | |
| 18:58:32 | Sundar | Anyway, please describe the full-card attach for a GPU | |
| 18:58:41 | efried | Again, I don't understand how a GPU does I/O; and for this example there's no V(irtualization). | |
| 18:59:08 | efried | Okay, so for GPU, we start with discovery. For the immediate future all of this is going to happen in nova, facilitated by the powervm virt driver. | |
| 18:59:38 | efried | Discovery consists of intersecting two sets: 1) What the operator wants to allow (whitelist) and 2) What's on the system. | |
| 18:59:53 | efried | Leaving aside the details of #1 (it's a file we parse) | |
| 19:00:37 | efried | We discover what's on the system by doing a GET of /rest/api/uom/ManagedSystem, which returns an XML payload for a <ManagedSystem/> object, in which is embedded a list of <IOSlot/> objects. | |
| 19:01:05 | efried | The <IOSlot/> objects have data fields like device ID, vendor ID, class, feature codes... like you would associate with a PCI device. | |
| 19:01:27 | efried | it doesn't have a PCI address. Instead it has Power-specific addressing fields DRC name and DRC index. | |
| 19:01:53 | efried | (the latter is 64 bits, which is one of the things that makes 32-bit PCI addresses suck for us - there's no reliable two-way mapping) | |
| 19:02:08 | efried | So now we intersect #1 and #2 and come up with a list of the devices for which we're going to allow passthrough. | |
| 19:02:25 | efried | We stuff those into child providers of the compute node. | |
| 19:03:28 | efried | The design we're currently working toward is to have one RP per device. Since we're not virtualizing yet, each will have an inventory of 1. The resource class will be CUSTOM_IOSLOT unless overridden in the whitelist file; for GPUs it'll likely be CUSTOM_GPU (unless we can swing a standard GPU resource class in Rocky). | |
| 19:03:37 | efried | (or even ACCELERATOR_GPU :) | |
| 19:04:29 | efried | We also tag that RP with some custom traits, including some that are made up out of the vendor ID, device ID, etc. | |
| 19:05:11 | efried | Now the operator comes along and builds a flavor with resources:GPU=1 and maybe trait:CUSTOM_VENDOR_ID_ABCD=required etc. | |
| 19:05:17 | efried | And does a spawn | |
| 19:05:49 | efried | Most of nova doesn't notice the GPU resource in the extra specs, just ignores it. But once we get into the powervm virt driver, we see it and act on it. | |
| 19:06:09 | efried | We see it in the allocation, to be clear. We don't care so much about the flavor at this point. | |
| 19:06:44 | efried | In the allocation we can see which provider UUID the resource came from. And we know which actual device that corresponds to on the system. (Because we cleverly named the RP after the DRC index, or something.) | |
| 19:07:08 | efried | We create the VM as normal, but before boot, we have to attach the device. | |
| 19:07:52 | efried | We do that by calling another REST API, probably POST /rest/api/uom/LogicalPartition/{uuid} with a payload including the <IOSlot/> entry corresponding to the device corresponding to the resource provider we peeled out of the allocation. | |
| 19:08:18 | efried | That REST API does magic on the platform to attach the device to the VM. Then we boot, and we're done. | |
| 19:08:21 | efried | So | |
| 19:08:24 | efried | In cyborg-land | |
| 19:08:49 | efried | that POST /rest/api/uom/LogicalPartition/{uuid} thing would be the guts of the os-acc plugin's plug() operation. | |
| 19:08:55 | efried | easy peasy. | |
| 19:09:00 | efried | But what about the discovery part? | |
| 19:10:04 | efried | This is where Cyborg shouldn't have to know anything about GET /rest/api/uom/ManagedSystem and how to parse the XML payload that comes in response. | |
| 19:10:16 | efried | or even the much easier pypowervm equivalent of ^ | |
| 19:10:35 | Sundar | Cool. The core of this flow -- representation in terms of RPs/RCs/traits and making Nova agnostic about the GPU resource class -- are the same as Cyborg ... | |
| 19:11:21 | Sundar | A few questions: who runs the discovery API (GET of /rest/api/uom/ManagedSystem)? Is it something on the compute node? | |
| 19:11:21 | efried | There's a reason for that :) | |
| 19:12:08 | efried | Yes, in pre-cyborg-land, the discovery happens in the powervm virt driver code, during get_available_resource / update_provider_tree | |
| 19:13:33 | efried | Note that this flow is in the early stages of development right now. We expect to have it working in our out-of-tree driver in Rocky. What we have currently implemented is for SR-IOV and is very different. | |
| 19:13:59 | Sundar | OK, for the other REST API (POST /rest/api/uom/LogicalPartition/{uuid}), something about the implementation has to come back to the compute node and poke around the device, right? If the pypowervm is doing that, why do you need the REST API? The pypowervm can just do the poking around. | |
| 19:14:06 | efried | The discovery and attach are similarly themed, in the sense that they involve PowerVM REST API calls. But we're not doing anything with resource providers yet. | |
| 19:14:32 | efried | "come back to the compute node and poke around the device"... | |
| 19:14:44 | efried | This is the thing about the power platform. | |
| 19:15:00 | efried | The management partition is not the hypervisor. | |
| 19:16:25 | efried | Power has true virtualization, in the sense that the host owns the hardware, and the "hypervisor" is a firmware thingy that I don't understand, and the management partition can only do its work by talking to that firmware gizmo. | |
| 19:17:04 | Sundar | So, the powervm virt driver is running in the mgmt partition and does not know about the hypervisor. Does the hypervisor run device drivers? | |