| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-02 | |||
| 18:18:56 | efried | hold a minute... | |
| 18:22:15 | efried | Sundar: Okay, sorry about that - was finishing up a soooper messy lunch, typing with one hand. I'm back to full speed now. | |
| 18:22:28 | efried | So here's the example: I want to program and attach an FPGA | |
| 18:23:08 | efried | Cyborg can do the programming, but in order to do so, it needs to be supplied with a file-like that's piped to the device. | |
| 18:23:18 | efried | (Again, no idea how realistic this is, but go with me for a bit here) | |
| 18:23:57 | efried | The user says "program the device with bitstream from image XYZ" | |
| 18:24:11 | efried | Cyborg calls the plugin and says "give me a file-like so I can program". | |
| 18:24:36 | efried | On libvirt, maybe the plugin just does f = open("/path/to/special_file") and gives back f. | |
| 18:24:49 | efried | On Power, we probably have to attach the device to the management partition and then do ^ | |
| 18:25:18 | efried | On Windows, gods forbid, who knows what happens, but it'd be like D:\path\to\special_file anyway... | |
| 18:25:36 | efried | So then control goes back to cyborg, which goes and happily flashes the device. | |
| 18:25:59 | efried | Now there has to be a "close/cleanup" step. Cyborg calls the plugin and says, "Okay, I'm done". | |
| 18:26:30 | efried | On libvirt, in this example, it's just close(f). On Power, close(f) and detach the device from the management partition. On Windows.... who knows, whatever. | |
| 18:27:47 | efried | Sundar: Now, I don't know where this fits in the overall "plug" flow. Does it happen before or after we attach the thing to the VM? I would assume before. | |
| 18:28:00 | efried | Except if there's a flow where we program the thing while it's already attached; I don't even know how that would work. | |
| 18:28:00 | Sundar | We already have provisions for programming from Cyborg drivers (not yet implemented fully). Re. "a file-like that's piped to the device", the bitstreams will reside in Glance and will be accessed via image UUID. | |
| 18:28:16 | efried | Sundar: Right, that's where the data comes *from*, but how does it get *to* the device? | |
| 18:28:36 | efried | The glance side is the source of the data. You need a sink. | |
| 18:29:29 | Sundar | The Cyborg FPGA driver knows what to do. Anyway, you will have a vendor stack -- like OPAE kernel drivers for Intel FPGAs, Xilinx stack etc. | |
| 18:29:29 | efried | Or maybe what you're saying is the program_fpga plugin method will be given the source handle and be responsible for pushing the data from that handle to the device. | |
| 18:29:32 | efried | That would be fine too. | |
| 18:29:49 | Sundar | The Cyborg FPGA driver knows how to interface with the vendor-specific stack | |
| 18:30:07 | efried | vendor-specific, sure, but on all platforms? That's the part I'm doubting. | |
| 18:31:51 | Sundar | If you have an FPGA on Power, you will presumably have a vendor-specific (Intel/Xilinx) and OS-specific (Linux/AIX/...) driver for it, right? Independent of Openstack/Kubernetes/..., you would need that | |
| 18:32:19 | Sundar | The Cyborg driver goes on top of that | |
| 18:32:21 | efried | I imagine there's two levels of "driver" we'll need. | |
| 18:32:59 | efried | One that the platform can use to talk to the device, and one per guest OS that the VM needs to actually *use* the thing. | |
| 18:33:27 | efried | See, on kvm that's basically one thing, because the hypervisor and the guests are running the same OS. | |
| 18:33:43 | efried | ...and the device is "attached" to the hypervisor to begin with. | |
| 18:34:14 | Sundar | Yes, the first is the PF driver and the latter is the VF driver for SR-IOV-based devices. They need not be the same OS -- you could run Centos on the host and Ubuntu (or maybe Windows) on the VM | |
| 18:34:35 | Sundar | Does Power use SR-IOV? | |
| 18:34:38 | efried | Yes | |
| 18:34:47 | efried | That's a good example. | |
| 18:35:03 | Sundar | That's what I'd expect :-P but good to confirm! | |
| 18:35:16 | Sundar | OK, so you could have a PF driver on the host for mgmt ,.. | |
| 18:35:26 | Sundar | and a VF driver on the guest for FPGA access | |
| 18:35:29 | efried | In order to carve off a VF and set up its QoS etc. you have to make REST calls to the platform. | |
| 18:35:40 | efried | And to attach it to a VM, you have to call REST as well. | |
| 18:35:52 | efried | But the VM itself has to have the right drivers installed in order to see that VF as a network device. | |
| 18:36:13 | Sundar | Why do you assume a specific implementation, with REST API etc.? Let's keep it generic | |
| 18:37:09 | Sundar | Yes, you need to 'get a VF' -- either grab one that already exists (and was discovered), or create one on the fly ... | |
| 18:37:27 | Sundar | Either way, it is a device-specific op in my view. | |
| 18:38:06 | Sundar | The specific calls you may make would probably depend on the OS, but that is abstracted by the Cyborg driver and/or vendor-specific driver for that OS | |
| 18:38:40 | Sundar | For example, with Linux ... | |
| 18:39:06 | Sundar | you can run pci_sriov_enable() as soon as Cybprg driver discovers the device (or even earlier), and you get the list of VFs | |
| 18:39:22 | efried | See, this is exactly my point. | |
| 18:39:31 | Sundar | The Cyborg db stores them, and they can later be attached to a VAN | |
| 18:40:25 | efried | "Why do you assume a specific implementation, with REST API etc.? Let's keep it generic" ==> I'm mentioning specifically how this needs to be done on a Power system, because it's fundamentally different than what happens on kvm. | |
| 18:40:48 | efried | "as soon as Cyborg driver discovers the device" ==> *How* does the cyborg driver discover the device? | |
| 18:40:50 | Sundar | What REST API do you need for Power? | |
| 18:41:30 | efried | Assuming you're writing in python, the REST API is available via https://github.com/powervm/pypowervm | |
| 18:41:46 | efried | It's a totally Power-specific API. | |
| 18:42:16 | efried | And, once again, I would not expect cyborg to be calling into pypowervm. I would expect anything at that level to be relegated to the plugin. | |
| 18:43:54 | efried | pypowervm is awesome, but still requires some pretty esoteric domain knowledge of power systems. Knowledge that I wouldn't really expect the cyborg team to want to come up to speed on. | |
| 18:44:21 | efried | Though I would be delighted to support you if you wanted to get into that business :) | |
| 18:47:12 | dansmith | efried: the hypervisor and guests are not running the same OS in kvm | |
| 18:47:18 | dansmith | they might be, but that would be coincidence | |
| 18:47:35 | efried | yeah, my bad. | |
| 18:48:00 | Sundar | :) We do want to support all architectures. IMHO, it would be simplest if there are Cyborg drivers for Power devices as well. There is very little that core Cyborg assumes about devices, esp. ... | |
| 18:48:37 | Sundar | with the recent changes to make the scheduling independent of PCI-centric things like whitelists | |
| 18:49:26 | Sundar | You may have concerns about whether Cyborg can support such widely different architectures, ... | |
| 18:49:41 | Sundar | but Cyborg should support them | |
| 18:49:55 | efried | I do. Or rather, whether Cyborg will be motivated to support the... "less popular" ones. | |
| 18:50:23 | efried | wouldn't be the first time Power got marginalized because it's not kvm. | |
| 18:50:30 | Sundar | As I tell everybody :), open source is democratic -- popularity does not (or should not) count! | |
| 18:51:28 | efried | whereas if we architect it such that platform specific pieces can be written by those with skin in that platform (i.e. me and my team) then we're covered either way. | |
| 18:51:40 | Sundar | Got it -- based on your input, I have worked to make the scheduling PCI-independent. We can carry that forward into the compute node/os-acc as well | |
| 18:51:46 | efried | If you want to write it, great; but if you don't (don't want to, don't have time/funding, don't get around to it, whatever) then we can. | |
| 18:52:59 | Sundar | You (or anybody else) can write Cyborg drivers that wrap around the vendor-specific ones (like Intel OPAE) | |
| 18:53:26 | efried | Just so we're clear, "PCI address" is only one element of making sure things are generic. Kind of been using it as the flagship, but there's more to it than that. | |
| 18:53:43 | efried | But your efforts to make things non-PCI-centric have been noted and much appreciated. | |
| 18:54:39 | Sundar | Can you give me an end-to-end workflow for deploying a device on Power, so i can understand what all things are different? | |
| 18:55:01 | efried | Well | |
| 18:55:19 | efried | I can give you an example for SRIOV. | |
| 18:55:29 | efried | I can't speak to FPGA or anything "programmable". | |
| 18:55:45 | Sundar | OK, let's take a GPU | |
| 18:56:06 | Sundar | say with SR-IOV support? | |
| 18:56:07 | efried | So for GPU, I can tell you how we're going to do full-card attach. We don't have support for vGPU yet. | |
| 18:56:34 | efried | So of course the GPU example is going to be very simple. | |
| 18:56:43 | efried | I don't know what "GPU with SR-IOV support" means. | |
| 18:57:15 | Sundar | OK, that was hypothetical. I'll let you give your example | |
| 18:57:17 | efried | I thought SR-IOV had I/O (i.e. wires out the back) and GPU didn't. | |
| 18:58:01 | Sundar | Any PCI device can support SR-IOV. Many GPU devices don't, in practice | |
| 18:58:32 | Sundar | Anyway, please describe the full-card attach for a GPU | |
| 18:58:41 | efried | Again, I don't understand how a GPU does I/O; and for this example there's no V(irtualization). | |
| 18:59:08 | efried | Okay, so for GPU, we start with discovery. For the immediate future all of this is going to happen in nova, facilitated by the powervm virt driver. | |
| 18:59:38 | efried | Discovery consists of intersecting two sets: 1) What the operator wants to allow (whitelist) and 2) What's on the system. | |
| 18:59:53 | efried | Leaving aside the details of #1 (it's a file we parse) | |
| 19:00:37 | efried | We discover what's on the system by doing a GET of /rest/api/uom/ManagedSystem, which returns an XML payload for a <ManagedSystem/> object, in which is embedded a list of <IOSlot/> objects. | |
| 19:01:05 | efried | The <IOSlot/> objects have data fields like device ID, vendor ID, class, feature codes... like you would associate with a PCI device. | |
| 19:01:27 | efried | it doesn't have a PCI address. Instead it has Power-specific addressing fields DRC name and DRC index. | |
| 19:01:53 | efried | (the latter is 64 bits, which is one of the things that makes 32-bit PCI addresses suck for us - there's no reliable two-way mapping) | |
| 19:02:08 | efried | So now we intersect #1 and #2 and come up with a list of the devices for which we're going to allow passthrough. | |
| 19:02:25 | efried | We stuff those into child providers of the compute node. | |
| 19:03:28 | efried | The design we're currently working toward is to have one RP per device. Since we're not virtualizing yet, each will have an inventory of 1. The resource class will be CUSTOM_IOSLOT unless overridden in the whitelist file; for GPUs it'll likely be CUSTOM_GPU (unless we can swing a standard GPU resource class in Rocky). | |
| 19:03:37 | efried | (or even ACCELERATOR_GPU :) | |
| 19:04:29 | efried | We also tag that RP with some custom traits, including some that are made up out of the vendor ID, device ID, etc. | |
| 19:05:11 | efried | Now the operator comes along and builds a flavor with resources:GPU=1 and maybe trait:CUSTOM_VENDOR_ID_ABCD=required etc. | |
| 19:05:17 | efried | And does a spawn | |
| 19:05:49 | efried | Most of nova doesn't notice the GPU resource in the extra specs, just ignores it. But once we get into the powervm virt driver, we see it and act on it. | |