Posted	Nick	Remark
#openstack-nova - 2018-07-02
18:43:54	efried	pypowervm is awesome, but still requires some pretty esoteric domain knowledge of power systems. Knowledge that I wouldn't really expect the cyborg team to want to come up to speed on.
18:44:21	efried	Though I would be delighted to support you if you wanted to get into that business :)
18:47:12	dansmith	efried: the hypervisor and guests are not running the same OS in kvm
18:47:18	dansmith	they might be, but that would be coincidence
18:47:35	efried	yeah, my bad.
18:48:00	Sundar	:) We do want to support all architectures. IMHO, it would be simplest if there are Cyborg drivers for Power devices as well. There is very little that core Cyborg assumes about devices, esp. ...
18:48:37	Sundar	with the recent changes to make the scheduling independent of PCI-centric things like whitelists
18:49:26	Sundar	You may have concerns about whether Cyborg can support such widely different architectures, ...
18:49:41	Sundar	but Cyborg should support them
18:49:55	efried	I do. Or rather, whether Cyborg will be motivated to support the... "less popular" ones.
18:50:23	efried	wouldn't be the first time Power got marginalized because it's not kvm.
18:50:30	Sundar	As I tell everybody :), open source is democratic -- popularity does not (or should not) count!
18:51:28	efried	whereas if we architect it such that platform specific pieces can be written by those with skin in that platform (i.e. me and my team) then we're covered either way.
18:51:40	Sundar	Got it -- based on your input, I have worked to make the scheduling PCI-independent. We can carry that forward into the compute node/os-acc as well
18:51:46	efried	If you want to write it, great; but if you don't (don't want to, don't have time/funding, don't get around to it, whatever) then we can.
18:52:59	Sundar	You (or anybody else) can write Cyborg drivers that wrap around the vendor-specific ones (like Intel OPAE)
18:53:26	efried	Just so we're clear, "PCI address" is only one element of making sure things are generic. Kind of been using it as the flagship, but there's more to it than that.
18:53:43	efried	But your efforts to make things non-PCI-centric have been noted and much appreciated.
18:54:39	Sundar	Can you give me an end-to-end workflow for deploying a device on Power, so i can understand what all things are different?
18:55:01	efried	Well
18:55:19	efried	I can give you an example for SRIOV.
18:55:29	efried	I can't speak to FPGA or anything "programmable".
18:55:45	Sundar	OK, let's take a GPU
18:56:06	Sundar	say with SR-IOV support?
18:56:07	efried	So for GPU, I can tell you how we're going to do full-card attach. We don't have support for vGPU yet.
18:56:34	efried	So of course the GPU example is going to be very simple.
18:56:43	efried	I don't know what "GPU with SR-IOV support" means.
18:57:15	Sundar	OK, that was hypothetical. I'll let you give your example
18:57:17	efried	I thought SR-IOV had I/O (i.e. wires out the back) and GPU didn't.
18:58:01	Sundar	Any PCI device can support SR-IOV. Many GPU devices don't, in practice
18:58:32	Sundar	Anyway, please describe the full-card attach for a GPU
18:58:41	efried	Again, I don't understand how a GPU does I/O; and for this example there's no V(irtualization).
18:59:08	efried	Okay, so for GPU, we start with discovery. For the immediate future all of this is going to happen in nova, facilitated by the powervm virt driver.
18:59:38	efried	Discovery consists of intersecting two sets: 1) What the operator wants to allow (whitelist) and 2) What's on the system.
18:59:53	efried	Leaving aside the details of #1 (it's a file we parse)
19:00:37	efried	We discover what's on the system by doing a GET of /rest/api/uom/ManagedSystem, which returns an XML payload for a <ManagedSystem/> object, in which is embedded a list of <IOSlot/> objects.
19:01:05	efried	The <IOSlot/> objects have data fields like device ID, vendor ID, class, feature codes... like you would associate with a PCI device.
19:01:27	efried	it doesn't have a PCI address. Instead it has Power-specific addressing fields DRC name and DRC index.
19:01:53	efried	(the latter is 64 bits, which is one of the things that makes 32-bit PCI addresses suck for us - there's no reliable two-way mapping)
19:02:08	efried	So now we intersect #1 and #2 and come up with a list of the devices for which we're going to allow passthrough.
19:02:25	efried	We stuff those into child providers of the compute node.
19:03:28	efried	The design we're currently working toward is to have one RP per device. Since we're not virtualizing yet, each will have an inventory of 1. The resource class will be CUSTOM_IOSLOT unless overridden in the whitelist file; for GPUs it'll likely be CUSTOM_GPU (unless we can swing a standard GPU resource class in Rocky).
19:03:37	efried	(or even ACCELERATOR_GPU :)
19:04:29	efried	We also tag that RP with some custom traits, including some that are made up out of the vendor ID, device ID, etc.
19:05:11	efried	Now the operator comes along and builds a flavor with resources:GPU=1 and maybe trait:CUSTOM_VENDOR_ID_ABCD=required etc.
19:05:17	efried	And does a spawn
19:05:49	efried	Most of nova doesn't notice the GPU resource in the extra specs, just ignores it. But once we get into the powervm virt driver, we see it and act on it.
19:06:09	efried	We see it in the allocation, to be clear. We don't care so much about the flavor at this point.
19:06:44	efried	In the allocation we can see which provider UUID the resource came from. And we know which actual device that corresponds to on the system. (Because we cleverly named the RP after the DRC index, or something.)
19:07:08	efried	We create the VM as normal, but before boot, we have to attach the device.
19:07:52	efried	We do that by calling another REST API, probably POST /rest/api/uom/LogicalPartition/{uuid} with a payload including the <IOSlot/> entry corresponding to the device corresponding to the resource provider we peeled out of the allocation.
19:08:18	efried	That REST API does magic on the platform to attach the device to the VM. Then we boot, and we're done.
19:08:21	efried	So
19:08:24	efried	In cyborg-land
19:08:49	efried	that POST /rest/api/uom/LogicalPartition/{uuid} thing would be the guts of the os-acc plugin's plug() operation.
19:08:55	efried	easy peasy.
19:09:00	efried	But what about the discovery part?
19:10:04	efried	This is where Cyborg shouldn't have to know anything about GET /rest/api/uom/ManagedSystem and how to parse the XML payload that comes in response.
19:10:16	efried	or even the much easier pypowervm equivalent of ^
19:10:35	Sundar	Cool. The core of this flow -- representation in terms of RPs/RCs/traits and making Nova agnostic about the GPU resource class -- are the same as Cyborg ...
19:11:21	Sundar	A few questions: who runs the discovery API (GET of /rest/api/uom/ManagedSystem)? Is it something on the compute node?
19:11:21	efried	There's a reason for that :)
19:12:08	efried	Yes, in pre-cyborg-land, the discovery happens in the powervm virt driver code, during get_available_resource / update_provider_tree
19:13:33	efried	Note that this flow is in the early stages of development right now. We expect to have it working in our out-of-tree driver in Rocky. What we have currently implemented is for SR-IOV and is very different.
19:13:59	Sundar	OK, for the other REST API (POST /rest/api/uom/LogicalPartition/{uuid}), something about the implementation has to come back to the compute node and poke around the device, right? If the pypowervm is doing that, why do you need the REST API? The pypowervm can just do the poking around.
19:14:06	efried	The discovery and attach are similarly themed, in the sense that they involve PowerVM REST API calls. But we're not doing anything with resource providers yet.
19:14:32	efried	"come back to the compute node and poke around the device"...
19:14:44	efried	This is the thing about the power platform.
19:15:00	efried	The management partition is not the hypervisor.
19:16:25	efried	Power has true virtualization, in the sense that the host owns the hardware, and the "hypervisor" is a firmware thingy that I don't understand, and the management partition can only do its work by talking to that firmware gizmo.
19:17:04	Sundar	So, the powervm virt driver is running in the mgmt partition and does not know about the hypervisor. Does the hypervisor run device drivers?
19:17:27	efried	not in the sense that you think of device drivers.
19:18:24	efried	One of the main points to note is that the devices themselves aren't "visible" to the management partition, unless you actually *attach* them to it.
19:18:53	Sundar	How would a GPU device vendor support Power? Does he have to write a plugin/whatever for the hypervisor?
19:18:55	efried	This is why pypowervm can't just "poke around the device" directly.
19:19:22	efried	Somebody has to write drivers, yeah.
19:19:36	efried	You're way out of my area of expertise here, though.
19:20:18	Sundar	Hmmm, I need to think about this.
19:20:37	Sundar	Would SR-IOV device flow differ from the above in any significant way?
19:20:38	efried	I have no idea how e.g. FPGA programming is going to work. I'm only guessing it'll work like I mentioned above, where we'll have to attach the device to the management partition, where we'll have the right device driver installed, and we'll do the programming there, and then detach from the mgmt partition and attach to the VM.
19:21:12	efried	So SR-IOV we actually have working today, just without placement in the picture; it's all hacked into the old PCI passthrough subsystem.
19:21:35	efried	But SR-IOV is special (even more so) on power.
19:22:28	efried	We virtualize the VF through the virtial I/O servers. (That's three virtuals in that sentence.) This allows us to do redundancy at the platform level (rather than aggregation at the VM) and it also enables live migration of SR-IOV VMs.
19:23:07	efried	But - the discovery and plug steps are much the same in the sense that we do them by making REST calls.
19:23:34	efried	It's just the nova flow that's totally different at the moment. I'm actually not sure how we're going to do SR-IOV with placement. Maybe just not, yet.
19:23:45	efried	edmondsw: Have we considered ^ ?
19:24:22	Sundar	Power's mgmt vs. hypervisor split sounds a bit like Xen's Dom 0 vs hypervisor split, except Xen drivers in dom0 can access devices directly (no REST API needed). Does that sound about right?
19:26:14	efried	I'm afraid I don't know.
19:27:59	Sundar	OK, let me process that a bit. Thanks, efried. I didn't think I would get a crash course in Power as part of Cyborg development. ;)
19:28:15	efried	Sundar: Nobody expects the Spanish Inquisition.
19:28:35	efried	And I'll thank you not to use "crash" and "Power" in the same sentence again.
19:28:41	efried	Never happens.
19:28:54	Sundar	haha, sure.
19:29:23	efried	Thanks for the chat, Sundar. Keep up the good work, sir.
19:29:54	Sundar	NP. Take care, efried.
19:56:10	edmondsw	efried that's long... what were you asking?
19:56:59	efried	edmondsw: What, if anything, our plan is for porting our SR-IOV support over to placement-land
19:57:55	edmondsw	efried I thought you were considering that, but I have not spent time on it
19:59:02	efried	edmondsw: I haven't really considered it yet, no. I think for the forseeable future if we continue to expose SR-IOV devices via PCI passthrough it will *probably* just work with the GPU-via-placement stuff.
19:59:13	efried	But eventually we'll want to get it all into placement-land.

Earlier   Later