Posted	Nick	Remark
#openstack-nova - 2018-07-02
18:31:51	Sundar	If you have an FPGA on Power, you will presumably have a vendor-specific (Intel/Xilinx) and OS-specific (Linux/AIX/...) driver for it, right? Independent of Openstack/Kubernetes/..., you would need that
18:32:19	Sundar	The Cyborg driver goes on top of that
18:32:21	efried	I imagine there's two levels of "driver" we'll need.
18:32:59	efried	One that the platform can use to talk to the device, and one per guest OS that the VM needs to actually *use* the thing.
18:33:27	efried	See, on kvm that's basically one thing, because the hypervisor and the guests are running the same OS.
18:33:43	efried	...and the device is "attached" to the hypervisor to begin with.
18:34:14	Sundar	Yes, the first is the PF driver and the latter is the VF driver for SR-IOV-based devices. They need not be the same OS -- you could run Centos on the host and Ubuntu (or maybe Windows) on the VM
18:34:35	Sundar	Does Power use SR-IOV?
18:34:38	efried	Yes
18:34:47	efried	That's a good example.
18:35:03	Sundar	That's what I'd expect :-P but good to confirm!
18:35:16	Sundar	OK, so you could have a PF driver on the host for mgmt ,..
18:35:26	Sundar	and a VF driver on the guest for FPGA access
18:35:29	efried	In order to carve off a VF and set up its QoS etc. you have to make REST calls to the platform.
18:35:40	efried	And to attach it to a VM, you have to call REST as well.
18:35:52	efried	But the VM itself has to have the right drivers installed in order to see that VF as a network device.
18:36:13	Sundar	Why do you assume a specific implementation, with REST API etc.? Let's keep it generic
18:37:09	Sundar	Yes, you need to 'get a VF' -- either grab one that already exists (and was discovered), or create one on the fly ...
18:37:27	Sundar	Either way, it is a device-specific op in my view.
18:38:06	Sundar	The specific calls you may make would probably depend on the OS, but that is abstracted by the Cyborg driver and/or vendor-specific driver for that OS
18:38:40	Sundar	For example, with Linux ...
18:39:06	Sundar	you can run pci_sriov_enable() as soon as Cybprg driver discovers the device (or even earlier), and you get the list of VFs
18:39:22	efried	See, this is exactly my point.
18:39:31	Sundar	The Cyborg db stores them, and they can later be attached to a VAN
18:40:25	efried	"Why do you assume a specific implementation, with REST API etc.? Let's keep it generic" ==> I'm mentioning specifically how this needs to be done on a Power system, because it's fundamentally different than what happens on kvm.
18:40:48	efried	"as soon as Cyborg driver discovers the device" ==> *How* does the cyborg driver discover the device?
18:40:50	Sundar	What REST API do you need for Power?
18:41:30	efried	Assuming you're writing in python, the REST API is available via https://github.com/powervm/pypowervm
18:41:46	efried	It's a totally Power-specific API.
18:42:16	efried	And, once again, I would not expect cyborg to be calling into pypowervm. I would expect anything at that level to be relegated to the plugin.
18:43:54	efried	pypowervm is awesome, but still requires some pretty esoteric domain knowledge of power systems. Knowledge that I wouldn't really expect the cyborg team to want to come up to speed on.
18:44:21	efried	Though I would be delighted to support you if you wanted to get into that business :)
18:47:12	dansmith	efried: the hypervisor and guests are not running the same OS in kvm
18:47:18	dansmith	they might be, but that would be coincidence
18:47:35	efried	yeah, my bad.
18:48:00	Sundar	:) We do want to support all architectures. IMHO, it would be simplest if there are Cyborg drivers for Power devices as well. There is very little that core Cyborg assumes about devices, esp. ...
18:48:37	Sundar	with the recent changes to make the scheduling independent of PCI-centric things like whitelists
18:49:26	Sundar	You may have concerns about whether Cyborg can support such widely different architectures, ...
18:49:41	Sundar	but Cyborg should support them
18:49:55	efried	I do. Or rather, whether Cyborg will be motivated to support the... "less popular" ones.
18:50:23	efried	wouldn't be the first time Power got marginalized because it's not kvm.
18:50:30	Sundar	As I tell everybody :), open source is democratic -- popularity does not (or should not) count!
18:51:28	efried	whereas if we architect it such that platform specific pieces can be written by those with skin in that platform (i.e. me and my team) then we're covered either way.
18:51:40	Sundar	Got it -- based on your input, I have worked to make the scheduling PCI-independent. We can carry that forward into the compute node/os-acc as well
18:51:46	efried	If you want to write it, great; but if you don't (don't want to, don't have time/funding, don't get around to it, whatever) then we can.
18:52:59	Sundar	You (or anybody else) can write Cyborg drivers that wrap around the vendor-specific ones (like Intel OPAE)
18:53:26	efried	Just so we're clear, "PCI address" is only one element of making sure things are generic. Kind of been using it as the flagship, but there's more to it than that.
18:53:43	efried	But your efforts to make things non-PCI-centric have been noted and much appreciated.
18:54:39	Sundar	Can you give me an end-to-end workflow for deploying a device on Power, so i can understand what all things are different?
18:55:01	efried	Well
18:55:19	efried	I can give you an example for SRIOV.
18:55:29	efried	I can't speak to FPGA or anything "programmable".
18:55:45	Sundar	OK, let's take a GPU
18:56:06	Sundar	say with SR-IOV support?
18:56:07	efried	So for GPU, I can tell you how we're going to do full-card attach. We don't have support for vGPU yet.
18:56:34	efried	So of course the GPU example is going to be very simple.
18:56:43	efried	I don't know what "GPU with SR-IOV support" means.
18:57:15	Sundar	OK, that was hypothetical. I'll let you give your example
18:57:17	efried	I thought SR-IOV had I/O (i.e. wires out the back) and GPU didn't.
18:58:01	Sundar	Any PCI device can support SR-IOV. Many GPU devices don't, in practice
18:58:32	Sundar	Anyway, please describe the full-card attach for a GPU
18:58:41	efried	Again, I don't understand how a GPU does I/O; and for this example there's no V(irtualization).
18:59:08	efried	Okay, so for GPU, we start with discovery. For the immediate future all of this is going to happen in nova, facilitated by the powervm virt driver.
18:59:38	efried	Discovery consists of intersecting two sets: 1) What the operator wants to allow (whitelist) and 2) What's on the system.
18:59:53	efried	Leaving aside the details of #1 (it's a file we parse)
19:00:37	efried	We discover what's on the system by doing a GET of /rest/api/uom/ManagedSystem, which returns an XML payload for a <ManagedSystem/> object, in which is embedded a list of <IOSlot/> objects.
19:01:05	efried	The <IOSlot/> objects have data fields like device ID, vendor ID, class, feature codes... like you would associate with a PCI device.
19:01:27	efried	it doesn't have a PCI address. Instead it has Power-specific addressing fields DRC name and DRC index.
19:01:53	efried	(the latter is 64 bits, which is one of the things that makes 32-bit PCI addresses suck for us - there's no reliable two-way mapping)
19:02:08	efried	So now we intersect #1 and #2 and come up with a list of the devices for which we're going to allow passthrough.
19:02:25	efried	We stuff those into child providers of the compute node.
19:03:28	efried	The design we're currently working toward is to have one RP per device. Since we're not virtualizing yet, each will have an inventory of 1. The resource class will be CUSTOM_IOSLOT unless overridden in the whitelist file; for GPUs it'll likely be CUSTOM_GPU (unless we can swing a standard GPU resource class in Rocky).
19:03:37	efried	(or even ACCELERATOR_GPU :)
19:04:29	efried	We also tag that RP with some custom traits, including some that are made up out of the vendor ID, device ID, etc.
19:05:11	efried	Now the operator comes along and builds a flavor with resources:GPU=1 and maybe trait:CUSTOM_VENDOR_ID_ABCD=required etc.
19:05:17	efried	And does a spawn
19:05:49	efried	Most of nova doesn't notice the GPU resource in the extra specs, just ignores it. But once we get into the powervm virt driver, we see it and act on it.
19:06:09	efried	We see it in the allocation, to be clear. We don't care so much about the flavor at this point.
19:06:44	efried	In the allocation we can see which provider UUID the resource came from. And we know which actual device that corresponds to on the system. (Because we cleverly named the RP after the DRC index, or something.)
19:07:08	efried	We create the VM as normal, but before boot, we have to attach the device.
19:07:52	efried	We do that by calling another REST API, probably POST /rest/api/uom/LogicalPartition/{uuid} with a payload including the <IOSlot/> entry corresponding to the device corresponding to the resource provider we peeled out of the allocation.
19:08:18	efried	That REST API does magic on the platform to attach the device to the VM. Then we boot, and we're done.
19:08:21	efried	So
19:08:24	efried	In cyborg-land
19:08:49	efried	that POST /rest/api/uom/LogicalPartition/{uuid} thing would be the guts of the os-acc plugin's plug() operation.
19:08:55	efried	easy peasy.
19:09:00	efried	But what about the discovery part?
19:10:04	efried	This is where Cyborg shouldn't have to know anything about GET /rest/api/uom/ManagedSystem and how to parse the XML payload that comes in response.
19:10:16	efried	or even the much easier pypowervm equivalent of ^
19:10:35	Sundar	Cool. The core of this flow -- representation in terms of RPs/RCs/traits and making Nova agnostic about the GPU resource class -- are the same as Cyborg ...
19:11:21	efried	There's a reason for that :)
19:11:21	Sundar	A few questions: who runs the discovery API (GET of /rest/api/uom/ManagedSystem)? Is it something on the compute node?
19:12:08	efried	Yes, in pre-cyborg-land, the discovery happens in the powervm virt driver code, during get_available_resource / update_provider_tree
19:13:33	efried	Note that this flow is in the early stages of development right now. We expect to have it working in our out-of-tree driver in Rocky. What we have currently implemented is for SR-IOV and is very different.
19:13:59	Sundar	OK, for the other REST API (POST /rest/api/uom/LogicalPartition/{uuid}), something about the implementation has to come back to the compute node and poke around the device, right? If the pypowervm is doing that, why do you need the REST API? The pypowervm can just do the poking around.
19:14:06	efried	The discovery and attach are similarly themed, in the sense that they involve PowerVM REST API calls. But we're not doing anything with resource providers yet.
19:14:32	efried	"come back to the compute node and poke around the device"...
19:14:44	efried	This is the thing about the power platform.
19:15:00	efried	The management partition is not the hypervisor.
19:16:25	efried	Power has true virtualization, in the sense that the host owns the hardware, and the "hypervisor" is a firmware thingy that I don't understand, and the management partition can only do its work by talking to that firmware gizmo.

Earlier   Later