| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-14 | |||
| 18:41:04 | Sundar | "Living the dream" -- great! Reviewing my specs is part of that dream.? :) | |
| 18:41:05 | efried | I was hoping to rope in stephenfin, who knows things about os-vif, but he's UTC. | |
| 18:41:33 | efried | Yeah, I've been sadly behind on that part of the dream, for which I apologize. | |
| 18:41:43 | efried | But trying to catch up now. | |
| 18:41:45 | Sundar | haha -- NP, JK | |
| 18:42:03 | efried | So I only yesterday read up on os-vif, which means two things: | |
| 18:42:11 | efried | 1) It's fresh in my mind | |
| 18:42:30 | efried | 2) It's still really new to me, and I'm sure I don't understand it as deeply as I should. | |
| 18:42:33 | Sundar | I am reviewing os-vif myself. I seem to have missed some important points, like using ovo. Thanks for pointing that out | |
| 18:42:40 | efried | which is why I wanted to bring in a ringer. | |
| 18:42:50 | Sundar | Sure | |
| 18:42:53 | efried | Right, so my understanding is this: | |
| 18:43:31 | efried | We define these base objects - in os-vif, they're subclasses of VIFBase. | |
| 18:44:00 | efried | I'm still not completely clear on what the category of those things is. But I think in the cyborg case, it might be AccBase and then the subclasses might be AccFPGA, AccGPU, AccHPTS, etc. | |
| 18:44:07 | efried | i.e. different broad categories of accelerators. | |
| 18:44:26 | efried | The AccBase has a 'plugin' field. | |
| 18:45:25 | efried | So for example if I want my platform, PowerVM, to provide support for GPUs, maybe I would publish a (separate, third-party) library called acc-powervm or cyborg-powervm or similar. | |
| 18:46:01 | efried | And maybe within that lib I have a class cyborg_powervm.plugin.gpu.PowerVMAccGPUPlugin | |
| 18:46:58 | efried | My package provides some kind of initialization code that registers that plugin and instantiates an instance of os_acc.objects.acc.AccGPU with the `plugin` field set to "cyborg_powervm.plugin.gpu.PowerVMAccGPUPlugin" | |
| 18:48:08 | edmondsw | melwitt mriedem could one of you whack https://blueprints.launchpad.net/nova/+spec/powervm-opts-group ... it's very old and obsolete | |
| 18:48:25 | Sundar | With os-vif and Neutron, most of the functionality is delegated to mechanism/type drivers, agents, etc. So, having a plugin field in each object may make sense. With Cyborg, I'd expect Cyborg agent to do the core work, and rely upon drivers for specifics | |
| 18:48:26 | efried | sorry, the package doesn't instantiate that guy - it just provides a registration hook so that cyborg knows about my plugin. And that registration tells you what kind of accelerator(s) it's good for. | |
| 18:49:52 | melwitt | edmondsw: done, thanks for letting us know | |
| 18:49:56 | efried | Now when cyborg gets a request to attach an accelerator, it knows what kind it is already (let's say GPU) so it knows it needs to instantiate an AccGPU. It looks around for registered plugins and discovers cyborg-powervm is registered; and it looks for plugins registered as being able to handle GPUs and finds cyborg_powervm.plugin.gpu.PowerVMAccGPUPlugin | |
| 18:50:14 | efried | so it sets `plugin` to that value in the AccGPU instance. | |
| 18:50:23 | Sundar | Cyborg already supports a notion of drivers, which cna be per-device or per-vendor, etc. Once the driver has discovered the necessary details, Cyborg has enough info to populate placement, and handle scheduling and attaches. For attaches, it may need to call the driver for specific aspects | |
| 18:50:46 | Sundar | Would/can your library/plugin really be a driver in the Cyborg sense? | |
| 18:50:48 | efried | And from this point on, any time anything platform-specific needs to be done, the plugin gets called upon to do that thing. | |
| 18:51:40 | efried | I don't know whether we're using the same definition of "driver" here, but what I can tell you is that there's no way cyborg knows how to do the plug operation on a Power system. | |
| 18:52:12 | efried | Whoever's doing the plug operation needs to call into Power-specific code. | |
| 18:53:08 | Sundar | You had mentioned that PCI devices in Power are treated differently than in x86. Can you expand on what is different about it? | |
| 18:53:21 | efried | The point of having this AccGPU ovo is that once it's built, you can send it over RPC between cyborg and nova (and maybe others), and as long as those guys have the right stevedore-isms to load up the thing pointed at by the `plugin` field, either side of that equation can invoke the plugin methods when it needs to do something platform-specific. | |
| 18:53:24 | Sundar | May be that will help me understand where the major differences are | |
| 18:53:52 | efried | okay | |
| 18:54:21 | efried | Devices on a Power system don't have PCI addresses. They have a location code, which is formatted completely differently. | |
| 18:54:48 | efried | And there's no way to do conversion/alias because location codes are 64 bits, whereas PCI addresses are 32. | |
| 18:55:57 | efried | We normally talk about these locations in terms of "DRC name", which looks something like U78CB.001.WZS0JZB-P1-C15 | |
| 18:56:42 | efried | which I think somehow represents like host serial, drawer, physical slot, physical function, and then more bits can go on the end for virtual functions... | |
| 18:57:04 | efried | There's also a "DRC index" which is just a 64-bit number, whose segments represent the same thing. | |
| 18:57:52 | efried | But the main issue is that, in order to attach a device to a VM on PowerVM, you have to call into the NovaLink REST API. | |
| 18:58:08 | efried | Which in no way resembles anything libvirt-ish. | |
| 18:58:26 | efried | We have a library called pypowervm that lets you make such calls in pythonic ways. | |
| 18:58:45 | efried | And, to come full circle, we have driver code in nova that calls into pypowervm. | |
| 18:59:20 | efried | There's no question in my mind that we need to decouple this level of platform-specific logic from cyborg itself. | |
| 18:59:42 | Sundar | So, attaching a device to a VM is not handled by nova virt driver directly but delegated to pypowervm lib? | |
| 19:00:06 | Sundar | Is there a virt driver for power? | |
| 19:00:09 | efried | well, yes, in the same way that the nova libvirt driver delegates to the libvirt... uhh, lib. | |
| 19:00:12 | efried | Yes. | |
| 19:00:51 | efried | nova.virt.powervm.driver.PowerVMDriver | |
| 19:01:04 | Sundar | OK. Thanks. Suppose, instead of PCI BDFs, we have a generic 'attach handle', which is an object that Nova virt drivers would need to do the attach | |
| 19:01:25 | Sundar | It could be a PCI BDF, a mediated device UUID, or a Power location code, DRC index, whatever... | |
| 19:01:32 | dansmith | better would be a uuid | |
| 19:01:35 | efried | which, like nova.virt.libvirt.driver.LibvirtDriver is a subclass of nova.virt.driver.ComputeDriver | |
| 19:02:33 | Sundar | OK. The os-acc API could deal with attach handles in general, instead of PCI BDFs in particular | |
| 19:02:42 | efried | Right, dansmith, a UUID for the thing that corresponds to the resource provider, and then enough information in the other fields in the AccGPU (or whatever) object to let me know how to configure things. | |
| 19:02:47 | efried | Sundar: YES | |
| 19:02:53 | Sundar | The handle would include a type: PCI BDF, mediated device, PowerVM device etc. | |
| 19:03:03 | dansmith | efried: I meant a uuid for the attach handle | |
| 19:03:12 | Sundar | Based on which we (Cyborg as well as nova virt) could invoke the right libraries | |
| 19:03:55 | efried | If we're following the os-vif model, they've got a field for a "port profile". They allow any number of those to be registered in os-vif, not sure if they're also allowed to be registered via the plugin packages. | |
| 19:04:40 | dansmith | if you follow the neutron model, you give nova a port uuid which you got from neutron, and nova can ask neutron what the vif_type is, if it needs that | |
| 19:05:54 | efried | dansmith: How does it go from that to a specific plugin? | |
| 19:06:10 | dansmith | well, maybe I'm mixing the external user into the workflow and you're just talking about the nova->cyborg bit | |
| 19:06:35 | Sundar | dansmith: The equiv of a port UUID would be a UUID for each accelerator, but that is problematic because the #accelerators per RP can change for multi-function devices (vGPUs, FPGAs, etc) | |
| 19:06:45 | dansmith | efried: but nova does a show on the port via uuid, and gets back a bunch of info, including the type, details, etc | |
| 19:07:08 | dansmith | Sundar: no, I mean a uuid for each piece you want the user to be able to attach | |
| 19:07:21 | dansmith | Sundar: not a uuid per accelerator device (although you need that too) | |
| 19:07:58 | efried | In the placement model, I'm not sure if we actually need the UUID of the specific attachy thing. | |
| 19:08:12 | dansmith | efried: compute will | |
| 19:08:36 | dansmith | and scheduler will need the uuid of the accelerator itself, because that's the RP with inventory | |
| 19:08:44 | mriedem | bpoulos: which osc plugin makes the openstack secret store command available? | |
| 19:09:20 | efried | n-sch talks to placement and gets back an allocated (provider, RC, amount). That info somehow gets to the driver's plug method, and the plug method knows how to attach <amount> <RC>s from <provider> to the VM. | |
| 19:10:17 | efried | but if there's some preconfiguration step, like telling it a bitstream UUID, or QoS params, or whatever, then yeah, we do the port model where we tell cyborg to set it up first. Then it'll create the Acc* instance filled in with everything *except* the provider UUID. | |
| 19:10:31 | dansmith | right, so n-sch does the claim to get that allocation, then n-cond talks to cyborg and says "okay, I've been told that you have 1 FOO_THING on compute $host, please give me an attachment" | |
| 19:10:52 | dansmith | then compute can use that to ask cyborg for the real details, pci address, or whatever | |
| 19:11:25 | efried | "give me an attachment" - what's an attachment? | |
| 19:11:44 | dansmith | efried: the equivalent of a port in neutron | |
| 19:12:04 | dansmith | efried: it's not an actual thing, it just represents the configuration and link between some network and the instance | |
| 19:12:09 | efried | okay, so basically it's just saying "register this thing and give me a UUID". | |
| 19:12:44 | efried | So cyborg stuffs that UUID into the Acc* object and wings it back over RPC to n-cpu, which ultimately calls the plug method. | |
| 19:12:44 | bpoulos | mriedem: barbican | |
| 19:12:45 | dansmith | efried: well, the point at which you do that is cyborg's opportunity to reconfigure the actual thing, and potentially adjust inventory if it has to reserve some amount of something because it gave you a principal or whatever | |
| 19:12:59 | mriedem | bpoulos: python-barbicanclient? | |
| 19:13:06 | dansmith | efried: no, cyborg does not talk RPC to nova-anything | |
| 19:13:09 | bpoulos | mriedem: in devstack, you can add the line "enable_plugin barbican https://git.openstack.org/openstack/barbican" | |
| 19:13:25 | mriedem | but that's the server | |
| 19:13:26 | bpoulos | mriedem: yeah | |
| 19:13:27 | mriedem | i just need the client | |
| 19:13:28 | mriedem | ok | |
| 19:13:30 | mriedem | http://git.openstack.org/cgit/openstack/python-barbicanclient/tree/barbicanclient/osc_plugin.py | |
| 19:13:31 | mriedem | yeah | |
| 19:13:31 | efried | dansmith: And when you say "cyborg's opportunity" you really mean "the platform-specific code as invoked by cyborg" | |
| 19:13:36 | dansmith | efried: yes | |
| 19:13:49 | efried | dansmith: And that's a separate step from the actual 'plug' | |
| 19:13:53 | dansmith | efried: right | |
| 19:14:06 | efried | which happens from n-cpu, and again should call into the cyborg plugin | |
| 19:14:15 | dansmith | efried: because in libvirt land, we can't do the plug until we've started the vm and havesomething to actually plug (in the network case anyway) | |
| 19:14:30 | dansmith | efried: that plug step is in os-acc, that nova-compute calls yeah | |
| 19:14:37 | dansmith | or compute/$virt | |
| 19:15:08 | efried | Right, so today network plug is a ComputeDriver method, but tomorrow it would live in the VIFThingy.plugin class - did I get that part right? | |