| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-14 | |||
| 18:55:57 | efried | We normally talk about these locations in terms of "DRC name", which looks something like U78CB.001.WZS0JZB-P1-C15 | |
| 18:56:42 | efried | which I think somehow represents like host serial, drawer, physical slot, physical function, and then more bits can go on the end for virtual functions... | |
| 18:57:04 | efried | There's also a "DRC index" which is just a 64-bit number, whose segments represent the same thing. | |
| 18:57:52 | efried | But the main issue is that, in order to attach a device to a VM on PowerVM, you have to call into the NovaLink REST API. | |
| 18:58:08 | efried | Which in no way resembles anything libvirt-ish. | |
| 18:58:26 | efried | We have a library called pypowervm that lets you make such calls in pythonic ways. | |
| 18:58:45 | efried | And, to come full circle, we have driver code in nova that calls into pypowervm. | |
| 18:59:20 | efried | There's no question in my mind that we need to decouple this level of platform-specific logic from cyborg itself. | |
| 18:59:42 | Sundar | So, attaching a device to a VM is not handled by nova virt driver directly but delegated to pypowervm lib? | |
| 19:00:06 | Sundar | Is there a virt driver for power? | |
| 19:00:09 | efried | well, yes, in the same way that the nova libvirt driver delegates to the libvirt... uhh, lib. | |
| 19:00:12 | efried | Yes. | |
| 19:00:51 | efried | nova.virt.powervm.driver.PowerVMDriver | |
| 19:01:04 | Sundar | OK. Thanks. Suppose, instead of PCI BDFs, we have a generic 'attach handle', which is an object that Nova virt drivers would need to do the attach | |
| 19:01:25 | Sundar | It could be a PCI BDF, a mediated device UUID, or a Power location code, DRC index, whatever... | |
| 19:01:32 | dansmith | better would be a uuid | |
| 19:01:35 | efried | which, like nova.virt.libvirt.driver.LibvirtDriver is a subclass of nova.virt.driver.ComputeDriver | |
| 19:02:33 | Sundar | OK. The os-acc API could deal with attach handles in general, instead of PCI BDFs in particular | |
| 19:02:42 | efried | Right, dansmith, a UUID for the thing that corresponds to the resource provider, and then enough information in the other fields in the AccGPU (or whatever) object to let me know how to configure things. | |
| 19:02:47 | efried | Sundar: YES | |
| 19:02:53 | Sundar | The handle would include a type: PCI BDF, mediated device, PowerVM device etc. | |
| 19:03:03 | dansmith | efried: I meant a uuid for the attach handle | |
| 19:03:12 | Sundar | Based on which we (Cyborg as well as nova virt) could invoke the right libraries | |
| 19:03:55 | efried | If we're following the os-vif model, they've got a field for a "port profile". They allow any number of those to be registered in os-vif, not sure if they're also allowed to be registered via the plugin packages. | |
| 19:04:40 | dansmith | if you follow the neutron model, you give nova a port uuid which you got from neutron, and nova can ask neutron what the vif_type is, if it needs that | |
| 19:05:54 | efried | dansmith: How does it go from that to a specific plugin? | |
| 19:06:10 | dansmith | well, maybe I'm mixing the external user into the workflow and you're just talking about the nova->cyborg bit | |
| 19:06:35 | Sundar | dansmith: The equiv of a port UUID would be a UUID for each accelerator, but that is problematic because the #accelerators per RP can change for multi-function devices (vGPUs, FPGAs, etc) | |
| 19:06:45 | dansmith | efried: but nova does a show on the port via uuid, and gets back a bunch of info, including the type, details, etc | |
| 19:07:08 | dansmith | Sundar: no, I mean a uuid for each piece you want the user to be able to attach | |
| 19:07:21 | dansmith | Sundar: not a uuid per accelerator device (although you need that too) | |
| 19:07:58 | efried | In the placement model, I'm not sure if we actually need the UUID of the specific attachy thing. | |
| 19:08:12 | dansmith | efried: compute will | |
| 19:08:36 | dansmith | and scheduler will need the uuid of the accelerator itself, because that's the RP with inventory | |
| 19:08:44 | mriedem | bpoulos: which osc plugin makes the openstack secret store command available? | |
| 19:09:20 | efried | n-sch talks to placement and gets back an allocated (provider, RC, amount). That info somehow gets to the driver's plug method, and the plug method knows how to attach <amount> <RC>s from <provider> to the VM. | |
| 19:10:17 | efried | but if there's some preconfiguration step, like telling it a bitstream UUID, or QoS params, or whatever, then yeah, we do the port model where we tell cyborg to set it up first. Then it'll create the Acc* instance filled in with everything *except* the provider UUID. | |
| 19:10:31 | dansmith | right, so n-sch does the claim to get that allocation, then n-cond talks to cyborg and says "okay, I've been told that you have 1 FOO_THING on compute $host, please give me an attachment" | |
| 19:10:52 | dansmith | then compute can use that to ask cyborg for the real details, pci address, or whatever | |
| 19:11:25 | efried | "give me an attachment" - what's an attachment? | |
| 19:11:44 | dansmith | efried: the equivalent of a port in neutron | |
| 19:12:04 | dansmith | efried: it's not an actual thing, it just represents the configuration and link between some network and the instance | |
| 19:12:09 | efried | okay, so basically it's just saying "register this thing and give me a UUID". | |
| 19:12:44 | efried | So cyborg stuffs that UUID into the Acc* object and wings it back over RPC to n-cpu, which ultimately calls the plug method. | |
| 19:12:44 | bpoulos | mriedem: barbican | |
| 19:12:45 | dansmith | efried: well, the point at which you do that is cyborg's opportunity to reconfigure the actual thing, and potentially adjust inventory if it has to reserve some amount of something because it gave you a principal or whatever | |
| 19:12:59 | mriedem | bpoulos: python-barbicanclient? | |
| 19:13:06 | dansmith | efried: no, cyborg does not talk RPC to nova-anything | |
| 19:13:09 | bpoulos | mriedem: in devstack, you can add the line "enable_plugin barbican https://git.openstack.org/openstack/barbican" | |
| 19:13:25 | mriedem | but that's the server | |
| 19:13:26 | bpoulos | mriedem: yeah | |
| 19:13:27 | mriedem | i just need the client | |
| 19:13:28 | mriedem | ok | |
| 19:13:30 | mriedem | http://git.openstack.org/cgit/openstack/python-barbicanclient/tree/barbicanclient/osc_plugin.py | |
| 19:13:31 | mriedem | yeah | |
| 19:13:31 | efried | dansmith: And when you say "cyborg's opportunity" you really mean "the platform-specific code as invoked by cyborg" | |
| 19:13:36 | dansmith | efried: yes | |
| 19:13:49 | efried | dansmith: And that's a separate step from the actual 'plug' | |
| 19:13:53 | dansmith | efried: right | |
| 19:14:06 | efried | which happens from n-cpu, and again should call into the cyborg plugin | |
| 19:14:15 | dansmith | efried: because in libvirt land, we can't do the plug until we've started the vm and havesomething to actually plug (in the network case anyway) | |
| 19:14:30 | dansmith | efried: that plug step is in os-acc, that nova-compute calls yeah | |
| 19:14:37 | dansmith | or compute/$virt | |
| 19:15:08 | efried | Right, so today network plug is a ComputeDriver method, but tomorrow it would live in the VIFThingy.plugin class - did I get that part right? | |
| 19:15:22 | dansmith | it's mostly in VIFThingy.plug now | |
| 19:15:26 | dansmith | except for a few cases | |
| 19:15:38 | efried | oh, so PowerVM is already behind the curve on this? | |
| 19:15:53 | dansmith | I mean, there's still a driver plug, but it's just saying "call into VIFTHing.plug" | |
| 19:16:04 | dansmith | efried: I dunno, probably :) | |
| 19:16:19 | Sundar | dansmith: ""n-cond talks to cyborg" in the current flow, the request comes to nova compute, and that invokes os-acc. The os-acc can talk to Cyborg agent to get any programmming done if needed, and that in turn could invoke any driver, or plugin if you prefer | |
| 19:16:37 | dansmith | although I'm not sure it's the same for power, as you're talking to the hypervisor in API terms, whereas in libvirt VIFThingy is mucking with bridges | |
| 19:17:11 | dansmith | Sundar: sure | |
| 19:17:21 | dansmith | Sundar: that's kinda how we *want* os-brick to work | |
| 19:17:24 | dansmith | but it doesn't | |
| 19:17:59 | efried | dansmith: "mucking with bridges" by running linux commands through processutils? | |
| 19:18:08 | Sundar | From nova-compute, how do you handle cinder-wide ops before getting into individual drivers? | |
| 19:18:20 | dansmith | efried: yep | |
| 19:18:35 | dansmith | Sundar: we talk to cinder ourselves | |
| 19:18:42 | efried | dansmith: Okay, so that's platform-specific stuff that really ought to live in the drivers/plugins | |
| 19:18:54 | dansmith | efried: that's what the VIFThingy is for | |
| 19:19:08 | dansmith | efried: to hold that stuff, per type | |
| 19:19:29 | dansmith | efried: and so that some vendor with a whizbang thing can encapsulate their own setup and attachment stuff without having to be in tree | |
| 19:20:12 | dansmith | gdi, how did you trick me out of ignoring your requests to talk about os-vif? | |
| 19:20:52 | melwitt | does anyone know if https://blueprints.launchpad.net/nova/+spec/optional-placement-database got rolled into https://blueprints.launchpad.net/nova/+spec/placement-extract and if I can obsolete the former? | |
| 19:21:03 | Sundar | There are some important differences between os-acc and os-vif | |
| 19:21:46 | Sundar | With os-acc/Cyborg, we have 2 concepts: accelerators and 'attach handles' (e.g. PCI VFs) | |
| 19:22:00 | dansmith | Sundar: same as neutron and same as cinder | |
| 19:22:24 | dansmith | neutron has a network and a port, and cinder has a volume and an attach_id | |
| 19:22:29 | Sundar | The correspondence between them can be complex. For example, a device may support afixed number of PCI VFs say 32, but the #accelerators depends on what is programmed in the device | |
| 19:22:49 | dansmith | yes, we've been over this, I get it :) | |
| 19:22:50 | efried | melwitt: Those are different afaik | |
| 19:23:08 | melwitt | efried: okay, I made a note. gonna ask cdent tomorrow if I remember | |
| 19:23:16 | Sundar | The #accelerators may be fewer than than #handles (no issues), or more (in which case not all accelerators can be exposed together) | |
| 19:23:48 | dansmith | Sundar: on SRIOV devices, you have PFs and VFs and if you hand off a PF, the VF inventory changes | |
| 19:23:54 | Sundar | Also, #accelerators per RP can vary over time (vGPUs, FPGAs) | |
| 19:23:57 | dansmith | I think you're more similar than different | |
| 19:24:16 | efried | melwitt: The former is a prereq of the latter I believe. Optional-db has I think four patches, ready for review (I keep +2ing and they keep getting rebased) and we should be able to close it out in rocky. Placement-extract is going to be a longer-term thing, with some stuff merging in rocky, but the major stuff happening in Stein. | |
| 19:24:59 | melwitt | efried: yeah, that makes sense but I found the optional separate db patches commit messages link to placement-extract bp https://review.openstack.org/#/q/topic:bp/placement-extract+(status:open+OR+status:merged) | |
| 19:25:02 | Sundar | For os-vif, what is the equivalent of accelerators? Attach handles are similar to ports | |
| 19:25:24 | efried | melwitt: That's probably a rebase error - do the commit messages list the right bp? | |