| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-14 | |||
| 19:03:55 | efried | If we're following the os-vif model, they've got a field for a "port profile". They allow any number of those to be registered in os-vif, not sure if they're also allowed to be registered via the plugin packages. | |
| 19:04:40 | dansmith | if you follow the neutron model, you give nova a port uuid which you got from neutron, and nova can ask neutron what the vif_type is, if it needs that | |
| 19:05:54 | efried | dansmith: How does it go from that to a specific plugin? | |
| 19:06:10 | dansmith | well, maybe I'm mixing the external user into the workflow and you're just talking about the nova->cyborg bit | |
| 19:06:35 | Sundar | dansmith: The equiv of a port UUID would be a UUID for each accelerator, but that is problematic because the #accelerators per RP can change for multi-function devices (vGPUs, FPGAs, etc) | |
| 19:06:45 | dansmith | efried: but nova does a show on the port via uuid, and gets back a bunch of info, including the type, details, etc | |
| 19:07:08 | dansmith | Sundar: no, I mean a uuid for each piece you want the user to be able to attach | |
| 19:07:21 | dansmith | Sundar: not a uuid per accelerator device (although you need that too) | |
| 19:07:58 | efried | In the placement model, I'm not sure if we actually need the UUID of the specific attachy thing. | |
| 19:08:12 | dansmith | efried: compute will | |
| 19:08:36 | dansmith | and scheduler will need the uuid of the accelerator itself, because that's the RP with inventory | |
| 19:08:44 | mriedem | bpoulos: which osc plugin makes the openstack secret store command available? | |
| 19:09:20 | efried | n-sch talks to placement and gets back an allocated (provider, RC, amount). That info somehow gets to the driver's plug method, and the plug method knows how to attach <amount> <RC>s from <provider> to the VM. | |
| 19:10:17 | efried | but if there's some preconfiguration step, like telling it a bitstream UUID, or QoS params, or whatever, then yeah, we do the port model where we tell cyborg to set it up first. Then it'll create the Acc* instance filled in with everything *except* the provider UUID. | |
| 19:10:31 | dansmith | right, so n-sch does the claim to get that allocation, then n-cond talks to cyborg and says "okay, I've been told that you have 1 FOO_THING on compute $host, please give me an attachment" | |
| 19:10:52 | dansmith | then compute can use that to ask cyborg for the real details, pci address, or whatever | |
| 19:11:25 | efried | "give me an attachment" - what's an attachment? | |
| 19:11:44 | dansmith | efried: the equivalent of a port in neutron | |
| 19:12:04 | dansmith | efried: it's not an actual thing, it just represents the configuration and link between some network and the instance | |
| 19:12:09 | efried | okay, so basically it's just saying "register this thing and give me a UUID". | |
| 19:12:44 | bpoulos | mriedem: barbican | |
| 19:12:44 | efried | So cyborg stuffs that UUID into the Acc* object and wings it back over RPC to n-cpu, which ultimately calls the plug method. | |
| 19:12:45 | dansmith | efried: well, the point at which you do that is cyborg's opportunity to reconfigure the actual thing, and potentially adjust inventory if it has to reserve some amount of something because it gave you a principal or whatever | |
| 19:12:59 | mriedem | bpoulos: python-barbicanclient? | |
| 19:13:06 | dansmith | efried: no, cyborg does not talk RPC to nova-anything | |
| 19:13:09 | bpoulos | mriedem: in devstack, you can add the line "enable_plugin barbican https://git.openstack.org/openstack/barbican" | |
| 19:13:25 | mriedem | but that's the server | |
| 19:13:26 | bpoulos | mriedem: yeah | |
| 19:13:27 | mriedem | i just need the client | |
| 19:13:28 | mriedem | ok | |
| 19:13:30 | mriedem | http://git.openstack.org/cgit/openstack/python-barbicanclient/tree/barbicanclient/osc_plugin.py | |
| 19:13:31 | efried | dansmith: And when you say "cyborg's opportunity" you really mean "the platform-specific code as invoked by cyborg" | |
| 19:13:31 | mriedem | yeah | |
| 19:13:36 | dansmith | efried: yes | |
| 19:13:49 | efried | dansmith: And that's a separate step from the actual 'plug' | |
| 19:13:53 | dansmith | efried: right | |
| 19:14:06 | efried | which happens from n-cpu, and again should call into the cyborg plugin | |
| 19:14:15 | dansmith | efried: because in libvirt land, we can't do the plug until we've started the vm and havesomething to actually plug (in the network case anyway) | |
| 19:14:30 | dansmith | efried: that plug step is in os-acc, that nova-compute calls yeah | |
| 19:14:37 | dansmith | or compute/$virt | |
| 19:15:08 | efried | Right, so today network plug is a ComputeDriver method, but tomorrow it would live in the VIFThingy.plugin class - did I get that part right? | |
| 19:15:22 | dansmith | it's mostly in VIFThingy.plug now | |
| 19:15:26 | dansmith | except for a few cases | |
| 19:15:38 | efried | oh, so PowerVM is already behind the curve on this? | |
| 19:15:53 | dansmith | I mean, there's still a driver plug, but it's just saying "call into VIFTHing.plug" | |
| 19:16:04 | dansmith | efried: I dunno, probably :) | |
| 19:16:19 | Sundar | dansmith: ""n-cond talks to cyborg" in the current flow, the request comes to nova compute, and that invokes os-acc. The os-acc can talk to Cyborg agent to get any programmming done if needed, and that in turn could invoke any driver, or plugin if you prefer | |
| 19:16:37 | dansmith | although I'm not sure it's the same for power, as you're talking to the hypervisor in API terms, whereas in libvirt VIFThingy is mucking with bridges | |
| 19:17:11 | dansmith | Sundar: sure | |
| 19:17:21 | dansmith | Sundar: that's kinda how we *want* os-brick to work | |
| 19:17:24 | dansmith | but it doesn't | |
| 19:17:59 | efried | dansmith: "mucking with bridges" by running linux commands through processutils? | |
| 19:18:08 | Sundar | From nova-compute, how do you handle cinder-wide ops before getting into individual drivers? | |
| 19:18:20 | dansmith | efried: yep | |
| 19:18:35 | dansmith | Sundar: we talk to cinder ourselves | |
| 19:18:42 | efried | dansmith: Okay, so that's platform-specific stuff that really ought to live in the drivers/plugins | |
| 19:18:54 | dansmith | efried: that's what the VIFThingy is for | |
| 19:19:08 | dansmith | efried: to hold that stuff, per type | |
| 19:19:29 | dansmith | efried: and so that some vendor with a whizbang thing can encapsulate their own setup and attachment stuff without having to be in tree | |
| 19:20:12 | dansmith | gdi, how did you trick me out of ignoring your requests to talk about os-vif? | |
| 19:20:52 | melwitt | does anyone know if https://blueprints.launchpad.net/nova/+spec/optional-placement-database got rolled into https://blueprints.launchpad.net/nova/+spec/placement-extract and if I can obsolete the former? | |
| 19:21:03 | Sundar | There are some important differences between os-acc and os-vif | |
| 19:21:46 | Sundar | With os-acc/Cyborg, we have 2 concepts: accelerators and 'attach handles' (e.g. PCI VFs) | |
| 19:22:00 | dansmith | Sundar: same as neutron and same as cinder | |
| 19:22:24 | dansmith | neutron has a network and a port, and cinder has a volume and an attach_id | |
| 19:22:29 | Sundar | The correspondence between them can be complex. For example, a device may support afixed number of PCI VFs say 32, but the #accelerators depends on what is programmed in the device | |
| 19:22:49 | dansmith | yes, we've been over this, I get it :) | |
| 19:22:50 | efried | melwitt: Those are different afaik | |
| 19:23:08 | melwitt | efried: okay, I made a note. gonna ask cdent tomorrow if I remember | |
| 19:23:16 | Sundar | The #accelerators may be fewer than than #handles (no issues), or more (in which case not all accelerators can be exposed together) | |
| 19:23:48 | dansmith | Sundar: on SRIOV devices, you have PFs and VFs and if you hand off a PF, the VF inventory changes | |
| 19:23:54 | Sundar | Also, #accelerators per RP can vary over time (vGPUs, FPGAs) | |
| 19:23:57 | dansmith | I think you're more similar than different | |
| 19:24:16 | efried | melwitt: The former is a prereq of the latter I believe. Optional-db has I think four patches, ready for review (I keep +2ing and they keep getting rebased) and we should be able to close it out in rocky. Placement-extract is going to be a longer-term thing, with some stuff merging in rocky, but the major stuff happening in Stein. | |
| 19:24:59 | melwitt | efried: yeah, that makes sense but I found the optional separate db patches commit messages link to placement-extract bp https://review.openstack.org/#/q/topic:bp/placement-extract+(status:open+OR+status:merged) | |
| 19:25:02 | Sundar | For os-vif, what is the equivalent of accelerators? Attach handles are similar to ports | |
| 19:25:24 | efried | melwitt: That's probably a rebase error - do the commit messages list the right bp? | |
| 19:25:37 | melwitt | gah, this one actually writes both https://review.openstack.org/362766 | |
| 19:25:37 | dansmith | Sundar: network or subnet, which has a finite number of attachments that can vary independently of ports | |
| 19:25:48 | dansmith | Sundar: and even more complicated if the port represents an SRIOV VF or PF | |
| 19:26:06 | efried | similarly with GPU vs VGPU passthrough ^ | |
| 19:26:20 | dansmith | yup | |
| 19:26:52 | efried | or maybe function vs region vs whole-FPGA | |
| 19:27:08 | dansmith | or function that takes the RJ45 with it | |
| 19:27:25 | dansmith | i.e. you could do 4 gzips, but only one SSL that needs the network | |
| 19:27:47 | efried | anyway, I gather we're looking at getting the accelerator thingy in the same way we get a port today: i.e. by precreating it through cyborg, NOT by specifying it along with the other resources in flavor extra_specs. | |
| 19:28:05 | dansmith | efried: that's easier, and a good place to start | |
| 19:28:13 | efried | and then we send it to the nove boot command same as we do with ports today. | |
| 19:28:19 | efried | s/nove/nova/ | |
| 19:28:26 | Sundar | Yes, accelerators are RCs which are pre-created. | |
| 19:28:37 | dansmith | nova still has to claim it in placement, but then only nova-compute needs to care about the rest | |
| 19:29:01 | efried | dansmith: Does the claim happen before or after the callout to the "prepare" thing? | |
| 19:29:02 | Sundar | We do not represent PCI PFs/VFs as RCs today, to avoid being PCI-centirc | |
| 19:29:22 | efried | Sundar: You can still represent PFs/VFs without them having to be PCI | |
| 19:29:23 | dansmith | efried: it needs to happen when we do the allocation for the instance | |
| 19:29:43 | dansmith | efried: once we know which host | |
| 19:29:47 | efried | dansmith: after the allocation itself | |
| 19:30:18 | efried | Or before the allocation, as part of testing for viable candidates? | |
| 19:30:20 | dansmith | efried: well, "the allocation" is the wrong thing to call the pre-create I think, because cyborg can't do a whole lot at that point, because it has no idea what host you're going to land on | |
| 19:32:16 | efried | dansmith: This is what I'm asking. By "allocation" I'm specifically talking about the placement allocation. Does the pre-create happen while we're still considering allocation candidates (which I guess would involve n-cond => n-cpu => os-acc => plugin and waiting for a response) and allow us to discard some of them, or does it happen after the allocation and if it fails we have to move on to our next of three tries within | |