| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-14 | |||
| 19:14:15 | dansmith | efried: because in libvirt land, we can't do the plug until we've started the vm and havesomething to actually plug (in the network case anyway) | |
| 19:14:30 | dansmith | efried: that plug step is in os-acc, that nova-compute calls yeah | |
| 19:14:37 | dansmith | or compute/$virt | |
| 19:15:08 | efried | Right, so today network plug is a ComputeDriver method, but tomorrow it would live in the VIFThingy.plugin class - did I get that part right? | |
| 19:15:22 | dansmith | it's mostly in VIFThingy.plug now | |
| 19:15:26 | dansmith | except for a few cases | |
| 19:15:38 | efried | oh, so PowerVM is already behind the curve on this? | |
| 19:15:53 | dansmith | I mean, there's still a driver plug, but it's just saying "call into VIFTHing.plug" | |
| 19:16:04 | dansmith | efried: I dunno, probably :) | |
| 19:16:19 | Sundar | dansmith: ""n-cond talks to cyborg" in the current flow, the request comes to nova compute, and that invokes os-acc. The os-acc can talk to Cyborg agent to get any programmming done if needed, and that in turn could invoke any driver, or plugin if you prefer | |
| 19:16:37 | dansmith | although I'm not sure it's the same for power, as you're talking to the hypervisor in API terms, whereas in libvirt VIFThingy is mucking with bridges | |
| 19:17:11 | dansmith | Sundar: sure | |
| 19:17:21 | dansmith | Sundar: that's kinda how we *want* os-brick to work | |
| 19:17:24 | dansmith | but it doesn't | |
| 19:17:59 | efried | dansmith: "mucking with bridges" by running linux commands through processutils? | |
| 19:18:08 | Sundar | From nova-compute, how do you handle cinder-wide ops before getting into individual drivers? | |
| 19:18:20 | dansmith | efried: yep | |
| 19:18:35 | dansmith | Sundar: we talk to cinder ourselves | |
| 19:18:42 | efried | dansmith: Okay, so that's platform-specific stuff that really ought to live in the drivers/plugins | |
| 19:18:54 | dansmith | efried: that's what the VIFThingy is for | |
| 19:19:08 | dansmith | efried: to hold that stuff, per type | |
| 19:19:29 | dansmith | efried: and so that some vendor with a whizbang thing can encapsulate their own setup and attachment stuff without having to be in tree | |
| 19:20:12 | dansmith | gdi, how did you trick me out of ignoring your requests to talk about os-vif? | |
| 19:20:52 | melwitt | does anyone know if https://blueprints.launchpad.net/nova/+spec/optional-placement-database got rolled into https://blueprints.launchpad.net/nova/+spec/placement-extract and if I can obsolete the former? | |
| 19:21:03 | Sundar | There are some important differences between os-acc and os-vif | |
| 19:21:46 | Sundar | With os-acc/Cyborg, we have 2 concepts: accelerators and 'attach handles' (e.g. PCI VFs) | |
| 19:22:00 | dansmith | Sundar: same as neutron and same as cinder | |
| 19:22:24 | dansmith | neutron has a network and a port, and cinder has a volume and an attach_id | |
| 19:22:29 | Sundar | The correspondence between them can be complex. For example, a device may support afixed number of PCI VFs say 32, but the #accelerators depends on what is programmed in the device | |
| 19:22:49 | dansmith | yes, we've been over this, I get it :) | |
| 19:22:50 | efried | melwitt: Those are different afaik | |
| 19:23:08 | melwitt | efried: okay, I made a note. gonna ask cdent tomorrow if I remember | |
| 19:23:16 | Sundar | The #accelerators may be fewer than than #handles (no issues), or more (in which case not all accelerators can be exposed together) | |
| 19:23:48 | dansmith | Sundar: on SRIOV devices, you have PFs and VFs and if you hand off a PF, the VF inventory changes | |
| 19:23:54 | Sundar | Also, #accelerators per RP can vary over time (vGPUs, FPGAs) | |
| 19:23:57 | dansmith | I think you're more similar than different | |
| 19:24:16 | efried | melwitt: The former is a prereq of the latter I believe. Optional-db has I think four patches, ready for review (I keep +2ing and they keep getting rebased) and we should be able to close it out in rocky. Placement-extract is going to be a longer-term thing, with some stuff merging in rocky, but the major stuff happening in Stein. | |
| 19:24:59 | melwitt | efried: yeah, that makes sense but I found the optional separate db patches commit messages link to placement-extract bp https://review.openstack.org/#/q/topic:bp/placement-extract+(status:open+OR+status:merged) | |
| 19:25:02 | Sundar | For os-vif, what is the equivalent of accelerators? Attach handles are similar to ports | |
| 19:25:24 | efried | melwitt: That's probably a rebase error - do the commit messages list the right bp? | |
| 19:25:37 | melwitt | gah, this one actually writes both https://review.openstack.org/362766 | |
| 19:25:37 | dansmith | Sundar: network or subnet, which has a finite number of attachments that can vary independently of ports | |
| 19:25:48 | dansmith | Sundar: and even more complicated if the port represents an SRIOV VF or PF | |
| 19:26:06 | efried | similarly with GPU vs VGPU passthrough ^ | |
| 19:26:20 | dansmith | yup | |
| 19:26:52 | efried | or maybe function vs region vs whole-FPGA | |
| 19:27:08 | dansmith | or function that takes the RJ45 with it | |
| 19:27:25 | dansmith | i.e. you could do 4 gzips, but only one SSL that needs the network | |
| 19:27:47 | efried | anyway, I gather we're looking at getting the accelerator thingy in the same way we get a port today: i.e. by precreating it through cyborg, NOT by specifying it along with the other resources in flavor extra_specs. | |
| 19:28:05 | dansmith | efried: that's easier, and a good place to start | |
| 19:28:13 | efried | and then we send it to the nove boot command same as we do with ports today. | |
| 19:28:19 | efried | s/nove/nova/ | |
| 19:28:26 | Sundar | Yes, accelerators are RCs which are pre-created. | |
| 19:28:37 | dansmith | nova still has to claim it in placement, but then only nova-compute needs to care about the rest | |
| 19:29:01 | efried | dansmith: Does the claim happen before or after the callout to the "prepare" thing? | |
| 19:29:02 | Sundar | We do not represent PCI PFs/VFs as RCs today, to avoid being PCI-centirc | |
| 19:29:22 | efried | Sundar: You can still represent PFs/VFs without them having to be PCI | |
| 19:29:23 | dansmith | efried: it needs to happen when we do the allocation for the instance | |
| 19:29:43 | dansmith | efried: once we know which host | |
| 19:29:47 | efried | dansmith: after the allocation itself | |
| 19:30:18 | efried | Or before the allocation, as part of testing for viable candidates? | |
| 19:30:20 | dansmith | efried: well, "the allocation" is the wrong thing to call the pre-create I think, because cyborg can't do a whole lot at that point, because it has no idea what host you're going to land on | |
| 19:32:16 | efried | dansmith: This is what I'm asking. By "allocation" I'm specifically talking about the placement allocation. Does the pre-create happen while we're still considering allocation candidates (which I guess would involve n-cond => n-cpu => os-acc => plugin and waiting for a response) and allow us to discard some of them, or does it happen after the allocation and if it fails we have to move on to our next of three tries within | |
| 19:33:05 | efried | Reading between the lines of your answers above I'm guessing it's the latter. | |
| 19:33:22 | dansmith | efried: when you said pre-create in cyborg, I assumed you meant before you do boot | |
| 19:33:48 | dansmith | in which case, it can't create an allocation in placement for you, because it doesn't know what RP to use (and more importantly, the compute node parent of the RP) | |
| 19:33:50 | efried | dansmith: I guess so, because boot includes plug, which has to come later. | |
| 19:34:56 | Sundar | How about this model? Accelerators are RCs, pre-published to Nova. Scheduling works with known RPs, traits and RCs, but attach handles are not part of it. Cyborg alone deals with attach handles. When a request comes to n-cpu, one or more attach handles get picked (nova virt either calls a plugin, or calls Cyborg agent, which int turn calls a Cyborg driver/plugin). The pre-create would be done by the plugin | |
| 19:35:47 | dansmith | um | |
| 19:35:51 | dansmith | I'm not sure what to do with that | |
| 19:36:07 | dansmith | Sundar: IMHO, cyborg or os-acc should not be mucking with the instance's allocation in placement | |
| 19:36:25 | Sundar | Yes, it does not | |
| 19:36:40 | dansmith | the discussion above is not about attach handles | |
| 19:36:53 | Sundar | Cyborg publishes RPs etc to placement. Nova/plcaement do all the allocation of candidates | |
| 19:36:58 | openstackgerrit | Brianna Poulos proposed openstack/python-novaclient master: Microversion 2.63 - Add trusted_image_certificates https://review.openstack.org/500396 | |
| 19:37:53 | Sundar | My difficulty is in relating attach handles to accelerators, and making sure all possibilities and use cases are covered :) | |
| 19:38:11 | dansmith | Sundar: I'm confused.. the things you're saying are true and not really negotiable, but it sounds like you're suggesting something else | |
| 19:38:34 | Sundar | Sorry, let me try again | |
| 19:38:44 | Sundar | Consider this flow: | |
| 19:39:52 | Sundar | A request comes to n-api. The n-sch works works with placement to decide allocation candidates and choose a compute node. The request then comes to n-cpu in the compute node. No Cyborg so far (except for populating Placement at the start). is this good so far? | |
| 19:40:26 | efried | cyborg was involved in creating the artifact to begin with. I think we're calling this an "attachment". | |
| 19:40:46 | efried | Sundar: How about this, let me draw up the flow as I think I understand it and then we'll have a starting point for discussion. | |
| 19:41:08 | efried | dansmith: What's this pre-create thing called in the neutron flow? | |
| 19:41:17 | dansmith | create port? | |
| 19:41:17 | Sundar | efried: Cyborg publishes accelerators as RCs. You are saying attach handles (like PCI VFs or Power location codes) should also be published? | |
| 19:42:04 | efried | Sundar: I'm saying the user does a "openstack create accelerator-attachment <details>". Cyborg creates what you're calling an "attachment" and returns its UUID. | |
| 19:42:23 | efried | Sundar: Then the user does "openstack server create <details> --accelerator <that UUID>" | |
| 19:43:04 | Sundar | efired: why is that needed? For a edevice, the #attachments can be discovered automatically., unlike Neutron ports whose # is not fixed or discoverable | |
| 19:43:14 | efried | This gets to n-sch, which asks cyborg for the Acc* object corresponding to the attachment UUID. Cyborg pulls the original profile info out of its database and feeds it back. | |
| 19:43:24 | dansmith | efried: and nova uses that id to ask cyborg what resources it should ask for based on what accelerator the user asked for | |
| 19:43:26 | Sundar | E.g. for a SR-IOV device, | |
| 19:43:30 | dansmith | efried: and include that in the allocation | |
| 19:43:46 | Sundar | the #VFs is known at device discovery time | |
| 19:43:52 | dansmith | it gets that claim, and then can immediately tell cyborg "okay, I just committed $foo to $user on host $bar, cool?" | |
| 19:44:08 | dansmith | and if cyborg says yes, then cyborg reswizzles its inventory if needed, and compute gets started | |
| 19:44:09 | efried | Right, n-sch converts the Acc* object into placement query material to be included in the GET /a-c query. | |
| 19:44:26 | dansmith | when ready, it asks cyborg for "okay how are we going to attach this mofo?", which might be os-acc, and then we're off | |
| 19:46:04 | efried | Okay, I think dansmith and I are on the same page. So dansmith in your view, the plugin needs to have methods for plug(), unplug(), and this prepare_for_plug() thing? | |
| 19:46:24 | dansmith | probablyish | |
| 19:46:42 | dansmith | it could also have the "tell cyborg we chose a host for attachment $foo", which n-cond could use in the early part | |