| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-28 | |||
| 15:00:49 | efried | Sundar: It seems reasonable, if nova only needs to talk to the cyborg API, to nix os-acc. | |
| 15:01:10 | jaypipes | Sundar: I was under the impression os-acc would function similarly to os-vif, where os-vif provides the data models that are used to understand what and how network interfaces are brought up on the host. | |
| 15:01:34 | efried | Sundar: My impression of os-acc was that it would be the clearinghouse for platform- and virt-specific plugins to do discovery and attachment. | |
| 15:01:39 | efried | yeah, what jaypipes said. | |
| 15:02:17 | Sundar | jaypipes: Yes, that's how we started. As the discussion progressed, and the spec was reviewed, it became clear that most of the work would be done by Cyborg APIs. | |
| 15:02:36 | jaypipes | Sundar: which I completely disagree with. | |
| 15:02:54 | jaypipes | Sundar: I have continued to maintain I see very little use in a Cyborg REST API at the moment. | |
| 15:03:30 | mriedem | gmann: yes https://review.openstack.org/#/c/604134/1/.zuul.yaml is a bad backport | |
| 15:03:54 | jaypipes | Sundar: let me tell you why. | |
| 15:04:06 | mriedem | gmann: the commit message in https://review.openstack.org/#/c/541474/ sounds like we don't need it in queens | |
| 15:04:18 | mriedem | the queens comment in https://github.com/openstack/nova/blob/master/.zuul.yaml#L140 might be wrong | |
| 15:04:35 | jaypipes | Sundar: focusing on the REST API in Cyborg right now means you are constrained by versioning in a way that just focusing on iterating a workable object/data model would not constrain you. | |
| 15:04:37 | Sundar | jaypipes: Without Cyborg APIs, how would one initiate device configuration, whether it is for a GPU, FPGA, ...? | |
| 15:04:40 | mriedem | gmann: https://review.openstack.org/#/c/396186/ isn't in queens | |
| 15:04:47 | jaypipes | Sundar: IMHO, a REST API is just a giant distraction right now for Cyborg. | |
| 15:05:09 | efried | jaypipes: The way I see it, we need a way to follow the thread of an ARQ from [set of resources and traits] to [actual accelerator instance plugged into a VM]. | |
| 15:05:11 | jaypipes | Sundar: what does Cyborg do "under the covers" in its drivers? | |
| 15:05:20 | jaypipes | Sundar: *that* is what I think os-acc should be doing. | |
| 15:06:06 | dansmith | bauzas: gonna circle back on https://review.openstack.org/#/c/599587 right? | |
| 15:06:17 | efried | jaypipes: Nova is going to be driving that workflow, but cyborg needs to be the one to make some of those transitions (like, "I landed on a host; pick me a specific device and configure it") and needs to be made aware of the ones Nova makes (like, "I plugged accelerator X into VM Y"). | |
| 15:06:27 | gmann | mriedem: ok. then we can remove the nova-next from queen pipeline list also + that job comment fix. i observed it while backporting the nova v2 job. | |
| 15:06:31 | Sundar | jaypipes: Are you saying that os-acc should have its own device-specific drivers and run the show by itself? Or that it should call into Cyborg drivers, without involving any REST API? | |
| 15:06:34 | bauzas | dansmith: yup, I'm just reviewing it | |
| 15:06:35 | efried | jaypipes: *some* kind of API seems like the appropriate way to do that. | |
| 15:06:40 | dansmith | bauzas: awesome, thanks | |
| 15:06:57 | jaypipes | Sundar: that is correct. | |
| 15:07:07 | mriedem | gmann: yeah it's either that or backport the devstack and nova change | |
| 15:07:41 | jaypipes | Sundar: I've said a number of times that I wish Cyborg would just accept that Nova is its sole consumer right now, stop working on some "stand-alone Cyborg" thing and just function like a Nova virt driver for right now. That way, you can iterate more quickly, actually get something that works (as opposed to just abstract, non-working things) and go from ther | |
| 15:08:08 | gmann | mriedem: i can give backport try also if you think it is worth to run on queens. | |
| 15:09:07 | mriedem | gmann: i backported the devstack change, you can backport https://review.openstack.org/#/c/513160/ if you want | |
| 15:09:10 | mriedem | it has merge conflicts | |
| 15:09:21 | mriedem | i'm guessing the nova-next/devstack testing just missed the queens GA | |
| 15:09:29 | mriedem | but the feature itself is in nova in queens | |
| 15:09:33 | mriedem | so it would probably be good to test it... | |
| 15:09:50 | Sundar | jaypipes: There are are several use cases for accelerators without direct Nova involvement. For example, we may want to set up OVS offload in an accelerator in a host, but that doesn't involve assigning to a VM. | |
| 15:10:05 | jaypipes | Sundar: there might be, yes. I' | |
| 15:10:28 | jaypipes | m saying I don't care about those use cases and I feel they are a giant distraction from getting anything working in Nova/Cyborg at the moment. | |
| 15:11:23 | gmann | mriedem: +1. will do tomorrow. | |
| 15:11:30 | mriedem | gmann: thanks | |
| 15:11:31 | sean-k-mooney | Sundar: "use cases for accelerators without direct Nova involvement" are by definition out side the scope of the nova/cyborge interaction sepc | |
| 15:11:42 | efried | jaypipes: Are you suggesting maintaining ARQ data in nova databases and manipulating them via OVOs within the nova code? | |
| 15:12:07 | jaypipes | Sundar: let me put this in the most direct way possible... I do not think it's appropriate to call a REST API to set up a local device on a host. | |
| 15:12:58 | mriedem | sounds like you want a thing that's like PlacementDirect | |
| 15:13:07 | mriedem | API interface is the same, but it doesn't go over http | |
| 15:13:31 | Sundar | jaypipes: As efried said above, manipulating devices from a virt driver means the device data is in Nova db. Is that what you are advocating? | |
| 15:14:13 | Sundar | sean-k-mooney: We were talking about stand-alone Cyborg | |
| 15:15:51 | jaypipes | Sundar: I would rather just get the small amount of code that is Cyborg's accelerators/drivers/ module (https://github.com/openstack/cyborg/blob/master/cyborg/accelerator/drivers/fpga/intel/driver.py), pull it into some library called "os-acc" and call it directly. | |
| 15:16:49 | jaypipes | Sundar: especially since it's basically just shelling out to some unknown /usr/bin/fpgaconf program which I assume is some Intel-specific binary | |
| 15:17:48 | jaypipes | Sundar: which is essentially what os-vif is, BTW... it just shells out to Linux binaries like "ip" or "ovsctl" etc. | |
| 15:18:00 | jaypipes | Sundar: which is why I say I'd like an os-vif for accelerators... | |
| 15:21:03 | Sundar | jaypipes: That means all the PCI details (#PFs, #VFs, etc.) and other data for programming devices etc. are all in Nova db. | |
| 15:21:29 | jaypipes | Sundar: which is exactly where they already are. | |
| 15:21:44 | jaypipes | Sundar: because Nova is the thing that owns compute node resources. | |
| 15:22:39 | efried | jaypipes: Please tell me you're not suggesting using/augmenting the existing database schemata | |
| 15:23:05 | Sundar | jaypipes: I thought there was a desire to pull all that complexity into a separate project. What do you see as the role for Cyborg in your model? | |
| 15:23:07 | sean-k-mooney | efried: that is a diffent topic | |
| 15:23:14 | sean-k-mooney | you could be we can do better | |
| 15:23:46 | efried | sean-k-mooney: It is exactly the topic. | |
| 15:23:50 | jaypipes | Sundar: on a sidenote, could you tell me where I can find the source code for fpgaconf? I can't seem to locate it.. | |
| 15:24:14 | jaypipes | efried: no, I'm not. I'm just saying that's where we *already* store this information. | |
| 15:24:41 | sean-k-mooney | efried: you can passthough gpugs and fpgas with nova pci pathough today | |
| 15:24:41 | efried | sean-k-mooney: We know we want the existing pci subsystem to diaf. Trying to retrofit it for cyborg purposes will just make it live on, like something from Walking Dead. | |
| 15:24:53 | sean-k-mooney | efried: we just cant program fpgas | |
| 15:25:26 | efried | Also, we know we want to separate eventually, so the tighter the integration with nova databases, the more painful that will be. | |
| 15:25:29 | artom | sean-k-mooney, yo, check downstream IRC, I'd like to skip our meeting since very few are around | |
| 15:25:39 | efried | As with placement, we should at the very least make it a separate database. | |
| 15:25:49 | jaypipes | Sundar: there is definitely a desire to standardize and clean up the mess that is the PCI device management code (and CPU pinning, NUMA topology, etc) code in Nova. I have never had a desire to create a new REST service to manage this data, however. | |
| 15:26:05 | efried | but cyborg is already a separate project, and already has existing use cases for operation and tracking/programming of devices independent of nova | |
| 15:26:11 | Sundar | jaypipes: While OPAE SDK has been released i github, I am not sure that all tools like fpgaconf got open sourced. I'll check and get back. | |
| 15:26:14 | efried | so why wouldn't we just put the new database there to begin with? | |
| 15:26:22 | jaypipes | Sundar: thx | |
| 15:28:00 | jaypipes | efried: perhaps there is. I'm trying to say that I think it's premature to do RESTful stuff versus cleaning up the device management and plugging/assignment code in nova. | |
| 15:29:24 | jaypipes | efried: I'm straining to think of a use case where I'd want programmatic listing and showing of device details across all my compute infrastructure. Other than just knowing what devices are there and what resources they provide (which is already taken care of by the placement service), I'm wondering what the use is of a REST API over this data. | |
| 15:31:59 | Sundar | jaypipes: if you were to apply that to networking, would you not want to see the details of the NICs, and manage their diversity in a structured way? | |
| 15:32:57 | jaypipes | Sundar: if you're talking about capabilities of NICs or bandwidth of physnets on NICs, we already have that in the placement DB. | |
| 15:33:39 | jangutter | Sundar: commenting as an engineer working for a NIC vendor, customers broadly fall into two categories: "Make everything look the same" and "handle this node specially". | |
| 15:34:35 | jangutter | Sundar: it would be nice to be able to say "provide me with a networking resource capable of _x_", but broadly, most requests to us have been to make things more uniform, not less. | |
| 15:35:37 | Sundar | jaypipes: jangutter: My point is, just as we have Neutron for networking etc., the diversity of accelerators is best handled by a separate project, rather than fold all in Nova. Esp. since there are use cases that don;t involve Nova, even if you don't agree with it. | |
| 15:36:58 | mriedem | as a casual observer, it sounds like the dilemma is doing something quick and dirty to get something done without a cyborg REST API, vs make a perfect external system that nova can leverage, which would likely delay this even longer | |
| 15:37:17 | mriedem | and if the former is done, how complicated is the extraction/decoupling later | |
| 15:37:28 | dansmith | not sure it's a dilemma, | |
| 15:37:32 | mriedem | and i think what was mentioned last week, | |
| 15:37:36 | dansmith | as I think only jaypipes is the one that feels it should go that way | |
| 15:37:47 | mriedem | was someone could already be doing a PoC for the former to see how it looks so we have an informed decision to make | |
| 15:37:47 | dansmith | (AFAICT at least) | |
| 15:38:02 | mriedem | as the latter is much more work i'd think | |
| 15:38:05 | dansmith | mriedem: yup, I think we probably all agree on that point | |
| 15:38:35 | dansmith | well, except maybe Sundar :) | |
| 15:39:23 | Sundar | mriedem: dansmith: I am working on a POC, a simple one focusing only on the Nova - Cyborg calls. It needs official clearance within my company before I can share it | |
| 15:39:59 | mriedem | the point of the rest api was so that the data and drivers live within cyborg, not nova, and nova hits those drivers over the API | |
| 15:40:06 | mriedem | if we could put a shim in like PlacementDirect, great | |
| 15:40:07 | dansmith | Sundar: yeah that's not going to work very well for collab, but.. good luck :/ | |
| 15:40:28 | jangutter | Sundar: I don't think that anybody disputes that it's better to have a separate entity to handle out-of-Nova scope things. The trick is that Nova already has a way to pass PCI devices to instances. There's also existing code in os-vif that gets called "on plug events". | |
| 15:40:32 | mriedem | we talked about all of this last week when we said we didn't need an os-acc library and could just use python-cyborgclient | |
| 15:40:34 | sean-k-mooney | dansmith: well i have some leaning towrad a libary solution too | |
| 15:40:46 | mriedem | and i was also confused why we needed to hit a rest api to get to the cyborg drivers, i.e. why it wasn't more like os-brick | |
| 15:41:13 | dansmith | I'm fine with the programming part being in a library instead of an agent if we want | |
| 15:41:37 | dansmith | mriedem: it's quite a bit more complicated than just manipulating standard system things to connect to an iscsi target | |
| 15:41:51 | mriedem | right that was explained last week | |
| 15:41:54 | mriedem | probably about this time :) | |