| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-28 | |||
| 15:13:07 | mriedem | API interface is the same, but it doesn't go over http | |
| 15:13:31 | Sundar | jaypipes: As efried said above, manipulating devices from a virt driver means the device data is in Nova db. Is that what you are advocating? | |
| 15:14:13 | Sundar | sean-k-mooney: We were talking about stand-alone Cyborg | |
| 15:15:51 | jaypipes | Sundar: I would rather just get the small amount of code that is Cyborg's accelerators/drivers/ module (https://github.com/openstack/cyborg/blob/master/cyborg/accelerator/drivers/fpga/intel/driver.py), pull it into some library called "os-acc" and call it directly. | |
| 15:16:49 | jaypipes | Sundar: especially since it's basically just shelling out to some unknown /usr/bin/fpgaconf program which I assume is some Intel-specific binary | |
| 15:17:48 | jaypipes | Sundar: which is essentially what os-vif is, BTW... it just shells out to Linux binaries like "ip" or "ovsctl" etc. | |
| 15:18:00 | jaypipes | Sundar: which is why I say I'd like an os-vif for accelerators... | |
| 15:21:03 | Sundar | jaypipes: That means all the PCI details (#PFs, #VFs, etc.) and other data for programming devices etc. are all in Nova db. | |
| 15:21:29 | jaypipes | Sundar: which is exactly where they already are. | |
| 15:21:44 | jaypipes | Sundar: because Nova is the thing that owns compute node resources. | |
| 15:22:39 | efried | jaypipes: Please tell me you're not suggesting using/augmenting the existing database schemata | |
| 15:23:05 | Sundar | jaypipes: I thought there was a desire to pull all that complexity into a separate project. What do you see as the role for Cyborg in your model? | |
| 15:23:07 | sean-k-mooney | efried: that is a diffent topic | |
| 15:23:14 | sean-k-mooney | you could be we can do better | |
| 15:23:46 | efried | sean-k-mooney: It is exactly the topic. | |
| 15:23:50 | jaypipes | Sundar: on a sidenote, could you tell me where I can find the source code for fpgaconf? I can't seem to locate it.. | |
| 15:24:14 | jaypipes | efried: no, I'm not. I'm just saying that's where we *already* store this information. | |
| 15:24:41 | sean-k-mooney | efried: you can passthough gpugs and fpgas with nova pci pathough today | |
| 15:24:41 | efried | sean-k-mooney: We know we want the existing pci subsystem to diaf. Trying to retrofit it for cyborg purposes will just make it live on, like something from Walking Dead. | |
| 15:24:53 | sean-k-mooney | efried: we just cant program fpgas | |
| 15:25:26 | efried | Also, we know we want to separate eventually, so the tighter the integration with nova databases, the more painful that will be. | |
| 15:25:29 | artom | sean-k-mooney, yo, check downstream IRC, I'd like to skip our meeting since very few are around | |
| 15:25:39 | efried | As with placement, we should at the very least make it a separate database. | |
| 15:25:49 | jaypipes | Sundar: there is definitely a desire to standardize and clean up the mess that is the PCI device management code (and CPU pinning, NUMA topology, etc) code in Nova. I have never had a desire to create a new REST service to manage this data, however. | |
| 15:26:05 | efried | but cyborg is already a separate project, and already has existing use cases for operation and tracking/programming of devices independent of nova | |
| 15:26:11 | Sundar | jaypipes: While OPAE SDK has been released i github, I am not sure that all tools like fpgaconf got open sourced. I'll check and get back. | |
| 15:26:14 | efried | so why wouldn't we just put the new database there to begin with? | |
| 15:26:22 | jaypipes | Sundar: thx | |
| 15:28:00 | jaypipes | efried: perhaps there is. I'm trying to say that I think it's premature to do RESTful stuff versus cleaning up the device management and plugging/assignment code in nova. | |
| 15:29:24 | jaypipes | efried: I'm straining to think of a use case where I'd want programmatic listing and showing of device details across all my compute infrastructure. Other than just knowing what devices are there and what resources they provide (which is already taken care of by the placement service), I'm wondering what the use is of a REST API over this data. | |
| 15:31:59 | Sundar | jaypipes: if you were to apply that to networking, would you not want to see the details of the NICs, and manage their diversity in a structured way? | |
| 15:32:57 | jaypipes | Sundar: if you're talking about capabilities of NICs or bandwidth of physnets on NICs, we already have that in the placement DB. | |
| 15:33:39 | jangutter | Sundar: commenting as an engineer working for a NIC vendor, customers broadly fall into two categories: "Make everything look the same" and "handle this node specially". | |
| 15:34:35 | jangutter | Sundar: it would be nice to be able to say "provide me with a networking resource capable of _x_", but broadly, most requests to us have been to make things more uniform, not less. | |
| 15:35:37 | Sundar | jaypipes: jangutter: My point is, just as we have Neutron for networking etc., the diversity of accelerators is best handled by a separate project, rather than fold all in Nova. Esp. since there are use cases that don;t involve Nova, even if you don't agree with it. | |
| 15:36:58 | mriedem | as a casual observer, it sounds like the dilemma is doing something quick and dirty to get something done without a cyborg REST API, vs make a perfect external system that nova can leverage, which would likely delay this even longer | |
| 15:37:17 | mriedem | and if the former is done, how complicated is the extraction/decoupling later | |
| 15:37:28 | dansmith | not sure it's a dilemma, | |
| 15:37:32 | mriedem | and i think what was mentioned last week, | |
| 15:37:36 | dansmith | as I think only jaypipes is the one that feels it should go that way | |
| 15:37:47 | mriedem | was someone could already be doing a PoC for the former to see how it looks so we have an informed decision to make | |
| 15:37:47 | dansmith | (AFAICT at least) | |
| 15:38:02 | mriedem | as the latter is much more work i'd think | |
| 15:38:05 | dansmith | mriedem: yup, I think we probably all agree on that point | |
| 15:38:35 | dansmith | well, except maybe Sundar :) | |
| 15:39:23 | Sundar | mriedem: dansmith: I am working on a POC, a simple one focusing only on the Nova - Cyborg calls. It needs official clearance within my company before I can share it | |
| 15:39:59 | mriedem | the point of the rest api was so that the data and drivers live within cyborg, not nova, and nova hits those drivers over the API | |
| 15:40:06 | mriedem | if we could put a shim in like PlacementDirect, great | |
| 15:40:07 | dansmith | Sundar: yeah that's not going to work very well for collab, but.. good luck :/ | |
| 15:40:28 | jangutter | Sundar: I don't think that anybody disputes that it's better to have a separate entity to handle out-of-Nova scope things. The trick is that Nova already has a way to pass PCI devices to instances. There's also existing code in os-vif that gets called "on plug events". | |
| 15:40:32 | mriedem | we talked about all of this last week when we said we didn't need an os-acc library and could just use python-cyborgclient | |
| 15:40:34 | sean-k-mooney | dansmith: well i have some leaning towrad a libary solution too | |
| 15:40:46 | mriedem | and i was also confused why we needed to hit a rest api to get to the cyborg drivers, i.e. why it wasn't more like os-brick | |
| 15:41:13 | dansmith | I'm fine with the programming part being in a library instead of an agent if we want | |
| 15:41:37 | dansmith | mriedem: it's quite a bit more complicated than just manipulating standard system things to connect to an iscsi target | |
| 15:41:51 | mriedem | right that was explained last week | |
| 15:41:54 | mriedem | probably about this time :) | |
| 15:42:10 | dansmith | but I don't think the tracking of devices, workflow to program, clean, etc these devices belongs in nova | |
| 15:42:31 | mriedem | i don't really think we should build on that existing stuff within nova either, | |
| 15:42:47 | mriedem | b/c once something is built on it, it makes it harder to clean it up later (it already is taking forever) | |
| 15:42:53 | mriedem | and that pci code went in in juno | |
| 15:43:11 | artom | Just... pay a bunch of interns, and stick a CORBA thing in front. | |
| 15:43:28 | mriedem | i was going to add, "says the guy that isn't working on cleaning any of it up" | |
| 15:43:28 | jangutter | artom: don't joke, xml will come out. | |
| 15:43:28 | jaypipes | artom: lol | |
| 15:43:36 | mriedem | for myself | |
| 15:43:39 | sean-k-mooney | artom: just no... | |
| 15:44:01 | mriedem | if we're talking xml, CIM model model this all so nicely! | |
| 15:44:06 | mriedem | *would model | |
| 15:44:35 | artom | If no one can agree on what to do, maybe if we all agree on what *not* to do, and then do that... | |
| 15:46:10 | sean-k-mooney | i think at this point we would all like to see a POC rather then talk in abstract terms about it in a loop | |
| 15:46:22 | dansmith | yup | |
| 15:46:47 | jangutter | we're falling into the trap of agreeing on stuff: which means it won't happen. | |
| 15:46:49 | jaypipes | Sundar: what's the likelihood of your PoC being available this year? | |
| 15:46:58 | jaypipes | Sundar: I understand you need permission from legal? | |
| 15:48:28 | Sundar | I'd like some agreement on the general direction of the POC. At the PTG, there was alignment on defining REST calls, and I wrote that up in a spec that is being reviewed. The discussion here seems more philosophical. I suggest that the POC should focus on clarifying the spec. | |
| 15:49:28 | artom | Sundar, I think at some point you'll have to accept that work might get wasted :( | |
| 15:49:41 | dansmith | Sundar: while everything is just words in a document, the discussion can get pretty philosophical pretty quick, as you have no doubt noticed | |
| 15:49:44 | artom | I've been observing this discussion from afar since... Dublin? | |
| 15:49:48 | sean-k-mooney | Sundar: can we do a simple poc of booting a vm with just a fixed fuction pci device | |
| 15:49:54 | Sundar | jaypipes: Yes, it requires legal clearance. I may get a requirement to follow a process, or likely multiple processes. I am hoping that, confining it to upstream code and no real devices will make it go faster | |
| 15:50:12 | artom | Bite the bullet, write some code, share it | |
| 15:50:23 | artom | If the code gets scrapped and needs to be rewritte, so be it | |
| 15:50:24 | sean-k-mooney | Sundar: the intel fpga stuff may but nova and cyborg working together dont | |
| 15:50:26 | jaypipes | Sundar: that's generally not what PoCs are, but ok :) My suggestion is to demo Nova actually being able to /usr/bin/fpgaconf some device. If you want to go through the trouble of getting all the REST API stuff done in order for Nova to shell out to /usr/bin/fpgaconf, go for it. :) | |
| 15:53:36 | jmlowe | If anybody remembers my slow placement api problem from a couple of days ago, it seems that NAT in the 4.15 kernel that ships with ubuntu 18.04 is problematic at scale | |
| 15:54:41 | cdent | jmlowe: I am a) relieved to hear that, b) sorry for your luck | |
| 15:55:06 | cdent | can you define "at scale"? | |
| 15:56:03 | mriedem | cern runs centos right? | |
| 15:56:05 | mriedem | tssurya: ^ | |
| 15:56:32 | jmlowe | I have a quick question about versioned notifications, if a notification has been transitioned to versioned and the setting is for legacy does that mean the notification will never be emitted? | |
| 15:57:10 | mriedem | jmlowe: what is the value of this option https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format | |
| 15:57:13 | mriedem | in your nova.conf | |
| 15:57:15 | jmlowe | cdent: I was falling over at ~280, threshold is probably significantly lower, ubuntu 16.04 didn't cause problems | |
| 15:57:20 | gibi | jmlowe: if you configured nova to emit legacy notifications only then nova only emts the legacy ones | |
| 15:58:01 | gibi | jmlowe: we did not removed the legacy notifications, and we did not plan to remove them | |
| 15:59:08 | jmlowe | I have unversioned, was looking at rocky release notes and saw that instance exits has been transitioned and that's the one event I really need to consume with ceilometer/panko which doesn't do versioned yet | |
| 15:59:18 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Support volume-backed server rebuild https://review.openstack.org/532407 | |
| 15:59:41 | mriedem | jmlowe: just leave as unversioned then, | |
| 15:59:47 | mriedem | as gibi said, we haven't removed the unversioned notifications | |