| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-02 | |||
| 17:33:03 | openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Handling a down cell https://review.openstack.org/557369 | |
| 17:52:04 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: return consumer info even when no alloc https://review.openstack.org/579654 | |
| 17:52:27 | Sundar | efried, please ping me when you can. Thanks. | |
| 17:52:47 | efried | Sundar: I'm here. What's up? | |
| 18:00:32 | openstackgerrit | Merged openstack/nova master: Remove unnecessary execute permissions of a file https://review.openstack.org/579511 | |
| 18:00:55 | Sundar | efried: Do you have a few minutes re. os-acc spec? | |
| 18:01:05 | efried | Sundar: sure | |
| 18:01:53 | efried | I'd like to understand how cyborg is going to do "discovery", actually. | |
| 18:02:36 | Sundar | Re. the idea that os-acc should operate in 3 stages (call Cyborg for prep, call plugin, and then call Cyborg again for post/cleanup), since the plugin has device/vendor-specific code, and Cyborg also has device/vendor-specific code, | |
| 18:03:00 | Sundar | the split in responsibilities, becomes unclear | |
| 18:03:41 | Sundar | If you delegate all device/vendor-specific code to the plugn, and let it either handle it or delegate to Cyborg as needed, that would be a cleaner split. What do you think? | |
| 18:03:42 | efried | It's "Cyborg also has device/vendor-specific code" that I'm confused about | |
| 18:04:23 | Sundar | Cyborg supports a system of drivers. Some drivers already exist. #link https://wiki.openstack.org/wiki/Cyborg#Pike.2FQueens_Initial_Architecture | |
| 18:04:30 | efried | ... | |
| 18:05:13 | efried | Sundar: Does this support exist on all platforms, or just kvm? | |
| 18:06:28 | efried | This is my concern. There's no way cyborg has written device discovery etc. on power. Or am I wrong? | |
| 18:07:00 | Sundar | The initial implementation is expected to be for libvirt, possibly qemu/KVM. However, based on our conversation, we could say all hypervisor/platform-specific stuff could be in the plugin. The Cyborg drivers would handle device-specific aspects | |
| 18:07:12 | Sundar | Cynorg architecture is hypervisor-neutral | |
| 18:07:20 | Sundar | * Cyborg | |
| 18:07:25 | efried | Right. I'm on board with that. | |
| 18:07:49 | efried | So we need to define the flows as if that's always the case. | |
| 18:09:07 | Sundar | Yes. Some things the plugin does (the Instance Half of the attach) are hypervisor/platform-specific. Other things (Device half) are device/vendor/platform-specific. ... | |
| 18:09:38 | Sundar | The split of roles between the plugin and Cyborg driver could be left to the plugin? | |
| 18:09:58 | efried | No, that's where we disagree. | |
| 18:11:00 | efried | The overall logic flow should live in cyborg-owned code (whether cyborg service or os-acc or whatever) so that it need only be written once. | |
| 18:11:58 | efried | The plugin should be responsible for the very granular leaf primitives that deal specifically with the platform, nothing else. | |
| 18:13:36 | Sundar | Does the 'platform' include the device-specific aspects, or hypervisor/architecture/system only? | |
| 18:14:40 | Sundar | If the latter, I am totally on board | |
| 18:15:01 | Sundar | If the former, we need to define what the plugin does vis-a-vis Cyborg driver(s) | |
| 18:15:18 | efried | I don't really know enough about the specifics of managing devices to be able to answer that, but yes, I'm hoping we can define the flow so the latter is possible. | |
| 18:16:24 | Sundar | Great. OK, how about this? Let's go with the 3-stage flow for os-acc: prep by calling Cyborg, call plugin, and finish by calling Cyborg. All device-specific aspects are left to the 1st and 3rd steps, to be done by a Cyborg driver. | |
| 18:16:45 | Sundar | All hypervisor/platform/system/arch-dependent steps are for the plugin | |
| 18:17:20 | efried | Yes. BUT it might not be as simple as three stages, I'm not sure. | |
| 18:18:07 | efried | let me walk through a (hypothetical) example to show you what I mean... | |
| 18:18:56 | efried | hold a minute... | |
| 18:22:15 | efried | Sundar: Okay, sorry about that - was finishing up a soooper messy lunch, typing with one hand. I'm back to full speed now. | |
| 18:22:28 | efried | So here's the example: I want to program and attach an FPGA | |
| 18:23:08 | efried | Cyborg can do the programming, but in order to do so, it needs to be supplied with a file-like that's piped to the device. | |
| 18:23:18 | efried | (Again, no idea how realistic this is, but go with me for a bit here) | |
| 18:23:57 | efried | The user says "program the device with bitstream from image XYZ" | |
| 18:24:11 | efried | Cyborg calls the plugin and says "give me a file-like so I can program". | |
| 18:24:36 | efried | On libvirt, maybe the plugin just does f = open("/path/to/special_file") and gives back f. | |
| 18:24:49 | efried | On Power, we probably have to attach the device to the management partition and then do ^ | |
| 18:25:18 | efried | On Windows, gods forbid, who knows what happens, but it'd be like D:\path\to\special_file anyway... | |
| 18:25:36 | efried | So then control goes back to cyborg, which goes and happily flashes the device. | |
| 18:25:59 | efried | Now there has to be a "close/cleanup" step. Cyborg calls the plugin and says, "Okay, I'm done". | |
| 18:26:30 | efried | On libvirt, in this example, it's just close(f). On Power, close(f) and detach the device from the management partition. On Windows.... who knows, whatever. | |
| 18:27:47 | efried | Sundar: Now, I don't know where this fits in the overall "plug" flow. Does it happen before or after we attach the thing to the VM? I would assume before. | |
| 18:28:00 | efried | Except if there's a flow where we program the thing while it's already attached; I don't even know how that would work. | |
| 18:28:00 | Sundar | We already have provisions for programming from Cyborg drivers (not yet implemented fully). Re. "a file-like that's piped to the device", the bitstreams will reside in Glance and will be accessed via image UUID. | |
| 18:28:16 | efried | Sundar: Right, that's where the data comes *from*, but how does it get *to* the device? | |
| 18:28:36 | efried | The glance side is the source of the data. You need a sink. | |
| 18:29:29 | Sundar | The Cyborg FPGA driver knows what to do. Anyway, you will have a vendor stack -- like OPAE kernel drivers for Intel FPGAs, Xilinx stack etc. | |
| 18:29:29 | efried | Or maybe what you're saying is the program_fpga plugin method will be given the source handle and be responsible for pushing the data from that handle to the device. | |
| 18:29:32 | efried | That would be fine too. | |
| 18:29:49 | Sundar | The Cyborg FPGA driver knows how to interface with the vendor-specific stack | |
| 18:30:07 | efried | vendor-specific, sure, but on all platforms? That's the part I'm doubting. | |
| 18:31:51 | Sundar | If you have an FPGA on Power, you will presumably have a vendor-specific (Intel/Xilinx) and OS-specific (Linux/AIX/...) driver for it, right? Independent of Openstack/Kubernetes/..., you would need that | |
| 18:32:19 | Sundar | The Cyborg driver goes on top of that | |
| 18:32:21 | efried | I imagine there's two levels of "driver" we'll need. | |
| 18:32:59 | efried | One that the platform can use to talk to the device, and one per guest OS that the VM needs to actually *use* the thing. | |
| 18:33:27 | efried | See, on kvm that's basically one thing, because the hypervisor and the guests are running the same OS. | |
| 18:33:43 | efried | ...and the device is "attached" to the hypervisor to begin with. | |
| 18:34:14 | Sundar | Yes, the first is the PF driver and the latter is the VF driver for SR-IOV-based devices. They need not be the same OS -- you could run Centos on the host and Ubuntu (or maybe Windows) on the VM | |
| 18:34:35 | Sundar | Does Power use SR-IOV? | |
| 18:34:38 | efried | Yes | |
| 18:34:47 | efried | That's a good example. | |
| 18:35:03 | Sundar | That's what I'd expect :-P but good to confirm! | |
| 18:35:16 | Sundar | OK, so you could have a PF driver on the host for mgmt ,.. | |
| 18:35:26 | Sundar | and a VF driver on the guest for FPGA access | |
| 18:35:29 | efried | In order to carve off a VF and set up its QoS etc. you have to make REST calls to the platform. | |
| 18:35:40 | efried | And to attach it to a VM, you have to call REST as well. | |
| 18:35:52 | efried | But the VM itself has to have the right drivers installed in order to see that VF as a network device. | |
| 18:36:13 | Sundar | Why do you assume a specific implementation, with REST API etc.? Let's keep it generic | |
| 18:37:09 | Sundar | Yes, you need to 'get a VF' -- either grab one that already exists (and was discovered), or create one on the fly ... | |
| 18:37:27 | Sundar | Either way, it is a device-specific op in my view. | |
| 18:38:06 | Sundar | The specific calls you may make would probably depend on the OS, but that is abstracted by the Cyborg driver and/or vendor-specific driver for that OS | |
| 18:38:40 | Sundar | For example, with Linux ... | |
| 18:39:06 | Sundar | you can run pci_sriov_enable() as soon as Cybprg driver discovers the device (or even earlier), and you get the list of VFs | |
| 18:39:22 | efried | See, this is exactly my point. | |
| 18:39:31 | Sundar | The Cyborg db stores them, and they can later be attached to a VAN | |
| 18:40:25 | efried | "Why do you assume a specific implementation, with REST API etc.? Let's keep it generic" ==> I'm mentioning specifically how this needs to be done on a Power system, because it's fundamentally different than what happens on kvm. | |
| 18:40:48 | efried | "as soon as Cyborg driver discovers the device" ==> *How* does the cyborg driver discover the device? | |
| 18:40:50 | Sundar | What REST API do you need for Power? | |
| 18:41:30 | efried | Assuming you're writing in python, the REST API is available via https://github.com/powervm/pypowervm | |
| 18:41:46 | efried | It's a totally Power-specific API. | |
| 18:42:16 | efried | And, once again, I would not expect cyborg to be calling into pypowervm. I would expect anything at that level to be relegated to the plugin. | |
| 18:43:54 | efried | pypowervm is awesome, but still requires some pretty esoteric domain knowledge of power systems. Knowledge that I wouldn't really expect the cyborg team to want to come up to speed on. | |
| 18:44:21 | efried | Though I would be delighted to support you if you wanted to get into that business :) | |
| 18:47:12 | dansmith | efried: the hypervisor and guests are not running the same OS in kvm | |
| 18:47:18 | dansmith | they might be, but that would be coincidence | |
| 18:47:35 | efried | yeah, my bad. | |
| 18:48:00 | Sundar | :) We do want to support all architectures. IMHO, it would be simplest if there are Cyborg drivers for Power devices as well. There is very little that core Cyborg assumes about devices, esp. ... | |
| 18:48:37 | Sundar | with the recent changes to make the scheduling independent of PCI-centric things like whitelists | |
| 18:49:26 | Sundar | You may have concerns about whether Cyborg can support such widely different architectures, ... | |
| 18:49:41 | Sundar | but Cyborg should support them | |
| 18:49:55 | efried | I do. Or rather, whether Cyborg will be motivated to support the... "less popular" ones. | |
| 18:50:23 | efried | wouldn't be the first time Power got marginalized because it's not kvm. | |
| 18:50:30 | Sundar | As I tell everybody :), open source is democratic -- popularity does not (or should not) count! | |
| 18:51:28 | efried | whereas if we architect it such that platform specific pieces can be written by those with skin in that platform (i.e. me and my team) then we're covered either way. | |