| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-02 | |||
| 16:06:27 | efried | mriedem: Yeah. | |
| 16:08:38 | mriedem | lyarwood: that's why test_os_vif_util doesn't fail then - no embedded serialized json | |
| 16:09:42 | efried | mriedem: This sucks a little, but it works: http://paste.openstack.org/show/724766/ | |
| 16:11:13 | mriedem | i was just going to deserialize in the test and pass those on to assertEqual as it's doing now | |
| 16:11:14 | mriedem | with a note | |
| 16:11:45 | efried | ight | |
| 16:16:35 | mriedem | i might also be able to handle this generically in assertJsonEqual | |
| 16:16:39 | mriedem | i'll try both | |
| 16:28:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle nested serialized json entries in assertJsonEqual https://review.openstack.org/579615 | |
| 16:28:57 | mriedem | efried: ok the generic one wins ^ | |
| 16:29:56 | efried | mriedem: I still don't see how that's going to help. The outermost object is not a string type. | |
| 16:30:25 | efried | nor is the first level in, which is I guess what you're targeting now. | |
| 16:32:43 | mriedem | i'm doing it in the recursion | |
| 16:33:14 | mriedem | bauzas: nvm i guess that's just something that shows up a lot in the gate | |
| 16:33:19 | efried | mriedem: Okay, yeah, I see the recursion now. | |
| 16:35:43 | mriedem | i'm pretty sure it's the greatest patch i've written in years | |
| 16:53:52 | openstackgerrit | Merged openstack/nova master: Fix missing versioned notification examples https://review.openstack.org/579436 | |
| 17:33:03 | openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Handling a down cell https://review.openstack.org/557369 | |
| 17:52:04 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: return consumer info even when no alloc https://review.openstack.org/579654 | |
| 17:52:27 | Sundar | efried, please ping me when you can. Thanks. | |
| 17:52:47 | efried | Sundar: I'm here. What's up? | |
| 18:00:32 | openstackgerrit | Merged openstack/nova master: Remove unnecessary execute permissions of a file https://review.openstack.org/579511 | |
| 18:00:55 | Sundar | efried: Do you have a few minutes re. os-acc spec? | |
| 18:01:05 | efried | Sundar: sure | |
| 18:01:53 | efried | I'd like to understand how cyborg is going to do "discovery", actually. | |
| 18:02:36 | Sundar | Re. the idea that os-acc should operate in 3 stages (call Cyborg for prep, call plugin, and then call Cyborg again for post/cleanup), since the plugin has device/vendor-specific code, and Cyborg also has device/vendor-specific code, | |
| 18:03:00 | Sundar | the split in responsibilities, becomes unclear | |
| 18:03:41 | Sundar | If you delegate all device/vendor-specific code to the plugn, and let it either handle it or delegate to Cyborg as needed, that would be a cleaner split. What do you think? | |
| 18:03:42 | efried | It's "Cyborg also has device/vendor-specific code" that I'm confused about | |
| 18:04:23 | Sundar | Cyborg supports a system of drivers. Some drivers already exist. #link https://wiki.openstack.org/wiki/Cyborg#Pike.2FQueens_Initial_Architecture | |
| 18:04:30 | efried | ... | |
| 18:05:13 | efried | Sundar: Does this support exist on all platforms, or just kvm? | |
| 18:06:28 | efried | This is my concern. There's no way cyborg has written device discovery etc. on power. Or am I wrong? | |
| 18:07:00 | Sundar | The initial implementation is expected to be for libvirt, possibly qemu/KVM. However, based on our conversation, we could say all hypervisor/platform-specific stuff could be in the plugin. The Cyborg drivers would handle device-specific aspects | |
| 18:07:12 | Sundar | Cynorg architecture is hypervisor-neutral | |
| 18:07:20 | Sundar | * Cyborg | |
| 18:07:25 | efried | Right. I'm on board with that. | |
| 18:07:49 | efried | So we need to define the flows as if that's always the case. | |
| 18:09:07 | Sundar | Yes. Some things the plugin does (the Instance Half of the attach) are hypervisor/platform-specific. Other things (Device half) are device/vendor/platform-specific. ... | |
| 18:09:38 | Sundar | The split of roles between the plugin and Cyborg driver could be left to the plugin? | |
| 18:09:58 | efried | No, that's where we disagree. | |
| 18:11:00 | efried | The overall logic flow should live in cyborg-owned code (whether cyborg service or os-acc or whatever) so that it need only be written once. | |
| 18:11:58 | efried | The plugin should be responsible for the very granular leaf primitives that deal specifically with the platform, nothing else. | |
| 18:13:36 | Sundar | Does the 'platform' include the device-specific aspects, or hypervisor/architecture/system only? | |
| 18:14:40 | Sundar | If the latter, I am totally on board | |
| 18:15:01 | Sundar | If the former, we need to define what the plugin does vis-a-vis Cyborg driver(s) | |
| 18:15:18 | efried | I don't really know enough about the specifics of managing devices to be able to answer that, but yes, I'm hoping we can define the flow so the latter is possible. | |
| 18:16:24 | Sundar | Great. OK, how about this? Let's go with the 3-stage flow for os-acc: prep by calling Cyborg, call plugin, and finish by calling Cyborg. All device-specific aspects are left to the 1st and 3rd steps, to be done by a Cyborg driver. | |
| 18:16:45 | Sundar | All hypervisor/platform/system/arch-dependent steps are for the plugin | |
| 18:17:20 | efried | Yes. BUT it might not be as simple as three stages, I'm not sure. | |
| 18:18:07 | efried | let me walk through a (hypothetical) example to show you what I mean... | |
| 18:18:56 | efried | hold a minute... | |
| 18:22:15 | efried | Sundar: Okay, sorry about that - was finishing up a soooper messy lunch, typing with one hand. I'm back to full speed now. | |
| 18:22:28 | efried | So here's the example: I want to program and attach an FPGA | |
| 18:23:08 | efried | Cyborg can do the programming, but in order to do so, it needs to be supplied with a file-like that's piped to the device. | |
| 18:23:18 | efried | (Again, no idea how realistic this is, but go with me for a bit here) | |
| 18:23:57 | efried | The user says "program the device with bitstream from image XYZ" | |
| 18:24:11 | efried | Cyborg calls the plugin and says "give me a file-like so I can program". | |
| 18:24:36 | efried | On libvirt, maybe the plugin just does f = open("/path/to/special_file") and gives back f. | |
| 18:24:49 | efried | On Power, we probably have to attach the device to the management partition and then do ^ | |
| 18:25:18 | efried | On Windows, gods forbid, who knows what happens, but it'd be like D:\path\to\special_file anyway... | |
| 18:25:36 | efried | So then control goes back to cyborg, which goes and happily flashes the device. | |
| 18:25:59 | efried | Now there has to be a "close/cleanup" step. Cyborg calls the plugin and says, "Okay, I'm done". | |
| 18:26:30 | efried | On libvirt, in this example, it's just close(f). On Power, close(f) and detach the device from the management partition. On Windows.... who knows, whatever. | |
| 18:27:47 | efried | Sundar: Now, I don't know where this fits in the overall "plug" flow. Does it happen before or after we attach the thing to the VM? I would assume before. | |
| 18:28:00 | Sundar | We already have provisions for programming from Cyborg drivers (not yet implemented fully). Re. "a file-like that's piped to the device", the bitstreams will reside in Glance and will be accessed via image UUID. | |
| 18:28:00 | efried | Except if there's a flow where we program the thing while it's already attached; I don't even know how that would work. | |
| 18:28:16 | efried | Sundar: Right, that's where the data comes *from*, but how does it get *to* the device? | |
| 18:28:36 | efried | The glance side is the source of the data. You need a sink. | |
| 18:29:29 | efried | Or maybe what you're saying is the program_fpga plugin method will be given the source handle and be responsible for pushing the data from that handle to the device. | |
| 18:29:29 | Sundar | The Cyborg FPGA driver knows what to do. Anyway, you will have a vendor stack -- like OPAE kernel drivers for Intel FPGAs, Xilinx stack etc. | |
| 18:29:32 | efried | That would be fine too. | |
| 18:29:49 | Sundar | The Cyborg FPGA driver knows how to interface with the vendor-specific stack | |
| 18:30:07 | efried | vendor-specific, sure, but on all platforms? That's the part I'm doubting. | |
| 18:31:51 | Sundar | If you have an FPGA on Power, you will presumably have a vendor-specific (Intel/Xilinx) and OS-specific (Linux/AIX/...) driver for it, right? Independent of Openstack/Kubernetes/..., you would need that | |
| 18:32:19 | Sundar | The Cyborg driver goes on top of that | |
| 18:32:21 | efried | I imagine there's two levels of "driver" we'll need. | |
| 18:32:59 | efried | One that the platform can use to talk to the device, and one per guest OS that the VM needs to actually *use* the thing. | |
| 18:33:27 | efried | See, on kvm that's basically one thing, because the hypervisor and the guests are running the same OS. | |
| 18:33:43 | efried | ...and the device is "attached" to the hypervisor to begin with. | |
| 18:34:14 | Sundar | Yes, the first is the PF driver and the latter is the VF driver for SR-IOV-based devices. They need not be the same OS -- you could run Centos on the host and Ubuntu (or maybe Windows) on the VM | |
| 18:34:35 | Sundar | Does Power use SR-IOV? | |
| 18:34:38 | efried | Yes | |
| 18:34:47 | efried | That's a good example. | |
| 18:35:03 | Sundar | That's what I'd expect :-P but good to confirm! | |
| 18:35:16 | Sundar | OK, so you could have a PF driver on the host for mgmt ,.. | |
| 18:35:26 | Sundar | and a VF driver on the guest for FPGA access | |
| 18:35:29 | efried | In order to carve off a VF and set up its QoS etc. you have to make REST calls to the platform. | |
| 18:35:40 | efried | And to attach it to a VM, you have to call REST as well. | |
| 18:35:52 | efried | But the VM itself has to have the right drivers installed in order to see that VF as a network device. | |
| 18:36:13 | Sundar | Why do you assume a specific implementation, with REST API etc.? Let's keep it generic | |
| 18:37:09 | Sundar | Yes, you need to 'get a VF' -- either grab one that already exists (and was discovered), or create one on the fly ... | |
| 18:37:27 | Sundar | Either way, it is a device-specific op in my view. | |
| 18:38:06 | Sundar | The specific calls you may make would probably depend on the OS, but that is abstracted by the Cyborg driver and/or vendor-specific driver for that OS | |
| 18:38:40 | Sundar | For example, with Linux ... | |
| 18:39:06 | Sundar | you can run pci_sriov_enable() as soon as Cybprg driver discovers the device (or even earlier), and you get the list of VFs | |
| 18:39:22 | efried | See, this is exactly my point. | |
| 18:39:31 | Sundar | The Cyborg db stores them, and they can later be attached to a VAN | |
| 18:40:25 | efried | "Why do you assume a specific implementation, with REST API etc.? Let's keep it generic" ==> I'm mentioning specifically how this needs to be done on a Power system, because it's fundamentally different than what happens on kvm. | |
| 18:40:48 | efried | "as soon as Cyborg driver discovers the device" ==> *How* does the cyborg driver discover the device? | |