| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-08-30 | |||
| 19:59:51 | efried | it may in fact be device *type* specific, not necessarily device specific. | |
| 20:00:02 | efried | Anyway, I think we're in violent agreement on the high points here. | |
| 20:00:36 | Sundar | Well, the actual act of writing to the device and manipulating it to create a new vGPU type or program a bitstream would in fcat be specific to device models or vendors. | |
| 20:02:19 | Sundar | So, the Cyborg and its driver(s) would have to handle the equivalent of port binding (device specific) and plug. That gives us a VAN with the device end configured, ready to be attached to an instance | |
| 20:02:46 | Sundar | After that, Nova virt can take that VAN and do the needful for that hypervisor, in a device-independen way | |
| 20:02:48 | efried | and does that happen in a separate step prior to spawn? | |
| 20:03:18 | Sundar | Yes. I shared a flow diagram with you this morning. | |
| 20:03:31 | Sundar | Before we get to that :), | |
| 20:04:11 | Sundar | the role of os-acc plugin is highly diminished with these aspects considered. The driver does the device end, Nova virt does the instance end | |
| 20:04:48 | efried | "nova virt" by invoking the plugin through os-acc? | |
| 20:04:55 | Sundar | The possible role for os-acc extensions is to handle device-compute interactions, such as NUMA affinity for interrupt vectors. | |
| 20:06:41 | Sundar | Nova virt calls os-acc, and that in turn calls Cyborg in some way. When that call returns, Nova virt has to persist the instance - VAN association in Nova's db. neither os-acc nor Cyborg can do that, so it has to go to Nova virt. The subsequent step of attaching to the VM is already in Nova virt. | |
| 20:06:48 | Sundar | So, what does the plugin do for attach? | |
| 20:07:07 | efried | no idea | |
| 20:07:19 | efried | if it's nothing, it's nothing. | |
| 20:08:03 | Sundar | I understand your need for keeping things hypervisor-specific, but Nova virt is already hypervisor-specific, right? | |
| 20:08:14 | efried | yup | |
| 20:08:39 | efried | I don't really have a stake in getting lots of code into these plugins. | |
| 20:08:48 | efried | I just want to make sure we don't end up with linux-isms in common code paths. | |
| 20:08:54 | efried | linux/libvirt | |
| 20:09:16 | Sundar | OK. Could you look at the flow diagram I shared this morning? | |
| 20:09:26 | efried | If I didn't have to write a separate driver/plugin at all, I would be pretty happy. But I'm not sure that is going to happen. | |
| 20:10:08 | Sundar | Not to spoil your afternoon, but I think you would need PowerVM drivers for GPUs, FPGAs, or whatever you support :) | |
| 20:11:23 | Sundar | Are you looking at Power+KVM hypervisor too? | |
| 20:12:10 | Sundar | brb in 5 min | |
| 20:12:19 | efried | I don't know anything about pkvm | |
| 20:14:00 | Sundar | back. ok | |
| 20:15:43 | Sundar | Back to os-acc plugins. The possible role for os-acc plugins is to handle device-compute interactions, such as NUMA affinity for interrupt vectors. But it may be too much to bring them in this spec. Shall we put os-acc plugins as a placeholder until we get to that? | |
| 20:18:26 | efried | "NUMA affinity for interrupt vectors" <== Greek to me. | |
| 20:18:51 | Sundar | OK, NUMA affinity for devices in general? | |
| 20:19:11 | efried | If you don't have an actual use case for os-acc plugins, then... | |
| 20:19:35 | efried | NUMA affinity is something we want to be able to handle via resource provider structure. | |
| 20:19:48 | efried | we can't yet | |
| 20:19:56 | efried | but it's something we should bring up (again) in Denver. | |
| 20:20:49 | Sundar | Sure. Shall we have a f2f chat at Denver? It'll be good to sync up and drive the specs to a close shortly thereafter | |
| 20:21:14 | efried | definitely | |
| 20:21:30 | efried | If I were you, I would ask for a nova/cyborg cross-project session. | |
| 20:21:52 | efried | Based on what I know of people's schedules, I would say Tuesday would be a good day for it. | |
| 20:22:03 | Sundar | I have entered a slot in Nova etherpad. May be I should ping melwitt? | |
| 20:23:02 | efried | It might be better to schedule something during cyborg's time (M/T) where you can get the "placement cores" to visit the cyborg room and spend a couple of hours hashing stuff out. | |
| 20:23:33 | efried | And keep it on the nova schedule in case we a) don't get everything sorted; and/or b) need a wider audience. | |
| 20:23:48 | Sundar | OK | |
| 20:24:04 | efried | Send something to the dev ML, like what Blazar did, to organize the Tuesday thing. | |
| 20:24:05 | efried | oh | |
| 20:24:14 | efried | Sorry, I forgot, Blazar sent that directly to people. | |
| 20:24:18 | efried | not to the ML. | |
| 20:24:34 | Sundar | Meanwhile, please LMK if you are ok with the flow diagram. Did you get it? | |
| 20:26:34 | efried | I got it. | |
| 20:27:47 | Sundar | Great. I'll grab lunch and get back in 15 min. We can shake it out after that. | |
| 20:29:57 | efried | give me 20 | |
| 21:07:14 | Sundar | efried: Ready any time | |
| 21:08:00 | efried | Sundar: Nova meeting in progress. Not sure how long it will last, but not beyond top of the hour. Will you be around? | |
| 21:09:31 | Sundar | Yes | |
| 21:21:08 | efried | Sundar: ō/ | |
| 21:23:57 | Sundar | That was quick. | |
| 21:24:32 | efried | now where did I put that diagram... | |
| 21:24:56 | efried | got it | |
| 21:25:14 | Sundar | The main thing to note in the flow is that there isn't much that os-acc is doing. | |
| 21:25:59 | Sundar | Most of it is Nova virt or Cyborg. I am still pondering what os-acc can do that is not device-specific (and so in Cyborg) or hypervisor-specific (and so in Nova virt) | |
| 21:26:59 | efried | Sundar: as a touchpoint/router and future extension point for those pieces of the requests, it may be useful. | |
| 21:27:05 | efried | but I'm really not sure. | |
| 21:27:30 | Sundar | Yes, I agree. That's what I was thinking | |
| 21:28:21 | efried | not sure it makes much sense for the calls from os-acc to cyborg API/conductor to be asynchronous if it's just going to poll for them to complete right away... | |
| 21:29:37 | Sundar | The calls may take milliseconds to possibly seconds, depending on whether Glance bitstreams need to fetched, one or more FPGAs need to be programmed, etc | |
| 21:30:20 | efried | so? | |
| 21:30:27 | Sundar | That is why it is async | |
| 21:30:31 | efried | The caller is blocking on their completion anyway | |
| 21:30:39 | efried | So why does it matter if they take "a long time"? | |
| 21:30:48 | Sundar | Ah, the n-cpu could do other things while this is blocked, right? | |
| 21:30:53 | efried | could it? | |
| 21:31:06 | efried | I guess. | |
| 21:31:18 | Sundar | The allocation of other resources -- networking, storage -- could go in parallel | |
| 21:31:21 | efried | Just sounds like it would make things pretty complicated. | |
| 21:31:36 | efried | If n-cpu wanted to parallelize, it could send that request off in its own thread. | |
| 21:31:46 | efried | But having it be async *forces* n-cpu to deal with that async-ness. | |
| 21:31:49 | efried | Anyway, this is really a nit. | |
| 21:31:59 | efried | not a substantive thing. | |
| 21:32:03 | Sundar | The n-cpu to os-acc could be async too. | |
| 21:32:15 | Sundar | So, os-acc deals with the async weirdness. | |
| 21:32:59 | Sundar | Hmm, well, os-acc to Cyborg can be sync if os-acc itself is called in an async way | |
| 21:33:13 | Sundar | Just wondering if a REST API call cna really block for seconds | |
| 21:33:48 | efried | absolutely. | |
| 21:34:10 | efried | I'm sure there's a connection timeout at the HTTP level. But seconds shouldn't be a problem. | |
| 21:34:16 | Sundar | Are there precedents in OpenStack where a REST API blocks for seconds? | |
| 21:35:57 | efried | I have no idea. | |
| 21:36:10 | efried | That might be something edleafe and/or cdent would know. | |
| 21:37:22 | Sundar | OK. Another note: The proposal calls for persisting VAN objects in Cyborg db, but Nova db maintains the association between the instance and its VAN UUIDs . That s because, on a VM suspend for example, Nova would need to call os-acc to detach VANs but not deallocate them | |
| 21:37:55 | Sundar | Never mind, the detach would be in Nova virt. | |
| 21:38:03 | Sundar | But you get the point | |
| 21:38:29 | Sundar | On a termination, Nova virt would detach each VAN and then call os-acc to release the resources | |
| 21:38:47 | efried | right, some kind of handle (i.e. the VAN UUID) would need to be associated with the instance in the nova db. | |
| 21:38:54 | Sundar | Yes | |
| 21:39:14 | efried | As long as we can query cyborg with the UUID to get the rest of the VAN info, the UUID should be the only thing we need to store, I would think. | |
| 21:39:21 | Sundar | Yes | |
| 21:42:33 | Sundar | Good. Since we are in agreement, I'll write this up in the spec. | |
| 21:43:55 | efried | Sundar: Note that I'm only one person, and my stake in the details we've just discussed is not an especially strong one. | |
| 21:45:20 | Sundar | We need some path to get this converged. I am incorporating sean-mooney's comments and responding to them. But, if somebody else were to come along in a month, we can't keep waiting. | |
| 21:45:40 | Sundar | I hope to get this closed during the PTG or at most a week after | |
| 21:46:25 | Sundar | I requested melwitt for a PTG session. Could I ask for your help in ensuring that all Nova feedback is given by that time? | |
| 22:13:03 | efried | hah | |