| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-03 | |||
| 16:05:49 | bauzas | efried: I think I reached the same conclusion | |
| 16:05:50 | mriedem | https://developer.openstack.org/api-ref/compute/#reset-server-state-os-resetstate-action | |
| 16:06:06 | bauzas | efried: for some reason, xen is better designed than libvirt | |
| 16:06:06 | efried | bauzas: But I think as much as we possibly can, we should create standard traits and ask the virt drivers to map to them. | |
| 16:06:15 | bauzas | efried: it already makes the translation I guess | |
| 16:06:33 | efried | bauzas: Which one is doing translation, though? | |
| 16:06:37 | bauzas | efried: because if you look the Xen GPU types, those look very identical to the nvidia doc | |
| 16:06:45 | bauzas | efried: the xen API | |
| 16:06:50 | efried | bauzas: Right, so it's libvirt that's translating. | |
| 16:06:57 | bauzas | efried: no | |
| 16:06:59 | efried | xen is just using 'em as is | |
| 16:07:01 | efried | no? | |
| 16:07:13 | bauzas | efried: libvirt is straight setting what the kernel module tells | |
| 16:07:14 | bauzas | https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt | |
| 16:07:49 | efried | oh, so xen is mapping those back to the model names in the literature | |
| 16:08:00 | exarlos | mriedem: Let me go have a read of that. | |
| 16:08:17 | exarlos | It is on the same host still. | |
| 16:08:24 | bauzas | efried: that's what I guess, yes | |
| 16:08:38 | efried | bauzas: so the point is that we need to choose one or the other, and then the docs for a given hypervisor have to tell me how to discover and determine the appropriate corresponding trait name. | |
| 16:08:52 | bauzas | efried: ok, I take the challenge | |
| 16:09:20 | bauzas | efried: I'm in favor of an unified trait name | |
| 16:09:20 | efried | bauzas: Which needs to happen regardless, because at the very least 'nvidia-11' will need to be translated to 'trait:HW_VGPU_TYPE_NVIDIA_11=required' | |
| 16:09:31 | bauzas | yup | |
| 16:10:14 | efried | bauzas: What would be neat is if each hypervisor provided a script to discover the devs and list them along with their trait names. | |
| 16:10:28 | efried | that might be asking too much, though :) | |
| 16:10:31 | bauzas | efried: what I wonder if whether the mapping should be hardcoded in the virt driver code, or if we should leave operators setting that | |
| 16:10:45 | bauzas | efried: heh, I have the same wish | |
| 16:10:50 | bauzas | efried: xen does that | |
| 16:10:57 | bauzas | efried: but see my docs on libvirt usage | |
| 16:11:02 | bauzas | that's crazy | |
| 16:11:10 | bauzas | you have to lookup sysfs | |
| 16:11:21 | efried | bauzas: The operator mapping thing - that's going back to the yaml config we started brainstorming in Denver. | |
| 16:11:26 | bauzas | we don't have any libvirt API that shows you that | |
| 16:11:52 | bauzas | efried: I know | |
| 16:12:06 | efried | bauzas: BUT that mapping is going to be more for device aliasing, white/blacklisting, etc. I don't think we should be using it to define traits, unless maybe they're CUSTOM_* | |
| 16:12:19 | bauzas | efried: just a thought | |
| 16:12:26 | bauzas | efried: about custom traits | |
| 16:12:29 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add 'numa-aware-vswitches' spec https://review.openstack.org/541290 | |
| 16:12:33 | bauzas | efried: can you set those with nested RPs ? | |
| 16:12:52 | bauzas | efried: because then, I wouldn't care a bit about that | |
| 16:13:07 | bauzas | efried: I could just ask operators to provide custom traits | |
| 16:13:19 | efried | bauzas: You can set traits on any RP. We originally talked about traits "inheriting" or "propagating" in various directions, but we're not doing that. | |
| 16:13:31 | efried | bauzas: "on any RP" including child, sharing, etc. | |
| 16:13:31 | bauzas | efried: I'm a bit rusty on traits | |
| 16:13:46 | bauzas | efried: but I guess operators have to play with placement client to set the trait ? | |
| 16:13:56 | bauzas | if so, that's perfect | |
| 16:14:44 | bauzas | jaypipes: for your sanity, don't read the vendor crazypants I wrote | |
| 16:15:08 | efried | bauzas: Well, what we talked about in Dublin is that each of various entities (nova-compute, neutron, operator) will have a set of traits it owns/controls for a given provider. It's allowed to set/unset those traits, but the other entities are not. | |
| 16:15:42 | efried | bauzas: So in this example, I would expect nova-compute (specifically the virt driver) to "own" the trait indicating the GPU type. The operator doesn't get to set that. That wouldn't make sense. | |
| 16:16:02 | bauzas | efried: keep in mind we will have one inventory per vGPU type | |
| 16:16:18 | efried | bauzas: Same entity owns the inventory. | |
| 16:16:49 | bauzas | efried: what if I want to set custom traits ? | |
| 16:17:26 | bauzas | efried: say the virt driver now provides a tree of a root RP and two children RP, each one being a vGPU type | |
| 16:17:44 | bauzas | efried: as an operator, I'd like to pick only one type based on a custom trait | |
| 16:17:50 | efried | bauzas: From this perspective, we don't really make a distinction between standard and custom traits. But I would say the op is allowed to invent and assign a random custom trait as she sees fit. | |
| 16:17:54 | bauzas | efried: exactly what we wrote in the spec, actually | |
| 16:18:13 | bauzas | https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html | |
| 16:18:14 | efried | bauzas: Legit. Basically the op just has to be careful not to stomp on a trait owned by compute. | |
| 16:18:24 | efried | bauzas: beyond that, sky's the limit. | |
| 16:18:38 | efried | bauzas: which is exactly why we made this a requirement in Dublin. | |
| 16:18:46 | efried | (dansmith and I almost came to blows over it) | |
| 16:18:51 | bauzas | yeah I remember | |
| 16:19:14 | efried | I still believe it's going to bite us, but time will tell. | |
| 16:19:36 | bauzas | efried: ok, so what's the way to set custom traits as of now ? | |
| 16:19:42 | bauzas | thru placement API ? (and the client) | |
| 16:19:55 | efried | bauzas: Sure, that works. | |
| 16:20:15 | bauzas | efried: ok, what I need is testing then | |
| 16:20:30 | bauzas | efried: my patch series is close to provide inventories using update_provider_tree() | |
| 16:20:44 | efried | bauzas: There's no change to the placement API for any of that; the only thing we did was update the u-p-t spec/docs to make it clear that a dance is required to make sure you're not stepping on each other. | |
| 16:21:12 | efried | bauzas: Oh, are you aware of https://review.openstack.org/#/c/560444/ ? | |
| 16:21:12 | bauzas | efried: ok, will look | |
| 16:21:40 | bauzas | efried: no, I wasn't | |
| 16:21:54 | bauzas | efried: my top patch in the series returns a dict of dicts | |
| 16:22:08 | bauzas | efried: I'll use that dict of dicts to populate the tree | |
| 16:22:31 | bauzas | efried: but I was considering to implement by a change the new public interface for libvirt | |
| 16:22:34 | openstackgerrit | Merged openstack/nova master: Get anchors for sharing providers https://review.openstack.org/565279 | |
| 16:22:42 | openstackgerrit | Merged openstack/nova master: Make association_refresh configurable https://review.openstack.org/565526 | |
| 16:23:00 | efried | bauzas: Have you been following the xen driver work for vgpus? | |
| 16:23:08 | bauzas | efried: barely but yes | |
| 16:23:10 | efried | bauzas: I think they're the furthest along with using upt in their driver. | |
| 16:23:30 | bauzas | efried: well, I'm pretty close too | |
| 16:23:35 | efried | bauzas: Of course, they're based on code that hasn't merged and/or won't work until the nrp-in-alloc-cands series lands. | |
| 16:23:55 | bauzas | efried: last change in my series is https://review.openstack.org/#/c/564215/ | |
| 16:24:13 | bauzas | efried: I know, and I was about to propose your series for a runway slot | |
| 16:24:42 | bauzas | I'm a bit out of review as of those days due to my patch series, but I can switch to reviewing if I'm blocked due to that series :p | |
| 16:25:06 | bauzas | anyway, I need to drop for today | |
| 16:25:08 | efried | bauzas: Granular is ready as well. (Except both series are going to bounce on merge conflict soon) | |
| 16:25:32 | bauzas | granular is already a runway candidate | |
| 16:25:32 | jaypipes | bauzas: I'm a strong -2 for having standard traits (or really, any traits) representing GPU vendor types | |
| 16:25:33 | efried | I guess I could put both of those series into the runway queue. | |
| 16:25:45 | efried | jaypipes: eh?? | |
| 16:25:50 | bauzas | jaypipes: I'm cool with that | |
| 16:25:57 | exarlos | mriedem: pastebin.com/SPpHu9Tu is the current error state of the VM. | |
| 16:26:11 | bauzas | jaypipes: I guess your concern is the os-traits explosion | |
| 16:26:14 | melwitt | looks like the gate is timing out a lot in response to me saying it had been good in the nova meeting | |
| 16:26:22 | efried | jaypipes: Oh, you're looking for GPU *features* as opposed to types. | |
| 16:26:25 | bauzas | melwitt: yup, I noticed that in the meeting too | |
| 16:26:33 | melwitt | :( sorry everyone | |
| 16:26:54 | exarlos | mriedem: I would think it's recoverable, but i'm still learning. | |
| 16:26:59 | bauzas | jaypipes: efried: anyway, I need to drop | |