| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-03 | |||
| 16:00:45 | bauzas | they could name the flavor GRID M60-0Q I'm fine | |
| 16:01:01 | efried | bauzas: So what I'm trying to figure out is whether there's such a thing as a flavor that would include a VGPU, that's possible to use to deploy instances on different hypervisors? | |
| 16:01:04 | bauzas | but the extra spec would be "requested=nvidia-11" | |
| 16:01:23 | bauzas | efried: mixing environments is highly discouraged | |
| 16:01:41 | efried | Well then I think it doesn't matter whether the traits are named the same from one hypervisor to the next. | |
| 16:01:41 | bauzas | you can do that but then split your cloud | |
| 16:01:56 | bauzas | efried: that said, there is a flaw | |
| 16:01:57 | exarlos | mriedem: and also, should I expect each "server" to have a respective folder in /var/lib/nova/instances/<uuid> ? | |
| 16:02:17 | bauzas | efried: the mdev GPU type name depends on the nvidia driver | |
| 16:02:48 | bauzas | efried: since it depends on a vendor driver, we have no guarantee that 'nvidia-11' will still correspond to a specific GPU type in the future | |
| 16:03:14 | efried | bauzas: Right, so that's what I meant by 1-1 | |
| 16:03:17 | bauzas | efried: if nvidia bumps a new driver that passes types that are not backwards compatible, then operators need to set again their flavors | |
| 16:03:40 | bauzas | that's not a public API, and I hate that | |
| 16:04:02 | bauzas | the kernel just leaks out what the VFIO device tells | |
| 16:04:03 | efried | bauzas: So wait, it's possible that 'nvidia-11' today corresponds to M60-0B and tomorrow will correspond to M99-0X ?? | |
| 16:04:10 | bauzas | efried: possibly yeah | |
| 16:04:14 | bauzas | crazy, isn't it ? | |
| 16:04:14 | efried | that's bullshit, yo. | |
| 16:04:23 | bauzas | that's a vendor API | |
| 16:04:34 | bauzas | vendors can do crazy things | |
| 16:04:44 | bauzas | including breaking changes | |
| 16:05:09 | mriedem | exarlos: without knowing where this actually failed and how it failed, it's hard to say | |
| 16:05:33 | mriedem | if the guest is still running on the source host, which you could verify via virsh (assuming libvirt), then you could probably just reset the state of the instance | |
| 16:05:35 | efried | bauzas: Well, we can only do so much. I think this mapping is something the virt driver will need to be able to keep track of. And if the stoopid vendor driver makes a breaking change, the virt driver will just have to scramble to accomodate. | |
| 16:05:49 | bauzas | efried: I think I reached the same conclusion | |
| 16:05:50 | mriedem | https://developer.openstack.org/api-ref/compute/#reset-server-state-os-resetstate-action | |
| 16:06:06 | bauzas | efried: for some reason, xen is better designed than libvirt | |
| 16:06:06 | efried | bauzas: But I think as much as we possibly can, we should create standard traits and ask the virt drivers to map to them. | |
| 16:06:15 | bauzas | efried: it already makes the translation I guess | |
| 16:06:33 | efried | bauzas: Which one is doing translation, though? | |
| 16:06:37 | bauzas | efried: because if you look the Xen GPU types, those look very identical to the nvidia doc | |
| 16:06:45 | bauzas | efried: the xen API | |
| 16:06:50 | efried | bauzas: Right, so it's libvirt that's translating. | |
| 16:06:57 | bauzas | efried: no | |
| 16:06:59 | efried | xen is just using 'em as is | |
| 16:07:01 | efried | no? | |
| 16:07:13 | bauzas | efried: libvirt is straight setting what the kernel module tells | |
| 16:07:14 | bauzas | https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt | |
| 16:07:49 | efried | oh, so xen is mapping those back to the model names in the literature | |
| 16:08:00 | exarlos | mriedem: Let me go have a read of that. | |
| 16:08:17 | exarlos | It is on the same host still. | |
| 16:08:24 | bauzas | efried: that's what I guess, yes | |
| 16:08:38 | efried | bauzas: so the point is that we need to choose one or the other, and then the docs for a given hypervisor have to tell me how to discover and determine the appropriate corresponding trait name. | |
| 16:08:52 | bauzas | efried: ok, I take the challenge | |
| 16:09:20 | bauzas | efried: I'm in favor of an unified trait name | |
| 16:09:20 | efried | bauzas: Which needs to happen regardless, because at the very least 'nvidia-11' will need to be translated to 'trait:HW_VGPU_TYPE_NVIDIA_11=required' | |
| 16:09:31 | bauzas | yup | |
| 16:10:14 | efried | bauzas: What would be neat is if each hypervisor provided a script to discover the devs and list them along with their trait names. | |
| 16:10:28 | efried | that might be asking too much, though :) | |
| 16:10:31 | bauzas | efried: what I wonder if whether the mapping should be hardcoded in the virt driver code, or if we should leave operators setting that | |
| 16:10:45 | bauzas | efried: heh, I have the same wish | |
| 16:10:50 | bauzas | efried: xen does that | |
| 16:10:57 | bauzas | efried: but see my docs on libvirt usage | |
| 16:11:02 | bauzas | that's crazy | |
| 16:11:10 | bauzas | you have to lookup sysfs | |
| 16:11:21 | efried | bauzas: The operator mapping thing - that's going back to the yaml config we started brainstorming in Denver. | |
| 16:11:26 | bauzas | we don't have any libvirt API that shows you that | |
| 16:11:52 | bauzas | efried: I know | |
| 16:12:06 | efried | bauzas: BUT that mapping is going to be more for device aliasing, white/blacklisting, etc. I don't think we should be using it to define traits, unless maybe they're CUSTOM_* | |
| 16:12:19 | bauzas | efried: just a thought | |
| 16:12:26 | bauzas | efried: about custom traits | |
| 16:12:29 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add 'numa-aware-vswitches' spec https://review.openstack.org/541290 | |
| 16:12:33 | bauzas | efried: can you set those with nested RPs ? | |
| 16:12:52 | bauzas | efried: because then, I wouldn't care a bit about that | |
| 16:13:07 | bauzas | efried: I could just ask operators to provide custom traits | |
| 16:13:19 | efried | bauzas: You can set traits on any RP. We originally talked about traits "inheriting" or "propagating" in various directions, but we're not doing that. | |
| 16:13:31 | efried | bauzas: "on any RP" including child, sharing, etc. | |
| 16:13:31 | bauzas | efried: I'm a bit rusty on traits | |
| 16:13:46 | bauzas | efried: but I guess operators have to play with placement client to set the trait ? | |
| 16:13:56 | bauzas | if so, that's perfect | |
| 16:14:44 | bauzas | jaypipes: for your sanity, don't read the vendor crazypants I wrote | |
| 16:15:08 | efried | bauzas: Well, what we talked about in Dublin is that each of various entities (nova-compute, neutron, operator) will have a set of traits it owns/controls for a given provider. It's allowed to set/unset those traits, but the other entities are not. | |
| 16:15:42 | efried | bauzas: So in this example, I would expect nova-compute (specifically the virt driver) to "own" the trait indicating the GPU type. The operator doesn't get to set that. That wouldn't make sense. | |
| 16:16:02 | bauzas | efried: keep in mind we will have one inventory per vGPU type | |
| 16:16:18 | efried | bauzas: Same entity owns the inventory. | |
| 16:16:49 | bauzas | efried: what if I want to set custom traits ? | |
| 16:17:26 | bauzas | efried: say the virt driver now provides a tree of a root RP and two children RP, each one being a vGPU type | |
| 16:17:44 | bauzas | efried: as an operator, I'd like to pick only one type based on a custom trait | |
| 16:17:50 | efried | bauzas: From this perspective, we don't really make a distinction between standard and custom traits. But I would say the op is allowed to invent and assign a random custom trait as she sees fit. | |
| 16:17:54 | bauzas | efried: exactly what we wrote in the spec, actually | |
| 16:18:13 | bauzas | https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html | |
| 16:18:14 | efried | bauzas: Legit. Basically the op just has to be careful not to stomp on a trait owned by compute. | |
| 16:18:24 | efried | bauzas: beyond that, sky's the limit. | |
| 16:18:38 | efried | bauzas: which is exactly why we made this a requirement in Dublin. | |
| 16:18:46 | efried | (dansmith and I almost came to blows over it) | |
| 16:18:51 | bauzas | yeah I remember | |
| 16:19:14 | efried | I still believe it's going to bite us, but time will tell. | |
| 16:19:36 | bauzas | efried: ok, so what's the way to set custom traits as of now ? | |
| 16:19:42 | bauzas | thru placement API ? (and the client) | |
| 16:19:55 | efried | bauzas: Sure, that works. | |
| 16:20:15 | bauzas | efried: ok, what I need is testing then | |
| 16:20:30 | bauzas | efried: my patch series is close to provide inventories using update_provider_tree() | |
| 16:20:44 | efried | bauzas: There's no change to the placement API for any of that; the only thing we did was update the u-p-t spec/docs to make it clear that a dance is required to make sure you're not stepping on each other. | |
| 16:21:12 | efried | bauzas: Oh, are you aware of https://review.openstack.org/#/c/560444/ ? | |
| 16:21:12 | bauzas | efried: ok, will look | |
| 16:21:40 | bauzas | efried: no, I wasn't | |
| 16:21:54 | bauzas | efried: my top patch in the series returns a dict of dicts | |
| 16:22:08 | bauzas | efried: I'll use that dict of dicts to populate the tree | |
| 16:22:31 | bauzas | efried: but I was considering to implement by a change the new public interface for libvirt | |
| 16:22:34 | openstackgerrit | Merged openstack/nova master: Get anchors for sharing providers https://review.openstack.org/565279 | |