| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 15:04:00 | jaypipes | gibi: maybe I'm just being thick... I still don't get it. You will have multiple backends on the same compute host supporting the same physical networks that support the same vNIC types and you want to be able to choose which backend to allocate a piece of bandwidth from? | |
| 15:04:12 | bauzas | jaypipes: I'm not sure we need a spec for https://review.openstack.org/#/c/556873/1/specs/rocky/approved/nested-resource-providers-allocation-candidates.rst | |
| 15:04:24 | bauzas | jaypipes: it's just fixes we need to merge IMHO | |
| 15:05:01 | mriedem | bauzas: it's an api change and microversion right? | |
| 15:05:03 | mriedem | so spec is required yeah | |
| 15:05:05 | mriedem | ? | |
| 15:05:17 | gibi | jaypipes: exactly. Neutron today does it in the following way: | |
| 15:05:28 | gibi | jaypipes: Neutron has a mechnism driver config | |
| 15:05:32 | bauzas | mriedem: from the spec itself, looks like it's not changing the API | |
| 15:05:37 | stephenfin | sean-k-mooney[m]: Potentially dumb question, but in what way would lack of guest memory be related? | |
| 15:05:47 | gibi | jaypipes: Neutron iterates throught that list and try binding the port with the given driver | |
| 15:05:48 | jaypipes | mriedem: no API change, no. | |
| 15:06:05 | stephenfin | I figured if you could do 'ovs-appctl dpif-netdev/pmd-rxq-show' to show the affinity for a given interface and just return that | |
| 15:06:12 | gibi | jaypipes: the first driver that returns a positive result from that bind call will be the one Neutron use | |
| 15:06:30 | gibi | jaypipes: so that config option defines a preference order between backends supporting the same physnet | |
| 15:06:34 | mriedem | jaypipes: bauzas: it's a behavior change for the alloc candidates api though | |
| 15:06:35 | stephenfin | Or does that even exist before the port is attached to the interface? | |
| 15:06:45 | bhagyashris | mriedem: thank you for review i will look into it as i am working in IST time zone it's end of day for me :) | |
| 15:06:53 | stephenfin | this would be so much easier if I have a machine to experiment on. Stupid fried motherboard is killing me :( | |
| 15:07:07 | jaypipes | mriedem: if "behaviour change" means "it will work when there are nested providers", then yes. :) | |
| 15:07:12 | bauzas | mriedem: technically, alloc-candidates doesn't work yet with nested RPs | |
| 15:07:19 | jaypipes | stephenfin: at least it's not an efried motherboard. | |
| 15:07:25 | bauzas | mriedem: it's not changing the existing | |
| 15:07:35 | mriedem | bauzas: technically volume-backed rebuild with a new image doesn't work either, | |
| 15:07:40 | mriedem | but if we make that work, it's an api change | |
| 15:07:51 | mriedem | even if the request params don't change | |
| 15:08:01 | jaypipes | does it even matter guys? the spec is up. | |
| 15:08:01 | bauzas | I agree it's a signal | |
| 15:08:12 | mriedem | yeah i'm saying it's a spec, | |
| 15:08:15 | mriedem | and likely a microversion bump | |
| 15:08:26 | bauzas | okay, I'll comment that too then | |
| 15:09:47 | bauzas | mriedem: jaypipes: speaking of specs | |
| 15:10:03 | bauzas | jaypipes: now I'm back, can we discuss about my point with vGPU types ? | |
| 15:10:17 | bauzas | looks like you were sad about that | |
| 15:11:05 | sahid | stephenfin, sean-k-mooney[m] I think the issue is on the scheduling, we will have first to select a host so then we could create the ports and query them | |
| 15:11:28 | stephenfin | sahid: I'd be perfectly fine saying we need pre-created ports for this thing | |
| 15:11:54 | stephenfin | That's already a requirement for SR-IOV and afaik is the plan for gibi's bandwidth-aware scheduling spec | |
| 15:12:18 | sahid | stephenfin: the problem is then you need to raise that re-schedule excpetion is the resources are not enough | |
| 15:12:27 | jaypipes | bauzas: "sad" would be one way to put it, yes. | |
| 15:12:32 | sahid | yes i think we do that somewhere but i can't really remember | |
| 15:12:47 | stephenfin | same issue with SR-IOV though, right? | |
| 15:12:47 | bauzas | jaypipes: let's be gentlemen :p | |
| 15:13:08 | bauzas | jaypipes: so, before discussing about a solution, do you understand the problem ? | |
| 15:13:18 | sahid | stephenfin: yes probably you are right | |
| 15:13:43 | jaypipes | bauzas: yes, I fully understand the problem. | |
| 15:13:57 | bauzas | jaypipes: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html we don't mention that a single pGPU can have multiple types and we just suppose a single inventory of VGPU for each | |
| 15:14:47 | jaypipes | bauzas: I had numerous conversations with jianghuaw_ about this. | |
| 15:14:48 | bauzas | the fact is, once *one* mediated device is created, then *all* the others types are not possible for that pGPU | |
| 15:15:03 | stephenfin | sahid, sean-k-mooney[m]: The bigger concern I have is that a vhost-user port is configured to a given PMD based on the guest - not the routes | |
| 15:15:04 | bauzas | jaypipes: and what are your thoughts on that ? | |
| 15:15:52 | bauzas | jaypipes: we already somehow set inventories based on config option thru enabled_gpu_types tho | |
| 15:16:00 | jaypipes | bauzas: the only thing I really don't want is on-the-fly re-configuration of providers based on *what the user requested*. | |
| 15:16:15 | stephenfin | sahid, sean-k-mooney[m]: e.g. all vhost-user ports will be handled by a random PMD until the guest is attached, when that reallocation happens | |
| 15:16:17 | bauzas | jaypipes: it's not that, if I understand correctly your concern | |
| 15:16:45 | stephenfin | sahid, sean-k-mooney[m]: Assuming vhost-user ports that aren't associated with a guest even appear in output of 'ovs-appctl dpif-netdev/pmd-rxq-show' (I can't test it, grrr) | |
| 15:17:12 | jaypipes | bauzas: if we want to pre-define configuration of multiple supported vGPU types using a CONF option, so be it. I would prefer to stop adding yet more CONF options and instead handle inventory of providers using a provider-config YAML file format, but that ain't gonna happen apparently, so be it. | |
| 15:17:33 | bauzas | jaypipes: if we go on a direction where each pGPU has multiple inventories, each for a GPU type, then that said, yes it would be dynamically modified on an instance creation | |
| 15:17:48 | kashyap | alex_xu_: You're right; I can remove that extra test. The main test in test_driver.py takes care of the full config. Thanks for catching. | |
| 15:17:52 | bauzas | jaypipes: I can try to spec it, you know | |
| 15:18:05 | jaypipes | bauzas: that's exactly what I *don't* want. | |
| 15:18:05 | bauzas | jaypipes: the YAML file | |
| 15:18:15 | bauzas | jaypipes: okay cool, so we're aligned | |
| 15:18:18 | edleafe | bauzas: multiple inventories is not going to work | |
| 15:18:31 | bauzas | edleafe: if each inventory is behind a child RP | |
| 15:18:38 | bauzas | either way, sounds we're in agreement | |
| 15:18:42 | edleafe | yeah | |
| 15:18:51 | bauzas | I *don't* want to make things complicated | |
| 15:19:03 | bauzas | the problem is more about the specific config option | |
| 15:19:08 | bauzas | I understand jaypipes on that | |
| 15:19:15 | jaypipes | bauzas: it would be the same inventory of VGPU. But different child providers would be tagged with specific VGPU_TYPE_XXX traits, right? | |
| 15:19:37 | sean-k-mooney[m] | stephenfin: if they are not associate with a guest im not sure. when they do get added to a guest they might also change. | |
| 15:19:41 | bauzas | jaypipes: you're talking of the possibility to have children, each of them being a type ? | |
| 15:19:56 | bauzas | jaypipes: if so, each inventory would be different | |
| 15:20:06 | jaypipes | bauzas: no | |
| 15:20:10 | bauzas | because the total number of vGPUs you can create depends on your type | |
| 15:20:20 | jaypipes | bauzas: I'm saying the resource class would all be "VGPU" | |
| 15:20:21 | bauzas | oh, with the conf opt solution ? | |
| 15:20:34 | stephenfin | sean-k-mooney[m]: Yeah, it seems like a lot of magic (even for ovs-dpdk) to say "we have this route that uses this NIC, therefore the vhost-user port should be processed by this PMD thread" | |
| 15:20:39 | bauzas | yeah, it's still VGPU resource class and a trait for that type | |
| 15:20:49 | jaypipes | bauzas: right. | |
| 15:20:58 | bauzas | so each PGPU will be a leaf | |
| 15:21:11 | sean-k-mooney[m] | stephenfin: this wont work in general however as this would only work with vhost-user. the approch you need to enable need to work for any switch backend | |
| 15:21:22 | bauzas | with one inventory about the total number of vGPUs it can create *for the type defined by the opt* | |
| 15:21:25 | jaypipes | bauzas: each pGPU group, but yes. | |
| 15:21:33 | bauzas | plus the trait telling which type it is | |
| 15:21:42 | bauzas | jaypipes: libvirt doesn't have the notion of groups | |
| 15:21:43 | jaypipes | bauzas: right. | |
| 15:21:49 | jaypipes | bauzas: I know, but xen does. | |
| 15:22:21 | bauzas | sure, but from a placement perspective, a "PGPU" RP is, from a libvirt perspective, a PCI device, and from a xen perspective, a PGPU group | |
| 15:22:25 | stephenfin | sean-k-mooney[m]: Sure, but vhost-user would be a start. Once we have a way to expose this information, we can extend other backends | |
| 15:22:30 | bauzas | but both are reconciled | |
| 15:22:37 | jaypipes | ack | |
| 15:22:41 | bauzas | it's just a driver-only thing | |
| 15:23:00 | bauzas | okay, now question | |
| 15:23:07 | bauzas | does that need to be spec'd up ? | |
| 15:23:11 | bauzas | jaypipes:^ | |
| 15:23:57 | sean-k-mooney[m] | stephenfin: if we wanted to do anything regardign the interface rx queue it likely should be an os-vif thing where we calulate teh best pmd our selves and set that not the other way around | |
| 15:24:13 | jaypipes | bauzas: yes. new CONF option, new way of behaving for the virt drivers. I would say yes. | |
| 15:24:35 | openstackgerrit | Merged openstack/nova master: tox: Remove unnecessary configuration https://review.openstack.org/556544 | |
| 15:26:09 | sean-k-mooney[m] | stephenfin: most backend wont that this info so vhost-user is not something we should build on | |