| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 15:05:47 | gibi | jaypipes: Neutron iterates throught that list and try binding the port with the given driver | |
| 15:05:48 | jaypipes | mriedem: no API change, no. | |
| 15:06:05 | stephenfin | I figured if you could do 'ovs-appctl dpif-netdev/pmd-rxq-show' to show the affinity for a given interface and just return that | |
| 15:06:12 | gibi | jaypipes: the first driver that returns a positive result from that bind call will be the one Neutron use | |
| 15:06:30 | gibi | jaypipes: so that config option defines a preference order between backends supporting the same physnet | |
| 15:06:34 | mriedem | jaypipes: bauzas: it's a behavior change for the alloc candidates api though | |
| 15:06:35 | stephenfin | Or does that even exist before the port is attached to the interface? | |
| 15:06:45 | bhagyashris | mriedem: thank you for review i will look into it as i am working in IST time zone it's end of day for me :) | |
| 15:06:53 | stephenfin | this would be so much easier if I have a machine to experiment on. Stupid fried motherboard is killing me :( | |
| 15:07:07 | jaypipes | mriedem: if "behaviour change" means "it will work when there are nested providers", then yes. :) | |
| 15:07:12 | bauzas | mriedem: technically, alloc-candidates doesn't work yet with nested RPs | |
| 15:07:19 | jaypipes | stephenfin: at least it's not an efried motherboard. | |
| 15:07:25 | bauzas | mriedem: it's not changing the existing | |
| 15:07:35 | mriedem | bauzas: technically volume-backed rebuild with a new image doesn't work either, | |
| 15:07:40 | mriedem | but if we make that work, it's an api change | |
| 15:07:51 | mriedem | even if the request params don't change | |
| 15:08:01 | bauzas | I agree it's a signal | |
| 15:08:01 | jaypipes | does it even matter guys? the spec is up. | |
| 15:08:12 | mriedem | yeah i'm saying it's a spec, | |
| 15:08:15 | mriedem | and likely a microversion bump | |
| 15:08:26 | bauzas | okay, I'll comment that too then | |
| 15:09:47 | bauzas | mriedem: jaypipes: speaking of specs | |
| 15:10:03 | bauzas | jaypipes: now I'm back, can we discuss about my point with vGPU types ? | |
| 15:10:17 | bauzas | looks like you were sad about that | |
| 15:11:05 | sahid | stephenfin, sean-k-mooney[m] I think the issue is on the scheduling, we will have first to select a host so then we could create the ports and query them | |
| 15:11:28 | stephenfin | sahid: I'd be perfectly fine saying we need pre-created ports for this thing | |
| 15:11:54 | stephenfin | That's already a requirement for SR-IOV and afaik is the plan for gibi's bandwidth-aware scheduling spec | |
| 15:12:18 | sahid | stephenfin: the problem is then you need to raise that re-schedule excpetion is the resources are not enough | |
| 15:12:27 | jaypipes | bauzas: "sad" would be one way to put it, yes. | |
| 15:12:32 | sahid | yes i think we do that somewhere but i can't really remember | |
| 15:12:47 | bauzas | jaypipes: let's be gentlemen :p | |
| 15:12:47 | stephenfin | same issue with SR-IOV though, right? | |
| 15:13:08 | bauzas | jaypipes: so, before discussing about a solution, do you understand the problem ? | |
| 15:13:18 | sahid | stephenfin: yes probably you are right | |
| 15:13:43 | jaypipes | bauzas: yes, I fully understand the problem. | |
| 15:13:57 | bauzas | jaypipes: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html we don't mention that a single pGPU can have multiple types and we just suppose a single inventory of VGPU for each | |
| 15:14:47 | jaypipes | bauzas: I had numerous conversations with jianghuaw_ about this. | |
| 15:14:48 | bauzas | the fact is, once *one* mediated device is created, then *all* the others types are not possible for that pGPU | |
| 15:15:03 | stephenfin | sahid, sean-k-mooney[m]: The bigger concern I have is that a vhost-user port is configured to a given PMD based on the guest - not the routes | |
| 15:15:04 | bauzas | jaypipes: and what are your thoughts on that ? | |
| 15:15:52 | bauzas | jaypipes: we already somehow set inventories based on config option thru enabled_gpu_types tho | |
| 15:16:00 | jaypipes | bauzas: the only thing I really don't want is on-the-fly re-configuration of providers based on *what the user requested*. | |
| 15:16:15 | stephenfin | sahid, sean-k-mooney[m]: e.g. all vhost-user ports will be handled by a random PMD until the guest is attached, when that reallocation happens | |
| 15:16:17 | bauzas | jaypipes: it's not that, if I understand correctly your concern | |
| 15:16:45 | stephenfin | sahid, sean-k-mooney[m]: Assuming vhost-user ports that aren't associated with a guest even appear in output of 'ovs-appctl dpif-netdev/pmd-rxq-show' (I can't test it, grrr) | |
| 15:17:12 | jaypipes | bauzas: if we want to pre-define configuration of multiple supported vGPU types using a CONF option, so be it. I would prefer to stop adding yet more CONF options and instead handle inventory of providers using a provider-config YAML file format, but that ain't gonna happen apparently, so be it. | |
| 15:17:33 | bauzas | jaypipes: if we go on a direction where each pGPU has multiple inventories, each for a GPU type, then that said, yes it would be dynamically modified on an instance creation | |
| 15:17:48 | kashyap | alex_xu_: You're right; I can remove that extra test. The main test in test_driver.py takes care of the full config. Thanks for catching. | |
| 15:17:52 | bauzas | jaypipes: I can try to spec it, you know | |
| 15:18:05 | bauzas | jaypipes: the YAML file | |
| 15:18:05 | jaypipes | bauzas: that's exactly what I *don't* want. | |
| 15:18:15 | bauzas | jaypipes: okay cool, so we're aligned | |
| 15:18:18 | edleafe | bauzas: multiple inventories is not going to work | |
| 15:18:31 | bauzas | edleafe: if each inventory is behind a child RP | |
| 15:18:38 | bauzas | either way, sounds we're in agreement | |
| 15:18:42 | edleafe | yeah | |
| 15:18:51 | bauzas | I *don't* want to make things complicated | |
| 15:19:03 | bauzas | the problem is more about the specific config option | |
| 15:19:08 | bauzas | I understand jaypipes on that | |
| 15:19:15 | jaypipes | bauzas: it would be the same inventory of VGPU. But different child providers would be tagged with specific VGPU_TYPE_XXX traits, right? | |
| 15:19:37 | sean-k-mooney[m] | stephenfin: if they are not associate with a guest im not sure. when they do get added to a guest they might also change. | |
| 15:19:41 | bauzas | jaypipes: you're talking of the possibility to have children, each of them being a type ? | |
| 15:19:56 | bauzas | jaypipes: if so, each inventory would be different | |
| 15:20:06 | jaypipes | bauzas: no | |
| 15:20:10 | bauzas | because the total number of vGPUs you can create depends on your type | |
| 15:20:20 | jaypipes | bauzas: I'm saying the resource class would all be "VGPU" | |
| 15:20:21 | bauzas | oh, with the conf opt solution ? | |
| 15:20:34 | stephenfin | sean-k-mooney[m]: Yeah, it seems like a lot of magic (even for ovs-dpdk) to say "we have this route that uses this NIC, therefore the vhost-user port should be processed by this PMD thread" | |
| 15:20:39 | bauzas | yeah, it's still VGPU resource class and a trait for that type | |
| 15:20:49 | jaypipes | bauzas: right. | |
| 15:20:58 | bauzas | so each PGPU will be a leaf | |
| 15:21:11 | sean-k-mooney[m] | stephenfin: this wont work in general however as this would only work with vhost-user. the approch you need to enable need to work for any switch backend | |
| 15:21:22 | bauzas | with one inventory about the total number of vGPUs it can create *for the type defined by the opt* | |
| 15:21:25 | jaypipes | bauzas: each pGPU group, but yes. | |
| 15:21:33 | bauzas | plus the trait telling which type it is | |
| 15:21:42 | bauzas | jaypipes: libvirt doesn't have the notion of groups | |
| 15:21:43 | jaypipes | bauzas: right. | |
| 15:21:49 | jaypipes | bauzas: I know, but xen does. | |
| 15:22:21 | bauzas | sure, but from a placement perspective, a "PGPU" RP is, from a libvirt perspective, a PCI device, and from a xen perspective, a PGPU group | |
| 15:22:25 | stephenfin | sean-k-mooney[m]: Sure, but vhost-user would be a start. Once we have a way to expose this information, we can extend other backends | |
| 15:22:30 | bauzas | but both are reconciled | |
| 15:22:37 | jaypipes | ack | |
| 15:22:41 | bauzas | it's just a driver-only thing | |
| 15:23:00 | bauzas | okay, now question | |
| 15:23:07 | bauzas | does that need to be spec'd up ? | |
| 15:23:11 | bauzas | jaypipes:^ | |
| 15:23:57 | sean-k-mooney[m] | stephenfin: if we wanted to do anything regardign the interface rx queue it likely should be an os-vif thing where we calulate teh best pmd our selves and set that not the other way around | |
| 15:24:13 | jaypipes | bauzas: yes. new CONF option, new way of behaving for the virt drivers. I would say yes. | |
| 15:24:35 | openstackgerrit | Merged openstack/nova master: tox: Remove unnecessary configuration https://review.openstack.org/556544 | |
| 15:26:09 | sean-k-mooney[m] | stephenfin: most backend wont that this info so vhost-user is not something we should build on | |
| 15:26:36 | stephenfin | sean-k-mooney[m]: What other backends would there be? | |
| 15:26:54 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | |
| 15:26:57 | sean-k-mooney[m] | stephenfin: for example kernel ovs with kernel vhost has kerne vhost-treads it spwans per interface, it dose not have pmds, so while we can taskset those threads we cant affinites them via looking a queues | |
| 15:27:18 | bauzas | jaypipes: fine by me | |
| 15:27:51 | bauzas | artom: had some concerns about the virt driver "calling" the conductor in https://review.openstack.org/#/c/552722/6 | |
| 15:28:24 | artom | bauzas, replied on the patch, but we can continue here if you want. Basically, no new calls are added, it's just existing methods/returns with new data in them | |
| 15:28:30 | bauzas | artom: maybe I misunderstood but you mean that the compute service would call back the conductor ? | |
| 15:28:30 | artom | Should I clarify the spec? | |
| 15:28:31 | alex_xu_ | kashyap: np | |
| 15:28:45 | bauzas | artom: that would help | |