| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 15:12:27 | jaypipes | bauzas: "sad" would be one way to put it, yes. | |
| 15:12:32 | sahid | yes i think we do that somewhere but i can't really remember | |
| 15:12:47 | stephenfin | same issue with SR-IOV though, right? | |
| 15:12:47 | bauzas | jaypipes: let's be gentlemen :p | |
| 15:13:08 | bauzas | jaypipes: so, before discussing about a solution, do you understand the problem ? | |
| 15:13:18 | sahid | stephenfin: yes probably you are right | |
| 15:13:43 | jaypipes | bauzas: yes, I fully understand the problem. | |
| 15:13:57 | bauzas | jaypipes: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html we don't mention that a single pGPU can have multiple types and we just suppose a single inventory of VGPU for each | |
| 15:14:47 | jaypipes | bauzas: I had numerous conversations with jianghuaw_ about this. | |
| 15:14:48 | bauzas | the fact is, once *one* mediated device is created, then *all* the others types are not possible for that pGPU | |
| 15:15:03 | stephenfin | sahid, sean-k-mooney[m]: The bigger concern I have is that a vhost-user port is configured to a given PMD based on the guest - not the routes | |
| 15:15:04 | bauzas | jaypipes: and what are your thoughts on that ? | |
| 15:15:52 | bauzas | jaypipes: we already somehow set inventories based on config option thru enabled_gpu_types tho | |
| 15:16:00 | jaypipes | bauzas: the only thing I really don't want is on-the-fly re-configuration of providers based on *what the user requested*. | |
| 15:16:15 | stephenfin | sahid, sean-k-mooney[m]: e.g. all vhost-user ports will be handled by a random PMD until the guest is attached, when that reallocation happens | |
| 15:16:17 | bauzas | jaypipes: it's not that, if I understand correctly your concern | |
| 15:16:45 | stephenfin | sahid, sean-k-mooney[m]: Assuming vhost-user ports that aren't associated with a guest even appear in output of 'ovs-appctl dpif-netdev/pmd-rxq-show' (I can't test it, grrr) | |
| 15:17:12 | jaypipes | bauzas: if we want to pre-define configuration of multiple supported vGPU types using a CONF option, so be it. I would prefer to stop adding yet more CONF options and instead handle inventory of providers using a provider-config YAML file format, but that ain't gonna happen apparently, so be it. | |
| 15:17:33 | bauzas | jaypipes: if we go on a direction where each pGPU has multiple inventories, each for a GPU type, then that said, yes it would be dynamically modified on an instance creation | |
| 15:17:48 | kashyap | alex_xu_: You're right; I can remove that extra test. The main test in test_driver.py takes care of the full config. Thanks for catching. | |
| 15:17:52 | bauzas | jaypipes: I can try to spec it, you know | |
| 15:18:05 | jaypipes | bauzas: that's exactly what I *don't* want. | |
| 15:18:05 | bauzas | jaypipes: the YAML file | |
| 15:18:15 | bauzas | jaypipes: okay cool, so we're aligned | |
| 15:18:18 | edleafe | bauzas: multiple inventories is not going to work | |
| 15:18:31 | bauzas | edleafe: if each inventory is behind a child RP | |
| 15:18:38 | bauzas | either way, sounds we're in agreement | |
| 15:18:42 | edleafe | yeah | |
| 15:18:51 | bauzas | I *don't* want to make things complicated | |
| 15:19:03 | bauzas | the problem is more about the specific config option | |
| 15:19:08 | bauzas | I understand jaypipes on that | |
| 15:19:15 | jaypipes | bauzas: it would be the same inventory of VGPU. But different child providers would be tagged with specific VGPU_TYPE_XXX traits, right? | |
| 15:19:37 | sean-k-mooney[m] | stephenfin: if they are not associate with a guest im not sure. when they do get added to a guest they might also change. | |
| 15:19:41 | bauzas | jaypipes: you're talking of the possibility to have children, each of them being a type ? | |
| 15:19:56 | bauzas | jaypipes: if so, each inventory would be different | |
| 15:20:06 | jaypipes | bauzas: no | |
| 15:20:10 | bauzas | because the total number of vGPUs you can create depends on your type | |
| 15:20:20 | jaypipes | bauzas: I'm saying the resource class would all be "VGPU" | |
| 15:20:21 | bauzas | oh, with the conf opt solution ? | |
| 15:20:34 | stephenfin | sean-k-mooney[m]: Yeah, it seems like a lot of magic (even for ovs-dpdk) to say "we have this route that uses this NIC, therefore the vhost-user port should be processed by this PMD thread" | |
| 15:20:39 | bauzas | yeah, it's still VGPU resource class and a trait for that type | |
| 15:20:49 | jaypipes | bauzas: right. | |
| 15:20:58 | bauzas | so each PGPU will be a leaf | |
| 15:21:11 | sean-k-mooney[m] | stephenfin: this wont work in general however as this would only work with vhost-user. the approch you need to enable need to work for any switch backend | |
| 15:21:22 | bauzas | with one inventory about the total number of vGPUs it can create *for the type defined by the opt* | |
| 15:21:25 | jaypipes | bauzas: each pGPU group, but yes. | |
| 15:21:33 | bauzas | plus the trait telling which type it is | |
| 15:21:42 | bauzas | jaypipes: libvirt doesn't have the notion of groups | |
| 15:21:43 | jaypipes | bauzas: right. | |
| 15:21:49 | jaypipes | bauzas: I know, but xen does. | |
| 15:22:21 | bauzas | sure, but from a placement perspective, a "PGPU" RP is, from a libvirt perspective, a PCI device, and from a xen perspective, a PGPU group | |
| 15:22:25 | stephenfin | sean-k-mooney[m]: Sure, but vhost-user would be a start. Once we have a way to expose this information, we can extend other backends | |
| 15:22:30 | bauzas | but both are reconciled | |
| 15:22:37 | jaypipes | ack | |
| 15:22:41 | bauzas | it's just a driver-only thing | |
| 15:23:00 | bauzas | okay, now question | |
| 15:23:07 | bauzas | does that need to be spec'd up ? | |
| 15:23:11 | bauzas | jaypipes:^ | |
| 15:23:57 | sean-k-mooney[m] | stephenfin: if we wanted to do anything regardign the interface rx queue it likely should be an os-vif thing where we calulate teh best pmd our selves and set that not the other way around | |
| 15:24:13 | jaypipes | bauzas: yes. new CONF option, new way of behaving for the virt drivers. I would say yes. | |
| 15:24:35 | openstackgerrit | Merged openstack/nova master: tox: Remove unnecessary configuration https://review.openstack.org/556544 | |
| 15:26:09 | sean-k-mooney[m] | stephenfin: most backend wont that this info so vhost-user is not something we should build on | |
| 15:26:36 | stephenfin | sean-k-mooney[m]: What other backends would there be? | |
| 15:26:54 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | |
| 15:26:57 | sean-k-mooney[m] | stephenfin: for example kernel ovs with kernel vhost has kerne vhost-treads it spwans per interface, it dose not have pmds, so while we can taskset those threads we cant affinites them via looking a queues | |
| 15:27:18 | bauzas | jaypipes: fine by me | |
| 15:27:51 | bauzas | artom: had some concerns about the virt driver "calling" the conductor in https://review.openstack.org/#/c/552722/6 | |
| 15:28:24 | artom | bauzas, replied on the patch, but we can continue here if you want. Basically, no new calls are added, it's just existing methods/returns with new data in them | |
| 15:28:30 | artom | Should I clarify the spec? | |
| 15:28:30 | bauzas | artom: maybe I misunderstood but you mean that the compute service would call back the conductor ? | |
| 15:28:31 | alex_xu_ | kashyap: np | |
| 15:28:45 | bauzas | artom: that would help | |
| 15:29:03 | edmondsw | mriedem I don't remember ever discussing a service-protected-server case in our RBAC discussions, but I don't think you need to use service tokens there... just have a role that's given permission to manage these | |
| 15:29:12 | sean-k-mooney[m] | stephenfin: in any case queue mapping or tasksetting of vhost threads i think is out of scope of nova | |
| 15:29:26 | edmondsw | that spec needs a lot of work if anything's to happen there | |
| 15:30:10 | bauzas | artom: also, on the fact we would check the compute version | |
| 15:30:32 | bauzas | artom: does a live migration work with two different compute versions already ? | |
| 15:30:34 | openstackgerrit | Merged openstack/nova master: Move placement test cases from db to placement https://review.openstack.org/553149 | |
| 15:30:40 | mriedem | bauzas: yes | |
| 15:30:48 | mriedem | we support mixed version computes for live migration | |
| 15:30:49 | artom | bauzas, I didn't see anything in the code that would suggest it wouldn't | |
| 15:30:58 | bauzas | mriedem: context is https://review.openstack.org/#/c/552722/6/specs/rocky/approved/numa-aware-live-migration.rst@244 | |
| 15:30:59 | mriedem | we have a grenade + live migratoin job that also does this back and forth | |
| 15:30:59 | sean-k-mooney[m] | linux bridge, sriov, hardware offloaded ovs, vpp, iovision(ebpf), mini net, calico and macvtap are the main ones used beyond ovs/ovs-dpdk | |
| 15:31:25 | bauzas | mriedem: then, I was claiming we could microversion the behavioural change artom is going to introduce | |
| 15:31:26 | sean-k-mooney[m] | oh there is also snabb switch | |
| 15:31:27 | mriedem | that's a bit drastic | |
| 15:31:32 | artom | bauzas, also, we'd only fail it if the instance has "NUMA" characteristics | |
| 15:31:36 | mriedem | sec | |
| 15:31:50 | artom | Which is currently broken anyways, so not as drastic as it sounds | |
| 15:31:55 | bauzas | artom: I agree, and we don't test that AFAIK | |
| 15:31:55 | mriedem | see https://review.openstack.org/#/c/522537/13/nova/conductor/tasks/live_migrate.py | |
| 15:32:06 | mriedem | you only need to know if the source and dest computes can do the new hotness | |
| 15:32:35 | mriedem | otherwise it's a novalidhost if you can't find a pair | |
| 15:32:42 | bauzas | mriedem: yeah I remember that change | |
| 15:33:29 | sean-k-mooney[m] | stephenfin: on and NTT have a dpdk soft patch panel thing , the point is nova cant have special case code for all of these | |
| 15:35:21 | bauzas | mriedem: if the implementation of artom's spec would go into that direction, then yes we wouldn't need a microversion | |
| 15:35:35 | mriedem | i think we can agree that we don't want to add subnet to the requested network turducken in the compute api https://review.openstack.org/#/c/518227/ | |
| 15:37:11 | bauzas | mriedem: oh please, yes | |
| 15:37:33 | bauzas | just create the port in neutron and pass it to nova | |