| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-25 | |||
| 13:44:27 | bauzas | like enabled_vgpu_types = ['nvidia-11', 'nvidia-10'] | |
| 13:44:38 | bauzas | yeah, that is too specific | |
| 13:45:08 | bauzas | jianghuaw: ^ | |
| 13:45:22 | edleafe | Scheduler subteam meeting in 15 minutes in #openstack-meeting-alt | |
| 13:45:54 | jianghuaw | how about use the vGPU type's name instead of the id? by considering the type name is common from both XenServer and kvm? enabled_vgpu_types = ['GRID K160Q'...] | |
| 13:46:44 | jianghuaw | As I explained in the comment, both virts can get the type name - "GRID K160" | |
| 13:47:09 | jianghuaw | And that's also the name recorded in the user guide. | |
| 13:48:16 | sdague | mriedem: with you and I both with fingers in the qemu 2.10 patch, you want a 4th core to look into it - https://review.openstack.org/#/c/505673/ - or you want to just make sure my update isn't crazy and put it in | |
| 13:48:26 | bauzas | jianghuaw: and what if I have two exact same cards ? | |
| 13:48:49 | sahid | mriedem: did you notice my request, continuing development on /pci and in parallel working on porting virt features on RP | |
| 13:49:01 | mriedem | sdague: i looked at it last week, which is how we went down the paused rabbit hole | |
| 13:49:09 | mriedem | sdague: but we have plenty of cores, what's one more to review | |
| 13:49:10 | sahid | by this way we could still provide proction-ready features and have at some point a deprecating phease | |
| 13:49:22 | sdague | mriedem: right, that's fixed now | |
| 13:49:49 | mriedem | sahid: this? https://review.openstack.org/#/c/485522/ | |
| 13:50:18 | mriedem | sahid: sorry no i'm not sure what you're referring to | |
| 13:50:34 | sahid | mriedem: no worries let me give you a pointer | |
| 13:50:48 | sahid | mriedem: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122591.html | |
| 13:50:56 | jianghuaw | bauzas, hmmm. yes, that means we can only expose one type per compute node. | |
| 13:51:13 | jianghuaw | I think that's the reason I used a pci-tied config option; | |
| 13:52:23 | jianghuaw | we can specify the vGPU enabled a a specific PGPU keyed by pci address. | |
| 13:52:32 | openstackgerrit | Merged openstack/nova master: Add some inline code docs tracing the cold migrate flow https://review.openstack.org/496861 | |
| 13:52:47 | mriedem | jaypipes: "The decision of whether to allow an approach that adds more to the existing /pci module is ultimately Matt's." | |
| 13:52:53 | mriedem | jaypipes: i didn't even see that bus coming | |
| 13:53:22 | jaypipes | mriedem: lol. well, sorry, but you are the PTL :) | |
| 13:54:02 | bauzas | jianghuaw: sec, reading your last comments then | |
| 13:56:51 | bauzas | jianghuaw: so, in case of libvirt, we have the capabilities XML that shows up different device IDs for two same cards (or types if you prefer) | |
| 13:56:52 | efried | jianghuaw Is it possible you have multiple identical GPUs (or mulitple pGPUs on the same card) where you would want to whitelist some but not others? | |
| 13:57:00 | efried | :) | |
| 13:57:02 | sahid | mriedem: jaypipes, my point is to continue make our users happy, have a deprecating phase to fix any issues for the work ongoing with RP and at some point, switch | |
| 13:57:26 | bauzas | jianghuaw: because each of them is corresponding to a mediated device | |
| 13:57:50 | jaypipes | sahid: well, we're never going to make users happy :) | |
| 13:57:56 | bauzas | jianghuaw: and that's the deployer's responsibility to configure libvirt so that the GPUs are shown as mediated devices | |
| 13:58:03 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Return Alternate Hosts https://review.openstack.org/504275 | |
| 13:58:09 | jaypipes | sahid: but anyway, like I said, I'm fine with you adding to the existing code. | |
| 13:58:21 | jaypipes | sahid: I just will be focusing on the n-r-p stuff, that's all | |
| 13:58:40 | bauzas | it's open-source development, anyone can just propose | |
| 13:58:58 | sahid | jaypipes: yes i understand that which it make sense and i will be happy to help and review some part if i can | |
| 13:59:03 | bauzas | I just felt we discussed on what could be achievable for Queens in parallel of the nested RPs implementation | |
| 14:00:02 | bauzas | the main focus for Queens was to provide the libvirt feature to returning RPs that were augmented by the VGPU resources | |
| 14:00:38 | bauzas | jianghuaw: can you clarify how Xen would see those GPU types ? | |
| 14:00:54 | edleafe | Scheduler subteam meeting running now in #openstack-meeting-alt | |
| 14:00:55 | sahid | bauzas: so you want to push on top of something under heavy developpement a feature which we are going to be used by large industries ? i think we should be reasonable | |
| 14:01:28 | bauzas | sahid: I'm just saying it's orthogonal :) | |
| 14:01:33 | sahid | RP is doind a lof of things, we need a deprecating phase, where some users can migrate and so we can fix the issues | |
| 14:01:35 | jianghuaw | bauzas, ok. XenServer will detect the pGPU types and make the same type of pGPUs into a single group. | |
| 14:01:46 | bauzas | vGPU tracking is done by providing a new set of resource classes | |
| 14:01:53 | bauzas | and traits | |
| 14:02:11 | bauzas | while nested resource providers is focusing on providing a tree of resource providers with dependencies | |
| 14:02:19 | bauzas | those don't overlap | |
| 14:02:24 | jianghuaw | And expose the vGPU types supported by the pGPU group. | |
| 14:02:47 | jaypipes | sahid: for the record, we *have* been doing that deprecation/migration period with resource providers. for example, we had a deprecation/migration period for tracking Ironic nodes as atomic resources. | |
| 14:02:49 | bauzas | snap, scheduler meeting | |
| 14:03:03 | jianghuaw | when requesting a vGPU, xenserver will schedule a PGPU and create a vGPU on it. | |
| 14:03:14 | jaypipes | sahid: so it's not true that we're just imposing resource providers modeling without providing a migration path for existing resource classes. | |
| 14:03:41 | jianghuaw | bauzas, so we can't define whitelist per pGPU by per pGPU group. | |
| 14:03:51 | bauzas | sahid: jaypipes: probably a good call for discussing that in the scheduler meeting | |
| 14:03:54 | sahid | jaypipes: oh yes right, but, what about NUMA, CPU Pinning, Huge Pages and all the virt specific features? | |
| 14:04:08 | sahid | don't we have to migr incrementally? | |
| 14:04:18 | jaypipes | sahid: yes, absolutely. | |
| 14:04:32 | jaypipes | sahid: none of those resources are planned to migrate in Queens, BTW. | |
| 14:04:41 | sahid | jaypipes: yes and what is the ETA for sometging production-ready? | |
| 14:04:48 | sahid | ok cool that is my point | |
| 14:04:50 | bauzas | well, with the fact that CPU pinning isn't targeted to be pushed to Placement, right? | |
| 14:04:50 | owalsh | bauzas: hey | |
| 14:04:53 | jianghuaw | bauzas, that's why we make the pGPU group as the vGPU's resource provider. | |
| 14:05:16 | bauzas | jianghuaw: so, say I have two same cards, I'd only see one pGPU group right? | |
| 14:05:27 | jianghuaw | true. | |
| 14:05:28 | sahid | jaypipes: it will take at least 2 or 3 releases and during that time we can't make any development? | |
| 14:05:39 | bauzas | jianghuaw: okay, then that's a bit different from libvirt | |
| 14:06:01 | sahid | i just suggest we work in parallel | |
| 14:06:29 | jianghuaw | bauzas, could you give an example for 'that shows up different device IDs for two same cards.' | |
| 14:06:32 | sahid | since the /pci is kind of production-ready and provide eveyrything we need | |
| 14:06:32 | jaypipes | sahid: depends on what you mean by "production-ready". (I personally don't think the existing pci manager is well-written or maintainable, but I guess there are different defninitions of "production-ready") | |
| 14:06:50 | jaypipes | sahid: again, I'm not disagreeing with you... | |
| 14:06:56 | sahid | jaypipes: yes yes :) | |
| 14:07:00 | jaypipes | not sure why you're acting like I am :) | |
| 14:07:26 | sahid | oops sorry really | |
| 14:07:45 | sahid | it's just that you are the only who are paying attention at me :) | |
| 14:08:10 | dansmith | jaypipes: we can provide pretty simple vgpu support via resource classes today right? without building more into the pci infrastructure we have | |
| 14:08:31 | jaypipes | sahid: we're paying attention but also in scheduler IRC meeting :) | |
| 14:08:42 | dansmith | jaypipes: if we just let virt drivers see that a vgpu was requested (i.e. see the flavor) and just configure the guest with an available one | |
| 14:08:43 | jianghuaw | bauzas, will the "GRID M60-0B" have different type id <type id='nvidia-11'> for two same pGPUs? | |
| 14:09:20 | bauzas | jianghuaw: I haven't tested yet, but I bet it | |
| 14:09:27 | jaypipes | dansmith: very simple resources, yes. we could add a (custom) resource class for the VGPU resources and have a flavor consume some amount of those | |
| 14:09:43 | dansmith | jaypipes: right, that'd be my preference for the first go-round | |
| 14:10:02 | jianghuaw | bauzas, I think if the type id is different, it will be a good solution to use 'type-id' for libvirt. | |
| 14:10:20 | bauzas | dansmith: jaypipes: I was only seeing the virt drivers adding the new VGPU resource class to the existing RPs as a first round, that's it :) | |
| 14:10:26 | dansmith | jaypipes: let people segregate different types of gpus into aggregates, and just assume equal portions of vgpu per instance without any smarts, which I think would be a perfectly reasonable first stab, which we can iterate on when we have traits and things | |
| 14:10:32 | dansmith | bauzas: ++ | |
| 14:11:06 | jaypipes | dansmith: agreed. | |
| 14:11:21 | bauzas | jianghuaw: yeah, so you agree with the fact that the listOpt can have different semantics based on the driver? | |
| 14:11:50 | jianghuaw | bauzas, yes. libvirt uses the type-id but XenServer continue to use the type name. As you suggested, virt driver should able to mapping them into real resources. | |
| 14:12:05 | bauzas | that would be strings for each of the drivers, but in the libvirt case, it would be device IDs while it would be pGPU resource type names for xen | |
| 14:12:08 | jianghuaw | but we need confirm if the type-id will be different. | |
| 14:12:19 | bauzas | jianghuaw: what I need is a testbed :p | |
| 14:13:02 | jianghuaw | bauzas, thanks. | |
| 14:13:28 | bauzas | jianghuaw: fancy updating the spec so I can do a pass once again ? | |
| 14:14:10 | jianghuaw | bauzas, sure. I will update it soon. | |
| 14:14:19 | bauzas | cool | |