Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-28
15:15:47 jianghuaw_ jaypipes, the display heads will always be consumed automatically when requesting VGPU.
15:15:58 jaypipes efried: I don't see any issue with that. (plus, I guarantee you nvidia and intel will end up changing that in the future. it's what they do. ALL THE TIME.
15:16:06 alex_xu mriedem: yea, you are right, without 'migrate' key, it won't route to that action
15:16:13 bauzas jaypipes: the problem here is that display heads are correlated by the number of VGPUs and their types
15:16:26 bauzas oh man
15:16:27 jaypipes bauzas: I don't see a problem?
15:16:47 jaypipes bauzas: set the amount of VGPU and display head resources in the flavor correctly, no?
15:16:59 bauzas jaypipes: once we have traits, it's doable
15:17:11 jaypipes bauzas: traits have nothing to do with this.
15:17:22 bauzas jaypipes: for the moment, the user has no clue about which type he will get
15:17:31 alex_xu efried: you mean you are working on the two testcases in this patch https://review.openstack.org/498737 ?
15:17:34 bauzas hence how many heads he could get
15:17:36 efried alex_xu Yes
15:17:41 jaypipes bauzas: it's not the user. it's the deployer/admin that needs to set shit up.
15:17:42 alex_xu efried: cool, thanks
15:17:51 efried alex_xu The first one is ported over, and passing. Working on the second one...
15:17:56 bauzas jaypipes: I don't disagree
15:18:15 bauzas jaypipes: the operator provides which supported types the compute should provide
15:18:26 bauzas so the VGPUs will depend on thaty
15:18:29 bauzas and the heads
15:18:54 alex_xu efried: I ported the first one on Monday, but interupt by other works... so nvm, thanks for porting those
15:19:01 bauzas but then, if the user wants a specific GRID-K5000, then the operator will set the flavor with a trait
15:19:41 jaypipes bauzas: correct. if the virt driver (Xen, libvirt, whatevs) wants to do some fancy-pants discovery on startup of nova-compute and ensure that inventory counts are proper for the types of GPUs found on the host, cool with me. But it's the deployer's responsibility to create flavors that make sense for these related resources.
15:19:44 bauzas anyway, I need to go
15:19:58 bauzas disappearing for 20 mins
15:20:01 bauzas -ish
15:20:25 jianghuaw_ jaypipes, as the display heads are optional (we have to allow that as the vGPU may not support display heads). If the flavor has resources=VGPU:1; then placement may return resource providers which have display heads. In that case, the display heads are consumed automatically.
15:20:33 jianghuaw_ That's the problem.
15:20:44 bauzas jaypipes: one last thing, libvirt doesn't support display heads yet
15:20:45 bauzas :p
15:20:49 efried jaypipes Here's the counterexample: One pGPU provides VGPU types that always allocate four display heads, so it exposes inventory of VGPU:2,DISPLAY_HEAD:8. Some other pGPU always allocates two display heads, so it exposes inventory of VGPU:2,DISPLAY_HEAD:4. In my flavor, I request VGPU:1,DISPLAY_HEAD:2. Scheduler happily schedules me to that first one, and allocates usage of 2 display heads. But we actually consume four.
15:20:53 bauzas (so I'm not fully on track with that)
15:20:55 jaypipes jianghuaw_: no, nothing should be "consumed automatically". that's the problem. :)
15:21:18 dansmith mriedem: are you not going to +2 this? https://review.openstack.org/#/c/521662/
15:21:36 jianghuaw_ jaypipes, agreed. but the problem is how to avoid that case.
15:21:55 jaypipes efried: that is what min_unit, max_unit, and step_size are for.
15:22:21 efried Hum, yeah, I guess that would work.
15:22:26 efried But
15:22:37 efried We still have the problem of when I request VGPU:1 and no display heads.
15:22:42 jianghuaw_ jaypipes, but how about if flavor only has resources=VGPU:1?
15:22:49 efried jinx
15:22:52 jianghuaw_ it doesn't request display heads.
15:23:21 efried Scheduler will happily schedule me to either one of those guys above, and not create a usage for display heads, even though they got consumed.
15:23:27 jianghuaw_ so it consumes a vGPU. But we know any consuming vGPU result also result the available display heads be reduced also.
15:23:46 efried (Because there's another RP somewhere whose vGPUs don't have display heads, and that's where I meant to go.)
15:24:08 jianghuaw_ efried, ack
15:24:24 efried (Or actually I didn't care whether I went there or not; but if I got onto the display-heads-havin' RPs, I have to consume its display heads)
15:25:16 jaypipes jianghuaw_, efried: this is the same problem that stephenfin and snikitin's "PCI affinity scheduling policies" thing is about...
15:26:10 jianghuaw_ jaypipes, is there solution for that problem already?
15:26:16 stephenfin pretty much, yeah
15:26:19 jaypipes jianghuaw_, efried: I think in that scenario, I would just say the flavor must always request a display head, regardless of if the user ends up using one.
15:26:30 openstackgerrit Merged openstack/nova master: Update document related to host aggregate https://review.openstack.org/514499
15:26:59 efried jaypipes That would make it impossible to use vGPUs that don't have display heads.
15:27:00 stephenfin jianghuaw_: https://review.openstack.org/#/c/390520/
15:27:08 jaypipes jianghuaw_: the "solution" is very specific to PCI, involves (yet another) configuration option/metadata key/value pair, and generally is going to make long-term porting of the PCIPassthroughFilter impossible. but yeah...
15:27:15 jianghuaw_ jaypipes, so making display heads as mandatory as long as it request vGPU?
15:27:51 jianghuaw_ stephenfin, jaypipes thanks. looking at the patch.
15:27:51 jaypipes jianghuaw_: IFF the compute hosts' pGPUs do not have the ability to only serve up a vGPU without a display head.
15:28:23 artom mriedem, oh, you've proposed the upstream rebuild CVE backports, thanks :)
15:29:12 jianghuaw_ jaypipes, but we do have cases where vGPUs don't support a display head. which means no inventory for display heads.
15:29:18 jaypipes jianghuaw_: bottom line for me on this is that I do not think this use case is particularly high priority, and given that there is a solution of "just have the flavor always request a display head (or >1 display head) if the underlying hardware always requires a display head, I don't want to add in any sort of trait hack here.
15:32:15 openstackgerrit Merged openstack/nova master: Add instance action record for attach/detach/swap volumes https://review.openstack.org/517205
15:32:16 jaypipes jianghuaw_: in that scenario, what about just attaching a custom trait CUSTOM_NO_DISPLAY_HEAD to the flavor that doesn't want a display head and attach the same trait to the resource providers representing those pGPUs?
15:32:49 jaypipes jianghuaw_: that way, we're not using any standard traits for things that are quantitative and you can still have the scheduler and placement service do what they're good at without any additional hacks.
15:34:04 jaypipes jianghuaw_: in that way, all the requests for vGPU with no need for display head support will go to those hosts and all other requests for vGPUs *with* display head support will go to the other hosts that have inventory of both VGPU and VGPU_DISPLAY_HEAD
15:34:08 jianghuaw_ jaypipes, hmmm. It should work.
15:34:58 jaypipes jianghuaw_: I'm fine using custom traits for something like that. it's just that I am strongly against using standard traits for things that are, in essence, quantities of some class of resource (like the proposed SINGLE_DISPLAY_HEAD_CAPABLE thing above)
15:35:04 jianghuaw_ jaypipes, another option is to making display head default as 1. If there is no display heads return from hypervisor.
15:35:56 jaypipes jianghuaw_: you mean "fake out" the inventory counts for VGPU_DISPLAY_HEADS for these hosts?
15:36:13 jianghuaw_ jaypipes, yes.
15:36:35 jaypipes jianghuaw_: nah, I'd prefer not to do that. otherwise, we'll go down the route of cdent and his "infinite inventories" :P
15:37:06 jianghuaw_ jaypipes, ok. Let's forget it:-)
15:37:35 cdent all my best ideas, like sands in the hourglass
15:38:22 jaypipes hehe
15:38:46 jaypipes jianghuaw_: no, in all seriousness, I think the custom trait CUSTOM_NO_DISPLAY_HEAD is the right approach to solve that remaining use case.
15:40:57 jianghuaw_ jaypipes, I think the above approaching by using custom traits actually strongly depends on the administrators to do right thing to set the flavors. I means it make break thing if someone wrongly created a flavor only request VGPU and don't have CUSTOM_NO_DISPLAY_HEAD.
15:41:56 jaypipes jianghuaw_: that's a tradeoff I'm willing to make.
15:42:47 jianghuaw_ jaypipes, got it. So we should record that in the document.
15:43:13 jianghuaw_ let's go with that approach.
15:43:16 jaypipes jianghuaw_: yeah. I think a "how to use vGPUs with OpenStack" article/tutorial/reference would be extremely useful.
15:43:51 jianghuaw_ jaypipes, thanks very much.j
15:43:57 jaypipes jianghuaw_: any time :)
15:44:53 jianghuaw_ efried, bauzas: thank you also for the discussion and advices.
15:45:13 efried jianghuaw_ Cool, good luck.
15:45:51 openstackgerrit Merged openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/519066
15:46:03 jianghuaw_ efried, :-)
15:46:18 efried alex_xu FYI, both of those tests pass on top of the stack; and I don't think we actually have other tests that match them exactly; and even if we did, I'm not opposed to having some duplication there.
15:46:41 efried alex_xu Not sure exactly how/where to propose the code, though.
15:48:34 efried alex_xu I guess we should propose the code under https://review.openstack.org/#/c/498737/ and abandon https://review.openstack.org/#/c/480379/
15:49:00 efried alex_xu But I'm not sure which patch to declare as having fixed bug #1731072
15:49:01 openstack bug 1731072 in OpenStack Compute (nova) "AllocationCandidates.get_by_filters returns garbage with multiple aggregates" [Medium,Confirmed] https://launchpad.net/bugs/1731072
15:49:41 efried oh, I guess that bug isn't fully fixed anyway, so we can just leave that alone.
15:49:48 efried Okay, I've answered all my questions :)
15:49:58 efried jaypipes mriedem FYI ^^
15:50:52 efried Dangit, wrong bug. Should have been bug #1702420 -- which is indeed fixed now.
15:50:53 openstack bug 1702420 in OpenStack Compute (nova) "The AllocationCandidates.get_by_filters returned wrong combination of AllocationRequests" [High,In progress] https://launchpad.net/bugs/1702420 - Assigned to Alex Xu (xuhj)
15:51:31 jaypipes efried: now as in now or now as in "will be shortly once some patch is merged"?
15:51:35 jianghuaw_ bauzas, Please check the above discussion after you come back. Hope you also agree with the approach: 1. always requesting display heads for VGPUs which have display heads; 2. use custom traits e.g. CUSTOM_NO_DISPLAY_HEAD for those cases where display head is not supported.
15:52:21 efried jaypipes The latter. I'm rebasing https://review.openstack.org/#/c/498737/ on top of the series formerly known as The Big Refactor.
15:52:37 jianghuaw_ bauzas, and don't forget to review my vGPU patches. Thanks:-)

Earlier   Later