Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-28
15:05:45 gibi mriedem: thanks, I
15:05:49 gibi mriedem: thanks, I'm +2 now
15:06:09 jianghuaw_ efried, it looks like you agree with me to make it as traits. right?
15:06:52 efried jianghuaw_ With the caveat that I don't fully understand the different ways you can request/configure these things, yes, I believe I agree.
15:07:26 mriedem sdague: could use some review on https://review.openstack.org/#/c/521391/ which is a fix for a regression introduced by a recent cve fix, so it's going to have to be backported through to newton - there is a regression test patch underneath it
15:07:36 openstackgerrit jiangpf proposed openstack/nova master: Encode libvirt domain XML in UTF-8 https://review.openstack.org/522161
15:07:45 efried jianghuaw_ A given VGPU type will only ever have *one* of the traits (e.g. SINGLE_DISPLAY_HEAD_CAPABLE, DUAL_DISPLAY_HEAD_CAPABLE, QUAD_DISPLAY_HEAD_CAPABLE) right?
15:08:37 mriedem sdague: actually that regression test might be busted now, checking
15:08:58 jianghuaw_ efried, at the moment I see the possible number is 1, 2, 4, 8
15:09:38 efried jianghuaw_ Okay; the question remains: a given VGPU type will *always* and *only* consume *one* of those numbers of display heads?
15:10:18 jianghuaw_ correct.
15:10:47 efried Then yes, I think it's appropriate *not* to inventory display heads separately, and I think it's okay to have those be traits.
15:11:18 dansmith mriedem: ack
15:11:19 efried Now if I want a VGPU with four display heads, I request resources=VGPU:1&required=QUAD_DISPLAY_HEAD_CAPABLE
15:11:57 jianghuaw_ efried, yeah. That's what I was thinking:-)
15:12:04 efried jaypipes Does this seem sane to you ^^
15:12:22 jaypipes gimme a minute, folks. trying to finish rebasing n-r-p
15:13:14 jaypipes ok, final tests running now... lemme read back, sorry
15:13:32 efried jaypipes Summary: A given VGPU type will *always* have a certain number of display heads. So display heads as traits (SINGLE_DISPLAY_HEAD_CAPABLE, DUAL_DISPLAY_HEAD_CAPABLE, etc.) and a RP of VGPUs (presumably the pGPU) would only ever have one of those traits.
15:13:55 jianghuaw_ efried, thanks for the summary.
15:14:34 jaypipes efried: if the display head is *consumed* by a request, it needs to be a resource class, not a trait.
15:15:17 efried jaypipes The issue is that the number of display heads are always correlated exactly with the VGPU type.
15:15:47 jianghuaw_ jaypipes, the display heads will always be consumed automatically when requesting VGPU.
15:15:58 jaypipes efried: I don't see any issue with that. (plus, I guarantee you nvidia and intel will end up changing that in the future. it's what they do. ALL THE TIME.
15:16:06 alex_xu mriedem: yea, you are right, without 'migrate' key, it won't route to that action
15:16:13 bauzas jaypipes: the problem here is that display heads are correlated by the number of VGPUs and their types
15:16:26 bauzas oh man
15:16:27 jaypipes bauzas: I don't see a problem?
15:16:47 jaypipes bauzas: set the amount of VGPU and display head resources in the flavor correctly, no?
15:16:59 bauzas jaypipes: once we have traits, it's doable
15:17:11 jaypipes bauzas: traits have nothing to do with this.
15:17:22 bauzas jaypipes: for the moment, the user has no clue about which type he will get
15:17:31 alex_xu efried: you mean you are working on the two testcases in this patch https://review.openstack.org/498737 ?
15:17:34 bauzas hence how many heads he could get
15:17:36 efried alex_xu Yes
15:17:41 jaypipes bauzas: it's not the user. it's the deployer/admin that needs to set shit up.
15:17:42 alex_xu efried: cool, thanks
15:17:51 efried alex_xu The first one is ported over, and passing. Working on the second one...
15:17:56 bauzas jaypipes: I don't disagree
15:18:15 bauzas jaypipes: the operator provides which supported types the compute should provide
15:18:26 bauzas so the VGPUs will depend on thaty
15:18:29 bauzas and the heads
15:18:54 alex_xu efried: I ported the first one on Monday, but interupt by other works... so nvm, thanks for porting those
15:19:01 bauzas but then, if the user wants a specific GRID-K5000, then the operator will set the flavor with a trait
15:19:41 jaypipes bauzas: correct. if the virt driver (Xen, libvirt, whatevs) wants to do some fancy-pants discovery on startup of nova-compute and ensure that inventory counts are proper for the types of GPUs found on the host, cool with me. But it's the deployer's responsibility to create flavors that make sense for these related resources.
15:19:44 bauzas anyway, I need to go
15:19:58 bauzas disappearing for 20 mins
15:20:01 bauzas -ish
15:20:25 jianghuaw_ jaypipes, as the display heads are optional (we have to allow that as the vGPU may not support display heads). If the flavor has resources=VGPU:1; then placement may return resource providers which have display heads. In that case, the display heads are consumed automatically.
15:20:33 jianghuaw_ That's the problem.
15:20:44 bauzas jaypipes: one last thing, libvirt doesn't support display heads yet
15:20:45 bauzas :p
15:20:49 efried jaypipes Here's the counterexample: One pGPU provides VGPU types that always allocate four display heads, so it exposes inventory of VGPU:2,DISPLAY_HEAD:8. Some other pGPU always allocates two display heads, so it exposes inventory of VGPU:2,DISPLAY_HEAD:4. In my flavor, I request VGPU:1,DISPLAY_HEAD:2. Scheduler happily schedules me to that first one, and allocates usage of 2 display heads. But we actually consume four.
15:20:53 bauzas (so I'm not fully on track with that)
15:20:55 jaypipes jianghuaw_: no, nothing should be "consumed automatically". that's the problem. :)
15:21:18 dansmith mriedem: are you not going to +2 this? https://review.openstack.org/#/c/521662/
15:21:36 jianghuaw_ jaypipes, agreed. but the problem is how to avoid that case.
15:21:55 jaypipes efried: that is what min_unit, max_unit, and step_size are for.
15:22:21 efried Hum, yeah, I guess that would work.
15:22:26 efried But
15:22:37 efried We still have the problem of when I request VGPU:1 and no display heads.
15:22:42 jianghuaw_ jaypipes, but how about if flavor only has resources=VGPU:1?
15:22:49 efried jinx
15:22:52 jianghuaw_ it doesn't request display heads.
15:23:21 efried Scheduler will happily schedule me to either one of those guys above, and not create a usage for display heads, even though they got consumed.
15:23:27 jianghuaw_ so it consumes a vGPU. But we know any consuming vGPU result also result the available display heads be reduced also.
15:23:46 efried (Because there's another RP somewhere whose vGPUs don't have display heads, and that's where I meant to go.)
15:24:08 jianghuaw_ efried, ack
15:24:24 efried (Or actually I didn't care whether I went there or not; but if I got onto the display-heads-havin' RPs, I have to consume its display heads)
15:25:16 jaypipes jianghuaw_, efried: this is the same problem that stephenfin and snikitin's "PCI affinity scheduling policies" thing is about...
15:26:10 jianghuaw_ jaypipes, is there solution for that problem already?
15:26:16 stephenfin pretty much, yeah
15:26:19 jaypipes jianghuaw_, efried: I think in that scenario, I would just say the flavor must always request a display head, regardless of if the user ends up using one.
15:26:30 openstackgerrit Merged openstack/nova master: Update document related to host aggregate https://review.openstack.org/514499
15:26:59 efried jaypipes That would make it impossible to use vGPUs that don't have display heads.
15:27:00 stephenfin jianghuaw_: https://review.openstack.org/#/c/390520/
15:27:08 jaypipes jianghuaw_: the "solution" is very specific to PCI, involves (yet another) configuration option/metadata key/value pair, and generally is going to make long-term porting of the PCIPassthroughFilter impossible. but yeah...
15:27:15 jianghuaw_ jaypipes, so making display heads as mandatory as long as it request vGPU?
15:27:51 jianghuaw_ stephenfin, jaypipes thanks. looking at the patch.
15:27:51 jaypipes jianghuaw_: IFF the compute hosts' pGPUs do not have the ability to only serve up a vGPU without a display head.
15:28:23 artom mriedem, oh, you've proposed the upstream rebuild CVE backports, thanks :)
15:29:12 jianghuaw_ jaypipes, but we do have cases where vGPUs don't support a display head. which means no inventory for display heads.
15:29:18 jaypipes jianghuaw_: bottom line for me on this is that I do not think this use case is particularly high priority, and given that there is a solution of "just have the flavor always request a display head (or >1 display head) if the underlying hardware always requires a display head, I don't want to add in any sort of trait hack here.
15:32:15 openstackgerrit Merged openstack/nova master: Add instance action record for attach/detach/swap volumes https://review.openstack.org/517205
15:32:16 jaypipes jianghuaw_: in that scenario, what about just attaching a custom trait CUSTOM_NO_DISPLAY_HEAD to the flavor that doesn't want a display head and attach the same trait to the resource providers representing those pGPUs?
15:32:49 jaypipes jianghuaw_: that way, we're not using any standard traits for things that are quantitative and you can still have the scheduler and placement service do what they're good at without any additional hacks.
15:34:04 jaypipes jianghuaw_: in that way, all the requests for vGPU with no need for display head support will go to those hosts and all other requests for vGPUs *with* display head support will go to the other hosts that have inventory of both VGPU and VGPU_DISPLAY_HEAD
15:34:08 jianghuaw_ jaypipes, hmmm. It should work.
15:34:58 jaypipes jianghuaw_: I'm fine using custom traits for something like that. it's just that I am strongly against using standard traits for things that are, in essence, quantities of some class of resource (like the proposed SINGLE_DISPLAY_HEAD_CAPABLE thing above)
15:35:04 jianghuaw_ jaypipes, another option is to making display head default as 1. If there is no display heads return from hypervisor.
15:35:56 jaypipes jianghuaw_: you mean "fake out" the inventory counts for VGPU_DISPLAY_HEADS for these hosts?
15:36:13 jianghuaw_ jaypipes, yes.
15:36:35 jaypipes jianghuaw_: nah, I'd prefer not to do that. otherwise, we'll go down the route of cdent and his "infinite inventories" :P
15:37:06 jianghuaw_ jaypipes, ok. Let's forget it:-)
15:37:35 cdent all my best ideas, like sands in the hourglass
15:38:22 jaypipes hehe
15:38:46 jaypipes jianghuaw_: no, in all seriousness, I think the custom trait CUSTOM_NO_DISPLAY_HEAD is the right approach to solve that remaining use case.
15:40:57 jianghuaw_ jaypipes, I think the above approaching by using custom traits actually strongly depends on the administrators to do right thing to set the flavors. I means it make break thing if someone wrongly created a flavor only request VGPU and don't have CUSTOM_NO_DISPLAY_HEAD.
15:41:56 jaypipes jianghuaw_: that's a tradeoff I'm willing to make.
15:42:47 jianghuaw_ jaypipes, got it. So we should record that in the document.

Earlier   Later