| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-28 | |||
| 14:28:10 | bauzas | right | |
| 14:28:27 | mriedem | efried: that's why it was marked 'unknown' | |
| 14:29:00 | mriedem | efried: i don't think libvirt ensures devices are re-attached in the same order on rebuild either | |
| 14:29:21 | mriedem | that's why mdbooth had the bdm metadata stuff for awhile | |
| 14:29:25 | mriedem | *i think* | |
| 14:30:08 | mdbooth | efried: They are definitely *not* reattached in the same order in all circumstances | |
| 14:30:24 | jianghuaw_ | bauzas, It's possible to allocate vGPUs which actually supports display heads for those flavors not requesting display heads. | |
| 14:30:27 | efried | mdbooth Yeah, so in PowerVM, we guarantee that. | |
| 14:30:55 | jianghuaw_ | as the resource provider have inventory for VGPU which can meet the request. | |
| 14:31:07 | mdbooth | efried: It would be nice. | |
| 14:31:37 | mdbooth | efried: Is your scheme reproducible? | |
| 14:31:39 | efried | mriedem Yeah, I confirmed by looking at our OOT driver - there's a bunch of special stuff we do in spawn for the rebuild case that we haven't ported in yet. So we want to declare that we *don't* support rebuild in the in-tree driver yet. | |
| 14:31:59 | bauzas | jianghuaw_: the problem I see is that if you provide a flavor asking for both VGPUs *and* heads, then the scheduler will ask placement for *both* | |
| 14:32:03 | mdbooth | efried: i.e. Could the libvirt driver use it? | |
| 14:32:26 | efried | mdbooth Bunch of caveats, but I think the concept might be portable. | |
| 14:32:42 | bauzas | jianghuaw_: AFAIR, we had a spec for modifying the placement call to ask for some resource classes not mandatory | |
| 14:32:51 | bauzas | efried: jaypipes: is it, right? | |
| 14:33:03 | efried | bauzas Not for Q | |
| 14:33:04 | mdbooth | The issue the libvirt driver has is the interaction between dynamic and 'static' disks | |
| 14:33:12 | bauzas | efried: yup, I know | |
| 14:33:25 | bauzas | efried: I was more commenting about the fact we discussed that | |
| 14:33:27 | efried | bauzas The one I remember was for "preferred traits", not for optional resource classes. | |
| 14:33:33 | bauzas | yeah that | |
| 14:33:37 | mdbooth | So if you have an instance with a root disk and a config disk, then you attach a volume, you'll have root disk, config disk, volume | |
| 14:33:49 | mdbooth | But in rebuild we always put the config disk last | |
| 14:33:58 | mdbooth | So on rebuild you'll have root disk, volume, config disk | |
| 14:33:59 | jianghuaw_ | bauzas, I think this case makes sense. What I'm think is that a flavor is asking only VGPU, but it placement may return RPs which contains display heads also. | |
| 14:34:10 | mdbooth | And in fact we have no way of knowing what the previous order was | |
| 14:34:13 | jianghuaw_ | it will result into the display heads leaking. | |
| 14:34:16 | bauzas | jianghuaw_: right | |
| 14:34:37 | bauzas | what do you mean by leaking ? | |
| 14:34:43 | efried | mdbooth For us, the dev order is determined by which virtual "slot" the disk is attached via. So we maintain a cache of which devices are associated with which slots, and then we make sure to restore that on rebuild. | |
| 14:34:50 | bauzas | jianghuaw_: that's depending on the virt driver, right ? | |
| 14:35:00 | mdbooth | efried: Right. Where do you store the cache, though? | |
| 14:35:15 | efried | mdbooth In our case, in order to support that, the user has to set up swift :$ | |
| 14:35:27 | mdbooth | Ah... | |
| 14:35:31 | efried | indeeed. | |
| 14:35:35 | jianghuaw_ | bauzas, I mean it automatically consumed display heads. but placement don't it. | |
| 14:35:51 | mdbooth | See, I actually think there's a lot of merit in having persistent driver state | |
| 14:36:14 | mdbooth | I think it should be local to the hypervisor, though | |
| 14:36:34 | efried | mdbooth We don't go crazy with persistent state, for sure. We save these slot mappings, and we save the partition's NVRAM. | |
| 14:36:55 | jianghuaw_ | bauzas, don't it => doesn't know of it. | |
| 14:37:16 | mdbooth | efried: Definitely in favour of the concept. | |
| 14:37:48 | efried | mdbooth I have to say, our implementation ain't pretty. | |
| 14:38:10 | jianghuaw_ | bauzas, virt driver can only stop a booting by checking the resoureces in the allocation. | |
| 14:38:21 | efried | mdbooth It was one of these rush jobs where we realized the problem (devices attached out of order) late in a cycle, and threw together a solution at the last minute. | |
| 14:38:55 | mdbooth | efried: Out of curiosity, why did you consider it a sufficiently big problem for a big rush job? | |
| 14:39:01 | jianghuaw_ | bauzas, or is there a way to correct the allocation if the allocated vGPU also supports display heads? | |
| 14:39:21 | mdbooth | Device ordering is also inconsistent in a Linux guest OS | |
| 14:39:29 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] re-use existing conf with auth token middleware https://review.openstack.org/523403 | |
| 14:39:38 | efried | mdbooth I don't remember all the reasons, but e.g. maybe you don't boot. | |
| 14:39:47 | efried | mdbooth Or possibly worse, boot to the wrong disk. | |
| 14:41:20 | efried | mdbooth Also could have something to do with applications in guests referring to devices by name - and if that name changes, you're kaput. | |
| 14:41:43 | mdbooth | efried: That last point isn't fixed by consistent device ordering | |
| 14:41:47 | efried | mdbooth Keep in mind we're running not just Linux, but also AIX and IBMi. | |
| 14:41:56 | mdbooth | Because as I say, Linux itself is also non-deterministic | |
| 14:42:31 | mdbooth | The first should be handled by the hypervisor specifying a boot order for its BIOS/UEFI | |
| 14:43:27 | mdbooth | Or whatever hopefully much better boot solution is in Power :) | |
| 14:43:43 | efried | mdbooth That's exactly the point: The boot list is specified based on hardware addresses, which are discovered based on slots. | |
| 14:45:04 | efried | mdbooth The big conceptual divide here is that PowerVM has a real hypervisor and the guests are real partitions. | |
| 14:46:10 | jianghuaw_ | bauzas, I think efried's opinion is that we shouldn't make inventory for VGPU_DISPLAY_HEAD as that's possible to be consumed automatically by requesting VGPU. | |
| 14:46:54 | efried | jianghuaw_ Only if display heads are *only* consumed automatically by requesting VGPU | |
| 14:47:29 | edleafe | jianghuaw_: if it can't be consumed independently, and is always consumed automatically, then it shouldn't be tracked as inventory. | |
| 14:47:45 | efried | jianghuaw_ If it's possible to request them separately, then you need to inventory them separately; and then have extra logic in scenarios where it would happen automatically such that, if the flavor doesn't request the appropriate number of each, you throw an error. | |
| 14:47:46 | bauzas | jianghuaw_: I agree with him | |
| 14:48:18 | bauzas | jianghuaw_: until we have a way to track optional resources classes, we shouldn't be providing inventories for display heads IMHO | |
| 14:48:42 | bauzas | and just do the display head consumption solely in the virt driver, if you can | |
| 14:48:59 | bauzas | tbh, libvirt doesn't support that yet, so I don't give a bit of that :p | |
| 14:49:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Refined fix for validating image on rebuild https://review.openstack.org/523427 | |
| 14:50:14 | jianghuaw_ | bauzas, efried Ok. Agreed. I will drop that patch for inventoring display heads. | |
| 14:51:05 | jianghuaw_ | Per my understand, the display heads will always be consumed automatically as long as the VGPUs supporting display heads. | |
| 14:53:11 | jianghuaw_ | Actually different vGPU types may support different display heads. And users may need some vGPUs which support specified amount of display heads. That's why I said it seems somehow traits for the vGPUs. | |
| 14:54:00 | efried | jianghuaw_ Yes, that sounds like it might be suitable for traits. | |
| 14:55:00 | jianghuaw_ | efried, but jaypipes seems don't agree ^ :-) | |
| 14:55:24 | bauzas | jianghuaw_: well, requesting a display head is not quantitative | |
| 14:55:35 | efried | jianghuaw_ It depends whether it can be consumed independently, or is always tied to the VGPU. | |
| 14:55:43 | efried | Which I'm still not clear on. | |
| 14:56:24 | jianghuaw_ | efried, yes, it's always tied to VGPU. | |
| 14:56:35 | efried | Okay. If it is *always* the case that requesting a specific VGPU results in a specific number (possibly zero) of display heads being consumed, then there's no reason to inventory separately. | |
| 14:57:55 | efried | jianghuaw_ And how does the user specify which type of VGPU they're getting? | |
| 14:58:02 | bauzas | by the trait | |
| 14:58:15 | jianghuaw_ | yes, by traits. | |
| 14:58:18 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Nits from Ic3ab7d60e4ac12b767fe70bef97b327545a86e74 https://review.openstack.org/523432 | |
| 14:58:30 | bauzas | given we don't support traits yet, we only ask for the operator to provide *one* type | |
| 14:58:47 | efried | So a given trait is in fact overloading more than one aspect of the VGPU's characteristics. | |
| 14:59:15 | bauzas | well | |
| 14:59:22 | efried | In other words, by saying I need a VGPU of type X, I'm also implicitly stating that I'm going to get two display heads (or whatever) | |
| 14:59:33 | bauzas | that is a correct assumption to me | |
| 14:59:59 | efried | Okay. Then if that correlation always holds true | |
| 15:00:12 | efried | that is, a given VGPU type always maps to a given number of display heads | |
| 15:00:21 | jianghuaw_ | the plan is to associate traits like display_solutions; supported features after we support traits. | |
| 15:00:26 | bauzas | efried: see the spec | |
| 15:00:33 | bauzas | there are some examples | |
| 15:00:40 | efried | then you shouldn't have inventory for the display heads. | |
| 15:00:45 | bauzas | I agree | |
| 15:00:52 | bauzas | if that's correlated | |
| 15:01:10 | efried | And there's even no need for extra traits - though those could be included and it wouldn't hurt anything. | |
| 15:01:12 | bauzas | tbh, I haven't thought that much on that problem, since libvirt doesn't return the heads :p | |
| 15:01:20 | bauzas | (yet) | |
| 15:02:00 | jianghuaw_ | efried, the display heads is one factor can be used to determine a vGPU type. | |