Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-19
16:12:36 cdent you're asserting that targeting a cluster is wrong, and what I'm trying to understand why that's wrong
16:12:40 dansmith it is a hack, because if you have memory available on one and cpu available on another, you can't represent that
16:12:48 dansmith no, that's not what I'm saying
16:13:04 dansmith I'm saying nova thinks the cluster is a single bag of resources that can be combined and that's wrong
16:13:29 dansmith and I'm saying that I think the spec we're discussing is trying to rectify that by using multiple providers with the proper amounts of inventory each
16:14:02 mriedem yeah it's trying to break the single cluster into multiple pools
16:14:03 cdent the spec is trying to allow resource pools (a vmware thing that enables tenanted clusters) to work
16:14:13 mriedem but it's also tying those pools to specific tenants
16:14:24 mriedem like the aggregate multitenenancy isolation filter
16:14:25 mriedem i think
16:14:49 dansmith if it's purely for the tenantization, then that's not what I think it's trying to do, and isn't really a thing the virt driver should be doing
16:14:55 cdent there will likely be other specs that try to turn clusters into hierarchies of nested providers so that an esxi host works much the same as targeting a particular numa setup
16:15:13 dansmith ^that is what I thought the actual goal was under the covers
16:15:15 mriedem so if the admin creates a resource pool A_foo in cluster 1, i think it means that nova is supposed to restrict all requests from tenant foo to resource pool A_foo in cluster 1
16:15:18 dansmith but if not, that's disappointing
16:15:25 cdent okay, stepping back a bit: why do we care what a virt driver does if it follows the interface and allows things to land
16:15:54 cdent I'm asking that as a real question, not as a rhetorical device
16:16:30 dansmith because nova is an abstraction?
16:16:45 cdent right, if the abstraction works and continues to work, what does it matter?
16:16:53 dansmith the virt driver has to do more than mechanically implement the python interface to be something that a user can depend on
16:17:16 dansmith as an example, it doesn't work properly right now but it does implement the interface
16:17:36 cdent and it is working to get better
16:17:41 cdent using the tools that the interface provides
16:17:48 dansmith this tenant grouping doesn't make it better
16:17:57 dansmith it makes it more different in behavior than the others
16:18:01 mriedem maybe the tenant isolation part of this should be split out
16:18:28 mriedem so it's just a generic way to model nested resource pools in a single cluster (which is represented by the compute node provider root in nova)
16:18:53 mriedem and then the compute node RP for vmware doesn't have cpu/ram/disk, the nested resource pool providers do
16:19:09 mriedem and then later bake tenant isolation logic into it
16:19:19 mriedem sorry for the bad timing but i've got a thing i've got to run to, back in awhile
16:19:40 dansmith at that point, the tenant isolation becomes much the same problem as efried pointed out with ensuring that you get CPU and memory from the same numa node on a regular box,
16:19:56 dansmith which probably helps, if we decide that the virt drivers should draw aggregates around those groupings or whatever,
16:20:05 dansmith or something
16:20:10 cdent I suspect the tenant isolation is on the spec at all so that the resource pool stuff can be justified: it's a use case for wanting to represent resource pools
16:20:23 cdent if that's needed and reasource pools are groovy, that's great
16:20:35 efried As I commented on https://review.openstack.org/#/c/534339/ the 1GB-from-each-of-two-children-to-make-one-2GB-instance thing shouldn't be a problem.
16:20:48 efried BUT the "get my CPU from one RP and my disk from another" would be.
16:21:14 dansmith efried: currently the former is a problem with the vmware driver AFAIK
16:22:03 dansmith cdent: implementing the structurally separated resources in a cluster as nested providers of the root would make progress towards fixing that ^ and then makes the tenant isolation thing a smaller delta we can probably have a more reasonable discussion over
16:22:11 efried To mitigate in UPT-land, you would have to do subtree lassoing like we talked about needing for NUMA; or you would have to model the cluster members as NOT being in the same tree.
16:22:17 cdent dansmith: that's not what the spec does
16:22:24 cdent resource pools are not physical
16:22:39 dansmith efried: that's what I'm saying would be an improvement over what it does today
16:22:45 efried which one dansmith?
16:22:57 cdent if you have a problem with the cluster presented agglomerated resources as a single thing, then you'd have a problem with resource pools too
16:23:01 dansmith efried: representing cluster members as children under the root
16:23:12 cdent max_unit, on either the cluster or the resource pool "fixes" that
16:23:30 efried dansmith: Only if we have the subtree-lassoing technology. I haven't caught up on my specs yet - did someone propose that yet?
16:23:33 dansmith cdent: max_unit does not prevent nova from thinking it can get cpu and memory from two different cluster members
16:23:39 efried Agree ^^
16:24:02 cdent dansmith: yes, and?
16:24:11 dansmith efried: right, I'm assuming the lassoing thing as well as changing this vmware RP exposure.. the two together would be required
16:24:55 dansmith cdent: so max_unit has not "fixed" the vmware driver reporting a whole cluster as a single RP/node
16:24:56 efried In that case, yes, I agree we can do this with nested.
16:25:02 cdent the only way we get what you seem to want is for every esxi host to represent all its resources, which breaks the DRS, unless the virt driver can write allocations
16:25:26 dansmith cdent: yeah, I think we've asserted that DRS under nova is broken, for that reason exactly
16:25:34 dansmith broken fundamentally I mean
16:25:59 efried cdent: I thought we talked about the fact that you want to hide the whole cluster-ness anyway?
16:26:20 efried Represent the whole thing as a single RP, and then vmware virt would do the individual node business under the covers.
16:26:32 cdent efried: yes, dan's saying that's broken
16:26:34 dansmith efried: that's what it does today, and it's breakable
16:26:58 dansmith efried: because nova will not know that one node is out of CPU but has some memory available, where another node has the opposite.. nova will think it can schedule to that, but it can't
16:27:07 gjayavelu @cdent @dansmith @efried I'm catching up on the discussions about the resource pools spec
16:27:45 efried dansmith: Okay, I agree with that. But I think the virt can probably mitigate
16:27:53 dansmith efried: it can't
16:27:55 efried ...by cleverly spoofing the inventories.
16:28:05 dansmith oh, it could work around it that way sure
16:28:13 dansmith that's fairly wasteful though :)
16:28:17 efried Basically by representing the least common denominator of the pool, as the inventory.
16:28:24 efried Is it?
16:28:30 dansmith sure
16:28:33 efried Because you wouldn't be able to schedule more than that anyway, wouldja?
16:28:59 dansmith well, maybe you're right
16:29:06 cdent yeah, you either fail to get allocation candidates, or you fail later
16:29:07 dansmith well, no,
16:29:13 cdent better to fail to get allocation candidates
16:29:43 dansmith you could have a node with small memory, small cpu, lots of disk, and another node with zero disk but lots of other resources.. you have to report yourself as fully committed at that point right?
16:29:47 efried Okay, I see, if you had one node with 1VCPU/1024MEM and one with 2VCPU/512MEM, you would wind up representing 1/512 even though you could technically schedule 1/1024 or 2/512.
16:29:49 dansmith cdent: also that
16:30:00 efried dansmith: Yeah, I think we're saying the same thing.
16:30:04 dansmith yar
16:30:23 cdent at the moment that is an accepted shortcoming
16:30:35 cdent mostly because the common case is for everything in the cluster to be the same
16:30:58 efried Yeah, was gonna say, homogeneity would make most of the issue moot.
16:31:02 dansmith right but your flavors have to basically fit together perfectly like puzzle pieces to avoid too much fragmentation
16:31:15 cdent ENOPARSE
16:31:18 openstackgerrit sahid proposed openstack/nova master: libvirt: slow live-migration to ensure network is ready https://review.openstack.org/497457
16:31:35 efried I think it's a pretty solid 80/20 that's worth going forward with.
16:32:32 efried Hypothetically the DRS will also reshuffle things to maintain/restore homogeneity, nah? That's kind of its job?
16:32:54 cdent yes
16:32:58 cdent well
16:33:09 cdent it will shuffle things. but it can't move hardware
16:33:21 dansmith that doesn't fix anything if your flavors are different sizes
16:33:21 efried understood.
16:33:27 efried It can
16:33:46 efried because it can consolidate VMs in ways that maximize utilization of a node.
16:33:50 efried If it's clever enough.
16:34:25 dansmith it may be able to if things fit right, yes
16:34:26 cdent even if its not clever enough, if vmware is being wasteful, that's vmware's problem, right?
16:34:27 efried But anyway, IMO it's acceptable to state that we know this solution is not 100% perfect, and move forward with it, because it will work *mostly* well, *most* of the time.

Earlier   Later