Posted	Nick	Remark
#openstack-nova - 2018-03-19
16:06:16	mriedem	compute node RP (created by nova) -> nested tenant-specific resource pool providers, created by the virt driver, proxied from vcenter
16:06:30	mriedem	the 1:M compute node thing isn't in this
16:06:30	dansmith	yep
16:06:34	dansmith	right
16:06:39	cfriesen	mriedem: is there anything in particular blocking https://review.openstack.org/#/c/465787/ ? I notice it hasn't really gotten any reviews other than me.
16:06:42	dansmith	I think jaypipes was asserting that it was
16:07:23	jaypipes	dansmith: sorry, on a video call for 45 mintues
16:07:38	mriedem	i guess since we don't model numa as a nested RP for the libvirt driver yet, i'm not sure how much proxy we'll be doing to compare to the vcenter thing
16:07:50	dansmith	the difference here I think,
16:08:02	dansmith	is that libvirt will report the static nature of the system as nested NUMA RPs,
16:08:11	openstackgerrit	Lee Yarwood proposed openstack/nova master: libvirt: Block swapping to an encrypted volume when using QEMU to decrypt https://review.openstack.org/544238
16:08:12	dansmith	where vmware would be representing cluster config, which is mutable
16:08:33	dansmith	and I agree that sucks, and I don't know what would happen if vmware swizzled things after instances are booted
16:08:35	dansmith	but..
16:08:55	dansmith	what we have today is also wrong (exposing a cluster as a single uniform set of resources)
16:09:06	elod	cfriesen: it's general and there's this patch https://review.openstack.org/#/c/554053/ (which unfortunately ends up in another bug with pip... see depends-on)
16:10:11	mriedem	this spec also bakes business logic into the driver about how those resource pools are created in vcenter, i think
16:10:15	cdent	dansmith: since I'm not aware of the many bits of history involved with this stuff (either internal or external to vmware) why is that "wrong" instead of "different"?
16:10:25	mriedem	basically based on the name of the resource pool, it's encoded into the pool name which tells the virt driver what to do about it
16:10:37	dansmith	cdent: because you can't schedule instances to it that span what nova thinks are uniform resources
16:11:02	dansmith	cdent: at least, that's my understanding of it at the moment
16:11:05	openstackgerrit	Lee Yarwood proposed openstack/nova stable/pike: libvirt: Block swap volume attempts with encrypted volumes prior to Queens https://review.openstack.org/543569
16:11:19	openstackgerrit	Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094
16:11:24	dansmith	cdent: i.e. if you have a cluster with two 1G nodes, nova thinks you should be able to boot a 2G instance there but you can't
16:12:01	cdent	that's recently fixed via inventory handling with max_unit
16:12:08	dansmith	s/fixed/hacked
16:12:13	cdent	no
16:12:36	cdent	you're asserting that targeting a cluster is wrong, and what I'm trying to understand why that's wrong
16:12:40	dansmith	it is a hack, because if you have memory available on one and cpu available on another, you can't represent that
16:12:48	dansmith	no, that's not what I'm saying
16:13:04	dansmith	I'm saying nova thinks the cluster is a single bag of resources that can be combined and that's wrong
16:13:29	dansmith	and I'm saying that I think the spec we're discussing is trying to rectify that by using multiple providers with the proper amounts of inventory each
16:14:02	mriedem	yeah it's trying to break the single cluster into multiple pools
16:14:03	cdent	the spec is trying to allow resource pools (a vmware thing that enables tenanted clusters) to work
16:14:13	mriedem	but it's also tying those pools to specific tenants
16:14:24	mriedem	like the aggregate multitenenancy isolation filter
16:14:25	mriedem	i think
16:14:49	dansmith	if it's purely for the tenantization, then that's not what I think it's trying to do, and isn't really a thing the virt driver should be doing
16:14:55	cdent	there will likely be other specs that try to turn clusters into hierarchies of nested providers so that an esxi host works much the same as targeting a particular numa setup
16:15:13	dansmith	^that is what I thought the actual goal was under the covers
16:15:15	mriedem	so if the admin creates a resource pool A_foo in cluster 1, i think it means that nova is supposed to restrict all requests from tenant foo to resource pool A_foo in cluster 1
16:15:18	dansmith	but if not, that's disappointing
16:15:25	cdent	okay, stepping back a bit: why do we care what a virt driver does if it follows the interface and allows things to land
16:15:54	cdent	I'm asking that as a real question, not as a rhetorical device
16:16:30	dansmith	because nova is an abstraction?
16:16:45	cdent	right, if the abstraction works and continues to work, what does it matter?
16:16:53	dansmith	the virt driver has to do more than mechanically implement the python interface to be something that a user can depend on
16:17:16	dansmith	as an example, it doesn't work properly right now but it does implement the interface
16:17:36	cdent	and it is working to get better
16:17:41	cdent	using the tools that the interface provides
16:17:48	dansmith	this tenant grouping doesn't make it better
16:17:57	dansmith	it makes it more different in behavior than the others
16:18:01	mriedem	maybe the tenant isolation part of this should be split out
16:18:28	mriedem	so it's just a generic way to model nested resource pools in a single cluster (which is represented by the compute node provider root in nova)
16:18:53	mriedem	and then the compute node RP for vmware doesn't have cpu/ram/disk, the nested resource pool providers do
16:19:09	mriedem	and then later bake tenant isolation logic into it
16:19:19	mriedem	sorry for the bad timing but i've got a thing i've got to run to, back in awhile
16:19:40	dansmith	at that point, the tenant isolation becomes much the same problem as efried pointed out with ensuring that you get CPU and memory from the same numa node on a regular box,
16:19:56	dansmith	which probably helps, if we decide that the virt drivers should draw aggregates around those groupings or whatever,
16:20:05	dansmith	or something
16:20:10	cdent	I suspect the tenant isolation is on the spec at all so that the resource pool stuff can be justified: it's a use case for wanting to represent resource pools
16:20:23	cdent	if that's needed and reasource pools are groovy, that's great
16:20:35	efried	As I commented on https://review.openstack.org/#/c/534339/ the 1GB-from-each-of-two-children-to-make-one-2GB-instance thing shouldn't be a problem.
16:20:48	efried	BUT the "get my CPU from one RP and my disk from another" would be.
16:21:14	dansmith	efried: currently the former is a problem with the vmware driver AFAIK
16:22:03	dansmith	cdent: implementing the structurally separated resources in a cluster as nested providers of the root would make progress towards fixing that ^ and then makes the tenant isolation thing a smaller delta we can probably have a more reasonable discussion over
16:22:11	efried	To mitigate in UPT-land, you would have to do subtree lassoing like we talked about needing for NUMA; or you would have to model the cluster members as NOT being in the same tree.
16:22:17	cdent	dansmith: that's not what the spec does
16:22:24	cdent	resource pools are not physical
16:22:39	dansmith	efried: that's what I'm saying would be an improvement over what it does today
16:22:45	efried	which one dansmith?
16:22:57	cdent	if you have a problem with the cluster presented agglomerated resources as a single thing, then you'd have a problem with resource pools too
16:23:01	dansmith	efried: representing cluster members as children under the root
16:23:12	cdent	max_unit, on either the cluster or the resource pool "fixes" that
16:23:30	efried	dansmith: Only if we have the subtree-lassoing technology. I haven't caught up on my specs yet - did someone propose that yet?
16:23:33	dansmith	cdent: max_unit does not prevent nova from thinking it can get cpu and memory from two different cluster members
16:23:39	efried	Agree ^^
16:24:02	cdent	dansmith: yes, and?
16:24:11	dansmith	efried: right, I'm assuming the lassoing thing as well as changing this vmware RP exposure.. the two together would be required
16:24:55	dansmith	cdent: so max_unit has not "fixed" the vmware driver reporting a whole cluster as a single RP/node
16:24:56	efried	In that case, yes, I agree we can do this with nested.
16:25:02	cdent	the only way we get what you seem to want is for every esxi host to represent all its resources, which breaks the DRS, unless the virt driver can write allocations
16:25:26	dansmith	cdent: yeah, I think we've asserted that DRS under nova is broken, for that reason exactly
16:25:34	dansmith	broken fundamentally I mean
16:25:59	efried	cdent: I thought we talked about the fact that you want to hide the whole cluster-ness anyway?
16:26:20	efried	Represent the whole thing as a single RP, and then vmware virt would do the individual node business under the covers.
16:26:32	cdent	efried: yes, dan's saying that's broken
16:26:34	dansmith	efried: that's what it does today, and it's breakable
16:26:58	dansmith	efried: because nova will not know that one node is out of CPU but has some memory available, where another node has the opposite.. nova will think it can schedule to that, but it can't
16:27:07	gjayavelu	@cdent @dansmith @efried I'm catching up on the discussions about the resource pools spec
16:27:45	efried	dansmith: Okay, I agree with that. But I think the virt can probably mitigate
16:27:53	dansmith	efried: it can't
16:27:55	efried	...by cleverly spoofing the inventories.
16:28:05	dansmith	oh, it could work around it that way sure
16:28:13	dansmith	that's fairly wasteful though :)
16:28:17	efried	Basically by representing the least common denominator of the pool, as the inventory.
16:28:24	efried	Is it?
16:28:30	dansmith	sure
16:28:33	efried	Because you wouldn't be able to schedule more than that anyway, wouldja?
16:28:59	dansmith	well, maybe you're right

Earlier   Later