Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-19
15:45:29 mriedem thanks for finding that issue in the multinode job config
15:46:31 sahid mriedem: yeah, it's that g-api service which looked very strange
15:47:06 openstackgerrit Jay Pipes proposed openstack/nova master: validate provider summary structure https://review.openstack.org/536516
15:47:07 openstackgerrit Jay Pipes proposed openstack/nova master: WIP - populate alloc cands for trees https://review.openstack.org/534968
15:47:39 kashyap sahid: One more nit there: s/TAPs devices/TAP devices/ :-)
15:47:46 kashyap (In driver.py)
15:48:49 cfriesen has anyone run into problems with setuptools recently? I got a report that 39.0.1 didn't work with our version of pike, wondering if it's a general problem.
15:49:05 sahid kashyap: :) sure i will address that too
15:49:41 kashyap Thx
15:50:42 dansmith cfriesen: I had an issue downstream last week where the parse_version was returning a tuple instead of a string
15:50:56 dansmith cfriesen: migi was looking at it, in case you or he know anything else
15:53:34 cfriesen dansmith: we ended up pinning it to 38.5.1 for now, which seems to work okay
16:00:41 mriedem jaypipes: on that vmware resource pools spec, https://review.openstack.org/#/c/549067/6/specs/rocky/approved/vmware-resource-pool-placement.rst@210
16:01:05 mriedem jaypipes: finally just realized that this is proposing that the admin changes things *in vcenter*, not placement directly, and that's all proxied through the virt driver
16:01:24 mriedem which kind of goes against something we've always said we woudn't do, which is support out of band changes on the hypervisor
16:02:27 cdent mriedem: I think those sorts of changes are pretty much why upt exists?
16:02:42 cdent the hardware underlaying the hypervisor changes in some fashion
16:02:43 jaypipes mriedem: vcenter virt driver is virtually identical (pun intended) to how ironic virt driver wants to do things. i.e. the virt driver (not the compute service worker ala the resource tracker) creates the compute node resource provider records.
16:02:45 cdent upt notices
16:03:10 cdent efried: powervm is much the same, yeah? ^
16:03:38 mriedem jaypipes: this isn't creating the compute node resource provider record,
16:03:59 mriedem it's seeing, oh the admin created some resource pools in this cluster in vcenter directly, and now the virt driver needs to mirror that into placement,
16:04:11 mriedem and what happens when the admin decides to delete those pools in vcenter? nova would also need to mirror that
16:04:43 dansmith cdent: powervm only has one node
16:05:19 mriedem powervm wants to manage shared storage pool aggregates from what i remember
16:05:19 dansmith jaypipes: I read that spec as not exposing more nodes, but exposing a cluster as a very NUMA single node
16:05:28 cdent dansmith: not talking about the node here, just what the node is aware of: efried has mentioned that the powervm driver wants to ... yeah what mriedem said
16:05:36 dansmith mriedem: right, but not multiple nodes from one service as it stands in tree right now
16:05:50 mriedem these would be nested RPs
16:06:02 dansmith yeah, I think jaypipes is saying multiple nodes a-la ironic
16:06:07 dansmith and I don't think that's on the table for either
16:06:16 mriedem compute node RP (created by nova) -> nested tenant-specific resource pool providers, created by the virt driver, proxied from vcenter
16:06:30 mriedem the 1:M compute node thing isn't in this
16:06:30 dansmith yep
16:06:34 dansmith right
16:06:39 cfriesen mriedem: is there anything in particular blocking https://review.openstack.org/#/c/465787/ ? I notice it hasn't really gotten any reviews other than me.
16:06:42 dansmith I think jaypipes was asserting that it was
16:07:23 jaypipes dansmith: sorry, on a video call for 45 mintues
16:07:38 mriedem i guess since we don't model numa as a nested RP for the libvirt driver yet, i'm not sure how much proxy we'll be doing to compare to the vcenter thing
16:07:50 dansmith the difference here I think,
16:08:02 dansmith is that libvirt will report the static nature of the system as nested NUMA RPs,
16:08:11 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Block swapping to an encrypted volume when using QEMU to decrypt https://review.openstack.org/544238
16:08:12 dansmith where vmware would be representing cluster config, which is mutable
16:08:33 dansmith and I agree that sucks, and I don't know what would happen if vmware swizzled things after instances are booted
16:08:35 dansmith but..
16:08:55 dansmith what we have today is also wrong (exposing a cluster as a single uniform set of resources)
16:09:06 elod cfriesen: it's general and there's this patch https://review.openstack.org/#/c/554053/ (which unfortunately ends up in another bug with pip... see depends-on)
16:10:11 mriedem this spec also bakes business logic into the driver about how those resource pools are created in vcenter, i think
16:10:15 cdent dansmith: since I'm not aware of the many bits of history involved with this stuff (either internal or external to vmware) why is that "wrong" instead of "different"?
16:10:25 mriedem basically based on the name of the resource pool, it's encoded into the pool name which tells the virt driver what to do about it
16:10:37 dansmith cdent: because you can't schedule instances to it that span what nova thinks are uniform resources
16:11:02 dansmith cdent: at least, that's my understanding of it at the moment
16:11:05 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: libvirt: Block swap volume attempts with encrypted volumes prior to Queens https://review.openstack.org/543569
16:11:19 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094
16:11:24 dansmith cdent: i.e. if you have a cluster with two 1G nodes, nova thinks you should be able to boot a 2G instance there but you can't
16:12:01 cdent that's recently fixed via inventory handling with max_unit
16:12:08 dansmith s/fixed/hacked
16:12:13 cdent no
16:12:36 cdent you're asserting that targeting a cluster is wrong, and what I'm trying to understand why that's wrong
16:12:40 dansmith it is a hack, because if you have memory available on one and cpu available on another, you can't represent that
16:12:48 dansmith no, that's not what I'm saying
16:13:04 dansmith I'm saying nova thinks the cluster is a single bag of resources that can be combined and that's wrong
16:13:29 dansmith and I'm saying that I think the spec we're discussing is trying to rectify that by using multiple providers with the proper amounts of inventory each
16:14:02 mriedem yeah it's trying to break the single cluster into multiple pools
16:14:03 cdent the spec is trying to allow resource pools (a vmware thing that enables tenanted clusters) to work
16:14:13 mriedem but it's also tying those pools to specific tenants
16:14:24 mriedem like the aggregate multitenenancy isolation filter
16:14:25 mriedem i think
16:14:49 dansmith if it's purely for the tenantization, then that's not what I think it's trying to do, and isn't really a thing the virt driver should be doing
16:14:55 cdent there will likely be other specs that try to turn clusters into hierarchies of nested providers so that an esxi host works much the same as targeting a particular numa setup
16:15:13 dansmith ^that is what I thought the actual goal was under the covers
16:15:15 mriedem so if the admin creates a resource pool A_foo in cluster 1, i think it means that nova is supposed to restrict all requests from tenant foo to resource pool A_foo in cluster 1
16:15:18 dansmith but if not, that's disappointing
16:15:25 cdent okay, stepping back a bit: why do we care what a virt driver does if it follows the interface and allows things to land
16:15:54 cdent I'm asking that as a real question, not as a rhetorical device
16:16:30 dansmith because nova is an abstraction?
16:16:45 cdent right, if the abstraction works and continues to work, what does it matter?
16:16:53 dansmith the virt driver has to do more than mechanically implement the python interface to be something that a user can depend on
16:17:16 dansmith as an example, it doesn't work properly right now but it does implement the interface
16:17:36 cdent and it is working to get better
16:17:41 cdent using the tools that the interface provides
16:17:48 dansmith this tenant grouping doesn't make it better
16:17:57 dansmith it makes it more different in behavior than the others
16:18:01 mriedem maybe the tenant isolation part of this should be split out
16:18:28 mriedem so it's just a generic way to model nested resource pools in a single cluster (which is represented by the compute node provider root in nova)
16:18:53 mriedem and then the compute node RP for vmware doesn't have cpu/ram/disk, the nested resource pool providers do
16:19:09 mriedem and then later bake tenant isolation logic into it
16:19:19 mriedem sorry for the bad timing but i've got a thing i've got to run to, back in awhile
16:19:40 dansmith at that point, the tenant isolation becomes much the same problem as efried pointed out with ensuring that you get CPU and memory from the same numa node on a regular box,
16:19:56 dansmith which probably helps, if we decide that the virt drivers should draw aggregates around those groupings or whatever,
16:20:05 dansmith or something
16:20:10 cdent I suspect the tenant isolation is on the spec at all so that the resource pool stuff can be justified: it's a use case for wanting to represent resource pools
16:20:23 cdent if that's needed and reasource pools are groovy, that's great
16:20:35 efried As I commented on https://review.openstack.org/#/c/534339/ the 1GB-from-each-of-two-children-to-make-one-2GB-instance thing shouldn't be a problem.
16:20:48 efried BUT the "get my CPU from one RP and my disk from another" would be.
16:21:14 dansmith efried: currently the former is a problem with the vmware driver AFAIK
16:22:03 dansmith cdent: implementing the structurally separated resources in a cluster as nested providers of the root would make progress towards fixing that ^ and then makes the tenant isolation thing a smaller delta we can probably have a more reasonable discussion over
16:22:11 efried To mitigate in UPT-land, you would have to do subtree lassoing like we talked about needing for NUMA; or you would have to model the cluster members as NOT being in the same tree.
16:22:17 cdent dansmith: that's not what the spec does
16:22:24 cdent resource pools are not physical

Earlier   Later