| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-29 | |||
| 15:31:24 | efried | Are physical hosts the underlying members of a resource pool? | |
| 15:32:25 | cdent | efried: that's not really the way it is modeled, no, but I'm not sure what you're actually asking | |
| 15:32:42 | efried | cdent I'm just not understanding where the physical resources are here. | |
| 15:32:44 | jaypipes | cdent: what does the vmware virt driver *currently* do? does it report the entire amount of VCPU, memory, etc for all resource pools and all clusters that vCenter knows about? | |
| 15:33:27 | jaypipes | cdent: and the scheduler always picks the single nova-compute service daemon and basically all scheduling/placement is simply deferred to the vCenter API? | |
| 15:33:35 | cdent | right now: the vcenter represents one cluster, and reports all vcpu, memory, etc that it knows abouts | |
| 15:34:32 | cdent | but there are a multiple nova-copute services, each for a given cluster | |
| 15:34:50 | jaypipes | cdent: I'm not seeing the benefit of modeling things using placement's resource providers data modeling if deployments using vCenter won't actually end up using any of that information when doing placement/scheduling decisions anyway. | |
| 15:37:40 | cdent | jaypipes: two reasons a) if you have multiple clusters, and thus multiple nova-compues, and those clusters differet in various ways (for example by having different tenant virtual data centers) you need to scheduler differently, b) despite our troubles, we've made a very good resource/provider/inventory/etc modelling system with placement and it is a good place to track and select where to place things | |
| 15:38:35 | cdent | "this cluster over there is the one with all the GPUs" | |
| 15:38:43 | efried | cdent It'll help me a lot to understand what is underneath a resource pool. If a resource pool reports a total of 8 VCPUs available, does that mean I am guaranteed to be able to satisfy a deploy requesting 8 VCPUs? (All other things being available) | |
| 15:39:27 | efried | Cause I got the impression that those 8 VCPUs might be spread across say four physical hosts, and an instance could only go to one of those, so the biggest one you could handle might only be 2 VCPUs or whatever. | |
| 15:40:20 | jaypipes | cdent: for a) if the end user is just saying "I'm in tenant X, so launch an instance on virtual data center X", that is a stupid-simple scheduling decision that needs nothing other than a single trait (CUSTOM_TENANT_X decorates a single resource provider representing the nova-compute service that manages that tenant virtual data center). For b) again, if vCenter isn't going to use that information in its intra-cluster scheduling decisions, I | |
| 15:40:20 | jaypipes | don't see the point. | |
| 15:42:02 | jaypipes | cdent: another question for you.. | |
| 15:42:22 | cdent | jaypipes: can I pause you for a minute while I answer efried, then your current question? | |
| 15:43:11 | jaypipes | cdent: even if placement ended up getting proper inventory information for these vCenter clusters, would VMWare OpenStack users actually be able looking at that inventory information? Or would they be using VMWare GUIs to look at inventory information for their specific tenant virtual datacenter? | |
| 15:43:19 | jaypipes | cdent: of course, no worries | |
| 15:44:42 | cdent | efried: a while back there were some adjustment made to the vmware driver to make sure that the max_unit was no greater than what any individual host could provide. so your second statement about "the biggest one" is an issue in some cases, but it's squishy because of the DRS (dynamic resource scheduler thing which can do unattended live migrations and what not, magic stuff I haven't dug into) | |
| 15:46:24 | efried | So it is in fact the case that there are some number of physical hosts behind a resource pool, and an instance can only be deployed to one, not spread out across more than one. True statement? | |
| 15:46:26 | cdent | efried: so, yes, there is a fair bit of complexity which is _why_ being able to manipulate the entire provider tree from within the virt driver has been of such interest to rado (and me). | |
| 15:46:49 | jaypipes | maciejjozefczyk: you around? wanted to chat with you about https://bugs.launchpad.net/nova/+bug/1742747 | |
| 15:46:50 | openstack | Launchpad bug 1742747 in OpenStack Compute (nova) "RT overrides default allocation_ratios for ram cpu and disk" [Undecided,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk) | |
| 15:46:56 | cdent | efried: it can move and you can never know that it did | |
| 15:47:52 | efried | "it" the physical host? "can move" to another resource pool? and "you" the virt driver or you the operator? | |
| 15:48:10 | jaypipes | cdent: with the shared-memory solution, can't VMs literally be on multiple hypervisor hosts at once? | |
| 15:48:11 | maciejjozefczyk | jaypipes: yes, hey! | |
| 15:48:33 | jaypipes | maciejjozefczyk: you see my note on that bug? | |
| 15:49:25 | maciejjozefczyk | jaypipes: reading | |
| 15:49:40 | cdent | jaypipes: I've done my damnedest thus far to stay out of the guts of what vcenter gets up to, but there's plenty of magical knobs, but I'm trying to avoid knowing about them. there's already too many things to know | |
| 15:49:57 | jaypipes | cdent: ack | |
| 15:50:03 | cdent | efried: it is the instance | |
| 15:50:37 | efried | okay. "move" where, inter-pool or just intra-pool? | |
| 15:50:46 | efried | not that it actually matters. | |
| 15:51:19 | maciejjozefczyk | jaypipes: so we should also check aggregation metadatas if there is already definition of allocation ratio defined? | |
| 15:51:42 | maciejjozefczyk | if zeros are in config? | |
| 15:51:50 | jaypipes | maciejjozefczyk: yeah. check out that operators mailing list thread I linked in there... | |
| 15:52:07 | cdent | efried: intra-cluster. | |
| 15:52:24 | efried | cdent So the thing is, we didn't have a way to model multiple providers before. Now we do. But it sounds to me like nested is *not* the multiple-provider model we want to use here. It just doesn't make sense in light of recently-merged capabilities for a single provider *or* tree to "own" inventory spanning more than one actual compute host. (Unless, as jaypipes implies, there's a memory sharing thing - we can talk abou | |
| 15:52:24 | efried | t that later.) | |
| 15:52:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add regression test wrinkle for rebuilding twice with invalid image https://review.openstack.org/538955 | |
| 15:52:49 | mriedem | artom: ^ | |
| 15:54:14 | efried | cdent If you do it in a single provider OR a tree, you'll always have to reconcile a request for some subset of the total resources that you can't actually satisfy from any one host. | |
| 15:54:16 | artom | mriedem, I think you'd have to get the server out of ERROR first | |
| 15:55:02 | efried | cdent With the single-provider model, which I gather is how it has needed to be done to this point, it sounds like some of that reconciliation was that max-unit hack. | |
| 15:55:20 | jaypipes | efried: yeah. I think it's a much better solution for vCenter to model their resource tracking in the vCenter virt driver in a similar fashion to how Ironic's virt driver works. | |
| 15:55:56 | cdent | efried: I have no issue with the inventory for hardware things being pushed all the way down the real hardware, but I _do_ think that representing the hierarchical relationships between things using a tool we've created that has hierarchy makes sense (in this case cluster->resource-pool->physical host) | |
| 15:56:27 | jaypipes | efried: that wouldn't *preclude* using nested providers if, say, certain compute nodes had PCI devices or NUMA nodes, etc. But I would use a new root provider for each compute node in the cluster and represent the cluster as a placement aggregate, not as a child node under a "fake root" | |
| 15:56:42 | efried | jaypipes ++ exactly | |
| 15:57:01 | jaypipes | cdent: I think we're mostly agreeing. just disagreeing on the "single root vs multiple roots" thing. | |
| 15:57:09 | gibi | mriedem: quick question: There is a bugfix https://review.openstack.org/#/c/525242 that needs rpc version bump. Is it something that needs to wait for Rocky? | |
| 15:57:24 | efried | cdent A placement tree is more than just a way to represent a hierarchy. It is also allows resource requests to span a tree. Which doesn't make sense in the vcenter model as I'm undestanding it. | |
| 15:57:52 | mriedem | artom: you can rebuild an ERROR server | |
| 15:58:05 | cdent | efried: then I think it probably hasn't been explained well enough | |
| 15:58:14 | artom | mriedem, hah, that makes it even easier to hit | |
| 15:58:29 | mriedem | gibi: that doesn't necessarily mean it has to wait for rocky, | |
| 15:58:35 | mriedem | gibi: but it does mean we wouldn't backport it | |
| 15:58:39 | maciejjozefczyk | jaypipes: I'm ok with the idea to check aggregation metadatas first | |
| 15:59:04 | gibi | mriedem: OK. Then I'm going to rebase the above patch | |
| 15:59:19 | gibi | mriedem: the backport impossibility was agreed earlier so that is clear | |
| 15:59:28 | jaypipes | maciejjozefczyk: ok. with your operator hat on, what do you think about my proposal, considering it involves a technically not backwards-compatible behaviour change? | |
| 15:59:59 | efried | cdent I think I get that logically, conductor < cluster <=> n-cpu < resource pool < compute hosts (right?) and from that perspective you want to be able to "model" that structure as a tree. | |
| 16:00:32 | cdent | efried, jaypipes: I'd be much better at this conversation if I wasn't still fighting off this cold, but it's not just the desired to model, there's also a desire to do some accounting | |
| 16:00:57 | efried | cdent 1 < many and 1 <=> 1 | |
| 16:01:30 | maciejjozefczyk | jaypipes: From my perspective it would not change anyting, I set allocation_ratio per host not per aggregate, so my opinion seems not be helpful | |
| 16:01:46 | jaypipes | mgagne: hey! :) | |
| 16:02:00 | mgagne | jaypipes: hi! | |
| 16:02:07 | jaypipes | mgagne: we're talking about your favorite new anti-feature regarding host aggregates and allocation ratios :) | |
| 16:02:21 | mgagne | jaypipes: I just saw your email =) | |
| 16:02:39 | jaypipes | mgagne: cool. I'm hoping to commandeer maciejjozefczyk's patch to add this fix for you. | |
| 16:03:01 | jaypipes | of course, I'm trying to convince maciejjozefczyk that this would be of benefit ;) | |
| 16:03:04 | mgagne | jaypipes: I have yet to properly analyze your suggestion and related bug report. | |
| 16:03:22 | jaypipes | mgagne: no worries, would be good to get your feedback some time today. | |
| 16:03:39 | jaypipes | mgagne: I'm going to ping melwitt on it when she's up, too. | |
| 16:03:45 | mgagne | jaypipes: sure | |
| 16:04:00 | sauloaugusto | Is it possible to setup pike without cells ? | |
| 16:04:11 | openstackgerrit | Sen Yang proposed openstack/python-novaclient master: Implement hypervisor hostname exact patten match for server cold migrate https://review.openstack.org/520187 | |
| 16:04:14 | jaypipes | sauloaugusto: nope. | |
| 16:04:56 | efried | cdent So I think where we're getting to is this: You could have your nice satisfying model, and have to hack the inventory distribution to make it work; or you could have a flatter model that's less closely representative of the "real world", but your inventorying is simple, and accounting flows automatically from scheduling/allocations. | |
| 16:05:25 | efried | cdent ...except for the part where vcenter is gonna move an instance from one host to another. Yeah, that gets a tad hairy without having done a real live migration or whatever. | |
| 16:05:40 | sauloaugusto | jaypipes: Is there any clue about the error when you migrate from a no cell env to pike and you can only list instances from admin tenant ? | |
| 16:06:39 | dansmith | sauloaugusto: that doesn't sound like a cells-related thing | |
| 16:07:30 | dansmith | (the cells code doesn't do anything special with users/tenants) | |
| 16:07:33 | cdent | efried: having allocations against physical hosts (instead of the cluster) cannot work because instances move and we've declared the virt driver is not allowed to manipulate allocations | |
| 16:07:56 | efried | cdent Yeah, I see where that's a problem. I don't have an answer for that. | |
| 16:08:08 | cdent | thus inventory on resource pools | |
| 16:08:17 | efried | and resultant hackage. | |
| 16:08:25 | efried | which is unfortunate. | |
| 16:08:32 | maciejjozefczyk | jaypipes mgagne: Need to go. Let me know in bug report about what you decided okey ;) I'll update the patch then | |
| 16:08:33 | cdent | because that is the level at which accounting is desired | |
| 16:08:43 | jaypipes | maciejjozefczyk: will do, thanks mate :) | |
| 16:09:49 | efried | cdent Well, at the very least the resource pools should not be children in the same tree. They should be roots. | |
| 16:10:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Rollback instance.image_ref on failed rebuild https://review.openstack.org/538961 | |
| 16:10:16 | mriedem | artom: here is the fix ^ let me know when you have an LP bug number so i can update the references in here | |
| 16:10:33 | artom | mriedem, yep, writing it up as we speak | |
| 16:11:02 | cdent | efried: remind me (because I'm half asleep) of the mechanism to go from a root (when in nova-scheduler) to the compute node where the spawn can be called? | |
| 16:11:14 | sauloaugusto | dansmith: I think that is related , because the error start after I do themigration steps from nova . After that I start to see the Can't upgrade a READER transaction to a WRITER mid-transaction error on nova-api.log . | |
| 16:11:20 | cdent | efried: if the root is in fact not that compute-monde | |
| 16:11:26 | artom | mriedem, so we're rolling back the image but not other things that could change, like keypair? | |
| 16:11:43 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add regression test for bug 1735407 https://review.openstack.org/526095 | |