Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-29
15:46:24 efried So it is in fact the case that there are some number of physical hosts behind a resource pool, and an instance can only be deployed to one, not spread out across more than one. True statement?
15:46:26 cdent efried: so, yes, there is a fair bit of complexity which is _why_ being able to manipulate the entire provider tree from within the virt driver has been of such interest to rado (and me).
15:46:49 jaypipes maciejjozefczyk: you around? wanted to chat with you about https://bugs.launchpad.net/nova/+bug/1742747
15:46:50 openstack Launchpad bug 1742747 in OpenStack Compute (nova) "RT overrides default allocation_ratios for ram cpu and disk" [Undecided,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)
15:46:56 cdent efried: it can move and you can never know that it did
15:47:52 efried "it" the physical host? "can move" to another resource pool? and "you" the virt driver or you the operator?
15:48:10 jaypipes cdent: with the shared-memory solution, can't VMs literally be on multiple hypervisor hosts at once?
15:48:11 maciejjozefczyk jaypipes: yes, hey!
15:48:33 jaypipes maciejjozefczyk: you see my note on that bug?
15:49:25 maciejjozefczyk jaypipes: reading
15:49:40 cdent jaypipes: I've done my damnedest thus far to stay out of the guts of what vcenter gets up to, but there's plenty of magical knobs, but I'm trying to avoid knowing about them. there's already too many things to know
15:49:57 jaypipes cdent: ack
15:50:03 cdent efried: it is the instance
15:50:37 efried okay. "move" where, inter-pool or just intra-pool?
15:50:46 efried not that it actually matters.
15:51:19 maciejjozefczyk jaypipes: so we should also check aggregation metadatas if there is already definition of allocation ratio defined?
15:51:42 maciejjozefczyk if zeros are in config?
15:51:50 jaypipes maciejjozefczyk: yeah. check out that operators mailing list thread I linked in there...
15:52:07 cdent efried: intra-cluster.
15:52:24 efried t that later.)
15:52:24 efried cdent So the thing is, we didn't have a way to model multiple providers before. Now we do. But it sounds to me like nested is *not* the multiple-provider model we want to use here. It just doesn't make sense in light of recently-merged capabilities for a single provider *or* tree to "own" inventory spanning more than one actual compute host. (Unless, as jaypipes implies, there's a memory sharing thing - we can talk abou
15:52:47 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add regression test wrinkle for rebuilding twice with invalid image https://review.openstack.org/538955
15:52:49 mriedem artom: ^
15:54:14 efried cdent If you do it in a single provider OR a tree, you'll always have to reconcile a request for some subset of the total resources that you can't actually satisfy from any one host.
15:54:16 artom mriedem, I think you'd have to get the server out of ERROR first
15:55:02 efried cdent With the single-provider model, which I gather is how it has needed to be done to this point, it sounds like some of that reconciliation was that max-unit hack.
15:55:20 jaypipes efried: yeah. I think it's a much better solution for vCenter to model their resource tracking in the vCenter virt driver in a similar fashion to how Ironic's virt driver works.
15:55:56 cdent efried: I have no issue with the inventory for hardware things being pushed all the way down the real hardware, but I _do_ think that representing the hierarchical relationships between things using a tool we've created that has hierarchy makes sense (in this case cluster->resource-pool->physical host)
15:56:27 jaypipes efried: that wouldn't *preclude* using nested providers if, say, certain compute nodes had PCI devices or NUMA nodes, etc. But I would use a new root provider for each compute node in the cluster and represent the cluster as a placement aggregate, not as a child node under a "fake root"
15:56:42 efried jaypipes ++ exactly
15:57:01 jaypipes cdent: I think we're mostly agreeing. just disagreeing on the "single root vs multiple roots" thing.
15:57:09 gibi mriedem: quick question: There is a bugfix https://review.openstack.org/#/c/525242 that needs rpc version bump. Is it something that needs to wait for Rocky?
15:57:24 efried cdent A placement tree is more than just a way to represent a hierarchy. It is also allows resource requests to span a tree. Which doesn't make sense in the vcenter model as I'm undestanding it.
15:57:52 mriedem artom: you can rebuild an ERROR server
15:58:05 cdent efried: then I think it probably hasn't been explained well enough
15:58:14 artom mriedem, hah, that makes it even easier to hit
15:58:29 mriedem gibi: that doesn't necessarily mean it has to wait for rocky,
15:58:35 mriedem gibi: but it does mean we wouldn't backport it
15:58:39 maciejjozefczyk jaypipes: I'm ok with the idea to check aggregation metadatas first
15:59:04 gibi mriedem: OK. Then I'm going to rebase the above patch
15:59:19 gibi mriedem: the backport impossibility was agreed earlier so that is clear
15:59:28 jaypipes maciejjozefczyk: ok. with your operator hat on, what do you think about my proposal, considering it involves a technically not backwards-compatible behaviour change?
15:59:59 efried cdent I think I get that logically, conductor < cluster <=> n-cpu < resource pool < compute hosts (right?) and from that perspective you want to be able to "model" that structure as a tree.
16:00:32 cdent efried, jaypipes: I'd be much better at this conversation if I wasn't still fighting off this cold, but it's not just the desired to model, there's also a desire to do some accounting
16:00:57 efried cdent 1 < many and 1 <=> 1
16:01:30 maciejjozefczyk jaypipes: From my perspective it would not change anyting, I set allocation_ratio per host not per aggregate, so my opinion seems not be helpful
16:01:46 jaypipes mgagne: hey! :)
16:02:00 mgagne jaypipes: hi!
16:02:07 jaypipes mgagne: we're talking about your favorite new anti-feature regarding host aggregates and allocation ratios :)
16:02:21 mgagne jaypipes: I just saw your email =)
16:02:39 jaypipes mgagne: cool. I'm hoping to commandeer maciejjozefczyk's patch to add this fix for you.
16:03:01 jaypipes of course, I'm trying to convince maciejjozefczyk that this would be of benefit ;)
16:03:04 mgagne jaypipes: I have yet to properly analyze your suggestion and related bug report.
16:03:22 jaypipes mgagne: no worries, would be good to get your feedback some time today.
16:03:39 jaypipes mgagne: I'm going to ping melwitt on it when she's up, too.
16:03:45 mgagne jaypipes: sure
16:04:00 sauloaugusto Is it possible to setup pike without cells ?
16:04:11 openstackgerrit Sen Yang proposed openstack/python-novaclient master: Implement hypervisor hostname exact patten match for server cold migrate https://review.openstack.org/520187
16:04:14 jaypipes sauloaugusto: nope.
16:04:56 efried cdent So I think where we're getting to is this: You could have your nice satisfying model, and have to hack the inventory distribution to make it work; or you could have a flatter model that's less closely representative of the "real world", but your inventorying is simple, and accounting flows automatically from scheduling/allocations.
16:05:25 efried cdent ...except for the part where vcenter is gonna move an instance from one host to another. Yeah, that gets a tad hairy without having done a real live migration or whatever.
16:05:40 sauloaugusto jaypipes: Is there any clue about the error when you migrate from a no cell env to pike and you can only list instances from admin tenant ?
16:06:39 dansmith sauloaugusto: that doesn't sound like a cells-related thing
16:07:30 dansmith (the cells code doesn't do anything special with users/tenants)
16:07:33 cdent efried: having allocations against physical hosts (instead of the cluster) cannot work because instances move and we've declared the virt driver is not allowed to manipulate allocations
16:07:56 efried cdent Yeah, I see where that's a problem. I don't have an answer for that.
16:08:08 cdent thus inventory on resource pools
16:08:17 efried and resultant hackage.
16:08:25 efried which is unfortunate.
16:08:32 maciejjozefczyk jaypipes mgagne: Need to go. Let me know in bug report about what you decided okey ;) I'll update the patch then
16:08:33 cdent because that is the level at which accounting is desired
16:08:43 jaypipes maciejjozefczyk: will do, thanks mate :)
16:09:49 efried cdent Well, at the very least the resource pools should not be children in the same tree. They should be roots.
16:10:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Rollback instance.image_ref on failed rebuild https://review.openstack.org/538961
16:10:16 mriedem artom: here is the fix ^ let me know when you have an LP bug number so i can update the references in here
16:10:33 artom mriedem, yep, writing it up as we speak
16:11:02 cdent efried: remind me (because I'm half asleep) of the mechanism to go from a root (when in nova-scheduler) to the compute node where the spawn can be called?
16:11:14 sauloaugusto dansmith: I think that is related , because the error start after I do themigration steps from nova . After that I start to see the Can't upgrade a READER transaction to a WRITER mid-transaction error on nova-api.log .
16:11:20 cdent efried: if the root is in fact not that compute-monde
16:11:26 artom mriedem, so we're rolling back the image but not other things that could change, like keypair?
16:11:43 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add regression test for bug 1735407 https://review.openstack.org/526095
16:11:45 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add late server group policy check to rebuild https://review.openstack.org/525242
16:11:45 openstack bug 1735407 in OpenStack Compute (nova) "[Nova] Evacuation doesn't respect anti-affinity rules" [Medium,In progress] https://launchpad.net/bugs/1735407 - Assigned to Balazs Gibizer (balazs-gibizer)
16:11:48 mriedem artom: yeah - which was my original complaint in the earlier fix
16:12:03 dansmith sauloaugusto: the migration steps from no cells to one cell were required before pike
16:12:09 efried cdent Not sure we quite have that mechanism yet.
16:12:13 mriedem artom: we can't really roll everything back with what we have today in the conductor code
16:12:35 mriedem we'd either have to change the rpc cast to a call, or move the code that changes the instance from the api to conductor
16:12:37 cdent efried: indeed, thus why I've been modelling it the way I've been
16:12:40 efried cdent Yeah, a way to tie these brother-roots back to the compute host for purposes of allocation and deploy.
16:12:44 cdent if it is a child you know where to go
16:12:50 artom mriedem, yeah... I guess it's fine as a bandaid for now? Ideally we need to come back to this and fix it properly, maybe like what Andrey was proposing
16:12:50 mriedem hell, maybe we should have just moved the instance.save() from api to conductor...
16:12:52 cdent because
16:12:54 cdent it is a child
16:13:04 mriedem artom: yes this is a "tactical fix"
16:13:13 cdent a resource provider withing a compute-node is _obviously_ a child
16:13:22 efried cdent But if it's a child, the scheduler will happily deploy an instance with VCPU from pool1, MEMORY_MB from pool2, and DISK_GB from pool3.
16:13:24 cdent and it seems to me that the ironic model might be the thing that's weird
16:13:25 artom mriedem, heh, code for "someone else can deal with it once I'm retired" ;)

Earlier   Later