Posted	Nick	Remark
#openstack-nova - 2018-02-23
15:52:55	dansmith	mriedem: it would be hard with a grace limit
15:53:01	bauwser	sometimes people want to hard-fail on a filter, but sometimes they prefer to soft-fail
15:53:17	dansmith	bauwser: I'm not sure what that has to do here,
15:53:31	dansmith	because this is still hard fail on this filter, it's just at a nonzero level of duplication
15:53:42	mriedem	ok, so create a server group with the anti-affinity policy (hard) with some limit on overlap
15:53:42	dansmith	if you go over that limit, which is currently zero, you hard fail all the time
15:53:45	dansmith	with what is proposed here
15:54:00	dansmith	mriedem: that's what is being proposed here
15:54:08	mriedem	no it's not, it's the opposite
15:54:14	mriedem	the spec is talking about the soft policy
15:54:18	mriedem	you're saying soft policy is bad optics,
15:54:24	mriedem	so flip it to be hard policy,
15:54:28	mriedem	with the overlap limit
15:54:39	mriedem	either way we get to the same thing
15:54:40	dansmith	mriedem: but the spec is wrong in how things work
15:54:42	leakypipes	did someone say flaccid affinity?
15:54:49	dansmith	mriedem: they think they're adding a limit to soft, but you can't do that
15:54:49	mriedem	leakypipes: yes, word of the day
15:55:01	dansmith	you can increase the limit on hard though
15:55:04	mriedem	dansmith: b/c soft is enforced via weigher
15:55:07	mriedem	sure,
15:55:10	dansmith	mriedem: exactly
15:55:11	mriedem	so yeah let's make that point in the spec
15:55:19	bauwser	I'm just commenting it
15:55:46	bauwser	dansmith: are you proposing to have a max limit defined on the filter ? I don't like that
15:56:17	dansmith	bauwser: no, I'm saying that is what they are proposing here
15:56:19	dansmith	they just don't know it
15:56:19	jroll	(disclaimer, haven't read the spec) so, use case: I want to be able to say "I want 20 instances spread across at least 5 hosts, but preferably across more" - I think that's what is proposed here (based on this conversation), is that correct?
15:56:34	dansmith	jroll: no, that's what I was saying though
15:56:39	dansmith	that's what I would prefer I mean
15:56:55	jroll	dansmith: and this is what the spec thinks it is proposing, but it's really not? :)
15:56:57	bauwser	jroll: that's exactly what weighers aim to achieve
15:57:11	bauwser	for that usecase
15:57:11	dansmith	jroll: no, that's not what the spec thinks its proposing
15:57:22	mriedem	jroll: dansmith: well one thing is when you create the group, you don't say how many instances are going to be in it
15:57:26	bauwser	ie. ponder your list of hosts based on some criterias that are given by weighers
15:57:35	dansmith	jroll: it's proposing that you know how big the group will ever be, and set the allowed overlap per host to some number so you get the desired amount of minimum redundancy
15:57:37	mriedem	so "I want 20 instances spread across at least 5 hosts" is a bit chicken and egg to me
15:57:41	dansmith	which is much harder to reason about
15:57:54	jroll	dansmith: ah, ok, will read again
15:57:55	mriedem	but i guess if you already know when you create the group how many VMs are going to be in it, then that works
15:58:12	dansmith	mriedem: that's what this spec requires, is you to know how many instances will be in there
15:58:46	dansmith	mriedem: but, if you say "I want minimum redundancy of three" then you don't get any overlap until you boot the fourth and now you can start to double up because you've achieved that minimum level
16:00:23	leakypipes	dansmith: I think you saw the use case section begin with "As a NFV user, ..." and immediately threw up in your mouth ;)
16:00:25	bauwser	dansmith: it's very different
16:00:47	bauwser	dansmith: but I agree, describing for anti-affinity how many groups you want is somehow reasonable
16:00:49	jroll	leakypipes: there's vomit everywhere in here from that
16:00:54	leakypipes	:)
16:01:18	dansmith	bauwser: how many groups? you lost me there :)
16:01:31	mriedem	i think he means group of VMs per host in the group?
16:01:32	bauwser	man
16:01:40	bauwser	mriedem: bonus point
16:01:44	dansmith	oh min-redundancy you mean?
16:01:54	jroll	dansmith: I read through it, I agree with what you're suggesting there, thanks
16:01:59	dansmith	I think that's how most people will want to define a thing like this
16:02:08	dansmith	and jroll does, so.. end of story
16:02:11	bauwser	call it failure zone
16:02:34	bauwser	but if we call it failure zone, then leakypipes would kill me
16:02:40	bauwser	and I'm scared
16:02:46	leakypipes	bauwser: indeed I would.
16:03:15	bauwser	why people are doing instance groups besides the idea that they expect groups of failure domains
16:03:16	bauwser	?
16:03:49	bauwser	leakypipes: you don't imagine my mood now... restating https://www.thelocal.fr/20180223/french-cheese-wars-are-the-days-of-the-real-normandy-camembert-numbered
16:03:52	leakypipes	bauwser: because... N. F. V.
16:04:06	leakypipes	hahaha
16:04:20	leakypipes	that's awesome.
16:04:41	leakypipes	only in La France
16:08:51	leakypipes	mriedem: did you say that https://review.openstack.org/#/c/546925/2/specs/rocky/approved/allow-specifying-limit-for-affrinity-group.rst was somehow related to something sgordon put together?
16:09:15	mriedem	see the bp
16:09:22	mriedem	https://blueprints.launchpad.net/nova/+spec/complex-soft-anti-affinity-policies
16:09:32	leakypipes	oh... 113355
16:10:00	leakypipes	ok, not 113355
16:10:16	mriedem	https://review.openstack.org/#/c/546925/
16:10:38	mriedem	https://review.openstack.org/#/c/224325/ is the old backlog spec
16:10:44	leakypipes	mriedem: yeah, was confused since the spec is called something different than the blueprint.
16:13:56	leakypipes	mriedem: ftr, k8s does this very well.
16:14:07	leakypipes	mriedem: because, you know, it's an *orchestrator*.
16:14:31	leakypipes	mriedem: would be much better, IMHO, to recommend to your group to install k8s on the nova compute hosts in the NFVi and just have at it.
16:14:36	dansmith	leakypipes: do we provide heat enough control to do this externally?
16:15:47	jroll	k8s does this well because the orchestrator and the scheduler can work together, imo
16:15:52	leakypipes	dansmith: initially, yes. but k8s will do it on an ongoing basis -- i.e. if the state of the pods on a node changes, k8s will re-evaluate the tolerances and scheduling conditions and place pods on different nodes accordingly.
16:16:09	leakypipes	jroll: actually, they don't.
16:16:11	dansmith	right, sure
16:16:19	dansmith	I guess I'm not sure how heat would do this with our api
16:19:00	leakypipes	jroll: what actually happens in k8s is the descheduler (an orchestrator piece, not part of the scheduler) will evict pods from nodes that no longer meet scheduling constraints and the scheduler will try to re-place those evicted pods somewhere else (or the same node if conditions happen to be right at that specific moment)
16:19:58	leakypipes	jroll: the descheduler was originally called the rescheduler, but there's actually a rescheduler thing in k8s that does a different type of eviction, and that rescheduler piece is being deprecated.
16:20:03	jroll	leakypipes: sure, but the scheduler is aware of the orchestrator's requirements, right? or does it randomly choose and the descheduler continues to evict until things are right?
16:20:23	leakypipes	jroll: the reason the k8s works well is specifically because there is no coupling between the orchestrator and the resource tracker/scheduler.
16:21:15	leakypipes	jroll: scheduler is not aware of descheduler, no.
16:21:52	jroll	leakypipes: I probably need to do some reading to fully understand
16:22:30	leakypipes	jroll: descheduler is basically just a daemon that runs a periodic loop that checks to see if the constraints that were met on initial placement are still met. and if not, simply evicts the pod. that puts the pod into a "ready for scheduling" state and the scheduler, on its next loop, will scheduler that pod again.
16:23:16	openstackgerrit	Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211
16:23:22	jroll	curious how heat would accomplish this affinity model, though. does heat know about compute nodes, which instances are on them, the affinity characteristics of said compute nodes (to know if two instances are actually in a separate power domain, for example?)
16:23:24	leakypipes	incidentally, this is precisely how I proposed that spot instances be handled in Nova instead of a highly-coupled-to-the-scheduler architecture that the guys at Inria proposed.
16:23:59	jroll	leakypipes: is the scheduler aware of the constraints, like affinity, that need to be met? (and therefore that the descheduler is trying to meet?)
16:25:00	leakypipes	jroll: sure. the constraints are simply part of the pod scheduling policy that is associated with the pod when it's originally created.
16:25:19	bauwser	okay, so let's pretend we have "affinity knobs" :p
16:26:07	bauwser	leakypipes: I guess you know that some other openstack projects are trying to achieve the descheduler thing in OpenStack right?
16:26:37	bauwser	the reconciliation of resource usage IIUC what the k8s descheduler is trying to achieve
16:26:43	jroll	leakypipes: cool, so does nova expose sufficient data (instances, instance groups, affinity policies, compute node information, etc) for an external thing like heat to accomplish things like this spec proposes?

Earlier   Later