Posted	Nick	Remark
#openstack-nova - 2018-02-23
16:26:07	bauwser	leakypipes: I guess you know that some other openstack projects are trying to achieve the descheduler thing in OpenStack right?
16:26:37	bauwser	the reconciliation of resource usage IIUC what the k8s descheduler is trying to achieve
16:26:43	jroll	leakypipes: cool, so does nova expose sufficient data (instances, instance groups, affinity policies, compute node information, etc) for an external thing like heat to accomplish things like this spec proposes?
16:26:55	leakypipes	jroll: yes.
16:27:07	leakypipes	jroll: for the most part :)
16:27:14	jroll	heh
16:27:22	leakypipes	jroll: but yes.
16:27:33	leakypipes	jroll: heat is admin user, no?
16:27:44	bauwser	leakypipes: heat can have trusted users
16:27:48	bauwser	I mean trusts
16:27:51	jroll	leakypipes: I've no clue, honestly
16:28:01	bauwser	it's user delegation
16:28:10	bauwser	but anyway
16:28:10	leakypipes	bauwser: and trusts don't work with federated keystone, so...
16:28:19	bauwser	it's 5pm here and I need to pack my stuff :)
16:28:22	leakypipes	:)
16:28:30	dansmith	leakypipes: so how would heat go about doing this?
16:28:34	bauwser	so, see you folks in a pub or somewhere else
16:28:44	bauwser	like in a stadium
16:28:52	dansmith	leakypipes: it could boot the first three instances in an anti-affinity group and then after that, the fourth and later go anywhere?
16:28:53	jroll	leakypipes: but, this means that an openstack user needs to rely on their cloud provider to provide heat to be able to do this, rather than being able to write a little program to diy it
16:29:00	jroll	which might be fine but kinda sucks
16:29:14	leakypipes	dansmith: are you asking about heat or k8s?
16:29:30	dansmith	jroll: does it? I kinda expected people would be able to use heat themselves with just their creds
16:29:33	dansmith	leakypipes: heat and nova
16:29:38	leakypipes	jroll: "a little program"? :)
16:29:43	jroll	:P
16:29:55	jroll	dansmith: not if you need admin rights to nova to get compute node information
16:30:10	dansmith	jroll: you don't need that, and even if you had it I don't think it would help
16:30:17	leakypipes	dansmith: no.. if heat specifies the same instance group for all 5 instances, Nova guarantees they will go on different compute hosts...
16:30:20	dansmith	jroll: users can tell compute nodes apart, just not the actual hostnames
16:30:29	leakypipes	dansmith: but you already knew that, so I suspect I am missing yoru point.
16:31:05	dansmith	leakypipes: right, but how would heat implement this "no more than three instances per host" requirement with nova?
16:31:07	jroll	dansmith: fair point, I guess I'm thinking ahead to affinity around power/network domains, sorry
16:31:32	dansmith	jroll: yeah, but unless you let heat create aggregates, I still don't think being admin is helpful
16:31:47	dansmith	jroll: and each tenant creating aggregates via heat would be kinda crazy I think
16:32:01	dansmith	maybe just using them with custom flavors? I dunno
16:32:20	jroll	dansmith: do regular users have access to aggregate info?
16:32:21	dansmith	I'm legit asking because I don't know much about heat
16:32:23	leakypipes	dansmith: how would it implement it in a transactional/atomic way? it couldn't. but it could certainly call a compute API to get the compute node -> instance association for all instances in a group and do the calculation itself, no?'
16:32:27	dansmith	if this is doable outside of nova with heat, then I'm super -2 on the spec, I just don't know that it's possible
16:32:37	jroll	I don't either, so I hope you aren't asking me :)
16:32:47	dansmith	jroll: no, but that's what I'm saying, I don't know that you'd give your heat admin user ability to do much with them either
16:33:17	dansmith	leakypipes: but it can't control placement of those things
16:34:03	leakypipes	dansmith: hmm, true nuf
16:34:33	leakypipes	dansmith: yeah, you're right. it would need to be a part of the request spec sent to placement/scheduler :(
16:34:36	jroll	dansmith: I would think you'd need enough info to be able to map instances -> aggregates, decide if the spread is sufficient (>3 power domains or whatever), and I was going to say whatever jay is thinking from there :)
16:34:54	dansmith	that's why I'm saying I expect this is a primitive we need to provide so heat can provide the automatic kill-one-spawn-one rebalancing sort of behavior
16:35:15	dansmith	jroll: well, you still can't control where a new instance lands enough to effect this policy I think
16:35:18	jroll	just boot and delete instances until it looks right :P
16:35:29	dansmith	jroll: monte carlo scheduling? I'm in
16:35:30	jroll	dansmith: right, I was hoping jay was going to solve that problem as I typed
16:35:46	jroll	:P
16:36:36	leakypipes	dansmith: there will be a point (already reached?) where this stuff is just too expensive to try and calculate for each scheduling request and makes the interface between scheduler and placement overly cumbersome.
16:37:04	dansmith	leakypipes: I don't think this has anything to do with what scheduler asks of placement
16:37:17	dansmith	leakypipes: what is described in the spec is a simple change to the current hard affinity filter
16:37:31	dansmith	and what I was proposing would be similar but with some different math
16:37:56	dansmith	the biggest change is "oh now we're going to have extra specs on instance groups"
16:38:08	dansmith	which I think is a very big step we should not take lightly
16:38:11	dansmith	(or maybe at all)
16:38:50	dansmith	especially my point about changing that value on an existing group, because people will expect that if they change that from 3 to 2, existing instances get moved around
16:38:54	dansmith	and that ain't hap'nan
16:48:02	leakypipes	dansmith, jroll: if you want some mind-bending reading from k8s on this subject: https://github.com/kubernetes/kubernetes/pull/18265/files
16:48:17	leakypipes	dansmith, jroll: note that k8s affinity stuff is still in beta...
16:48:50	jroll	oh my
16:49:29	mriedem	dansmith: i think the 'extra spec' term in his spec is a mistake, or just a WIP thing,
16:49:39	mriedem	i don't think we should add a random bag of extra specs to instance groups either
16:50:04	mriedem	dansmith: i also said updating the max-per-host value on an existing group isn't going to happen
16:50:15	mriedem	for the same reasons we don't allow changing policy on an existing group, or changing membership
16:51:00	leakypipes	what about an awkward bag of flaccid specs?
16:51:27	mriedem	if it's as simple as putting an attribute on a group in a hard affinity policy that says, instead of allowing no more than 1 vm from this group per host, you can allow up to 3 (or whatever), then that seems to fit the bill
16:51:35	mriedem	leakypipes: they have pills for that
16:51:41	leakypipes	lol
16:51:49	mriedem	let me get you frank thomas' number
16:52:00	mriedem	actually, jimmy johnson sells them too and he already lives in FL
16:52:32	mriedem	sorry i got distracted, was hastily throwing together a PBC...
16:52:32	cfriesen	dansmith: we originally had metadata in instance groups, but it got pulled out due to not really having any users
16:52:54	mriedem	cfriesen: yeah that was linked into the spec and was something i didn't even know existed
16:53:04	mriedem	metadata in general makes our lives terrible
16:53:12	mriedem	like aggregate meta, and flavor extra specs
16:53:23	mriedem	i realize it's use though
16:54:06	cfriesen	dansmith: it'd be possible to prohibit changing the value on an existing group to something that would result in the current spread being invalid....alternately you could just allow that and document that it'll only affect the *next* scheduling decision.
16:54:39	mriedem	cfriesen: to determine if the new requested value would invalidate things would mean running through the scheduler all over again
16:54:44	mriedem	and you could still get it wrong
16:54:53	cfriesen	mriedem: there has been that recurring spec for allowing instances to be added/subtracted from a group
16:54:54	mriedem	which is why we have the late affinity check on the compute hosts
16:55:12	mriedem	cfriesen: i remember powervc pushing it back in kilo but that's been abandoned for a long time
16:55:39	cfriesen	arguably we've got races all over with instance group affinity, so what's one more. :)
16:55:41	dansmith	cfriesen: yeah then the group is in violation of policy with no way of getting it out, unless you do your own shuffling
16:56:07	dansmith	mriedem: and yeah I'd rather a real attribute, but I also think we're just going to end up with unlimited attributes for other things like this
16:56:15	dansmith	mriedem: so I dunno.. it's a can of worms
16:57:53	mriedem	idk, if this is the first time somtehing like this has come up in the last what 5 years?
16:58:23	mriedem	it would get annoying if, over time, we had a bunch of attributes on groups that only applied to specific policies
16:58:23	cfriesen	for what it's worth, internally we added a "best-effort" flag to the "hard" affinity/antiaffinity policies to allow us to migrate instances off a compute node for maintenance
16:58:37	mriedem	cfriesen: so soft affinity?
16:58:56	cfriesen	no, you can turn the best-effort flag on and off dynamically
16:59:17	cfriesen	so you'd normally run with it strict, but if you need to take down a compute node you can set best-effort, move everything, then turn it back off
16:59:41	cfriesen	otherwise if you've got hard-affinity you can't migrate any of them
17:00:04	mriedem	which is why people want a force flag on evacuate, live migrate (and now cold migrate)
17:00:06	leakypipes	mriedem: and the late affinity check is (the only?) remaining upcall from a cell to API, no?

Earlier   Later