Posted	Nick	Remark
#openstack-nova - 2018-02-23
16:29:38	leakypipes	jroll: "a little program"? :)
16:29:43	jroll	:P
16:29:55	jroll	dansmith: not if you need admin rights to nova to get compute node information
16:30:10	dansmith	jroll: you don't need that, and even if you had it I don't think it would help
16:30:17	leakypipes	dansmith: no.. if heat specifies the same instance group for all 5 instances, Nova guarantees they will go on different compute hosts...
16:30:20	dansmith	jroll: users can tell compute nodes apart, just not the actual hostnames
16:30:29	leakypipes	dansmith: but you already knew that, so I suspect I am missing yoru point.
16:31:05	dansmith	leakypipes: right, but how would heat implement this "no more than three instances per host" requirement with nova?
16:31:07	jroll	dansmith: fair point, I guess I'm thinking ahead to affinity around power/network domains, sorry
16:31:32	dansmith	jroll: yeah, but unless you let heat create aggregates, I still don't think being admin is helpful
16:31:47	dansmith	jroll: and each tenant creating aggregates via heat would be kinda crazy I think
16:32:01	dansmith	maybe just using them with custom flavors? I dunno
16:32:20	jroll	dansmith: do regular users have access to aggregate info?
16:32:21	dansmith	I'm legit asking because I don't know much about heat
16:32:23	leakypipes	dansmith: how would it implement it in a transactional/atomic way? it couldn't. but it could certainly call a compute API to get the compute node -> instance association for all instances in a group and do the calculation itself, no?'
16:32:27	dansmith	if this is doable outside of nova with heat, then I'm super -2 on the spec, I just don't know that it's possible
16:32:37	jroll	I don't either, so I hope you aren't asking me :)
16:32:47	dansmith	jroll: no, but that's what I'm saying, I don't know that you'd give your heat admin user ability to do much with them either
16:33:17	dansmith	leakypipes: but it can't control placement of those things
16:34:03	leakypipes	dansmith: hmm, true nuf
16:34:33	leakypipes	dansmith: yeah, you're right. it would need to be a part of the request spec sent to placement/scheduler :(
16:34:36	jroll	dansmith: I would think you'd need enough info to be able to map instances -> aggregates, decide if the spread is sufficient (>3 power domains or whatever), and I was going to say whatever jay is thinking from there :)
16:34:54	dansmith	that's why I'm saying I expect this is a primitive we need to provide so heat can provide the automatic kill-one-spawn-one rebalancing sort of behavior
16:35:15	dansmith	jroll: well, you still can't control where a new instance lands enough to effect this policy I think
16:35:18	jroll	just boot and delete instances until it looks right :P
16:35:29	dansmith	jroll: monte carlo scheduling? I'm in
16:35:30	jroll	dansmith: right, I was hoping jay was going to solve that problem as I typed
16:35:46	jroll	:P
16:36:36	leakypipes	dansmith: there will be a point (already reached?) where this stuff is just too expensive to try and calculate for each scheduling request and makes the interface between scheduler and placement overly cumbersome.
16:37:04	dansmith	leakypipes: I don't think this has anything to do with what scheduler asks of placement
16:37:17	dansmith	leakypipes: what is described in the spec is a simple change to the current hard affinity filter
16:37:31	dansmith	and what I was proposing would be similar but with some different math
16:37:56	dansmith	the biggest change is "oh now we're going to have extra specs on instance groups"
16:38:08	dansmith	which I think is a very big step we should not take lightly
16:38:11	dansmith	(or maybe at all)
16:38:50	dansmith	especially my point about changing that value on an existing group, because people will expect that if they change that from 3 to 2, existing instances get moved around
16:38:54	dansmith	and that ain't hap'nan
16:48:02	leakypipes	dansmith, jroll: if you want some mind-bending reading from k8s on this subject: https://github.com/kubernetes/kubernetes/pull/18265/files
16:48:17	leakypipes	dansmith, jroll: note that k8s affinity stuff is still in beta...
16:48:50	jroll	oh my
16:49:29	mriedem	dansmith: i think the 'extra spec' term in his spec is a mistake, or just a WIP thing,
16:49:39	mriedem	i don't think we should add a random bag of extra specs to instance groups either
16:50:04	mriedem	dansmith: i also said updating the max-per-host value on an existing group isn't going to happen
16:50:15	mriedem	for the same reasons we don't allow changing policy on an existing group, or changing membership
16:51:00	leakypipes	what about an awkward bag of flaccid specs?
16:51:27	mriedem	if it's as simple as putting an attribute on a group in a hard affinity policy that says, instead of allowing no more than 1 vm from this group per host, you can allow up to 3 (or whatever), then that seems to fit the bill
16:51:35	mriedem	leakypipes: they have pills for that
16:51:41	leakypipes	lol
16:51:49	mriedem	let me get you frank thomas' number
16:52:00	mriedem	actually, jimmy johnson sells them too and he already lives in FL
16:52:32	cfriesen	dansmith: we originally had metadata in instance groups, but it got pulled out due to not really having any users
16:52:32	mriedem	sorry i got distracted, was hastily throwing together a PBC...
16:52:54	mriedem	cfriesen: yeah that was linked into the spec and was something i didn't even know existed
16:53:04	mriedem	metadata in general makes our lives terrible
16:53:12	mriedem	like aggregate meta, and flavor extra specs
16:53:23	mriedem	i realize it's use though
16:54:06	cfriesen	dansmith: it'd be possible to prohibit changing the value on an existing group to something that would result in the current spread being invalid....alternately you could just allow that and document that it'll only affect the *next* scheduling decision.
16:54:39	mriedem	cfriesen: to determine if the new requested value would invalidate things would mean running through the scheduler all over again
16:54:44	mriedem	and you could still get it wrong
16:54:53	cfriesen	mriedem: there has been that recurring spec for allowing instances to be added/subtracted from a group
16:54:54	mriedem	which is why we have the late affinity check on the compute hosts
16:55:12	mriedem	cfriesen: i remember powervc pushing it back in kilo but that's been abandoned for a long time
16:55:39	cfriesen	arguably we've got races all over with instance group affinity, so what's one more. :)
16:55:41	dansmith	cfriesen: yeah then the group is in violation of policy with no way of getting it out, unless you do your own shuffling
16:56:07	dansmith	mriedem: and yeah I'd rather a real attribute, but I also think we're just going to end up with unlimited attributes for other things like this
16:56:15	dansmith	mriedem: so I dunno.. it's a can of worms
16:57:53	mriedem	idk, if this is the first time somtehing like this has come up in the last what 5 years?
16:58:23	cfriesen	for what it's worth, internally we added a "best-effort" flag to the "hard" affinity/antiaffinity policies to allow us to migrate instances off a compute node for maintenance
16:58:23	mriedem	it would get annoying if, over time, we had a bunch of attributes on groups that only applied to specific policies
16:58:37	mriedem	cfriesen: so soft affinity?
16:58:56	cfriesen	no, you can turn the best-effort flag on and off dynamically
16:59:17	cfriesen	so you'd normally run with it strict, but if you need to take down a compute node you can set best-effort, move everything, then turn it back off
16:59:41	cfriesen	otherwise if you've got hard-affinity you can't migrate any of them
17:00:04	mriedem	which is why people want a force flag on evacuate, live migrate (and now cold migrate)
17:00:06	leakypipes	mriedem: and the late affinity check is (the only?) remaining upcall from a cell to API, no?
17:00:15	mriedem	leakypipes: hells no
17:00:32	leakypipes	mriedem: it's not an upcall?
17:00:35	mriedem	https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls
17:00:38	mriedem	it is an upcall
17:00:42	mriedem	but it's not the only one
17:00:44	leakypipes	oh, not the only reminaing..
17:00:46	mriedem	we've got aggregates too
17:01:04	mriedem	so aggregates and affinity are the remaining upcall issues
17:01:21	mriedem	but right now there are 2 each
17:01:24	mriedem	so 4 upcall issues
17:01:33	leakypipes	mriedem: then you'll LOVE my aggregate affinity spec! :) now with moar AFFINITY and moar AGGREGATES!
17:01:48	mriedem	leakypipes: i already said 'upcall upcall upcall' on that spec several times :)
17:01:54	leakypipes	I know :)
17:02:20	mriedem	i think i also hedged with something like, 'but we already do this in a few other places so people already have to rely on it, so maybe another log on the fire doesn't kill us'
17:02:55	cfriesen	On a totally different topic...has anyone ever heard of nova allocating duplicate network interfaces? (So the user boots while asking for 2 network interfaces, and nova allocates two ports on each network.)
17:03:08	mriedem	yes
17:03:10	mriedem	that's old news
17:03:25	mriedem	tempest has a test for it also i think
17:03:31	mriedem	been around since juno?
17:05:28	cfriesen	I mean nova is allocating twice as many as were asked for.
17:05:47	mriedem	double your pleasure
17:05:49	mriedem	idk, going to lunch
17:14:43	efried	Greetings from JFK airport
17:27:30	mnaser	ok
17:27:33	mnaser	im convinced grenade is broken

Earlier   Later