Posted	Nick	Remark
#openstack-nova - 2018-02-06
21:30:23	efried	mriedem: Thanks. Will dig.
21:30:38	efried	edleafe: So first of all, it would only apply to the unnumbered request group. Because you're already getting same-provider "affinity" in the numbered groups.
21:31:29	efried	edleafe: So within the unnumbered request group, the API would force the resources from classes you list to come from either the same provider, or the same aggregate.
21:32:19	efried	edleafe: I'm not sure an anti_affine_resources makes any sense, but could be added if it does.
21:32:46	edleafe	efried: "affine"?
21:33:12	edleafe	efried: How would that specify RAM, PCI deveice, etc., associated with a given NUMA?
21:33:33	efried	edleafe: Okay, let me give a more complete example, make more sense.
21:34:29	sean-k-mooney	edleafe: anti_affine_resources for nic bonding e.g. antiafintiy of the pf for the VF request
21:34:40	sean-k-mooney	efried: ^
21:34:44	efried	sean-k-mooney: ack
21:35:20	efried	edleafe: To start with, model NUMA nodes as subtrees under the root compute host RP (which may not in fact provide any resources). So maybe the first tier children provide CPU & memory, and the second tier (grandchildren) are the network devices providing VFs.
21:35:53	efried	edleafe: Associate each child/grandchildren in a given NUMA node with the same aggregate UUID.
21:36:49	efried	edleafe: Then your request would be like GET /allocation_candidates?resources=VCPU:2,MEMORY_MB:2048,SRIOV_NET_VF:1,DISK_GB=2048&affine_resources=VCPU,MEMORY_MB,SRIOV_NET_VF
21:37:03	edleafe	why can't the network devices be children of the NUMA node?
21:37:11	efried	They are
21:37:14	efried	swhat I said
21:37:33	efried	Sorry, to rephrase:
21:37:36	sean-k-mooney	efried: but the cpu and memory shoudl also be childeren of the numa node too
21:37:51	edleafe	"and the second tier (grandchildren) are the network devices providing VFs."
21:37:53	efried	sean-k-mooney: The CPU/mem are resources provided by the numa node which is the RP.
21:38:47	sean-k-mooney	efried: ah they are inventories of the numa node rather then sub resouce providres but that means you can have traints on the cpu and ram
21:39:12	sean-k-mooney	*can't have traits
21:39:17	efried	So root RP is the compute node. It provides no resources. It has two child RPs, representing NUMA nodes. They provide VCPU and memory inventory. Then those guys have child RPs representing network devices, providing VF inventory.
21:39:34	edleafe	efried: one thing we learned back in Atlanta was that this stuff was much better discussed with a drawing surface so that all could see the relationships
21:40:04	edleafe	Those of us who think visually need that for anything other than the trivial stuff
21:40:22	mriedem	https://awwapp.com/# ?
21:40:25	efried	sean-k-mooney: Yeah, that would be another way to do it: model the NUMA provider as an empty child of the empty root compute host and give the NUMA provider three grandchild tiers which provide CPU, memory, and VFs.
21:40:48	edleafe	mriedem: sure. efried, have at it!
21:40:48	efried	But not to get hung up on those details.
21:41:07	mriedem	or https://webwhiteboard.com/
21:41:15	edleafe	And I'm still not clear how "distance" figures into any of this
21:41:16	mriedem	i won't be joining, but just being super helpful
21:41:51	efried	edleafe: It doesn't. This is purely one-dimensional affinity. Either they're in the same agg or they're not.
21:42:17	edleafe	efried: ok, that's where I was not clear
21:43:18	edleafe	efried: won't the current NRP stuff handle this without additional work? IOW, the CPU,RAM,VF all have to
21:43:31	edleafe	be from the same provider
21:43:41	efried	Only if we can model them as belonging to the same provider.
21:43:59	sean-k-mooney	edleafe: distance was discused in 2 forms. 1 a distance in nodes along the provider tree and 2 a distence to a shared resouce provider and an aggreate member to model things like network latency for share storage
21:44:00	efried	Which we can do for just CPU/memory, but not for VFs.
21:44:31	efried	And also splitting things up more allows us to tie traits more specifically to their resources, if that matters.
21:44:31	edleafe	efried: yeah, I guess it could think of the compute node as the root provider, and then stuff from different NUMA nodes would match
21:45:50	edleafe	sean-k-mooney: I remember the shared distance thing. I thought that was discarded as not needed, because you could just create tighter aggregates so that only "near" storage would ever be used
21:46:11	efried	sean-k-mooney: I tried thinking about distance along a tree, but kept painting myself into a corner where that would severely restrict how things would have to be modeled.
21:46:38	sean-k-mooney	edleafe: yes proably its been a while since denver
21:46:39	efried	sean-k-mooney: Do you understand the concept of distance as relates to aggregates? Cause I don't.
21:47:53	efried	https://awwapp.com/b/umrhofjje/
21:49:41	edleafe	efried: so we would use aggs to denote a NUMA node?
21:50:04	efried	aggzactly.
21:50:13	edleafe	ugh
21:51:50	efried	I don't know how we get anti-affinity with this, though. Anything down the anti_affine_resources thought path is bringing my head crashing against a brick wall.
21:51:50	edleafe	you'd probably have to extend the aggs to include the compute node, no?
21:52:03	efried	The least of the problems being the lack of symmetry.
21:52:07	sean-k-mooney	efried: not very well. the distance along the tree i understand. all inventoies of the same resouce provider have a distance of 0, parent/direct chile is distance or 1 and so on
21:52:13	efried	edleafe: I don't think so, in this case, since the compute node isn't providing any resources.
21:52:51	edleafe	efried: I was just thinking of the eventual monster SQL that would have to return the CN
21:53:45	efried	edleafe: That's another topic, but IMO we could be doing stuff with way simpler (though possibly less efficient) SQL that would be way more flexible and provably correct.
21:54:36	openstackgerrit	Matt Riedemann proposed openstack/nova master: Address comments from I51adbbdf13711e463b4d25c2ffd4a3123cd65675 https://review.openstack.org/541455
21:54:52	efried	sean-k-mooney: Right. And I suppose as long as the model is somewhat symmetrical (that is, all providers of a given resource are always at the same level of the tree - which shouldn't be too hard to encourage) and mutually understood by the dev and the op for a given driver (which pretty much has to be the case anyway) it could probably be made to work.
21:55:21	sean-k-mooney	edleafe: well if recall correctly jay did not want this to be handeled in the sql code but rater in a filter due to the complexity of the sql
21:56:33	sean-k-mooney	that said perhaps that was just for numa and not distance in general
21:59:55	sean-k-mooney	if there is a relitivly effcient way to handel numa/distance in sql without lots of complexity and ram the sql would certenly be better.
22:00:37	sean-k-mooney	actuly when i said filter i think i should have said python e.g. placement would still do the filting not the nova filters
22:01:55	efried	sean-k-mooney: I'm on board with that.
22:02:34	openstackgerrit	melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442
22:02:47	efried	Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it.
22:04:28	edleafe	efried: 100% on the non-SQL processing
22:04:54	edleafe	efried: I think Jay likes the challenge. :)
22:05:33	efried	No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff.
22:05:37	sean-k-mooney	edleafe: perhaps buy would you like the code review :)
22:06:05	edleafe	I'd rather review python than sqla
22:06:10	efried	++
22:07:44	efried	Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky.
22:07:58	edleafe	I also think that subtle bugs are more readily spotted in code
22:08:51	edleafe	I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement
22:09:04	efried	...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper.
22:09:33	edleafe	at some point you have to specify NUMA in the request, no?
22:09:56	efried	Well, no.
22:10:33	efried	With the model as I've described it, you would never have to say NUMA in any context.
22:10:43	efried	You just have to say "keep these resources together"
22:11:00	efried	and placement understands what you mean by "together" (because that's how we design it)
22:11:07	edleafe	efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM
22:11:36	efried	edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree.
22:12:15	efried	edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise.
22:12:30	edleafe	efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node.
22:12:57	efried	edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node>
22:13:27	edleafe	efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways.
22:14:05	edleafe	efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned?
22:14:21	sean-k-mooney	edleafe: yes it they can. i know more about this topic then is good for my health
22:15:04	sean-k-mooney	edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node
22:15:07	efried	edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait.
22:15:20	edleafe	ah good. So efried, just direct your questions to sean-k-mooney
22:15:32	sean-k-mooney	so all compute nodes always have at least 1 numa node
22:16:29	efried	Okay, so the only way to get non
22:16:47	efried	non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them.
22:17:05	efried	In which case we *don't* need the trait.
22:17:11	sean-k-mooney	efried: and if you incldue a distance to restict it
22:18:07	sean-k-mooney	if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes
22:18:17	sean-k-mooney	your performace will be less but it will still work
22:18:37	sean-k-mooney	and you never asked for any afintiy so it fine
22:21:11	efried	Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity.

Earlier   Later