Posted	Nick	Remark
#openstack-nova - 2018-02-06
21:39:12	sean-k-mooney	*can't have traits
21:39:17	efried	So root RP is the compute node. It provides no resources. It has two child RPs, representing NUMA nodes. They provide VCPU and memory inventory. Then those guys have child RPs representing network devices, providing VF inventory.
21:39:34	edleafe	efried: one thing we learned back in Atlanta was that this stuff was much better discussed with a drawing surface so that all could see the relationships
21:40:04	edleafe	Those of us who think visually need that for anything other than the trivial stuff
21:40:22	mriedem	https://awwapp.com/# ?
21:40:25	efried	sean-k-mooney: Yeah, that would be another way to do it: model the NUMA provider as an empty child of the empty root compute host and give the NUMA provider three grandchild tiers which provide CPU, memory, and VFs.
21:40:48	edleafe	mriedem: sure. efried, have at it!
21:40:48	efried	But not to get hung up on those details.
21:41:07	mriedem	or https://webwhiteboard.com/
21:41:15	edleafe	And I'm still not clear how "distance" figures into any of this
21:41:16	mriedem	i won't be joining, but just being super helpful
21:41:51	efried	edleafe: It doesn't. This is purely one-dimensional affinity. Either they're in the same agg or they're not.
21:42:17	edleafe	efried: ok, that's where I was not clear
21:43:18	edleafe	efried: won't the current NRP stuff handle this without additional work? IOW, the CPU,RAM,VF all have to
21:43:31	edleafe	be from the same provider
21:43:41	efried	Only if we can model them as belonging to the same provider.
21:43:59	sean-k-mooney	edleafe: distance was discused in 2 forms. 1 a distance in nodes along the provider tree and 2 a distence to a shared resouce provider and an aggreate member to model things like network latency for share storage
21:44:00	efried	Which we can do for just CPU/memory, but not for VFs.
21:44:31	efried	And also splitting things up more allows us to tie traits more specifically to their resources, if that matters.
21:44:31	edleafe	efried: yeah, I guess it could think of the compute node as the root provider, and then stuff from different NUMA nodes would match
21:45:50	edleafe	sean-k-mooney: I remember the shared distance thing. I thought that was discarded as not needed, because you could just create tighter aggregates so that only "near" storage would ever be used
21:46:11	efried	sean-k-mooney: I tried thinking about distance along a tree, but kept painting myself into a corner where that would severely restrict how things would have to be modeled.
21:46:38	sean-k-mooney	edleafe: yes proably its been a while since denver
21:46:39	efried	sean-k-mooney: Do you understand the concept of distance as relates to aggregates? Cause I don't.
21:47:53	efried	https://awwapp.com/b/umrhofjje/
21:49:41	edleafe	efried: so we would use aggs to denote a NUMA node?
21:50:04	efried	aggzactly.
21:50:13	edleafe	ugh
21:51:50	efried	I don't know how we get anti-affinity with this, though. Anything down the anti_affine_resources thought path is bringing my head crashing against a brick wall.
21:51:50	edleafe	you'd probably have to extend the aggs to include the compute node, no?
21:52:03	efried	The least of the problems being the lack of symmetry.
21:52:07	sean-k-mooney	efried: not very well. the distance along the tree i understand. all inventoies of the same resouce provider have a distance of 0, parent/direct chile is distance or 1 and so on
21:52:13	efried	edleafe: I don't think so, in this case, since the compute node isn't providing any resources.
21:52:51	edleafe	efried: I was just thinking of the eventual monster SQL that would have to return the CN
21:53:45	efried	edleafe: That's another topic, but IMO we could be doing stuff with way simpler (though possibly less efficient) SQL that would be way more flexible and provably correct.
21:54:36	openstackgerrit	Matt Riedemann proposed openstack/nova master: Address comments from I51adbbdf13711e463b4d25c2ffd4a3123cd65675 https://review.openstack.org/541455
21:54:52	efried	sean-k-mooney: Right. And I suppose as long as the model is somewhat symmetrical (that is, all providers of a given resource are always at the same level of the tree - which shouldn't be too hard to encourage) and mutually understood by the dev and the op for a given driver (which pretty much has to be the case anyway) it could probably be made to work.
21:55:21	sean-k-mooney	edleafe: well if recall correctly jay did not want this to be handeled in the sql code but rater in a filter due to the complexity of the sql
21:56:33	sean-k-mooney	that said perhaps that was just for numa and not distance in general
21:59:55	sean-k-mooney	if there is a relitivly effcient way to handel numa/distance in sql without lots of complexity and ram the sql would certenly be better.
22:00:37	sean-k-mooney	actuly when i said filter i think i should have said python e.g. placement would still do the filting not the nova filters
22:01:55	efried	sean-k-mooney: I'm on board with that.
22:02:34	openstackgerrit	melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442
22:02:47	efried	Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it.
22:04:28	edleafe	efried: 100% on the non-SQL processing
22:04:54	edleafe	efried: I think Jay likes the challenge. :)
22:05:33	efried	No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff.
22:05:37	sean-k-mooney	edleafe: perhaps buy would you like the code review :)
22:06:05	edleafe	I'd rather review python than sqla
22:06:10	efried	++
22:07:44	efried	Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky.
22:07:58	edleafe	I also think that subtle bugs are more readily spotted in code
22:08:51	edleafe	I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement
22:09:04	efried	...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper.
22:09:33	edleafe	at some point you have to specify NUMA in the request, no?
22:09:56	efried	Well, no.
22:10:33	efried	With the model as I've described it, you would never have to say NUMA in any context.
22:10:43	efried	You just have to say "keep these resources together"
22:11:00	efried	and placement understands what you mean by "together" (because that's how we design it)
22:11:07	edleafe	efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM
22:11:36	efried	edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree.
22:12:15	efried	edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise.
22:12:30	edleafe	efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node.
22:12:57	efried	edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node>
22:13:27	edleafe	efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways.
22:14:05	edleafe	efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned?
22:14:21	sean-k-mooney	edleafe: yes it they can. i know more about this topic then is good for my health
22:15:04	sean-k-mooney	edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node
22:15:07	efried	edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait.
22:15:20	edleafe	ah good. So efried, just direct your questions to sean-k-mooney
22:15:32	sean-k-mooney	so all compute nodes always have at least 1 numa node
22:16:29	efried	Okay, so the only way to get non
22:16:47	efried	non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them.
22:17:05	efried	In which case we *don't* need the trait.
22:17:11	sean-k-mooney	efried: and if you incldue a distance to restict it
22:18:07	sean-k-mooney	if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes
22:18:17	sean-k-mooney	your performace will be less but it will still work
22:18:37	sean-k-mooney	and you never asked for any afintiy so it fine
22:21:11	efried	Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity.
22:21:55	openstackgerrit	Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474
22:22:03	sean-k-mooney	efried: if you dont have to deal with aggreates then i think it will work not sure how distance is compute for aggregates however.
22:22:40	efried	sean-k-mooney: I'm talking about using one aggregate per NUMA node (so nothing to do with distance).
22:23:03	sean-k-mooney	when modeling afinity via distance affinty becoures deistnace < X and anti afinity is distance > X
22:23:48	sean-k-mooney	efried: oh then ya if numa nodes are aggrages then that is different
22:23:50	efried	Yeah, I get that, but *modeling* and *expressing* distance seems really complicated for people to understand.
22:24:10	cfriesen	efried: one aggregate per numa node? would you then associate a pool of RAM, CPUs, PCI devices, etc, with that aggregate?
22:24:48	cfriesen	at probably 2-4 numa nodes per compute node that seems like a lot of aggregates
22:24:55	sean-k-mooney	you would make the cpus/ram/pci devices resouce providers members of the aggregate
22:25:03	efried	cfriesen: You associate the resource *providers* with the aggregate. So whatever providers provide the resources associated with your NUMA node.
22:25:06	efried	yeah, what sean said.
22:25:18	efried	No, it would be one aggregate per NUMA node, period.
22:25:39	sean-k-mooney	efried: not one aggregate per numa node per host?
22:25:51	efried	sean-k-mooney: Yes.
22:26:02	efried	One aggregate per NUMA node, in the global sense :)
22:26:24	sean-k-mooney	oh ame that aggreate will be am large
22:26:35	efried	Is that "a lot" of aggregates? Meh. Aggregates are cheap.
22:27:16	efried	I feel like we're not talking in the same terms. Here: If you have 10 hosts, and each host has 2 NUMA nodes, you have 20 aggregates total.
22:27:18	sean-k-mooney	i think i would prefer lots of small aggregate then a few giant cluster wide ones
22:27:58	sean-k-mooney	efried: ah ok that what i was thinking we should have if we used aggregates so im cool with that
22:29:04	sean-k-mooney	we dont have traits on aggregates but at least for numa i dont think we need them on teh aggreages so it could work

Earlier   Later