Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
21:40:25 efried sean-k-mooney: Yeah, that would be another way to do it: model the NUMA provider as an empty child of the empty root compute host and give the NUMA provider three grandchild tiers which provide CPU, memory, and VFs.
21:40:48 edleafe mriedem: sure. efried, have at it!
21:40:48 efried But not to get hung up on those details.
21:41:07 mriedem or https://webwhiteboard.com/
21:41:15 edleafe And I'm still not clear how "distance" figures into any of this
21:41:16 mriedem i won't be joining, but just being super helpful
21:41:51 efried edleafe: It doesn't. This is purely one-dimensional affinity. Either they're in the same agg or they're not.
21:42:17 edleafe efried: ok, that's where I was not clear
21:43:18 edleafe efried: won't the current NRP stuff handle this without additional work? IOW, the CPU,RAM,VF all have to
21:43:31 edleafe be from the same provider
21:43:41 efried Only if we can model them as belonging to the same provider.
21:43:59 sean-k-mooney edleafe: distance was discused in 2 forms. 1 a distance in nodes along the provider tree and 2 a distence to a shared resouce provider and an aggreate member to model things like network latency for share storage
21:44:00 efried Which we can do for just CPU/memory, but not for VFs.
21:44:31 efried And also splitting things up more allows us to tie traits more specifically to their resources, if that matters.
21:44:31 edleafe efried: yeah, I guess it could think of the compute node as the root provider, and then stuff from different NUMA nodes would match
21:45:50 edleafe sean-k-mooney: I remember the shared distance thing. I thought that was discarded as not needed, because you could just create tighter aggregates so that only "near" storage would ever be used
21:46:11 efried sean-k-mooney: I tried thinking about distance along a tree, but kept painting myself into a corner where that would severely restrict how things would have to be modeled.
21:46:38 sean-k-mooney edleafe: yes proably its been a while since denver
21:46:39 efried sean-k-mooney: Do you understand the concept of distance as relates to aggregates? Cause I don't.
21:47:53 efried https://awwapp.com/b/umrhofjje/
21:49:41 edleafe efried: so we would use aggs to denote a NUMA node?
21:50:04 efried aggzactly.
21:50:13 edleafe ugh
21:51:50 efried I don't know how we get anti-affinity with this, though. Anything down the anti_affine_resources thought path is bringing my head crashing against a brick wall.
21:51:50 edleafe you'd probably have to extend the aggs to include the compute node, no?
21:52:03 efried The least of the problems being the lack of symmetry.
21:52:07 sean-k-mooney efried: not very well. the distance along the tree i understand. all inventoies of the same resouce provider have a distance of 0, parent/direct chile is distance or 1 and so on
21:52:13 efried edleafe: I don't think so, in this case, since the compute node isn't providing any resources.
21:52:51 edleafe efried: I was just thinking of the eventual monster SQL that would have to return the CN
21:53:45 efried edleafe: That's another topic, but IMO we could be doing stuff with way simpler (though possibly less efficient) SQL that would be way more flexible and provably correct.
21:54:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Address comments from I51adbbdf13711e463b4d25c2ffd4a3123cd65675 https://review.openstack.org/541455
21:54:52 efried sean-k-mooney: Right. And I suppose as long as the model is somewhat symmetrical (that is, all providers of a given resource are always at the same level of the tree - which shouldn't be too hard to encourage) and mutually understood by the dev and the op for a given driver (which pretty much has to be the case anyway) it could probably be made to work.
21:55:21 sean-k-mooney edleafe: well if recall correctly jay did not want this to be handeled in the sql code but rater in a filter due to the complexity of the sql
21:56:33 sean-k-mooney that said perhaps that was just for numa and not distance in general
21:59:55 sean-k-mooney if there is a relitivly effcient way to handel numa/distance in sql without lots of complexity and ram the sql would certenly be better.
22:00:37 sean-k-mooney actuly when i said filter i think i should have said python e.g. placement would still do the filting not the nova filters
22:01:55 efried sean-k-mooney: I'm on board with that.
22:02:34 openstackgerrit melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442
22:02:47 efried Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it.
22:04:28 edleafe efried: 100% on the non-SQL processing
22:04:54 edleafe efried: I think Jay likes the challenge. :)
22:05:33 efried No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff.
22:05:37 sean-k-mooney edleafe: perhaps buy would you like the code review :)
22:06:05 edleafe I'd rather review python than sqla
22:06:10 efried ++
22:07:44 efried Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky.
22:07:58 edleafe I also think that subtle bugs are more readily spotted in code
22:08:51 edleafe I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement
22:09:04 efried ...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper.
22:09:33 edleafe at some point you have to specify NUMA in the request, no?
22:09:56 efried Well, no.
22:10:33 efried With the model as I've described it, you would never have to say NUMA in any context.
22:10:43 efried You just have to say "keep these resources together"
22:11:00 efried and placement understands what you mean by "together" (because that's how we design it)
22:11:07 edleafe efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM
22:11:36 efried edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree.
22:12:15 efried edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise.
22:12:30 edleafe efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node.
22:12:57 efried edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node>
22:13:27 edleafe efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways.
22:14:05 edleafe efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned?
22:14:21 sean-k-mooney edleafe: yes it they can. i know more about this topic then is good for my health
22:15:04 sean-k-mooney edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node
22:15:07 efried edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait.
22:15:20 edleafe ah good. So efried, just direct your questions to sean-k-mooney
22:15:32 sean-k-mooney so all compute nodes always have at least 1 numa node
22:16:29 efried Okay, so the only way to get non
22:16:47 efried non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them.
22:17:05 efried In which case we *don't* need the trait.
22:17:11 sean-k-mooney efried: and if you incldue a distance to restict it
22:18:07 sean-k-mooney if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes
22:18:17 sean-k-mooney your performace will be less but it will still work
22:18:37 sean-k-mooney and you never asked for any afintiy so it fine
22:21:11 efried Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity.
22:21:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474
22:22:03 sean-k-mooney efried: if you dont have to deal with aggreates then i think it will work not sure how distance is compute for aggregates however.
22:22:40 efried sean-k-mooney: I'm talking about using one aggregate per NUMA node (so nothing to do with distance).
22:23:03 sean-k-mooney when modeling afinity via distance affinty becoures deistnace < X and anti afinity is distance > X
22:23:48 sean-k-mooney efried: oh then ya if numa nodes are aggrages then that is different
22:23:50 efried Yeah, I get that, but *modeling* and *expressing* distance seems really complicated for people to understand.
22:24:10 cfriesen efried: one aggregate per numa node? would you then associate a pool of RAM, CPUs, PCI devices, etc, with that aggregate?
22:24:48 cfriesen at probably 2-4 numa nodes per compute node that seems like a lot of aggregates
22:24:55 sean-k-mooney you would make the cpus/ram/pci devices resouce providers members of the aggregate
22:25:03 efried cfriesen: You associate the resource *providers* with the aggregate. So whatever providers provide the resources associated with your NUMA node.
22:25:06 efried yeah, what sean said.
22:25:18 efried No, it would be one aggregate per NUMA node, period.
22:25:39 sean-k-mooney efried: not one aggregate per numa node per host?
22:25:51 efried sean-k-mooney: Yes.
22:26:02 efried One aggregate per NUMA node, in the global sense :)
22:26:24 sean-k-mooney oh ame that aggreate will be am large
22:26:35 efried Is that "a lot" of aggregates? Meh. Aggregates are cheap.
22:27:16 efried I feel like we're not talking in the same terms. Here: If you have 10 hosts, and each host has 2 NUMA nodes, you have 20 aggregates total.
22:27:18 sean-k-mooney i think i would prefer lots of small aggregate then a few giant cluster wide ones
22:27:58 sean-k-mooney efried: ah ok that what i was thinking we should have if we used aggregates so im cool with that
22:29:04 sean-k-mooney we dont have traits on aggregates but at least for numa i dont think we need them on teh aggreages so it could work
22:29:21 cfriesen so you'd have these per-numa-node aggregates for system use, then per-host aggregates to match against flavor/image properties and act as availability zones?
22:29:44 efried sean-k-mooney: We can always stamp the involved providers with traits. But not sure we need 'em. At least to mark NUMA nodes.
22:30:02 efried cfriesen: I know nothing about host aggregates or availability zones.
22:30:16 sean-k-mooney cfriesen: well an aggreaget is just a bag of resouce providers nothing more or less. you can use them to group anything you want
22:30:18 efried cfriesen: Except that placement aggregates != host aggregates

Earlier   Later