Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
21:31:29 efried edleafe: So within the unnumbered request group, the API would force the resources from classes you list to come from either the same provider, or the same aggregate.
21:32:19 efried edleafe: I'm not sure an anti_affine_resources makes any sense, but could be added if it does.
21:32:46 edleafe efried: "affine"?
21:33:12 edleafe efried: How would that specify RAM, PCI deveice, etc., associated with a given NUMA?
21:33:33 efried edleafe: Okay, let me give a more complete example, make more sense.
21:34:29 sean-k-mooney edleafe: anti_affine_resources for nic bonding e.g. antiafintiy of the pf for the VF request
21:34:40 sean-k-mooney efried: ^
21:34:44 efried sean-k-mooney: ack
21:35:20 efried edleafe: To start with, model NUMA nodes as subtrees under the root compute host RP (which may not in fact provide any resources). So maybe the first tier children provide CPU & memory, and the second tier (grandchildren) are the network devices providing VFs.
21:35:53 efried edleafe: Associate each child/grandchildren in a given NUMA node with the same aggregate UUID.
21:36:49 efried edleafe: Then your request would be like GET /allocation_candidates?resources=VCPU:2,MEMORY_MB:2048,SRIOV_NET_VF:1,DISK_GB=2048&affine_resources=VCPU,MEMORY_MB,SRIOV_NET_VF
21:37:03 edleafe why can't the network devices be children of the NUMA node?
21:37:11 efried They are
21:37:14 efried swhat I said
21:37:33 efried Sorry, to rephrase:
21:37:36 sean-k-mooney efried: but the cpu and memory shoudl also be childeren of the numa node too
21:37:51 edleafe "and the second tier (grandchildren) are the network devices providing VFs."
21:37:53 efried sean-k-mooney: The CPU/mem are resources provided by the numa node which is the RP.
21:38:47 sean-k-mooney efried: ah they are inventories of the numa node rather then sub resouce providres but that means you can have traints on the cpu and ram
21:39:12 sean-k-mooney *can't have traits
21:39:17 efried So root RP is the compute node. It provides no resources. It has two child RPs, representing NUMA nodes. They provide VCPU and memory inventory. Then those guys have child RPs representing network devices, providing VF inventory.
21:39:34 edleafe efried: one thing we learned back in Atlanta was that this stuff was much better discussed with a drawing surface so that all could see the relationships
21:40:04 edleafe Those of us who think visually need that for anything other than the trivial stuff
21:40:22 mriedem https://awwapp.com/# ?
21:40:25 efried sean-k-mooney: Yeah, that would be another way to do it: model the NUMA provider as an empty child of the empty root compute host and give the NUMA provider three grandchild tiers which provide CPU, memory, and VFs.
21:40:48 edleafe mriedem: sure. efried, have at it!
21:40:48 efried But not to get hung up on those details.
21:41:07 mriedem or https://webwhiteboard.com/
21:41:15 edleafe And I'm still not clear how "distance" figures into any of this
21:41:16 mriedem i won't be joining, but just being super helpful
21:41:51 efried edleafe: It doesn't. This is purely one-dimensional affinity. Either they're in the same agg or they're not.
21:42:17 edleafe efried: ok, that's where I was not clear
21:43:18 edleafe efried: won't the current NRP stuff handle this without additional work? IOW, the CPU,RAM,VF all have to
21:43:31 edleafe be from the same provider
21:43:41 efried Only if we can model them as belonging to the same provider.
21:43:59 sean-k-mooney edleafe: distance was discused in 2 forms. 1 a distance in nodes along the provider tree and 2 a distence to a shared resouce provider and an aggreate member to model things like network latency for share storage
21:44:00 efried Which we can do for just CPU/memory, but not for VFs.
21:44:31 efried And also splitting things up more allows us to tie traits more specifically to their resources, if that matters.
21:44:31 edleafe efried: yeah, I guess it could think of the compute node as the root provider, and then stuff from different NUMA nodes would match
21:45:50 edleafe sean-k-mooney: I remember the shared distance thing. I thought that was discarded as not needed, because you could just create tighter aggregates so that only "near" storage would ever be used
21:46:11 efried sean-k-mooney: I tried thinking about distance along a tree, but kept painting myself into a corner where that would severely restrict how things would have to be modeled.
21:46:38 sean-k-mooney edleafe: yes proably its been a while since denver
21:46:39 efried sean-k-mooney: Do you understand the concept of distance as relates to aggregates? Cause I don't.
21:47:53 efried https://awwapp.com/b/umrhofjje/
21:49:41 edleafe efried: so we would use aggs to denote a NUMA node?
21:50:04 efried aggzactly.
21:50:13 edleafe ugh
21:51:50 efried I don't know how we get anti-affinity with this, though. Anything down the anti_affine_resources thought path is bringing my head crashing against a brick wall.
21:51:50 edleafe you'd probably have to extend the aggs to include the compute node, no?
21:52:03 efried The least of the problems being the lack of symmetry.
21:52:07 sean-k-mooney efried: not very well. the distance along the tree i understand. all inventoies of the same resouce provider have a distance of 0, parent/direct chile is distance or 1 and so on
21:52:13 efried edleafe: I don't think so, in this case, since the compute node isn't providing any resources.
21:52:51 edleafe efried: I was just thinking of the eventual monster SQL that would have to return the CN
21:53:45 efried edleafe: That's another topic, but IMO we could be doing stuff with way simpler (though possibly less efficient) SQL that would be way more flexible and provably correct.
21:54:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Address comments from I51adbbdf13711e463b4d25c2ffd4a3123cd65675 https://review.openstack.org/541455
21:54:52 efried sean-k-mooney: Right. And I suppose as long as the model is somewhat symmetrical (that is, all providers of a given resource are always at the same level of the tree - which shouldn't be too hard to encourage) and mutually understood by the dev and the op for a given driver (which pretty much has to be the case anyway) it could probably be made to work.
21:55:21 sean-k-mooney edleafe: well if recall correctly jay did not want this to be handeled in the sql code but rater in a filter due to the complexity of the sql
21:56:33 sean-k-mooney that said perhaps that was just for numa and not distance in general
21:59:55 sean-k-mooney if there is a relitivly effcient way to handel numa/distance in sql without lots of complexity and ram the sql would certenly be better.
22:00:37 sean-k-mooney actuly when i said filter i think i should have said python e.g. placement would still do the filting not the nova filters
22:01:55 efried sean-k-mooney: I'm on board with that.
22:02:34 openstackgerrit melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442
22:02:47 efried Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it.
22:04:28 edleafe efried: 100% on the non-SQL processing
22:04:54 edleafe efried: I think Jay likes the challenge. :)
22:05:33 efried No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff.
22:05:37 sean-k-mooney edleafe: perhaps buy would you like the code review :)
22:06:05 edleafe I'd rather review python than sqla
22:06:10 efried ++
22:07:44 efried Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky.
22:07:58 edleafe I also think that subtle bugs are more readily spotted in code
22:08:51 edleafe I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement
22:09:04 efried ...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper.
22:09:33 edleafe at some point you have to specify NUMA in the request, no?
22:09:56 efried Well, no.
22:10:33 efried With the model as I've described it, you would never have to say NUMA in any context.
22:10:43 efried You just have to say "keep these resources together"
22:11:00 efried and placement understands what you mean by "together" (because that's how we design it)
22:11:07 edleafe efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM
22:11:36 efried edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree.
22:12:15 efried edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise.
22:12:30 edleafe efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node.
22:12:57 efried edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node>
22:13:27 edleafe efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways.
22:14:05 edleafe efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned?
22:14:21 sean-k-mooney edleafe: yes it they can. i know more about this topic then is good for my health
22:15:04 sean-k-mooney edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node
22:15:07 efried edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait.
22:15:20 edleafe ah good. So efried, just direct your questions to sean-k-mooney
22:15:32 sean-k-mooney so all compute nodes always have at least 1 numa node
22:16:29 efried Okay, so the only way to get non
22:16:47 efried non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them.
22:17:05 efried In which case we *don't* need the trait.
22:17:11 sean-k-mooney efried: and if you incldue a distance to restict it
22:18:07 sean-k-mooney if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes
22:18:17 sean-k-mooney your performace will be less but it will still work
22:18:37 sean-k-mooney and you never asked for any afintiy so it fine
22:21:11 efried Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity.
22:21:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474
22:22:03 sean-k-mooney efried: if you dont have to deal with aggreates then i think it will work not sure how distance is compute for aggregates however.

Earlier   Later