Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
21:47:53 efried https://awwapp.com/b/umrhofjje/
21:49:41 edleafe efried: so we would use aggs to denote a NUMA node?
21:50:04 efried aggzactly.
21:50:13 edleafe ugh
21:51:50 efried I don't know how we get anti-affinity with this, though. Anything down the anti_affine_resources thought path is bringing my head crashing against a brick wall.
21:51:50 edleafe you'd probably have to extend the aggs to include the compute node, no?
21:52:03 efried The least of the problems being the lack of symmetry.
21:52:07 sean-k-mooney efried: not very well. the distance along the tree i understand. all inventoies of the same resouce provider have a distance of 0, parent/direct chile is distance or 1 and so on
21:52:13 efried edleafe: I don't think so, in this case, since the compute node isn't providing any resources.
21:52:51 edleafe efried: I was just thinking of the eventual monster SQL that would have to return the CN
21:53:45 efried edleafe: That's another topic, but IMO we could be doing stuff with way simpler (though possibly less efficient) SQL that would be way more flexible and provably correct.
21:54:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Address comments from I51adbbdf13711e463b4d25c2ffd4a3123cd65675 https://review.openstack.org/541455
21:54:52 efried sean-k-mooney: Right. And I suppose as long as the model is somewhat symmetrical (that is, all providers of a given resource are always at the same level of the tree - which shouldn't be too hard to encourage) and mutually understood by the dev and the op for a given driver (which pretty much has to be the case anyway) it could probably be made to work.
21:55:21 sean-k-mooney edleafe: well if recall correctly jay did not want this to be handeled in the sql code but rater in a filter due to the complexity of the sql
21:56:33 sean-k-mooney that said perhaps that was just for numa and not distance in general
21:59:55 sean-k-mooney if there is a relitivly effcient way to handel numa/distance in sql without lots of complexity and ram the sql would certenly be better.
22:00:37 sean-k-mooney actuly when i said filter i think i should have said python e.g. placement would still do the filting not the nova filters
22:01:55 efried sean-k-mooney: I'm on board with that.
22:02:34 openstackgerrit melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442
22:02:47 efried Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it.
22:04:28 edleafe efried: 100% on the non-SQL processing
22:04:54 edleafe efried: I think Jay likes the challenge. :)
22:05:33 efried No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff.
22:05:37 sean-k-mooney edleafe: perhaps buy would you like the code review :)
22:06:05 edleafe I'd rather review python than sqla
22:06:10 efried ++
22:07:44 efried Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky.
22:07:58 edleafe I also think that subtle bugs are more readily spotted in code
22:08:51 edleafe I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement
22:09:04 efried ...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper.
22:09:33 edleafe at some point you have to specify NUMA in the request, no?
22:09:56 efried Well, no.
22:10:33 efried With the model as I've described it, you would never have to say NUMA in any context.
22:10:43 efried You just have to say "keep these resources together"
22:11:00 efried and placement understands what you mean by "together" (because that's how we design it)
22:11:07 edleafe efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM
22:11:36 efried edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree.
22:12:15 efried edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise.
22:12:30 edleafe efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node.
22:12:57 efried edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node>
22:13:27 edleafe efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways.
22:14:05 edleafe efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned?
22:14:21 sean-k-mooney edleafe: yes it they can. i know more about this topic then is good for my health
22:15:04 sean-k-mooney edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node
22:15:07 efried edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait.
22:15:20 edleafe ah good. So efried, just direct your questions to sean-k-mooney
22:15:32 sean-k-mooney so all compute nodes always have at least 1 numa node
22:16:29 efried Okay, so the only way to get non
22:16:47 efried non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them.
22:17:05 efried In which case we *don't* need the trait.
22:17:11 sean-k-mooney efried: and if you incldue a distance to restict it
22:18:07 sean-k-mooney if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes
22:18:17 sean-k-mooney your performace will be less but it will still work
22:18:37 sean-k-mooney and you never asked for any afintiy so it fine
22:21:11 efried Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity.
22:21:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474
22:22:03 sean-k-mooney efried: if you dont have to deal with aggreates then i think it will work not sure how distance is compute for aggregates however.
22:22:40 efried sean-k-mooney: I'm talking about using one aggregate per NUMA node (so nothing to do with distance).
22:23:03 sean-k-mooney when modeling afinity via distance affinty becoures deistnace < X and anti afinity is distance > X
22:23:48 sean-k-mooney efried: oh then ya if numa nodes are aggrages then that is different
22:23:50 efried Yeah, I get that, but *modeling* and *expressing* distance seems really complicated for people to understand.
22:24:10 cfriesen efried: one aggregate per numa node? would you then associate a pool of RAM, CPUs, PCI devices, etc, with that aggregate?
22:24:48 cfriesen at probably 2-4 numa nodes per compute node that seems like a lot of aggregates
22:24:55 sean-k-mooney you would make the cpus/ram/pci devices resouce providers members of the aggregate
22:25:03 efried cfriesen: You associate the resource *providers* with the aggregate. So whatever providers provide the resources associated with your NUMA node.
22:25:06 efried yeah, what sean said.
22:25:18 efried No, it would be one aggregate per NUMA node, period.
22:25:39 sean-k-mooney efried: not one aggregate per numa node per host?
22:25:51 efried sean-k-mooney: Yes.
22:26:02 efried One aggregate per NUMA node, in the global sense :)
22:26:24 sean-k-mooney oh ame that aggreate will be am large
22:26:35 efried Is that "a lot" of aggregates? Meh. Aggregates are cheap.
22:27:16 efried I feel like we're not talking in the same terms. Here: If you have 10 hosts, and each host has 2 NUMA nodes, you have 20 aggregates total.
22:27:18 sean-k-mooney i think i would prefer lots of small aggregate then a few giant cluster wide ones
22:27:58 sean-k-mooney efried: ah ok that what i was thinking we should have if we used aggregates so im cool with that
22:29:04 sean-k-mooney we dont have traits on aggregates but at least for numa i dont think we need them on teh aggreages so it could work
22:29:21 cfriesen so you'd have these per-numa-node aggregates for system use, then per-host aggregates to match against flavor/image properties and act as availability zones?
22:29:44 efried sean-k-mooney: We can always stamp the involved providers with traits. But not sure we need 'em. At least to mark NUMA nodes.
22:30:02 efried cfriesen: I know nothing about host aggregates or availability zones.
22:30:16 sean-k-mooney cfriesen: well an aggreaget is just a bag of resouce providers nothing more or less. you can use them to group anything you want
22:30:18 efried cfriesen: Except that placement aggregates != host aggregates
22:30:39 efried cfriesen: The other main use case for aggregates is to associate sharing resource providers, like a shared storage pool.
22:31:25 efried ^ placement aggregates, that is.
22:32:07 cfriesen efried: okay...I'm not up on the placement terminology. :)
22:32:35 efried cfriesen: I'm not up on... pretty much everything else. Between the two of us, we're gonna need a pidgin.
22:33:22 sean-k-mooney cfriesen: a sharing resouce provider is jsut a normal resouce provider with a trait and then its inventores are consumable by anything in the same aggreates as it
22:34:36 cfriesen so how do we handle the idea of an instance consuming cpu/ram from one placement aggregate but a PCI device from another placement aggregate? (for the non-strict PCI affinity case)
22:34:44 openstackgerrit Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160
22:34:49 mriedem melwitt: cleaned that up ^
22:35:28 sean-k-mooney cfriesen: instance consume resocues form invetories provided by resouce providers
22:35:40 melwitt mriedem: thanks. sorry I lost track of that one
22:35:47 sean-k-mooney cfriesen: and resouce provires can be member of 0-n aggregates
22:36:30 sean-k-mooney so you are not consuming form the aggregates
22:37:22 efried cfriesen: In the design I'm strawpersoning, if you care about CPU/mem affinity and you don't care about VF affinity, you would say affine_resources=VCPU,MEMORY_MB but not SRIOV_NET_VF
22:37:46 efried cfriesen: s/VF/PCI device/ if you wish.
22:39:08 sean-k-mooney efried: that may break down a bit with anti afinity
22:39:33 efried sean-k-mooney: I'm saying we're not handling anti-affinity at all with this model.
22:39:34 sean-k-mooney e.g. i want 2 VF whoes parent PF are anti affined
22:39:48 sean-k-mooney efried: ah ok
22:39:52 efried sean-k-mooney: Yeah, I get that. But that's not a NUMA use case, right?

Earlier   Later