| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 21:41:16 | mriedem | i won't be joining, but just being super helpful | |
| 21:41:51 | efried | edleafe: It doesn't. This is purely one-dimensional affinity. Either they're in the same agg or they're not. | |
| 21:42:17 | edleafe | efried: ok, that's where I was not clear | |
| 21:43:18 | edleafe | efried: won't the current NRP stuff handle this without additional work? IOW, the CPU,RAM,VF all have to | |
| 21:43:31 | edleafe | be from the same provider | |
| 21:43:41 | efried | Only if we can model them as belonging to the same provider. | |
| 21:43:59 | sean-k-mooney | edleafe: distance was discused in 2 forms. 1 a distance in nodes along the provider tree and 2 a distence to a shared resouce provider and an aggreate member to model things like network latency for share storage | |
| 21:44:00 | efried | Which we can do for just CPU/memory, but not for VFs. | |
| 21:44:31 | efried | And also splitting things up more allows us to tie traits more specifically to their resources, if that matters. | |
| 21:44:31 | edleafe | efried: yeah, I guess it could think of the compute node as the root provider, and then stuff from different NUMA nodes would match | |
| 21:45:50 | edleafe | sean-k-mooney: I remember the shared distance thing. I thought that was discarded as not needed, because you could just create tighter aggregates so that only "near" storage would ever be used | |
| 21:46:11 | efried | sean-k-mooney: I tried thinking about distance along a tree, but kept painting myself into a corner where that would severely restrict how things would have to be modeled. | |
| 21:46:38 | sean-k-mooney | edleafe: yes proably its been a while since denver | |
| 21:46:39 | efried | sean-k-mooney: Do you understand the concept of distance as relates to aggregates? Cause I don't. | |
| 21:47:53 | efried | https://awwapp.com/b/umrhofjje/ | |
| 21:49:41 | edleafe | efried: so we would use aggs to denote a NUMA node? | |
| 21:50:04 | efried | aggzactly. | |
| 21:50:13 | edleafe | ugh | |
| 21:51:50 | efried | I don't know how we get anti-affinity with this, though. Anything down the anti_affine_resources thought path is bringing my head crashing against a brick wall. | |
| 21:51:50 | edleafe | you'd probably have to extend the aggs to include the compute node, no? | |
| 21:52:03 | efried | The least of the problems being the lack of symmetry. | |
| 21:52:07 | sean-k-mooney | efried: not very well. the distance along the tree i understand. all inventoies of the same resouce provider have a distance of 0, parent/direct chile is distance or 1 and so on | |
| 21:52:13 | efried | edleafe: I don't think so, in this case, since the compute node isn't providing any resources. | |
| 21:52:51 | edleafe | efried: I was just thinking of the eventual monster SQL that would have to return the CN | |
| 21:53:45 | efried | edleafe: That's another topic, but IMO we could be doing stuff with way simpler (though possibly less efficient) SQL that would be way more flexible and provably correct. | |
| 21:54:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Address comments from I51adbbdf13711e463b4d25c2ffd4a3123cd65675 https://review.openstack.org/541455 | |
| 21:54:52 | efried | sean-k-mooney: Right. And I suppose as long as the model is somewhat symmetrical (that is, all providers of a given resource are always at the same level of the tree - which shouldn't be too hard to encourage) and mutually understood by the dev and the op for a given driver (which pretty much has to be the case anyway) it could probably be made to work. | |
| 21:55:21 | sean-k-mooney | edleafe: well if recall correctly jay did not want this to be handeled in the sql code but rater in a filter due to the complexity of the sql | |
| 21:56:33 | sean-k-mooney | that said perhaps that was just for numa and not distance in general | |
| 21:59:55 | sean-k-mooney | if there is a relitivly effcient way to handel numa/distance in sql without lots of complexity and ram the sql would certenly be better. | |
| 22:00:37 | sean-k-mooney | actuly when i said filter i think i should have said python e.g. placement would still do the filting not the nova filters | |
| 22:01:55 | efried | sean-k-mooney: I'm on board with that. | |
| 22:02:34 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442 | |
| 22:02:47 | efried | Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it. | |
| 22:04:28 | edleafe | efried: 100% on the non-SQL processing | |
| 22:04:54 | edleafe | efried: I think Jay likes the challenge. :) | |
| 22:05:33 | efried | No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff. | |
| 22:05:37 | sean-k-mooney | edleafe: perhaps buy would you like the code review :) | |
| 22:06:05 | edleafe | I'd rather review python than sqla | |
| 22:06:10 | efried | ++ | |
| 22:07:44 | efried | Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky. | |
| 22:07:58 | edleafe | I also think that subtle bugs are more readily spotted in code | |
| 22:08:51 | edleafe | I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement | |
| 22:09:04 | efried | ...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper. | |
| 22:09:33 | edleafe | at some point you have to specify NUMA in the request, no? | |
| 22:09:56 | efried | Well, no. | |
| 22:10:33 | efried | With the model as I've described it, you would never have to say NUMA in any context. | |
| 22:10:43 | efried | You just have to say "keep these resources together" | |
| 22:11:00 | efried | and placement understands what you mean by "together" (because that's how we design it) | |
| 22:11:07 | edleafe | efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM | |
| 22:11:36 | efried | edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree. | |
| 22:12:15 | efried | edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise. | |
| 22:12:30 | edleafe | efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node. | |
| 22:12:57 | efried | edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node> | |
| 22:13:27 | edleafe | efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways. | |
| 22:14:05 | edleafe | efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned? | |
| 22:14:21 | sean-k-mooney | edleafe: yes it they can. i know more about this topic then is good for my health | |
| 22:15:04 | sean-k-mooney | edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node | |
| 22:15:07 | efried | edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait. | |
| 22:15:20 | edleafe | ah good. So efried, just direct your questions to sean-k-mooney | |
| 22:15:32 | sean-k-mooney | so all compute nodes always have at least 1 numa node | |
| 22:16:29 | efried | Okay, so the only way to get non | |
| 22:16:47 | efried | non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them. | |
| 22:17:05 | efried | In which case we *don't* need the trait. | |
| 22:17:11 | sean-k-mooney | efried: and if you incldue a distance to restict it | |
| 22:18:07 | sean-k-mooney | if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes | |
| 22:18:17 | sean-k-mooney | your performace will be less but it will still work | |
| 22:18:37 | sean-k-mooney | and you never asked for any afintiy so it fine | |
| 22:21:11 | efried | Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity. | |
| 22:21:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474 | |
| 22:22:03 | sean-k-mooney | efried: if you dont have to deal with aggreates then i think it will work not sure how distance is compute for aggregates however. | |
| 22:22:40 | efried | sean-k-mooney: I'm talking about using one aggregate per NUMA node (so nothing to do with distance). | |
| 22:23:03 | sean-k-mooney | when modeling afinity via distance affinty becoures deistnace < X and anti afinity is distance > X | |
| 22:23:48 | sean-k-mooney | efried: oh then ya if numa nodes are aggrages then that is different | |
| 22:23:50 | efried | Yeah, I get that, but *modeling* and *expressing* distance seems really complicated for people to understand. | |
| 22:24:10 | cfriesen | efried: one aggregate per numa node? would you then associate a pool of RAM, CPUs, PCI devices, etc, with that aggregate? | |
| 22:24:48 | cfriesen | at probably 2-4 numa nodes per compute node that seems like a lot of aggregates | |
| 22:24:55 | sean-k-mooney | you would make the cpus/ram/pci devices resouce providers members of the aggregate | |
| 22:25:03 | efried | cfriesen: You associate the resource *providers* with the aggregate. So whatever providers provide the resources associated with your NUMA node. | |
| 22:25:06 | efried | yeah, what sean said. | |
| 22:25:18 | efried | No, it would be one aggregate per NUMA node, period. | |
| 22:25:39 | sean-k-mooney | efried: not one aggregate per numa node per host? | |
| 22:25:51 | efried | sean-k-mooney: Yes. | |
| 22:26:02 | efried | One aggregate per NUMA node, in the global sense :) | |
| 22:26:24 | sean-k-mooney | oh ame that aggreate will be am large | |
| 22:26:35 | efried | Is that "a lot" of aggregates? Meh. Aggregates are cheap. | |
| 22:27:16 | efried | I feel like we're not talking in the same terms. Here: If you have 10 hosts, and each host has 2 NUMA nodes, you have 20 aggregates total. | |
| 22:27:18 | sean-k-mooney | i think i would prefer lots of small aggregate then a few giant cluster wide ones | |
| 22:27:58 | sean-k-mooney | efried: ah ok that what i was thinking we should have if we used aggregates so im cool with that | |
| 22:29:04 | sean-k-mooney | we dont have traits on aggregates but at least for numa i dont think we need them on teh aggreages so it could work | |
| 22:29:21 | cfriesen | so you'd have these per-numa-node aggregates for system use, then per-host aggregates to match against flavor/image properties and act as availability zones? | |
| 22:29:44 | efried | sean-k-mooney: We can always stamp the involved providers with traits. But not sure we need 'em. At least to mark NUMA nodes. | |
| 22:30:02 | efried | cfriesen: I know nothing about host aggregates or availability zones. | |
| 22:30:16 | sean-k-mooney | cfriesen: well an aggreaget is just a bag of resouce providers nothing more or less. you can use them to group anything you want | |
| 22:30:18 | efried | cfriesen: Except that placement aggregates != host aggregates | |
| 22:30:39 | efried | cfriesen: The other main use case for aggregates is to associate sharing resource providers, like a shared storage pool. | |
| 22:31:25 | efried | ^ placement aggregates, that is. | |
| 22:32:07 | cfriesen | efried: okay...I'm not up on the placement terminology. :) | |
| 22:32:35 | efried | cfriesen: I'm not up on... pretty much everything else. Between the two of us, we're gonna need a pidgin. | |
| 22:33:22 | sean-k-mooney | cfriesen: a sharing resouce provider is jsut a normal resouce provider with a trait and then its inventores are consumable by anything in the same aggreates as it | |