| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 22:01:55 | efried | sean-k-mooney: I'm on board with that. | |
| 22:02:34 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442 | |
| 22:02:47 | efried | Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it. | |
| 22:04:28 | edleafe | efried: 100% on the non-SQL processing | |
| 22:04:54 | edleafe | efried: I think Jay likes the challenge. :) | |
| 22:05:33 | efried | No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff. | |
| 22:05:37 | sean-k-mooney | edleafe: perhaps buy would you like the code review :) | |
| 22:06:05 | edleafe | I'd rather review python than sqla | |
| 22:06:10 | efried | ++ | |
| 22:07:44 | efried | Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky. | |
| 22:07:58 | edleafe | I also think that subtle bugs are more readily spotted in code | |
| 22:08:51 | edleafe | I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement | |
| 22:09:04 | efried | ...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper. | |
| 22:09:33 | edleafe | at some point you have to specify NUMA in the request, no? | |
| 22:09:56 | efried | Well, no. | |
| 22:10:33 | efried | With the model as I've described it, you would never have to say NUMA in any context. | |
| 22:10:43 | efried | You just have to say "keep these resources together" | |
| 22:11:00 | efried | and placement understands what you mean by "together" (because that's how we design it) | |
| 22:11:07 | edleafe | efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM | |
| 22:11:36 | efried | edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree. | |
| 22:12:15 | efried | edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise. | |
| 22:12:30 | edleafe | efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node. | |
| 22:12:57 | efried | edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node> | |
| 22:13:27 | edleafe | efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways. | |
| 22:14:05 | edleafe | efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned? | |
| 22:14:21 | sean-k-mooney | edleafe: yes it they can. i know more about this topic then is good for my health | |
| 22:15:04 | sean-k-mooney | edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node | |
| 22:15:07 | efried | edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait. | |
| 22:15:20 | edleafe | ah good. So efried, just direct your questions to sean-k-mooney | |
| 22:15:32 | sean-k-mooney | so all compute nodes always have at least 1 numa node | |
| 22:16:29 | efried | Okay, so the only way to get non | |
| 22:16:47 | efried | non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them. | |
| 22:17:05 | efried | In which case we *don't* need the trait. | |
| 22:17:11 | sean-k-mooney | efried: and if you incldue a distance to restict it | |
| 22:18:07 | sean-k-mooney | if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes | |
| 22:18:17 | sean-k-mooney | your performace will be less but it will still work | |
| 22:18:37 | sean-k-mooney | and you never asked for any afintiy so it fine | |
| 22:21:11 | efried | Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity. | |
| 22:21:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474 | |
| 22:22:03 | sean-k-mooney | efried: if you dont have to deal with aggreates then i think it will work not sure how distance is compute for aggregates however. | |
| 22:22:40 | efried | sean-k-mooney: I'm talking about using one aggregate per NUMA node (so nothing to do with distance). | |
| 22:23:03 | sean-k-mooney | when modeling afinity via distance affinty becoures deistnace < X and anti afinity is distance > X | |
| 22:23:48 | sean-k-mooney | efried: oh then ya if numa nodes are aggrages then that is different | |
| 22:23:50 | efried | Yeah, I get that, but *modeling* and *expressing* distance seems really complicated for people to understand. | |
| 22:24:10 | cfriesen | efried: one aggregate per numa node? would you then associate a pool of RAM, CPUs, PCI devices, etc, with that aggregate? | |
| 22:24:48 | cfriesen | at probably 2-4 numa nodes per compute node that seems like a lot of aggregates | |
| 22:24:55 | sean-k-mooney | you would make the cpus/ram/pci devices resouce providers members of the aggregate | |
| 22:25:03 | efried | cfriesen: You associate the resource *providers* with the aggregate. So whatever providers provide the resources associated with your NUMA node. | |
| 22:25:06 | efried | yeah, what sean said. | |
| 22:25:18 | efried | No, it would be one aggregate per NUMA node, period. | |
| 22:25:39 | sean-k-mooney | efried: not one aggregate per numa node per host? | |
| 22:25:51 | efried | sean-k-mooney: Yes. | |
| 22:26:02 | efried | One aggregate per NUMA node, in the global sense :) | |
| 22:26:24 | sean-k-mooney | oh ame that aggreate will be am large | |
| 22:26:35 | efried | Is that "a lot" of aggregates? Meh. Aggregates are cheap. | |
| 22:27:16 | efried | I feel like we're not talking in the same terms. Here: If you have 10 hosts, and each host has 2 NUMA nodes, you have 20 aggregates total. | |
| 22:27:18 | sean-k-mooney | i think i would prefer lots of small aggregate then a few giant cluster wide ones | |
| 22:27:58 | sean-k-mooney | efried: ah ok that what i was thinking we should have if we used aggregates so im cool with that | |
| 22:29:04 | sean-k-mooney | we dont have traits on aggregates but at least for numa i dont think we need them on teh aggreages so it could work | |
| 22:29:21 | cfriesen | so you'd have these per-numa-node aggregates for system use, then per-host aggregates to match against flavor/image properties and act as availability zones? | |
| 22:29:44 | efried | sean-k-mooney: We can always stamp the involved providers with traits. But not sure we need 'em. At least to mark NUMA nodes. | |
| 22:30:02 | efried | cfriesen: I know nothing about host aggregates or availability zones. | |
| 22:30:16 | sean-k-mooney | cfriesen: well an aggreaget is just a bag of resouce providers nothing more or less. you can use them to group anything you want | |
| 22:30:18 | efried | cfriesen: Except that placement aggregates != host aggregates | |
| 22:30:39 | efried | cfriesen: The other main use case for aggregates is to associate sharing resource providers, like a shared storage pool. | |
| 22:31:25 | efried | ^ placement aggregates, that is. | |
| 22:32:07 | cfriesen | efried: okay...I'm not up on the placement terminology. :) | |
| 22:32:35 | efried | cfriesen: I'm not up on... pretty much everything else. Between the two of us, we're gonna need a pidgin. | |
| 22:33:22 | sean-k-mooney | cfriesen: a sharing resouce provider is jsut a normal resouce provider with a trait and then its inventores are consumable by anything in the same aggreates as it | |
| 22:34:36 | cfriesen | so how do we handle the idea of an instance consuming cpu/ram from one placement aggregate but a PCI device from another placement aggregate? (for the non-strict PCI affinity case) | |
| 22:34:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160 | |
| 22:34:49 | mriedem | melwitt: cleaned that up ^ | |
| 22:35:28 | sean-k-mooney | cfriesen: instance consume resocues form invetories provided by resouce providers | |
| 22:35:40 | melwitt | mriedem: thanks. sorry I lost track of that one | |
| 22:35:47 | sean-k-mooney | cfriesen: and resouce provires can be member of 0-n aggregates | |
| 22:36:30 | sean-k-mooney | so you are not consuming form the aggregates | |
| 22:37:22 | efried | cfriesen: In the design I'm strawpersoning, if you care about CPU/mem affinity and you don't care about VF affinity, you would say affine_resources=VCPU,MEMORY_MB but not SRIOV_NET_VF | |
| 22:37:46 | efried | cfriesen: s/VF/PCI device/ if you wish. | |
| 22:39:08 | sean-k-mooney | efried: that may break down a bit with anti afinity | |
| 22:39:33 | efried | sean-k-mooney: I'm saying we're not handling anti-affinity at all with this model. | |
| 22:39:34 | sean-k-mooney | e.g. i want 2 VF whoes parent PF are anti affined | |
| 22:39:48 | sean-k-mooney | efried: ah ok | |
| 22:39:52 | efried | sean-k-mooney: Yeah, I get that. But that's not a NUMA use case, right? | |
| 22:40:01 | efried | It's more about HA | |
| 22:40:05 | efried | redundancy | |
| 22:40:15 | efried | powerfail grouping | |
| 22:40:18 | efried | that kind of thing. | |
| 22:40:21 | sean-k-mooney | ya its for ha bonding so my bond dont come from the same card | |
| 22:40:51 | efried | So yeah, this doesn't cover that. And if there's a generic solution that encompasses both, great. But I'm gonna have to pick Jay's brain for that. | |
| 22:40:51 | sean-k-mooney | e.g. pulling out one cable wont bring donw the bond | |
| 22:42:24 | sean-k-mooney | efried: if you can show your propasl support affinity in a clean way and does not block future addtion of anti afinity by design then its better then nothing | |
| 22:43:02 | sean-k-mooney | e.g. we dont have to support everything in one go in rocky | |
| 22:43:33 | sean-k-mooney | it would be nice however | |
| 22:44:25 | efried | That's kinda what I was thinking. Propose it this way, show that it wouldn't prevent a more generic solution in the future, and then Jay can propose (or not) that more generic solution. | |
| 22:46:14 | cfriesen | efried: what about multi-numa-node guests? they would need to essentially do numa anti-affinity (but host affinity) between the two pools of memory | |
| 22:47:02 | efried | cfriesen: What does that mean? Multi-numa-node guests? You mean guests that don't care about NUMA affinity? Or do they explicitly care about *anti* affinity? | |
| 22:47:05 | sean-k-mooney | cfriesen: multi numa guest do not need to be on multiple host numa nodes | |
| 22:47:14 | cfriesen | efried: no, I mean guests with multi numa nodes | |
| 22:47:22 | sean-k-mooney | cfriesen: i consider that limitation in libvirt to be a bug not a feature | |
| 22:47:30 | sean-k-mooney | libvirt virt driver that is | |