| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-09 | |||
| 16:12:41 | edleafe | ok, I think I have the gist of it | |
| 16:13:54 | sean-k-mooney | edleafe: actully black on colour is easier to read on paper then black on white if you have dislexia but on pcs not so much. thats why they make tinted glasses for peopel with dislexia | |
| 16:14:20 | leakypipes | sean-k-mooney: sense makes that me to. | |
| 16:14:27 | edleafe | sean-k-mooney: good to know | |
| 16:14:46 | leakypipes | finucannot: that's the GOOD stuff, man... | |
| 16:14:52 | edleafe | finucannot: there's not much else :) | |
| 16:14:58 | finucannot | leakypipes: You have a problem, sir. | |
| 16:15:04 | leakypipes | finucannot: yes, well aware :) | |
| 16:15:19 | sean-k-mooney | finucannot: so you read the titles and nothing else :P | |
| 16:15:52 | sean-k-mooney | sql is precise | |
| 16:17:58 | cfriesen | are we still planning on using this for numa as well? | |
| 16:18:10 | finucannot | ^ that was my question | |
| 16:18:13 | leakypipes | cfriesen: yes | |
| 16:18:22 | fried_rice | leakypipes: I'm ready. Hangout? | |
| 16:18:29 | leakypipes | cfriesen: though... meh? :) | |
| 16:18:43 | leakypipes | cfriesen: it would be a giant headache for operators to do NUMA affinity this way. | |
| 16:19:13 | leakypipes | cfriesen: due to the pinning logic that needs to occur in the filtering, I'd just as likely recommend keeping the NUMATopologyFilter as-is in Nova scheduler side. | |
| 16:19:13 | fried_rice | oh. Well, that's a point in favor of the spec I'm writing up. It handles NUMA specifically, but not much else. | |
| 16:20:15 | cfriesen | leakypipes: why would operators need to do anything? | |
| 16:20:25 | leakypipes | cfriesen: the primary use cases I've had in my mind with this are the use cases around multi-failure-domain (HA) placement as well as affinity placement for groups of instances (with tolerances for relative distances between where those instances land) | |
| 16:21:02 | leakypipes | cfriesen: well, perhaps the operator wouldn't... maybe the virt driver could auto-create the aggregates and set distances. | |
| 16:21:09 | leakypipes | cfriesen: for numa nodes, etc | |
| 16:22:01 | cfriesen | leakypipes: makes sense. do you see us being able to drop the server group affinity/antiaffinity filters? | |
| 16:22:44 | leakypipes | cfriesen, finucannot, sean-k-mooney, fried_rice, edleafe: hmm, perhaps it wasn't obvious, but I am not hardcoding the whole "region", "rack", "row", "power-unit" "power-domain" entries in that distance_types table. those would be for operators to fill in what they liked. | |
| 16:23:10 | fried_rice | Which would allow for "numa-node" in theory. | |
| 16:23:25 | finucannot | Yeah, they're all...distance_codes? | |
| 16:23:32 | cfriesen | leakypipes: for the numa case we'd need something like "give me cpus and ram on the same numa node, and some more cpus and ram from a different numa node of the same host, and a pci device from the same host" | |
| 16:23:49 | edleafe | leakypipes: so they would have to define the distance codes, as well as the value for the distances? | |
| 16:24:42 | leakypipes | cfriesen: re: the server group filters... maybe. the concept that I'm espousing here does away with the concept of a server group entirely and relies on point-in-time policy from the caller of nova boot. so instead a scheduler hint for the instance group, the caller is specifying "I want to be near (for *this* definition of "near") this other instance..." | |
| 16:25:07 | leakypipes | edleafe: yep. but keep in mind those distances are simply relative, nothing more. | |
| 16:25:50 | edleafe | I would certainly like to get input from operators about this, then | |
| 16:25:50 | leakypipes | edleafe: we could of course provide tutorial/examples for common use cases. | |
| 16:25:52 | sean-k-mooney | leakypipes: yes i assumed that the operator would fill thoes in but a compute driver could also in some cases | |
| 16:26:07 | fried_rice | One thing that concerns me is how much the operator needs to know about how resource providers are modeled by the driver. | |
| 16:26:30 | leakypipes | fried_rice: in what way? could you elaborate? | |
| 16:26:44 | edleafe | fried_rice: I was just going to say that it seems that we are making the operators fit the solution rather than the other way around | |
| 16:27:20 | fried_rice | If the providers are modeled in tree fashion, the operator needs to know that so she can know which provider(s) to specify what distances for. | |
| 16:27:35 | leakypipes | edleafe: I'm not sure how any solution that didn't rely on the operator describing the relationship of their compute nodes to their groupings of compute nodes (i.e. placement aggregates) would work. | |
| 16:27:36 | sean-k-mooney | leakypipes: you konw the distance types you are proposing are similar to the crush map valuse used by default in ceph | |
| 16:27:42 | leakypipes | edleafe: I mean, there's so many variations... | |
| 16:27:42 | fried_rice | Which is different if they're modeled in *this* shape tree versus *that* shape tree. | |
| 16:28:57 | edleafe | leakypipes: I'm not saying that this isn't acceptable. I'm just saying that it's important to get operators input on whether this is acceptable. | |
| 16:29:11 | leakypipes | sean-k-mooney: I'm not familiar with ceph crush maps :( however, this distance modeling is kinda similar to the distance maps for CPU cores to different levels of cache in NUMA setups, for sure. | |
| 16:29:41 | leakypipes | edleafe: ack, totes. | |
| 16:29:43 | fried_rice | leakypipes: Can you give an example (in the etherpad) for the CLI syntax that would deploy an instance which gets its storage from a sharing provider "near" its compute host? | |
| 16:30:09 | leakypipes | fried_rice: sure, one sec. | |
| 16:30:52 | fried_rice | And when you're done with that, I wouldn't mind delving a little bit into how real NUMA use cases would be modeled/expressed. Like the one cfriesen mentions above. | |
| 16:31:58 | sean-k-mooney | leakypipes: http://docs.ceph.com/docs/master/rados/operations/crush-map/#types-and-buckets just as an fyi. if you swap osd for RP that could work for placement too | |
| 16:34:55 | sean-k-mooney | by the way my brain is working at like 50% today partly due to a headache but i might still joing to listen but no going to be able to add too much. | |
| 16:35:12 | leakypipes | sean-k-mooney: ah, cool. thanks for the link :) | |
| 16:37:05 | sean-k-mooney | ceph have a very similar problem when selecting where to place data on to any give disk they control given the ha/stroage profiles requested for a given pool. | |
| 16:43:22 | leakypipes | fried_rice: aren't conversations on etherpad fun? ;) | |
| 16:43:48 | fried_rice | It's nice to be able to tie the conversation to the context from which it sprungeth. | |
| 16:44:01 | finucannot | Remind me: why do we need the basic NUMA stuff to be mapped this way? | |
| 16:44:20 | leakypipes | finucannot: we don't. | |
| 16:44:22 | openstackgerrit | Merged openstack/nova master: Use with method to consistent oslo timeutils usage https://review.openstack.org/542105 | |
| 16:44:26 | finucannot | As opposed to what we normally do, namely, N guest nodes are mapped to N host nodes | |
| 16:44:27 | fried_rice | finucannot: I kinda don't think we do. I'm trying to get us to that conclusion... | |
| 16:44:35 | finucannot | Gotcha :) | |
| 16:47:14 | leakypipes | finucannot, cfriesen: which one of you is light green? | |
| 16:47:19 | cfriesen | not me | |
| 16:47:23 | finucannot | no me | |
| 16:47:27 | finucannot | *not | |
| 16:47:37 | leakypipes | sean-k-mooney: are you light green? | |
| 16:47:59 | sean-k-mooney | yes i need to add my name to a list at the top | |
| 16:48:33 | leakypipes | sean-k-mooney: or just set your name in the ehterpad box... | |
| 16:48:36 | leakypipes | top right | |
| 16:49:09 | sean-k-mooney | ah yes that gets lost when you log out but done | |
| 16:49:39 | fried_rice | Okay, so leakypipes, this model is satisfying use cases involving placing resource consumers (usually/often instances) at certain distances from other resource consumers. It's not (real good at) placing resources at certain distances from other resources within one consumer/allocation. | |
| 16:49:56 | edleafe | leakypipes: will distances have to be defined for every possible relation in a system? | |
| 16:50:05 | edleafe | leakypipes: I have a rough diagram here: https://awwapp.com/b/um4k85oc1/ | |
| 16:50:45 | edleafe | leakypipes: will every compute node need a distance defined to all shared storages? | |
| 16:57:41 | leakypipes | edleafe: no, every resource provider would need a distance entry to the aggregate representing the row that the storage pool serviced. | |
| 16:58:04 | edleafe | leakypipes: ok, gotcha | |
| 16:59:06 | edleafe | leakypipes: so if there were 100 rows in a DC, and 10000 compute nodes, that would mean there would have to be 100K distances defined? | |
| 16:59:31 | edleafe | oops, 1M distances | |
| 17:03:12 | mriedem | hrw: thanks, some small tweaks and we're golden | |
| 17:03:29 | hrw | mriedem: cool, thanks | |
| 17:04:14 | edleafe | I can see the usefulness of "near" and "not near", but how is the granularity of distance useful? Would there be a use case like "give me a shared storage that is more than 2 away but less than 7"? | |
| 17:04:58 | leakypipes | edleafe: the affinity/anti-affinity use cases are for deployments that have more than one tier of distance. so, you'd want to have some additional distance breakdown, say, for a rack. you'd create aggregates for each rack and assign the compute nodes in that rack to the associated aggregate. the resource_provider_distances table would necessarily have an order of magnitude more records than the resource_provider_aggregates table. | |
| 17:05:35 | leakypipes | edleafe: so if we have 1M records in resource_provider_aggregates, we'd likely have something like 10M records in the resource_provider_distances table. | |
| 17:06:16 | leakypipes | edleafe: that 10M records would take approximately 200M of memory. | |
| 17:06:22 | edleafe | leakypipes: understood. And these would all be set by the operators? | |
| 17:07:35 | leakypipes | edleafe: yes, but in groups of course... so, a CLI tool that did this: openstack placement aggregate $AGG_UUID set-distance-to $OTHER_AGG_UUID $DISTANCE_AMOUNT would be something we would create for operators. | |
| 17:08:07 | leakypipes | edleafe: that would basically do the INSERT INTO ... SELECT ... FROM statements that appear in the etherpad. | |
| 17:08:47 | edleafe | leakypipes: hmmm... so couldn't we just define agg distances, rather than distances between everything in AggA to everything in AggB? | |
| 17:09:16 | leakypipes | edleafe: yeah, I thought about that. it may be possible to do that. | |
| 17:09:38 | edleafe | leakypipes: ok, just thought I was missing something | |
| 17:10:39 | fried_rice | In that simplified form, we just have one extra table: agg1, agg2, distance ? | |
| 17:11:18 | fried_rice | whose entries may be redundant (agg2, agg1, distance had better have the same distance as ^) | |
| 17:11:28 | fried_rice | (unless distances are "directional" - ye gods) | |
| 17:11:49 | edleafe | fried_rice: yeesh | |
| 17:12:09 | leakypipes | fried_rice: right, that's the issue I ran into when I originally modeled things like that. basically needed a CHECK constraint to ensure aggA != aggB | |
| 17:12:24 | edleafe | fried_rice: it might be simpler to assign a "position" to each agg, rather than a distance between them | |
| 17:12:35 | fried_rice | mm | |
| 17:12:47 | edleafe | the distance is just abs(posA - posB) | |
| 17:14:02 | fried_rice | edleafe: And we could support directionality there too by taking out the abs(). negative distance is "uphill", positive "downhill". | |
| 17:14:09 | fried_rice | edleafe: (Yes, I'm kidding) | |
| 17:15:30 | sean-k-mooney | fried_rice: never try to model distace over netwrk links with mpls multi path routing because in that case your distance ro a request can be different then the distance for a respocne | |
| 17:16:53 | fried_rice | Okay folks, I think I've seen enough to "get it". Just want to confirm a couple of takeaways here: | |