Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-09
16:18:29 leakypipes cfriesen: though... meh? :)
16:18:43 leakypipes cfriesen: it would be a giant headache for operators to do NUMA affinity this way.
16:19:13 fried_rice oh. Well, that's a point in favor of the spec I'm writing up. It handles NUMA specifically, but not much else.
16:19:13 leakypipes cfriesen: due to the pinning logic that needs to occur in the filtering, I'd just as likely recommend keeping the NUMATopologyFilter as-is in Nova scheduler side.
16:20:15 cfriesen leakypipes: why would operators need to do anything?
16:20:25 leakypipes cfriesen: the primary use cases I've had in my mind with this are the use cases around multi-failure-domain (HA) placement as well as affinity placement for groups of instances (with tolerances for relative distances between where those instances land)
16:21:02 leakypipes cfriesen: well, perhaps the operator wouldn't... maybe the virt driver could auto-create the aggregates and set distances.
16:21:09 leakypipes cfriesen: for numa nodes, etc
16:22:01 cfriesen leakypipes: makes sense. do you see us being able to drop the server group affinity/antiaffinity filters?
16:22:44 leakypipes cfriesen, finucannot, sean-k-mooney, fried_rice, edleafe: hmm, perhaps it wasn't obvious, but I am not hardcoding the whole "region", "rack", "row", "power-unit" "power-domain" entries in that distance_types table. those would be for operators to fill in what they liked.
16:23:10 fried_rice Which would allow for "numa-node" in theory.
16:23:25 finucannot Yeah, they're all...distance_codes?
16:23:32 cfriesen leakypipes: for the numa case we'd need something like "give me cpus and ram on the same numa node, and some more cpus and ram from a different numa node of the same host, and a pci device from the same host"
16:23:49 edleafe leakypipes: so they would have to define the distance codes, as well as the value for the distances?
16:24:42 leakypipes cfriesen: re: the server group filters... maybe. the concept that I'm espousing here does away with the concept of a server group entirely and relies on point-in-time policy from the caller of nova boot. so instead a scheduler hint for the instance group, the caller is specifying "I want to be near (for *this* definition of "near") this other instance..."
16:25:07 leakypipes edleafe: yep. but keep in mind those distances are simply relative, nothing more.
16:25:50 leakypipes edleafe: we could of course provide tutorial/examples for common use cases.
16:25:50 edleafe I would certainly like to get input from operators about this, then
16:25:52 sean-k-mooney leakypipes: yes i assumed that the operator would fill thoes in but a compute driver could also in some cases
16:26:07 fried_rice One thing that concerns me is how much the operator needs to know about how resource providers are modeled by the driver.
16:26:30 leakypipes fried_rice: in what way? could you elaborate?
16:26:44 edleafe fried_rice: I was just going to say that it seems that we are making the operators fit the solution rather than the other way around
16:27:20 fried_rice If the providers are modeled in tree fashion, the operator needs to know that so she can know which provider(s) to specify what distances for.
16:27:35 leakypipes edleafe: I'm not sure how any solution that didn't rely on the operator describing the relationship of their compute nodes to their groupings of compute nodes (i.e. placement aggregates) would work.
16:27:36 sean-k-mooney leakypipes: you konw the distance types you are proposing are similar to the crush map valuse used by default in ceph
16:27:42 fried_rice Which is different if they're modeled in *this* shape tree versus *that* shape tree.
16:27:42 leakypipes edleafe: I mean, there's so many variations...
16:28:57 edleafe leakypipes: I'm not saying that this isn't acceptable. I'm just saying that it's important to get operators input on whether this is acceptable.
16:29:11 leakypipes sean-k-mooney: I'm not familiar with ceph crush maps :( however, this distance modeling is kinda similar to the distance maps for CPU cores to different levels of cache in NUMA setups, for sure.
16:29:41 leakypipes edleafe: ack, totes.
16:29:43 fried_rice leakypipes: Can you give an example (in the etherpad) for the CLI syntax that would deploy an instance which gets its storage from a sharing provider "near" its compute host?
16:30:09 leakypipes fried_rice: sure, one sec.
16:30:52 fried_rice And when you're done with that, I wouldn't mind delving a little bit into how real NUMA use cases would be modeled/expressed. Like the one cfriesen mentions above.
16:31:58 sean-k-mooney leakypipes: http://docs.ceph.com/docs/master/rados/operations/crush-map/#types-and-buckets just as an fyi. if you swap osd for RP that could work for placement too
16:34:55 sean-k-mooney by the way my brain is working at like 50% today partly due to a headache but i might still joing to listen but no going to be able to add too much.
16:35:12 leakypipes sean-k-mooney: ah, cool. thanks for the link :)
16:37:05 sean-k-mooney ceph have a very similar problem when selecting where to place data on to any give disk they control given the ha/stroage profiles requested for a given pool.
16:43:22 leakypipes fried_rice: aren't conversations on etherpad fun? ;)
16:43:48 fried_rice It's nice to be able to tie the conversation to the context from which it sprungeth.
16:44:01 finucannot Remind me: why do we need the basic NUMA stuff to be mapped this way?
16:44:20 leakypipes finucannot: we don't.
16:44:22 openstackgerrit Merged openstack/nova master: Use with method to consistent oslo timeutils usage https://review.openstack.org/542105
16:44:26 finucannot As opposed to what we normally do, namely, N guest nodes are mapped to N host nodes
16:44:27 fried_rice finucannot: I kinda don't think we do. I'm trying to get us to that conclusion...
16:44:35 finucannot Gotcha :)
16:47:14 leakypipes finucannot, cfriesen: which one of you is light green?
16:47:19 cfriesen not me
16:47:23 finucannot no me
16:47:27 finucannot *not
16:47:37 leakypipes sean-k-mooney: are you light green?
16:47:59 sean-k-mooney yes i need to add my name to a list at the top
16:48:33 leakypipes sean-k-mooney: or just set your name in the ehterpad box...
16:48:36 leakypipes top right
16:49:09 sean-k-mooney ah yes that gets lost when you log out but done
16:49:39 fried_rice Okay, so leakypipes, this model is satisfying use cases involving placing resource consumers (usually/often instances) at certain distances from other resource consumers. It's not (real good at) placing resources at certain distances from other resources within one consumer/allocation.
16:49:56 edleafe leakypipes: will distances have to be defined for every possible relation in a system?
16:50:05 edleafe leakypipes: I have a rough diagram here: https://awwapp.com/b/um4k85oc1/
16:50:45 edleafe leakypipes: will every compute node need a distance defined to all shared storages?
16:57:41 leakypipes edleafe: no, every resource provider would need a distance entry to the aggregate representing the row that the storage pool serviced.
16:58:04 edleafe leakypipes: ok, gotcha
16:59:06 edleafe leakypipes: so if there were 100 rows in a DC, and 10000 compute nodes, that would mean there would have to be 100K distances defined?
16:59:31 edleafe oops, 1M distances
17:03:12 mriedem hrw: thanks, some small tweaks and we're golden
17:03:29 hrw mriedem: cool, thanks
17:04:14 edleafe I can see the usefulness of "near" and "not near", but how is the granularity of distance useful? Would there be a use case like "give me a shared storage that is more than 2 away but less than 7"?
17:04:58 leakypipes edleafe: the affinity/anti-affinity use cases are for deployments that have more than one tier of distance. so, you'd want to have some additional distance breakdown, say, for a rack. you'd create aggregates for each rack and assign the compute nodes in that rack to the associated aggregate. the resource_provider_distances table would necessarily have an order of magnitude more records than the resource_provider_aggregates table.
17:05:35 leakypipes edleafe: so if we have 1M records in resource_provider_aggregates, we'd likely have something like 10M records in the resource_provider_distances table.
17:06:16 leakypipes edleafe: that 10M records would take approximately 200M of memory.
17:06:22 edleafe leakypipes: understood. And these would all be set by the operators?
17:07:35 leakypipes edleafe: yes, but in groups of course... so, a CLI tool that did this: openstack placement aggregate $AGG_UUID set-distance-to $OTHER_AGG_UUID $DISTANCE_AMOUNT would be something we would create for operators.
17:08:07 leakypipes edleafe: that would basically do the INSERT INTO ... SELECT ... FROM statements that appear in the etherpad.
17:08:47 edleafe leakypipes: hmmm... so couldn't we just define agg distances, rather than distances between everything in AggA to everything in AggB?
17:09:16 leakypipes edleafe: yeah, I thought about that. it may be possible to do that.
17:09:38 edleafe leakypipes: ok, just thought I was missing something
17:10:39 fried_rice In that simplified form, we just have one extra table: agg1, agg2, distance ?
17:11:18 fried_rice whose entries may be redundant (agg2, agg1, distance had better have the same distance as ^)
17:11:28 fried_rice (unless distances are "directional" - ye gods)
17:11:49 edleafe fried_rice: yeesh
17:12:09 leakypipes fried_rice: right, that's the issue I ran into when I originally modeled things like that. basically needed a CHECK constraint to ensure aggA != aggB
17:12:24 edleafe fried_rice: it might be simpler to assign a "position" to each agg, rather than a distance between them
17:12:35 fried_rice mm
17:12:47 edleafe the distance is just abs(posA - posB)
17:14:02 fried_rice edleafe: And we could support directionality there too by taking out the abs(). negative distance is "uphill", positive "downhill".
17:14:09 fried_rice edleafe: (Yes, I'm kidding)
17:15:30 sean-k-mooney fried_rice: never try to model distace over netwrk links with mpls multi path routing because in that case your distance ro a request can be different then the distance for a respocne
17:16:53 fried_rice Okay folks, I think I've seen enough to "get it". Just want to confirm a couple of takeaways here:
17:16:55 fried_rice (1) This isn't intended, and not particularly suited, for solving NUMA.
17:17:25 fried_rice (2) This is post-Rocky?
17:17:53 fried_rice leakypipes: ^ -^
17:18:15 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: Provide basic data for AArch64 support matrix/functionality https://review.openstack.org/541728
17:19:29 hrw mriedem: implemented
17:20:22 fried_rice gotta run for a couple hours. Thanks leakypipes edleafe finucannot cfriesen sean-k-mooney, this has been very helpful.
17:20:49 finucannot :D
17:34:07 sean-k-mooney ok my brain has had it for today. im going to take some panadol and go lie down on the couch and not think for a while. enjoy your weekends o/
17:34:22 leakypipes sean-k-mooney: ciao :)
17:42:16 leakypipes fried_rolls: did you say you had a spec/proposal up for the distances stuff? I'd certainly be eager to read it.
17:42:21 leakypipes or was it edleafe?
17:42:39 leakypipes edleafe: what happened to figleaf?
17:43:50 edleafe dunno
17:44:00 leakypipes there he is.

Earlier   Later