Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-09
16:26:44 edleafe fried_rice: I was just going to say that it seems that we are making the operators fit the solution rather than the other way around
16:27:20 fried_rice If the providers are modeled in tree fashion, the operator needs to know that so she can know which provider(s) to specify what distances for.
16:27:35 leakypipes edleafe: I'm not sure how any solution that didn't rely on the operator describing the relationship of their compute nodes to their groupings of compute nodes (i.e. placement aggregates) would work.
16:27:36 sean-k-mooney leakypipes: you konw the distance types you are proposing are similar to the crush map valuse used by default in ceph
16:27:42 fried_rice Which is different if they're modeled in *this* shape tree versus *that* shape tree.
16:27:42 leakypipes edleafe: I mean, there's so many variations...
16:28:57 edleafe leakypipes: I'm not saying that this isn't acceptable. I'm just saying that it's important to get operators input on whether this is acceptable.
16:29:11 leakypipes sean-k-mooney: I'm not familiar with ceph crush maps :( however, this distance modeling is kinda similar to the distance maps for CPU cores to different levels of cache in NUMA setups, for sure.
16:29:41 leakypipes edleafe: ack, totes.
16:29:43 fried_rice leakypipes: Can you give an example (in the etherpad) for the CLI syntax that would deploy an instance which gets its storage from a sharing provider "near" its compute host?
16:30:09 leakypipes fried_rice: sure, one sec.
16:30:52 fried_rice And when you're done with that, I wouldn't mind delving a little bit into how real NUMA use cases would be modeled/expressed. Like the one cfriesen mentions above.
16:31:58 sean-k-mooney leakypipes: http://docs.ceph.com/docs/master/rados/operations/crush-map/#types-and-buckets just as an fyi. if you swap osd for RP that could work for placement too
16:34:55 sean-k-mooney by the way my brain is working at like 50% today partly due to a headache but i might still joing to listen but no going to be able to add too much.
16:35:12 leakypipes sean-k-mooney: ah, cool. thanks for the link :)
16:37:05 sean-k-mooney ceph have a very similar problem when selecting where to place data on to any give disk they control given the ha/stroage profiles requested for a given pool.
16:43:22 leakypipes fried_rice: aren't conversations on etherpad fun? ;)
16:43:48 fried_rice It's nice to be able to tie the conversation to the context from which it sprungeth.
16:44:01 finucannot Remind me: why do we need the basic NUMA stuff to be mapped this way?
16:44:20 leakypipes finucannot: we don't.
16:44:22 openstackgerrit Merged openstack/nova master: Use with method to consistent oslo timeutils usage https://review.openstack.org/542105
16:44:26 finucannot As opposed to what we normally do, namely, N guest nodes are mapped to N host nodes
16:44:27 fried_rice finucannot: I kinda don't think we do. I'm trying to get us to that conclusion...
16:44:35 finucannot Gotcha :)
16:47:14 leakypipes finucannot, cfriesen: which one of you is light green?
16:47:19 cfriesen not me
16:47:23 finucannot no me
16:47:27 finucannot *not
16:47:37 leakypipes sean-k-mooney: are you light green?
16:47:59 sean-k-mooney yes i need to add my name to a list at the top
16:48:33 leakypipes sean-k-mooney: or just set your name in the ehterpad box...
16:48:36 leakypipes top right
16:49:09 sean-k-mooney ah yes that gets lost when you log out but done
16:49:39 fried_rice Okay, so leakypipes, this model is satisfying use cases involving placing resource consumers (usually/often instances) at certain distances from other resource consumers. It's not (real good at) placing resources at certain distances from other resources within one consumer/allocation.
16:49:56 edleafe leakypipes: will distances have to be defined for every possible relation in a system?
16:50:05 edleafe leakypipes: I have a rough diagram here: https://awwapp.com/b/um4k85oc1/
16:50:45 edleafe leakypipes: will every compute node need a distance defined to all shared storages?
16:57:41 leakypipes edleafe: no, every resource provider would need a distance entry to the aggregate representing the row that the storage pool serviced.
16:58:04 edleafe leakypipes: ok, gotcha
16:59:06 edleafe leakypipes: so if there were 100 rows in a DC, and 10000 compute nodes, that would mean there would have to be 100K distances defined?
16:59:31 edleafe oops, 1M distances
17:03:12 mriedem hrw: thanks, some small tweaks and we're golden
17:03:29 hrw mriedem: cool, thanks
17:04:14 edleafe I can see the usefulness of "near" and "not near", but how is the granularity of distance useful? Would there be a use case like "give me a shared storage that is more than 2 away but less than 7"?
17:04:58 leakypipes edleafe: the affinity/anti-affinity use cases are for deployments that have more than one tier of distance. so, you'd want to have some additional distance breakdown, say, for a rack. you'd create aggregates for each rack and assign the compute nodes in that rack to the associated aggregate. the resource_provider_distances table would necessarily have an order of magnitude more records than the resource_provider_aggregates table.
17:05:35 leakypipes edleafe: so if we have 1M records in resource_provider_aggregates, we'd likely have something like 10M records in the resource_provider_distances table.
17:06:16 leakypipes edleafe: that 10M records would take approximately 200M of memory.
17:06:22 edleafe leakypipes: understood. And these would all be set by the operators?
17:07:35 leakypipes edleafe: yes, but in groups of course... so, a CLI tool that did this: openstack placement aggregate $AGG_UUID set-distance-to $OTHER_AGG_UUID $DISTANCE_AMOUNT would be something we would create for operators.
17:08:07 leakypipes edleafe: that would basically do the INSERT INTO ... SELECT ... FROM statements that appear in the etherpad.
17:08:47 edleafe leakypipes: hmmm... so couldn't we just define agg distances, rather than distances between everything in AggA to everything in AggB?
17:09:16 leakypipes edleafe: yeah, I thought about that. it may be possible to do that.
17:09:38 edleafe leakypipes: ok, just thought I was missing something
17:10:39 fried_rice In that simplified form, we just have one extra table: agg1, agg2, distance ?
17:11:18 fried_rice whose entries may be redundant (agg2, agg1, distance had better have the same distance as ^)
17:11:28 fried_rice (unless distances are "directional" - ye gods)
17:11:49 edleafe fried_rice: yeesh
17:12:09 leakypipes fried_rice: right, that's the issue I ran into when I originally modeled things like that. basically needed a CHECK constraint to ensure aggA != aggB
17:12:24 edleafe fried_rice: it might be simpler to assign a "position" to each agg, rather than a distance between them
17:12:35 fried_rice mm
17:12:47 edleafe the distance is just abs(posA - posB)
17:14:02 fried_rice edleafe: And we could support directionality there too by taking out the abs(). negative distance is "uphill", positive "downhill".
17:14:09 fried_rice edleafe: (Yes, I'm kidding)
17:15:30 sean-k-mooney fried_rice: never try to model distace over netwrk links with mpls multi path routing because in that case your distance ro a request can be different then the distance for a respocne
17:16:53 fried_rice Okay folks, I think I've seen enough to "get it". Just want to confirm a couple of takeaways here:
17:16:55 fried_rice (1) This isn't intended, and not particularly suited, for solving NUMA.
17:17:25 fried_rice (2) This is post-Rocky?
17:17:53 fried_rice leakypipes: ^ -^
17:18:15 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: Provide basic data for AArch64 support matrix/functionality https://review.openstack.org/541728
17:19:29 hrw mriedem: implemented
17:20:22 fried_rice gotta run for a couple hours. Thanks leakypipes edleafe finucannot cfriesen sean-k-mooney, this has been very helpful.
17:20:49 finucannot :D
17:34:07 sean-k-mooney ok my brain has had it for today. im going to take some panadol and go lie down on the couch and not think for a while. enjoy your weekends o/
17:34:22 leakypipes sean-k-mooney: ciao :)
17:42:16 leakypipes fried_rolls: did you say you had a spec/proposal up for the distances stuff? I'd certainly be eager to read it.
17:42:21 leakypipes or was it edleafe?
17:42:39 leakypipes edleafe: what happened to figleaf?
17:43:50 edleafe dunno
17:44:00 leakypipes there he is.
17:44:07 figleaf maybe my bouncer hiccupped
17:46:41 figleaf leakypipes: fried_rolls and I had a convo about modeling NUMA using aggs instead of distances
17:46:55 figleaf leakypipes: I don't think there's a spec posted yet
17:46:59 leakypipes gotcha
17:49:29 fried_rolls figleaf, leakypipes: I'm working on it. The summary paragraph is up in a bp template: https://blueprints.launchpad.net/nova/+spec/resource-class-affinity
17:55:04 openstackgerrit Merged openstack/nova master: [placement] Move body examples to an isolated directory https://review.openstack.org/529998
17:55:12 openstackgerrit Merged openstack/nova stable/ocata: Add release note for Aggregate[Core|Ram|Disk]Filter change https://review.openstack.org/542323
17:55:21 openstackgerrit Merged openstack/nova master: Make bdms querying in multi-cell use scatter-gather and ignore down cell https://review.openstack.org/541246
17:55:30 openstackgerrit Merged openstack/nova master: Remove unnecessary arguments in notification methods https://review.openstack.org/535599
17:55:42 openstackgerrit Merged openstack/nova master: Remove microversion fallback code from report client https://review.openstack.org/528794
17:59:45 melwitt dansmith: I had to replace assertEqual with assertItemsEqual for the list comparison so that ordering wouldn't result in sporadic test failures, if you could re-review at your convenience https://review.openstack.org/#/c/540258
18:00:49 openstackgerrit Surya Seetharaman proposed openstack/nova master: Unmap compute nodes when deleting host mappings during delete cell operation https://review.openstack.org/542964
18:05:21 openstackgerrit Surya Seetharaman proposed openstack/nova master: Unmap compute nodes when deleting host mappings in delete cell operation https://review.openstack.org/542964
18:12:07 dansmith melwitt: yeah I saw that this morning, thought I already +2d
18:12:28 dansmith just never clicked the button I guess
18:12:40 melwitt dansmith: super, thanks
18:13:19 melwitt :)
18:13:27 itlinux hello all.. I have a strange behaviors my coworker sent me. this: I'm seeing this weird behavior on the poc. I'm deploying the avi controllers with heat, this creates a nova instance with a cinder volume. the volumes get created, but seems like the last one loses(a timeout?) and another volume gets created and that one is attached to the last instance.
18:55:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide basic data for AArch64 support matrix/functionality https://review.openstack.org/541728
18:58:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Check for leaked server resource allocations in post_test_hook https://review.openstack.org/538510
19:07:26 amoralej i'm finding some failures in jobs for queens when security groups are failing to create with unauthorized error

Earlier   Later