Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-09
16:49:39 fried_rice Okay, so leakypipes, this model is satisfying use cases involving placing resource consumers (usually/often instances) at certain distances from other resource consumers. It's not (real good at) placing resources at certain distances from other resources within one consumer/allocation.
16:49:56 edleafe leakypipes: will distances have to be defined for every possible relation in a system?
16:50:05 edleafe leakypipes: I have a rough diagram here: https://awwapp.com/b/um4k85oc1/
16:50:45 edleafe leakypipes: will every compute node need a distance defined to all shared storages?
16:57:41 leakypipes edleafe: no, every resource provider would need a distance entry to the aggregate representing the row that the storage pool serviced.
16:58:04 edleafe leakypipes: ok, gotcha
16:59:06 edleafe leakypipes: so if there were 100 rows in a DC, and 10000 compute nodes, that would mean there would have to be 100K distances defined?
16:59:31 edleafe oops, 1M distances
17:03:12 mriedem hrw: thanks, some small tweaks and we're golden
17:03:29 hrw mriedem: cool, thanks
17:04:14 edleafe I can see the usefulness of "near" and "not near", but how is the granularity of distance useful? Would there be a use case like "give me a shared storage that is more than 2 away but less than 7"?
17:04:58 leakypipes edleafe: the affinity/anti-affinity use cases are for deployments that have more than one tier of distance. so, you'd want to have some additional distance breakdown, say, for a rack. you'd create aggregates for each rack and assign the compute nodes in that rack to the associated aggregate. the resource_provider_distances table would necessarily have an order of magnitude more records than the resource_provider_aggregates table.
17:05:35 leakypipes edleafe: so if we have 1M records in resource_provider_aggregates, we'd likely have something like 10M records in the resource_provider_distances table.
17:06:16 leakypipes edleafe: that 10M records would take approximately 200M of memory.
17:06:22 edleafe leakypipes: understood. And these would all be set by the operators?
17:07:35 leakypipes edleafe: yes, but in groups of course... so, a CLI tool that did this: openstack placement aggregate $AGG_UUID set-distance-to $OTHER_AGG_UUID $DISTANCE_AMOUNT would be something we would create for operators.
17:08:07 leakypipes edleafe: that would basically do the INSERT INTO ... SELECT ... FROM statements that appear in the etherpad.
17:08:47 edleafe leakypipes: hmmm... so couldn't we just define agg distances, rather than distances between everything in AggA to everything in AggB?
17:09:16 leakypipes edleafe: yeah, I thought about that. it may be possible to do that.
17:09:38 edleafe leakypipes: ok, just thought I was missing something
17:10:39 fried_rice In that simplified form, we just have one extra table: agg1, agg2, distance ?
17:11:18 fried_rice whose entries may be redundant (agg2, agg1, distance had better have the same distance as ^)
17:11:28 fried_rice (unless distances are "directional" - ye gods)
17:11:49 edleafe fried_rice: yeesh
17:12:09 leakypipes fried_rice: right, that's the issue I ran into when I originally modeled things like that. basically needed a CHECK constraint to ensure aggA != aggB
17:12:24 edleafe fried_rice: it might be simpler to assign a "position" to each agg, rather than a distance between them
17:12:35 fried_rice mm
17:12:47 edleafe the distance is just abs(posA - posB)
17:14:02 fried_rice edleafe: And we could support directionality there too by taking out the abs(). negative distance is "uphill", positive "downhill".
17:14:09 fried_rice edleafe: (Yes, I'm kidding)
17:15:30 sean-k-mooney fried_rice: never try to model distace over netwrk links with mpls multi path routing because in that case your distance ro a request can be different then the distance for a respocne
17:16:53 fried_rice Okay folks, I think I've seen enough to "get it". Just want to confirm a couple of takeaways here:
17:16:55 fried_rice (1) This isn't intended, and not particularly suited, for solving NUMA.
17:17:25 fried_rice (2) This is post-Rocky?
17:17:53 fried_rice leakypipes: ^ -^
17:18:15 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: Provide basic data for AArch64 support matrix/functionality https://review.openstack.org/541728
17:19:29 hrw mriedem: implemented
17:20:22 fried_rice gotta run for a couple hours. Thanks leakypipes edleafe finucannot cfriesen sean-k-mooney, this has been very helpful.
17:20:49 finucannot :D
17:34:07 sean-k-mooney ok my brain has had it for today. im going to take some panadol and go lie down on the couch and not think for a while. enjoy your weekends o/
17:34:22 leakypipes sean-k-mooney: ciao :)
17:42:16 leakypipes fried_rolls: did you say you had a spec/proposal up for the distances stuff? I'd certainly be eager to read it.
17:42:21 leakypipes or was it edleafe?
17:42:39 leakypipes edleafe: what happened to figleaf?
17:43:50 edleafe dunno
17:44:00 leakypipes there he is.
17:44:07 figleaf maybe my bouncer hiccupped
17:46:41 figleaf leakypipes: fried_rolls and I had a convo about modeling NUMA using aggs instead of distances
17:46:55 figleaf leakypipes: I don't think there's a spec posted yet
17:46:59 leakypipes gotcha
17:49:29 fried_rolls figleaf, leakypipes: I'm working on it. The summary paragraph is up in a bp template: https://blueprints.launchpad.net/nova/+spec/resource-class-affinity
17:55:04 openstackgerrit Merged openstack/nova master: [placement] Move body examples to an isolated directory https://review.openstack.org/529998
17:55:12 openstackgerrit Merged openstack/nova stable/ocata: Add release note for Aggregate[Core|Ram|Disk]Filter change https://review.openstack.org/542323
17:55:21 openstackgerrit Merged openstack/nova master: Make bdms querying in multi-cell use scatter-gather and ignore down cell https://review.openstack.org/541246
17:55:30 openstackgerrit Merged openstack/nova master: Remove unnecessary arguments in notification methods https://review.openstack.org/535599
17:55:42 openstackgerrit Merged openstack/nova master: Remove microversion fallback code from report client https://review.openstack.org/528794
17:59:45 melwitt dansmith: I had to replace assertEqual with assertItemsEqual for the list comparison so that ordering wouldn't result in sporadic test failures, if you could re-review at your convenience https://review.openstack.org/#/c/540258
18:00:49 openstackgerrit Surya Seetharaman proposed openstack/nova master: Unmap compute nodes when deleting host mappings during delete cell operation https://review.openstack.org/542964
18:05:21 openstackgerrit Surya Seetharaman proposed openstack/nova master: Unmap compute nodes when deleting host mappings in delete cell operation https://review.openstack.org/542964
18:12:07 dansmith melwitt: yeah I saw that this morning, thought I already +2d
18:12:28 dansmith just never clicked the button I guess
18:12:40 melwitt dansmith: super, thanks
18:13:19 melwitt :)
18:13:27 itlinux hello all.. I have a strange behaviors my coworker sent me. this: I'm seeing this weird behavior on the poc. I'm deploying the avi controllers with heat, this creates a nova instance with a cinder volume. the volumes get created, but seems like the last one loses(a timeout?) and another volume gets created and that one is attached to the last instance.
18:55:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide basic data for AArch64 support matrix/functionality https://review.openstack.org/541728
18:58:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Check for leaked server resource allocations in post_test_hook https://review.openstack.org/538510
19:07:26 amoralej i'm finding some failures in jobs for queens when security groups are failing to create with unauthorized error
19:07:39 amoralej https://[::1]:8774/v2.1/os-security-groups returned with HTTP 401
19:07:48 amoralej https://centos.logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario001/4365/weirdo-project/logs/apache/nova_api_wsgi_error_ssl.txt.gz
19:07:52 openstackgerrit Merged openstack/python-novaclient master: Remove 2 redundant methods https://review.openstack.org/528325
19:07:57 amoralej i see the corresponding error in neutron
19:08:03 amoralej https://centos.logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario001/4365/weirdo-project/logs/neutron/server.txt.gz#_2018-02-09_17_53_21_219
19:08:11 amoralej any idea what could be the cause?
19:08:24 amoralej it's just happens from time to time
19:24:02 melwitt amoralej: is that a tempest test? the first thing I notice is that the nova api isn't being called with the latest microversion, as we deprecated the os-security-groups proxy api back in newton https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id33
19:24:19 amoralej melwitt, yes, it's tempest test
19:24:33 amoralej tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
19:24:50 melwitt that's not causing the 401, but just to note
19:25:21 melwitt other than that, neutron is rejecting the request because of failed token validation. the only thing I can think there is if the token expired over a long running time?
19:25:50 melwitt I don't know off the top of my head how tempest gets/refreshes auth tokens throughout a job run
19:25:55 amoralej but expiration time for tokens were hours, right?
19:26:20 melwitt yeah, I would think so. have to check
19:26:45 amoralej the full job, deployment + tempest run took 51 minutes
19:27:40 amoralej but yeah, i thought something like that too
19:30:02 melwitt looks like the default is only an hour https://github.com/openstack/keystone/blob/master/keystone/conf/token.py#L49
19:30:29 melwitt but I see in your keystone.conf it's set to 2400 seconds, even shorter https://centos.logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario001/4365/weirdo-project/logs/etc/keystone/keystone.conf.txt.gz
19:30:55 melwitt 40 minutes
19:31:17 melwitt but, that only matters if tempest gets a token once and keeps reusing it throughout the run, which I don't know if it does
19:32:13 amoralej yeah, i don't know that, but i think the tempest run itself took much less that 40 mins
19:34:17 melwitt oh, okay, you said deployment + tempest
19:34:52 amoralej yes
19:35:06 amoralej let's see how next runs go
19:35:28 amoralej but hitting the same issue twice in a day is normally a bad sign
19:36:01 melwitt I get that it's bad, it's just there's almost no info there. neutron is rejecting with a 401, token auth failure
19:36:41 amoralej yes, no idea about how to debug deeper
19:36:53 amoralej that's why i came to see the experts :)
19:42:30 melwitt looks like upstream tempest jobs leave keystone.conf [token]expiration at the default 3600 seconds, so maybe you could try that. I would think it's an expired token, else there's something seriously wrong in the keystone auth middleware
20:42:30 openstackgerrit Eric Fried proposed openstack/nova master: Ensure resource classes correctly https://review.openstack.org/539738
20:42:31 openstackgerrit Eric Fried proposed openstack/nova master: New-style _set_inventory_for_provider https://review.openstack.org/537648
20:42:31 openstackgerrit Eric Fried proposed openstack/nova master: SchedulerReportClient.update_from_provider_tree https://review.openstack.org/533821

Earlier   Later