Earlier  
Later
New Search
Posted Nick Remark
#openstack-sdks - 2018-10-19
12:51:58 cmurphy ah hmm that is a somewhat compelling case
12:52:18 openstackgerrit Monty Taylor proposed openstack/python-openstackclient master: Make use of keystoneauth service-type filtering for versions https://review.openstack.org/585030
12:53:24 mordred cmurphy: yah - I wrote more words on that review - but dtantsur had a patch to sdk to make use of the retriable_status_codes feature - and that made me realize that if ksa is doing retries under the covers, it makes higher-level rate limiting have a hole where it doesn't know about some of the interactions
12:53:43 cmurphy yeah that makes sense
12:54:22 frickler mordred: that patch seems to have a merge conflict now that you updated it
12:54:26 mordred that said - there's still a potential slight weirdness, as retries are implemented in the session, while a rate limit is really an adapter-level concept (so that it can be per-service)
12:54:28 mordred frickler: *awesome*
12:54:30 mordred lemme fix
12:55:47 openstackgerrit Monty Taylor proposed openstack/python-openstackclient master: Make use of keystoneauth service-type filtering for versions https://review.openstack.org/585030
12:55:51 mordred frickler: that should do it :)
12:57:52 mordred cmurphy: so I guess the important next step for that patch would be "write tests"?
12:58:51 cmurphy mordred: I'm kind of hoping there is a non-DIY way of implementing the semaphore thing
12:59:12 cmurphy mordred: but assuming there's not then tests would be a good next step
12:59:22 mordred cmurphy: I didn't find one when I was looking - although I totally agree with you
12:59:29 mordred I'm a bit sad there isn't one of these in the standard lib
12:59:44 frickler mordred: oh, that's osc, I can only +1 there anyway, but lgtm
13:00:01 mordred frickler: awesome, thanks!
13:00:24 mordred cmurphy: maybe we should upstream this one to the python stdlib once we're happy with it. I'm sure that'll only take 8 years or something
13:00:44 cmurphy :D
13:00:53 dtantsur heh
13:00:58 dtantsur Python 4.2
13:01:06 cmurphy lol
13:12:52 samueldmq morning sdk
13:15:50 mordred morning samueldmq !
13:25:17 openstackgerrit Samuel de Medeiros Queiroz proposed openstack/openstacksdk master: DNM: Logging image code path https://review.openstack.org/611868
13:29:33 openstackgerrit Samuel de Medeiros Queiroz proposed openstack/openstacksdk master: DNM: Logging public server code path https://review.openstack.org/611869
14:15:41 openstackgerrit Monty Taylor proposed openstack/keystoneauth master: Add support for client-side rate limiting https://review.openstack.org/605043
14:18:19 mordred dtantsur: ^^ I reworked it a bit based on your review and put the rate limit wait into the semaphore itself
14:18:33 mordred dtantsur: I tried to add some comments - but it's still a pretty dense class
14:18:41 dtantsur cool, will check
14:19:34 mordred kmalloc, cmurphy: ^^ also y'all. of course, it can't land without tests - so I still have to figure that out - but if you feel like attacking it logically, awesome
14:25:43 dtantsur mordred: I have a feeling there is a deadlock lurking there
14:25:46 dtantsur see inline
14:28:11 mordred dtantsur: ah - yeah. it should be fine as long as there are more than one thread executing at the same time - but not for the 1st thread or in a single-threaded mode ... good catch
14:30:43 dtantsur mordred: I'm not sure how multiple thread would help. You're calling wait() while holding a lock. For wait() to return something needs to acquire the same lock and fire the event.
14:30:55 dtantsur It cannot acquire the lock held during wait(). Am I missing something?
14:31:57 mordred dtantsur: no - you're totally right
14:32:04 mordred dtantsur: I need another coffee :)
14:32:33 dtantsur mordred: the resource starvation problem you mention in the commit message.. is it really so bad to justify writing your own semaphore class? :)
14:33:16 mordred dtantsur: yup :)
14:34:35 openstackgerrit Stephen Finucane proposed openstack/cliff master: Remove dead files https://review.openstack.org/611888
14:34:35 openstackgerrit Stephen Finucane proposed openstack/cliff master: Stop wildcard importing argparse https://review.openstack.org/611889
14:34:36 openstackgerrit Stephen Finucane proposed openstack/cliff master: WIP: Add support for environment variables https://review.openstack.org/611890
14:34:37 mordred dtantsur: amazingly enough - all of the varied use cases this solves for are represented by actual real-world issues
14:35:11 mordred dtantsur: you're think that the server side would just be able to handle all of this without the client needing to care ... but oh golly
14:35:15 dtantsur that damn reality; I like dealing with round things in vacuum much more
14:35:18 mordred right?
14:35:23 dtantsur ofc
14:35:32 mordred dtantsur: remind me to tell you the story of the time we crashed the HP Public Cloud
14:35:45 mordred just by running nodepool against its public api
14:35:58 dtantsur OH I'd LOVE to hear that
14:37:02 mordred they had a hardware load balancer / api gateway in front of the api services - which would time out client connections for requests that took too long
14:37:23 mordred so, on receiving a 500 from the load balancer - we'd auto retry again
14:37:55 mordred problem was - the requests were taking so long because there was a missing index on a sql table, so each request was causing a long running query to get started in mysql
14:39:00 dtantsur LOOOL
14:39:47 mordred bcause it was doing a full table scan on the server table for every operation - oh, and also they hadn't been cleaning out old deleted entries from the table
14:40:19 dtantsur hashtag successstory
14:40:25 mordred so we'd to, while True: try: create_server ; break except: delete_server
14:40:42 mordred every create would fail, at which point we'd attempt to clean up after ourselves
14:40:53 mordred in a tight loop with no pausing
14:41:11 mordred and then everything melted to the ground :)
14:41:35 dtantsur well, you gave them a free DoS attack testing :D some people pay for that
14:41:41 mordred RIGHT?
14:41:53 mordred this is how we try to convince more people to give us resources for nodepool
14:42:03 mordred "we promise we can find the errors in your cloud deployment"
14:43:11 dtantsur okay, I should stop laughing so loudly, my wife will become suspicious :D
14:50:11 mordred hehehe
14:50:41 openstackgerrit Monty Taylor proposed openstack/keystoneauth master: Add support for client-side rate limiting https://review.openstack.org/605043
14:51:07 mordred dtantsur: figuring out how to write unittests for this thing is going to be ... fun
14:51:18 kmalloc Mock
14:51:22 kmalloc Lots of mock
14:51:54 mordred kmalloc: I dunno ... seems like a good risk of testing that we wrote some mocks
14:51:58 kmalloc My recommendation is mock plus freezegun, check to make sure x calls were made
14:52:05 dtantsur mock the heck out of it \o/
14:52:11 kmalloc With the right args
14:53:37 kmalloc You can introspect the magicmock object and see what was called
14:54:27 kmalloc The underlying request stuff is tested elsewhere.
14:54:41 mordred oh - totally - definitely not worried about testing that
14:54:57 kmalloc So we just want to check the ratelimiting and locking.
14:55:12 dtantsur oh, how I "love" ironicclient+keystoneauth spitting "Service catalog is empty" on wrong magical combination of arguments..
14:55:14 mordred mostly just want to make sure we're hitting things like dead locks and max concurrency and stuff
14:55:19 kmalloc All doable with simple wsgiref webtest app.
14:56:00 kmalloc You're looking to do a simple reflector thing :)
14:56:09 mordred you're a simple reflector thing
14:56:42 dtantsur lol
14:59:08 mordred kmalloc: this is more moist than I prefer
14:59:45 kmalloc I suggest a lemur dryer then ;)
15:45:23 mnaser mordred: i was just hit by this .. https://github.com/ansible/ansible/pull/42913 -- do you mind giving it another shipit if it makes sense to you?
15:45:36 mnaser cc Shrews rcarrillocruz ^
15:55:14 mordred mnaser: seems reasonable - but also that kinda feels like a default behavior sdk should have on the backend
15:55:34 mordred oh - wait- I misread soemthing
16:09:31 mnaser mordred: in this case a cloud with multiple users having “admin” would fail (in different domains)
16:09:54 mnaser mordred: thank you!
17:07:42 kmalloc mordred: the limits code for SDK looks correct
18:17:58 mordred kmalloc: yay!
#openstack-sdks - 2018-10-20
11:03:51 openstackgerrit Yuval Shalev proposed openstack/openstacksdk master: Added assign function to identity proxy https://review.openstack.org/612006
11:08:05 openstackgerrit Yuval Shalev proposed openstack/openstacksdk master: Added assign function to identity v3 proxy https://review.openstack.org/612006
14:14:46 openstackgerrit Yuval Shalev proposed openstack/openstacksdk master: Added assign function to identity v3 proxy https://review.openstack.org/612006
#openstack-sdks - 2018-10-21
02:17:45 openstackgerrit Daniel Speichert proposed openstack/openstacksdk master: Fix upload of Swift object smaller than segment limit (create_object) https://review.openstack.org/612113
12:43:27 openstackgerrit Maxim Babushkin proposed openstack/openstacksdk master: Add vnic_type to create_port valid kwargs https://review.openstack.org/612162

Earlier   Later