Earlier  
Later
New Search
Posted Nick Remark
#openstack-sdks - 2019-08-06
13:27:30 rm_work kk
13:27:36 rm_work np maybe can talk after
13:31:02 mordred rm_work: yes - deprecating python-*client and moving to SDK is the right direction for sure
13:43:02 rm_work ok cool thanks :)
13:55:43 efried mordred, dtantsur: I just added that global_request_id kwarg in ksa's Session.request so you can set it per call
13:56:01 efried I haven't looked at what you're doing in sdk yet...
13:56:02 mordred efried: yah - I bumped the sdk min so we can consume it
13:56:09 efried ah cool
13:56:17 mordred (and added some hooks and stuff)
13:56:48 efried okay, I'll take a closer look.
14:04:06 dtantsur great!
14:20:43 Shrews dtantsur: oops, sorry for missing the traits fix
16:37:30 Shrews mordred: so that code... seems to delete a single object, yet from my cleanup, i see multiple objects for each image. does sdk split that up into separate objects, or is that something rax does behind the scenes?
16:38:03 mordred Shrews: it's something we do ... one sec
16:38:30 mordred Shrews: if you look in openstack/cloud/_object_store.py - and look for "use_slo"
16:39:02 mordred Shrews: and then _upload_large_object
16:40:24 mordred Shrews: https://docs.openstack.org/swift/latest/overview_large_objects.html is the docs for the feature
16:42:01 Shrews ok. i'll have to study this
16:42:23 mordred Shrews: and in delete_object in openstack/cloud/_object_store.py - we check to see if it was a static-large-object. if not, we pass the multipart-manifest = delete param which shoudl cause deleting the dynamic-large-object to delete all the segments. and if it is a static large object deleting the manifest is _supposed_ to delete the segments *I think*
16:42:45 mordred it's also possibel that with static large objects it's our responsibility to delete the segments
16:42:47 mordred or with dynamic
16:42:52 mordred I might be getting the boolean backwards
16:43:11 mordred anyway - yes - please learn things - I'm probably wrong about many parts
16:48:08 timburke ?multipart-manifest=delete works on SLOs -- it does *not* work on DLOs
16:50:55 mordred timburke: on DLOs, we're supposed to delete all of the segments ourselves, yes?
17:16:13 timburke mordred, yup, client's gotta manage the segments for DLO
18:14:03 mordred Shrews: ^^ - ok, so on DLO we're almost certainly leaking segments
18:14:53 mordred Shrews: but I don't see anywhere in nodepool where we're setting use_slo to false so I'd expect we're using SLO
18:15:44 Shrews mordred: would that come from a clouds.yaml setting?
18:16:08 Shrews rackspace profile maybe?
18:17:30 Shrews hrm, all i see is "image_api_use_tasks": true
18:29:05 mordred yeah - it's not a setting we set in clouds.yaml
18:29:26 mordred Shrews: so - a few things ...
18:30:07 mordred Shrews: I think it's entirely possible that we will leak swift objects any time sometihng _else_ goes wrong in an image upload that doesn't trigger that finally
18:30:27 mordred also - maybe we leaked a bunch of things a while back - I think there was a time when the default of use_slo was the opposite
18:31:08 mordred Shrews: maybe on use_tasks clouds we should have delete_image go and look for the object that would have been used and try to delete it?
18:31:41 Shrews maybe
18:34:19 Shrews mordred: is the slo manifest an object within itself? where would i find it?
18:35:39 Shrews wondering if i can view it from the openstack client. all i see are the segments in 'images'
18:40:27 mordred Shrews: it is an object
18:40:55 mordred Shrews: iirc, there should be a bunch of segment objects like foo/123456 - and then there should be a manifest object foo
18:41:36 mordred which should contain a json document pointing to all of the segment objects
18:47:41 Shrews mordred: ah, maybe something...
18:47:46 Shrews openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://storage101.ord1.clouddrive.com/v1/MossoCloudFS_28a933f9-900d-40b8-9de3-deb24d213aa2/images/opensuse-15-1564920980?multipart-manifest=put, Bad Request
18:48:09 Shrews mordred: if we fail to save the manifest, i bet we don't (can't?) delete the segments
18:49:25 Shrews that one happened 2 days ago, and is the only image segments i did not delete in ORD because it was too recent
18:50:21 Shrews mordred: so feeling like your first suspicion might be viable
18:51:29 mordred Shrews: ah - yeah
18:52:11 mordred Shrews: sounds like we should maybe retry the manifest put - and if it exceeds retries maybe unwind deleting teh segments since we do have the list of segments in the manifest at that point?
18:52:57 Shrews mordred: yup
18:53:30 mordred Shrews: also - a retry at that point is WAY cheaper than a hard-fail followed by re-uploading all of the segments later
18:56:28 Shrews totally
18:56:47 Shrews will see what kind of code i can barf out after the meeting
18:57:49 Shrews mordred: also, i'm guessing the 'images_segments' container is an old artifact? doesn't seem to be used
18:58:46 Shrews it exists in rax, but empty across the board
19:08:41 mordred yeah - I thnk old
19:12:06 mordred efried: dammit. your sensible request for tests has caused me to remember that something else is broken and find a whole new thing is broken
19:17:20 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807
19:17:45 mordred efried, dtantsur|afk: ^^ that adds a test and fixes the other issues - and also 2 things are borked that I would like to get fixed
19:49:22 efried mordred: ack, and #sorrynotsorry
20:00:53 mordred Shrews: we're naming the swift object after the image name, right?
20:01:20 Shrews mordred: i believe so
20:01:33 mordred Shrews: so we could also do a scan of objects matching the pattern ${imagename}/${index} on image delete perhaps?
20:02:26 Shrews mordred: well, yeah, but why? as an extra precaution?
20:03:49 efried mordred: reviewed
20:05:29 Shrews mordred: unless in rax, the images automatically reference the swift object? i had assumed that there was some step after uploading to make it a proper "image", but maybe that's not correct
20:07:40 mordred Shrews: well - there's no swift object anywhere but rax
20:07:52 mordred Shrews: but yeah - extra precaution - probably not worth the energy
#openstack-sdks - 2019-08-07
07:52:48 openstackgerrit Adrian Turjak proposed openstack/keystoneauth master: add support for auth_receipts and multi-method auth https://review.opendev.org/675049
07:59:10 openstackgerrit Adrian Turjak proposed openstack/keystoneauth master: add support for auth_receipts and multi-method auth https://review.opendev.org/675049
10:10:06 dtantsur mordred: we don't have anything to build swift temporary URLs in SDK, do we?
10:13:05 dtantsur we could probably just copy https://opendev.org/openstack/python-swiftclient/src/branch/master/swiftclient/utils.py#L71 as it is :)
13:04:47 mordred dtantsur: we've got support for form_post which uses the temp-url-key infrastructure
13:05:04 mordred dtantsur: so yeah, we could probably lift in that code for generating temp_urls themselves pretty easily
13:05:09 openstackgerrit Loan Harrouin proposed openstack/openstacksdk master: Add get_actions for servers https://review.opendev.org/675113
13:05:55 dtantsur makes sense. it seems that the code in swiftclient doesn't depend on anything specific to swiftclient
13:08:40 openstackgerrit Loan Harrouin proposed openstack/openstacksdk master: Add get_actions for server resource https://review.opendev.org/675113
13:22:30 edleafe Who knew that our API work extended to snail mail? https://twitter.com/shawnacscott/status/1158813468704698368
13:24:26 dtantsur oh lol
13:39:20 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807
13:39:20 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Initalize pool_executor so close works https://review.opendev.org/675120
13:39:21 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Fix discovery cache sharing https://review.opendev.org/675121
13:39:38 mordred dtantsur, efried: ^^ there we go - with discovery cache fixed and also with context manager fixed
13:46:21 Shrews mordred: do we have any existing tests in sdk that exercise the large object code? I'm not finding any
13:46:37 mordred Shrews: yes - becuase they sucked to write
13:46:45 Shrews :)
13:46:56 Shrews i must be looking in the wrong places then
13:48:09 mordred Shrews: test_create_dynamic_large_object and test_create_static_large_object in openstack/tests/unit/cloud/test_object.py
13:48:26 Shrews ah ha
13:48:33 mordred also test_object_segment_retry_failure
13:48:53 Shrews i forgot we still have mixed tests
13:52:47 mordred yeah. one of these days it's all going to be perfect and shiny
14:03:12 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Fix discovery cache sharing https://review.opendev.org/675121
14:03:12 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807
14:04:02 mordred Shrews: if you have a sec, the first two in that stack are bugfixes I noticed while working on the third for efried
14:04:52 openstackgerrit Loan Harrouin proposed openstack/openstacksdk master: Add get_actions for server resource https://review.opendev.org/675113
14:05:19 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807
14:05:30 mordred efried: you are quick at re-reviewing things
14:06:39 mordred dtantsur: if we do lift in the temp-url stuff, we should probably look to see what from the form_post method we can reuse. (we're using that form-post method in zuul log uploading to swift, so I'm pretty sure it's solid)
14:07:06 dtantsur makes sense, yeah

Earlier   Later