| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-sdks - 2018-07-24 | |||
| 16:01:41 | Shrews | yeah | |
| 16:01:47 | samueldmq | or is there a special thing that was setup for devstack | |
| 16:01:48 | samueldmq | kk | |
| 16:02:04 | Shrews | and the tags were to selectively choose which things you wanted to test (in case the cloud didn't support something) | |
| 16:02:49 | samueldmq | gotcha | |
| 16:03:10 | Shrews | looks like we carried that over to sdk, but i haven't tested it there | |
| 16:03:23 | samueldmq | I'll take a look to get familiar | |
| 16:03:30 | samueldmq | and see what works and what doesn't | |
| 16:04:14 | Shrews | \o/ | |
| 16:04:29 | samueldmq | :D | |
| 17:38:52 | openstackgerrit | Merged openstack/openstacksdk master: Run ansible tests against specific public cloud https://review.openstack.org/582459 | |
| 18:22:44 | openstackgerrit | Merged openstack/keystoneauth master: Add ability to filter version data by service-type https://review.openstack.org/585029 | |
| 18:37:40 | mordred | Shrews, dtroyer, dhellmann: feel like reviewing some os-service-types patches? (https://review.openstack.org/#/q/project:openstack/os-service-types+status:open) | |
| 18:51:02 | Shrews | i don't even know what that repo is (though i can infer from the name) lol | |
| 18:54:48 | mordred | Shrews: :) | |
| 19:32:05 | corvus | mordred: i'm having trouble using the new data= parameter to create_object: | |
| 19:32:07 | corvus | TypeError: object of type 'DeflateFilter' has no len() | |
| 19:32:07 | corvus | file_size = len(data) | |
| 19:32:07 | corvus | File "/home/corvus/git/openstack-infra/zuul-jobs/.tox/py35/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7525, in create_object | |
| 19:32:45 | corvus | mordred: the session.put code i was using before handled data being an iterator | |
| 19:33:40 | mordred | corvus: ah! hrm. SO ... | |
| 19:34:32 | corvus | er, cloud.object_store.put which i guess is the proxy put. which, i think, is basically going to call session.put. you get the idea. | |
| 19:34:33 | mordred | corvus: that len() is there in service of figuring out whether the data to be uploaded needs to be uploaded as multiple segments | |
| 19:34:54 | corvus | mordred: i believe httplib handles that automatically | |
| 19:35:31 | mordred | well, not for swift it doesn't | |
| 19:35:36 | corvus | (briefly, if it is an iterator with no len, it gets uploaded in multiple segments) | |
| 19:35:45 | corvus | sorry, chunked encoding | |
| 19:35:55 | corvus | are you talking about something else? | |
| 19:36:07 | mordred | yah - this is about uploading to multiple swift objects if the object size is > max_objet_size | |
| 19:36:23 | mordred | and then creating either a DLO or SLO object out of that | |
| 19:36:30 | mordred | (dynamic large object or static large object) | |
| 19:36:45 | corvus | mordred: that's something you have to decide on before you upload? | |
| 19:37:21 | mordred | yup- it impacts how you upload things - and also causes sdk to spawn up a set of worker threads to handle it if it's needed | |
| 19:38:54 | corvus | that sounds fundamentally incompatible with streaming compression | |
| 19:39:32 | mordred | well ... | |
| 19:40:12 | mordred | I believe each segment upload itself could be uploaded with streaming compression | |
| 19:40:40 | mordred | we should ask notmyname about it though ... | |
| 19:41:43 | mordred | corvus: the simple approach could be skip the length/segment step if data is passed as a parameter | |
| 19:42:17 | mordred | corvus: fwiw, vexxhost has 'max_file_size': 5368709120 | |
| 19:42:22 | corvus | mordred: just assume the object isn't too large? | |
| 19:42:44 | mordred | yeah. I mean, the 'standard' max_file_size is what vexxhost has -which is 5G | |
| 19:43:59 | mordred | corvus: but - the way these work is one of two ways (each resulting in a set of discreet objects being uploaded) | |
| 19:44:50 | mordred | in both cases the data is uploaded in a set of 'segment' objects. in static large objects there is then a manifest object that lists the segment objects and the order in which they should be served | |
| 19:45:33 | mordred | in dynamic the segment objects follow a naming scheme with an empty manifest object (so you don't have to keep track of the list and then upload the manifest object) | |
| 19:45:52 | mordred | each of the segment objects is a completely independent swift object, so I'd expect uploading to them with deflate would work | |
| 19:46:40 | mordred | oh - then when you fetch the objects, as a client, you just fetch the manifest object, and swift knows that what you want to do is stream the concatenation of the segment objects | |
| 19:47:11 | mordred | it's the download that I'm not sure what happens | |
| 19:47:42 | corvus | the download? | |
| 19:47:49 | mordred | the fetch | |
| 19:48:32 | corvus | i'm confused, i thought you just described what happens in that case | |
| 19:48:58 | mordred | yes - in the standard case - I do not know how it interacts with streaming compression | |
| 19:49:12 | mordred | largely becaues I've never tested it - I'd imagine it would DTRT | |
| 19:49:17 | mordred | but don't actually know | |
| 19:49:26 | corvus | oh i see what you're saying | |
| 19:50:54 | mordred | corvus: out of curiosity - what is the iterator that has the data in it? | |
| 19:51:28 | mordred | oh. the deflatefilter | |
| 19:51:35 | mordred | *duh* it's in the traceback | |
| 19:52:29 | corvus | yeah. it's a thing i wrote that compresses 16k at a time and produces an unknown amount of data each chunk | |
| 19:55:14 | mordred | corvus: well - here's another question then ... | |
| 19:57:26 | mordred | corvus: does max_file_size apply to the compresesd or uncompresed value | |
| 19:57:41 | corvus | excellent question! it's all very vague | |
| 19:58:21 | corvus | i'm assuming, for the moment, that swift doesn't attempt to do any decoding. so probably it applies to the compressed value. | |
| 19:58:29 | corvus | (i'm assuming it is stored in the way i send it) | |
| 19:59:04 | corvus | i have little factual basis for making that assumption | |
| 20:00:39 | mordred | yeah | |
| 20:00:52 | mordred | corvus: for now, if you make a len method on your object that just returns 0 | |
| 20:00:57 | mordred | if should get you past your issue | |
| 20:01:30 | mordred | file_size is only used a little later to determine if file_size < max_segment_size ... | |
| 20:01:43 | mordred | so if you return 0, the code pass data on through directly | |
| 20:02:33 | openstackgerrit | Merged openstack/os-service-types master: Updated from OpenStack Service Type Authority https://review.openstack.org/585062 | |
| 20:02:33 | openstackgerrit | Merged openstack/os-service-types master: Switch to stestr https://review.openstack.org/585349 | |
| 20:03:03 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data https://review.openstack.org/585532 | |
| 20:03:14 | mordred | corvus: ^^ also, I think that should at least do a workaround | |
| 20:03:42 | corvus | mordred: though does it get passed through to the underlying session method? i need that not to have a len method, or it'll avoid chunked uploads. | |
| 20:04:54 | corvus | mordred: i'll try your patch out in a little bit | |
| 20:05:13 | mordred | corvus: ah. | |
| 20:05:48 | mordred | corvus: oh for the love of ... | |
| 20:06:34 | mordred | corvus: we don't use file_size in the data path at all | |
| 20:07:58 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data https://review.openstack.org/585532 | |
| 20:08:18 | mordred | corvus: ^^ that should fix it more appropriately | |
| 20:09:08 | corvus | mordred: ack. i'll give it a spin when i finish dealing with trailing slashes :) | |
| 20:09:19 | mordred | corvus: trailing slashes are the worst | |
| 20:09:29 | mordred | corvus: they're almost as terrible as INCORRECT WHITESPACE | |
| 20:09:38 | notmyname | mordred: hello | |
| 20:09:45 | mordred | yay it's notmyname ! | |
| 20:10:30 | mordred | notmyname: we have several questions - I will try to ask them in some semblance of order | |
| 20:11:41 | mordred | notmyname: first of all, if you are uploading a large object and you want to upload compressed with a deflate header - does the compressed or uncompressed size count towards max_file_size? | |
| 20:12:36 | corvus | specifically, "content-encoding: deflate" is what's happening here | |
| 20:12:46 | mordred | yeah. corvus is likely to say smarter words than me | |
| 20:14:55 | notmyname | that is likely to be dependent on something between the client and swift itself | |
| 20:15:15 | notmyname | there's not anything in swift that will accept compressed data and store in uncompressed | |
| 20:15:39 | notmyname | however, if you've got some caching thing (CDN or otherwise) that understands those headers, that work can be done there | |
| 20:16:02 | notmyname | swift will happily store the content-encoding header, if you send it, and return it on a read request. | |
| 20:17:49 | notmyname | lol, rackspace took my name off the author byline ;-) https://blog.rackspace.com/cloud-files-cdn-compresses-at-the-edge | |
| 20:17:57 | mordred | ok. so - what about SLO/DLO objects and concatenation? if the segments are uploaded compressed and swift doesnt' natively do any uncompression, I'm guessing that could get weird for the read? | |
| 20:18:08 | mordred | notmyname: haha | |
| 20:18:45 | mordred | and by 'get weird' I mean 'not work" | |
| 20:19:31 | notmyname | let me try something... | |
| 20:21:17 | timburke | i'd expect you'd want to open the large object, stream it through a compressor, and break out segments from that compressed stream. that way when you go to download the large object, you'll get a singular large, compressed stream | |
| 20:23:05 | timburke | breaking the large object into segments first then compressing is unlikely to end well | |