| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-sdks - 2018-07-24 | |||
| 15:58:09 | samueldmq | openstack/ocnfig that is | |
| 15:58:25 | mordred | in practice, I think there are still a good amount of code in fixtures that assumes we're running against a devstack | |
| 15:58:34 | mordred | and could be make more smarter, we just haven't had the time yet | |
| 15:58:38 | Shrews | yeah | |
| 15:58:39 | samueldmq | "whatever that can happen will happen" someone said | |
| 15:58:45 | mordred | ++ | |
| 15:58:56 | samueldmq | and then shade exists :-) | |
| 15:59:33 | samueldmq | kk I have some info to mull over | |
| 16:00:00 | samueldmq | I'm happy I got an old laptop running linux again, was bored using windows all the time for company stuff | |
| 16:00:01 | Shrews | somewhere in shade i had a script that allowed us to run those tests against another (non-devstack) cloud | |
| 16:00:13 | samueldmq | which means I am expecting to start having fun around here | |
| 16:00:28 | mordred | samueldmq: yay! | |
| 16:00:29 | samueldmq | Shrews: that'd be a good start | |
| 16:00:32 | mordred | Shrews: yeah- I remember that ... | |
| 16:00:34 | samueldmq | \o/ | |
| 16:00:48 | Shrews | oh, even tox support for that | |
| 16:00:56 | Shrews | tox -e ansible -- -c cloudX [TAG ...] | |
| 16:01:05 | Shrews | how clever i was | |
| 16:01:37 | samueldmq | and that looks up on the regular /opt/openstack/clouds.yaml? | |
| 16:01:41 | Shrews | yeah | |
| 16:01:47 | samueldmq | or is there a special thing that was setup for devstack | |
| 16:01:48 | samueldmq | kk | |
| 16:02:04 | Shrews | and the tags were to selectively choose which things you wanted to test (in case the cloud didn't support something) | |
| 16:02:49 | samueldmq | gotcha | |
| 16:03:10 | Shrews | looks like we carried that over to sdk, but i haven't tested it there | |
| 16:03:23 | samueldmq | I'll take a look to get familiar | |
| 16:03:30 | samueldmq | and see what works and what doesn't | |
| 16:04:14 | Shrews | \o/ | |
| 16:04:29 | samueldmq | :D | |
| 17:38:52 | openstackgerrit | Merged openstack/openstacksdk master: Run ansible tests against specific public cloud https://review.openstack.org/582459 | |
| 18:22:44 | openstackgerrit | Merged openstack/keystoneauth master: Add ability to filter version data by service-type https://review.openstack.org/585029 | |
| 18:37:40 | mordred | Shrews, dtroyer, dhellmann: feel like reviewing some os-service-types patches? (https://review.openstack.org/#/q/project:openstack/os-service-types+status:open) | |
| 18:51:02 | Shrews | i don't even know what that repo is (though i can infer from the name) lol | |
| 18:54:48 | mordred | Shrews: :) | |
| 19:32:05 | corvus | mordred: i'm having trouble using the new data= parameter to create_object: | |
| 19:32:07 | corvus | File "/home/corvus/git/openstack-infra/zuul-jobs/.tox/py35/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7525, in create_object | |
| 19:32:07 | corvus | file_size = len(data) | |
| 19:32:07 | corvus | TypeError: object of type 'DeflateFilter' has no len() | |
| 19:32:45 | corvus | mordred: the session.put code i was using before handled data being an iterator | |
| 19:33:40 | mordred | corvus: ah! hrm. SO ... | |
| 19:34:32 | corvus | er, cloud.object_store.put which i guess is the proxy put. which, i think, is basically going to call session.put. you get the idea. | |
| 19:34:33 | mordred | corvus: that len() is there in service of figuring out whether the data to be uploaded needs to be uploaded as multiple segments | |
| 19:34:54 | corvus | mordred: i believe httplib handles that automatically | |
| 19:35:31 | mordred | well, not for swift it doesn't | |
| 19:35:36 | corvus | (briefly, if it is an iterator with no len, it gets uploaded in multiple segments) | |
| 19:35:45 | corvus | sorry, chunked encoding | |
| 19:35:55 | corvus | are you talking about something else? | |
| 19:36:07 | mordred | yah - this is about uploading to multiple swift objects if the object size is > max_objet_size | |
| 19:36:23 | mordred | and then creating either a DLO or SLO object out of that | |
| 19:36:30 | mordred | (dynamic large object or static large object) | |
| 19:36:45 | corvus | mordred: that's something you have to decide on before you upload? | |
| 19:37:21 | mordred | yup- it impacts how you upload things - and also causes sdk to spawn up a set of worker threads to handle it if it's needed | |
| 19:38:54 | corvus | that sounds fundamentally incompatible with streaming compression | |
| 19:39:32 | mordred | well ... | |
| 19:40:12 | mordred | I believe each segment upload itself could be uploaded with streaming compression | |
| 19:40:40 | mordred | we should ask notmyname about it though ... | |
| 19:41:43 | mordred | corvus: the simple approach could be skip the length/segment step if data is passed as a parameter | |
| 19:42:17 | mordred | corvus: fwiw, vexxhost has 'max_file_size': 5368709120 | |
| 19:42:22 | corvus | mordred: just assume the object isn't too large? | |
| 19:42:44 | mordred | yeah. I mean, the 'standard' max_file_size is what vexxhost has -which is 5G | |
| 19:43:59 | mordred | corvus: but - the way these work is one of two ways (each resulting in a set of discreet objects being uploaded) | |
| 19:44:50 | mordred | in both cases the data is uploaded in a set of 'segment' objects. in static large objects there is then a manifest object that lists the segment objects and the order in which they should be served | |
| 19:45:33 | mordred | in dynamic the segment objects follow a naming scheme with an empty manifest object (so you don't have to keep track of the list and then upload the manifest object) | |
| 19:45:52 | mordred | each of the segment objects is a completely independent swift object, so I'd expect uploading to them with deflate would work | |
| 19:46:40 | mordred | oh - then when you fetch the objects, as a client, you just fetch the manifest object, and swift knows that what you want to do is stream the concatenation of the segment objects | |
| 19:47:11 | mordred | it's the download that I'm not sure what happens | |
| 19:47:42 | corvus | the download? | |
| 19:47:49 | mordred | the fetch | |
| 19:48:32 | corvus | i'm confused, i thought you just described what happens in that case | |
| 19:48:58 | mordred | yes - in the standard case - I do not know how it interacts with streaming compression | |
| 19:49:12 | mordred | largely becaues I've never tested it - I'd imagine it would DTRT | |
| 19:49:17 | mordred | but don't actually know | |
| 19:49:26 | corvus | oh i see what you're saying | |
| 19:50:54 | mordred | corvus: out of curiosity - what is the iterator that has the data in it? | |
| 19:51:28 | mordred | oh. the deflatefilter | |
| 19:51:35 | mordred | *duh* it's in the traceback | |
| 19:52:29 | corvus | yeah. it's a thing i wrote that compresses 16k at a time and produces an unknown amount of data each chunk | |
| 19:55:14 | mordred | corvus: well - here's another question then ... | |
| 19:57:26 | mordred | corvus: does max_file_size apply to the compresesd or uncompresed value | |
| 19:57:41 | corvus | excellent question! it's all very vague | |
| 19:58:21 | corvus | i'm assuming, for the moment, that swift doesn't attempt to do any decoding. so probably it applies to the compressed value. | |
| 19:58:29 | corvus | (i'm assuming it is stored in the way i send it) | |
| 19:59:04 | corvus | i have little factual basis for making that assumption | |
| 20:00:39 | mordred | yeah | |
| 20:00:52 | mordred | corvus: for now, if you make a len method on your object that just returns 0 | |
| 20:00:57 | mordred | if should get you past your issue | |
| 20:01:30 | mordred | file_size is only used a little later to determine if file_size < max_segment_size ... | |
| 20:01:43 | mordred | so if you return 0, the code pass data on through directly | |
| 20:02:33 | openstackgerrit | Merged openstack/os-service-types master: Switch to stestr https://review.openstack.org/585349 | |
| 20:02:33 | openstackgerrit | Merged openstack/os-service-types master: Updated from OpenStack Service Type Authority https://review.openstack.org/585062 | |
| 20:03:03 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data https://review.openstack.org/585532 | |
| 20:03:14 | mordred | corvus: ^^ also, I think that should at least do a workaround | |
| 20:03:42 | corvus | mordred: though does it get passed through to the underlying session method? i need that not to have a len method, or it'll avoid chunked uploads. | |
| 20:04:54 | corvus | mordred: i'll try your patch out in a little bit | |
| 20:05:13 | mordred | corvus: ah. | |
| 20:05:48 | mordred | corvus: oh for the love of ... | |
| 20:06:34 | mordred | corvus: we don't use file_size in the data path at all | |
| 20:07:58 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data https://review.openstack.org/585532 | |
| 20:08:18 | mordred | corvus: ^^ that should fix it more appropriately | |
| 20:09:08 | corvus | mordred: ack. i'll give it a spin when i finish dealing with trailing slashes :) | |