| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-sdks - 2018-07-24 | |||
| 15:54:06 | mordred | in some clouds we may not have admin, for instance, so making a new project and testing keystone admin functions should all be skipped | |
| 15:54:08 | samueldmq | to allow oyu input the cloud to talk against | |
| 15:54:11 | mordred | but we could still test that booting a vm works | |
| 15:54:15 | samueldmq | when running tests right | |
| 15:55:22 | mordred | the tests already will skip if the service doesn't exist - then we have a devstack-specific test that responds to env vars and fails if an envvar asserts a service should exist but the shade tests don't find it | |
| 15:55:31 | mordred | that way we can make sure we don't accidentally fail open | |
| 15:55:36 | mordred | i the gate | |
| 15:55:48 | mordred | so we could probably expand on that somehow | |
| 15:56:10 | Shrews | mordred: lgtm | |
| 15:56:12 | mordred | to allow manual running against public clouds to be more flexible without sacrificing gate stuff | |
| 15:56:14 | mordred | Shrews: woot | |
| 15:56:18 | samueldmq | mordred: how are those vars set up then? | |
| 15:56:29 | samueldmq | I assume that cn be automatic by looking up the service catalog | |
| 15:56:34 | samueldmq | but would require some work | |
| 15:56:48 | mordred | well - "has_service('network')" uses the service catalog | |
| 15:56:54 | mordred | but the env var is defined in the test job | |
| 15:57:18 | mordred | so the test job (which sets up the services in the first place) also sets vars that say "I believe I set up magnum" | |
| 15:57:32 | mordred | then the shade test says "the test config expects magnum, do I see it in the catalog" | |
| 15:57:41 | samueldmq | ah makes sense | |
| 15:57:48 | samueldmq | but none of that apply to manual run | |
| 15:57:51 | mordred | exactly | |
| 15:58:01 | samueldmq | also, runnig against real public clouds wouldn't require any setup | |
| 15:58:08 | mordred | that's the theory :) | |
| 15:58:09 | samueldmq | openstack/ocnfig that is | |
| 15:58:25 | mordred | in practice, I think there are still a good amount of code in fixtures that assumes we're running against a devstack | |
| 15:58:34 | mordred | and could be make more smarter, we just haven't had the time yet | |
| 15:58:38 | Shrews | yeah | |
| 15:58:39 | samueldmq | "whatever that can happen will happen" someone said | |
| 15:58:45 | mordred | ++ | |
| 15:58:56 | samueldmq | and then shade exists :-) | |
| 15:59:33 | samueldmq | kk I have some info to mull over | |
| 16:00:00 | samueldmq | I'm happy I got an old laptop running linux again, was bored using windows all the time for company stuff | |
| 16:00:01 | Shrews | somewhere in shade i had a script that allowed us to run those tests against another (non-devstack) cloud | |
| 16:00:13 | samueldmq | which means I am expecting to start having fun around here | |
| 16:00:28 | mordred | samueldmq: yay! | |
| 16:00:29 | samueldmq | Shrews: that'd be a good start | |
| 16:00:32 | mordred | Shrews: yeah- I remember that ... | |
| 16:00:34 | samueldmq | \o/ | |
| 16:00:48 | Shrews | oh, even tox support for that | |
| 16:00:56 | Shrews | tox -e ansible -- -c cloudX [TAG ...] | |
| 16:01:05 | Shrews | how clever i was | |
| 16:01:37 | samueldmq | and that looks up on the regular /opt/openstack/clouds.yaml? | |
| 16:01:41 | Shrews | yeah | |
| 16:01:47 | samueldmq | or is there a special thing that was setup for devstack | |
| 16:01:48 | samueldmq | kk | |
| 16:02:04 | Shrews | and the tags were to selectively choose which things you wanted to test (in case the cloud didn't support something) | |
| 16:02:49 | samueldmq | gotcha | |
| 16:03:10 | Shrews | looks like we carried that over to sdk, but i haven't tested it there | |
| 16:03:23 | samueldmq | I'll take a look to get familiar | |
| 16:03:30 | samueldmq | and see what works and what doesn't | |
| 16:04:14 | Shrews | \o/ | |
| 16:04:29 | samueldmq | :D | |
| 17:38:52 | openstackgerrit | Merged openstack/openstacksdk master: Run ansible tests against specific public cloud https://review.openstack.org/582459 | |
| 18:22:44 | openstackgerrit | Merged openstack/keystoneauth master: Add ability to filter version data by service-type https://review.openstack.org/585029 | |
| 18:37:40 | mordred | Shrews, dtroyer, dhellmann: feel like reviewing some os-service-types patches? (https://review.openstack.org/#/q/project:openstack/os-service-types+status:open) | |
| 18:51:02 | Shrews | i don't even know what that repo is (though i can infer from the name) lol | |
| 18:54:48 | mordred | Shrews: :) | |
| 19:32:05 | corvus | mordred: i'm having trouble using the new data= parameter to create_object: | |
| 19:32:07 | corvus | TypeError: object of type 'DeflateFilter' has no len() | |
| 19:32:07 | corvus | file_size = len(data) | |
| 19:32:07 | corvus | File "/home/corvus/git/openstack-infra/zuul-jobs/.tox/py35/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7525, in create_object | |
| 19:32:45 | corvus | mordred: the session.put code i was using before handled data being an iterator | |
| 19:33:40 | mordred | corvus: ah! hrm. SO ... | |
| 19:34:32 | corvus | er, cloud.object_store.put which i guess is the proxy put. which, i think, is basically going to call session.put. you get the idea. | |
| 19:34:33 | mordred | corvus: that len() is there in service of figuring out whether the data to be uploaded needs to be uploaded as multiple segments | |
| 19:34:54 | corvus | mordred: i believe httplib handles that automatically | |
| 19:35:31 | mordred | well, not for swift it doesn't | |
| 19:35:36 | corvus | (briefly, if it is an iterator with no len, it gets uploaded in multiple segments) | |
| 19:35:45 | corvus | sorry, chunked encoding | |
| 19:35:55 | corvus | are you talking about something else? | |
| 19:36:07 | mordred | yah - this is about uploading to multiple swift objects if the object size is > max_objet_size | |
| 19:36:23 | mordred | and then creating either a DLO or SLO object out of that | |
| 19:36:30 | mordred | (dynamic large object or static large object) | |
| 19:36:45 | corvus | mordred: that's something you have to decide on before you upload? | |
| 19:37:21 | mordred | yup- it impacts how you upload things - and also causes sdk to spawn up a set of worker threads to handle it if it's needed | |
| 19:38:54 | corvus | that sounds fundamentally incompatible with streaming compression | |
| 19:39:32 | mordred | well ... | |
| 19:40:12 | mordred | I believe each segment upload itself could be uploaded with streaming compression | |
| 19:40:40 | mordred | we should ask notmyname about it though ... | |
| 19:41:43 | mordred | corvus: the simple approach could be skip the length/segment step if data is passed as a parameter | |
| 19:42:17 | mordred | corvus: fwiw, vexxhost has 'max_file_size': 5368709120 | |
| 19:42:22 | corvus | mordred: just assume the object isn't too large? | |
| 19:42:44 | mordred | yeah. I mean, the 'standard' max_file_size is what vexxhost has -which is 5G | |
| 19:43:59 | mordred | corvus: but - the way these work is one of two ways (each resulting in a set of discreet objects being uploaded) | |
| 19:44:50 | mordred | in both cases the data is uploaded in a set of 'segment' objects. in static large objects there is then a manifest object that lists the segment objects and the order in which they should be served | |
| 19:45:33 | mordred | in dynamic the segment objects follow a naming scheme with an empty manifest object (so you don't have to keep track of the list and then upload the manifest object) | |
| 19:45:52 | mordred | each of the segment objects is a completely independent swift object, so I'd expect uploading to them with deflate would work | |
| 19:46:40 | mordred | oh - then when you fetch the objects, as a client, you just fetch the manifest object, and swift knows that what you want to do is stream the concatenation of the segment objects | |
| 19:47:11 | mordred | it's the download that I'm not sure what happens | |
| 19:47:42 | corvus | the download? | |
| 19:47:49 | mordred | the fetch | |
| 19:48:32 | corvus | i'm confused, i thought you just described what happens in that case | |
| 19:48:58 | mordred | yes - in the standard case - I do not know how it interacts with streaming compression | |
| 19:49:12 | mordred | largely becaues I've never tested it - I'd imagine it would DTRT | |
| 19:49:17 | mordred | but don't actually know | |
| 19:49:26 | corvus | oh i see what you're saying | |
| 19:50:54 | mordred | corvus: out of curiosity - what is the iterator that has the data in it? | |
| 19:51:28 | mordred | oh. the deflatefilter | |
| 19:51:35 | mordred | *duh* it's in the traceback | |
| 19:52:29 | corvus | yeah. it's a thing i wrote that compresses 16k at a time and produces an unknown amount of data each chunk | |