| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-sdks - 2018-07-24 | |||
| 20:38:11 | notmyname | corvus: mordred: so the general answer is that swift will store the bytestream you send it and also headers that may have some definition for clients (eg content-encoding). SLOs aren't special in that the segments are simply slices of the resulting range. swift doesn't do any interpretation of the contents of objects | |
| 20:39:05 | mordred | timburke: yah - server-side copy or always SLO both sound like good general options | |
| 20:39:45 | mordred | there's definitely a balancing act we're trying to do here with wanting SDK to DTRT and yet also providing enough knobs so that we can do the zuul log upload thing efficiently | |
| 20:41:45 | mordred | notmyname, timburke: thanks both of you - this has been super helpful | |
| 20:42:02 | corvus | ++ | |
| 20:43:46 | timburke | fwiw, swiftclient opts for the buffering thing when uploading from stdin -- i think we go up to 16MB (or something like that?) then start uploading 16MB segments. since its stdin, there's only one upload per-process, so we don't feel too bad about the memory | |
| 20:49:33 | mordred | timburke: yah - the fun part of this story is that once sdk switches to "oh, you wanted an SLO" - it does so with a pool of threads (similar to swiftuploader in swiftclient) | |
| 20:49:56 | mordred | of course, actually ... now that I think about it - that won't work for iterators that don't have seek anyway | |
| 20:50:51 | mordred | since it does it in parallel for files by opening multiple handles and seeking on them ... so to support SLO from an input stream we'll need to reengineer what we're doing anyway | |
| 20:50:53 | mordred | corvus: ^^ | |
| 20:51:38 | corvus | this may be a limited use case. we won't want to use the deflatefilter for, say, already compressed images. i've only got it set up to engage for text/ types with no encoding right now. | |
| 20:51:57 | mordred | corvus: ah - cool. | |
| 20:52:15 | mordred | corvus: I'm almost starting to feel like we should add your compressiong streaming code into sdk itself so that we can put it further down the stack | |
| 20:52:45 | mordred | corvus: like, put it around the file reads after the seek in the SLO segment uploads | |
| 20:53:07 | mordred | and make an option to create_object "compress=False" or something like that (just thinking out loud) | |
| 20:53:07 | corvus | mordred: i think it may be pretty domain-specific; i don't think it's appropriate for everything | |
| 20:53:23 | mordred | good point. oh - and also that would be the wrong place anyway | |
| 20:53:33 | notmyname | eg https://github.com/openstack/swift/blob/master/swift/common/internal_client.py#L54 ? | |
| 20:53:36 | corvus | (i actually anticipate some period of us tweaking when this gets used) | |
| 20:54:15 | openstackgerrit | Merged openstack/os-service-types master: Allow passing in service types with _ in them https://review.openstack.org/585410 | |
| 20:54:19 | corvus | notmyname: why in the world isn't that in the standard library? :) | |
| 20:55:04 | corvus | notmyname: i wrote one of those too. i may improve it now :) | |
| 20:58:52 | mordred | corvus: it seems like the 'don't run len on data' patch from above should be the only thing you'd need for the easier case - and that we can probably wait until later to deal with SLO and streamed input | |
| 21:13:32 | corvus | mordred: your patch works. though using that method as opposed to the proxy results in a HEAD request to the container before each PUT | |
| 21:19:55 | mordred | corvus: yeah. we should really cache that container status | |
| 21:20:27 | corvus | mordred: that's probably going to be an extra 1500 requests for, say, a devstack job, yeah? | |
| 21:20:31 | corvus | what's it for? | |
| 21:20:44 | mordred | corvus: create_object will create the container for you if it doesn't exist | |
| 21:20:54 | mordred | head is checking container existence | |
| 21:22:21 | corvus | mordred: i want to set some things on the container when it's created; i assumed it'd be best for me to handle that explicitly before doing the upload | |
| 21:22:37 | mordred | corvus: yes - it's totally best for you to do that | |
| 21:23:14 | mordred | corvus: is this multiple executions or a single program with a single long-lived session? | |
| 21:23:28 | corvus | mordred: a single program with 30 threads uploading in parallel | |
| 21:24:11 | mordred | corvus: hrm. there is a container cache already - it seems like it should only make one HEAD | |
| 21:24:40 | corvus | mordred: my test isn't big enough to re-use a thread. maybe they're all racing to get the first head | |
| 21:24:57 | mordred | probably so. we could make the container cache more threadsafe though | |
| 21:25:06 | corvus | mordred: i'm not yet performing my existence check -- would that prime the cache? | |
| 21:25:08 | mordred | similar to how we do for servers | |
| 21:25:13 | mordred | yes | |
| 21:25:16 | mordred | get_container | |
| 21:25:18 | mordred | will do it | |
| 21:25:29 | corvus | cool, then we may not need to do anything to resolve this. i'll plumb that code in now. | |
| 21:25:33 | mordred | cool | |
| 21:28:06 | corvus | mordred: yep that took care of it. sorry for the false alarm :) | |
| 21:32:39 | mordred | yay! | |
| 21:45:26 | openstackgerrit | Merged openstack/python-openstackclient master: Fix error with image show when image name is None https://review.openstack.org/529464 | |
| 22:48:09 | openstackgerrit | Dean Troyer proposed openstack/python-openstackclient master: Support --community in openstack image list https://review.openstack.org/565152 | |
| 23:10:52 | mordred | dtroyer: https://review.openstack.org/#/c/582334 <-- I left a novel of a comment there - I'm guessing we should probably just land it in that form since it's definitely an error it's fixing and leave making it more better to next cycle ... | |
| 23:11:05 | mordred | dtroyer: but whatever you think | |
| 23:17:07 | corvus | mordred: you said i could pass a dict to the connection constructor... how's that go again? | |
| 23:18:05 | corvus | just openstack.connect(**args) ? | |
| 23:18:57 | corvus | hrm, that gets me: keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url | |
| 23:19:48 | dtroyer | mordred: agreed, there are a number of microversion-affected changes to compute outstanding, some merged already, all of them are likely in this situation, this gets things working to some extent, we'll clean it up when we SDK-enable things and that may influence the order we do that work in... | |
| 23:26:54 | corvus | mordred: oh apparently that's because profile is ignored, so i need to add auth_url in there | |
| 23:28:41 | corvus | mordred: reading https://docs.openstack.org/openstacksdk/latest/user/transition_from_profile.html it's not clear how to accomplish what i want (construct a connection from a dict, using profile=vexxhost and not specifying auth_url) | |
| #openstack-sdks - 2018-07-25 | |||
| 00:42:32 | openstackgerrit | Merged openstack/python-openstackclient master: Support filtering port with IP address substring https://review.openstack.org/532324 | |
| 00:52:39 | openstackgerrit | Merged openstack/python-openstackclient master: compute: host: expand kwargs in host_set() call https://review.openstack.org/583225 | |
| 00:52:41 | openstackgerrit | Merged openstack/python-openstackclient master: Implement support for registered limits https://review.openstack.org/574475 | |
| 00:52:42 | openstackgerrit | Merged openstack/python-openstackclient master: Implement support for project limits https://review.openstack.org/575211 | |
| 01:19:08 | openstackgerrit | Artom Lifshitz proposed openstack/openstacksdk master: Send disk_over_commit if nova api < 2.25 https://review.openstack.org/585579 | |
| 02:49:03 | openstackgerrit | Merged openstack/python-openstackclient master: Support --community in openstack image list https://review.openstack.org/565152 | |
| 13:20:15 | openstackgerrit | Ryan Brady proposed openstack-infra/shade master: Add support for static routes https://review.openstack.org/580661 | |
| 14:32:36 | mordred | corvus: sorry about that ... profile is getting consumed by the old deprecated soon-to-be-removed old sdk profile | |
| 14:33:10 | mordred | corvus: the easiest way around it would be to construct a CloudRegion from a dict and pass that to the connection ctr | |
| 14:34:02 | corvus | mordred: it doesn't look like cloudregion would accept the clouds.yaml form of arguments: https://docs.openstack.org/openstacksdk/latest/user/config/reference.html#openstack.config.cloud_region.CloudRegion | |
| 14:34:10 | corvus | mordred: i don't see 'profile' or 'auth' there...? | |
| 14:35:22 | mordred | corvus: you are right. this is a bug | |
| 14:35:37 | corvus | mordred: what's the "openstack_config" argument for that? | |
| 14:36:31 | mordred | well, that's the bug - I do not think we support passing profile as an argument- itlooks like it's only procesed during config file reading | |
| 14:39:24 | mordred | corvus: oh no, it should work | |
| 14:39:28 | mordred | corvus: do this: | |
| 14:41:35 | mordred | corvus: from openstack.config import loader ; config = loader.OpenStackConfig().get_one(**args) ; conn = openstack.connection.Connection(config=config) | |
| 14:42:48 | corvus | mordred: still wants auth_url | |
| 14:43:32 | corvus | File \"/home/corvus/git/openstack-infra/zuul-jobs/.tox/py35/lib/python3.5/site-packages/keystoneauth1/loading/base.py\", line 162, in load_from_options\n raise exceptions.MissingRequiredOptions(missing_required)\nkeystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url\n", | |
| 14:44:09 | corvus | that's from the get_one line | |
| 14:45:18 | mordred | poo. lemme poke further 0 sorry about that | |
| 14:48:39 | mordred | corvus: ok. I jave reproduced that lcaly | |
| 14:49:39 | dtantsur | folks, how do I deal with "The service catalog is empty" from session.get_endpoint of the service catalog is NOT empty? | |
| 14:49:50 | dtantsur | in what other cases can this happen? | |
| 14:50:02 | mordred | dtantsur: I've never seen that | |
| 14:50:08 | dtantsur | :( | |
| 14:50:19 | mordred | dtantsur: are you sure you have a token/catalog? | |
| 14:50:48 | dtantsur | mordred: the session is created with keystoneauth1.identity.v3.Token, the token comes from Mistral, no idea how it creates it | |
| 14:51:01 | dtantsur | it's tripleo, there is nothing I can be sure about :D | |
| 14:51:35 | mordred | dtantsur: it's possible the catalog got stripped from the token if it came from middleware | |
| 14:52:00 | dtantsur | mordred: the token - is it a string or an object? I think I get a string | |
| 14:52:01 | mordred | people like doing that to reduce size of the payload for reasons I don't really understand | |
| 14:52:20 | dtantsur | and now a funny thing: if I wrap it in an Adapter, it seems to work Oo | |
| 14:52:29 | mordred | weird | |
| 14:53:54 | dtantsur | extremely weird | |
| 14:57:02 | dtantsur | so, this apparently blows up https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identity/base.py#L275 | |
| 15:02:16 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Support passing profile to get_one https://review.openstack.org/585780 | |
| 15:02:24 | mordred | corvus: ^^ | |
| 15:04:06 | dtantsur | I guess I short report it against keystoneauth. it should not report EmptyCatalog if it's not empty in reality. | |
| 15:04:24 | corvus | mordred: that works, thanks! | |
| 15:04:38 | dtantsur | s/short/should/ (WUT) | |
| 15:05:13 | mordred | corvus: yay! | |
| 15:13:37 | dtantsur | filed https://bugs.launchpad.net/keystoneauth/+bug/1783590 | |
| 15:13:38 | openstack | Launchpad bug 1783590 in keystoneauth "EmptyCatalog raised when the catalog is not empty" [Undecided,New] | |
| 16:10:59 | dtantsur | mordred: do you know if I can pass the catalog to keystoneauth somehow? given that Token auth is, well, broken apparently in this regard | |