Earlier  
Later
New Search
Posted Nick Remark
#openstack-sdks - 2018-07-19
17:22:14 Shrews why upload data you don't have?
17:22:50 corvus Shrews: continued conversation from yesterday about proxy vs connection
17:23:01 corvus Shrews: apparently i need to create a zero byte object in swift as a directory marker
17:23:31 Shrews corvus: hrm, that seems like something i had to deal with at some point in the past
17:23:38 corvus aiui, if i were to use the connection object, i would have to create a zero byte file on disk and pass the path to it. if i use the proxy, i can say data=''
17:24:12 Shrews corvus: but now i have context (so thx) :)
17:24:23 corvus np
17:27:37 Shrews actually, i think it was with the shade create_container code i originally wrote. there was weirdness with it
17:28:09 Shrews err, create_object
17:28:09 corvus i'm happy to use the proxy object for now. i'm trying that, but i'm not sure it's sending the headers i'm telling it to.
17:33:21 corvus Shrews: this is what i think i should be doing: http://paste.openstack.org/show/726298/ but the additional headers don't show up in the curl debug output. should they?
17:34:10 Shrews corvus: not sure, tbh
17:36:19 mordred Shrews, corvus: we should add support for specifying the data to upload to create_object
17:39:10 mordred yes - I would expect extra headers to be sent ...
17:40:25 mordred corvus: you could also just do the put directly ... cloud.object_store.put('/{container}/{relative_path}', headers=headers)
17:43:42 Shrews is it too late to switch to using the shade-version of things?
17:43:49 Shrews through Connection
17:44:17 mordred that's what he's mostly using - but the shade vesion doesn't allow zero-byte uploads
17:44:25 mordred corvus: I'm making a quick patch to add a data param
17:44:26 Shrews ah, i see
17:44:51 corvus also, would be nice to avoid the checksums
17:45:06 mordred yup. I made that patch lat night - this one is on top of it
17:45:11 corvus ack
17:45:26 mordred corvus: also, I would like to cut an sdk release today - so it shouldn't be super terrible to get both in
17:45:28 corvus i will switch to put for now to keep moving
17:45:35 mordred corvus: ++
17:47:14 Shrews i could have swore the shade create_container handled the "directory" nature he's looking to achieve, but i'm probably misremembering
17:50:12 Shrews hrm, my old test code doesn't do that, so i guess my brain is telling lies
17:55:16 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add flag for disabling object checksum generation https://review.openstack.org/584044
17:55:17 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add ability to pass data to create_object https://review.openstack.org/584045
17:55:19 mordred Shrews, corvus: ^^
17:56:05 mordred corvus, Shrews: should we add a method for "create_object_directory" or something? that will do the right thing to create the zero-byte file as a pseudo-directory?
17:56:36 Shrews mordred: i think that was what i was remembering... i *wanted* to make an api for doing that, but then just forgot to do it
17:57:11 Shrews seems like a common enough thing we could help users out with
17:57:15 mordred yah
18:03:07 corvus just make sure you can still pass in the other parameters (like expire-at, etc)
18:03:17 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add create_object_pseudo_directory method https://review.openstack.org/584050
18:03:38 mordred corvus: yup! ^^ how's that?
18:04:05 Shrews mordred: https://docs.openstack.org/newton/user-guide/cli-swift-pseudo-hierarchical-folders-directories.html
18:04:44 Shrews i think if you just create the top-level container, then put the heirarchy in the object name, it works as expected
18:04:49 Shrews i think i remember trying this now
18:04:58 Shrews corvus: ^^
18:05:36 mordred corvus: your code was passing text/html - but the docs from mnaser said that the directory objects wanted to be application/directory so I put that in the helper method - but obviously I havent' tested that in the wild
18:05:56 corvus mordred: yeah my snippet was an example of an actual file
18:06:06 mordred Shrews: hrm that makes me want to make a list_object_directories method or something similar that does delimiter='/' and whatnot from that doc
18:06:13 mordred corvus: ah - gotcha
18:06:32 mordred I should probably add some unittests of those
18:06:38 mordred or else Shrews might be unhappy
18:17:57 Shrews mordred: corvus: so i confirmed that naming the object with heirarchy works as expected
18:18:23 Shrews corvus: so you can avoid the whole empty file workaround if you do that
18:19:31 mordred Shrews: I'm not sure what you mean?
18:20:01 corvus Shrews: apparently that's true with actual swift. but perhaps not with ceph/rados. at least, that's what i heard. so i wanted to do a quick 5 minute experiment to verify. :)
18:20:02 mordred Shrews: oh - the empty file thing is to make the directory listing work on some installations (it's not always required, but it never hurts)
18:20:03 Shrews mordred: create_container("foo") ; create_object("foo", "path/to/object.txt")
18:20:28 corvus i've got access to one of each right now, and am testing in parallel
18:20:28 mordred Shrews: yes. that will work - but on radosgw path/to won't render as a directory without the 0byte marker object
18:20:52 Shrews mordred: are you saying that we HAVE AN INCONSISTENCY WITH AN OPENSTACK API????????
18:21:11 mordred Shrews: shocking right?
18:21:22 Shrews speechless
18:21:23 corvus i'm not weighing in on what anyone should do in the long run. i'm only just trying to write the code to perform the experiment to formulate whatever may or may not become a bug report. :)
18:21:32 mordred Shrews: although to be fair to swift in this case, it's an inconsistency wtih swift and something else that is not swift that iplements the same api
18:22:07 Shrews mordred: well, i know nothing of the other swift-like-yet-not-swift things
18:22:47 Shrews this just bugged me b/c i know i went through this at some point
18:23:41 mordred yah
18:33:55 mnaser mordred: did that end up resolving it?
18:36:20 mordred mnaser: we are still validating
18:40:42 openstackgerrit Merged openstack-infra/shade master: Fix doc mistake https://review.openstack.org/584010
19:03:00 corvus mnaser, mordred: yeah, this works as expected with the directory marker: https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/corvustest-1532020307/test/
19:15:42 mnaser sweet
19:29:14 mordred corvus: woot!
19:30:17 openstackgerrit Hongbin Lu proposed openstack-infra/shade master: Use valid filters to list floating IPs in neutron https://review.openstack.org/584076
19:35:25 mordred corvus, mnaser: ok - so at this point I think I should send a note to sage about the behavior difference, yeah?
19:35:57 openstackgerrit Hongbin Lu proposed openstack/openstacksdk master: Use valid filters to list floating IPs in neutron https://review.openstack.org/584029
19:41:03 mnaser mordred: I think so. It looks like rados is much more explicit
19:41:27 mnaser Where as swift is okay with the directory not actually existing
19:46:55 mordred https://etherpad.openstack.org/p/EAYOUb8toz <-- mnaser, corvus this look ok and/or correct?
19:50:25 mnaser mordred: makes sense. I have a long 6 hour flight today. Maybe I can try hacking on this but I haven’t touched C in a while :)
19:52:56 mordred mnaser: :)
19:53:07 mordred mnaser: 6 hours is plenty of time to get your compiler working
19:53:33 mnaser Compiling ceph, hah
19:57:52 openstackgerrit Toure Dunnon proposed openstack-infra/shade master: python-shade expose MTU setting. https://review.openstack.org/578861
20:03:37 corvus mordred: oh i'd like to suggest some changes to the etherpad
20:06:08 corvus mordred: 2 things. 1) this is specifically the behavior where swift serves existing foo/index.html when you request foo/. that's like "DirectoryIndex" in apache, which is distinct from mod_autoindex, which is auto-generate a directory listing if there is no index.html. swift *also* has that capability, but i haven't done anything with it yet, and i worry that the message appears a little muddied.
20:06:39 corvus mordred: 2) i believe i can point at some swift docs that describe the swift behavior (so the ceph behavior doesn't match the docs)
20:06:45 corvus gimme a sec to dig those up
20:07:19 corvus okay, well, the docs are ambiguous
20:07:26 corvus to me at any rate
20:07:29 corvus https://docs.openstack.org/mitaka/user-guide/cli_swift_static_website.html#static-website
20:07:51 corvus If you create sub-directories for your site by creating pseudo-directories in your container, the index page for each sub-directory is displayed by default. If your pseudo-directory does not have a file with the same name as your index file, visits to the sub-directory return a 404 error.
20:08:55 corvus so, i guess it hinges on the question "what is a pseudo-directory?". I had assumed it was merely including '/' in the paths of uploaded files. if, instead, it means "actually create an application/directory object" then i agree ceph behavior matches docs.
20:12:47 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add flag for disabling object checksum generation https://review.openstack.org/584044
20:12:48 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add ability to pass data to create_object https://review.openstack.org/584045
20:12:48 openstackgerrit Monty Taylor proposed openstack/openstacksdk master: Add create_directory_marker_object method https://review.openstack.org/584050
20:13:28 mordred corvus: yeah - I agree, those are a bit ambiguous
20:14:01 mordred Shrews, corvus: ^^ same patches, but this time with tests
20:16:23 mordred corvus: it might be worth pinging notmyname about the doc ambiguity
20:16:49 corvus mordred: i just started doing that actually in -swift
20:16:53 mordred neat
20:18:06 mordred corvus: should I wait for his response before sending email?
20:21:06 corvus mordred: yeah, maybe a few mins

Earlier   Later