Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-12
21:02:43 melwitt mnaser: they are, it's just the ceph docs say they don't support qcow2 format
21:02:58 mnaser melwitt: it will create a centralized cache so to speak, so an image is downloaded from cell A to cell B once only
21:03:12 mnaser rather than being downloaded potenitally N times where N is the # of hypervisors
21:03:18 mnaser potentially*
21:03:24 melwitt okay, I'm just missing how it's centralized if each ceph is separate and can't find each other like you said
21:04:39 melwitt if you configure things unlike the ceph docs instruct, then yes it will download and convert. that's probably why it says don't do that, use raw
21:05:06 mnaser melwitt: okay, but if i use raw, it'll still download it across the network and cache it inside every hypervisor
21:05:27 mnaser ideally, i'd imagine it caching it inside the ceph cluster in cell B, so that it is downloaded once only, and nova can do fast clone
21:05:49 melwitt so cache one copy per cell is what you're saying
21:05:56 mnaser correct
21:06:12 mnaser so currently: cell B HVs downloads raw image from glance (which is in another location), caches it locally, imports to create disk
21:06:34 mnaser idea: cell B HV downloads raw image from glance, stores it in ceph, fast-clone to create disk. further hypervisors would fast-clone that same image
21:06:35 melwitt okay. yeah, I guess I didn't know ceph images are cached on disk on hypervisors. I thought they just accessed the ceph pool each time
21:07:05 mnaser melwitt: generally they are not, but because the cell wouldnt be able to reach that cluster, it will fall back to plain o'l downloading image via http and importing it
21:07:42 mnaser and that can be a pretty big bottleneck as things scale
21:08:31 melwitt that seems reasonable to me, mdbooth is the imagebackend guru so he'd be the best person to run the idea by
21:09:40 mnaser melwitt: just drumming up things to hear what it looks like. this whole idea can be worked around by storing things twice in glance (it supports multiple image locations)
21:10:08 mnaser but then it would be a glance thing rather than nova (nova would just try all locations and find that one of them is in a ceph cluster it can access)
21:10:22 mnaser it's been a fun little problem to think about
21:13:14 melwitt yeah, good points. let me dig up our etherpad for the to-resolve things we've been keeping track of
21:14:49 mnaser im sure ill run into plenty of other fun stuff as we'll use segments and routed networks too
21:16:43 melwitt I expect so too
21:17:31 melwitt mriedem: what's the current multi-cells issues/todos/considerations etherpad we have?
21:17:51 melwitt I found https://etherpad.openstack.org/p/nova-pike-cells-v2-todos but wasn't sure if it's the latest thing
21:18:17 mriedem this? https://etherpad.openstack.org/p/cellsv1-to-v2-migration
21:18:48 mriedem i'm fairly sure https://etherpad.openstack.org/p/nova-pike-cells-v2-todos hasn't been touched in a long time
21:19:20 mriedem so yeah https://etherpad.openstack.org/p/nova-pike-cells-v2-todos is probably the most recent since it was the running list of stuff for pike
21:19:45 melwitt okay, maybe I just assumed we had one. a place where we can track concerns from operators as we gather more problems to solve
21:20:14 mriedem there is also the old wiki page https://wiki.openstack.org/wiki/Nova-Cells-v2
21:23:51 melwitt okay, maybe I'll add it to Open Questions on the wiki so it doesn't get lost
21:39:38 openstackgerrit Merged openstack/nova stable/pike: [placement] Fix foreign key constraint error https://review.openstack.org/525620
21:41:56 penick We're doing the first Juno->Ocata upgrade right now.. :|
21:42:00 melwitt mnaser: wrote down the ideas here, feel free to edit/correct https://wiki.openstack.org/wiki/Nova-Cells-v2#Open_Questions
21:46:26 penick I'm sure it'll be fine. Heck, it worked in devstack.
21:46:55 melwitt penick: yall just now started?
21:46:57 mriedem penick: are you going to live stream it on facebook for your aunt?
21:47:33 penick melwitt: well we started a bit earlier. We were scheduled to start at 10. So was a fire drill, apparently..
21:47:47 penick mriedem: My aunt? Which one? And why?
21:47:59 mriedem because i thought live stream,
21:48:04 mriedem and then i thought facebook,
21:48:13 mriedem and then i thought, who would like to watch that on facebook - someone's aunt
21:48:19 mriedem because that's what i associate with facebook
21:48:26 penick that actually parses
21:48:40 melwitt you can live-tweet it
21:48:53 penick tweet 1: We've started!
21:48:57 mriedem aunt sheryl isn't on the twitters yet
21:48:59 penick Tweet 37: everything's fucked
21:49:42 penick I now have a new interview question.. "Describe a data structure most appropriate for simulating mriedem's brain. Here's a pen. Godspeed."
21:50:34 penick My aunt Lisa is on twitter, she mostly uses it to insult political figures. I respect that.
21:51:15 penick if we live tweeted the process we could mine it later to make a how-to guide
21:52:25 melwitt you should definitely record what you've done if you want to do that first draft of the ops guide for fast-forward-upgrade
21:53:12 openstackgerrit melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614
21:53:13 openstackgerrit melanie witt proposed openstack/nova master: Add console_auth_token_get() method to DB API https://review.openstack.org/481700
21:53:13 openstackgerrit melanie witt proposed openstack/nova master: Add console connection object https://review.openstack.org/320063
21:53:14 openstackgerrit melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381
21:53:14 openstackgerrit melanie witt proposed openstack/nova master: Use ConsoleConnection object to generate authorizations https://review.openstack.org/325414
21:53:15 openstackgerrit melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990
21:53:16 dims godspeed penick !
21:53:32 penick dims thanks! :)
21:53:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Add log in conductor to mark host for instance creation https://review.openstack.org/520369
21:55:20 penick melwitt: Yep
22:00:17 jroll I like this new interview question
22:00:57 melwitt it's jroll! o/
22:01:06 jroll \o :)
22:02:49 dims hey jroll !
22:06:11 openstackgerrit Merged openstack/nova master: remove reserve_quota_delta https://review.openstack.org/521469
22:08:08 openstackgerrit Merged openstack/nova stable/ocata: Skip test_rebuild_server_in_error_state for cells v1 https://review.openstack.org/496359
22:08:17 jroll hiya dims
22:20:43 mriedem jaypipes: so with the in_tree filter,
22:20:51 mriedem is the value for that a parent provider?
22:20:59 mriedem like, leaf nodes would always return an empty list?
22:22:08 mriedem if if i have parent A and children B and C, if i query for in_tree=C, do i get B because it's in the "same tree"?
22:22:56 jaypipes mriedem: the value is the UUID of any resource provider. the returned results are all providers in that provider's tree. you can provider a child, a root, a grandchild, it doesn't matter.
22:23:32 mriedem ok that wasn't clear to me from the api ref description, "A UUID of a resource provider. The returned resource providers will be in the same “provider tree” as the specified provider." and looking at the tests used in https://review.openstack.org/#/c/521097/
22:23:56 jaypipes mriedem: the tests specifically show that behaviour.
22:24:31 jaypipes mriedem: you can specify the child, the root or the grandchild as the in_tree parameter value and all the providers in the tree will be returned.
22:24:34 mriedem sorry i mean https://review.openstack.org/#/c/520663/9
22:24:40 mriedem https://review.openstack.org/#/c/520663/9/nova/tests/functional/api/openstack/placement/test_report_client.py
22:24:49 mriedem that's not a parent/child relationship so it wasn't clear to me
22:26:29 jaypipes mriedem: oh, yeah, sorry, I didn't write that particular patch
22:26:58 jaypipes mriedem: eric is always passing the root (which happens to be the compute node UUID)
22:27:50 mriedem yeah see my questoin at https://review.openstack.org/#/c/520663/9/nova/tests/unit/scheduler/client/test_report.py
22:27:57 mriedem L1400
22:28:59 mriedem would probably be good to have a tree structure sample like that in the api ref description to explain the scenarios
22:42:00 mnaser melwitt: https://wiki.openstack.org/wiki/Nova-Cells-v2#Open_Questions appended the discussion re cellsv2+ceph+images
22:42:04 mnaser thanks for putting it up there
22:43:36 melwitt mnaser: cool, thanks for adding to it. so, with my question I was wondering how does glance manage multiple locations then. AFAIK it can only connect to one ceph cluster at a time itself, but I'm pretty ignorant about glance
22:44:26 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
22:44:45 mnaser melwitt: thats a good question, i think in this context, the nova <=> glance interactions, glance wont really ever talk to ceph because its just supplying locations
22:45:07 mnaser melwitt: now when you delete an image with multiple locations, some which are not accessible, what's going to happen...
22:45:11 mnaser another fun item to the list lol
22:45:33 melwitt mnaser: I see. so the workflow for the admin would be, store image in ceph cluster A, call glance api to add it to the DB and point at location A, and repeat
22:46:17 melwitt I saw that glance stores ceph credentials (one set) so that's what I was thinking of with the question
22:46:29 melwitt in its conf file I mean
22:46:41 mnaser yeah glance currently works with a single set of credentials, so the second location would manually be set (the API allows you up add/remove a location)
22:46:49 melwitt got it
22:47:16 mnaser so you can have some sort of cronjob that "replicates" stuff over across cluster but thats hacky and becomes confusing to the end user
22:47:41 mnaser and as you brought up, operations in glance might fail like deletes (unless you remove the location manually then delete but thats no fun for useres)
22:47:56 melwitt right
22:48:22 mnaser (this macbook keyboard is driving me insane with this weird things where it types 2 characters of the same letter when clicking it :()

Earlier   Later