Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-12
19:07:31 mriedem yeah i'm assuming he can host it himself
19:08:51 mriedem er yeah http://openlabtesting.org/
19:09:22 cdent even comes with uml
19:30:27 mriedem dansmith: want to check out the first 2 changes in the alternate hosts series here? https://review.openstack.org/#/c/516707/ - just needs final +2
19:30:48 dansmith okay in a bit
19:39:22 mriedem for anyone that cares, the instance action paging series is also ready to go i think https://review.openstack.org/#/q/topic:bp/pagination-add-changes-since-for-instance-action-list+status:open
19:39:50 mriedem i've started moving fully complete series up to the top of https://etherpad.openstack.org/p/nova-queens-blueprint-status
20:15:49 openstackgerrit Matt Riedemann proposed openstack/nova master: [WIP] POC to use neutron port_list when filtering instance by ip https://review.openstack.org/525505
20:18:43 openstackgerrit Michael Still proposed openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516
20:18:44 openstackgerrit Michael Still proposed openstack/nova master: Start moving users of parted to privsep. https://review.openstack.org/519011
20:18:44 openstackgerrit Michael Still proposed openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010
20:18:45 openstackgerrit Michael Still proposed openstack/nova master: Convert users of tune2fs to privsep. https://review.openstack.org/519484
20:18:45 openstackgerrit Michael Still proposed openstack/nova master: Move remaining uses of parted to privsep. https://review.openstack.org/519483
20:18:46 openstackgerrit Michael Still proposed openstack/nova master: Move makefs to privsep https://review.openstack.org/527510
20:51:45 mnaser throwing this up for discussion - how would folks feel if image caching used the imagebackend instead of just it's own cache
20:52:15 mnaser it would be beneficial for scenarios when you have a qcow2 image stored, with ceph backend for example, avoiding the import and convert on every iteration
20:53:01 mnaser another interesting scenario is multiple cells with independent ceph clusters (thereby, image is not directly accessible to do a cow new image)
20:53:55 mnaser (we're going to use cells v2 to add another az in an entirely different physical facility with a 10g link between them, fun times ahead :>)
20:56:26 melwitt mnaser: if you're using ceph and its native qcow2 format, you will be using glance images of type RAW and will get fast-clone and no image conversions, right?
20:56:48 melwitt that is, what do you mean by 'import and convert'?
20:56:55 mnaser melwitt: ceph with images stored in raw will get fast clone
20:57:33 mnaser melwitt: so - current scenario - images stored in raw in `images` pool in ceph, vm disks stored in `vms` pool in ceph. when nova boots a new vm, it can access `images` directly so it does a fast-clone and you get a disk very quickly
20:58:12 mnaser melwitt: if the image format is qcow2, ceph wont be able to do fast clone. nova will pull the image down locally, convert it to raw, and upload/import it back to ceph, then boot the machine
20:58:21 melwitt right
20:59:21 mnaser melwitt: now in an env where you have 2 cells, each one has a ceph cluster of its own, cell A which uses the same ceph cluster as glance can do fast clone. cell B will check the location and realize it has no way to reach that ceph cluster, so it will download the raw image locally and import/upload it to ceph
20:59:56 mnaser and for every new VM, it will do this (obviously leveraging local image cache), but that can be a lot of traffic before an image gets cached
21:00:39 mnaser my two ideas to work this around is... a) use the multiple locations feature of glance to mirror the images and add an rbd location in both clusters (nova loops over all locations till it finds one, then falls back to download)
21:00:46 melwitt from the ceph docs "Important Ceph doesn’t support QCOW2 for hosting a virtual machine disk. Thus if you want to boot virtual machines in Ceph (ephemeral backend or boot from volume), the Glance image format must be RAW." http://docs.ceph.com/docs/master/rbd/rbd-openstack/
21:01:04 melwitt the cells thing is different though, and something to think about
21:01:23 mnaser or b) refactor image cache to use the image backend so that images can be cached in the ceph cluster, avoiding multiple downloads
21:01:49 mnaser melwitt: let me check the code, i am fairly confident qcow2 images will be downloaded by nova and converted before being imported. *me checks*
21:02:02 melwitt how does b) help if each cell is using an independent ceph?
21:02:43 melwitt mnaser: they are, it's just the ceph docs say they don't support qcow2 format
21:02:58 mnaser melwitt: it will create a centralized cache so to speak, so an image is downloaded from cell A to cell B once only
21:03:12 mnaser rather than being downloaded potenitally N times where N is the # of hypervisors
21:03:18 mnaser potentially*
21:03:24 melwitt okay, I'm just missing how it's centralized if each ceph is separate and can't find each other like you said
21:04:39 melwitt if you configure things unlike the ceph docs instruct, then yes it will download and convert. that's probably why it says don't do that, use raw
21:05:06 mnaser melwitt: okay, but if i use raw, it'll still download it across the network and cache it inside every hypervisor
21:05:27 mnaser ideally, i'd imagine it caching it inside the ceph cluster in cell B, so that it is downloaded once only, and nova can do fast clone
21:05:49 melwitt so cache one copy per cell is what you're saying
21:05:56 mnaser correct
21:06:12 mnaser so currently: cell B HVs downloads raw image from glance (which is in another location), caches it locally, imports to create disk
21:06:34 mnaser idea: cell B HV downloads raw image from glance, stores it in ceph, fast-clone to create disk. further hypervisors would fast-clone that same image
21:06:35 melwitt okay. yeah, I guess I didn't know ceph images are cached on disk on hypervisors. I thought they just accessed the ceph pool each time
21:07:05 mnaser melwitt: generally they are not, but because the cell wouldnt be able to reach that cluster, it will fall back to plain o'l downloading image via http and importing it
21:07:42 mnaser and that can be a pretty big bottleneck as things scale
21:08:31 melwitt that seems reasonable to me, mdbooth is the imagebackend guru so he'd be the best person to run the idea by
21:09:40 mnaser melwitt: just drumming up things to hear what it looks like. this whole idea can be worked around by storing things twice in glance (it supports multiple image locations)
21:10:08 mnaser but then it would be a glance thing rather than nova (nova would just try all locations and find that one of them is in a ceph cluster it can access)
21:10:22 mnaser it's been a fun little problem to think about
21:13:14 melwitt yeah, good points. let me dig up our etherpad for the to-resolve things we've been keeping track of
21:14:49 mnaser im sure ill run into plenty of other fun stuff as we'll use segments and routed networks too
21:16:43 melwitt I expect so too
21:17:31 melwitt mriedem: what's the current multi-cells issues/todos/considerations etherpad we have?
21:17:51 melwitt I found https://etherpad.openstack.org/p/nova-pike-cells-v2-todos but wasn't sure if it's the latest thing
21:18:17 mriedem this? https://etherpad.openstack.org/p/cellsv1-to-v2-migration
21:18:48 mriedem i'm fairly sure https://etherpad.openstack.org/p/nova-pike-cells-v2-todos hasn't been touched in a long time
21:19:20 mriedem so yeah https://etherpad.openstack.org/p/nova-pike-cells-v2-todos is probably the most recent since it was the running list of stuff for pike
21:19:45 melwitt okay, maybe I just assumed we had one. a place where we can track concerns from operators as we gather more problems to solve
21:20:14 mriedem there is also the old wiki page https://wiki.openstack.org/wiki/Nova-Cells-v2
21:23:51 melwitt okay, maybe I'll add it to Open Questions on the wiki so it doesn't get lost
21:39:38 openstackgerrit Merged openstack/nova stable/pike: [placement] Fix foreign key constraint error https://review.openstack.org/525620
21:41:56 penick We're doing the first Juno->Ocata upgrade right now.. :|
21:42:00 melwitt mnaser: wrote down the ideas here, feel free to edit/correct https://wiki.openstack.org/wiki/Nova-Cells-v2#Open_Questions
21:46:26 penick I'm sure it'll be fine. Heck, it worked in devstack.
21:46:55 melwitt penick: yall just now started?
21:46:57 mriedem penick: are you going to live stream it on facebook for your aunt?
21:47:33 penick melwitt: well we started a bit earlier. We were scheduled to start at 10. So was a fire drill, apparently..
21:47:47 penick mriedem: My aunt? Which one? And why?
21:47:59 mriedem because i thought live stream,
21:48:04 mriedem and then i thought facebook,
21:48:13 mriedem and then i thought, who would like to watch that on facebook - someone's aunt
21:48:19 mriedem because that's what i associate with facebook
21:48:26 penick that actually parses
21:48:40 melwitt you can live-tweet it
21:48:53 penick tweet 1: We've started!
21:48:57 mriedem aunt sheryl isn't on the twitters yet
21:48:59 penick Tweet 37: everything's fucked
21:49:42 penick I now have a new interview question.. "Describe a data structure most appropriate for simulating mriedem's brain. Here's a pen. Godspeed."
21:50:34 penick My aunt Lisa is on twitter, she mostly uses it to insult political figures. I respect that.
21:51:15 penick if we live tweeted the process we could mine it later to make a how-to guide
21:52:25 melwitt you should definitely record what you've done if you want to do that first draft of the ops guide for fast-forward-upgrade
21:53:12 openstackgerrit melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614
21:53:13 openstackgerrit melanie witt proposed openstack/nova master: Add console connection object https://review.openstack.org/320063
21:53:13 openstackgerrit melanie witt proposed openstack/nova master: Add console_auth_token_get() method to DB API https://review.openstack.org/481700
21:53:14 openstackgerrit melanie witt proposed openstack/nova master: Use ConsoleConnection object to generate authorizations https://review.openstack.org/325414
21:53:14 openstackgerrit melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381
21:53:15 openstackgerrit melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990
21:53:16 dims godspeed penick !
21:53:32 penick dims thanks! :)
21:53:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Add log in conductor to mark host for instance creation https://review.openstack.org/520369
21:55:20 penick melwitt: Yep
22:00:17 jroll I like this new interview question
22:00:57 melwitt it's jroll! o/
22:01:06 jroll \o :)
22:02:49 dims hey jroll !
22:06:11 openstackgerrit Merged openstack/nova master: remove reserve_quota_delta https://review.openstack.org/521469
22:08:08 openstackgerrit Merged openstack/nova stable/ocata: Skip test_rebuild_server_in_error_state for cells v1 https://review.openstack.org/496359
22:08:17 jroll hiya dims

Earlier   Later