| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-07 | |||
| 18:27:33 | melwitt | yeah, gotcha | |
| 18:29:33 | melwitt | yeah, I dunno. that's a tough one, bad timing. mdbooth might be able to look at it tomorrow | |
| 18:29:51 | dansmith | the spec also kinda describes this as just bringing his extension into tree, which will still use the extension point, which is not what I think we should do | |
| 18:30:01 | dansmith | which is also why I think this needs another go-round at least :) | |
| 18:30:03 | dansmith | yeah | |
| 18:37:24 | melwitt | the concept makes a lot of sense, similar to the ceph fast clone stuff we have, I just don't yet understand why the extension point is needed to grab the image if we didn't need it for fast clone | |
| 18:39:49 | melwitt | I think the feature is definitely legit but the implementation details are the potential snag, but I think everyone said that already | |
| 18:44:37 | mriedem | dansmith: melwitt: i'm happy to sit on that until mdbooth can look it over, i'll drop the +2 | |
| 18:46:08 | mriedem | sounds like, if image isn't in cache, and images_type!=rbd but CONF.libvirt.rbd_user is configured, then look for the image in ceph, else download | |
| 18:46:38 | mriedem | which is presumably how this works today with the extension download handler | |
| 18:47:06 | melwitt | yeah, agreed sounds like it would be something like that | |
| 18:48:59 | melwitt | the only catch might be that you have to get it from the glance ceph pool. I don't think that's the case for the fast clone snapshot | |
| 18:49:41 | melwitt | but I guess it must be ... thinking about the glance direct urls | |
| 18:49:43 | dansmith | agreed that it sounds like the right thing to do, I just am unsure on the mechanics | |
| 18:49:50 | dansmith | if we can wait for mdbooth to look, I'd feel a lot better | |
| 18:50:11 | melwitt | yeah, that's fair enough | |
| 18:55:01 | mnaser | melwitt, mriedem: that spec is nice but it doesn't figure out our issue because we have 2 ceph clusters | |
| 18:55:29 | mnaser | for me the value would be if the image cache would use rbd instead of local storage | |
| 18:55:45 | mnaser | so we download the image once and then it remains cached inside the cluster | |
| 18:56:20 | mnaser | which is a problem i'm about to run into next week that i haven't fully thought out what/how i go about this :( | |
| 18:56:22 | mriedem | so maybe that's an alternative for the spec | |
| 18:56:26 | mriedem | make image cache ceph aware | |
| 18:56:53 | mriedem | mnaser: there are nova guidance counselors here to help :) | |
| 18:57:08 | mnaser | image cache and backend are decoupled.. or not? | |
| 18:58:13 | mnaser | mriedem: https://wiki.openstack.org/wiki/Technical_Committee_Tracker#Project_Teams i'm signed up ha | |
| 18:58:30 | mnaser | so if they are decoupled, that would solve that problem for that use case in that spec | |
| 18:58:36 | mriedem | ha | |
| 18:58:39 | mnaser | it would involve an extra import/export | |
| 18:58:58 | mnaser | because you'd be exporting it from glance once slowly and importing it to ceph, storing it potentially twice if its the same cluster | |
| 18:59:33 | mnaser | in my case i'd be storing it twice but in different clusters | |
| 19:01:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: libvirt: Don't report DISK_GB if sharing https://review.openstack.org/560459 | |
| 19:01:59 | melwitt | cloned from it) | |
| 19:01:59 | melwitt | yeah, I dunno if I'm missing something but it sounds like we could do this by adding an extra step in the process, like instead of cow cloning from the glance pool, we take an extra step first to copy it and flatten to the nova pool (if does not exist) and then cow clone off of that instead. it would also decouple instances from the images in glance (problem of being unable to delete a glance image if any instance is in existing cow | |
| 19:02:32 | melwitt | *instance is existing that was cow cloned from it | |
| 19:02:38 | mnaser | melwitt: thats a pretty good way of going about things and it does remove that weird | |
| 19:02:45 | mnaser | "i cant delete my image!!!" complaints with rbd | |
| 19:02:48 | melwitt | yeah | |
| 19:03:22 | mnaser | the only thing is that does potentially slow down first boots | |
| 19:03:37 | mnaser | and i'm not sure how that would be handled if someone uploads an image and boots 10 VMs of it right away | |
| 19:03:38 | melwitt | yeah | |
| 19:04:44 | melwitt | heh, maybe this ties into that spec, rbd export, is that faster than clone + flatten or the same I would think? | |
| 19:05:07 | mnaser | i think flatten is faster because flatten doesnt involve the host that does it | |
| 19:05:42 | mnaser | export means you download as fast as your compute node, and upload as fast as your compute node in import | |
| 19:05:58 | melwitt | oh, okay | |
| 19:06:12 | mnaser | where as flatten is this cluster operation where osds talk to themselves | |
| 19:06:20 | melwitt | got it | |
| 19:06:39 | mnaser | but maybe this is what you're talking about but if we decouple the 'export' and 'import' then both my and this users use case can be solved | |
| 19:07:01 | mnaser | if we can do cow, export is noop and import is create cow volume | |
| 19:07:22 | mnaser | if compute node uses local storage and image is on glance, export using rbd export, import using <write-on-disk> | |
| 19:07:45 | mnaser | if compute node uses ceph, image in ceph, but not directly accessible, export by glance-api download, import to ceph cluster | |
| 19:08:29 | mnaser | nova/nova/virt/libvirt/imagecache.py havent been touched since mid-2016 other than privsep stuff, i imagine its flawless :) | |
| 19:08:47 | melwitt | totally flawless | |
| 19:09:19 | mnaser | "This cleanup code will delete all info files the first time it runs in Ocata, which means we can delete this block entirely in P." | |
| 19:10:04 | melwitt | heh | |
| 19:11:29 | mnaser | this doesn't look too crazy, ImageCacheManager could be changed up to have multiple interfaces without too much crazy work scrolling through it | |
| 19:11:48 | melwitt | wanna do it? :) | |
| 19:12:10 | mnaser | i can give it a shot, the only thing that is a concern is locking | |
| 19:12:16 | mnaser | i think ceph allows you to lock an image | |
| 19:12:31 | mriedem | i know someone in the UK that loves locks and the image cache manager | |
| 19:12:32 | melwitt | I was gonna say, maybe you can write your ideas on the spec review https://review.openstack.org/#/c/572805 if that helps. if these two use cases can be solved with one stone | |
| 19:12:41 | melwitt | hah, good point | |
| 19:13:01 | dansmith | yes, mnaser please comment | |
| 19:13:15 | dansmith | sounds like a good reason to let this spec live for more than 24 hours :) | |
| 19:16:43 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:19:37 | mnaser | dansmith, melwitt, mriedem: i left comments | |
| 19:19:58 | mriedem | thanks | |
| 19:20:01 | mnaser | i wonder how much pain i'm signing myself up for, but this is something we'll eventually need | |
| 19:20:02 | melwitt | ++ | |
| 19:20:20 | dansmith | mnaser: you know you like the pain | |
| 19:20:27 | mnaser | ha | |
| 19:20:32 | dansmith | hard drive until it megahertz | |
| 19:21:16 | mnaser | i'm going to start on some wip giant patch just to see what i'm dealing with | |
| 19:21:42 | jmlowe | Quick question, should this log line indicate that an allocation is being deleted? DEBUG nova.compute.resource_tracker [Instance XXXX has been moved to another host XXX. There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 2048, u'DISK_GB': 8}}. _remove_deleted_instances_allocations /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:12 | |
| 19:21:44 | jmlowe | 86 | |
| 19:23:01 | openstackgerrit | Simon Dodsley proposed openstack/nova master: Add enhanced KVM storage QoS quotas https://review.openstack.org/558530 | |
| 19:23:33 | melwitt | mnaser: I think lots of people need it and I think I could help with some of it, our customers run into things with the glance clone to nova part like this https://review.openstack.org/457886 | |
| 19:23:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment https://review.openstack.org/572790 | |
| 19:24:27 | melwitt | mnaser: and like you've said before, we could see people wanting to use separate ceph clusters per cell and it would help that too. and then the issue this other person has in the spec. etc | |
| 19:24:55 | mnaser | melwitt: cool, i'll start reading through code and pushing something preliminary, but yeah this is something that has to happen at some point | |
| 19:26:04 | melwitt | jmlowe: maybe, probably. dansmith did we have a bug or doc that explains/helps with knowing whether allocations need to be cleaned up via the osc-placement cli? | |
| 19:26:21 | mriedem | jmlowe: the code has a big comment with more context on that message http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/resource_tracker.py#n1329 | |
| 19:26:35 | dansmith | it means it would have been cleaned up in ocata, but not later I think | |
| 19:26:44 | mriedem | this is a mixed ocata compute / auto delete allocation thing from pike | |
| 19:27:07 | jmlowe | looking at the placement logs I don't see a delete, it's allocated on two hosts now following a host-evacuate-live | |
| 19:27:09 | dansmith | melwitt: no doc that I know of | |
| 19:27:39 | melwitt | jmlowe: yeah, sorry, misread your question. it doesn't mean it deleted them but is saying it found allocations for an instance that is no longer on the host | |
| 19:27:41 | mriedem | if the source host is dead (evacuated from) then those allocations aren't causing any problems | |
| 19:27:54 | dansmith | host-evacuate-live is live migration | |
| 19:27:56 | dansmith | not evacuation | |
| 19:28:00 | mriedem | oh wait host-evacuate-live is not evac | |
| 19:28:02 | mriedem | yeah | |
| 19:28:06 | mriedem | link me! | |
| 19:28:30 | mriedem | jmlowe: is the instance running properly on the dest host? | |
| 19:28:39 | mriedem | and is the source host pike or later? | |
| 19:28:47 | jmlowe | all pike | |
| 19:29:04 | mriedem | ok if the instance is happy on the dest host then you can manually remove the allocations for that instance on the source host | |
| 19:29:29 | mriedem | mayhap we need a placement FAQ like our cells v2 FAQ | |
| 19:29:59 | melwitt | yeah. we had someone run into this same problem internally but I don't have the bz handy | |
| 19:30:24 | mriedem | i bet i do | |
| 19:30:26 | melwitt | or if there was one or if this happened during a testing situation | |