| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-07 | |||
| 18:22:03 | melwitt | that is, I don't want us to lose the functionality but I don't know what the right way to provide it is | |
| 18:22:04 | dansmith | and/or we could just give it a spec exception until next week in hopes that we could get a quick read from anyone who knows anything about this to trivially thumb it up or something | |
| 18:22:19 | dansmith | the spec is about adding a feature, | |
| 18:22:24 | dansmith | it's not about the extension point itself | |
| 18:22:40 | dansmith | the guy is offering to upstream his thing so that we could remove the extension point basically | |
| 18:22:48 | dansmith | but I dunno if his thing is a total hack or not | |
| 18:22:53 | melwitt | but it's about adding a feature that provides the same functionality that the deprecated extension point is currently taking care of? | |
| 18:22:57 | dansmith | no | |
| 18:23:06 | dansmith | it uses the extension point to add the feature out of tree currently | |
| 18:23:27 | dansmith | it's coming up because if the extension point goes away, he can't sideload his extra thing | |
| 18:23:48 | dansmith | the confusion may be that he asserted that bringing it into tree means we keep the extension point, but that's not really a thing | |
| 18:25:27 | melwitt | hm, okay. yeah, I would be okay with a freeze exception to review this since it showed up only yesterday and if it's an important operator issue | |
| 18:26:06 | dansmith | I'm not sure it is an important operator issue, because the thread on the ML was a week ago and nobody has really jumped on it | |
| 18:26:12 | melwitt | I just haven't connected the dots between it and the ML post, I had assumed it was proposed in reaction to the deprecation of the extension point and as a way to keep the fast download behavior | |
| 18:26:24 | dansmith | we don't have the fast download behavior now | |
| 18:26:30 | dansmith | this guy hacks it in with an out of tree extension, | |
| 18:26:34 | dansmith | which uses this extension point | |
| 18:26:43 | dansmith | he wants to upstream it so he can keep the functionality after we remove the extension point | |
| 18:26:47 | melwitt | oh, I see | |
| 18:26:53 | dansmith | which is awesome and stuff, | |
| 18:27:22 | dansmith | I just don't know if what he's describing that he's doing is a gross hack or legit, because Idon't know the image cache maze of caves very wekk | |
| 18:27:23 | dansmith | *well | |
| 18:27:33 | melwitt | yeah, gotcha | |
| 18:29:33 | melwitt | yeah, I dunno. that's a tough one, bad timing. mdbooth might be able to look at it tomorrow | |
| 18:29:51 | dansmith | the spec also kinda describes this as just bringing his extension into tree, which will still use the extension point, which is not what I think we should do | |
| 18:30:01 | dansmith | which is also why I think this needs another go-round at least :) | |
| 18:30:03 | dansmith | yeah | |
| 18:37:24 | melwitt | the concept makes a lot of sense, similar to the ceph fast clone stuff we have, I just don't yet understand why the extension point is needed to grab the image if we didn't need it for fast clone | |
| 18:39:49 | melwitt | I think the feature is definitely legit but the implementation details are the potential snag, but I think everyone said that already | |
| 18:44:37 | mriedem | dansmith: melwitt: i'm happy to sit on that until mdbooth can look it over, i'll drop the +2 | |
| 18:46:08 | mriedem | sounds like, if image isn't in cache, and images_type!=rbd but CONF.libvirt.rbd_user is configured, then look for the image in ceph, else download | |
| 18:46:38 | mriedem | which is presumably how this works today with the extension download handler | |
| 18:47:06 | melwitt | yeah, agreed sounds like it would be something like that | |
| 18:48:59 | melwitt | the only catch might be that you have to get it from the glance ceph pool. I don't think that's the case for the fast clone snapshot | |
| 18:49:41 | melwitt | but I guess it must be ... thinking about the glance direct urls | |
| 18:49:43 | dansmith | agreed that it sounds like the right thing to do, I just am unsure on the mechanics | |
| 18:49:50 | dansmith | if we can wait for mdbooth to look, I'd feel a lot better | |
| 18:50:11 | melwitt | yeah, that's fair enough | |
| 18:55:01 | mnaser | melwitt, mriedem: that spec is nice but it doesn't figure out our issue because we have 2 ceph clusters | |
| 18:55:29 | mnaser | for me the value would be if the image cache would use rbd instead of local storage | |
| 18:55:45 | mnaser | so we download the image once and then it remains cached inside the cluster | |
| 18:56:20 | mnaser | which is a problem i'm about to run into next week that i haven't fully thought out what/how i go about this :( | |
| 18:56:22 | mriedem | so maybe that's an alternative for the spec | |
| 18:56:26 | mriedem | make image cache ceph aware | |
| 18:56:53 | mriedem | mnaser: there are nova guidance counselors here to help :) | |
| 18:57:08 | mnaser | image cache and backend are decoupled.. or not? | |
| 18:58:13 | mnaser | mriedem: https://wiki.openstack.org/wiki/Technical_Committee_Tracker#Project_Teams i'm signed up ha | |
| 18:58:30 | mnaser | so if they are decoupled, that would solve that problem for that use case in that spec | |
| 18:58:36 | mriedem | ha | |
| 18:58:39 | mnaser | it would involve an extra import/export | |
| 18:58:58 | mnaser | because you'd be exporting it from glance once slowly and importing it to ceph, storing it potentially twice if its the same cluster | |
| 18:59:33 | mnaser | in my case i'd be storing it twice but in different clusters | |
| 19:01:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: libvirt: Don't report DISK_GB if sharing https://review.openstack.org/560459 | |
| 19:01:59 | melwitt | cloned from it) | |
| 19:01:59 | melwitt | yeah, I dunno if I'm missing something but it sounds like we could do this by adding an extra step in the process, like instead of cow cloning from the glance pool, we take an extra step first to copy it and flatten to the nova pool (if does not exist) and then cow clone off of that instead. it would also decouple instances from the images in glance (problem of being unable to delete a glance image if any instance is in existing cow | |
| 19:02:32 | melwitt | *instance is existing that was cow cloned from it | |
| 19:02:38 | mnaser | melwitt: thats a pretty good way of going about things and it does remove that weird | |
| 19:02:45 | mnaser | "i cant delete my image!!!" complaints with rbd | |
| 19:02:48 | melwitt | yeah | |
| 19:03:22 | mnaser | the only thing is that does potentially slow down first boots | |
| 19:03:37 | mnaser | and i'm not sure how that would be handled if someone uploads an image and boots 10 VMs of it right away | |
| 19:03:38 | melwitt | yeah | |
| 19:04:44 | melwitt | heh, maybe this ties into that spec, rbd export, is that faster than clone + flatten or the same I would think? | |
| 19:05:07 | mnaser | i think flatten is faster because flatten doesnt involve the host that does it | |
| 19:05:42 | mnaser | export means you download as fast as your compute node, and upload as fast as your compute node in import | |
| 19:05:58 | melwitt | oh, okay | |
| 19:06:12 | mnaser | where as flatten is this cluster operation where osds talk to themselves | |
| 19:06:20 | melwitt | got it | |
| 19:06:39 | mnaser | but maybe this is what you're talking about but if we decouple the 'export' and 'import' then both my and this users use case can be solved | |
| 19:07:01 | mnaser | if we can do cow, export is noop and import is create cow volume | |
| 19:07:22 | mnaser | if compute node uses local storage and image is on glance, export using rbd export, import using <write-on-disk> | |
| 19:07:45 | mnaser | if compute node uses ceph, image in ceph, but not directly accessible, export by glance-api download, import to ceph cluster | |
| 19:08:29 | mnaser | nova/nova/virt/libvirt/imagecache.py havent been touched since mid-2016 other than privsep stuff, i imagine its flawless :) | |
| 19:08:47 | melwitt | totally flawless | |
| 19:09:19 | mnaser | "This cleanup code will delete all info files the first time it runs in Ocata, which means we can delete this block entirely in P." | |
| 19:10:04 | melwitt | heh | |
| 19:11:29 | mnaser | this doesn't look too crazy, ImageCacheManager could be changed up to have multiple interfaces without too much crazy work scrolling through it | |
| 19:11:48 | melwitt | wanna do it? :) | |
| 19:12:10 | mnaser | i can give it a shot, the only thing that is a concern is locking | |
| 19:12:16 | mnaser | i think ceph allows you to lock an image | |
| 19:12:31 | mriedem | i know someone in the UK that loves locks and the image cache manager | |
| 19:12:32 | melwitt | I was gonna say, maybe you can write your ideas on the spec review https://review.openstack.org/#/c/572805 if that helps. if these two use cases can be solved with one stone | |
| 19:12:41 | melwitt | hah, good point | |
| 19:13:01 | dansmith | yes, mnaser please comment | |
| 19:13:15 | dansmith | sounds like a good reason to let this spec live for more than 24 hours :) | |
| 19:16:43 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:19:37 | mnaser | dansmith, melwitt, mriedem: i left comments | |
| 19:19:58 | mriedem | thanks | |
| 19:20:01 | mnaser | i wonder how much pain i'm signing myself up for, but this is something we'll eventually need | |
| 19:20:02 | melwitt | ++ | |
| 19:20:20 | dansmith | mnaser: you know you like the pain | |
| 19:20:27 | mnaser | ha | |
| 19:20:32 | dansmith | hard drive until it megahertz | |
| 19:21:16 | mnaser | i'm going to start on some wip giant patch just to see what i'm dealing with | |
| 19:21:42 | jmlowe | Quick question, should this log line indicate that an allocation is being deleted? DEBUG nova.compute.resource_tracker [Instance XXXX has been moved to another host XXX. There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 2048, u'DISK_GB': 8}}. _remove_deleted_instances_allocations /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:12 | |
| 19:21:44 | jmlowe | 86 | |
| 19:23:01 | openstackgerrit | Simon Dodsley proposed openstack/nova master: Add enhanced KVM storage QoS quotas https://review.openstack.org/558530 | |
| 19:23:33 | melwitt | mnaser: I think lots of people need it and I think I could help with some of it, our customers run into things with the glance clone to nova part like this https://review.openstack.org/457886 | |
| 19:23:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment https://review.openstack.org/572790 | |
| 19:24:27 | melwitt | mnaser: and like you've said before, we could see people wanting to use separate ceph clusters per cell and it would help that too. and then the issue this other person has in the spec. etc | |