| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-07 | |||
| 18:55:45 | mnaser | so we download the image once and then it remains cached inside the cluster | |
| 18:56:20 | mnaser | which is a problem i'm about to run into next week that i haven't fully thought out what/how i go about this :( | |
| 18:56:22 | mriedem | so maybe that's an alternative for the spec | |
| 18:56:26 | mriedem | make image cache ceph aware | |
| 18:56:53 | mriedem | mnaser: there are nova guidance counselors here to help :) | |
| 18:57:08 | mnaser | image cache and backend are decoupled.. or not? | |
| 18:58:13 | mnaser | mriedem: https://wiki.openstack.org/wiki/Technical_Committee_Tracker#Project_Teams i'm signed up ha | |
| 18:58:30 | mnaser | so if they are decoupled, that would solve that problem for that use case in that spec | |
| 18:58:36 | mriedem | ha | |
| 18:58:39 | mnaser | it would involve an extra import/export | |
| 18:58:58 | mnaser | because you'd be exporting it from glance once slowly and importing it to ceph, storing it potentially twice if its the same cluster | |
| 18:59:33 | mnaser | in my case i'd be storing it twice but in different clusters | |
| 19:01:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: libvirt: Don't report DISK_GB if sharing https://review.openstack.org/560459 | |
| 19:01:59 | melwitt | cloned from it) | |
| 19:01:59 | melwitt | yeah, I dunno if I'm missing something but it sounds like we could do this by adding an extra step in the process, like instead of cow cloning from the glance pool, we take an extra step first to copy it and flatten to the nova pool (if does not exist) and then cow clone off of that instead. it would also decouple instances from the images in glance (problem of being unable to delete a glance image if any instance is in existing cow | |
| 19:02:32 | melwitt | *instance is existing that was cow cloned from it | |
| 19:02:38 | mnaser | melwitt: thats a pretty good way of going about things and it does remove that weird | |
| 19:02:45 | mnaser | "i cant delete my image!!!" complaints with rbd | |
| 19:02:48 | melwitt | yeah | |
| 19:03:22 | mnaser | the only thing is that does potentially slow down first boots | |
| 19:03:37 | mnaser | and i'm not sure how that would be handled if someone uploads an image and boots 10 VMs of it right away | |
| 19:03:38 | melwitt | yeah | |
| 19:04:44 | melwitt | heh, maybe this ties into that spec, rbd export, is that faster than clone + flatten or the same I would think? | |
| 19:05:07 | mnaser | i think flatten is faster because flatten doesnt involve the host that does it | |
| 19:05:42 | mnaser | export means you download as fast as your compute node, and upload as fast as your compute node in import | |
| 19:05:58 | melwitt | oh, okay | |
| 19:06:12 | mnaser | where as flatten is this cluster operation where osds talk to themselves | |
| 19:06:20 | melwitt | got it | |
| 19:06:39 | mnaser | but maybe this is what you're talking about but if we decouple the 'export' and 'import' then both my and this users use case can be solved | |
| 19:07:01 | mnaser | if we can do cow, export is noop and import is create cow volume | |
| 19:07:22 | mnaser | if compute node uses local storage and image is on glance, export using rbd export, import using <write-on-disk> | |
| 19:07:45 | mnaser | if compute node uses ceph, image in ceph, but not directly accessible, export by glance-api download, import to ceph cluster | |
| 19:08:29 | mnaser | nova/nova/virt/libvirt/imagecache.py havent been touched since mid-2016 other than privsep stuff, i imagine its flawless :) | |
| 19:08:47 | melwitt | totally flawless | |
| 19:09:19 | mnaser | "This cleanup code will delete all info files the first time it runs in Ocata, which means we can delete this block entirely in P." | |
| 19:10:04 | melwitt | heh | |
| 19:11:29 | mnaser | this doesn't look too crazy, ImageCacheManager could be changed up to have multiple interfaces without too much crazy work scrolling through it | |
| 19:11:48 | melwitt | wanna do it? :) | |
| 19:12:10 | mnaser | i can give it a shot, the only thing that is a concern is locking | |
| 19:12:16 | mnaser | i think ceph allows you to lock an image | |
| 19:12:31 | mriedem | i know someone in the UK that loves locks and the image cache manager | |
| 19:12:32 | melwitt | I was gonna say, maybe you can write your ideas on the spec review https://review.openstack.org/#/c/572805 if that helps. if these two use cases can be solved with one stone | |
| 19:12:41 | melwitt | hah, good point | |
| 19:13:01 | dansmith | yes, mnaser please comment | |
| 19:13:15 | dansmith | sounds like a good reason to let this spec live for more than 24 hours :) | |
| 19:16:43 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:19:37 | mnaser | dansmith, melwitt, mriedem: i left comments | |
| 19:19:58 | mriedem | thanks | |
| 19:20:01 | mnaser | i wonder how much pain i'm signing myself up for, but this is something we'll eventually need | |
| 19:20:02 | melwitt | ++ | |
| 19:20:20 | dansmith | mnaser: you know you like the pain | |
| 19:20:27 | mnaser | ha | |
| 19:20:32 | dansmith | hard drive until it megahertz | |
| 19:21:16 | mnaser | i'm going to start on some wip giant patch just to see what i'm dealing with | |
| 19:21:42 | jmlowe | Quick question, should this log line indicate that an allocation is being deleted? DEBUG nova.compute.resource_tracker [Instance XXXX has been moved to another host XXX. There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 2048, u'DISK_GB': 8}}. _remove_deleted_instances_allocations /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:12 | |
| 19:21:44 | jmlowe | 86 | |
| 19:23:01 | openstackgerrit | Simon Dodsley proposed openstack/nova master: Add enhanced KVM storage QoS quotas https://review.openstack.org/558530 | |
| 19:23:33 | melwitt | mnaser: I think lots of people need it and I think I could help with some of it, our customers run into things with the glance clone to nova part like this https://review.openstack.org/457886 | |
| 19:23:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment https://review.openstack.org/572790 | |
| 19:24:27 | melwitt | mnaser: and like you've said before, we could see people wanting to use separate ceph clusters per cell and it would help that too. and then the issue this other person has in the spec. etc | |
| 19:24:55 | mnaser | melwitt: cool, i'll start reading through code and pushing something preliminary, but yeah this is something that has to happen at some point | |
| 19:26:04 | melwitt | jmlowe: maybe, probably. dansmith did we have a bug or doc that explains/helps with knowing whether allocations need to be cleaned up via the osc-placement cli? | |
| 19:26:21 | mriedem | jmlowe: the code has a big comment with more context on that message http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/resource_tracker.py#n1329 | |
| 19:26:35 | dansmith | it means it would have been cleaned up in ocata, but not later I think | |
| 19:26:44 | mriedem | this is a mixed ocata compute / auto delete allocation thing from pike | |
| 19:27:07 | jmlowe | looking at the placement logs I don't see a delete, it's allocated on two hosts now following a host-evacuate-live | |
| 19:27:09 | dansmith | melwitt: no doc that I know of | |
| 19:27:39 | melwitt | jmlowe: yeah, sorry, misread your question. it doesn't mean it deleted them but is saying it found allocations for an instance that is no longer on the host | |
| 19:27:41 | mriedem | if the source host is dead (evacuated from) then those allocations aren't causing any problems | |
| 19:27:54 | dansmith | host-evacuate-live is live migration | |
| 19:27:56 | dansmith | not evacuation | |
| 19:28:00 | mriedem | oh wait host-evacuate-live is not evac | |
| 19:28:02 | mriedem | yeah | |
| 19:28:06 | mriedem | link me! | |
| 19:28:30 | mriedem | jmlowe: is the instance running properly on the dest host? | |
| 19:28:39 | mriedem | and is the source host pike or later? | |
| 19:28:47 | jmlowe | all pike | |
| 19:29:04 | mriedem | ok if the instance is happy on the dest host then you can manually remove the allocations for that instance on the source host | |
| 19:29:29 | mriedem | mayhap we need a placement FAQ like our cells v2 FAQ | |
| 19:29:59 | melwitt | yeah. we had someone run into this same problem internally but I don't have the bz handy | |
| 19:30:24 | mriedem | i bet i do | |
| 19:30:26 | melwitt | or if there was one or if this happened during a testing situation | |
| 19:30:34 | melwitt | can't remember | |
| 19:30:46 | mriedem | https://github.com/openstack/nova/commit/12a3b698d17672ec0277dff4e96c44a92186aab1#diff-afb9c0c0ca5276c7eacd987bbf51d8e6 | |
| 19:31:58 | mnaser | ok the flood of questions may start sorry | |
| 19:32:06 | mnaser | imagebackend and imagecache api's are nothing exposed right? | |
| 19:32:23 | mnaser | as in, we don't expect anyone to be extending an image backend as a consumer of nova | |
| 19:32:35 | melwitt | not that I know of, but that would be an mdbooth question | |
| 19:33:20 | mriedem | none of that stuff is exposed via rest api | |
| 19:33:34 | mriedem | not pluggable either | |
| 19:38:09 | jmlowe | mriedem: that instance seems happy on the dest, also it seems that after a round of host-evacuate-live I now have 4 instances that are double allocated | |
| 19:38:35 | jmlowe | more accurately 3 double allocated and one triple allocated | |
| 19:39:15 | mriedem | errors in the logs? | |
| 19:39:51 | mriedem | jmlowe: btw, the allocation tracking for migrations all got smarter in queens https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html | |
| 19:40:02 | mriedem | hint hint | |
| 19:40:14 | melwitt | jmlowe: do you have the latest release of pike? | |
| 19:40:48 | melwitt | there were some bugs around allocations that got fixed in subsequent point releases of pike, I thought | |
| 19:40:52 | jmlowe | ooh, I did find an error, not in placement-api where I expected but in nova-compute | |
| 19:41:18 | jmlowe | "Failed to clean allocation of migrated instance on the source node" | |
| 19:41:26 | jmlowe | not especially helpful | |