| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-07 | |||
| 19:06:39 | mnaser | but maybe this is what you're talking about but if we decouple the 'export' and 'import' then both my and this users use case can be solved | |
| 19:07:01 | mnaser | if we can do cow, export is noop and import is create cow volume | |
| 19:07:22 | mnaser | if compute node uses local storage and image is on glance, export using rbd export, import using <write-on-disk> | |
| 19:07:45 | mnaser | if compute node uses ceph, image in ceph, but not directly accessible, export by glance-api download, import to ceph cluster | |
| 19:08:29 | mnaser | nova/nova/virt/libvirt/imagecache.py havent been touched since mid-2016 other than privsep stuff, i imagine its flawless :) | |
| 19:08:47 | melwitt | totally flawless | |
| 19:09:19 | mnaser | "This cleanup code will delete all info files the first time it runs in Ocata, which means we can delete this block entirely in P." | |
| 19:10:04 | melwitt | heh | |
| 19:11:29 | mnaser | this doesn't look too crazy, ImageCacheManager could be changed up to have multiple interfaces without too much crazy work scrolling through it | |
| 19:11:48 | melwitt | wanna do it? :) | |
| 19:12:10 | mnaser | i can give it a shot, the only thing that is a concern is locking | |
| 19:12:16 | mnaser | i think ceph allows you to lock an image | |
| 19:12:31 | mriedem | i know someone in the UK that loves locks and the image cache manager | |
| 19:12:32 | melwitt | I was gonna say, maybe you can write your ideas on the spec review https://review.openstack.org/#/c/572805 if that helps. if these two use cases can be solved with one stone | |
| 19:12:41 | melwitt | hah, good point | |
| 19:13:01 | dansmith | yes, mnaser please comment | |
| 19:13:15 | dansmith | sounds like a good reason to let this spec live for more than 24 hours :) | |
| 19:16:43 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:19:37 | mnaser | dansmith, melwitt, mriedem: i left comments | |
| 19:19:58 | mriedem | thanks | |
| 19:20:01 | mnaser | i wonder how much pain i'm signing myself up for, but this is something we'll eventually need | |
| 19:20:02 | melwitt | ++ | |
| 19:20:20 | dansmith | mnaser: you know you like the pain | |
| 19:20:27 | mnaser | ha | |
| 19:20:32 | dansmith | hard drive until it megahertz | |
| 19:21:16 | mnaser | i'm going to start on some wip giant patch just to see what i'm dealing with | |
| 19:21:42 | jmlowe | Quick question, should this log line indicate that an allocation is being deleted? DEBUG nova.compute.resource_tracker [Instance XXXX has been moved to another host XXX. There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 2048, u'DISK_GB': 8}}. _remove_deleted_instances_allocations /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:12 | |
| 19:21:44 | jmlowe | 86 | |
| 19:23:01 | openstackgerrit | Simon Dodsley proposed openstack/nova master: Add enhanced KVM storage QoS quotas https://review.openstack.org/558530 | |
| 19:23:33 | melwitt | mnaser: I think lots of people need it and I think I could help with some of it, our customers run into things with the glance clone to nova part like this https://review.openstack.org/457886 | |
| 19:23:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment https://review.openstack.org/572790 | |
| 19:24:27 | melwitt | mnaser: and like you've said before, we could see people wanting to use separate ceph clusters per cell and it would help that too. and then the issue this other person has in the spec. etc | |
| 19:24:55 | mnaser | melwitt: cool, i'll start reading through code and pushing something preliminary, but yeah this is something that has to happen at some point | |
| 19:26:04 | melwitt | jmlowe: maybe, probably. dansmith did we have a bug or doc that explains/helps with knowing whether allocations need to be cleaned up via the osc-placement cli? | |
| 19:26:21 | mriedem | jmlowe: the code has a big comment with more context on that message http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/resource_tracker.py#n1329 | |
| 19:26:35 | dansmith | it means it would have been cleaned up in ocata, but not later I think | |
| 19:26:44 | mriedem | this is a mixed ocata compute / auto delete allocation thing from pike | |
| 19:27:07 | jmlowe | looking at the placement logs I don't see a delete, it's allocated on two hosts now following a host-evacuate-live | |
| 19:27:09 | dansmith | melwitt: no doc that I know of | |
| 19:27:39 | melwitt | jmlowe: yeah, sorry, misread your question. it doesn't mean it deleted them but is saying it found allocations for an instance that is no longer on the host | |
| 19:27:41 | mriedem | if the source host is dead (evacuated from) then those allocations aren't causing any problems | |
| 19:27:54 | dansmith | host-evacuate-live is live migration | |
| 19:27:56 | dansmith | not evacuation | |
| 19:28:00 | mriedem | oh wait host-evacuate-live is not evac | |
| 19:28:02 | mriedem | yeah | |
| 19:28:06 | mriedem | link me! | |
| 19:28:30 | mriedem | jmlowe: is the instance running properly on the dest host? | |
| 19:28:39 | mriedem | and is the source host pike or later? | |
| 19:28:47 | jmlowe | all pike | |
| 19:29:04 | mriedem | ok if the instance is happy on the dest host then you can manually remove the allocations for that instance on the source host | |
| 19:29:29 | mriedem | mayhap we need a placement FAQ like our cells v2 FAQ | |
| 19:29:59 | melwitt | yeah. we had someone run into this same problem internally but I don't have the bz handy | |
| 19:30:24 | mriedem | i bet i do | |
| 19:30:26 | melwitt | or if there was one or if this happened during a testing situation | |
| 19:30:34 | melwitt | can't remember | |
| 19:30:46 | mriedem | https://github.com/openstack/nova/commit/12a3b698d17672ec0277dff4e96c44a92186aab1#diff-afb9c0c0ca5276c7eacd987bbf51d8e6 | |
| 19:31:58 | mnaser | ok the flood of questions may start sorry | |
| 19:32:06 | mnaser | imagebackend and imagecache api's are nothing exposed right? | |
| 19:32:23 | mnaser | as in, we don't expect anyone to be extending an image backend as a consumer of nova | |
| 19:32:35 | melwitt | not that I know of, but that would be an mdbooth question | |
| 19:33:20 | mriedem | none of that stuff is exposed via rest api | |
| 19:33:34 | mriedem | not pluggable either | |
| 19:38:09 | jmlowe | mriedem: that instance seems happy on the dest, also it seems that after a round of host-evacuate-live I now have 4 instances that are double allocated | |
| 19:38:35 | jmlowe | more accurately 3 double allocated and one triple allocated | |
| 19:39:15 | mriedem | errors in the logs? | |
| 19:39:51 | mriedem | jmlowe: btw, the allocation tracking for migrations all got smarter in queens https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html | |
| 19:40:02 | mriedem | hint hint | |
| 19:40:14 | melwitt | jmlowe: do you have the latest release of pike? | |
| 19:40:48 | melwitt | there were some bugs around allocations that got fixed in subsequent point releases of pike, I thought | |
| 19:40:52 | jmlowe | ooh, I did find an error, not in placement-api where I expected but in nova-compute | |
| 19:41:18 | jmlowe | "Failed to clean allocation of migrated instance on the source node" | |
| 19:41:26 | jmlowe | not especially helpful | |
| 19:42:06 | openstackgerrit | Eric Fried proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480 | |
| 19:42:08 | jmlowe | bah, multiline html error confusing my grep | |
| 19:42:53 | mriedem | jmlowe: there should be something before that error in the logs | |
| 19:43:05 | jmlowe | it's a 409 unable to allocated would exceed vcpu capacity | |
| 19:43:43 | jmlowe | which begs the question, how did it get put there in the first place? | |
| 19:44:59 | jmlowe | melwitt: I should be running 16.1.1 | |
| 19:45:46 | mriedem | if you have debug logging enabled in the compute service, you'd see this before it fails http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1644 | |
| 19:46:13 | mriedem | and http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1681 | |
| 19:46:56 | mriedem | what it's trying to do is take the existing 'doubled up' allocations that the instance has on both the source and dest, and remove the source node allocations while maintaining the dest host allocations | |
| 19:47:22 | mriedem | since the instance already has those dest host allocations, i'm not sure why it would complain about that...unless it's considering it a new allocation rather than a replacement | |
| 19:47:43 | jmlowe | http://paste.openstack.org/show/722922/ | |
| 19:47:58 | mriedem | i bet that's where you're running into double/triple allocations | |
| 19:48:05 | mriedem | jaypipes: ^ | |
| 19:48:08 | mriedem | efried: ^ | |
| 19:48:31 | efried | mriedem: tl;dr? | |
| 19:49:43 | jmlowe | get 409 post live migration with placement causing old allocation to not be deleted, wind up with allocations for one instance on 2+ resource providers | |
| 19:49:43 | mriedem | efried: jmlowe is doing live migrations in pike, and at the end when we go to cleanup allocations from the source host, the PUT /allocations is failing because *i think* placement is considering this like a new allocation request against a provider | |
| 19:50:00 | mriedem | yeah in here http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1594 | |
| 19:50:14 | mriedem | in pike we have allocations against the source and dest node during a live migratoin, | |
| 19:50:22 | jmlowe | all fixed in queens? | |
| 19:50:31 | mriedem | post live migration, we try to remove the source node allocation whilst maintaining the dest node allocation by munging the allocations | |
| 19:50:37 | mriedem | yes fixed in queens | |
| 19:50:57 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:51:02 | jmlowe | lol, rolling cluster to get to centos 7.5 in preparation for upgrade to queens | |
| 19:51:25 | mriedem | what i don't get is https://developer.openstack.org/api-ref/placement/#update-allocations says " If allocations already exist for this consumer, they are replaced." | |
| 19:51:30 | mriedem | so it shouldn't consider it a new allocation | |
| 19:51:59 | mriedem | we're doing a PUT /allocatoins here but really it's a PATCH | |
| 19:52:15 | melwitt | is the fix not backportable? | |