| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-07 | |||
| 19:29:04 | mriedem | ok if the instance is happy on the dest host then you can manually remove the allocations for that instance on the source host | |
| 19:29:29 | mriedem | mayhap we need a placement FAQ like our cells v2 FAQ | |
| 19:29:59 | melwitt | yeah. we had someone run into this same problem internally but I don't have the bz handy | |
| 19:30:24 | mriedem | i bet i do | |
| 19:30:26 | melwitt | or if there was one or if this happened during a testing situation | |
| 19:30:34 | melwitt | can't remember | |
| 19:30:46 | mriedem | https://github.com/openstack/nova/commit/12a3b698d17672ec0277dff4e96c44a92186aab1#diff-afb9c0c0ca5276c7eacd987bbf51d8e6 | |
| 19:31:58 | mnaser | ok the flood of questions may start sorry | |
| 19:32:06 | mnaser | imagebackend and imagecache api's are nothing exposed right? | |
| 19:32:23 | mnaser | as in, we don't expect anyone to be extending an image backend as a consumer of nova | |
| 19:32:35 | melwitt | not that I know of, but that would be an mdbooth question | |
| 19:33:20 | mriedem | none of that stuff is exposed via rest api | |
| 19:33:34 | mriedem | not pluggable either | |
| 19:38:09 | jmlowe | mriedem: that instance seems happy on the dest, also it seems that after a round of host-evacuate-live I now have 4 instances that are double allocated | |
| 19:38:35 | jmlowe | more accurately 3 double allocated and one triple allocated | |
| 19:39:15 | mriedem | errors in the logs? | |
| 19:39:51 | mriedem | jmlowe: btw, the allocation tracking for migrations all got smarter in queens https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html | |
| 19:40:02 | mriedem | hint hint | |
| 19:40:14 | melwitt | jmlowe: do you have the latest release of pike? | |
| 19:40:48 | melwitt | there were some bugs around allocations that got fixed in subsequent point releases of pike, I thought | |
| 19:40:52 | jmlowe | ooh, I did find an error, not in placement-api where I expected but in nova-compute | |
| 19:41:18 | jmlowe | "Failed to clean allocation of migrated instance on the source node" | |
| 19:41:26 | jmlowe | not especially helpful | |
| 19:42:06 | openstackgerrit | Eric Fried proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480 | |
| 19:42:08 | jmlowe | bah, multiline html error confusing my grep | |
| 19:42:53 | mriedem | jmlowe: there should be something before that error in the logs | |
| 19:43:05 | jmlowe | it's a 409 unable to allocated would exceed vcpu capacity | |
| 19:43:43 | jmlowe | which begs the question, how did it get put there in the first place? | |
| 19:44:59 | jmlowe | melwitt: I should be running 16.1.1 | |
| 19:45:46 | mriedem | if you have debug logging enabled in the compute service, you'd see this before it fails http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1644 | |
| 19:46:13 | mriedem | and http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1681 | |
| 19:46:56 | mriedem | what it's trying to do is take the existing 'doubled up' allocations that the instance has on both the source and dest, and remove the source node allocations while maintaining the dest host allocations | |
| 19:47:22 | mriedem | since the instance already has those dest host allocations, i'm not sure why it would complain about that...unless it's considering it a new allocation rather than a replacement | |
| 19:47:43 | jmlowe | http://paste.openstack.org/show/722922/ | |
| 19:47:58 | mriedem | i bet that's where you're running into double/triple allocations | |
| 19:48:05 | mriedem | jaypipes: ^ | |
| 19:48:08 | mriedem | efried: ^ | |
| 19:48:31 | efried | mriedem: tl;dr? | |
| 19:49:43 | jmlowe | get 409 post live migration with placement causing old allocation to not be deleted, wind up with allocations for one instance on 2+ resource providers | |
| 19:49:43 | mriedem | efried: jmlowe is doing live migrations in pike, and at the end when we go to cleanup allocations from the source host, the PUT /allocations is failing because *i think* placement is considering this like a new allocation request against a provider | |
| 19:50:00 | mriedem | yeah in here http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1594 | |
| 19:50:14 | mriedem | in pike we have allocations against the source and dest node during a live migratoin, | |
| 19:50:22 | jmlowe | all fixed in queens? | |
| 19:50:31 | mriedem | post live migration, we try to remove the source node allocation whilst maintaining the dest node allocation by munging the allocations | |
| 19:50:37 | mriedem | yes fixed in queens | |
| 19:50:57 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:51:02 | jmlowe | lol, rolling cluster to get to centos 7.5 in preparation for upgrade to queens | |
| 19:51:25 | mriedem | what i don't get is https://developer.openstack.org/api-ref/placement/#update-allocations says " If allocations already exist for this consumer, they are replaced." | |
| 19:51:30 | mriedem | so it shouldn't consider it a new allocation | |
| 19:51:59 | mriedem | we're doing a PUT /allocatoins here but really it's a PATCH | |
| 19:52:15 | melwitt | is the fix not backportable? | |
| 19:52:21 | mriedem | no | |
| 19:52:43 | melwitt | k, that explains that | |
| 19:52:51 | mriedem | there are rpc version cahnges | |
| 19:52:53 | mriedem | is why | |
| 19:53:06 | mriedem | things that require specs usually aren't backportable | |
| 19:53:08 | jmlowe | I can probably suffer through now that I know what I'm looking for | |
| 19:53:22 | melwitt | mriedem: I didn't know what the change was part of, sorry | |
| 19:53:42 | mriedem | i'm worried that we might not be doing an allocation replacement | |
| 19:53:53 | mriedem | trying to think of a recreate | |
| 19:53:57 | efried | mriedem: I can't understand it either. We ought to be replacing allocs with a subset of same allocs. | |
| 19:54:08 | mriedem | right | |
| 19:54:37 | mriedem | i think if you had a functional test with 2 compute nodes with 1 VCPU each for inventory, create a 1 VCPU instance, live migrate it, that would tell us if we are not replacing | |
| 19:55:11 | efried | And by "you" you mean you. | |
| 19:55:15 | mriedem | server create would claim the 1 VCPU from the source node, live migrate would claim the 1 VCPU from the dest node (doubled up allocs), and post-live migration should remove the 1 VCPU allocation from the source node and maintain the 1 VCPU allocatoin against the dest node | |
| 19:55:25 | mriedem | :( | |
| 19:56:07 | efried | interestingly, we're using 1.10 in that method, which seems unnecessary. | |
| 19:56:48 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L1342 | |
| 19:58:10 | mriedem | i think this is the test we'd want, but it's not using the right fake virt driver https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2313 | |
| 19:58:36 | mriedem | although it does check the allocations that the server has on both the source and dest after the live migration completes | |
| 19:58:51 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2337 | |
| 19:59:02 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2341 | |
| 19:59:07 | mriedem | so we're not doubling | |
| 20:00:31 | mriedem | i also seem to remember cases where "Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider '3f288dee-8183-491e-b488-ad4698141927'. The requested amount would exceed the capacity." is misleading | |
| 20:03:46 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/placement/objects/resource_provider.py#n2086 | |
| 20:04:19 | efried | mriedem: I just walked through that code quickish and it seems sane. | |
| 20:04:25 | mnaser | does anyone know what imagebackend means when it says 'template' | |
| 20:04:38 | mriedem | jmlowe: do you see this warning in the placement-api logs for that same resource provider? http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/placement/objects/resource_provider.py#n2078 | |
| 20:04:52 | mnaser | the cache function for backends says: "Creates image from template." and i've gathered image really a 'storage device' | |
| 20:05:15 | mriedem | for this provider 3f288dee-8183-491e-b488-ad4698141927 | |
| 20:08:58 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 20:10:17 | melwitt | mnaser: looks like that's just supposed to be the function object that will fetch the image, example fetch_raw_image | |
| 20:10:46 | melwitt | (by looking at the only place create_image is called) | |
| 20:10:56 | mnaser | melwitt: thanks.. i've switched between "rip this apart and implement it cleanly and sanely" and "... just make it work" a few times now | |
| 20:11:05 | melwitt | mnaser: heh. yeah. | |
| 20:11:51 | melwitt | I have a feeling that "prepare_template" might have been something else at first and morphed into the fetch function. I dunno | |
| 20:12:09 | mnaser | i was thinking of creating a concept of 'storage pools' in nova, storage pool has many volumes, all the operations for the pool are there depending on the driver | |
| 20:12:28 | mnaser | with a common function for downloading images from glance into said storage pool | |
| 20:13:05 | mnaser | like download_from_glance (using whatever best efficent way to download), with it calling write_to_backend with the data | |
| 20:13:14 | mriedem | mnaser: what you're saying has all been said by mdbooth at least a dozen times | |
| 20:13:31 | mriedem | and a few others re: "re-implement" | |
| 20:13:35 | mnaser | i guess they havent had the time to get aronud it | |
| 20:13:46 | jmlowe | mriedem: chock a block full of those errors | |
| 20:13:47 | mriedem | i don't know if we have the time to tell the tale | |
| 20:14:22 | openstackgerrit | Merged openstack/nova master: Follow up changes to granular placement policy reviews https://review.openstack.org/570588 | |
| 20:14:30 | melwitt | mnaser: if that could be a subclass of backend and then have rbd derive from that, that might work? unless you think the storage pool thing could apply to all drivers? | |
| 20:14:41 | melwitt | I know I'm majorly oversimplifying this | |
| 20:14:50 | mriedem | jmlowe: hmm, not really sure; we have a test case that does this same scenario and asserts the allocations before and after the live migration across 2 computes are what we'd expect | |
| 20:15:11 | mriedem | jmlowe: if you have 4 bad instances, maybe we just need to get those cleaned up, upgrade to queens and forget this ever happened | |
| 20:15:24 | mriedem | w/o a clear recreate it's hard to say what the fix is | |