| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-07 | |||
| 19:49:43 | mriedem | efried: jmlowe is doing live migrations in pike, and at the end when we go to cleanup allocations from the source host, the PUT /allocations is failing because *i think* placement is considering this like a new allocation request against a provider | |
| 19:49:43 | jmlowe | get 409 post live migration with placement causing old allocation to not be deleted, wind up with allocations for one instance on 2+ resource providers | |
| 19:50:00 | mriedem | yeah in here http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1594 | |
| 19:50:14 | mriedem | in pike we have allocations against the source and dest node during a live migratoin, | |
| 19:50:22 | jmlowe | all fixed in queens? | |
| 19:50:31 | mriedem | post live migration, we try to remove the source node allocation whilst maintaining the dest node allocation by munging the allocations | |
| 19:50:37 | mriedem | yes fixed in queens | |
| 19:50:57 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:51:02 | jmlowe | lol, rolling cluster to get to centos 7.5 in preparation for upgrade to queens | |
| 19:51:25 | mriedem | what i don't get is https://developer.openstack.org/api-ref/placement/#update-allocations says " If allocations already exist for this consumer, they are replaced." | |
| 19:51:30 | mriedem | so it shouldn't consider it a new allocation | |
| 19:51:59 | mriedem | we're doing a PUT /allocatoins here but really it's a PATCH | |
| 19:52:15 | melwitt | is the fix not backportable? | |
| 19:52:21 | mriedem | no | |
| 19:52:43 | melwitt | k, that explains that | |
| 19:52:51 | mriedem | there are rpc version cahnges | |
| 19:52:53 | mriedem | is why | |
| 19:53:06 | mriedem | things that require specs usually aren't backportable | |
| 19:53:08 | jmlowe | I can probably suffer through now that I know what I'm looking for | |
| 19:53:22 | melwitt | mriedem: I didn't know what the change was part of, sorry | |
| 19:53:42 | mriedem | i'm worried that we might not be doing an allocation replacement | |
| 19:53:53 | mriedem | trying to think of a recreate | |
| 19:53:57 | efried | mriedem: I can't understand it either. We ought to be replacing allocs with a subset of same allocs. | |
| 19:54:08 | mriedem | right | |
| 19:54:37 | mriedem | i think if you had a functional test with 2 compute nodes with 1 VCPU each for inventory, create a 1 VCPU instance, live migrate it, that would tell us if we are not replacing | |
| 19:55:11 | efried | And by "you" you mean you. | |
| 19:55:15 | mriedem | server create would claim the 1 VCPU from the source node, live migrate would claim the 1 VCPU from the dest node (doubled up allocs), and post-live migration should remove the 1 VCPU allocation from the source node and maintain the 1 VCPU allocatoin against the dest node | |
| 19:55:25 | mriedem | :( | |
| 19:56:07 | efried | interestingly, we're using 1.10 in that method, which seems unnecessary. | |
| 19:56:48 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L1342 | |
| 19:58:10 | mriedem | i think this is the test we'd want, but it's not using the right fake virt driver https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2313 | |
| 19:58:36 | mriedem | although it does check the allocations that the server has on both the source and dest after the live migration completes | |
| 19:58:51 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2337 | |
| 19:59:02 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2341 | |
| 19:59:07 | mriedem | so we're not doubling | |
| 20:00:31 | mriedem | i also seem to remember cases where "Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider '3f288dee-8183-491e-b488-ad4698141927'. The requested amount would exceed the capacity." is misleading | |
| 20:03:46 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/placement/objects/resource_provider.py#n2086 | |
| 20:04:19 | efried | mriedem: I just walked through that code quickish and it seems sane. | |
| 20:04:25 | mnaser | does anyone know what imagebackend means when it says 'template' | |
| 20:04:38 | mriedem | jmlowe: do you see this warning in the placement-api logs for that same resource provider? http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/placement/objects/resource_provider.py#n2078 | |
| 20:04:52 | mnaser | the cache function for backends says: "Creates image from template." and i've gathered image really a 'storage device' | |
| 20:05:15 | mriedem | for this provider 3f288dee-8183-491e-b488-ad4698141927 | |
| 20:08:58 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 20:10:17 | melwitt | mnaser: looks like that's just supposed to be the function object that will fetch the image, example fetch_raw_image | |
| 20:10:46 | melwitt | (by looking at the only place create_image is called) | |
| 20:10:56 | mnaser | melwitt: thanks.. i've switched between "rip this apart and implement it cleanly and sanely" and "... just make it work" a few times now | |
| 20:11:05 | melwitt | mnaser: heh. yeah. | |
| 20:11:51 | melwitt | I have a feeling that "prepare_template" might have been something else at first and morphed into the fetch function. I dunno | |
| 20:12:09 | mnaser | i was thinking of creating a concept of 'storage pools' in nova, storage pool has many volumes, all the operations for the pool are there depending on the driver | |
| 20:12:28 | mnaser | with a common function for downloading images from glance into said storage pool | |
| 20:13:05 | mnaser | like download_from_glance (using whatever best efficent way to download), with it calling write_to_backend with the data | |
| 20:13:14 | mriedem | mnaser: what you're saying has all been said by mdbooth at least a dozen times | |
| 20:13:31 | mriedem | and a few others re: "re-implement" | |
| 20:13:35 | mnaser | i guess they havent had the time to get aronud it | |
| 20:13:46 | jmlowe | mriedem: chock a block full of those errors | |
| 20:13:47 | mriedem | i don't know if we have the time to tell the tale | |
| 20:14:22 | openstackgerrit | Merged openstack/nova master: Follow up changes to granular placement policy reviews https://review.openstack.org/570588 | |
| 20:14:30 | melwitt | mnaser: if that could be a subclass of backend and then have rbd derive from that, that might work? unless you think the storage pool thing could apply to all drivers? | |
| 20:14:41 | melwitt | I know I'm majorly oversimplifying this | |
| 20:14:50 | mriedem | jmlowe: hmm, not really sure; we have a test case that does this same scenario and asserts the allocations before and after the live migration across 2 computes are what we'd expect | |
| 20:15:11 | mriedem | jmlowe: if you have 4 bad instances, maybe we just need to get those cleaned up, upgrade to queens and forget this ever happened | |
| 20:15:24 | jmlowe | mriedem: I'm down with that | |
| 20:15:24 | mriedem | w/o a clear recreate it's hard to say what the fix is | |
| 20:15:32 | mnaser | melwitt: something that can be applied to all drivers, leveraging the 'Backend' class that exists in imagebackend.py and the existing implemented backends | |
| 20:15:44 | melwitt | k | |
| 20:15:50 | openstackgerrit | Merged openstack/nova master: Support nested alloc cands with sharing providers https://review.openstack.org/567508 | |
| 20:15:58 | openstackgerrit | Merged openstack/nova master: Use list instead of set for duplicate check https://review.openstack.org/569720 | |
| 20:16:02 | mnaser | but again that's like the 6th solution i've thought of so far | |
| 20:16:32 | melwitt | jmlowe: you'll need to install osc-placement (it's a plugin for openstackclient) and use the CLI to clean up the stale allocations after confirming the instances are all good on the destination | |
| 20:16:38 | melwitt | if you want to go that route | |
| 20:16:50 | jmlowe | melwitt: way ahead of you there | |
| 20:16:55 | melwitt | hah, cool | |
| 20:23:05 | melwitt | mnaser: fwiw, if there's a minimalistic way we could add the stuff for ceph that isn't fugly, I'd go that route. and maybe try to work backward to make it nicer and more general for other drivers. just MHO | |
| 20:23:58 | mnaser | melwitt: that's what i'm thinking.. like adding some extra code before `clone` and `create_image` for caching.. | |
| 20:24:54 | melwitt | yeah, something. maybe an intermediate layer that does the caching part | |
| 20:35:20 | efried | mriedem, jmlowe: It's not consistent? | |
| 20:50:07 | melwitt | nova meeting in 10 min | |
| 20:51:01 | dansmith | melwitt: I have a thing distracting me right now - can you just summarize the cells meeting for me? | |
| 20:51:11 | melwitt | sure | |
| 20:51:41 | mnaser | melwitt: thinking this through a bit more.. i'm thinking maybe glance image cache could be a cleaner way... if glance adds locations when it caches things.. this little bit here .. https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L923-L926 | |
| 20:52:05 | mnaser | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L199-L225 | |
| 20:52:18 | mnaser | would just skip the ones with different fsid and do the normal cow for the one matching | |
| 20:54:00 | melwitt | oh, hm | |
| 20:54:49 | melwitt | have to think about it more, I'm not _that_ familiar with these | |
| 20:55:00 | mnaser | yeah, i'm still kinda hoping from one possible solutino to another | |
| 20:56:44 | mnaser | https://bugzilla.redhat.com/show_bug.cgi?id=1225775 | |
| 20:56:45 | openstack | bugzilla.redhat.com bug 1225775 in openstack-glance "[RFE] Glance should be able to manage multiple Ceph clusters as backends" [Urgent,Assigned] - Assigned to akekane | |
| 20:57:10 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 20:57:54 | mriedem1 | another pretty trivial spec ^ | |
| 20:58:02 | mriedem1 | pass through more volume qos specs to the guest | |
| 20:58:10 | mriedem | probably didn't need a spec actually | |
| 21:26:32 | openstackgerrit | Merged openstack/nova master: Fix issues in nova-show-usage-statistics-for-hosts-instances.rst https://review.openstack.org/573077 | |
| 21:26:39 | openstackgerrit | Merged openstack/nova master: Fix bug to doc:nova-status https://review.openstack.org/573149 | |
| 21:26:46 | openstackgerrit | Merged openstack/nova master: Delete duplicate functions in placement test https://review.openstack.org/573193 | |
| 21:26:55 | openstackgerrit | Merged openstack/nova master: Enhance api-guide general info some updates https://review.openstack.org/561773 | |
| 21:36:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_compute_api.py (3) https://review.openstack.org/568184 | |
| 21:36:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_compute_api.py (4) https://review.openstack.org/568462 | |
| 21:36:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (4) https://review.openstack.org/570750 | |
| 21:37:11 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (5) https://review.openstack.org/570842 | |
| 21:37:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (6) https://review.openstack.org/571330 | |