| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-07 | |||
| 19:20:01 | mnaser | i wonder how much pain i'm signing myself up for, but this is something we'll eventually need | |
| 19:20:02 | melwitt | ++ | |
| 19:20:20 | dansmith | mnaser: you know you like the pain | |
| 19:20:27 | mnaser | ha | |
| 19:20:32 | dansmith | hard drive until it megahertz | |
| 19:21:16 | mnaser | i'm going to start on some wip giant patch just to see what i'm dealing with | |
| 19:21:42 | jmlowe | Quick question, should this log line indicate that an allocation is being deleted? DEBUG nova.compute.resource_tracker [Instance XXXX has been moved to another host XXX. There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 2048, u'DISK_GB': 8}}. _remove_deleted_instances_allocations /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:12 | |
| 19:21:44 | jmlowe | 86 | |
| 19:23:01 | openstackgerrit | Simon Dodsley proposed openstack/nova master: Add enhanced KVM storage QoS quotas https://review.openstack.org/558530 | |
| 19:23:33 | melwitt | mnaser: I think lots of people need it and I think I could help with some of it, our customers run into things with the glance clone to nova part like this https://review.openstack.org/457886 | |
| 19:23:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment https://review.openstack.org/572790 | |
| 19:24:27 | melwitt | mnaser: and like you've said before, we could see people wanting to use separate ceph clusters per cell and it would help that too. and then the issue this other person has in the spec. etc | |
| 19:24:55 | mnaser | melwitt: cool, i'll start reading through code and pushing something preliminary, but yeah this is something that has to happen at some point | |
| 19:26:04 | melwitt | jmlowe: maybe, probably. dansmith did we have a bug or doc that explains/helps with knowing whether allocations need to be cleaned up via the osc-placement cli? | |
| 19:26:21 | mriedem | jmlowe: the code has a big comment with more context on that message http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/resource_tracker.py#n1329 | |
| 19:26:35 | dansmith | it means it would have been cleaned up in ocata, but not later I think | |
| 19:26:44 | mriedem | this is a mixed ocata compute / auto delete allocation thing from pike | |
| 19:27:07 | jmlowe | looking at the placement logs I don't see a delete, it's allocated on two hosts now following a host-evacuate-live | |
| 19:27:09 | dansmith | melwitt: no doc that I know of | |
| 19:27:39 | melwitt | jmlowe: yeah, sorry, misread your question. it doesn't mean it deleted them but is saying it found allocations for an instance that is no longer on the host | |
| 19:27:41 | mriedem | if the source host is dead (evacuated from) then those allocations aren't causing any problems | |
| 19:27:54 | dansmith | host-evacuate-live is live migration | |
| 19:27:56 | dansmith | not evacuation | |
| 19:28:00 | mriedem | oh wait host-evacuate-live is not evac | |
| 19:28:02 | mriedem | yeah | |
| 19:28:06 | mriedem | link me! | |
| 19:28:30 | mriedem | jmlowe: is the instance running properly on the dest host? | |
| 19:28:39 | mriedem | and is the source host pike or later? | |
| 19:28:47 | jmlowe | all pike | |
| 19:29:04 | mriedem | ok if the instance is happy on the dest host then you can manually remove the allocations for that instance on the source host | |
| 19:29:29 | mriedem | mayhap we need a placement FAQ like our cells v2 FAQ | |
| 19:29:59 | melwitt | yeah. we had someone run into this same problem internally but I don't have the bz handy | |
| 19:30:24 | mriedem | i bet i do | |
| 19:30:26 | melwitt | or if there was one or if this happened during a testing situation | |
| 19:30:34 | melwitt | can't remember | |
| 19:30:46 | mriedem | https://github.com/openstack/nova/commit/12a3b698d17672ec0277dff4e96c44a92186aab1#diff-afb9c0c0ca5276c7eacd987bbf51d8e6 | |
| 19:31:58 | mnaser | ok the flood of questions may start sorry | |
| 19:32:06 | mnaser | imagebackend and imagecache api's are nothing exposed right? | |
| 19:32:23 | mnaser | as in, we don't expect anyone to be extending an image backend as a consumer of nova | |
| 19:32:35 | melwitt | not that I know of, but that would be an mdbooth question | |
| 19:33:20 | mriedem | none of that stuff is exposed via rest api | |
| 19:33:34 | mriedem | not pluggable either | |
| 19:38:09 | jmlowe | mriedem: that instance seems happy on the dest, also it seems that after a round of host-evacuate-live I now have 4 instances that are double allocated | |
| 19:38:35 | jmlowe | more accurately 3 double allocated and one triple allocated | |
| 19:39:15 | mriedem | errors in the logs? | |
| 19:39:51 | mriedem | jmlowe: btw, the allocation tracking for migrations all got smarter in queens https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html | |
| 19:40:02 | mriedem | hint hint | |
| 19:40:14 | melwitt | jmlowe: do you have the latest release of pike? | |
| 19:40:48 | melwitt | there were some bugs around allocations that got fixed in subsequent point releases of pike, I thought | |
| 19:40:52 | jmlowe | ooh, I did find an error, not in placement-api where I expected but in nova-compute | |
| 19:41:18 | jmlowe | "Failed to clean allocation of migrated instance on the source node" | |
| 19:41:26 | jmlowe | not especially helpful | |
| 19:42:06 | openstackgerrit | Eric Fried proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480 | |
| 19:42:08 | jmlowe | bah, multiline html error confusing my grep | |
| 19:42:53 | mriedem | jmlowe: there should be something before that error in the logs | |
| 19:43:05 | jmlowe | it's a 409 unable to allocated would exceed vcpu capacity | |
| 19:43:43 | jmlowe | which begs the question, how did it get put there in the first place? | |
| 19:44:59 | jmlowe | melwitt: I should be running 16.1.1 | |
| 19:45:46 | mriedem | if you have debug logging enabled in the compute service, you'd see this before it fails http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1644 | |
| 19:46:13 | mriedem | and http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1681 | |
| 19:46:56 | mriedem | what it's trying to do is take the existing 'doubled up' allocations that the instance has on both the source and dest, and remove the source node allocations while maintaining the dest host allocations | |
| 19:47:22 | mriedem | since the instance already has those dest host allocations, i'm not sure why it would complain about that...unless it's considering it a new allocation rather than a replacement | |
| 19:47:43 | jmlowe | http://paste.openstack.org/show/722922/ | |
| 19:47:58 | mriedem | i bet that's where you're running into double/triple allocations | |
| 19:48:05 | mriedem | jaypipes: ^ | |
| 19:48:08 | mriedem | efried: ^ | |
| 19:48:31 | efried | mriedem: tl;dr? | |
| 19:49:43 | jmlowe | get 409 post live migration with placement causing old allocation to not be deleted, wind up with allocations for one instance on 2+ resource providers | |
| 19:49:43 | mriedem | efried: jmlowe is doing live migrations in pike, and at the end when we go to cleanup allocations from the source host, the PUT /allocations is failing because *i think* placement is considering this like a new allocation request against a provider | |
| 19:50:00 | mriedem | yeah in here http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1594 | |
| 19:50:14 | mriedem | in pike we have allocations against the source and dest node during a live migratoin, | |
| 19:50:22 | jmlowe | all fixed in queens? | |
| 19:50:31 | mriedem | post live migration, we try to remove the source node allocation whilst maintaining the dest node allocation by munging the allocations | |
| 19:50:37 | mriedem | yes fixed in queens | |
| 19:50:57 | openstackgerrit | Simon Dodsley proposed openstack/nova-specs master: Enhanced KVM Storage QoS https://review.openstack.org/340168 | |
| 19:51:02 | jmlowe | lol, rolling cluster to get to centos 7.5 in preparation for upgrade to queens | |
| 19:51:25 | mriedem | what i don't get is https://developer.openstack.org/api-ref/placement/#update-allocations says " If allocations already exist for this consumer, they are replaced." | |
| 19:51:30 | mriedem | so it shouldn't consider it a new allocation | |
| 19:51:59 | mriedem | we're doing a PUT /allocatoins here but really it's a PATCH | |
| 19:52:15 | melwitt | is the fix not backportable? | |
| 19:52:21 | mriedem | no | |
| 19:52:43 | melwitt | k, that explains that | |
| 19:52:51 | mriedem | there are rpc version cahnges | |
| 19:52:53 | mriedem | is why | |
| 19:53:06 | mriedem | things that require specs usually aren't backportable | |
| 19:53:08 | jmlowe | I can probably suffer through now that I know what I'm looking for | |
| 19:53:22 | melwitt | mriedem: I didn't know what the change was part of, sorry | |
| 19:53:42 | mriedem | i'm worried that we might not be doing an allocation replacement | |
| 19:53:53 | mriedem | trying to think of a recreate | |
| 19:53:57 | efried | mriedem: I can't understand it either. We ought to be replacing allocs with a subset of same allocs. | |
| 19:54:08 | mriedem | right | |
| 19:54:37 | mriedem | i think if you had a functional test with 2 compute nodes with 1 VCPU each for inventory, create a 1 VCPU instance, live migrate it, that would tell us if we are not replacing | |
| 19:55:11 | efried | And by "you" you mean you. | |
| 19:55:15 | mriedem | server create would claim the 1 VCPU from the source node, live migrate would claim the 1 VCPU from the dest node (doubled up allocs), and post-live migration should remove the 1 VCPU allocation from the source node and maintain the 1 VCPU allocatoin against the dest node | |
| 19:55:25 | mriedem | :( | |
| 19:56:07 | efried | interestingly, we're using 1.10 in that method, which seems unnecessary. | |
| 19:56:48 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L1342 | |
| 19:58:10 | mriedem | i think this is the test we'd want, but it's not using the right fake virt driver https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2313 | |
| 19:58:36 | mriedem | although it does check the allocations that the server has on both the source and dest after the live migration completes | |
| 19:58:51 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/tests/functional/test_servers.py#L2337 | |