| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-19 | |||
| 15:01:54 | belmoreira | what's the best way to fix wrong allocations? (requires_allocation_refresh = False) | |
| 15:02:46 | mriedem | belmoreira: coincidentally bauzas and i are talking about busted allocations in -placement | |
| 15:02:52 | mriedem | but should probably be talking about that here since it's a nova problem | |
| 15:03:03 | bauzas | actually yeah | |
| 15:03:34 | mriedem | i'm not sure what requires_allocation_refresh is | |
| 15:03:49 | mriedem | resource_provider_association_refresh ? | |
| 15:04:00 | mriedem | resource_provider_association_refresh doesn't touch allocations anyway | |
| 15:05:03 | belmoreira | this is only enable for ironic (the default is false). I can confirm but I'm assuming it always updates the allocations | |
| 15:05:09 | mriedem | belmoreira: well one part is fixing the many bugs that cause incorrect allocatoins, e.g. https://review.opendev.org/#/c/654067/ | |
| 15:05:40 | mriedem | belmoreira: oh, requires_allocation_refresh in the ironic driver, that was removed awhile ago | |
| 15:05:52 | belmoreira | mriedem I agree. But after hitting the bugs operators need something to fix the busted allocations | |
| 15:06:27 | mriedem | yeah i know | |
| 15:06:39 | mriedem | i think https://bugs.launchpad.net/nova/+bug/1793569 is where we've been collecting the various notes on what some operators are doing for scripts | |
| 15:06:39 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 15:07:18 | mriedem | belmoreira: one option is if you know that some instance has the wrong allocations, you can delete the allocations in placement and then run the nova-manage placement heal_allocations command and it should fix the allocations | |
| 15:07:38 | mriedem | but that won't do anything about allocations held by migration records from some failed migration | |
| 15:08:06 | ohwhyosa | mloza, I recommend you checkout also with the people at #openstack-kolla | |
| 15:08:57 | ohwhyosa | I can't help you more, I'm kinda new myself (without the kinda) but since I had a similar problem recently and was helped by sean-k-mooney , I thought I'd pay the favor forward in case it would help | |
| 15:14:21 | bauzas | mriedem: you need to stop your APIs before cleaning up allocations, right? | |
| 15:15:10 | mriedem | umm | |
| 15:15:19 | belmoreira | mriedem in that case the operator needs to identify somehow the wrong allocations. I was thinking in having a "nova-manage" for that and fix or a periodic task | |
| 15:15:21 | mriedem | i guess it depends on what you mean by cleaning up | |
| 15:15:36 | mriedem | belmoreira: see comment 5 that i just posted https://bugs.launchpad.net/nova/+bug/1793569 | |
| 15:15:37 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 15:15:39 | bauzas | mriedem: what you just said 'delete allocations and run nova-manage' | |
| 15:16:04 | mriedem | bauzas: heal_allocations won't do anything to an instance whose task_state is not None, | |
| 15:16:05 | belmoreira | mriedem we had that periodic task in the past, if I remember correctly | |
| 15:16:05 | bauzas | if you delete allocations, you could race if you leave users to be able to create new instances ;) | |
| 15:16:22 | mriedem | bauzas: creating new servers and their allocations isn't the problem you're trying to solve | |
| 15:16:39 | bauzas | I'm misunderstanding what you propose then | |
| 15:16:49 | mriedem | belmoreira: once everything is upgraded to pike, the resource tracker would stop updating allocations for non-ironic nodes | |
| 15:17:57 | mriedem | bauzas: if you're trying to heal allocations for a single instance, you could lock it, delete its allocatoins in placement, and then run heal_allocations and then unlock it | |
| 15:18:25 | bauzas | hah ok, that I understand | |
| 15:18:30 | mriedem | but as i've said in that bug above, and in irc many a time, heal_allocations nor the scripts in that bug from larsks and mnaser deal with stale allocations held by migration consumers | |
| 15:18:53 | bauzas | I thought you were saying about scrubbing the whole allocations table and just do the nova-manage placement heal stuff to recreate all the records | |
| 15:19:11 | mriedem | bauzas: i probably wouldn't do that on a production cloud with a lot of instances | |
| 15:19:17 | bauzas | of course | |
| 15:19:27 | mriedem | unless you use the --dry-run option which was also recently added and not backported | |
| 15:19:40 | mriedem | heal_allocations also doesn't deal with nested allocations (yet) | |
| 15:19:49 | belmoreira | mriedem right. Why it shouldn't continue to happen for libvirt if we continue to require it for ironic? (at least optional) We will have always bugs and this "sync" will be always required | |
| 15:20:02 | mriedem | so our tooling for operators is definitely lagging the complicated features we're shoving in | |
| 15:20:32 | mriedem | belmoreira: first, we no longer have that flag in the ironic driver as i said - it was removed some time ago, either rocky or stein | |
| 15:20:37 | bauzas | belmoreira: for the context, I have some customers that complain about the nova_api (because Queens) DB be growing because of orphaned allocations | |
| 15:21:12 | mriedem | belmoreira: the reason the RT doesn't manage allocations since pike is because starting in pike the scheduler creates the allocations and the RT would trample the allocations during a migration afterward, which is its own problem | |
| 15:21:38 | mriedem | trust me, we had a bunch of bugs going into Pike RC1 and post-GA related to that | |
| 15:22:02 | mriedem | that was addressed in queens with https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html | |
| 15:22:25 | mriedem | but because of ^ and the lack of the RT auto healing things, if a migraiton fails and we don't cleanup properly, you've got stale allocations in placement held by migration records | |
| 15:22:31 | mriedem | and heal_allocations doesn't yet deal with those | |
| 15:22:47 | belmoreira | mridem fair enough | |
| 15:22:50 | mriedem | which leads to the scheduler thinking you have far less capacity than you probably do | |
| 15:23:13 | belmoreira | yeah, heal_allocations only create new allocations | |
| 15:23:32 | bauzas | I guess a "fix my cloud" button would be awesome | |
| 15:23:57 | belmoreira | we need a nova-manage for placement consistency | |
| 15:24:00 | mriedem | or just more people (developers/cores) working on nova that care about fixing these latent issues for operators | |
| 15:24:26 | bauzas | but then https://media.giphy.com/media/xThuW45pxrB820tD0c/giphy.gif | |
| 15:24:29 | mriedem | b/c frankly i'm getting burned out on caring about finding and fixing this stuff | |
| 15:25:04 | bauzas | mriedem: oh yeah I understand you, don't make me wrong | |
| 15:25:13 | belmoreira | mriedem don't get me wrong. I really appreciate all your work on this | |
| 15:25:25 | bauzas | I just think about the possible DB locks a "fix my cloud" button would make | |
| 15:25:40 | mriedem | bauzas: invariably we'd fuck up a "fix my cloud" thing anyway | |
| 15:26:12 | bauzas | ideally, this needs to work online | |
| 15:26:29 | bauzas | but this is the ideal | |
| 15:27:02 | mriedem | as i said in https://bugs.launchpad.net/nova/+bug/1793569/comments/5 i think one low hanging fruit is probably adding something to heal_allocations that scans for allocations held by migratoins which are not in progress and reports on those | |
| 15:27:02 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 15:27:06 | bauzas | belmoreira: you'd be okay with some tool that 'd fix allocations by a maintenance window ? | |
| 15:27:36 | bauzas | I mean, I guess the answer, it'd depend on the required time window :-) | |
| 15:28:38 | bauzas | mriedem: couldn't we just report allocations that are either for not-in-progress migrations or just not related to any instance or migration UUID ? | |
| 15:28:48 | mriedem | i'm torn about how much functionality to shove into heal_allocations b/c it's going to get more complicated https://review.opendev.org/#/q/topic:bug/1819923+(status:open+OR+status:merged) | |
| 15:29:35 | mriedem | bauzas: sure we *could* but that's likely a separate command i'd think | |
| 15:29:44 | mriedem | nova-manage placement audit_allocations or something | |
| 15:30:33 | cdent | which is part of how we ended up wanting consumer types | |
| 15:30:42 | cdent | for most clouds (so far) it won't matter because all allocations will come from nova | |
| 15:30:53 | mriedem | i thought about that, and we could fudge around consumer types for now by checking the resource classes involved to know it's coming from nova | |
| 15:31:12 | bauzas | cdent: heh touché | |
| 15:31:45 | bauzas | you know what ? | |
| 15:31:50 | mriedem | we could also use some troubleshooting docs about "my allocations seem all f'ed up, how can i tell for sure?" but i'm not sure how easy that is to write, but it's something that comes up in here almost every week i think | |
| 15:32:08 | bauzas | I've been given that escalation so I have free time to work on it | |
| 15:33:15 | cdent | having those docs and tools coming from the people who are experiencing and/or fixing the bugs would be ideal as they have the most idea of what's going on. in a perfect universe we'd have lots of people running around to do it, but we're long past that | |
| 15:33:22 | bauzas | mriedem: specless BP, there is ? | |
| 15:33:46 | mriedem | bauzas: i don't think providing tooling for fixing our messes is a feature | |
| 15:34:05 | sean-k-mooney | its not really a bug either | |
| 15:34:16 | sean-k-mooney | the thing that is cause the mess is a bug | |
| 15:34:26 | sean-k-mooney | i guess a tool could be a partial fix for that but | |
| 15:34:51 | sean-k-mooney | are you ok with tieing the tool to the bug rather then a seperate thing | |
| 15:34:57 | mriedem | https://bugs.launchpad.net/nova/+bug/1793569 is a bug | |
| 15:34:58 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 15:35:09 | bauzas | cool, I'll use this bug | |
| 15:35:51 | sean-k-mooney | am ok that feels more like an RFE then a bug but ok | |
| 15:36:03 | sean-k-mooney | no need to invent more paperwork when we dont have too | |
| 15:36:13 | mriedem | bauzas: i wouldn't mind you posting something to the ML with what you plan on adding before starting a bunch of work on it | |
| 15:36:43 | bauzas | ack, tagging ops too | |
| 15:36:56 | belmoreira | bauzas yes please | |
| 15:39:01 | mriedem | bauzas: you can reply to this thread i started last september http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004223.html | |
| 15:39:30 | bauzas | mriedem: cool | |
| 15:40:30 | mriedem | i'll also say i expect functional tests for anything added b/c unit tests don't cut it with this kind of stuff that involves placement | |
| 15:42:04 | mloza | ohwhyosa: how did fix your issue? | |
| 15:43:40 | bauzas | belmoreira: do you know if [ops] tag in the email subject is enough for getting ops' eye on what I write ? | |
| 15:44:29 | mriedem | yes, it should be | |
| 15:44:47 | belmoreira | is the tag that we are using | |
| 15:44:58 | mriedem | it's what i used on http://lists.openstack.org/pipermail/openstack-discuss/2019-June/thread.html#7097 | |