Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
15:07:38 mriedem but that won't do anything about allocations held by migration records from some failed migration
15:08:06 ohwhyosa mloza, I recommend you checkout also with the people at #openstack-kolla
15:08:57 ohwhyosa I can't help you more, I'm kinda new myself (without the kinda) but since I had a similar problem recently and was helped by sean-k-mooney , I thought I'd pay the favor forward in case it would help
15:14:21 bauzas mriedem: you need to stop your APIs before cleaning up allocations, right?
15:15:10 mriedem umm
15:15:19 belmoreira mriedem in that case the operator needs to identify somehow the wrong allocations. I was thinking in having a "nova-manage" for that and fix or a periodic task
15:15:21 mriedem i guess it depends on what you mean by cleaning up
15:15:36 mriedem belmoreira: see comment 5 that i just posted https://bugs.launchpad.net/nova/+bug/1793569
15:15:37 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:15:39 bauzas mriedem: what you just said 'delete allocations and run nova-manage'
15:16:04 mriedem bauzas: heal_allocations won't do anything to an instance whose task_state is not None,
15:16:05 belmoreira mriedem we had that periodic task in the past, if I remember correctly
15:16:05 bauzas if you delete allocations, you could race if you leave users to be able to create new instances ;)
15:16:22 mriedem bauzas: creating new servers and their allocations isn't the problem you're trying to solve
15:16:39 bauzas I'm misunderstanding what you propose then
15:16:49 mriedem belmoreira: once everything is upgraded to pike, the resource tracker would stop updating allocations for non-ironic nodes
15:17:57 mriedem bauzas: if you're trying to heal allocations for a single instance, you could lock it, delete its allocatoins in placement, and then run heal_allocations and then unlock it
15:18:25 bauzas hah ok, that I understand
15:18:30 mriedem but as i've said in that bug above, and in irc many a time, heal_allocations nor the scripts in that bug from larsks and mnaser deal with stale allocations held by migration consumers
15:18:53 bauzas I thought you were saying about scrubbing the whole allocations table and just do the nova-manage placement heal stuff to recreate all the records
15:19:11 mriedem bauzas: i probably wouldn't do that on a production cloud with a lot of instances
15:19:17 bauzas of course
15:19:27 mriedem unless you use the --dry-run option which was also recently added and not backported
15:19:40 mriedem heal_allocations also doesn't deal with nested allocations (yet)
15:19:49 belmoreira mriedem right. Why it shouldn't continue to happen for libvirt if we continue to require it for ironic? (at least optional) We will have always bugs and this "sync" will be always required
15:20:02 mriedem so our tooling for operators is definitely lagging the complicated features we're shoving in
15:20:32 mriedem belmoreira: first, we no longer have that flag in the ironic driver as i said - it was removed some time ago, either rocky or stein
15:20:37 bauzas belmoreira: for the context, I have some customers that complain about the nova_api (because Queens) DB be growing because of orphaned allocations
15:21:12 mriedem belmoreira: the reason the RT doesn't manage allocations since pike is because starting in pike the scheduler creates the allocations and the RT would trample the allocations during a migration afterward, which is its own problem
15:21:38 mriedem trust me, we had a bunch of bugs going into Pike RC1 and post-GA related to that
15:22:02 mriedem that was addressed in queens with https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html
15:22:25 mriedem but because of ^ and the lack of the RT auto healing things, if a migraiton fails and we don't cleanup properly, you've got stale allocations in placement held by migration records
15:22:31 mriedem and heal_allocations doesn't yet deal with those
15:22:47 belmoreira mridem fair enough
15:22:50 mriedem which leads to the scheduler thinking you have far less capacity than you probably do
15:23:13 belmoreira yeah, heal_allocations only create new allocations
15:23:32 bauzas I guess a "fix my cloud" button would be awesome
15:23:57 belmoreira we need a nova-manage for placement consistency
15:24:00 mriedem or just more people (developers/cores) working on nova that care about fixing these latent issues for operators
15:24:26 bauzas but then https://media.giphy.com/media/xThuW45pxrB820tD0c/giphy.gif
15:24:29 mriedem b/c frankly i'm getting burned out on caring about finding and fixing this stuff
15:25:04 bauzas mriedem: oh yeah I understand you, don't make me wrong
15:25:13 belmoreira mriedem don't get me wrong. I really appreciate all your work on this
15:25:25 bauzas I just think about the possible DB locks a "fix my cloud" button would make
15:25:40 mriedem bauzas: invariably we'd fuck up a "fix my cloud" thing anyway
15:26:12 bauzas ideally, this needs to work online
15:26:29 bauzas but this is the ideal
15:27:02 mriedem as i said in https://bugs.launchpad.net/nova/+bug/1793569/comments/5 i think one low hanging fruit is probably adding something to heal_allocations that scans for allocations held by migratoins which are not in progress and reports on those
15:27:02 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:27:06 bauzas belmoreira: you'd be okay with some tool that 'd fix allocations by a maintenance window ?
15:27:36 bauzas I mean, I guess the answer, it'd depend on the required time window :-)
15:28:38 bauzas mriedem: couldn't we just report allocations that are either for not-in-progress migrations or just not related to any instance or migration UUID ?
15:28:48 mriedem i'm torn about how much functionality to shove into heal_allocations b/c it's going to get more complicated https://review.opendev.org/#/q/topic:bug/1819923+(status:open+OR+status:merged)
15:29:35 mriedem bauzas: sure we *could* but that's likely a separate command i'd think
15:29:44 mriedem nova-manage placement audit_allocations or something
15:30:33 cdent which is part of how we ended up wanting consumer types
15:30:42 cdent for most clouds (so far) it won't matter because all allocations will come from nova
15:30:53 mriedem i thought about that, and we could fudge around consumer types for now by checking the resource classes involved to know it's coming from nova
15:31:12 bauzas cdent: heh touché
15:31:45 bauzas you know what ?
15:31:50 mriedem we could also use some troubleshooting docs about "my allocations seem all f'ed up, how can i tell for sure?" but i'm not sure how easy that is to write, but it's something that comes up in here almost every week i think
15:32:08 bauzas I've been given that escalation so I have free time to work on it
15:33:15 cdent having those docs and tools coming from the people who are experiencing and/or fixing the bugs would be ideal as they have the most idea of what's going on. in a perfect universe we'd have lots of people running around to do it, but we're long past that
15:33:22 bauzas mriedem: specless BP, there is ?
15:33:46 mriedem bauzas: i don't think providing tooling for fixing our messes is a feature
15:34:05 sean-k-mooney its not really a bug either
15:34:16 sean-k-mooney the thing that is cause the mess is a bug
15:34:26 sean-k-mooney i guess a tool could be a partial fix for that but
15:34:51 sean-k-mooney are you ok with tieing the tool to the bug rather then a seperate thing
15:34:57 mriedem https://bugs.launchpad.net/nova/+bug/1793569 is a bug
15:34:58 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:35:09 bauzas cool, I'll use this bug
15:35:51 sean-k-mooney am ok that feels more like an RFE then a bug but ok
15:36:03 sean-k-mooney no need to invent more paperwork when we dont have too
15:36:13 mriedem bauzas: i wouldn't mind you posting something to the ML with what you plan on adding before starting a bunch of work on it
15:36:43 bauzas ack, tagging ops too
15:36:56 belmoreira bauzas yes please
15:39:01 mriedem bauzas: you can reply to this thread i started last september http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004223.html
15:39:30 bauzas mriedem: cool
15:40:30 mriedem i'll also say i expect functional tests for anything added b/c unit tests don't cut it with this kind of stuff that involves placement
15:42:04 mloza ohwhyosa: how did fix your issue?
15:43:40 bauzas belmoreira: do you know if [ops] tag in the email subject is enough for getting ops' eye on what I write ?
15:44:29 mriedem yes, it should be
15:44:47 belmoreira is the tag that we are using
15:44:58 mriedem it's what i used on http://lists.openstack.org/pipermail/openstack-discuss/2019-June/thread.html#7097
15:45:11 mriedem which is related to all of this also - nova mis-managing placement resources
15:46:46 artom sean-k-mooney, remind me again, what's the situation where we can have a single instance with bind-time and plug-time events?
15:46:53 mriedem bauzas: mayhap while you have been escalated to spend time upstream, you'd like to review https://review.opendev.org/#/q/topic:bug/1825537+(status:open+OR+status:merged) as well
15:47:11 artom OVS hybrid-plugged ports for the former, what's the latter?
15:47:23 artom 'cuz hybrid-plug is a neutron-wide setting, no?
15:47:28 mriedem artom: sriov direct-physical i thought
15:47:37 sean-k-mooney ovs + sriov or just tow different sriov ports
15:47:38 mriedem artom: different types of ports attached to the server
15:48:18 artom mriedem, that's what I have in my commit message currently (not pushed yet), but it means I got https://review.opendev.org/#/c/664431/ completely wrong
15:48:44 sean-k-mooney e.g. one port that is vnic_type=direct-physical + another port that is vnic_type=direct
15:48:50 artom I think I'l push, sean-k-mooney can shoot it down ;)
15:48:58 mriedem belmoreira: while you're around, here is another tool for ops for you https://review.opendev.org/#/c/655908/
15:49:35 sean-k-mooney artom: if i think its incorrect ills leave a comment or updated it for you when i review
15:49:45 artom sean-k-mooney, appreciated :)
15:50:17 sean-k-mooney but ya it basicaly only happens if you have two different network backend attached to the same vm which only happens if you are using sriov really

Earlier   Later