Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-25
12:59:00 efried FYI there's a community-wide goal getting set up for Train to try to get more of the CLIs up to parity
12:59:27 efried If you have the means, it would be great if you wanted to propose a fix for this.
13:00:07 shilpasd <efried> sure
13:00:16 shilpasd efried: sure
13:00:18 sean-k-mooney efried: in this case its an openstacsdk limitation not a cli issue
13:00:33 efried sean-k-mooney: Yes, I agree. shilpasd opened the bug against the right component.
13:00:41 sean-k-mooney nova cli and osc do not use the sdk for the api currently
13:00:56 sean-k-mooney ah yes they did
13:02:45 sean-k-mooney efried: stephenfin and i were talking internally about ho to improve cli partity
13:04:00 sean-k-mooney efried: we were thinking a phased approch make the most sence. first make osc have partity with nova cli by just delegating to it internall. then port osc to openstack sdk once that is done
13:04:28 sean-k-mooney so form a user point of view we get to cli parity first then we can get teh sdk to partity and swap over to useing it
13:05:10 sean-k-mooney but there are still big gaps in that plan. e.g. should clint only things like nova host-evac be ported and how to deal with micorverions
13:07:57 ganso Hi folks. I am about to log a bug and would like to confirm if this is an intentional behavior or not: If an exception is raised by the driver during confirm_migration, the migration record is stuck in confirming state and the allocations are not removed. The instance is fine at the destination in this stage, but the source host has allocations that is not possible to clear without going to the database or invoking
13:07:57 ganso the Placement API via curl. After several migration attempts that fail in the same spot, the source node is filled with these allocations that prevent new instances from being created or instances migrated to this node.
13:08:39 sean-k-mooney efried: but ya in anycase i do hope we can make progress on that goal in train
13:16:18 efried ganso: What release are you running?
13:16:41 ganso efried: I tried pike, queens and stein
13:17:07 ganso efried: same problem in all of them ^
13:18:50 efried ganso: Well, that doesn't sound great. mriedem is probably your best bet to say whether this is a known issue. If he doesn't chime in here I think you can safely open a bug.
13:19:34 ganso efried: thanks! I will wait for mriedem's confirmation first.
13:20:58 mriedem looking, but i wouldn't be at all surprised
13:21:58 mriedem ack on the migration.status not being set to error because the @errors_out_migration_ctx decorator is not used here https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L3933
13:22:11 mriedem so you never get this far https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L4011
13:22:22 mriedem nor do the allocations get cleaned up https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L4017
13:22:32 mriedem so yes it's a big
13:22:38 mriedem setting the migration status to error is easy
13:22:48 mriedem knowing whether or not the allocations should be changed is harder
13:23:22 mriedem ganso: the instance status goes to ERROR right? https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L3993
13:23:32 ganso mriedem: yes the instance goes to error
13:23:33 mriedem which means you can't revert the resize after confirm fails
13:23:45 mriedem unless you manually reset the vm_state to 'resized' in the db
13:24:16 mriedem so at this point, the instance is on the dest, the db points the instance at the dest, and confirm fails on the dest and you're stuck in error state,
13:24:17 ganso mriedem: I didn't try that. I noticed I could only change the status to either error or active, so I changed the status to active and could use the instance fine after that, even migrate it again.
13:24:25 mriedem i would say we probably just consider the instance is stuck there now and cleanup the allocations
13:24:39 mriedem ganso: yeah reset status only changes to active/errror
13:24:40 mriedem *error
13:24:53 mriedem so you could have tried hard rebooting the instance to fix it also since you can do that with an error instance
13:25:06 mriedem but that's not going to cleanup the allocations as you found
13:25:17 mriedem ganso: anyway, it's definitely a bug
13:25:24 ganso mriedem: hmmm seems to me the migration_confirm runs on the source, and fails there. The libvirt driver only cleans up on that method's implementation
13:25:51 mriedem oh right, yeah confirm_resize runs on the source host
13:25:54 mriedem to cleanup the source host
13:26:07 efried no confirm_resize --force?
13:26:18 efried "I know this thing is copacetic on the destination, make it so"
13:26:29 mriedem probably another good reason to just cleanup the guest on the hypervisor as best effort but still cleanup the allocations
13:26:43 ganso mriedem: thanks! I will log the bug =)
13:26:47 mriedem efried: that would assume you know you need to force it before the instance gets stuck in error status
13:27:01 mriedem you can't confirm resize on an error instance
13:27:10 efried That's what the force would be
13:27:20 mriedem ew
13:27:21 efried "I done checked the instance, and it's good. Make it look good in the databases and stuff"
13:27:30 efried shrug, just spitballing.
13:27:38 mriedem you can fix it up with a hard reboot
13:27:41 mriedem on the dest
13:27:44 efried except the allocations
13:27:48 mriedem but that won't clean the allocations
13:28:05 mriedem right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way
13:28:12 ganso efried, mriedem: yea the main problem are the allocations.
13:28:27 efried does nova manage heal_allocations work on something like this?
13:28:28 mriedem these giant compute / conductor manager methods are so convoluted that it gets hard to deal with errors in isolation
13:28:31 mriedem and if they should be fatal or not
13:28:53 mriedem efried: no, that only creates missing allocations, doesn't cleanup ones that shouldn't exist
13:29:01 mriedem mnaser has a script for that though
13:29:02 mriedem sec
13:29:16 cdent I'm gonna get a tattoo of [t c6v1]
13:29:16 purplerbot <mriedem> right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way [2019-03-25 13:28:05.178818] [n c6v1]
13:29:22 efried What I'm getting at is, if there's some way to clean up the allocations other than curl'ing placement, I'm less concerned about this than otherwise
13:29:42 mriedem not built into nova no
13:29:52 mriedem there are various operator scripts floating around
13:31:59 mriedem https://bugs.launchpad.net/nova/+bug/1793569
13:32:00 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
13:32:30 mriedem there is a script in there from mnaser and one from larsks about cleaning up allocations in placement based on the nova db
13:33:01 mriedem i can't find the email, but last year i posted to the ML looking for someone to incorporate mnaser's into nova-manage
13:34:29 efried ganso: ^^ :P
13:36:06 ganso efried, mriedem: it would be great if such functionality existed in nova-manage, so the customer can run if they run into this situation.
13:36:30 tssurya mriedem: manser's script: http://paste.openstack.org/show/734146/
13:36:37 efried ganso: Above was me subtly suggesting that you might be interested in making that a reality :)
13:37:32 tssurya oh nvm, had a question but got the answer
13:39:29 ganso efried: on the other hand, it is a workaround for a bug. Once the bug is fixed there shouldn't be a need for the customer to run those scripts. I am not aware of which situations it is acceptable to have stale resources and perform cleanup of them.
13:39:31 mriedem i just re-posted to the ML
13:40:15 efried ganso: Yeah, if we never had bugs, we wouldn't need scripts like this :)
13:40:31 mriedem i'm sure the same problem exists on revert resize
13:40:35 mriedem and lots of other move ops
13:40:43 efried ganso: This won't be the last time we encounter a bug for which this workaround ... yeah, what mriedem said.
13:41:11 mriedem there is an assumption in these 200 LOC methods that we'll get to the things at the end because everything at the top and middle of the method will pass without error
13:42:19 mriedem even when we're trying to help https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L4703 i'm not sure how correct that is now
13:42:32 ganso efried: yup. What I am trying to say is: if such workaround was common and of easy access, perhaps those bugs wouldn't even be considered bugs and wouldn't be reported by customers. Not sure if that's a good route to go.
13:42:38 mriedem resize_instance changes the instance.host/node to the dest machine, and finish_resize then runs on the dest which could fail and if so, will revert allocations
13:42:52 efried mriedem: Restructure so the critical stuff is in a finally: block?
13:43:06 efried ganso: btw, did you figure out what the driver failure was?
13:43:42 efried mriedem: Are we supposed to be fast-approving the s,git://,https://, patches?
13:43:52 ganso efried: no, unfortunately my customer's logs don't go far back. I was able to find out where it fails because of the migration record in 'confirming' state
13:43:54 mriedem efried: which ones?
13:44:05 efried mriedem: https://review.openstack.org/#/c/646682/
13:44:09 efried and brethren
13:45:11 mriedem i guess i haven't been paying attention to the opendev changes
13:46:20 mriedem approved
13:49:11 mriedem oh god it's on all branches too
13:49:50 mriedem ianw will be king of all stats
13:49:51 openstackgerrit Eric Fried proposed openstack/nova master: s,git://github.com/,https://git.openstack.org/, https://review.openstack.org/647472

Earlier   Later