| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-25 | |||
| 13:22:38 | mriedem | setting the migration status to error is easy | |
| 13:22:48 | mriedem | knowing whether or not the allocations should be changed is harder | |
| 13:23:22 | mriedem | ganso: the instance status goes to ERROR right? https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L3993 | |
| 13:23:32 | ganso | mriedem: yes the instance goes to error | |
| 13:23:33 | mriedem | which means you can't revert the resize after confirm fails | |
| 13:23:45 | mriedem | unless you manually reset the vm_state to 'resized' in the db | |
| 13:24:16 | mriedem | so at this point, the instance is on the dest, the db points the instance at the dest, and confirm fails on the dest and you're stuck in error state, | |
| 13:24:17 | ganso | mriedem: I didn't try that. I noticed I could only change the status to either error or active, so I changed the status to active and could use the instance fine after that, even migrate it again. | |
| 13:24:25 | mriedem | i would say we probably just consider the instance is stuck there now and cleanup the allocations | |
| 13:24:39 | mriedem | ganso: yeah reset status only changes to active/errror | |
| 13:24:40 | mriedem | *error | |
| 13:24:53 | mriedem | so you could have tried hard rebooting the instance to fix it also since you can do that with an error instance | |
| 13:25:06 | mriedem | but that's not going to cleanup the allocations as you found | |
| 13:25:17 | mriedem | ganso: anyway, it's definitely a bug | |
| 13:25:24 | ganso | mriedem: hmmm seems to me the migration_confirm runs on the source, and fails there. The libvirt driver only cleans up on that method's implementation | |
| 13:25:51 | mriedem | oh right, yeah confirm_resize runs on the source host | |
| 13:25:54 | mriedem | to cleanup the source host | |
| 13:26:07 | efried | no confirm_resize --force? | |
| 13:26:18 | efried | "I know this thing is copacetic on the destination, make it so" | |
| 13:26:29 | mriedem | probably another good reason to just cleanup the guest on the hypervisor as best effort but still cleanup the allocations | |
| 13:26:43 | ganso | mriedem: thanks! I will log the bug =) | |
| 13:26:47 | mriedem | efried: that would assume you know you need to force it before the instance gets stuck in error status | |
| 13:27:01 | mriedem | you can't confirm resize on an error instance | |
| 13:27:10 | efried | That's what the force would be | |
| 13:27:20 | mriedem | ew | |
| 13:27:21 | efried | "I done checked the instance, and it's good. Make it look good in the databases and stuff" | |
| 13:27:30 | efried | shrug, just spitballing. | |
| 13:27:38 | mriedem | you can fix it up with a hard reboot | |
| 13:27:41 | mriedem | on the dest | |
| 13:27:44 | efried | except the allocations | |
| 13:27:48 | mriedem | but that won't clean the allocations | |
| 13:28:05 | mriedem | right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way | |
| 13:28:12 | ganso | efried, mriedem: yea the main problem are the allocations. | |
| 13:28:27 | efried | does nova manage heal_allocations work on something like this? | |
| 13:28:28 | mriedem | these giant compute / conductor manager methods are so convoluted that it gets hard to deal with errors in isolation | |
| 13:28:31 | mriedem | and if they should be fatal or not | |
| 13:28:53 | mriedem | efried: no, that only creates missing allocations, doesn't cleanup ones that shouldn't exist | |
| 13:29:01 | mriedem | mnaser has a script for that though | |
| 13:29:02 | mriedem | sec | |
| 13:29:16 | purplerbot | <mriedem> right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way [2019-03-25 13:28:05.178818] [n c6v1] | |
| 13:29:16 | cdent | I'm gonna get a tattoo of [t c6v1] | |
| 13:29:22 | efried | What I'm getting at is, if there's some way to clean up the allocations other than curl'ing placement, I'm less concerned about this than otherwise | |
| 13:29:42 | mriedem | not built into nova no | |
| 13:29:52 | mriedem | there are various operator scripts floating around | |
| 13:31:59 | mriedem | https://bugs.launchpad.net/nova/+bug/1793569 | |
| 13:32:00 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 13:32:30 | mriedem | there is a script in there from mnaser and one from larsks about cleaning up allocations in placement based on the nova db | |
| 13:33:01 | mriedem | i can't find the email, but last year i posted to the ML looking for someone to incorporate mnaser's into nova-manage | |
| 13:34:29 | efried | ganso: ^^ :P | |
| 13:36:06 | ganso | efried, mriedem: it would be great if such functionality existed in nova-manage, so the customer can run if they run into this situation. | |
| 13:36:30 | tssurya | mriedem: manser's script: http://paste.openstack.org/show/734146/ | |
| 13:36:37 | efried | ganso: Above was me subtly suggesting that you might be interested in making that a reality :) | |
| 13:37:32 | tssurya | oh nvm, had a question but got the answer | |
| 13:39:29 | ganso | efried: on the other hand, it is a workaround for a bug. Once the bug is fixed there shouldn't be a need for the customer to run those scripts. I am not aware of which situations it is acceptable to have stale resources and perform cleanup of them. | |
| 13:39:31 | mriedem | i just re-posted to the ML | |
| 13:40:15 | efried | ganso: Yeah, if we never had bugs, we wouldn't need scripts like this :) | |
| 13:40:31 | mriedem | i'm sure the same problem exists on revert resize | |
| 13:40:35 | mriedem | and lots of other move ops | |
| 13:40:43 | efried | ganso: This won't be the last time we encounter a bug for which this workaround ... yeah, what mriedem said. | |
| 13:41:11 | mriedem | there is an assumption in these 200 LOC methods that we'll get to the things at the end because everything at the top and middle of the method will pass without error | |
| 13:42:19 | mriedem | even when we're trying to help https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L4703 i'm not sure how correct that is now | |
| 13:42:32 | ganso | efried: yup. What I am trying to say is: if such workaround was common and of easy access, perhaps those bugs wouldn't even be considered bugs and wouldn't be reported by customers. Not sure if that's a good route to go. | |
| 13:42:38 | mriedem | resize_instance changes the instance.host/node to the dest machine, and finish_resize then runs on the dest which could fail and if so, will revert allocations | |
| 13:42:52 | efried | mriedem: Restructure so the critical stuff is in a finally: block? | |
| 13:43:06 | efried | ganso: btw, did you figure out what the driver failure was? | |
| 13:43:42 | efried | mriedem: Are we supposed to be fast-approving the s,git://,https://, patches? | |
| 13:43:52 | ganso | efried: no, unfortunately my customer's logs don't go far back. I was able to find out where it fails because of the migration record in 'confirming' state | |
| 13:43:54 | mriedem | efried: which ones? | |
| 13:44:05 | efried | mriedem: https://review.openstack.org/#/c/646682/ | |
| 13:44:09 | efried | and brethren | |
| 13:45:11 | mriedem | i guess i haven't been paying attention to the opendev changes | |
| 13:46:20 | mriedem | approved | |
| 13:49:11 | mriedem | oh god it's on all branches too | |
| 13:49:50 | mriedem | ianw will be king of all stats | |
| 13:49:51 | openstackgerrit | Eric Fried proposed openstack/nova master: s,git://github.com/,https://git.openstack.org/, https://review.openstack.org/647472 | |
| 13:52:08 | mriedem | sean-k-mooney: is the kuryr-kubernetes-tempest-daemon-octavia failure in https://review.openstack.org/#/c/646868/ on stable/rocky a known issue? | |
| 13:55:14 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Refactor CONTRIBUTING.rst https://review.openstack.org/640970 | |
| 13:55:28 | bauzas | stephenfin: I'd appreciate some doc-nits ;) | |
| 13:55:34 | mriedem | ganso: let me know when you have a bug posted and i can at least fix the migration.status = 'error' thing quickly | |
| 13:55:40 | bauzas | stephenfin: oops, https://review.openstack.org/640970 | |
| 13:55:48 | mriedem | bauzas: unrelated, but for you https://review.openstack.org/#/c/646006/ | |
| 13:56:10 | bauzas | mriedem: ack, will look | |
| 13:56:22 | ganso | mriedem: https://bugs.launchpad.net/nova/+bug/1821594 | |
| 13:56:23 | openstack | Launchpad bug 1821594 in OpenStack Compute (nova) "Error in confirm_migration leaves stale allocations and 'confirming' migration state" [Undecided,New] | |
| 13:56:35 | ganso | mriedem: thanks! | |
| 14:05:52 | efried | thanks ganso | |
| 14:07:38 | mriedem | i need to email claudiub about https://review.openstack.org/#/c/543971/ | |
| 14:14:17 | sean-k-mooney | mriedem: sorry was on 1:1 will check thatreview now | |
| 14:16:33 | sean-k-mooney | mriedem: we made kuryr-kubernetes-tempest-daemon-octavia non-voting on master and kuryr removed it a few weeks ago so im not surpised that it failed | |
| 14:20:04 | sean-k-mooney | mriedem: it looked like it failed becasue the devstack-plugin-container tried to install linux kernel package that are not found on the repos anymore | |
| 14:20:34 | sean-k-mooney | it looks like it was installin kernel 4.4.0 packages which would correspond to ubuntu 16.04 | |
| 14:20:52 | sean-k-mooney | if this ran on ubunut bionic then it would explain why it was not found | |
| 14:21:37 | sean-k-mooney | actully its a ubuntu-xenial node i would guess the package is just not in te repo anymore | |
| 14:28:14 | openstackgerrit | Merged openstack/nova-specs master: Fix warnings in the document generation https://review.openstack.org/631150 | |
| 14:33:59 | sean-k-mooney | mriedem: ill propose the backport of making that non voting to the sable branches | |
| 14:34:29 | openstackgerrit | Merged openstack/nova master: Update contributor guide for Train https://review.openstack.org/645581 | |
| 14:34:33 | mriedem | thanks | |
| 14:43:28 | mnaser | btw, I raised this on the mailing list .. I dunno how release-critical this can be considered considering it's been a long time bug -- http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004206.html | |
| 15:09:10 | openstackgerrit | sean mooney proposed openstack/os-vif stable/rocky: make kuryr-kubernetes-tempest-daemon-octavia non voting https://review.openstack.org/647500 | |
| 15:10:02 | sean-k-mooney | mriedem: that will do it for rocky ^ but i need to check project-conifg for older branches | |