| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-25 | |||
| 13:27:44 | efried | except the allocations | |
| 13:27:48 | mriedem | but that won't clean the allocations | |
| 13:28:05 | mriedem | right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way | |
| 13:28:12 | ganso | efried, mriedem: yea the main problem are the allocations. | |
| 13:28:27 | efried | does nova manage heal_allocations work on something like this? | |
| 13:28:28 | mriedem | these giant compute / conductor manager methods are so convoluted that it gets hard to deal with errors in isolation | |
| 13:28:31 | mriedem | and if they should be fatal or not | |
| 13:28:53 | mriedem | efried: no, that only creates missing allocations, doesn't cleanup ones that shouldn't exist | |
| 13:29:01 | mriedem | mnaser has a script for that though | |
| 13:29:02 | mriedem | sec | |
| 13:29:16 | cdent | I'm gonna get a tattoo of [t c6v1] | |
| 13:29:16 | purplerbot | <mriedem> right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way [2019-03-25 13:28:05.178818] [n c6v1] | |
| 13:29:22 | efried | What I'm getting at is, if there's some way to clean up the allocations other than curl'ing placement, I'm less concerned about this than otherwise | |
| 13:29:42 | mriedem | not built into nova no | |
| 13:29:52 | mriedem | there are various operator scripts floating around | |
| 13:31:59 | mriedem | https://bugs.launchpad.net/nova/+bug/1793569 | |
| 13:32:00 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 13:32:30 | mriedem | there is a script in there from mnaser and one from larsks about cleaning up allocations in placement based on the nova db | |
| 13:33:01 | mriedem | i can't find the email, but last year i posted to the ML looking for someone to incorporate mnaser's into nova-manage | |
| 13:34:29 | efried | ganso: ^^ :P | |
| 13:36:06 | ganso | efried, mriedem: it would be great if such functionality existed in nova-manage, so the customer can run if they run into this situation. | |
| 13:36:30 | tssurya | mriedem: manser's script: http://paste.openstack.org/show/734146/ | |
| 13:36:37 | efried | ganso: Above was me subtly suggesting that you might be interested in making that a reality :) | |
| 13:37:32 | tssurya | oh nvm, had a question but got the answer | |
| 13:39:29 | ganso | efried: on the other hand, it is a workaround for a bug. Once the bug is fixed there shouldn't be a need for the customer to run those scripts. I am not aware of which situations it is acceptable to have stale resources and perform cleanup of them. | |
| 13:39:31 | mriedem | i just re-posted to the ML | |
| 13:40:15 | efried | ganso: Yeah, if we never had bugs, we wouldn't need scripts like this :) | |
| 13:40:31 | mriedem | i'm sure the same problem exists on revert resize | |
| 13:40:35 | mriedem | and lots of other move ops | |
| 13:40:43 | efried | ganso: This won't be the last time we encounter a bug for which this workaround ... yeah, what mriedem said. | |
| 13:41:11 | mriedem | there is an assumption in these 200 LOC methods that we'll get to the things at the end because everything at the top and middle of the method will pass without error | |
| 13:42:19 | mriedem | even when we're trying to help https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L4703 i'm not sure how correct that is now | |
| 13:42:32 | ganso | efried: yup. What I am trying to say is: if such workaround was common and of easy access, perhaps those bugs wouldn't even be considered bugs and wouldn't be reported by customers. Not sure if that's a good route to go. | |
| 13:42:38 | mriedem | resize_instance changes the instance.host/node to the dest machine, and finish_resize then runs on the dest which could fail and if so, will revert allocations | |
| 13:42:52 | efried | mriedem: Restructure so the critical stuff is in a finally: block? | |
| 13:43:06 | efried | ganso: btw, did you figure out what the driver failure was? | |
| 13:43:42 | efried | mriedem: Are we supposed to be fast-approving the s,git://,https://, patches? | |
| 13:43:52 | ganso | efried: no, unfortunately my customer's logs don't go far back. I was able to find out where it fails because of the migration record in 'confirming' state | |
| 13:43:54 | mriedem | efried: which ones? | |
| 13:44:05 | efried | mriedem: https://review.openstack.org/#/c/646682/ | |
| 13:44:09 | efried | and brethren | |
| 13:45:11 | mriedem | i guess i haven't been paying attention to the opendev changes | |
| 13:46:20 | mriedem | approved | |
| 13:49:11 | mriedem | oh god it's on all branches too | |
| 13:49:50 | mriedem | ianw will be king of all stats | |
| 13:49:51 | openstackgerrit | Eric Fried proposed openstack/nova master: s,git://github.com/,https://git.openstack.org/, https://review.openstack.org/647472 | |
| 13:52:08 | mriedem | sean-k-mooney: is the kuryr-kubernetes-tempest-daemon-octavia failure in https://review.openstack.org/#/c/646868/ on stable/rocky a known issue? | |
| 13:55:14 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Refactor CONTRIBUTING.rst https://review.openstack.org/640970 | |
| 13:55:28 | bauzas | stephenfin: I'd appreciate some doc-nits ;) | |
| 13:55:34 | mriedem | ganso: let me know when you have a bug posted and i can at least fix the migration.status = 'error' thing quickly | |
| 13:55:40 | bauzas | stephenfin: oops, https://review.openstack.org/640970 | |
| 13:55:48 | mriedem | bauzas: unrelated, but for you https://review.openstack.org/#/c/646006/ | |
| 13:56:10 | bauzas | mriedem: ack, will look | |
| 13:56:22 | ganso | mriedem: https://bugs.launchpad.net/nova/+bug/1821594 | |
| 13:56:23 | openstack | Launchpad bug 1821594 in OpenStack Compute (nova) "Error in confirm_migration leaves stale allocations and 'confirming' migration state" [Undecided,New] | |
| 13:56:35 | ganso | mriedem: thanks! | |
| 14:05:52 | efried | thanks ganso | |
| 14:07:38 | mriedem | i need to email claudiub about https://review.openstack.org/#/c/543971/ | |
| 14:14:17 | sean-k-mooney | mriedem: sorry was on 1:1 will check thatreview now | |
| 14:16:33 | sean-k-mooney | mriedem: we made kuryr-kubernetes-tempest-daemon-octavia non-voting on master and kuryr removed it a few weeks ago so im not surpised that it failed | |
| 14:20:04 | sean-k-mooney | mriedem: it looked like it failed becasue the devstack-plugin-container tried to install linux kernel package that are not found on the repos anymore | |
| 14:20:34 | sean-k-mooney | it looks like it was installin kernel 4.4.0 packages which would correspond to ubuntu 16.04 | |
| 14:20:52 | sean-k-mooney | if this ran on ubunut bionic then it would explain why it was not found | |
| 14:21:37 | sean-k-mooney | actully its a ubuntu-xenial node i would guess the package is just not in te repo anymore | |
| 14:28:14 | openstackgerrit | Merged openstack/nova-specs master: Fix warnings in the document generation https://review.openstack.org/631150 | |
| 14:33:59 | sean-k-mooney | mriedem: ill propose the backport of making that non voting to the sable branches | |
| 14:34:29 | openstackgerrit | Merged openstack/nova master: Update contributor guide for Train https://review.openstack.org/645581 | |
| 14:34:33 | mriedem | thanks | |
| 14:43:28 | mnaser | btw, I raised this on the mailing list .. I dunno how release-critical this can be considered considering it's been a long time bug -- http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004206.html | |
| 15:09:10 | openstackgerrit | sean mooney proposed openstack/os-vif stable/rocky: make kuryr-kubernetes-tempest-daemon-octavia non voting https://review.openstack.org/647500 | |
| 15:10:02 | sean-k-mooney | mriedem: that will do it for rocky ^ but i need to check project-conifg for older branches | |
| 15:10:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add falvor to requested_resources in RequestSpec https://review.openstack.org/647396 | |
| 15:11:29 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add flavor to requested_resources in RequestSpec https://review.openstack.org/647396 | |
| 15:13:12 | sean-k-mooney | actully no never mind | |
| 15:13:39 | sean-k-mooney | we have zuul config in repo for older release se just do not run that job | |
| 15:16:11 | openstackgerrit | Matt Riedemann proposed openstack/os-vif stable/rocky: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646868 | |
| 15:22:55 | openstackgerrit | Merged openstack/python-novaclient stable/stein: Replace openstack.org git:// URLs with https:// https://review.openstack.org/647075 | |
| 15:32:14 | openstackgerrit | Merged openstack/os-vif master: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646867 | |
| 15:37:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Overwrite RequestSpec.ignore_hosts during evacuate https://review.openstack.org/647512 | |
| 15:37:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/647513 | |
| 15:37:21 | mriedem | bauzas: cfriesen: dansmith: ^ fixes that super latent bug | |
| 15:39:30 | cfriesen | mriedem: nice. will take a look. | |
| 15:39:51 | mriedem | only took 2+ years | |
| 15:41:43 | mriedem | i can't wait until my next "don't persist field x in request spec" adventure | |
| 15:45:12 | mriedem | i guess i need to squash the 2nd and 3rd changes anyway...doing that | |
| 15:48:58 | bauzas | mriedem: nice | |
| 15:49:16 | bauzas | mriedem: once I'm done with some internal meeting + the vgpu docs, you're next in the queue | |
| 15:52:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug 1669054 https://review.openstack.org/646006 | |
| 15:52:26 | openstack | bug 1669054 in OpenStack Compute (nova) "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem) | |
| 15:52:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/647512 | |
| 15:55:38 | cfriesen | found a grammar bug in the commit message for https://review.openstack.org/#/c/647512 | |
| 15:55:41 | cfriesen | mriedem: ^ | |
| 15:55:51 | cfriesen | see review | |
| 15:56:34 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Remove expiremental note in the VGPU docs https://review.openstack.org/647518 | |
| 15:56:35 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/647519 | |
| 15:56:51 | bauzas | mriedem: you could be interested in ^ | |
| 15:59:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/647512 | |
| 15:59:29 | mriedem | cfriesen: done | |
| 15:59:43 | dansmith | wow a grammar nit in a mriedem patch | |
| 16:02:28 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/647519 | |