| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-17 | |||
| 17:11:10 | sean-k-mooney | is there a way to search for bugs you commented on? | |
| 17:12:01 | sean-k-mooney | oh there is | |
| 17:12:21 | melwitt | here's the IRC convo about it from the same day we commented on the oslo.messaging review http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-06-06.log.html#t2019-06-06T22:47:09 | |
| 17:13:04 | sean-k-mooney | oh thats clever i would not have thought of that. i rememerned it was mnaser that hit it | |
| 17:15:13 | melwitt | rest of the convo is here and I don't see any nova lp bug mentioned other than an old one that got no additional info on it | |
| 17:15:15 | melwitt | http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-06-07.log.html | |
| 17:15:19 | sean-k-mooney | ok so there wasnt a specifc nova bug but there were 3 related bugs | |
| 17:15:23 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1794706 | |
| 17:15:23 | openstack | Launchpad bug 1794706 in OpenStack Compute (nova) "The instance left stuck when oslo.messaging raised MessageDeliveryFailure exception" [Undecided,Expired] | |
| 17:15:27 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1437955 | |
| 17:15:27 | openstack | Launchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 17:15:31 | melwitt | yeah | |
| 17:15:32 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1661510 | |
| 17:15:32 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 17:16:12 | sean-k-mooney | eandersson: so i think you were hitting the same issue as mnaser | |
| 17:17:02 | sean-k-mooney | melwitt: thanks :) | |
| 17:17:37 | melwitt | :) | |
| 17:18:17 | sean-k-mooney | shoudl i un expire https://bugs.launchpad.net/nova/+bug/1794706 by the way | |
| 17:18:17 | openstack | Launchpad bug 1794706 in OpenStack Compute (nova) "The instance left stuck when oslo.messaging raised MessageDeliveryFailure exception" [Undecided,Expired] | |
| 17:18:56 | sean-k-mooney | i guess there is no point we dont have the info we need | |
| 17:22:09 | eandersson | Yea that sounds like the exact issue | |
| 17:23:30 | eandersson | btw another major issue with this is that unless you know what you are doing it's very difficult to identify the bad compute | |
| 17:23:42 | melwitt | sean-k-mooney: yeah, I'd say don't bother, it's from 2017 and not enough info to move forward, unless I've missed something | |
| 17:23:56 | melwitt | oh, nvm 2018. I can't read | |
| 17:27:23 | melwitt | sean-k-mooney: if you wanted to unexpire and add more info to it based on eandersson experience (if it's the same thing) then I think that makes sense | |
| 17:27:27 | sean-k-mooney | eandersson: yes. would you feel comfortable writing this up as a bug | |
| 17:27:51 | sean-k-mooney | melwitt: its not exactly the same thing as the old nova bug | |
| 17:27:57 | sean-k-mooney | but i think its the same oslo bug | |
| 17:28:01 | melwitt | ack | |
| 17:28:06 | sean-k-mooney | which is what mnaser was hitting | |
| 17:28:11 | eandersson | I am pretty sure what happened here was that we had a rabbitmq network partition weeks ago | |
| 17:28:20 | eandersson | and that partition somehow damaged the queue | |
| 17:28:32 | sean-k-mooney | yep | |
| 17:28:43 | eandersson | but only from openstack perspective, because I could consume the queue using the ui etc. | |
| 17:28:43 | sean-k-mooney | basically if the que gets deleted | |
| 17:28:52 | sean-k-mooney | and you dont restart the nova-compute agent | |
| 17:28:57 | sean-k-mooney | then it wont recreate it | |
| 17:28:58 | eandersson | and conductor | |
| 17:28:58 | eandersson | yea | |
| 17:29:10 | eandersson | the weird thing is that it existed in a "healthy" state with the bindings etc | |
| 17:29:20 | melwitt | and if we leverage the earlier mentioned oslo.messaging change, we can make it recover in nova? | |
| 17:29:51 | eandersson | If I can find another bad queue I can test | |
| 17:29:57 | sean-k-mooney | i think so | |
| 17:30:05 | melwitt | kewl | |
| 17:30:40 | sean-k-mooney | i think either the condocot or compute node would get and excpetion if tehy tried to do a topic send to the queue and that would allwo use to fix it by recreating the queues | |
| 17:30:50 | openstackgerrit | Merged openstack/nova stable/queens: lxc: make use of filter python3 compatible https://review.opendev.org/676500 | |
| 17:31:02 | openstackgerrit | Merged openstack/nova master: Make sure tox install requirements.txt with upper-constraints https://review.opendev.org/689152 | |
| 17:31:24 | sean-k-mooney | melwitt: i would need to fully re read the oslo feature but they had a recovery mechanium in mind | |
| 17:32:22 | melwitt | sean-k-mooney: yeah. if I'm remembering right, they were saying just setting an option called 'mandatory' would make it do the things by itself | |
| 17:33:01 | melwitt | and didn't want to change the default to mandatory=1/true, so they added this config option passing mechanism. and we just needed to pass 'mandatory' to it in nova | |
| 17:33:11 | melwitt | something like that | |
| 17:33:21 | sean-k-mooney | ya something like that | |
| 17:33:35 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1437955 | |
| 17:33:35 | openstack | Launchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 17:33:47 | sean-k-mooney | this kind fo explains it but i think there was more in teh review | |
| 17:36:57 | sean-k-mooney | while this is only the 2nd time someone has reported this in the last 4 months it is really hard to traige/identify this issue so we proably shoudl try to fix this. i guess we soudl follow up with the oslo folks and see if it ready to use | |
| 17:37:51 | sean-k-mooney | im surpiesed more people have not hit this to be honest | |
| 17:38:15 | sean-k-mooney | mayeb they have and just rebooted the node | |
| 17:41:54 | melwitt | yeah, should just be a matter of bumping our minimum oslo.messaging version and finding where to set the mandatory flag | |
| 17:42:15 | melwitt | that doesn't help for stable branches obvs. not sure if we have any options there | |
| 17:42:47 | melwitt | I can give it a go unless someone else wants to | |
| 17:43:54 | melwitt | looks like the change was released in version 9.8.0 | |
| 17:46:00 | melwitt | thinking more, there's no way we could have this for stable branches | |
| 17:46:36 | melwitt | and also, how to reproduce and prove the mandatory flag fixes anything | |
| 18:05:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'os-consoles' API https://review.opendev.org/687907 | |
| 18:05:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova-console' service, 'os-consoles' API https://review.opendev.org/687908 | |
| 18:05:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova-xvpvncproxy' https://review.opendev.org/687909 | |
| 18:12:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Change order of two classes https://review.opendev.org/689178 | |
| 18:12:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Rework '_delete_server' https://review.opendev.org/689179 | |
| 18:12:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Make '_wait_for_state_change' behave consistently https://review.opendev.org/689180 | |
| 18:12:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Unify '_wait_until_deleted' implementations https://review.opendev.org/689181 | |
| 18:12:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Make 'ServerTestBase' subclass 'InstanceHelperMixin' https://review.opendev.org/689182 | |
| 18:17:51 | mnaser | eandersson: i have hit that exact issue many times :( | |
| 18:18:01 | mnaser | it really has to do with rabbitmq being unhappy when the queues come back | |
| 18:18:08 | mnaser | curious -- what version of it are you running? | |
| 18:31:28 | eandersson | 3.7.5 | |
| 18:37:56 | mordred | stephenfin: did y'all sort out the requirements thing from earlier/ | |
| 18:37:57 | mordred | ? | |
| 18:38:16 | dansmith | mordred: with the networx thing or whatever? | |
| 18:38:40 | dansmith | mordred: https://review.opendev.org/#/c/689152/ | |
| 18:40:38 | mordred | dansmith: cool! | |
| 19:07:21 | eandersson | sean-k-mooney, mnaser > Message not delivered: NO_ROUTE (312) to queue 'compute.<compute_name>' from exchange 'nova' | |
| 19:07:24 | eandersson | This is the actual issue | |
| 19:07:37 | eandersson | It's an super odd bug because the queue is fine, and the compute can consume from it | |
| 19:07:46 | eandersson | but the binding (that is 100% there) simply does not work | |
| 19:09:07 | eandersson | if confirm-deliveries isn't enabled, which might be the problem, amqp isn't going to raise an error | |
| 19:09:49 | eandersson | mandatory flag isn't enough afaik | |
| 19:10:05 | eandersson | mandatory + confirm_deliveries is what I used | |
| 19:12:52 | eandersson | http://paste.openstack.org/show/784547/ | |
| 19:13:55 | openstackgerrit | Merged openstack/nova master: Add functional recreate test for bug 1848343 https://review.opendev.org/688980 | |
| 19:13:55 | openstack | bug 1848343 in OpenStack Compute (nova) "Reverting migration-based allocations leaks allocations if the server is deleted" [Medium,In progress] https://launchpad.net/bugs/1848343 - Assigned to Matt Riedemann (mriedem) | |
| 19:14:06 | eandersson | * http://paste.openstack.org/show/784548/ | |
| 19:14:13 | openstackgerrit | Merged openstack/nova master: Add live migration recreate test for bug 1848343 https://review.opendev.org/688994 | |
| 19:17:21 | openstackgerrit | Merged openstack/nova master: Add compute side revert allocation test for bug 1848343 https://review.opendev.org/689013 | |
| 19:17:21 | openstack | bug 1848343 in OpenStack Compute (nova) "Reverting migration-based allocations leaks allocations if the server is deleted" [Medium,In progress] https://launchpad.net/bugs/1848343 - Assigned to Matt Riedemann (mriedem) | |
| 19:44:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete source allocations in move_allocations if target no longer exists https://review.opendev.org/689049 | |
| 20:06:23 | mriedem | easy couple of patches to fix an upgrade issue since stein https://review.opendev.org/#/q/topic:bug/1824435+(status:open+OR+status:merged) | |
| 20:51:01 | eandersson | mnaser, https://github.com/rabbitmq/rabbitmq-server/pull/1884#issuecomment-464277810 | |
| 20:51:04 | eandersson | I think this is the issue | |
| 20:52:40 | eandersson | https://github.com/rabbitmq/rabbitmq-server/pull/1879 | |