| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-17 | |||
| 17:28:01 | melwitt | ack | |
| 17:28:06 | sean-k-mooney | which is what mnaser was hitting | |
| 17:28:11 | eandersson | I am pretty sure what happened here was that we had a rabbitmq network partition weeks ago | |
| 17:28:20 | eandersson | and that partition somehow damaged the queue | |
| 17:28:32 | sean-k-mooney | yep | |
| 17:28:43 | eandersson | but only from openstack perspective, because I could consume the queue using the ui etc. | |
| 17:28:43 | sean-k-mooney | basically if the que gets deleted | |
| 17:28:52 | sean-k-mooney | and you dont restart the nova-compute agent | |
| 17:28:57 | sean-k-mooney | then it wont recreate it | |
| 17:28:58 | eandersson | and conductor | |
| 17:28:58 | eandersson | yea | |
| 17:29:10 | eandersson | the weird thing is that it existed in a "healthy" state with the bindings etc | |
| 17:29:20 | melwitt | and if we leverage the earlier mentioned oslo.messaging change, we can make it recover in nova? | |
| 17:29:51 | eandersson | If I can find another bad queue I can test | |
| 17:29:57 | sean-k-mooney | i think so | |
| 17:30:05 | melwitt | kewl | |
| 17:30:40 | sean-k-mooney | i think either the condocot or compute node would get and excpetion if tehy tried to do a topic send to the queue and that would allwo use to fix it by recreating the queues | |
| 17:30:50 | openstackgerrit | Merged openstack/nova stable/queens: lxc: make use of filter python3 compatible https://review.opendev.org/676500 | |
| 17:31:02 | openstackgerrit | Merged openstack/nova master: Make sure tox install requirements.txt with upper-constraints https://review.opendev.org/689152 | |
| 17:31:24 | sean-k-mooney | melwitt: i would need to fully re read the oslo feature but they had a recovery mechanium in mind | |
| 17:32:22 | melwitt | sean-k-mooney: yeah. if I'm remembering right, they were saying just setting an option called 'mandatory' would make it do the things by itself | |
| 17:33:01 | melwitt | and didn't want to change the default to mandatory=1/true, so they added this config option passing mechanism. and we just needed to pass 'mandatory' to it in nova | |
| 17:33:11 | melwitt | something like that | |
| 17:33:21 | sean-k-mooney | ya something like that | |
| 17:33:35 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1437955 | |
| 17:33:35 | openstack | Launchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 17:33:47 | sean-k-mooney | this kind fo explains it but i think there was more in teh review | |
| 17:36:57 | sean-k-mooney | while this is only the 2nd time someone has reported this in the last 4 months it is really hard to traige/identify this issue so we proably shoudl try to fix this. i guess we soudl follow up with the oslo folks and see if it ready to use | |
| 17:37:51 | sean-k-mooney | im surpiesed more people have not hit this to be honest | |
| 17:38:15 | sean-k-mooney | mayeb they have and just rebooted the node | |
| 17:41:54 | melwitt | yeah, should just be a matter of bumping our minimum oslo.messaging version and finding where to set the mandatory flag | |
| 17:42:15 | melwitt | that doesn't help for stable branches obvs. not sure if we have any options there | |
| 17:42:47 | melwitt | I can give it a go unless someone else wants to | |
| 17:43:54 | melwitt | looks like the change was released in version 9.8.0 | |
| 17:46:00 | melwitt | thinking more, there's no way we could have this for stable branches | |
| 17:46:36 | melwitt | and also, how to reproduce and prove the mandatory flag fixes anything | |
| 18:05:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'os-consoles' API https://review.opendev.org/687907 | |
| 18:05:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova-console' service, 'os-consoles' API https://review.opendev.org/687908 | |
| 18:05:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova-xvpvncproxy' https://review.opendev.org/687909 | |
| 18:12:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Change order of two classes https://review.opendev.org/689178 | |
| 18:12:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Rework '_delete_server' https://review.opendev.org/689179 | |
| 18:12:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Make '_wait_for_state_change' behave consistently https://review.opendev.org/689180 | |
| 18:12:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Unify '_wait_until_deleted' implementations https://review.opendev.org/689181 | |
| 18:12:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Make 'ServerTestBase' subclass 'InstanceHelperMixin' https://review.opendev.org/689182 | |
| 18:17:51 | mnaser | eandersson: i have hit that exact issue many times :( | |
| 18:18:01 | mnaser | it really has to do with rabbitmq being unhappy when the queues come back | |
| 18:18:08 | mnaser | curious -- what version of it are you running? | |
| 18:31:28 | eandersson | 3.7.5 | |
| 18:37:56 | mordred | stephenfin: did y'all sort out the requirements thing from earlier/ | |
| 18:37:57 | mordred | ? | |
| 18:38:16 | dansmith | mordred: with the networx thing or whatever? | |
| 18:38:40 | dansmith | mordred: https://review.opendev.org/#/c/689152/ | |
| 18:40:38 | mordred | dansmith: cool! | |
| 19:07:21 | eandersson | sean-k-mooney, mnaser > Message not delivered: NO_ROUTE (312) to queue 'compute.<compute_name>' from exchange 'nova' | |
| 19:07:24 | eandersson | This is the actual issue | |
| 19:07:37 | eandersson | It's an super odd bug because the queue is fine, and the compute can consume from it | |
| 19:07:46 | eandersson | but the binding (that is 100% there) simply does not work | |
| 19:09:07 | eandersson | if confirm-deliveries isn't enabled, which might be the problem, amqp isn't going to raise an error | |
| 19:09:49 | eandersson | mandatory flag isn't enough afaik | |
| 19:10:05 | eandersson | mandatory + confirm_deliveries is what I used | |
| 19:12:52 | eandersson | http://paste.openstack.org/show/784547/ | |
| 19:13:55 | openstackgerrit | Merged openstack/nova master: Add functional recreate test for bug 1848343 https://review.opendev.org/688980 | |
| 19:13:55 | openstack | bug 1848343 in OpenStack Compute (nova) "Reverting migration-based allocations leaks allocations if the server is deleted" [Medium,In progress] https://launchpad.net/bugs/1848343 - Assigned to Matt Riedemann (mriedem) | |
| 19:14:06 | eandersson | * http://paste.openstack.org/show/784548/ | |
| 19:14:13 | openstackgerrit | Merged openstack/nova master: Add live migration recreate test for bug 1848343 https://review.opendev.org/688994 | |
| 19:17:21 | openstackgerrit | Merged openstack/nova master: Add compute side revert allocation test for bug 1848343 https://review.opendev.org/689013 | |
| 19:17:21 | openstack | bug 1848343 in OpenStack Compute (nova) "Reverting migration-based allocations leaks allocations if the server is deleted" [Medium,In progress] https://launchpad.net/bugs/1848343 - Assigned to Matt Riedemann (mriedem) | |
| 19:44:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete source allocations in move_allocations if target no longer exists https://review.opendev.org/689049 | |
| 20:06:23 | mriedem | easy couple of patches to fix an upgrade issue since stein https://review.opendev.org/#/q/topic:bug/1824435+(status:open+OR+status:merged) | |
| 20:51:01 | eandersson | mnaser, https://github.com/rabbitmq/rabbitmq-server/pull/1884#issuecomment-464277810 | |
| 20:51:04 | eandersson | I think this is the issue | |
| 20:52:40 | eandersson | https://github.com/rabbitmq/rabbitmq-server/pull/1879 | |
| 20:59:57 | melwitt | if that's the case, then all that should be needed is upgrade rabbit to 3.7 and no need to consume the mandatory flag option in nova, iiuc | |
| 21:02:05 | melwitt | oh wait, according to the backscroll, need both the flag and the fix in 3.7 then sounds like | |
| 21:03:08 | melwitt | eandersson: can you correct me pls? ^ | |
| 21:03:54 | melwitt | just trying to understand whether we need to do anything in nova | |
| 21:46:12 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add new test base for libvirt functional tests https://review.opendev.org/689186 | |
| 21:46:12 | openstackgerrit | Matthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks https://review.opendev.org/663382 | |
| 21:46:13 | openstackgerrit | Matthew Booth proposed openstack/nova master: Fix incorrect instance state after build failure https://review.opendev.org/689278 | |
| 21:52:32 | eandersson | melwitt, so the queue can still get stuck in a bad state. It happens. | |
| 21:52:39 | eandersson | So having that extra layer of protection is still good. | |
| 21:53:23 | eandersson | (e.g. network partitions etc) | |
| 21:53:40 | melwitt | eandersson: sorry, what I mean is, what do we need to do in nova, if anything? pass the mandatory flag? pass the mandatory flag and the confirm_deliveries flag? or is nothing needed? | |
| 21:53:56 | eandersson | I think this probably has to be done on the oslo.messaging side | |
| 21:54:17 | melwitt | I wasn't sure whether your findings meant a change in the plan for | |
| 21:54:21 | melwitt | nova | |
| 21:54:23 | eandersson | but I still feel like a vm shouldn't be able to stay in a state like that indefinitely | |
| 21:55:24 | eandersson | maybe that could be changed to an rpc call instead? | |
| 21:55:51 | melwitt | no, it shouldn't. sorry, somehow what I'm saying is not coming across. you posted links about rabbit bugs earlier, so I wasn't sure if you were saying it's a rabbit bug and there's nothing we can do to help in nova | |
| 21:57:29 | melwitt | this is the change they made in oslo.messaging to allow people (like nova) to pass flags https://review.opendev.org/660373 | |
| 21:58:40 | eandersson | Yea - if they is exposed I would add mandatory to these calls in nova. | |
| 21:59:16 | melwitt | ok, cool | |
| 22:02:11 | eandersson | It will help protect against a bunch of potential RabbitMQ issues | |
| 22:03:47 | melwitt | thanks for clarifying that. I'll find how to add the flag on the nova side | |
| 22:27:08 | eandersson | melwitt, some information on mandatory here https://www.rabbitmq.com/reliability.html#routing | |
| 22:50:02 | melwitt | eandersson: thx | |
| 23:17:48 | openstackgerrit | Merged openstack/nova stable/stein: Stop sending bad values from libosinfo to libvirt https://review.opendev.org/688067 | |
| 23:17:55 | openstackgerrit | Merged openstack/nova stable/queens: Add functional recreate test for regression bug 1825537 https://review.opendev.org/675355 | |
| 23:17:55 | openstack | bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,In progress] https://launchpad.net/bugs/1825537 - Assigned to Matt Riedemann (mriedem) | |
| #openstack-nova - 2019-10-18 | |||
| 00:18:12 | openstackgerrit | Merged openstack/nova stable/queens: Fix rebuild of baremetal instance when vm_state is ERROR https://review.opendev.org/680873 | |