| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 11:42:44 | sean-k-mooney | ok ill see if i can play with this quickly but i think we should really fix this in ovsdbapp by allowing the connect to be created without poolling | |
| 11:43:48 | sean-k-mooney | https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L98-L102 | |
| 11:43:52 | sean-k-mooney | this does concern me a bit | |
| 11:44:32 | lyarwood | https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure we really need to clean this out | |
| 11:45:10 | sean-k-mooney | but yes ovs is defently un monkypatching https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270 | |
| 11:55:03 | opendevmeet | Launchpad bug 1793364 in Cinder "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] | |
| 11:55:03 | lyarwood | gibi: is https://bugs.launchpad.net/nova/+bug/1793364 resolved now by https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%22+(status:open%20OR%20status:merged) ? | |
| 11:55:12 | lyarwood | your comment suggests it was supposed to be | |
| 11:55:55 | gibi | lyarwood: it helped but I think we saw timeouts even after the compation | |
| 11:56:06 | gibi | let me find the info | |
| 11:56:21 | sean-k-mooney | i think we did too | |
| 11:56:26 | lyarwood | kk do we want to keep this open or move it to incomplete and update it with fresh logs etc the next time it happens? | |
| 11:56:37 | gibi | https://review.opendev.org/c/openstack/nova/+/774889/1#message-db5303e41b22ebfb7b33067d815a12ece2df510b | |
| 11:56:57 | gibi | lyarwood: ack | |
| 11:57:07 | gibi | I will check logstash for fresh occurences | |
| 12:00:35 | masterpe[m] | If I run ./nova-manage placement heal_allocations on Train I get the error: "Compute host scpuko57 could not be found." But it is in the "openstack hypervisor list" and "openstack compute service list" | |
| 12:02:10 | opendevmeet | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 12:02:10 | gibi | I made 1793364 duplicate of https://bugs.launchpad.net/nova/+bug/1823251 (as the newer report has more info) and added a link to the log of a recent occurence | |
| 12:03:26 | gibi | I have https://review.opendev.org/c/openstack/nova/+/775094 for more logs (now rechecked) but honestly I looked at this problem so many times that I don't think I can solve it. | |
| 12:24:34 | opendevreview | sean mooney proposed openstack/os-vif master: [WIP] mock ovs.poller.Poller https://review.opendev.org/c/openstack/os-vif/+/795770 | |
| 12:25:14 | sean-k-mooney | gibi: that passes the os-vif functional tests which actully create ports in ovs and locally i had it assert that the Poller was called | |
| 12:25:27 | sean-k-mooney | but im not sure if this will actully fix the issue | |
| 12:25:56 | sean-k-mooney | ill take a look at the tempest run when its done and we can see if the repeating dbug message is still present or not | |
| 12:29:21 | gibi | sean-k-mooney: cool | |
| 12:29:21 | kashyap | lyarwood: </me back after eclipse hunting> Hey. Have you got a bug filed, or shall I file one? | |
| 12:29:41 | kashyap | [OT] A couple of pictures of projections, if you missed it: https://kashyapc.fedorapeople.org/partial_solar_eclipse_2021/ | |
| 12:29:47 | lyarwood | kashyap: I don't yet so feel free to write one up if you have time | |
| 12:29:59 | kashyap | lyarwood: Sure; I'll do it right now | |
| 12:30:16 | kashyap | But would be good to see if we can reproduce this at least twice... | |
| 12:32:09 | gibi | kashyap: I think we hit it 7 times in the last 7 days http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22libvirt.libvirtError%3A%20unable%20to%20connect%20to%20server%20at%5C%22 | |
| 12:32:57 | kashyap | gibi: Ah, that's good to know. | |
| 12:33:13 | kashyap | So the trigger is block-migrating a puased instance | |
| 12:33:35 | kashyap | (Where "block-migrating" == migrating the instance along with its storage) | |
| 13:21:38 | opendevmeet | Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed] | |
| 13:21:38 | lyarwood | kashyap: https://bugs.launchpad.net/nova/+bug/1912310 found an older bug we could mark as a duplicate if you've already written a fresh bug up | |
| 13:22:35 | kashyap | lyarwood: Ah, good find; I'm just drafting it locally | |
| 13:22:43 | opendevmeet | Launchpad bug 1797185 in OpenStack Compute (nova) "live migration intermittently fails in CI with "Connection refused" during guest transfer" [Low,Confirmed] | |
| 13:22:43 | lyarwood | https://bugs.launchpad.net/nova/+bug/1797185 is another | |
| 13:22:48 | kashyap | Sigh | |
| 13:23:33 | kashyap | lyarwood: What I wonder is - how can we check _why_ the destination is unreahable... | |
| 13:23:41 | kashyap | Is it something peculiar to our upstream CI; or something else | |
| 13:26:10 | lyarwood | I honestly think it's a bug in the libvirt python bindings where they miss that the migration has already failed, try to poll it's progress and don't handle the fact that it's no longer there correctly | |
| 13:26:23 | lyarwood | the unreachable error is a red herring IMHO | |
| 13:27:47 | kashyap | lyarwood: Yeah; I asked Michal from libvirt; and he suggested Jiri Denemark | |
| 13:28:07 | kashyap | He isn't around; but I'll ask him to comment on this once he's back | |
| 13:28:24 | kashyap | lyarwood: Also, you only speak of Python bindings - why won't it be a bug in the C API itself? | |
| 13:28:37 | kashyap | Asking out of ignorance, not challenging :) | |
| 13:28:49 | bauzas | hola | |
| 13:29:00 | bauzas | lyarwood: gibican we know how many jobs have problems ? | |
| 13:29:04 | bauzas | gibi: ^ | |
| 13:29:35 | bauzas | if we have problems with fixing those problems, can we make those specific jobs non-voting until we fix the main issues ? | |
| 13:29:39 | lyarwood | kashyap: it could be that as well but n-cpu is calling the python bindings and they are raising the error here so it easily be an issue there | |
| 13:30:20 | bauzas | fwiw, I saw that https://review.opendev.org/c/openstack/nova/+/791506 passed the check pipeline but we don't know yet whether the gate one will accept it | |
| 13:30:34 | kashyap | lyarwood: Ah, okay; you're just speaking of the error trace in our case. Reasonable | |
| 13:31:08 | kashyap | lyarwood: Can you copy/paste your comment from the review to start with in there? | |
| 13:31:18 | kashyap | (In the latest bug you filed 2021 Jan) | |
| 13:31:30 | lyarwood | bauzas: There are various intermittent failures at the moment, I'm trying to clean up the gate-failures bug tag to get a better handle on what is failing and how often | |
| 13:32:02 | lyarwood | bauzas: we could move them to non-voting but this has been going on for weeks so I'd rather we try to get a better handle on this first | |
| 13:32:04 | bauzas | okay, fair enough | |
| 13:32:27 | lyarwood | I honestly think we haven't been landing enough to notice recently | |
| 13:32:28 | bauzas | lyarwood: I guess my point is that we're still having *all* the changes getting -1 because of the mypy issue | |
| 13:32:51 | bauzas | so I want the gate back asap | |
| 13:33:06 | lyarwood | yup fair | |
| 13:33:07 | bauzas | we have two possibilities, the main fix https://review.opendev.org/c/openstack/nova/+/791506 | |
| 13:33:38 | bauzas | but I also prepared an alternative change that's simplier for the gate and doesn't hit the transient issues we know https://review.opendev.org/c/openstack/nova/+/795744 | |
| 13:34:17 | bauzas | so, I guess, I'd recommend to wait for the main change to merge, but if we get a Zuul -2 from the gate on it, I'd honestly recommend to let https://review.opendev.org/c/openstack/nova/+/795744 go | |
| 13:42:23 | opendevmeet | Launchpad bug 1863889 in OpenStack Compute (nova) "Revert resize problem in neutron-tempest-dvr-ha-multinode-full" [Medium,Confirmed] | |
| 13:42:23 | lyarwood | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1863889/comments/3 another possible hit of the ovsdbapp issue | |
| 13:44:06 | sean-k-mooney | lyarwood: im talking to otherwiseguy and ralonsoh in #openstack-neutron about it now but ill let them know | |
| 13:44:28 | lyarwood | it's just another random timeout FWIW | |
| 13:44:44 | lyarwood | with lots of spam from ovsdbapp so no hard proof | |
| 13:55:31 | bauzas | lyarwood: the problem is that logstash isn't reliable | |
| 13:55:44 | bauzas | despite some people like me and dansmith expressing our needs | |
| 13:56:07 | bauzas | we need to focus on the few things we saw occuring often | |
| 13:56:20 | bauzas | and either fix them or hide them under the carpet until we are in a better situation | |
| 14:01:25 | lyarwood | yup agreed | |
| 14:01:49 | opendevmeet | Launchpad bug 1929710 in OpenStack Compute (nova) "virDomainGetBlockJobInfo fails during swap_volume as disk '$disk' not found in domain" [Medium,New] | |
| 14:01:49 | lyarwood | and that's what I have been doing prior to now with things like https://bugs.launchpad.net/nova/+bug/1929710 for example | |
| 14:03:08 | lyarwood | https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure should be in a much better state now, I've moved loads of things to incomplete where I couldn't find any evidence of recent hits etc. | |
| 14:03:23 | lyarwood | these should close automatically in the next 60 days | |
| 14:08:31 | bauzas | lyarwood: ++ thanks for triaging | |
| 14:22:05 | opendevreview | Merged openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 14:36:57 | opendevmeet | Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed] | |
| 14:36:57 | kashyap | lyarwood: Before I forget, for now, I've copy/pasted (with attribution) your comment from the change here: https://bugs.launchpad.net/nova/+bug/1912310 | |
| 14:40:12 | gibi | lyarwood: thanks for the cleanup. I add that query to the weekly agenda | |
| 14:40:48 | lyarwood | kashyap: ack thanks | |
| 14:40:53 | lyarwood | gibi: awesome cheers | |
| 14:54:23 | opendevmeet | Launchpad bug 1930926 in neutron "Failing over OVN dbs can cause original controller to permanently lose connection" [Medium,Fix released] - Assigned to Terry Wilson (otherwiseguy) | |
| 14:54:23 | sean-k-mooney | gibi: lyarwood mocking the poller wont work but i have a seperate patch to move it to a real thread against ovsdb and talking to otherwiseguy https://bugs.launchpad.net/neutron/+bug/1930926 and https://review.opendev.org/c/openstack/neutron/+/794892 might also help if we ported the same change to os-vif | |
| 14:54:38 | sean-k-mooney | mocking the poller actully cause the delay | |
| 14:54:59 | sean-k-mooney | as it forces use to reconnect every 5 seconds sicne we dont respond to the server side echo | |
| 14:56:05 | lyarwood | `# Overwriting globals in a library is clearly a good idea` lol | |
| 14:56:49 | lyarwood | sean-k-mooney: ack kk | |
| 14:58:54 | sean-k-mooney | ya... | |
| 15:00:04 | gibi | sean-k-mooney: will look at it shortly | |
| 15:32:25 | bauzas | mmm, /me needs to leave, unfortunately too early to see outcomes of https://zuul.opendev.org/t/openstack/status#795533 | |
| 15:32:51 | bauzas | hopefully the change will be merged, but the failing jobs are currently still running | |
| 15:54:17 | gibi | sean-k-mooney: is the the new directon? https://review.opendev.org/c/openstack/ovsdbapp/+/795789 | |
| 16:10:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move fake_notifier impl under NotificationFixture https://review.opendev.org/c/openstack/nova/+/758451 | |
| 16:10:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Test the NotificationFixture https://review.opendev.org/c/openstack/nova/+/758450 | |