| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 12:24:34 | opendevreview | sean mooney proposed openstack/os-vif master: [WIP] mock ovs.poller.Poller https://review.opendev.org/c/openstack/os-vif/+/795770 | |
| 12:25:14 | sean-k-mooney | gibi: that passes the os-vif functional tests which actully create ports in ovs and locally i had it assert that the Poller was called | |
| 12:25:27 | sean-k-mooney | but im not sure if this will actully fix the issue | |
| 12:25:56 | sean-k-mooney | ill take a look at the tempest run when its done and we can see if the repeating dbug message is still present or not | |
| 12:29:21 | kashyap | lyarwood: </me back after eclipse hunting> Hey. Have you got a bug filed, or shall I file one? | |
| 12:29:21 | gibi | sean-k-mooney: cool | |
| 12:29:41 | kashyap | [OT] A couple of pictures of projections, if you missed it: https://kashyapc.fedorapeople.org/partial_solar_eclipse_2021/ | |
| 12:29:47 | lyarwood | kashyap: I don't yet so feel free to write one up if you have time | |
| 12:29:59 | kashyap | lyarwood: Sure; I'll do it right now | |
| 12:30:16 | kashyap | But would be good to see if we can reproduce this at least twice... | |
| 12:32:09 | gibi | kashyap: I think we hit it 7 times in the last 7 days http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22libvirt.libvirtError%3A%20unable%20to%20connect%20to%20server%20at%5C%22 | |
| 12:32:57 | kashyap | gibi: Ah, that's good to know. | |
| 12:33:13 | kashyap | So the trigger is block-migrating a puased instance | |
| 12:33:35 | kashyap | (Where "block-migrating" == migrating the instance along with its storage) | |
| 13:21:38 | lyarwood | kashyap: https://bugs.launchpad.net/nova/+bug/1912310 found an older bug we could mark as a duplicate if you've already written a fresh bug up | |
| 13:21:38 | opendevmeet | Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed] | |
| 13:22:35 | kashyap | lyarwood: Ah, good find; I'm just drafting it locally | |
| 13:22:43 | lyarwood | https://bugs.launchpad.net/nova/+bug/1797185 is another | |
| 13:22:43 | opendevmeet | Launchpad bug 1797185 in OpenStack Compute (nova) "live migration intermittently fails in CI with "Connection refused" during guest transfer" [Low,Confirmed] | |
| 13:22:48 | kashyap | Sigh | |
| 13:23:33 | kashyap | lyarwood: What I wonder is - how can we check _why_ the destination is unreahable... | |
| 13:23:41 | kashyap | Is it something peculiar to our upstream CI; or something else | |
| 13:26:10 | lyarwood | I honestly think it's a bug in the libvirt python bindings where they miss that the migration has already failed, try to poll it's progress and don't handle the fact that it's no longer there correctly | |
| 13:26:23 | lyarwood | the unreachable error is a red herring IMHO | |
| 13:27:47 | kashyap | lyarwood: Yeah; I asked Michal from libvirt; and he suggested Jiri Denemark | |
| 13:28:07 | kashyap | He isn't around; but I'll ask him to comment on this once he's back | |
| 13:28:24 | kashyap | lyarwood: Also, you only speak of Python bindings - why won't it be a bug in the C API itself? | |
| 13:28:37 | kashyap | Asking out of ignorance, not challenging :) | |
| 13:28:49 | bauzas | hola | |
| 13:29:00 | bauzas | lyarwood: gibican we know how many jobs have problems ? | |
| 13:29:04 | bauzas | gibi: ^ | |
| 13:29:35 | bauzas | if we have problems with fixing those problems, can we make those specific jobs non-voting until we fix the main issues ? | |
| 13:29:39 | lyarwood | kashyap: it could be that as well but n-cpu is calling the python bindings and they are raising the error here so it easily be an issue there | |
| 13:30:20 | bauzas | fwiw, I saw that https://review.opendev.org/c/openstack/nova/+/791506 passed the check pipeline but we don't know yet whether the gate one will accept it | |
| 13:30:34 | kashyap | lyarwood: Ah, okay; you're just speaking of the error trace in our case. Reasonable | |
| 13:31:08 | kashyap | lyarwood: Can you copy/paste your comment from the review to start with in there? | |
| 13:31:18 | kashyap | (In the latest bug you filed 2021 Jan) | |
| 13:31:30 | lyarwood | bauzas: There are various intermittent failures at the moment, I'm trying to clean up the gate-failures bug tag to get a better handle on what is failing and how often | |
| 13:32:02 | lyarwood | bauzas: we could move them to non-voting but this has been going on for weeks so I'd rather we try to get a better handle on this first | |
| 13:32:04 | bauzas | okay, fair enough | |
| 13:32:27 | lyarwood | I honestly think we haven't been landing enough to notice recently | |
| 13:32:28 | bauzas | lyarwood: I guess my point is that we're still having *all* the changes getting -1 because of the mypy issue | |
| 13:32:51 | bauzas | so I want the gate back asap | |
| 13:33:06 | lyarwood | yup fair | |
| 13:33:07 | bauzas | we have two possibilities, the main fix https://review.opendev.org/c/openstack/nova/+/791506 | |
| 13:33:38 | bauzas | but I also prepared an alternative change that's simplier for the gate and doesn't hit the transient issues we know https://review.opendev.org/c/openstack/nova/+/795744 | |
| 13:34:17 | bauzas | so, I guess, I'd recommend to wait for the main change to merge, but if we get a Zuul -2 from the gate on it, I'd honestly recommend to let https://review.opendev.org/c/openstack/nova/+/795744 go | |
| 13:42:23 | lyarwood | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1863889/comments/3 another possible hit of the ovsdbapp issue | |
| 13:42:23 | opendevmeet | Launchpad bug 1863889 in OpenStack Compute (nova) "Revert resize problem in neutron-tempest-dvr-ha-multinode-full" [Medium,Confirmed] | |
| 13:44:06 | sean-k-mooney | lyarwood: im talking to otherwiseguy and ralonsoh in #openstack-neutron about it now but ill let them know | |
| 13:44:28 | lyarwood | it's just another random timeout FWIW | |
| 13:44:44 | lyarwood | with lots of spam from ovsdbapp so no hard proof | |
| 13:55:31 | bauzas | lyarwood: the problem is that logstash isn't reliable | |
| 13:55:44 | bauzas | despite some people like me and dansmith expressing our needs | |
| 13:56:07 | bauzas | we need to focus on the few things we saw occuring often | |
| 13:56:20 | bauzas | and either fix them or hide them under the carpet until we are in a better situation | |
| 14:01:25 | lyarwood | yup agreed | |
| 14:01:49 | lyarwood | and that's what I have been doing prior to now with things like https://bugs.launchpad.net/nova/+bug/1929710 for example | |
| 14:01:49 | opendevmeet | Launchpad bug 1929710 in OpenStack Compute (nova) "virDomainGetBlockJobInfo fails during swap_volume as disk '$disk' not found in domain" [Medium,New] | |
| 14:03:08 | lyarwood | https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure should be in a much better state now, I've moved loads of things to incomplete where I couldn't find any evidence of recent hits etc. | |
| 14:03:23 | lyarwood | these should close automatically in the next 60 days | |
| 14:08:31 | bauzas | lyarwood: ++ thanks for triaging | |
| 14:22:05 | opendevreview | Merged openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 14:36:57 | kashyap | lyarwood: Before I forget, for now, I've copy/pasted (with attribution) your comment from the change here: https://bugs.launchpad.net/nova/+bug/1912310 | |
| 14:36:57 | opendevmeet | Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed] | |
| 14:40:12 | gibi | lyarwood: thanks for the cleanup. I add that query to the weekly agenda | |
| 14:40:48 | lyarwood | kashyap: ack thanks | |
| 14:40:53 | lyarwood | gibi: awesome cheers | |
| 14:54:23 | sean-k-mooney | gibi: lyarwood mocking the poller wont work but i have a seperate patch to move it to a real thread against ovsdb and talking to otherwiseguy https://bugs.launchpad.net/neutron/+bug/1930926 and https://review.opendev.org/c/openstack/neutron/+/794892 might also help if we ported the same change to os-vif | |
| 14:54:23 | opendevmeet | Launchpad bug 1930926 in neutron "Failing over OVN dbs can cause original controller to permanently lose connection" [Medium,Fix released] - Assigned to Terry Wilson (otherwiseguy) | |
| 14:54:38 | sean-k-mooney | mocking the poller actully cause the delay | |
| 14:54:59 | sean-k-mooney | as it forces use to reconnect every 5 seconds sicne we dont respond to the server side echo | |
| 14:56:05 | lyarwood | `# Overwriting globals in a library is clearly a good idea` lol | |
| 14:56:49 | lyarwood | sean-k-mooney: ack kk | |
| 14:58:54 | sean-k-mooney | ya... | |
| 15:00:04 | gibi | sean-k-mooney: will look at it shortly | |
| 15:32:25 | bauzas | mmm, /me needs to leave, unfortunately too early to see outcomes of https://zuul.opendev.org/t/openstack/status#795533 | |
| 15:32:51 | bauzas | hopefully the change will be merged, but the failing jobs are currently still running | |
| 15:54:17 | gibi | sean-k-mooney: is the the new directon? https://review.opendev.org/c/openstack/ovsdbapp/+/795789 | |
| 16:10:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Test the NotificationFixture https://review.opendev.org/c/openstack/nova/+/758450 | |
| 16:10:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move fake_notifier impl under NotificationFixture https://review.opendev.org/c/openstack/nova/+/758451 | |
| 16:10:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: rpc: Mark attributes as private https://review.opendev.org/c/openstack/nova/+/792803 | |
| 16:11:09 | sean-k-mooney | gibi: its potentally part of it where are some other optimistaion that they have done in neutron that we could port ot os-vif that might help | |
| 16:11:23 | stephenfin | grenade-multinode failed on the deps change again :( tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_with_attached_volume this time | |
| 16:11:37 | gibi | stephenfin: :/ | |
| 16:11:40 | sean-k-mooney | gibi: it sound like the issue really need to be fixed in the ovs python bindindings | |
| 16:12:00 | gibi | sean-k-mooney: so no easy fix? | |
| 16:12:20 | gibi | sean-k-mooney: should we switch back to vsctl in the gate until we make a fix for the bindig? | |
| 16:12:34 | gibi | stephenfin: looking | |
| 16:12:37 | sean-k-mooney | gibi: my next attempt will be to patch get_system_poll in the ovs binding | |
| 16:12:46 | gibi | sean-k-mooney: OK, lets try that | |
| 16:12:46 | sean-k-mooney | to not return the unpatch version form os-vif | |
| 16:13:14 | sean-k-mooney | gibi: the short term fix would be to revert to the deprecated driver in devstack | |
| 16:13:29 | sean-k-mooney | untill we figure out a way to work around this properly | |
| 16:13:51 | sean-k-mooney | so we can certenly do that if we want too | |
| 16:13:54 | gibi | sean-k-mooney: lets keep the devstack option open, but I think it is OK to try patching get_system_poll first | |
| 16:14:10 | gibi | let see if that helps, if not then go with the devstack change | |
| 16:14:40 | sean-k-mooney | basicaly if i have get_system_poll returng _SelectSelect then i think it shoudl not block | |
| 16:14:43 | sean-k-mooney | on reconnect | |
| 16:15:20 | sean-k-mooney | then the neutron chagnes to disbale the echo and instead use tcp keepalive and limit the tables we cache woudl also miniums reconnect time | |