Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-10
13:31:30 lyarwood bauzas: There are various intermittent failures at the moment, I'm trying to clean up the gate-failures bug tag to get a better handle on what is failing and how often
13:32:02 lyarwood bauzas: we could move them to non-voting but this has been going on for weeks so I'd rather we try to get a better handle on this first
13:32:04 bauzas okay, fair enough
13:32:27 lyarwood I honestly think we haven't been landing enough to notice recently
13:32:28 bauzas lyarwood: I guess my point is that we're still having *all* the changes getting -1 because of the mypy issue
13:32:51 bauzas so I want the gate back asap
13:33:06 lyarwood yup fair
13:33:07 bauzas we have two possibilities, the main fix https://review.opendev.org/c/openstack/nova/+/791506
13:33:38 bauzas but I also prepared an alternative change that's simplier for the gate and doesn't hit the transient issues we know https://review.opendev.org/c/openstack/nova/+/795744
13:34:17 bauzas so, I guess, I'd recommend to wait for the main change to merge, but if we get a Zuul -2 from the gate on it, I'd honestly recommend to let https://review.opendev.org/c/openstack/nova/+/795744 go
13:42:23 opendevmeet Launchpad bug 1863889 in OpenStack Compute (nova) "Revert resize problem in neutron-tempest-dvr-ha-multinode-full" [Medium,Confirmed]
13:42:23 lyarwood sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1863889/comments/3 another possible hit of the ovsdbapp issue
13:44:06 sean-k-mooney lyarwood: im talking to otherwiseguy and ralonsoh in #openstack-neutron about it now but ill let them know
13:44:28 lyarwood it's just another random timeout FWIW
13:44:44 lyarwood with lots of spam from ovsdbapp so no hard proof
13:55:31 bauzas lyarwood: the problem is that logstash isn't reliable
13:55:44 bauzas despite some people like me and dansmith expressing our needs
13:56:07 bauzas we need to focus on the few things we saw occuring often
13:56:20 bauzas and either fix them or hide them under the carpet until we are in a better situation
14:01:25 lyarwood yup agreed
14:01:49 opendevmeet Launchpad bug 1929710 in OpenStack Compute (nova) "virDomainGetBlockJobInfo fails during swap_volume as disk '$disk' not found in domain" [Medium,New]
14:01:49 lyarwood and that's what I have been doing prior to now with things like https://bugs.launchpad.net/nova/+bug/1929710 for example
14:03:08 lyarwood https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure should be in a much better state now, I've moved loads of things to incomplete where I couldn't find any evidence of recent hits etc.
14:03:23 lyarwood these should close automatically in the next 60 days
14:08:31 bauzas lyarwood: ++ thanks for triaging
14:22:05 opendevreview Merged openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620
14:36:57 opendevmeet Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed]
14:36:57 kashyap lyarwood: Before I forget, for now, I've copy/pasted (with attribution) your comment from the change here: https://bugs.launchpad.net/nova/+bug/1912310
14:40:12 gibi lyarwood: thanks for the cleanup. I add that query to the weekly agenda
14:40:48 lyarwood kashyap: ack thanks
14:40:53 lyarwood gibi: awesome cheers
14:54:23 opendevmeet Launchpad bug 1930926 in neutron "Failing over OVN dbs can cause original controller to permanently lose connection" [Medium,Fix released] - Assigned to Terry Wilson (otherwiseguy)
14:54:23 sean-k-mooney gibi: lyarwood mocking the poller wont work but i have a seperate patch to move it to a real thread against ovsdb and talking to otherwiseguy https://bugs.launchpad.net/neutron/+bug/1930926 and https://review.opendev.org/c/openstack/neutron/+/794892 might also help if we ported the same change to os-vif
14:54:38 sean-k-mooney mocking the poller actully cause the delay
14:54:59 sean-k-mooney as it forces use to reconnect every 5 seconds sicne we dont respond to the server side echo
14:56:05 lyarwood `# Overwriting globals in a library is clearly a good idea` lol
14:56:49 lyarwood sean-k-mooney: ack kk
14:58:54 sean-k-mooney ya...
15:00:04 gibi sean-k-mooney: will look at it shortly
15:32:25 bauzas mmm, /me needs to leave, unfortunately too early to see outcomes of https://zuul.opendev.org/t/openstack/status#795533
15:32:51 bauzas hopefully the change will be merged, but the failing jobs are currently still running
15:54:17 gibi sean-k-mooney: is the the new directon? https://review.opendev.org/c/openstack/ovsdbapp/+/795789
16:10:01 opendevreview Balazs Gibizer proposed openstack/nova master: Move fake_notifier impl under NotificationFixture https://review.opendev.org/c/openstack/nova/+/758451
16:10:01 opendevreview Balazs Gibizer proposed openstack/nova master: Test the NotificationFixture https://review.opendev.org/c/openstack/nova/+/758450
16:10:02 opendevreview Balazs Gibizer proposed openstack/nova master: rpc: Mark attributes as private https://review.opendev.org/c/openstack/nova/+/792803
16:11:09 sean-k-mooney gibi: its potentally part of it where are some other optimistaion that they have done in neutron that we could port ot os-vif that might help
16:11:23 stephenfin grenade-multinode failed on the deps change again :( tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_with_attached_volume this time
16:11:37 gibi stephenfin: :/
16:11:40 sean-k-mooney gibi: it sound like the issue really need to be fixed in the ovs python bindindings
16:12:00 gibi sean-k-mooney: so no easy fix?
16:12:20 gibi sean-k-mooney: should we switch back to vsctl in the gate until we make a fix for the bindig?
16:12:34 gibi stephenfin: looking
16:12:37 sean-k-mooney gibi: my next attempt will be to patch get_system_poll in the ovs binding
16:12:46 sean-k-mooney to not return the unpatch version form os-vif
16:12:46 gibi sean-k-mooney: OK, lets try that
16:13:14 sean-k-mooney gibi: the short term fix would be to revert to the deprecated driver in devstack
16:13:29 sean-k-mooney untill we figure out a way to work around this properly
16:13:51 sean-k-mooney so we can certenly do that if we want too
16:13:54 gibi sean-k-mooney: lets keep the devstack option open, but I think it is OK to try patching get_system_poll first
16:14:10 gibi let see if that helps, if not then go with the devstack change
16:14:40 sean-k-mooney basicaly if i have get_system_poll returng _SelectSelect then i think it shoudl not block
16:14:43 sean-k-mooney on reconnect
16:15:20 sean-k-mooney then the neutron chagnes to disbale the echo and instead use tcp keepalive and limit the tables we cache woudl also miniums reconnect time
16:16:00 gibi stephenfin: that is the volume detaching issue again that where lyarwood wants to look at the guest console log for hints
16:17:14 gibi stephenfin: https://review.opendev.org/c/openstack/tempest/+/794757
16:18:06 gibi stephenfin: should we just retry or try to land https://review.opendev.org/c/openstack/nova/+/795744 in parallel as well to see which lands first?
16:19:52 gibi or turn off these unstable tests?
16:49:15 melwitt the gate is so angry :( seeing so many of Jun 10 16:08:13.476629 ubuntu-focal-rax-dfw-0025058768 nova-compute[50365]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance 63ddd2de-502b-4467-b7b5-350e601e4d1b from the live domain config. Device is still attached to the guest.
16:51:09 gibi melwitt: yes we discussed it with lyarwood during the day
16:51:11 melwitt whyyyyyy /o\
16:51:34 gibi he suspects that something in the guest OS makes the detach hanging
16:52:01 gibi hence https://review.opendev.org/c/openstack/tempest/+/794757
16:52:17 lyarwood I'm going to fix that up quickly now
16:52:39 gibi (I looked at the failures in the tempest test but did not figured out what was wrong:/)
16:53:09 melwitt how does that help the guest OS? sorry I don't understand
16:53:51 gibi that tempest patch will make sure that the guest console log is dumped
16:54:01 melwitt ohhh ok
16:54:18 lyarwood yeah it's not going to resolve anything
16:54:39 lyarwood I had to rework the cleanup ordering to get wait_for_volume_attachment_remove first
16:54:41 lyarwood as that is able to dump the console
16:55:46 melwitt gotcha
18:02:53 ganso melwitt, gibi: hi! could you please take a look at this backport? It already has a +2. Thanks in advance!
18:04:30 melwitt ganso: link?
18:06:00 ganso melwitt: sorry! forgot the link! https://review.opendev.org/c/openstack/nova/+/794328
18:06:16 melwitt thanks
18:35:32 opendevreview Lee Yarwood proposed openstack/nova master: DNM testing tempest volume detach failure capture of console https://review.opendev.org/c/openstack/nova/+/794766
19:15:39 sean-k-mooney gibi: looking at the ci result https://review.opendev.org/c/openstack/ovsdbapp/+/795789 actully does seam to if not fix greatly imporve the issue
19:15:55 sean-k-mooney gibi: there are no recoonection or stalls with that patch
19:16:40 sean-k-mooney im still going to look at addtional impormennt within os-vif but that looks to me like a viable path forward
22:16:29 opendevreview Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519
#openstack-nova - 2021-06-11
06:17:00 lyarwood melwitt: \o/ awesome thanks!
07:53:42 opendevreview liujiong proposed openstack/nova master: Do not create attachment for old root volume https://review.opendev.org/c/openstack/nova/+/795950
08:17:48 bauzas gibi: stephenfin: so, https://review.opendev.org/c/openstack/nova/+/795533 got a gate failure again
08:18:05 bauzas can we please accept then https://review.opendev.org/c/openstack/nova/+/795744 ?
08:24:27 bauzas looking at project:openstack/nova label:Verified>=-2,Zuul -age:3d is:open we have a lot of changes getting -1
08:29:52 opendevreview Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362
08:29:54 opendevreview Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363
08:29:57 opendevreview Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944
08:29:59 opendevreview Yongli He proposed openstack/nova master: smartnic support - reject server move and suspend https://review.opendev.org/c/openstack/nova/+/779913
08:30:00 opendevreview Yongli He proposed openstack/nova master: smartnic support - functional tests https://review.opendev.org/c/openstack/nova/+/780147

Earlier   Later