| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 16:10:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move fake_notifier impl under NotificationFixture https://review.opendev.org/c/openstack/nova/+/758451 | |
| 16:10:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Test the NotificationFixture https://review.opendev.org/c/openstack/nova/+/758450 | |
| 16:10:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: rpc: Mark attributes as private https://review.opendev.org/c/openstack/nova/+/792803 | |
| 16:11:09 | sean-k-mooney | gibi: its potentally part of it where are some other optimistaion that they have done in neutron that we could port ot os-vif that might help | |
| 16:11:23 | stephenfin | grenade-multinode failed on the deps change again :( tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_with_attached_volume this time | |
| 16:11:37 | gibi | stephenfin: :/ | |
| 16:11:40 | sean-k-mooney | gibi: it sound like the issue really need to be fixed in the ovs python bindindings | |
| 16:12:00 | gibi | sean-k-mooney: so no easy fix? | |
| 16:12:20 | gibi | sean-k-mooney: should we switch back to vsctl in the gate until we make a fix for the bindig? | |
| 16:12:34 | gibi | stephenfin: looking | |
| 16:12:37 | sean-k-mooney | gibi: my next attempt will be to patch get_system_poll in the ovs binding | |
| 16:12:46 | sean-k-mooney | to not return the unpatch version form os-vif | |
| 16:12:46 | gibi | sean-k-mooney: OK, lets try that | |
| 16:13:14 | sean-k-mooney | gibi: the short term fix would be to revert to the deprecated driver in devstack | |
| 16:13:29 | sean-k-mooney | untill we figure out a way to work around this properly | |
| 16:13:51 | sean-k-mooney | so we can certenly do that if we want too | |
| 16:13:54 | gibi | sean-k-mooney: lets keep the devstack option open, but I think it is OK to try patching get_system_poll first | |
| 16:14:10 | gibi | let see if that helps, if not then go with the devstack change | |
| 16:14:40 | sean-k-mooney | basicaly if i have get_system_poll returng _SelectSelect then i think it shoudl not block | |
| 16:14:43 | sean-k-mooney | on reconnect | |
| 16:15:20 | sean-k-mooney | then the neutron chagnes to disbale the echo and instead use tcp keepalive and limit the tables we cache woudl also miniums reconnect time | |
| 16:16:00 | gibi | stephenfin: that is the volume detaching issue again that where lyarwood wants to look at the guest console log for hints | |
| 16:17:14 | gibi | stephenfin: https://review.opendev.org/c/openstack/tempest/+/794757 | |
| 16:18:06 | gibi | stephenfin: should we just retry or try to land https://review.opendev.org/c/openstack/nova/+/795744 in parallel as well to see which lands first? | |
| 16:19:52 | gibi | or turn off these unstable tests? | |
| 16:49:15 | melwitt | the gate is so angry :( seeing so many of Jun 10 16:08:13.476629 ubuntu-focal-rax-dfw-0025058768 nova-compute[50365]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance 63ddd2de-502b-4467-b7b5-350e601e4d1b from the live domain config. Device is still attached to the guest. | |
| 16:51:09 | gibi | melwitt: yes we discussed it with lyarwood during the day | |
| 16:51:11 | melwitt | whyyyyyy /o\ | |
| 16:51:34 | gibi | he suspects that something in the guest OS makes the detach hanging | |
| 16:52:01 | gibi | hence https://review.opendev.org/c/openstack/tempest/+/794757 | |
| 16:52:17 | lyarwood | I'm going to fix that up quickly now | |
| 16:52:39 | gibi | (I looked at the failures in the tempest test but did not figured out what was wrong:/) | |
| 16:53:09 | melwitt | how does that help the guest OS? sorry I don't understand | |
| 16:53:51 | gibi | that tempest patch will make sure that the guest console log is dumped | |
| 16:54:01 | melwitt | ohhh ok | |
| 16:54:18 | lyarwood | yeah it's not going to resolve anything | |
| 16:54:39 | lyarwood | I had to rework the cleanup ordering to get wait_for_volume_attachment_remove first | |
| 16:54:41 | lyarwood | as that is able to dump the console | |
| 16:55:46 | melwitt | gotcha | |
| 18:02:53 | ganso | melwitt, gibi: hi! could you please take a look at this backport? It already has a +2. Thanks in advance! | |
| 18:04:30 | melwitt | ganso: link? | |
| 18:06:00 | ganso | melwitt: sorry! forgot the link! https://review.opendev.org/c/openstack/nova/+/794328 | |
| 18:06:16 | melwitt | thanks | |
| 18:35:32 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM testing tempest volume detach failure capture of console https://review.opendev.org/c/openstack/nova/+/794766 | |
| 19:15:39 | sean-k-mooney | gibi: looking at the ci result https://review.opendev.org/c/openstack/ovsdbapp/+/795789 actully does seam to if not fix greatly imporve the issue | |
| 19:15:55 | sean-k-mooney | gibi: there are no recoonection or stalls with that patch | |
| 19:16:40 | sean-k-mooney | im still going to look at addtional impormennt within os-vif but that looks to me like a viable path forward | |
| 22:16:29 | opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | |
| #openstack-nova - 2021-06-11 | |||
| 06:17:00 | lyarwood | melwitt: \o/ awesome thanks! | |
| 07:53:42 | opendevreview | liujiong proposed openstack/nova master: Do not create attachment for old root volume https://review.opendev.org/c/openstack/nova/+/795950 | |
| 08:17:48 | bauzas | gibi: stephenfin: so, https://review.opendev.org/c/openstack/nova/+/795533 got a gate failure again | |
| 08:18:05 | bauzas | can we please accept then https://review.opendev.org/c/openstack/nova/+/795744 ? | |
| 08:24:27 | bauzas | looking at project:openstack/nova label:Verified>=-2,Zuul -age:3d is:open we have a lot of changes getting -1 | |
| 08:29:52 | opendevreview | Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362 | |
| 08:29:54 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363 | |
| 08:29:57 | opendevreview | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 08:29:59 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - reject server move and suspend https://review.opendev.org/c/openstack/nova/+/779913 | |
| 08:30:00 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - functional tests https://review.opendev.org/c/openstack/nova/+/780147 | |
| 08:59:48 | lyarwood | bauzas: ack I've +W'd it to chip away at the gate failures | |
| 09:00:46 | bauzas | lyarwood: ok, I'll also provide the revert change | |
| 09:47:53 | opendevreview | Sylvain Bauza proposed openstack/nova master: Revert "Removing mypy to fix the nova CI" https://review.opendev.org/c/openstack/nova/+/795973 | |
| 09:48:05 | bauzas | lyarwood: revert is up | |
| 11:39:31 | opendevreview | Merged openstack/nova master: Removing mypy to fix the nova CI https://review.opendev.org/c/openstack/nova/+/795744 | |
| 11:51:08 | gibi | lyarwood, stephenfin, kashyap: rechecked https://review.opendev.org/c/openstack/nova/+/795533 as it hit https://bugs.launchpad.net/nova/+bug/1912310 | |
| 11:51:23 | kashyap | gibi: Hi; /me clicks | |
| 11:51:27 | gibi | the bug being "libvirt.libvirtError: unable to connect to server at" | |
| 11:51:33 | kashyap | I guess it's the dreaded connection refused thing | |
| 11:51:38 | kashyap | Yep | |
| 11:52:02 | gibi | around the same time when the migration fails | |
| 11:52:23 | gibi | there is an error | |
| 11:52:24 | gibi | multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory\ | |
| 11:53:16 | gibi | on the dest compute | |
| 11:53:32 | gibi | I don't know if it is related but at least it correlates by time | |
| 11:53:46 | gibi | I've added the log links to the bug | |
| 11:55:44 | lyarwood | That shouldn't be related no | |
| 11:55:55 | lyarwood | it's likely n-cpu fetching the connector from os-brick | |
| 11:56:05 | lyarwood | and os-brick is checking if multipathd is present on the host | |
| 11:57:14 | lyarwood | https://bugs.launchpad.net/nova/+bug/1931702 - FWIW I've raised this and will push a skip of the live migration with attached volume tests now | |
| 11:58:31 | gibi | lyarwood: good stuff | |
| 11:59:15 | gibi | lyarwood: is there a way forward with the lockup other than skiping the test? | |
| 11:59:48 | lyarwood | gibi: yeah we don't have the complete console log as I guess it rotated | |
| 12:00:00 | lyarwood | gibi: so I'll work out how to capture that and then raise a QEMU bug depending on what we see | |
| 12:00:10 | gibi | ohh, OK, cool | |
| 12:00:18 | lyarwood | gibi: as I'm assuming live migration has had some kind of impact here | |
| 12:01:16 | kashyap | lyarwood: For the QEMU bug, do you think we have hope of reproducing this outside of the CI Gate? | |
| 12:01:38 | kashyap | lyarwood: E.g. in the same Ubuntu env; same versions, migrating a paused instance -- can that do it? | |
| 12:01:39 | lyarwood | really depends on what if anything we get back from the guestOS | |
| 12:01:49 | kashyap | Yeah; good point | |
| 12:01:56 | lyarwood | the instance isn't paused in this latest failure | |
| 12:12:10 | masterpe[m] | I have instances in the placement.allocations table but these instances does not exists anymore. "nova-manage placement audit" does not exists on Train. And I have tryed "nova-manage placement heal_allocations" but that does not work. Can I savely deleted those records in database? | |
| 12:17:23 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration tests until bug #1931702 is resolved https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:06 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:07 | lyarwood | forgot to update the title sorry | |
| 12:22:16 | frickler | lyarwood: interesting issue, is that with cirros as guest or something else? | |
| 12:22:34 | lyarwood | frickler: yeah it's Cirros 0.5.2 | |
| 12:32:41 | lyarwood | ^ should give us console logs at the time we send the detach to see how borked the guestOS is | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM Revert "zuul: Skip block migration with attached volumes tests due to #1931702" https://review.opendev.org/c/openstack/nova/+/795997 | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to bug #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:33:36 | lyarwood | whops, ^ there we go, git-review being slow again. | |
| 12:38:37 | lyarwood | gibi: https://review.opendev.org/c/openstack/nova/+/792415 simple docs review if you have time btw | |