Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-23
14:42:19 sean-k-mooney oh ya its the tear down method
14:42:29 elodilles gibi: well I've seen that there was an ongoing discussion so I did not want to interrupt earlier :)
14:42:29 sean-k-mooney ok that make a liit more sense
14:43:42 sean-k-mooney ok so the teardown failed becasue presumable the volume was still attached to the vm
14:43:54 sean-k-mooney it was in state detaching
14:46:28 gibi it is trying to detach in a loop so I guess it is the original detach problem
14:46:39 gibi for what we created the notification based solution
14:46:47 gibi but that solution is not in victoria
14:46:58 sean-k-mooney ya the detach starts at Jun 21 09:20:16.007823
14:48:00 gibi so either we increase the timeout value before we retry in victoria or backport the notification based solution to victoria
14:48:39 sean-k-mooney we are calling os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume on it
14:48:59 sean-k-mooney we would not do that if the detach ahs not commpeted right?
14:49:37 gibi sean-k-mooney: I don't know. what I know that the detach is not completed from libvirt perspective, it still reports the device in the live domain after 7 retries
14:50:03 sean-k-mooney this is happening as part of a live migration
14:51:08 sean-k-mooney this looks odd what is the test actully doing
14:51:38 sean-k-mooney it look like we are runign post live migration on the souce which is tearing odwn the volume while its potentioanly still detaching
14:51:58 gibi I think lyarwood has more context on this detach problem
14:52:13 sean-k-mooney im wonderign if the tempest test is broken
14:52:42 sean-k-mooney e.g. is it issuing a detach and then a live migrate without waiting for the detach to complete
14:53:45 sean-k-mooney this is the test https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_live_migration.py#L178-L204
14:54:30 sean-k-mooney thats doing an attach
14:54:41 sean-k-mooney but its not doing a detach
14:54:53 sean-k-mooney its also not waiting to ensure its actully attached
14:55:06 sean-k-mooney unless self.attach_volume is doint that internally
14:56:31 sean-k-mooney ok if the attach fails its cleanup funciton does a detach im assuming https://github.com/openstack/tempest/blob/53c02181f87804a4ba8ddf6288ea1f7717234c2a/tempest/api/compute/base.py#L554-L591
14:56:47 sean-k-mooney it is waiting internally
14:57:42 sean-k-mooney https://github.com/openstack/tempest/blob/53c02181f87804a4ba8ddf6288ea1f7717234c2a/tempest/api/compute/base.py#L590
14:58:02 sean-k-mooney can we see the volume attach complete in the logs
15:00:39 sean-k-mooney we do the attachment in libvirt at un 21 09:20:00.476733
15:10:10 lyarwood if we are still talking about https://c5d6a707d1df71acd55f-fbeed0693cac4a6e5441d43111515edc.ssl.cf5.rackcdn.com/787252/4/gate/nova-live-migration/706ec17/testr_results.html it's just another basic detach failure from the live domain
15:10:22 lyarwood after the test has passed and we are cleaning up
15:12:12 lyarwood I really need to land https://review.opendev.org/c/openstack/tempest/+/794757 so we can get some guestOS logs in this case
15:13:30 gibi lyarwood: that run is from stable/victoria so we don't have the new detach code there. Do you think it would help if we try to backport the new detach code? or is it possibly unrelated?
15:13:31 lyarwood or we could just Depends-On that from a DNM stable/victoria change
15:13:38 elodilles lyarwood: so you are saying that this is similar like bug #1931716 but not the exact same case?
15:14:04 lyarwood gibi: we've seen failures of this test on master so this time I don't think backporting that code is going to help
15:14:06 lyarwood brb
15:15:15 gibi lyarwood: OK, then I agree to land the tempest improvement first to get more info
15:18:44 opendevreview Elod Illes proposed openstack/nova stable/victoria: DNM: volume detach test https://review.opendev.org/c/openstack/nova/+/797675
15:19:30 elodilles I've created the DNM patch what lyarwood suggested ^^^
15:21:51 lyarwood elodilles: thanks
15:22:23 elodilles np. let's see what happens :)
15:30:06 lyarwood elodilles: I have a change to skip the test on master btw that we could also backport if we see the same soft lockups
15:30:42 opendevreview Stephen Finucane proposed openstack/nova stable/victoria: Test numa and vcpu topologies bug: #1910466 https://review.opendev.org/c/openstack/nova/+/797680
15:30:43 opendevreview Stephen Finucane proposed openstack/nova stable/victoria: Fix max cpu topologies with numa affinity https://review.opendev.org/c/openstack/nova/+/797681
15:35:54 elodilles lyarwood: thanks, good to know that, we might need to backport that then if needed
15:46:02 kashyap gibi: lyarwood: sean-k-mooney: When you get a min, have a look to see if you spot any holes there: https://blueprints.launchpad.net/nova/+spec/virtio-as-default-display-device
15:50:50 sean-k-mooney that will need a spec most likely
15:51:11 sean-k-mooney im not agaisnt it but we need to record the current modele in use
15:51:17 sean-k-mooney and then only change the default for new instnaces
15:59:48 sean-k-mooney lyarwood: so in this case we dont actully want to do the detach on succesful migration right
16:00:01 sean-k-mooney that is jsut an artifcat of the way the cleanup is done
16:00:11 sean-k-mooney we just want to delete the vm then delete the volume
16:00:19 lyarwood sean-k-mooney: yeah it's just part of the tempest cleanup code that's added when we initially attach
16:00:33 sean-k-mooney could we modify that
16:00:45 lyarwood indeed, we don't need to detach as part of this test
16:00:52 sean-k-mooney add a flag to attach e.g cleanup=false
16:00:55 lyarwood so we could just nuke the instance and move on with our lives
16:01:18 sean-k-mooney yep
16:01:21 lyarwood yup, the only issue is that there's duplication in the tempest code base with lots of this utility code
16:01:32 lyarwood but I can do this for the compute attach volume part at least
16:01:42 lyarwood I think there's another helper in the volume api tests
16:01:47 lyarwood and maybe the scenario manager
16:02:21 sean-k-mooney it should in thoery decrease test execution time
16:03:12 lyarwood Yup we would however drop some coverage of detaching volumes
16:34:00 sean-k-mooney lyarwood: we would but in principal we have tests for that specifically
16:34:23 sean-k-mooney detach is not really a part of what we are testing in this case
16:34:37 sean-k-mooney it does expose racecondition more as we see here
16:34:49 sean-k-mooney but im not sure that is always a good thing
17:19:37 opendevreview Lee Yarwood proposed openstack/nova master: WIP compute: Avoid calling detach with src connection_info during LM rollback https://review.opendev.org/c/openstack/nova/+/797725
17:27:29 sean-k-mooney that was quick
17:39:33 ganso lyarwood: hi! is there anything pending on https://review.opendev.org/c/openstack/nova/+/795432 so it can be merged? That patch was the latest one to pass CI. There are several patches (including one with +W) that haven't had a single positive CI run
19:37:58 lyarwood elodilles: https://review.opendev.org/c/openstack/nova/+/795432 - can you hit this in the morning? I've upgraded my +1 to +2 to move it along.
19:38:17 lyarwood ganso: apologies, I'll work with elodilles and the other stable cores to unblock things in the morning
19:39:47 ganso lyarwood: np! thank you very much! I was mostly wondering if something else was pending because I had rebased on top of it and it still failed. See latest comments in https://review.opendev.org/c/openstack/nova/+/796719
#openstack-nova - 2021-06-24
07:25:05 opendevreview Slawek Kaplonski proposed openstack/nova stable/ussuri: [neutron] Get only ID and name of the SGs from Neutron https://review.opendev.org/c/openstack/nova/+/787253
07:47:10 opendevreview Elod Illes proposed openstack/nova stable/ussuri: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/794675
07:51:55 elodilles lyarwood: uhh, I almost messed up the supermegasquash :S
07:52:19 elodilles lyarwood: btw, do we really need there the "Replace nova-live-migration with zuulv3 jobs" ?
07:52:26 elodilles :/
08:18:32 elodilles lyarwood: anyway, I've +2+W'd the supermegasquash
08:23:16 elodilles lyarwood: I also abandoned the ussuri patches that were squashed into the supermegasquash
08:23:40 elodilles just to clean up a bit
08:40:05 lyarwood ganso: ah apologies I see now, the nova-next failure in your stable/ussuri change looks like a known issue with the bionic version of QEMU
08:40:59 lyarwood ganso: we used the zuulv3 change to the live migration jobs to fix this in later releases by moving to Focal but that isn't an option on Ussuri, Train etc.
08:41:20 lyarwood ganso: so tl;dr the zuulv3 changes might not resolve this issue
08:41:42 lyarwood ganso: however we could skip the test if it's failing constantly across the remaining Bionic based branches
08:42:35 lyarwood elodilles: as above I'm not sure that we do need that back on stable/ussuri but at the same time it shouldn't hurt
08:43:18 lyarwood elodilles: the label is defined in the parent tempest job and everything in our job definition should be generic (with the edits I made to the evacuation test role) to run on Bionic
08:46:34 elodilles lyarwood: ok, thanks, meanwhile I came to the same conclusion that it probably doesn't hurt but on the other hand fixes the gate, so i +w'd the patch :)
08:47:04 lyarwood that's the thing, I've not seen an example where it clearly does just yet but hopefully I've missed something
08:48:00 lyarwood the paused instance live migration failure is a known bionic issue and something that we saw back during Victoria that prompted us to move the jobs to Focal as part of the zuulv3 migration
09:38:38 opendevreview Lee Yarwood proposed openstack/nova master: compute: Avoid calling detach with src connection_info during LM rollback https://review.opendev.org/c/openstack/nova/+/797725
09:42:19 opendevreview Lee Yarwood proposed openstack/nova master: compute: Avoid calling detach with src connection_info during LM rollback https://review.opendev.org/c/openstack/nova/+/797725
10:14:21 opendevreview Lee Yarwood proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519
10:20:20 opendevreview Lee Yarwood proposed openstack/nova master: libvirt: Create qcow2 disks with the correct size without extending https://review.opendev.org/c/openstack/nova/+/779275
11:24:58 opendevreview Stephen Finucane proposed openstack/nova stable/ussuri: Test numa and vcpu topologies bug: #1910466 https://review.opendev.org/c/openstack/nova/+/797878
11:24:59 opendevreview Stephen Finucane proposed openstack/nova stable/ussuri: Fix max cpu topologies with numa affinity https://review.opendev.org/c/openstack/nova/+/797879
11:35:20 opendevreview Stephen Finucane proposed openstack/nova stable/train: Test numa and vcpu topologies bug: #1910466 https://review.opendev.org/c/openstack/nova/+/797880
11:35:21 opendevreview Stephen Finucane proposed openstack/nova stable/train: Fix max cpu topologies with numa affinity https://review.opendev.org/c/openstack/nova/+/797881

Earlier   Later