| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-09 | |||
| 09:27:57 | gibi | based on this it started failing yesterday | |
| 09:28:06 | gibi | but it is small sample | |
| 09:32:14 | gibi | it seems other branches (wallaby, xena, master) are not affected but only master has good amount of runs to be sure | |
| 09:33:41 | gibi | but master uses cirros 0.5.2 | |
| 09:35:38 | gibi | ohh both wallaby and xena also uses 0.5.2 | |
| 09:35:44 | gibi | maybe it is the cirros version | |
| 09:36:06 | gibi | I'm wondering where we define the cirros version | |
| 09:37:12 | gibi | ok, that is devstack | |
| 09:37:20 | gibi | the 0.5.2. bump was this patch https://review.opendev.org/c/openstack/devstack/+/779179 | |
| 09:37:21 | kashyap | Hmm | |
| 09:37:38 | kashyap | Okay, so failing since yesterday; and only affects stable/victoria | |
| 09:39:07 | gibi | stable/ussuri uses cirros 0.4.0 and it seems that is also not affected (still small sample) | |
| 09:39:07 | kashyap | (The bump was only this year - it shouldn't affect stable/victoria?) | |
| 09:39:48 | gibi | kashyap: the bump to 0.5.2 does not effect victora, that uses 0.5.1 still as devstack has stable branches too | |
| 09:40:06 | kashyap | Aaah, right | |
| 09:47:22 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: [stable-only]Bump cirros to 0.5.2 for live migration https://review.opendev.org/c/openstack/nova/+/817173 | |
| 09:47:44 | gibi | lyarwood, kashyap: that is my guess ^^ lets see what happens | |
| 09:47:59 | gibi | lyarwood: a totally different failure from stable/victoria https://zuul.opendev.org/t/openstack/build/3f48404a55904986b6f5bcd2ce7d1908/log/job-output.txt#2517 | |
| 09:48:06 | gibi | die 276 'Support for rhel8 is incomplete: no support for installing packages' | |
| 09:48:11 | lyarwood | hmm that only adds in the ahci module so I dunno maybe | |
| 09:48:15 | gibi | it is from tempest-integrated-compute-centos-8-stream | |
| 09:49:04 | lyarwood | oh I think that has never worked but I've added it in on master | |
| 09:49:11 | lyarwood | and because tempest is branchless | |
| 09:49:14 | lyarwood | fun times | |
| 09:49:22 | gibi | :) | |
| 09:49:36 | lyarwood | https://review.opendev.org/c/openstack/tempest/+/797614 was the change | |
| 09:49:44 | lyarwood | landed overnight | |
| 09:50:27 | lyarwood | so I need to add a branch conditional in there I guess | |
| 09:51:36 | gibi | for which branch? | |
| 09:52:20 | gibi | ahh I see it passed on master | |
| 09:52:54 | gibi | but now it fails on master too https://zuul.opendev.org/t/openstack/build/a637bf6e68c545e59c8d091393a2307e/log/job-output.txt but with a totally different issue | |
| 09:53:20 | gibi | with botocore version conflict | |
| 09:56:14 | lyarwood | hmmm https://review.opendev.org/c/openstack/devstack/+/688614 is in stable/victoria | |
| 09:57:00 | lyarwood | oh it's CentOSStream | |
| 09:59:08 | lyarwood | https://review.opendev.org/c/openstack/devstack/+/803023 was rejected so I need the branch conditional in master gah | |
| 10:00:51 | gibi | I've opened a gate bug for the tempest-integrated-compute-centos-8-stream job failing on master with version conflict as it seems to be 100% hit | |
| 10:00:54 | gibi | tempest-integrated-compute-centos-8-stream | |
| 10:01:19 | gibi | https://bugs.launchpad.net/nova/+bug/1950291 | |
| 10:03:48 | lyarwood | weird | |
| 10:04:12 | lyarwood | `Cannot install cinder because these package versions have conflicting dependencies.` FWIW | |
| 10:10:38 | gibi | interesing nothing recent in cinder bumped version | |
| 10:10:50 | gibi | and nothing in the requirements repo since the 6th | |
| 10:12:06 | gibi | this was on the 6th bumping boto https://review.opendev.org/c/openstack/requirements/+/816611/2/upper-constraints.txt#314 | |
| 10:12:30 | gibi | sorry 4th | |
| 10:15:11 | lyarwood | gibi: did you have a bug for the stable/victoria issue? | |
| 10:15:37 | gibi | nope | |
| 10:15:45 | lyarwood | ack | |
| 10:15:47 | gibi | please raise one | |
| 10:18:46 | kashyap | gibi: lyarwood: Catching up ... is it because of this not merging yet? https://review.opendev.org/c/openstack/devstack/+/803023 (fix is_fedora for centos 8 stream) | |
| 10:20:24 | lyarwood | yeah but as we didn't support centos8stream at that point we shouldn't backport this anyway | |
| 10:20:37 | lyarwood | I'll just land some regex shortly to fix this | |
| 10:21:12 | opendevreview | Jun Chen proposed openstack/nova master: Catch an exception in power off procedure https://review.opendev.org/c/openstack/nova/+/817176 | |
| 10:23:08 | kashyap | lyarwood: Ah, noted. Sorry, regex based on what? To selectively check if 8stream is available, if not fallback to vanilla CentoS? | |
| 10:24:44 | lyarwood | kashyap: regex to stop the centos8stream job from running on branches older than wallaby | |
| 10:24:50 | lyarwood | https://review.opendev.org/c/openstack/tempest/+/817179 | |
| 10:25:23 | kashyap | Aah, like that; thx! | |
| 10:25:53 | opendevreview | Lee Yarwood proposed openstack/nova stable/victoria: DNM - Test integrated-gate-compute fix for centos8stream https://review.opendev.org/c/openstack/nova/+/817180 | |
| 10:25:54 | lyarwood | ^ testing here | |
| 10:28:46 | gibi | I cannot reproduce the version conflict locally seen in https://bugs.launchpad.net/nova/+bug/1950291 | |
| 10:30:28 | gibi | but I don't have python3.6 :/ | |
| 10:35:07 | frickler | gibi: that looks like a failure in the index from pypi CDN, there is no conflict if you look at the version numbers. pip just fails to generate a proper error when it can't find that specific version in the index | |
| 10:35:40 | gibi | frickler: ohh, good point, then I guess the error will go away after the recheck | |
| 10:35:45 | bauzas | gibi: should we mark the bug Critical as it holds the gate ? | |
| 10:35:59 | bauzas | https://bugs.launchpad.net/nova/+bug/1950291 | |
| 10:36:08 | gibi | bauzas: wait a bit, frickler has an explanation that might mean it was a transient only | |
| 10:36:13 | bauzas | ok | |
| 10:36:15 | gibi | I have recheck running now | |
| 10:36:22 | bauzas | that's what I see | |
| 10:36:53 | frickler | we have some way of telling the CDN to refresh its cache, I can look that up in a bit | |
| 10:41:28 | gibi | still failing with boto conflict after recheck https://zuul.opendev.org/t/openstack/build/0ba1dd59972d48a98fe29b47dbb82e1e/log/job-output.txt | |
| 10:42:13 | bauzas | gibi: marking it Critical until we figure out a better vision | |
| 10:43:09 | bauzas | gibi: just to be clear, this is an unrelated issue from the stable/victoria gate, right? | |
| 10:43:22 | bauzas | here, we have centos-stream on wallaby and later | |
| 10:43:43 | gibi | bauzas: right | |
| 10:44:11 | bauzas | and I see lyarwood providing a tempest fix for the stable branches that are impacted | |
| 10:44:42 | frickler | bauzas: gibi: I did "curl -XPURGE https://pypi.org/simple/botocore" and the same without the "/botocore". please try another recheck | |
| 10:45:10 | gibi | frickler: ack I will | |
| 10:45:13 | gibi | and thanks | |
| 10:45:34 | bauzas | but I'll scroll when I'm back | |
| 10:46:04 | frickler | if it is still failing for jobs starting now, please ping infra-root in #opendev, I'll be afk for a bit | |
| 10:46:18 | kevko | sean-k-mooney: hi, here ? :) | |
| 11:03:27 | gibi | frickler: ack thanks | |
| 11:03:52 | opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: Create qcow2 disks with the correct size without extending https://review.opendev.org/c/openstack/nova/+/779275 | |
| 11:10:20 | gibi | lyarwood: fyi cirros 0.5.2 is not a solution for the kernel panic as https://review.opendev.org/c/openstack/nova/+/817173 still triggers it | |
| 11:24:17 | lyarwood | gibi: Yeah I didn't think it would tbh | |
| 11:24:25 | gibi | so back to square one | |
| 12:00:43 | gibi | lyarwood: should I file a bug for the kernel panic problem on stable/victoria? | |
| 12:00:52 | gibi | or you already did? | |
| 12:00:59 | lyarwood | I haven't so go ahead | |
| 12:01:07 | gibi | ok | |
| 12:01:09 | gibi | I will do | |
| 12:09:35 | gibi | lyarwood: https://bugs.launchpad.net/nova/+bug/1950310 | |
| 12:21:24 | gibi | lyarwood: could this be the appearence of our old volume detach bug ^^ where the fix was the redesigned detach code in https://review.opendev.org/q/topic:bug/1882521 | |
| 12:21:35 | gibi | that was backported only to wallaby | |
| 12:22:21 | gibi | and I do see in the nova log that the _do_wait_and_retry_detach function goes through the 7 iteration | |
| 13:21:11 | opendevreview | Merged openstack/nova master: Remove SESSION_CONFIGURED global from DB fixture https://review.opendev.org/c/openstack/nova/+/815689 | |
| 13:26:41 | gibi | frickler: seems your PURGE command helped later runs does not hit the boto version conflict | |
| 13:27:21 | frickler | gibi: great, thanks for confirming | |
| 13:51:08 | gibi | lyarwood: I backported the libvirt event based detach series to stable/victoria let's see if that helps with the kernel panic | |
| 13:51:09 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: libvirt: Define and emit DeviceRemovedEvent and DeviceRemovalFailedEvent https://review.opendev.org/c/openstack/nova/+/817209 | |