| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-09 | |||
| 08:18:01 | bauzas | gibi: thanks | |
| 08:18:10 | bauzas | appreciated | |
| 08:22:25 | gibi | no worries | |
| 08:54:13 | gibi | lyarwood, elodilles: I'm seeing multiple guest kernel panics in stable/victoria volume related tests | |
| 08:55:53 | gibi | 1) https://zuul.opendev.org/t/openstack/build/67c89daf17e3475cb1d632f87beeb60d/log/controller/logs/tempest_log.txt#5950 | |
| 08:56:28 | lyarwood | Just jumping on a call but I wonder if we were still using cirros 0.4.0 back then? | |
| 08:57:13 | lyarwood | /opt/stack/devstack/files/cirros-0.5.1-x86_64-disk.img | |
| 08:57:15 | lyarwood | maybe not | |
| 08:57:32 | gibi | 2) https://1a59031cf12ee85b5b8a-5c947c8d22eb7769ff9d2de46bec4cc9.ssl.cf5.rackcdn.com/810915/2/gate/nova-grenade-multinode/ebc944c/testr_results.html | |
| 08:57:32 | lyarwood | however I also see image.http_image = http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz | |
| 08:58:13 | lyarwood | anyway I'll take a look after this call | |
| 08:58:36 | gibi | ack, I see both 0.3.1 and 0.5.1 in the logs | |
| 09:00:31 | gibi | hm the issue in the grenade job uses a different cirros as it has kernel 4.4.0 while the failed test case in the live migration job has kernel 5.3.0 | |
| 09:01:52 | gibi | both kernel stack trace shows page fault but in different processes | |
| 09:11:33 | kashyap | gibi: Got a link to the traceback? | |
| 09:11:44 | gibi | kashyap: https://zuul.opendev.org/t/openstack/build/67c89daf17e3475cb1d632f87beeb60d/log/controller/logs/tempest_log.txt#5950 | |
| 09:11:48 | gibi | that is one | |
| 09:12:04 | kashyap | Yep, finally it loaded; thanks | |
| 09:12:24 | kashyap | So, the above trace is with kernel 4.4.0? (i.e. CirrOS 0.5.1?) | |
| 09:13:08 | gibi | this one is kernel 5.3 | |
| 09:13:24 | gibi | [ 15.489062] CPU: 0 PID: 284 Comm: ip Not tainted 5.3.0-26-generic #28~18.04.1-Ubuntu | |
| 09:14:07 | kashyap | Yes, just saw it. Silly me | |
| 09:14:08 | gibi | sorry wrong buffer | |
| 09:14:08 | gibi | ... | |
| 09:14:23 | gibi | [ 15.302770] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 5.3.0-26-generic #28~18.04.1-Ubuntu | |
| 09:14:31 | gibi | this one is from the stack trace you are looking at | |
| 09:16:53 | gibi | and that is matching with cirros 0.5.1 | |
| 09:17:02 | kashyap | Yeah, figured as much. The trace seems to go into kernel RCU (read-copy update) code in the kernel ... which I was told can be used to "frighten small children and adults alike" | |
| 09:17:23 | kashyap | s/in the kernel// | |
| 09:17:47 | gibi | :) | |
| 09:18:15 | kashyap | Hm, I wonder what changed suddenly in stable/victoria for us to hit these | |
| 09:18:45 | gibi | I can try to check how frequently we hit kernel panics in stable/victoria and when we get the increawse | |
| 09:26:17 | gibi | we don't have much logs going backward in time for nova-live-migration as it was turned off for a while on stable | |
| 09:27:21 | gibi | https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&branch=stable%2Fvictoria | |
| 09:27:57 | gibi | based on this it started failing yesterday | |
| 09:28:06 | gibi | but it is small sample | |
| 09:32:14 | gibi | it seems other branches (wallaby, xena, master) are not affected but only master has good amount of runs to be sure | |
| 09:33:41 | gibi | but master uses cirros 0.5.2 | |
| 09:35:38 | gibi | ohh both wallaby and xena also uses 0.5.2 | |
| 09:35:44 | gibi | maybe it is the cirros version | |
| 09:36:06 | gibi | I'm wondering where we define the cirros version | |
| 09:37:12 | gibi | ok, that is devstack | |
| 09:37:20 | gibi | the 0.5.2. bump was this patch https://review.opendev.org/c/openstack/devstack/+/779179 | |
| 09:37:21 | kashyap | Hmm | |
| 09:37:38 | kashyap | Okay, so failing since yesterday; and only affects stable/victoria | |
| 09:39:07 | kashyap | (The bump was only this year - it shouldn't affect stable/victoria?) | |
| 09:39:07 | gibi | stable/ussuri uses cirros 0.4.0 and it seems that is also not affected (still small sample) | |
| 09:39:48 | gibi | kashyap: the bump to 0.5.2 does not effect victora, that uses 0.5.1 still as devstack has stable branches too | |
| 09:40:06 | kashyap | Aaah, right | |
| 09:47:22 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: [stable-only]Bump cirros to 0.5.2 for live migration https://review.opendev.org/c/openstack/nova/+/817173 | |
| 09:47:44 | gibi | lyarwood, kashyap: that is my guess ^^ lets see what happens | |
| 09:47:59 | gibi | lyarwood: a totally different failure from stable/victoria https://zuul.opendev.org/t/openstack/build/3f48404a55904986b6f5bcd2ce7d1908/log/job-output.txt#2517 | |
| 09:48:06 | gibi | die 276 'Support for rhel8 is incomplete: no support for installing packages' | |
| 09:48:11 | lyarwood | hmm that only adds in the ahci module so I dunno maybe | |
| 09:48:15 | gibi | it is from tempest-integrated-compute-centos-8-stream | |
| 09:49:04 | lyarwood | oh I think that has never worked but I've added it in on master | |
| 09:49:11 | lyarwood | and because tempest is branchless | |
| 09:49:14 | lyarwood | fun times | |
| 09:49:22 | gibi | :) | |
| 09:49:36 | lyarwood | https://review.opendev.org/c/openstack/tempest/+/797614 was the change | |
| 09:49:44 | lyarwood | landed overnight | |
| 09:50:27 | lyarwood | so I need to add a branch conditional in there I guess | |
| 09:51:36 | gibi | for which branch? | |
| 09:52:20 | gibi | ahh I see it passed on master | |
| 09:52:54 | gibi | but now it fails on master too https://zuul.opendev.org/t/openstack/build/a637bf6e68c545e59c8d091393a2307e/log/job-output.txt but with a totally different issue | |
| 09:53:20 | gibi | with botocore version conflict | |
| 09:56:14 | lyarwood | hmmm https://review.opendev.org/c/openstack/devstack/+/688614 is in stable/victoria | |
| 09:57:00 | lyarwood | oh it's CentOSStream | |
| 09:59:08 | lyarwood | https://review.opendev.org/c/openstack/devstack/+/803023 was rejected so I need the branch conditional in master gah | |
| 10:00:51 | gibi | I've opened a gate bug for the tempest-integrated-compute-centos-8-stream job failing on master with version conflict as it seems to be 100% hit | |
| 10:00:54 | gibi | tempest-integrated-compute-centos-8-stream | |
| 10:01:19 | gibi | https://bugs.launchpad.net/nova/+bug/1950291 | |
| 10:03:48 | lyarwood | weird | |
| 10:04:12 | lyarwood | `Cannot install cinder because these package versions have conflicting dependencies.` FWIW | |
| 10:10:38 | gibi | interesing nothing recent in cinder bumped version | |
| 10:10:50 | gibi | and nothing in the requirements repo since the 6th | |
| 10:12:06 | gibi | this was on the 6th bumping boto https://review.opendev.org/c/openstack/requirements/+/816611/2/upper-constraints.txt#314 | |
| 10:12:30 | gibi | sorry 4th | |
| 10:15:11 | lyarwood | gibi: did you have a bug for the stable/victoria issue? | |
| 10:15:37 | gibi | nope | |
| 10:15:45 | lyarwood | ack | |
| 10:15:47 | gibi | please raise one | |
| 10:18:46 | kashyap | gibi: lyarwood: Catching up ... is it because of this not merging yet? https://review.opendev.org/c/openstack/devstack/+/803023 (fix is_fedora for centos 8 stream) | |
| 10:20:24 | lyarwood | yeah but as we didn't support centos8stream at that point we shouldn't backport this anyway | |
| 10:20:37 | lyarwood | I'll just land some regex shortly to fix this | |
| 10:21:12 | opendevreview | Jun Chen proposed openstack/nova master: Catch an exception in power off procedure https://review.opendev.org/c/openstack/nova/+/817176 | |
| 10:23:08 | kashyap | lyarwood: Ah, noted. Sorry, regex based on what? To selectively check if 8stream is available, if not fallback to vanilla CentoS? | |
| 10:24:44 | lyarwood | kashyap: regex to stop the centos8stream job from running on branches older than wallaby | |
| 10:24:50 | lyarwood | https://review.opendev.org/c/openstack/tempest/+/817179 | |
| 10:25:23 | kashyap | Aah, like that; thx! | |
| 10:25:53 | opendevreview | Lee Yarwood proposed openstack/nova stable/victoria: DNM - Test integrated-gate-compute fix for centos8stream https://review.opendev.org/c/openstack/nova/+/817180 | |
| 10:25:54 | lyarwood | ^ testing here | |
| 10:28:46 | gibi | I cannot reproduce the version conflict locally seen in https://bugs.launchpad.net/nova/+bug/1950291 | |
| 10:30:28 | gibi | but I don't have python3.6 :/ | |
| 10:35:07 | frickler | gibi: that looks like a failure in the index from pypi CDN, there is no conflict if you look at the version numbers. pip just fails to generate a proper error when it can't find that specific version in the index | |
| 10:35:40 | gibi | frickler: ohh, good point, then I guess the error will go away after the recheck | |
| 10:35:45 | bauzas | gibi: should we mark the bug Critical as it holds the gate ? | |
| 10:35:59 | bauzas | https://bugs.launchpad.net/nova/+bug/1950291 | |
| 10:36:08 | gibi | bauzas: wait a bit, frickler has an explanation that might mean it was a transient only | |
| 10:36:13 | bauzas | ok | |