| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-08 | |||
| 23:24:41 | gmann | +1, thanks | |
| 23:41:35 | clarkb | nova should consider removing nova-live-migration-ceph from the gate queue as the job is non voting | |
| 23:54:05 | opendevreview | Merged openstack/nova master: Add autopep8 to tox and pre-commit https://review.opendev.org/c/openstack/nova/+/806182 | |
| #openstack-nova - 2021-11-09 | |||
| 04:36:53 | opendevreview | Merged openstack/nova master: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/813263 | |
| 08:02:51 | gibi | lyarwood: I saw elodilles explained the setuptools pin question. thanks elodilles | |
| 08:10:09 | bauzas | hola folks | |
| 08:10:42 | bauzas | gibi: I'm asked to present some PTG updates in a company session today at the same time of the upstream meeting | |
| 08:10:58 | gibi | bauzas: o/ | |
| 08:10:59 | bauzas | gibi: it would be a 2 min presentation about Nova | |
| 08:11:18 | bauzas | gibi: could you help me by chairing the meeting when I'm asked to discuss ? | |
| 08:11:27 | gibi | bauzas: sure | |
| 08:11:37 | bauzas | I could run the meeting, then passing it to you for 5 mins | |
| 08:11:49 | bauzas | and then, either you continue or me :) | |
| 08:12:57 | gibi | ok | |
| 08:13:11 | gibi | I will handle it when you need to switch | |
| 08:18:01 | bauzas | gibi: thanks | |
| 08:18:10 | bauzas | appreciated | |
| 08:22:25 | gibi | no worries | |
| 08:54:13 | gibi | lyarwood, elodilles: I'm seeing multiple guest kernel panics in stable/victoria volume related tests | |
| 08:55:53 | gibi | 1) https://zuul.opendev.org/t/openstack/build/67c89daf17e3475cb1d632f87beeb60d/log/controller/logs/tempest_log.txt#5950 | |
| 08:56:28 | lyarwood | Just jumping on a call but I wonder if we were still using cirros 0.4.0 back then? | |
| 08:57:13 | lyarwood | /opt/stack/devstack/files/cirros-0.5.1-x86_64-disk.img | |
| 08:57:15 | lyarwood | maybe not | |
| 08:57:32 | lyarwood | however I also see image.http_image = http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz | |
| 08:57:32 | gibi | 2) https://1a59031cf12ee85b5b8a-5c947c8d22eb7769ff9d2de46bec4cc9.ssl.cf5.rackcdn.com/810915/2/gate/nova-grenade-multinode/ebc944c/testr_results.html | |
| 08:58:13 | lyarwood | anyway I'll take a look after this call | |
| 08:58:36 | gibi | ack, I see both 0.3.1 and 0.5.1 in the logs | |
| 09:00:31 | gibi | hm the issue in the grenade job uses a different cirros as it has kernel 4.4.0 while the failed test case in the live migration job has kernel 5.3.0 | |
| 09:01:52 | gibi | both kernel stack trace shows page fault but in different processes | |
| 09:11:33 | kashyap | gibi: Got a link to the traceback? | |
| 09:11:44 | gibi | kashyap: https://zuul.opendev.org/t/openstack/build/67c89daf17e3475cb1d632f87beeb60d/log/controller/logs/tempest_log.txt#5950 | |
| 09:11:48 | gibi | that is one | |
| 09:12:04 | kashyap | Yep, finally it loaded; thanks | |
| 09:12:24 | kashyap | So, the above trace is with kernel 4.4.0? (i.e. CirrOS 0.5.1?) | |
| 09:13:08 | gibi | this one is kernel 5.3 | |
| 09:13:24 | gibi | [ 15.489062] CPU: 0 PID: 284 Comm: ip Not tainted 5.3.0-26-generic #28~18.04.1-Ubuntu | |
| 09:14:07 | kashyap | Yes, just saw it. Silly me | |
| 09:14:08 | gibi | ... | |
| 09:14:08 | gibi | sorry wrong buffer | |
| 09:14:23 | gibi | [ 15.302770] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 5.3.0-26-generic #28~18.04.1-Ubuntu | |
| 09:14:31 | gibi | this one is from the stack trace you are looking at | |
| 09:16:53 | gibi | and that is matching with cirros 0.5.1 | |
| 09:17:02 | kashyap | Yeah, figured as much. The trace seems to go into kernel RCU (read-copy update) code in the kernel ... which I was told can be used to "frighten small children and adults alike" | |
| 09:17:23 | kashyap | s/in the kernel// | |
| 09:17:47 | gibi | :) | |
| 09:18:15 | kashyap | Hm, I wonder what changed suddenly in stable/victoria for us to hit these | |
| 09:18:45 | gibi | I can try to check how frequently we hit kernel panics in stable/victoria and when we get the increawse | |
| 09:26:17 | gibi | we don't have much logs going backward in time for nova-live-migration as it was turned off for a while on stable | |
| 09:27:21 | gibi | https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&branch=stable%2Fvictoria | |
| 09:27:57 | gibi | based on this it started failing yesterday | |
| 09:28:06 | gibi | but it is small sample | |
| 09:32:14 | gibi | it seems other branches (wallaby, xena, master) are not affected but only master has good amount of runs to be sure | |
| 09:33:41 | gibi | but master uses cirros 0.5.2 | |
| 09:35:38 | gibi | ohh both wallaby and xena also uses 0.5.2 | |
| 09:35:44 | gibi | maybe it is the cirros version | |
| 09:36:06 | gibi | I'm wondering where we define the cirros version | |
| 09:37:12 | gibi | ok, that is devstack | |
| 09:37:20 | gibi | the 0.5.2. bump was this patch https://review.opendev.org/c/openstack/devstack/+/779179 | |
| 09:37:21 | kashyap | Hmm | |
| 09:37:38 | kashyap | Okay, so failing since yesterday; and only affects stable/victoria | |
| 09:39:07 | gibi | stable/ussuri uses cirros 0.4.0 and it seems that is also not affected (still small sample) | |
| 09:39:07 | kashyap | (The bump was only this year - it shouldn't affect stable/victoria?) | |
| 09:39:48 | gibi | kashyap: the bump to 0.5.2 does not effect victora, that uses 0.5.1 still as devstack has stable branches too | |
| 09:40:06 | kashyap | Aaah, right | |
| 09:47:22 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: [stable-only]Bump cirros to 0.5.2 for live migration https://review.opendev.org/c/openstack/nova/+/817173 | |
| 09:47:44 | gibi | lyarwood, kashyap: that is my guess ^^ lets see what happens | |
| 09:47:59 | gibi | lyarwood: a totally different failure from stable/victoria https://zuul.opendev.org/t/openstack/build/3f48404a55904986b6f5bcd2ce7d1908/log/job-output.txt#2517 | |
| 09:48:06 | gibi | die 276 'Support for rhel8 is incomplete: no support for installing packages' | |
| 09:48:11 | lyarwood | hmm that only adds in the ahci module so I dunno maybe | |
| 09:48:15 | gibi | it is from tempest-integrated-compute-centos-8-stream | |
| 09:49:04 | lyarwood | oh I think that has never worked but I've added it in on master | |
| 09:49:11 | lyarwood | and because tempest is branchless | |
| 09:49:14 | lyarwood | fun times | |
| 09:49:22 | gibi | :) | |
| 09:49:36 | lyarwood | https://review.opendev.org/c/openstack/tempest/+/797614 was the change | |
| 09:49:44 | lyarwood | landed overnight | |
| 09:50:27 | lyarwood | so I need to add a branch conditional in there I guess | |
| 09:51:36 | gibi | for which branch? | |
| 09:52:20 | gibi | ahh I see it passed on master | |
| 09:52:54 | gibi | but now it fails on master too https://zuul.opendev.org/t/openstack/build/a637bf6e68c545e59c8d091393a2307e/log/job-output.txt but with a totally different issue | |
| 09:53:20 | gibi | with botocore version conflict | |
| 09:56:14 | lyarwood | hmmm https://review.opendev.org/c/openstack/devstack/+/688614 is in stable/victoria | |
| 09:57:00 | lyarwood | oh it's CentOSStream | |
| 09:59:08 | lyarwood | https://review.opendev.org/c/openstack/devstack/+/803023 was rejected so I need the branch conditional in master gah | |
| 10:00:51 | gibi | I've opened a gate bug for the tempest-integrated-compute-centos-8-stream job failing on master with version conflict as it seems to be 100% hit | |
| 10:00:54 | gibi | tempest-integrated-compute-centos-8-stream | |
| 10:01:19 | gibi | https://bugs.launchpad.net/nova/+bug/1950291 | |
| 10:03:48 | lyarwood | weird | |
| 10:04:12 | lyarwood | `Cannot install cinder because these package versions have conflicting dependencies.` FWIW | |
| 10:10:38 | gibi | interesing nothing recent in cinder bumped version | |
| 10:10:50 | gibi | and nothing in the requirements repo since the 6th | |
| 10:12:06 | gibi | this was on the 6th bumping boto https://review.opendev.org/c/openstack/requirements/+/816611/2/upper-constraints.txt#314 | |
| 10:12:30 | gibi | sorry 4th | |
| 10:15:11 | lyarwood | gibi: did you have a bug for the stable/victoria issue? | |
| 10:15:37 | gibi | nope | |
| 10:15:45 | lyarwood | ack | |
| 10:15:47 | gibi | please raise one | |
| 10:18:46 | kashyap | gibi: lyarwood: Catching up ... is it because of this not merging yet? https://review.opendev.org/c/openstack/devstack/+/803023 (fix is_fedora for centos 8 stream) | |
| 10:20:24 | lyarwood | yeah but as we didn't support centos8stream at that point we shouldn't backport this anyway | |
| 10:20:37 | lyarwood | I'll just land some regex shortly to fix this | |