| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-14 | |||
| 15:57:58 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Optionally delete mdev on instance cleanup https://review.opendev.org/c/openstack/nova/+/899407 | |
| #openstack-nova - 2024-01-15 | |||
| 07:46:29 | gibi | let me know if there are urgent+important things I should be aware of | |
| 08:17:28 | elodilles | gibi: welcome back o/ maybe a nice clean cherry pick for an easy start? :D o:) https://review.opendev.org/c/openstack/nova/+/904371 | |
| 08:24:10 | gibi | elodilles: thanks. Easy win indeed. Done | |
| 08:24:42 | elodilles | \o/ | |
| 08:24:48 | elodilles | thanks, too | |
| 08:28:48 | gibi | ping me when the next round of backport from that is up | |
| 08:31:16 | elodilles | sure thing :) | |
| 08:58:12 | opendevreview | Merged openstack/nova stable/2023.2: Reproduce bug #2025480 in a functional test https://review.opendev.org/c/openstack/nova/+/904370 | |
| 09:43:57 | bauzas | gibi: want some paperwork change ? sure https://review.opendev.org/c/openstack/nova-specs/+/905342 :p | |
| 09:45:27 | gibi | bauzas: sure. done. | |
| 09:45:33 | bauzas | <3 | |
| 09:58:15 | opendevreview | Merged openstack/nova-specs master: add the blueprint link to the specs https://review.opendev.org/c/openstack/nova-specs/+/905342 | |
| 10:24:07 | opendevreview | Merged openstack/nova stable/2023.2: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/904371 | |
| 10:45:44 | gibi | gate seems to be healthy on stable | |
| 10:48:31 | elodilles | gibi: yepp, it looks good \o/ and the 2023.2 version of the backport has merged ;) | |
| 10:49:59 | elodilles | so I've +2'd the patch on 2023.1 | |
| 11:07:01 | gibi | elodilles: lets see how the 2023.1 gate performs :) | |
| 11:09:00 | bauzas | fwiw, back in 2024 finding why we race for https://42ba7385992f85feadcc-7de3ea29d2988c6d3fb53b5f30580c1f.ssl.cf2.rackcdn.com/904177/5/check/nova-tox-functional-py38/b878af0/testr_results.html | |
| 11:09:25 | bauzas | 3 weeks after testing, I don't remember why | |
| 11:26:20 | sean-k-mooney[m] | bauzas: is it because you added the test before https://review.opendev.org/c/openstack/nova/+/904209/4 | |
| 11:30:21 | sean-k-mooney[m] | i also looks like whatever fixture your using to providie the mdevs likely is not taking effect | |
| 12:32:59 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 12:36:42 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 12:46:03 | opendevreview | Merged openstack/nova stable/2023.1: Reproduce bug #2025480 in a functional test https://review.opendev.org/c/openstack/nova/+/904372 | |
| 13:01:33 | opendevreview | Amit Uniyal proposed openstack/nova master: Allow swap resize from non-zero to zero https://review.opendev.org/c/openstack/nova/+/857339 | |
| 13:01:34 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: temp commit for tests https://review.opendev.org/c/openstack/nova/+/905597 | |
| 13:25:35 | bauzas | sean-k-mooney: nope, sorry but you're wrong, the test works correctly | |
| 13:26:23 | bauzas | I'm running stestr with --until-failure from 2 hours for all functional tests with mdevs, np for the moment | |
| 13:27:04 | bauzas | stestr --test-path=./nova/tests/functional run nova.tests.functional.libvirt.test_vgpu --until-failure | |
| 13:27:13 | bauzas | Ran: 3924 tests in 8192.6080 sec. | |
| 13:27:13 | bauzas | ====== | |
| 13:27:13 | bauzas | Totals | |
| 13:27:15 | bauzas | - Passed: 3924 | |
| 13:27:17 | bauzas | - Skipped: 0 | |
| 13:27:19 | bauzas | - Expected Fail: 0 | |
| 13:27:21 | bauzas | - Unexpected Success: 0 | |
| 13:27:23 | bauzas | - Failed: 0 | |
| 13:27:25 | bauzas | Sum of execute time for each test: 35255.4411 sec. | |
| 16:49:46 | kashyap | zigo: Or anyone: Do you know if #ubuntu-kernel on Libera is still the place to talk to Ubuntu kernel maintainers? | |
| 16:50:44 | clarkb | kashyap: the ubuntu wiki says that is the correct location and the ubuntu security team is in a similarly named channel on libera | |
| 16:58:11 | kashyap | clarkb: Thank you! Mid last year I had chat with 'juergh' about a certain kernel panic. I was hoping to catch them again, but the channel right now has about 4 folks. | |
| 17:04:37 | sean-k-mooney | http://tinyurl.com/bdfk6j9e | |
| 17:04:38 | kashyap | melwitt: dansmith: Hi, so even with CirrOS 0.5.3 (with the updated stable kernel), we're seeing these sort of failures, yeah? - https://bugs.launchpad.net/nova/+bug/2018612 | |
| 17:05:03 | sean-k-mooney | so for context we have had 226 kernel panics in the last 10 ish days | |
| 17:05:05 | kashyap | (Meta note: If so, we (I can do) have to file a separate issue as per the above) | |
| 17:05:23 | sean-k-mooney | many have been in the cinder project jobs | |
| 17:05:27 | kashyap | sean-k-mooney: Yeah, I was just about to run a query in OpenSearch too; thanks for the link | |
| 17:05:39 | melwitt | kashyap: we're using cirros 0.6.2 and 0.6.1 | |
| 17:05:43 | melwitt | orly? ok, that's good to know | |
| 17:06:10 | kashyap | melwitt: Ah, thanks! I'm running on an outdated brain model | |
| 17:06:19 | melwitt | :) | |
| 17:06:29 | sean-k-mooney | they are in neutron, glance, cinder,nova | |
| 17:06:55 | dansmith | the first one i see in that stack from sean-k-mooney is not like the one from the bug kashyap | |
| 17:07:21 | dansmith | it's failing to load userspace libs, which is what sean-k-mooney was mentioning, but is different from the page fault in the bug.. similar time in boot, but different behaivor | |
| 17:07:28 | sean-k-mooney | " Kernel panic - not syncing: IO-APIC + timer " | |
| 17:07:34 | sean-k-mooney | that is form using cirros 5.x | |
| 17:07:52 | sean-k-mooney | its a fixed kernel bug that was just not in the cirros images so we can ignore those | |
| 17:09:12 | kashyap | dansmith: Yeah, the one I just searched for is this (also in melwitt's CI list): | |
| 17:09:15 | kashyap | "Kernel panic - not syncing: Attempted to kill init! exitcode=0x00001000" | |
| 17:09:27 | sean-k-mooney | right the "/sbin/init: can't load library 'libtirpc.so.3'" is the main painic we still see after the recent changes we made | |
| 17:09:37 | kashyap | The above panic has 36 global hits (12 Tempest) in the last 7 days. Nova and Cinder projects affected | |
| 17:10:07 | dansmith | sean-k-mooney: meaning we see that even with the split boot right? | |
| 17:10:18 | dansmith | I hadn't seen that manifestation until just now | |
| 17:10:30 | melwitt | I guess that's "good" that we can still get some samples for showing kernel devs if we can get some help | |
| 17:10:38 | dansmith | or hadn't noticed. I've mostly been in "see a panic, recheck" mode for a while now | |
| 17:10:59 | kashyap | sean-k-mooney: IIUC, "5.x" needs to 0.5.3 or higher | |
| 17:11:16 | sean-k-mooney | dansmith: yep but form what i have seen its mostly (perhaps always) been a boot form volume test | |
| 17:11:48 | sean-k-mooney | kashyap: i dont think the io api issues was ever fixed in 0.5.3 | |
| 17:12:00 | sean-k-mooney | i tried to get it rebuilt but it didnt happen in the past | |
| 17:12:26 | sean-k-mooney | if 0.5.3 happned in the last year then it might have the updated kernel | |
| 17:12:38 | sean-k-mooney | if not then it is still using the old ubuntu package | |
| 17:13:17 | dansmith | sean-k-mooney: ack, but it seems like it's still in initramfs at this point so still seems like bfv "shouldn't matter" | |
| 17:13:45 | kashyap | By the "io api issues" you mean the IO-APIC related patch "no_timer_check" that was at some pointed added to a CirrOS image? | |
| 17:14:11 | sean-k-mooney | kashyap: no | |
| 17:14:41 | sean-k-mooney | the reason this happens in 5.2 for example is ubuntu had already fixed the kernel bug but cirros was not rebuilt with the fixed kernel package | |
| 17:14:42 | kashyap | sean-k-mooney: CirrOS 0.5.3 happened in May 2023 (after the bug Dan filed): https://github.com/cirros-dev/cirros/releases/tag/0.5.3 | |
| 17:14:44 | sean-k-mooney | eventhough it was released | |
| 17:15:20 | sean-k-mooney | ok so it did hapen in the last year | |
| 17:16:03 | kashyap | Yep; I filed this request after Dan's bug above: https://github.com/cirros-dev/cirros/issues/102 | |
| 17:16:11 | sean-k-mooney | i asked to do this 2-3 years ago and they said no at the time | |
| 17:16:50 | kashyap | Maybe because at that time the older kernel was still "in support" | |
| 17:17:09 | sean-k-mooney | in support didnt matter | |
| 17:17:19 | sean-k-mooney | it was pinned ot a specific deb package | |
| 17:17:26 | sean-k-mooney | so it was reciving no backports | |
| 17:17:47 | sean-k-mooney | it was using 5.3.0-26.28~18.04.1 explcitly | |
| 17:18:48 | sean-k-mooney | anyay ya if we are using cirros 5.2 anywhere they shoudl be updated to 5.3 | |
| 17:20:14 | sean-k-mooney | https://codesearch.opendev.org/?q=cirros-0.5.2&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 17:20:27 | sean-k-mooney | so unfortunetly we are usin 0.5.2 in some places | |
| 17:20:47 | sean-k-mooney | obviouly on master we shoudl be using 6.x | |
| 17:21:17 | sean-k-mooney | but we likely shoudl fix that if nova is pinend and inform others to update there repos to avoid the io-apic issue | |
| 17:21:54 | sean-k-mooney | the cinder-tempest-plugin-lvm-lio-barbican-centos-9-stream is using 0.5.2 for example | |
| 17:22:06 | sean-k-mooney | or at leat havign the io-apic panic | |
| 17:22:38 | sean-k-mooney | nova is using it ofr the arm job | |
| 17:23:00 | sean-k-mooney | thats the only usage of 5.2 that affects us directly | |
| 17:23:18 | sean-k-mooney | im not sure the kernel bug exists in the arm builds | |
| 17:27:09 | kashyap | melwitt: sean-k-mooney: I'll start a "kernel panics" only Etherpad to track some of these in one place | |
| 17:27:38 | melwitt | thanks kashyap | |
| 17:29:24 | sean-k-mooney | im in the middle of something else but dansmith melwitt do you want me to select one of the jobs and go back to the normal image and larger nodeset lable | |
| 17:29:38 | sean-k-mooney | or will ye have time to do that | |