| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-15 | |||
| 13:01:33 | opendevreview | Amit Uniyal proposed openstack/nova master: Allow swap resize from non-zero to zero https://review.opendev.org/c/openstack/nova/+/857339 | |
| 13:01:34 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: temp commit for tests https://review.opendev.org/c/openstack/nova/+/905597 | |
| 13:25:35 | bauzas | sean-k-mooney: nope, sorry but you're wrong, the test works correctly | |
| 13:26:23 | bauzas | I'm running stestr with --until-failure from 2 hours for all functional tests with mdevs, np for the moment | |
| 13:27:04 | bauzas | stestr --test-path=./nova/tests/functional run nova.tests.functional.libvirt.test_vgpu --until-failure | |
| 13:27:13 | bauzas | Ran: 3924 tests in 8192.6080 sec. | |
| 13:27:13 | bauzas | ====== | |
| 13:27:13 | bauzas | Totals | |
| 13:27:15 | bauzas | - Passed: 3924 | |
| 13:27:17 | bauzas | - Skipped: 0 | |
| 13:27:19 | bauzas | - Expected Fail: 0 | |
| 13:27:21 | bauzas | - Unexpected Success: 0 | |
| 13:27:23 | bauzas | - Failed: 0 | |
| 13:27:25 | bauzas | Sum of execute time for each test: 35255.4411 sec. | |
| 16:49:46 | kashyap | zigo: Or anyone: Do you know if #ubuntu-kernel on Libera is still the place to talk to Ubuntu kernel maintainers? | |
| 16:50:44 | clarkb | kashyap: the ubuntu wiki says that is the correct location and the ubuntu security team is in a similarly named channel on libera | |
| 16:58:11 | kashyap | clarkb: Thank you! Mid last year I had chat with 'juergh' about a certain kernel panic. I was hoping to catch them again, but the channel right now has about 4 folks. | |
| 17:04:37 | sean-k-mooney | http://tinyurl.com/bdfk6j9e | |
| 17:04:38 | kashyap | melwitt: dansmith: Hi, so even with CirrOS 0.5.3 (with the updated stable kernel), we're seeing these sort of failures, yeah? - https://bugs.launchpad.net/nova/+bug/2018612 | |
| 17:05:03 | sean-k-mooney | so for context we have had 226 kernel panics in the last 10 ish days | |
| 17:05:05 | kashyap | (Meta note: If so, we (I can do) have to file a separate issue as per the above) | |
| 17:05:23 | sean-k-mooney | many have been in the cinder project jobs | |
| 17:05:27 | kashyap | sean-k-mooney: Yeah, I was just about to run a query in OpenSearch too; thanks for the link | |
| 17:05:39 | melwitt | kashyap: we're using cirros 0.6.2 and 0.6.1 | |
| 17:05:43 | melwitt | orly? ok, that's good to know | |
| 17:06:10 | kashyap | melwitt: Ah, thanks! I'm running on an outdated brain model | |
| 17:06:19 | melwitt | :) | |
| 17:06:29 | sean-k-mooney | they are in neutron, glance, cinder,nova | |
| 17:06:55 | dansmith | the first one i see in that stack from sean-k-mooney is not like the one from the bug kashyap | |
| 17:07:21 | dansmith | it's failing to load userspace libs, which is what sean-k-mooney was mentioning, but is different from the page fault in the bug.. similar time in boot, but different behaivor | |
| 17:07:28 | sean-k-mooney | " Kernel panic - not syncing: IO-APIC + timer " | |
| 17:07:34 | sean-k-mooney | that is form using cirros 5.x | |
| 17:07:52 | sean-k-mooney | its a fixed kernel bug that was just not in the cirros images so we can ignore those | |
| 17:09:12 | kashyap | dansmith: Yeah, the one I just searched for is this (also in melwitt's CI list): | |
| 17:09:15 | kashyap | "Kernel panic - not syncing: Attempted to kill init! exitcode=0x00001000" | |
| 17:09:27 | sean-k-mooney | right the "/sbin/init: can't load library 'libtirpc.so.3'" is the main painic we still see after the recent changes we made | |
| 17:09:37 | kashyap | The above panic has 36 global hits (12 Tempest) in the last 7 days. Nova and Cinder projects affected | |
| 17:10:07 | dansmith | sean-k-mooney: meaning we see that even with the split boot right? | |
| 17:10:18 | dansmith | I hadn't seen that manifestation until just now | |
| 17:10:30 | melwitt | I guess that's "good" that we can still get some samples for showing kernel devs if we can get some help | |
| 17:10:38 | dansmith | or hadn't noticed. I've mostly been in "see a panic, recheck" mode for a while now | |
| 17:10:59 | kashyap | sean-k-mooney: IIUC, "5.x" needs to 0.5.3 or higher | |
| 17:11:16 | sean-k-mooney | dansmith: yep but form what i have seen its mostly (perhaps always) been a boot form volume test | |
| 17:11:48 | sean-k-mooney | kashyap: i dont think the io api issues was ever fixed in 0.5.3 | |
| 17:12:00 | sean-k-mooney | i tried to get it rebuilt but it didnt happen in the past | |
| 17:12:26 | sean-k-mooney | if 0.5.3 happned in the last year then it might have the updated kernel | |
| 17:12:38 | sean-k-mooney | if not then it is still using the old ubuntu package | |
| 17:13:17 | dansmith | sean-k-mooney: ack, but it seems like it's still in initramfs at this point so still seems like bfv "shouldn't matter" | |
| 17:13:45 | kashyap | By the "io api issues" you mean the IO-APIC related patch "no_timer_check" that was at some pointed added to a CirrOS image? | |
| 17:14:11 | sean-k-mooney | kashyap: no | |
| 17:14:41 | sean-k-mooney | the reason this happens in 5.2 for example is ubuntu had already fixed the kernel bug but cirros was not rebuilt with the fixed kernel package | |
| 17:14:42 | kashyap | sean-k-mooney: CirrOS 0.5.3 happened in May 2023 (after the bug Dan filed): https://github.com/cirros-dev/cirros/releases/tag/0.5.3 | |
| 17:14:44 | sean-k-mooney | eventhough it was released | |
| 17:15:20 | sean-k-mooney | ok so it did hapen in the last year | |
| 17:16:03 | kashyap | Yep; I filed this request after Dan's bug above: https://github.com/cirros-dev/cirros/issues/102 | |
| 17:16:11 | sean-k-mooney | i asked to do this 2-3 years ago and they said no at the time | |
| 17:16:50 | kashyap | Maybe because at that time the older kernel was still "in support" | |
| 17:17:09 | sean-k-mooney | in support didnt matter | |
| 17:17:19 | sean-k-mooney | it was pinned ot a specific deb package | |
| 17:17:26 | sean-k-mooney | so it was reciving no backports | |
| 17:17:47 | sean-k-mooney | it was using 5.3.0-26.28~18.04.1 explcitly | |
| 17:18:48 | sean-k-mooney | anyay ya if we are using cirros 5.2 anywhere they shoudl be updated to 5.3 | |
| 17:20:14 | sean-k-mooney | https://codesearch.opendev.org/?q=cirros-0.5.2&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 17:20:27 | sean-k-mooney | so unfortunetly we are usin 0.5.2 in some places | |
| 17:20:47 | sean-k-mooney | obviouly on master we shoudl be using 6.x | |
| 17:21:17 | sean-k-mooney | but we likely shoudl fix that if nova is pinend and inform others to update there repos to avoid the io-apic issue | |
| 17:21:54 | sean-k-mooney | the cinder-tempest-plugin-lvm-lio-barbican-centos-9-stream is using 0.5.2 for example | |
| 17:22:06 | sean-k-mooney | or at leat havign the io-apic panic | |
| 17:22:38 | sean-k-mooney | nova is using it ofr the arm job | |
| 17:23:00 | sean-k-mooney | thats the only usage of 5.2 that affects us directly | |
| 17:23:18 | sean-k-mooney | im not sure the kernel bug exists in the arm builds | |
| 17:27:09 | kashyap | melwitt: sean-k-mooney: I'll start a "kernel panics" only Etherpad to track some of these in one place | |
| 17:27:38 | melwitt | thanks kashyap | |
| 17:29:24 | sean-k-mooney | im in the middle of something else but dansmith melwitt do you want me to select one of the jobs and go back to the normal image and larger nodeset lable | |
| 17:29:38 | sean-k-mooney | or will ye have time to do that | |
| 17:30:03 | dansmith | not just larger nodeset, but also with a larger flavor | |
| 17:30:14 | sean-k-mooney | melwitt: also i had hopped to start reviewing your serise today but its looking like it will we wednessday | |
| 17:30:29 | sean-k-mooney | dansmith: ya that what i ment but didnt say | |
| 17:30:40 | dansmith | I'm actually supposed to be off doing something else right now too, but I can work on that later if you never get to it | |
| 17:31:02 | dansmith | (later this week) | |
| 17:31:08 | sean-k-mooney | ack ill see if i have time before i finish for today | |
| 17:31:21 | sean-k-mooney | to that end ill go back to my ansibel patch | |
| 17:33:48 | kashyap | sean-k-mooney: melwitt: A quick start here: https://etherpad.opendev.org/p/Kernel-panics-in-Nova-CI | |
| 17:44:43 | gibi | why a privsep decorated function like https://github.com/openstack/nova/blob/master/nova/virt/libvirt/cpu/core.py#L63-L66 does not get escalated privileges? I see https://paste.opendev.org/show/bFZ9WgE2828iGamQoDaB/ and sure /sys/devices/system/cpu/cpu1/online is only writtable by root but I would assume that is why we have the privesep decorator on the call | |
| 17:46:37 | sean-k-mooney | it shold be elevated | |
| 17:46:59 | sean-k-mooney | that is a file not found issue | |
| 17:47:28 | gibi | -rw-r--r--. 1 root root 4.0K Jan 15 14:38 /sys/devices/system/cpu/cpu1/online | |
| 17:47:28 | gibi | bash-5.1$ ls -alh /sys/devices/system/cpu/cpu1/online | |
| 17:47:28 | gibi | [root@edpm-compute-1 ~]# podman exec -it nova_compute /bin/bash | |
| 17:47:28 | gibi | no the file is visible | |
| 17:47:43 | sean-k-mooney | can you write ot it with sudo ? | |
| 17:47:59 | sean-k-mooney | im wondering if its selinux | |
| 17:48:12 | sean-k-mooney | im guesing your on centso becasue fo py 3.9 | |
| 17:48:33 | gibi | yes I can | |
| 17:48:34 | gibi | [root@edpm-compute-1 ~]# podman exec -it -u root nova_compute /bin/bash | |
| 17:48:35 | gibi | [root@edpm-compute-1 /]# echo "online" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:38 | gibi | [root@edpm-compute-1 /]# echo "offline" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:41 | gibi | [root@edpm-compute-1 /]# cat /sys/devices/system/cpu/cpu1/online | |
| 17:48:43 | gibi | 0 | |
| 17:49:03 | sean-k-mooney | odd | |