| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-15 | |||
| 17:05:27 | kashyap | sean-k-mooney: Yeah, I was just about to run a query in OpenSearch too; thanks for the link | |
| 17:05:39 | melwitt | kashyap: we're using cirros 0.6.2 and 0.6.1 | |
| 17:05:43 | melwitt | orly? ok, that's good to know | |
| 17:06:10 | kashyap | melwitt: Ah, thanks! I'm running on an outdated brain model | |
| 17:06:19 | melwitt | :) | |
| 17:06:29 | sean-k-mooney | they are in neutron, glance, cinder,nova | |
| 17:06:55 | dansmith | the first one i see in that stack from sean-k-mooney is not like the one from the bug kashyap | |
| 17:07:21 | dansmith | it's failing to load userspace libs, which is what sean-k-mooney was mentioning, but is different from the page fault in the bug.. similar time in boot, but different behaivor | |
| 17:07:28 | sean-k-mooney | " Kernel panic - not syncing: IO-APIC + timer " | |
| 17:07:34 | sean-k-mooney | that is form using cirros 5.x | |
| 17:07:52 | sean-k-mooney | its a fixed kernel bug that was just not in the cirros images so we can ignore those | |
| 17:09:12 | kashyap | dansmith: Yeah, the one I just searched for is this (also in melwitt's CI list): | |
| 17:09:15 | kashyap | "Kernel panic - not syncing: Attempted to kill init! exitcode=0x00001000" | |
| 17:09:27 | sean-k-mooney | right the "/sbin/init: can't load library 'libtirpc.so.3'" is the main painic we still see after the recent changes we made | |
| 17:09:37 | kashyap | The above panic has 36 global hits (12 Tempest) in the last 7 days. Nova and Cinder projects affected | |
| 17:10:07 | dansmith | sean-k-mooney: meaning we see that even with the split boot right? | |
| 17:10:18 | dansmith | I hadn't seen that manifestation until just now | |
| 17:10:30 | melwitt | I guess that's "good" that we can still get some samples for showing kernel devs if we can get some help | |
| 17:10:38 | dansmith | or hadn't noticed. I've mostly been in "see a panic, recheck" mode for a while now | |
| 17:10:59 | kashyap | sean-k-mooney: IIUC, "5.x" needs to 0.5.3 or higher | |
| 17:11:16 | sean-k-mooney | dansmith: yep but form what i have seen its mostly (perhaps always) been a boot form volume test | |
| 17:11:48 | sean-k-mooney | kashyap: i dont think the io api issues was ever fixed in 0.5.3 | |
| 17:12:00 | sean-k-mooney | i tried to get it rebuilt but it didnt happen in the past | |
| 17:12:26 | sean-k-mooney | if 0.5.3 happned in the last year then it might have the updated kernel | |
| 17:12:38 | sean-k-mooney | if not then it is still using the old ubuntu package | |
| 17:13:17 | dansmith | sean-k-mooney: ack, but it seems like it's still in initramfs at this point so still seems like bfv "shouldn't matter" | |
| 17:13:45 | kashyap | By the "io api issues" you mean the IO-APIC related patch "no_timer_check" that was at some pointed added to a CirrOS image? | |
| 17:14:11 | sean-k-mooney | kashyap: no | |
| 17:14:41 | sean-k-mooney | the reason this happens in 5.2 for example is ubuntu had already fixed the kernel bug but cirros was not rebuilt with the fixed kernel package | |
| 17:14:42 | kashyap | sean-k-mooney: CirrOS 0.5.3 happened in May 2023 (after the bug Dan filed): https://github.com/cirros-dev/cirros/releases/tag/0.5.3 | |
| 17:14:44 | sean-k-mooney | eventhough it was released | |
| 17:15:20 | sean-k-mooney | ok so it did hapen in the last year | |
| 17:16:03 | kashyap | Yep; I filed this request after Dan's bug above: https://github.com/cirros-dev/cirros/issues/102 | |
| 17:16:11 | sean-k-mooney | i asked to do this 2-3 years ago and they said no at the time | |
| 17:16:50 | kashyap | Maybe because at that time the older kernel was still "in support" | |
| 17:17:09 | sean-k-mooney | in support didnt matter | |
| 17:17:19 | sean-k-mooney | it was pinned ot a specific deb package | |
| 17:17:26 | sean-k-mooney | so it was reciving no backports | |
| 17:17:47 | sean-k-mooney | it was using 5.3.0-26.28~18.04.1 explcitly | |
| 17:18:48 | sean-k-mooney | anyay ya if we are using cirros 5.2 anywhere they shoudl be updated to 5.3 | |
| 17:20:14 | sean-k-mooney | https://codesearch.opendev.org/?q=cirros-0.5.2&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 17:20:27 | sean-k-mooney | so unfortunetly we are usin 0.5.2 in some places | |
| 17:20:47 | sean-k-mooney | obviouly on master we shoudl be using 6.x | |
| 17:21:17 | sean-k-mooney | but we likely shoudl fix that if nova is pinend and inform others to update there repos to avoid the io-apic issue | |
| 17:21:54 | sean-k-mooney | the cinder-tempest-plugin-lvm-lio-barbican-centos-9-stream is using 0.5.2 for example | |
| 17:22:06 | sean-k-mooney | or at leat havign the io-apic panic | |
| 17:22:38 | sean-k-mooney | nova is using it ofr the arm job | |
| 17:23:00 | sean-k-mooney | thats the only usage of 5.2 that affects us directly | |
| 17:23:18 | sean-k-mooney | im not sure the kernel bug exists in the arm builds | |
| 17:27:09 | kashyap | melwitt: sean-k-mooney: I'll start a "kernel panics" only Etherpad to track some of these in one place | |
| 17:27:38 | melwitt | thanks kashyap | |
| 17:29:24 | sean-k-mooney | im in the middle of something else but dansmith melwitt do you want me to select one of the jobs and go back to the normal image and larger nodeset lable | |
| 17:29:38 | sean-k-mooney | or will ye have time to do that | |
| 17:30:03 | dansmith | not just larger nodeset, but also with a larger flavor | |
| 17:30:14 | sean-k-mooney | melwitt: also i had hopped to start reviewing your serise today but its looking like it will we wednessday | |
| 17:30:29 | sean-k-mooney | dansmith: ya that what i ment but didnt say | |
| 17:30:40 | dansmith | I'm actually supposed to be off doing something else right now too, but I can work on that later if you never get to it | |
| 17:31:02 | dansmith | (later this week) | |
| 17:31:08 | sean-k-mooney | ack ill see if i have time before i finish for today | |
| 17:31:21 | sean-k-mooney | to that end ill go back to my ansibel patch | |
| 17:33:48 | kashyap | sean-k-mooney: melwitt: A quick start here: https://etherpad.opendev.org/p/Kernel-panics-in-Nova-CI | |
| 17:44:43 | gibi | why a privsep decorated function like https://github.com/openstack/nova/blob/master/nova/virt/libvirt/cpu/core.py#L63-L66 does not get escalated privileges? I see https://paste.opendev.org/show/bFZ9WgE2828iGamQoDaB/ and sure /sys/devices/system/cpu/cpu1/online is only writtable by root but I would assume that is why we have the privesep decorator on the call | |
| 17:46:37 | sean-k-mooney | it shold be elevated | |
| 17:46:59 | sean-k-mooney | that is a file not found issue | |
| 17:47:28 | gibi | no the file is visible | |
| 17:47:28 | gibi | [root@edpm-compute-1 ~]# podman exec -it nova_compute /bin/bash | |
| 17:47:28 | gibi | bash-5.1$ ls -alh /sys/devices/system/cpu/cpu1/online | |
| 17:47:28 | gibi | -rw-r--r--. 1 root root 4.0K Jan 15 14:38 /sys/devices/system/cpu/cpu1/online | |
| 17:47:43 | sean-k-mooney | can you write ot it with sudo ? | |
| 17:47:59 | sean-k-mooney | im wondering if its selinux | |
| 17:48:12 | sean-k-mooney | im guesing your on centso becasue fo py 3.9 | |
| 17:48:33 | gibi | yes I can | |
| 17:48:34 | gibi | [root@edpm-compute-1 ~]# podman exec -it -u root nova_compute /bin/bash | |
| 17:48:35 | gibi | [root@edpm-compute-1 /]# echo "online" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:38 | gibi | [root@edpm-compute-1 /]# echo "offline" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:41 | gibi | [root@edpm-compute-1 /]# cat /sys/devices/system/cpu/cpu1/online | |
| 17:48:43 | gibi | 0 | |
| 17:49:03 | sean-k-mooney | odd | |
| 17:49:09 | sean-k-mooney | anythign in the audit log? | |
| 17:49:48 | sean-k-mooney | im wondering if hte permission issue is due to user/group, or due to selinux or something else | |
| 17:51:39 | gibi | no related log in audit.log | |
| 17:52:08 | gibi | wouldn't I should see the code that escalates priviledges in the stack trace? | |
| 17:52:22 | gibi | * shouldn't I see ... | |
| 17:52:36 | sean-k-mooney | well it should be runing via privsep | |
| 17:52:58 | sean-k-mooney | so you shoudl see some privsep debug logs | |
| 17:53:24 | sean-k-mooney | im not sure why this is saying ERROR oslo_service.service | |
| 17:54:02 | sean-k-mooney | thats the wrong module logger | |
| 17:54:13 | gibi | there is no debug log about privsep either | |
| 17:54:45 | gibi | but we have `oslo.privsep.daemon=INFO` in the config | |
| 17:55:03 | sean-k-mooney | you should see message form nova sayign the deamon was started | |
| 17:55:07 | sean-k-mooney | even at info | |
| 17:55:15 | sean-k-mooney | perhaps nto the detailed logs but you should see it start | |
| 18:01:45 | gibi | something is very strange | |
| 18:02:52 | gibi | at the first nova-compute startup the privsep daemon logs that it is started, but subsequent container restarts it does not log it | |
| 18:03:33 | gibi | I'm going to redeploy the computes to see if I can reproduce it | |
| 18:04:46 | sean-k-mooney | this is nova compute container | |
| 18:04:59 | sean-k-mooney | hum ok i woner if we are leakign the unix socket | |
| 18:05:11 | sean-k-mooney | or what we are using to detech the start between container starts | |
| 18:05:24 | sean-k-mooney | and then not recreating the privsep deamon | |
| 18:09:21 | gibi | I pushed revert https://github.com/openstack-k8s-operators/nova-operator/pull/650 to unblock Andrew and the folks testing cpu pinning | |