| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-15 | |||
| 17:13:17 | dansmith | sean-k-mooney: ack, but it seems like it's still in initramfs at this point so still seems like bfv "shouldn't matter" | |
| 17:13:45 | kashyap | By the "io api issues" you mean the IO-APIC related patch "no_timer_check" that was at some pointed added to a CirrOS image? | |
| 17:14:11 | sean-k-mooney | kashyap: no | |
| 17:14:41 | sean-k-mooney | the reason this happens in 5.2 for example is ubuntu had already fixed the kernel bug but cirros was not rebuilt with the fixed kernel package | |
| 17:14:42 | kashyap | sean-k-mooney: CirrOS 0.5.3 happened in May 2023 (after the bug Dan filed): https://github.com/cirros-dev/cirros/releases/tag/0.5.3 | |
| 17:14:44 | sean-k-mooney | eventhough it was released | |
| 17:15:20 | sean-k-mooney | ok so it did hapen in the last year | |
| 17:16:03 | kashyap | Yep; I filed this request after Dan's bug above: https://github.com/cirros-dev/cirros/issues/102 | |
| 17:16:11 | sean-k-mooney | i asked to do this 2-3 years ago and they said no at the time | |
| 17:16:50 | kashyap | Maybe because at that time the older kernel was still "in support" | |
| 17:17:09 | sean-k-mooney | in support didnt matter | |
| 17:17:19 | sean-k-mooney | it was pinned ot a specific deb package | |
| 17:17:26 | sean-k-mooney | so it was reciving no backports | |
| 17:17:47 | sean-k-mooney | it was using 5.3.0-26.28~18.04.1 explcitly | |
| 17:18:48 | sean-k-mooney | anyay ya if we are using cirros 5.2 anywhere they shoudl be updated to 5.3 | |
| 17:20:14 | sean-k-mooney | https://codesearch.opendev.org/?q=cirros-0.5.2&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 17:20:27 | sean-k-mooney | so unfortunetly we are usin 0.5.2 in some places | |
| 17:20:47 | sean-k-mooney | obviouly on master we shoudl be using 6.x | |
| 17:21:17 | sean-k-mooney | but we likely shoudl fix that if nova is pinend and inform others to update there repos to avoid the io-apic issue | |
| 17:21:54 | sean-k-mooney | the cinder-tempest-plugin-lvm-lio-barbican-centos-9-stream is using 0.5.2 for example | |
| 17:22:06 | sean-k-mooney | or at leat havign the io-apic panic | |
| 17:22:38 | sean-k-mooney | nova is using it ofr the arm job | |
| 17:23:00 | sean-k-mooney | thats the only usage of 5.2 that affects us directly | |
| 17:23:18 | sean-k-mooney | im not sure the kernel bug exists in the arm builds | |
| 17:27:09 | kashyap | melwitt: sean-k-mooney: I'll start a "kernel panics" only Etherpad to track some of these in one place | |
| 17:27:38 | melwitt | thanks kashyap | |
| 17:29:24 | sean-k-mooney | im in the middle of something else but dansmith melwitt do you want me to select one of the jobs and go back to the normal image and larger nodeset lable | |
| 17:29:38 | sean-k-mooney | or will ye have time to do that | |
| 17:30:03 | dansmith | not just larger nodeset, but also with a larger flavor | |
| 17:30:14 | sean-k-mooney | melwitt: also i had hopped to start reviewing your serise today but its looking like it will we wednessday | |
| 17:30:29 | sean-k-mooney | dansmith: ya that what i ment but didnt say | |
| 17:30:40 | dansmith | I'm actually supposed to be off doing something else right now too, but I can work on that later if you never get to it | |
| 17:31:02 | dansmith | (later this week) | |
| 17:31:08 | sean-k-mooney | ack ill see if i have time before i finish for today | |
| 17:31:21 | sean-k-mooney | to that end ill go back to my ansibel patch | |
| 17:33:48 | kashyap | sean-k-mooney: melwitt: A quick start here: https://etherpad.opendev.org/p/Kernel-panics-in-Nova-CI | |
| 17:44:43 | gibi | why a privsep decorated function like https://github.com/openstack/nova/blob/master/nova/virt/libvirt/cpu/core.py#L63-L66 does not get escalated privileges? I see https://paste.opendev.org/show/bFZ9WgE2828iGamQoDaB/ and sure /sys/devices/system/cpu/cpu1/online is only writtable by root but I would assume that is why we have the privesep decorator on the call | |
| 17:46:37 | sean-k-mooney | it shold be elevated | |
| 17:46:59 | sean-k-mooney | that is a file not found issue | |
| 17:47:28 | gibi | no the file is visible | |
| 17:47:28 | gibi | [root@edpm-compute-1 ~]# podman exec -it nova_compute /bin/bash | |
| 17:47:28 | gibi | bash-5.1$ ls -alh /sys/devices/system/cpu/cpu1/online | |
| 17:47:28 | gibi | -rw-r--r--. 1 root root 4.0K Jan 15 14:38 /sys/devices/system/cpu/cpu1/online | |
| 17:47:43 | sean-k-mooney | can you write ot it with sudo ? | |
| 17:47:59 | sean-k-mooney | im wondering if its selinux | |
| 17:48:12 | sean-k-mooney | im guesing your on centso becasue fo py 3.9 | |
| 17:48:33 | gibi | yes I can | |
| 17:48:34 | gibi | [root@edpm-compute-1 ~]# podman exec -it -u root nova_compute /bin/bash | |
| 17:48:35 | gibi | [root@edpm-compute-1 /]# echo "online" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:38 | gibi | [root@edpm-compute-1 /]# echo "offline" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:41 | gibi | [root@edpm-compute-1 /]# cat /sys/devices/system/cpu/cpu1/online | |
| 17:48:43 | gibi | 0 | |
| 17:49:03 | sean-k-mooney | odd | |
| 17:49:09 | sean-k-mooney | anythign in the audit log? | |
| 17:49:48 | sean-k-mooney | im wondering if hte permission issue is due to user/group, or due to selinux or something else | |
| 17:51:39 | gibi | no related log in audit.log | |
| 17:52:08 | gibi | wouldn't I should see the code that escalates priviledges in the stack trace? | |
| 17:52:22 | gibi | * shouldn't I see ... | |
| 17:52:36 | sean-k-mooney | well it should be runing via privsep | |
| 17:52:58 | sean-k-mooney | so you shoudl see some privsep debug logs | |
| 17:53:24 | sean-k-mooney | im not sure why this is saying ERROR oslo_service.service | |
| 17:54:02 | sean-k-mooney | thats the wrong module logger | |
| 17:54:13 | gibi | there is no debug log about privsep either | |
| 17:54:45 | gibi | but we have `oslo.privsep.daemon=INFO` in the config | |
| 17:55:03 | sean-k-mooney | you should see message form nova sayign the deamon was started | |
| 17:55:07 | sean-k-mooney | even at info | |
| 17:55:15 | sean-k-mooney | perhaps nto the detailed logs but you should see it start | |
| 18:01:45 | gibi | something is very strange | |
| 18:02:52 | gibi | at the first nova-compute startup the privsep daemon logs that it is started, but subsequent container restarts it does not log it | |
| 18:03:33 | gibi | I'm going to redeploy the computes to see if I can reproduce it | |
| 18:04:46 | sean-k-mooney | this is nova compute container | |
| 18:04:59 | sean-k-mooney | hum ok i woner if we are leakign the unix socket | |
| 18:05:11 | sean-k-mooney | or what we are using to detech the start between container starts | |
| 18:05:24 | sean-k-mooney | and then not recreating the privsep deamon | |
| 18:09:21 | gibi | I pushed revert https://github.com/openstack-k8s-operators/nova-operator/pull/650 to unblock Andrew and the folks testing cpu pinning | |
| 18:10:15 | sean-k-mooney | hehe ok im not sure htat is super relevent to upstream nova folks but perhaps we have a bug | |
| 18:10:30 | gibi | nah, that repo is also upstream :D | |
| 18:10:40 | sean-k-mooney | i supose | |
| 18:11:00 | gibi | spreading the word that a new deployer engine exists ;) | |
| 18:11:03 | sean-k-mooney | anyway i suspect we need to look into how privsep is being executed | |
| 18:11:37 | sean-k-mooney | i didnt look at that previosly becuase i was expecting it to run the same as triplo did | |
| 18:11:51 | sean-k-mooney | but perhasp something has changed there | |
| 18:12:16 | gibi | I will redeploy tomorrow and check for the privesp process before touching anything | |
| 18:12:32 | gibi | then I will try a container restart to see if privesp also restarted | |
| 18:13:03 | sean-k-mooney | so you might need to do an operation that needs privsep before its started | |
| 18:13:13 | sean-k-mooney | so you might need to boot a vm or something like that | |
| 18:13:18 | sean-k-mooney | i think its lazy loaded | |
| 18:13:33 | sean-k-mooney | we are not usign the fork modle i belive | |
| 18:13:51 | sean-k-mooney | there are 2 ways to run privsep but i think we are using the privsep helper way | |
| 18:13:58 | sean-k-mooney | vai sudo/rootwrap | |
| 18:14:14 | gibi | I will check tomorrow and report back | |
| 18:14:26 | sean-k-mooney | cool. are you almost done for today | |
| 18:14:49 | sean-k-mooney | im going to be aroudn until the top of the hour but i approved the revert quickly to unblock folks | |
| 18:15:03 | gibi | I'm done for today | |
| 18:15:08 | gibi | see you all tomorrow | |
| 18:15:11 | sean-k-mooney | o/ | |
| 19:07:49 | opendevreview | Merged openstack/nova stable/2023.2: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/897297 | |
| 19:31:22 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| 19:32:59 | sean-k-mooney | dansmith: it looks like the 16g nodepool lables are not aviable so i used the 32G ones ^ i can see about readding the 16G flavors but i just want to see if that is correct first. the vxhost flaovr still exists as we have 16G lables but with bionic | |
| 19:33:11 | sean-k-mooney | so we just need a 16G lable with jammy | |