| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-15 | |||
| 17:27:38 | melwitt | thanks kashyap | |
| 17:29:24 | sean-k-mooney | im in the middle of something else but dansmith melwitt do you want me to select one of the jobs and go back to the normal image and larger nodeset lable | |
| 17:29:38 | sean-k-mooney | or will ye have time to do that | |
| 17:30:03 | dansmith | not just larger nodeset, but also with a larger flavor | |
| 17:30:14 | sean-k-mooney | melwitt: also i had hopped to start reviewing your serise today but its looking like it will we wednessday | |
| 17:30:29 | sean-k-mooney | dansmith: ya that what i ment but didnt say | |
| 17:30:40 | dansmith | I'm actually supposed to be off doing something else right now too, but I can work on that later if you never get to it | |
| 17:31:02 | dansmith | (later this week) | |
| 17:31:08 | sean-k-mooney | ack ill see if i have time before i finish for today | |
| 17:31:21 | sean-k-mooney | to that end ill go back to my ansibel patch | |
| 17:33:48 | kashyap | sean-k-mooney: melwitt: A quick start here: https://etherpad.opendev.org/p/Kernel-panics-in-Nova-CI | |
| 17:44:43 | gibi | why a privsep decorated function like https://github.com/openstack/nova/blob/master/nova/virt/libvirt/cpu/core.py#L63-L66 does not get escalated privileges? I see https://paste.opendev.org/show/bFZ9WgE2828iGamQoDaB/ and sure /sys/devices/system/cpu/cpu1/online is only writtable by root but I would assume that is why we have the privesep decorator on the call | |
| 17:46:37 | sean-k-mooney | it shold be elevated | |
| 17:46:59 | sean-k-mooney | that is a file not found issue | |
| 17:47:28 | gibi | -rw-r--r--. 1 root root 4.0K Jan 15 14:38 /sys/devices/system/cpu/cpu1/online | |
| 17:47:28 | gibi | bash-5.1$ ls -alh /sys/devices/system/cpu/cpu1/online | |
| 17:47:28 | gibi | [root@edpm-compute-1 ~]# podman exec -it nova_compute /bin/bash | |
| 17:47:28 | gibi | no the file is visible | |
| 17:47:43 | sean-k-mooney | can you write ot it with sudo ? | |
| 17:47:59 | sean-k-mooney | im wondering if its selinux | |
| 17:48:12 | sean-k-mooney | im guesing your on centso becasue fo py 3.9 | |
| 17:48:33 | gibi | yes I can | |
| 17:48:34 | gibi | [root@edpm-compute-1 ~]# podman exec -it -u root nova_compute /bin/bash | |
| 17:48:35 | gibi | [root@edpm-compute-1 /]# echo "online" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:38 | gibi | [root@edpm-compute-1 /]# echo "offline" > /sys/devices/system/cpu/cpu1/online | |
| 17:48:41 | gibi | [root@edpm-compute-1 /]# cat /sys/devices/system/cpu/cpu1/online | |
| 17:48:43 | gibi | 0 | |
| 17:49:03 | sean-k-mooney | odd | |
| 17:49:09 | sean-k-mooney | anythign in the audit log? | |
| 17:49:48 | sean-k-mooney | im wondering if hte permission issue is due to user/group, or due to selinux or something else | |
| 17:51:39 | gibi | no related log in audit.log | |
| 17:52:08 | gibi | wouldn't I should see the code that escalates priviledges in the stack trace? | |
| 17:52:22 | gibi | * shouldn't I see ... | |
| 17:52:36 | sean-k-mooney | well it should be runing via privsep | |
| 17:52:58 | sean-k-mooney | so you shoudl see some privsep debug logs | |
| 17:53:24 | sean-k-mooney | im not sure why this is saying ERROR oslo_service.service | |
| 17:54:02 | sean-k-mooney | thats the wrong module logger | |
| 17:54:13 | gibi | there is no debug log about privsep either | |
| 17:54:45 | gibi | but we have `oslo.privsep.daemon=INFO` in the config | |
| 17:55:03 | sean-k-mooney | you should see message form nova sayign the deamon was started | |
| 17:55:07 | sean-k-mooney | even at info | |
| 17:55:15 | sean-k-mooney | perhaps nto the detailed logs but you should see it start | |
| 18:01:45 | gibi | something is very strange | |
| 18:02:52 | gibi | at the first nova-compute startup the privsep daemon logs that it is started, but subsequent container restarts it does not log it | |
| 18:03:33 | gibi | I'm going to redeploy the computes to see if I can reproduce it | |
| 18:04:46 | sean-k-mooney | this is nova compute container | |
| 18:04:59 | sean-k-mooney | hum ok i woner if we are leakign the unix socket | |
| 18:05:11 | sean-k-mooney | or what we are using to detech the start between container starts | |
| 18:05:24 | sean-k-mooney | and then not recreating the privsep deamon | |
| 18:09:21 | gibi | I pushed revert https://github.com/openstack-k8s-operators/nova-operator/pull/650 to unblock Andrew and the folks testing cpu pinning | |
| 18:10:15 | sean-k-mooney | hehe ok im not sure htat is super relevent to upstream nova folks but perhaps we have a bug | |
| 18:10:30 | gibi | nah, that repo is also upstream :D | |
| 18:10:40 | sean-k-mooney | i supose | |
| 18:11:00 | gibi | spreading the word that a new deployer engine exists ;) | |
| 18:11:03 | sean-k-mooney | anyway i suspect we need to look into how privsep is being executed | |
| 18:11:37 | sean-k-mooney | i didnt look at that previosly becuase i was expecting it to run the same as triplo did | |
| 18:11:51 | sean-k-mooney | but perhasp something has changed there | |
| 18:12:16 | gibi | I will redeploy tomorrow and check for the privesp process before touching anything | |
| 18:12:32 | gibi | then I will try a container restart to see if privesp also restarted | |
| 18:13:03 | sean-k-mooney | so you might need to do an operation that needs privsep before its started | |
| 18:13:13 | sean-k-mooney | so you might need to boot a vm or something like that | |
| 18:13:18 | sean-k-mooney | i think its lazy loaded | |
| 18:13:33 | sean-k-mooney | we are not usign the fork modle i belive | |
| 18:13:51 | sean-k-mooney | there are 2 ways to run privsep but i think we are using the privsep helper way | |
| 18:13:58 | sean-k-mooney | vai sudo/rootwrap | |
| 18:14:14 | gibi | I will check tomorrow and report back | |
| 18:14:26 | sean-k-mooney | cool. are you almost done for today | |
| 18:14:49 | sean-k-mooney | im going to be aroudn until the top of the hour but i approved the revert quickly to unblock folks | |
| 18:15:03 | gibi | I'm done for today | |
| 18:15:08 | gibi | see you all tomorrow | |
| 18:15:11 | sean-k-mooney | o/ | |
| 19:07:49 | opendevreview | Merged openstack/nova stable/2023.2: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/897297 | |
| 19:31:22 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| 19:32:59 | sean-k-mooney | dansmith: it looks like the 16g nodepool lables are not aviable so i used the 32G ones ^ i can see about readding the 16G flavors but i just want to see if that is correct first. the vxhost flaovr still exists as we have 16G lables but with bionic | |
| 19:33:11 | sean-k-mooney | so we just need a 16G lable with jammy | |
| 19:35:35 | sean-k-mooney | with that said im not sure but it looks like vexhost might be providing 32G vms by defult now https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.opendev.org.yaml#L147-L187 | |
| 20:06:49 | opendevreview | Merged openstack/nova stable/2023.1: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/904373 | |
| 21:51:58 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| #openstack-nova - 2024-01-16 | |||
| 01:26:02 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| 06:11:14 | auniyal | Hi sean-k-mooney, gibi, bauzas, can you please llok at this change its a simple and important change which can help many tests. | |
| 06:11:15 | auniyal | https://review.opendev.org/c/openstack/nova/+/893584 | |
| 08:03:18 | gibi | elodilles: we can land the zed backport of https://review.opendev.org/q/topic:%22bug/2025480%22 now | |
| 08:06:32 | gibi | auniyal: done | |
| 08:07:34 | auniyal | gibi, thanks there are 2 more patches with same topic - can you please review them as well https://review.opendev.org/q/topic:%22refactor-volumeAttachment-calls%22 | |
| 08:31:26 | elodilles | gibi: ACK, thanks! +2+W'd \o/ | |
| 09:13:15 | opendevreview | Merged openstack/nova stable/zed: Reproduce bug #2025480 in a functional test https://review.opendev.org/c/openstack/nova/+/904374 | |
| 09:23:36 | gibi | sean-k-mooney bauzas: re: powermgmt; it seems that during compute startup when power management offlines unused pcpus the privsep daemon is not spawned automatically. During VM boot the privsep daemon spawned automatically | |
| 09:43:46 | gibi | it seems the privsep decorator fails to apply | |
| 09:43:47 | gibi | https://paste.opendev.org/show/b3m4lKm2z7c9yP2psvSJ/ | |
| 09:45:15 | Uggla | Hello, can someone clarify this: This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo). I mean in which case it is a FQDN or not ? | |
| 09:46:04 | Uggla | And do you know if the behavior of libvirt changed recently ? | |
| 09:54:54 | gibi | bauzas: sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/885293 it is not backported to 2023.1 :/ | |
| 09:55:33 | gibi | bauzas: do you happen to have a list of bugfixes you implemented for the powermgmt feature. I want to check all to see what is not backported to 2023.1 | |
| 09:56:20 | bauzas | gibi: I don't have a lot of bugfixes for this | |
| 09:56:27 | bauzas | I think you found the only one missing :( | |
| 09:56:37 | bauzas | (sorry, was in meeting) | |
| 09:57:37 | gibi | OK | |
| 09:57:40 | gibi | I will backport it | |
| 09:58:54 | bauzas | thanks | |
| 10:02:16 | bauzas | Uggla: are you talking of hypervisor_hostname field ? | |