Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-01-15
17:30:03 dansmith not just larger nodeset, but also with a larger flavor
17:30:14 sean-k-mooney melwitt: also i had hopped to start reviewing your serise today but its looking like it will we wednessday
17:30:29 sean-k-mooney dansmith: ya that what i ment but didnt say
17:30:40 dansmith I'm actually supposed to be off doing something else right now too, but I can work on that later if you never get to it
17:31:02 dansmith (later this week)
17:31:08 sean-k-mooney ack ill see if i have time before i finish for today
17:31:21 sean-k-mooney to that end ill go back to my ansibel patch
17:33:48 kashyap sean-k-mooney: melwitt: A quick start here: https://etherpad.opendev.org/p/Kernel-panics-in-Nova-CI
17:44:43 gibi why a privsep decorated function like https://github.com/openstack/nova/blob/master/nova/virt/libvirt/cpu/core.py#L63-L66 does not get escalated privileges? I see https://paste.opendev.org/show/bFZ9WgE2828iGamQoDaB/ and sure /sys/devices/system/cpu/cpu1/online is only writtable by root but I would assume that is why we have the privesep decorator on the call
17:46:37 sean-k-mooney it shold be elevated
17:46:59 sean-k-mooney that is a file not found issue
17:47:28 gibi no the file is visible
17:47:28 gibi [root@edpm-compute-1 ~]# podman exec -it nova_compute /bin/bash
17:47:28 gibi bash-5.1$ ls -alh /sys/devices/system/cpu/cpu1/online
17:47:28 gibi -rw-r--r--. 1 root root 4.0K Jan 15 14:38 /sys/devices/system/cpu/cpu1/online
17:47:43 sean-k-mooney can you write ot it with sudo ?
17:47:59 sean-k-mooney im wondering if its selinux
17:48:12 sean-k-mooney im guesing your on centso becasue fo py 3.9
17:48:33 gibi yes I can
17:48:34 gibi [root@edpm-compute-1 ~]# podman exec -it -u root nova_compute /bin/bash
17:48:35 gibi [root@edpm-compute-1 /]# echo "online" > /sys/devices/system/cpu/cpu1/online
17:48:38 gibi [root@edpm-compute-1 /]# echo "offline" > /sys/devices/system/cpu/cpu1/online
17:48:41 gibi [root@edpm-compute-1 /]# cat /sys/devices/system/cpu/cpu1/online
17:48:43 gibi 0
17:49:03 sean-k-mooney odd
17:49:09 sean-k-mooney anythign in the audit log?
17:49:48 sean-k-mooney im wondering if hte permission issue is due to user/group, or due to selinux or something else
17:51:39 gibi no related log in audit.log
17:52:08 gibi wouldn't I should see the code that escalates priviledges in the stack trace?
17:52:22 gibi * shouldn't I see ...
17:52:36 sean-k-mooney well it should be runing via privsep
17:52:58 sean-k-mooney so you shoudl see some privsep debug logs
17:53:24 sean-k-mooney im not sure why this is saying ERROR oslo_service.service
17:54:02 sean-k-mooney thats the wrong module logger
17:54:13 gibi there is no debug log about privsep either
17:54:45 gibi but we have `oslo.privsep.daemon=INFO` in the config
17:55:03 sean-k-mooney you should see message form nova sayign the deamon was started
17:55:07 sean-k-mooney even at info
17:55:15 sean-k-mooney perhaps nto the detailed logs but you should see it start
18:01:45 gibi something is very strange
18:02:52 gibi at the first nova-compute startup the privsep daemon logs that it is started, but subsequent container restarts it does not log it
18:03:33 gibi I'm going to redeploy the computes to see if I can reproduce it
18:04:46 sean-k-mooney this is nova compute container
18:04:59 sean-k-mooney hum ok i woner if we are leakign the unix socket
18:05:11 sean-k-mooney or what we are using to detech the start between container starts
18:05:24 sean-k-mooney and then not recreating the privsep deamon
18:09:21 gibi I pushed revert https://github.com/openstack-k8s-operators/nova-operator/pull/650 to unblock Andrew and the folks testing cpu pinning
18:10:15 sean-k-mooney hehe ok im not sure htat is super relevent to upstream nova folks but perhaps we have a bug
18:10:30 gibi nah, that repo is also upstream :D
18:10:40 sean-k-mooney i supose
18:11:00 gibi spreading the word that a new deployer engine exists ;)
18:11:03 sean-k-mooney anyway i suspect we need to look into how privsep is being executed
18:11:37 sean-k-mooney i didnt look at that previosly becuase i was expecting it to run the same as triplo did
18:11:51 sean-k-mooney but perhasp something has changed there
18:12:16 gibi I will redeploy tomorrow and check for the privesp process before touching anything
18:12:32 gibi then I will try a container restart to see if privesp also restarted
18:13:03 sean-k-mooney so you might need to do an operation that needs privsep before its started
18:13:13 sean-k-mooney so you might need to boot a vm or something like that
18:13:18 sean-k-mooney i think its lazy loaded
18:13:33 sean-k-mooney we are not usign the fork modle i belive
18:13:51 sean-k-mooney there are 2 ways to run privsep but i think we are using the privsep helper way
18:13:58 sean-k-mooney vai sudo/rootwrap
18:14:14 gibi I will check tomorrow and report back
18:14:26 sean-k-mooney cool. are you almost done for today
18:14:49 sean-k-mooney im going to be aroudn until the top of the hour but i approved the revert quickly to unblock folks
18:15:03 gibi I'm done for today
18:15:08 gibi see you all tomorrow
18:15:11 sean-k-mooney o/
19:07:49 opendevreview Merged openstack/nova stable/2023.2: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/897297
19:31:22 opendevreview sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628
19:32:59 sean-k-mooney dansmith: it looks like the 16g nodepool lables are not aviable so i used the 32G ones ^ i can see about readding the 16G flavors but i just want to see if that is correct first. the vxhost flaovr still exists as we have 16G lables but with bionic
19:33:11 sean-k-mooney so we just need a 16G lable with jammy
19:35:35 sean-k-mooney with that said im not sure but it looks like vexhost might be providing 32G vms by defult now https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.opendev.org.yaml#L147-L187
20:06:49 opendevreview Merged openstack/nova stable/2023.1: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/904373
21:51:58 opendevreview sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628
#openstack-nova - 2024-01-16
01:26:02 opendevreview sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628
06:11:14 auniyal Hi sean-k-mooney, gibi, bauzas, can you please llok at this change its a simple and important change which can help many tests.
06:11:15 auniyal https://review.opendev.org/c/openstack/nova/+/893584
08:03:18 gibi elodilles: we can land the zed backport of https://review.opendev.org/q/topic:%22bug/2025480%22 now
08:06:32 gibi auniyal: done
08:07:34 auniyal gibi, thanks there are 2 more patches with same topic - can you please review them as well https://review.opendev.org/q/topic:%22refactor-volumeAttachment-calls%22
08:31:26 elodilles gibi: ACK, thanks! +2+W'd \o/
09:13:15 opendevreview Merged openstack/nova stable/zed: Reproduce bug #2025480 in a functional test https://review.opendev.org/c/openstack/nova/+/904374
09:23:36 gibi sean-k-mooney bauzas: re: powermgmt; it seems that during compute startup when power management offlines unused pcpus the privsep daemon is not spawned automatically. During VM boot the privsep daemon spawned automatically
09:43:46 gibi it seems the privsep decorator fails to apply
09:43:47 gibi https://paste.opendev.org/show/b3m4lKm2z7c9yP2psvSJ/
09:45:15 Uggla Hello, can someone clarify this: This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo). I mean in which case it is a FQDN or not ?
09:46:04 Uggla And do you know if the behavior of libvirt changed recently ?
09:54:54 gibi bauzas: sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/885293 it is not backported to 2023.1 :/
09:55:33 gibi bauzas: do you happen to have a list of bugfixes you implemented for the powermgmt feature. I want to check all to see what is not backported to 2023.1
09:56:20 bauzas gibi: I don't have a lot of bugfixes for this
09:56:27 bauzas I think you found the only one missing :(
09:56:37 bauzas (sorry, was in meeting)
09:57:37 gibi OK
09:57:40 gibi I will backport it
09:58:54 bauzas thanks
10:02:16 bauzas Uggla: are you talking of hypervisor_hostname field ?
10:03:06 bauzas if so, this is virt-dependent
10:04:15 Uggla yep, in fact it seems we have users blocked by the patched introduced by Dan to check rename compute. They registered their nodes with the short name and libvirt seems to provide the FQDN and thus making the comparison to fail and block the compute startup.
10:05:35 bauzas Uggla: libvirt populates this field by https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9611

Earlier   Later