| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-15 | |||
| 17:54:45 | gibi | but we have `oslo.privsep.daemon=INFO` in the config | |
| 17:55:03 | sean-k-mooney | you should see message form nova sayign the deamon was started | |
| 17:55:07 | sean-k-mooney | even at info | |
| 17:55:15 | sean-k-mooney | perhaps nto the detailed logs but you should see it start | |
| 18:01:45 | gibi | something is very strange | |
| 18:02:52 | gibi | at the first nova-compute startup the privsep daemon logs that it is started, but subsequent container restarts it does not log it | |
| 18:03:33 | gibi | I'm going to redeploy the computes to see if I can reproduce it | |
| 18:04:46 | sean-k-mooney | this is nova compute container | |
| 18:04:59 | sean-k-mooney | hum ok i woner if we are leakign the unix socket | |
| 18:05:11 | sean-k-mooney | or what we are using to detech the start between container starts | |
| 18:05:24 | sean-k-mooney | and then not recreating the privsep deamon | |
| 18:09:21 | gibi | I pushed revert https://github.com/openstack-k8s-operators/nova-operator/pull/650 to unblock Andrew and the folks testing cpu pinning | |
| 18:10:15 | sean-k-mooney | hehe ok im not sure htat is super relevent to upstream nova folks but perhaps we have a bug | |
| 18:10:30 | gibi | nah, that repo is also upstream :D | |
| 18:10:40 | sean-k-mooney | i supose | |
| 18:11:00 | gibi | spreading the word that a new deployer engine exists ;) | |
| 18:11:03 | sean-k-mooney | anyway i suspect we need to look into how privsep is being executed | |
| 18:11:37 | sean-k-mooney | i didnt look at that previosly becuase i was expecting it to run the same as triplo did | |
| 18:11:51 | sean-k-mooney | but perhasp something has changed there | |
| 18:12:16 | gibi | I will redeploy tomorrow and check for the privesp process before touching anything | |
| 18:12:32 | gibi | then I will try a container restart to see if privesp also restarted | |
| 18:13:03 | sean-k-mooney | so you might need to do an operation that needs privsep before its started | |
| 18:13:13 | sean-k-mooney | so you might need to boot a vm or something like that | |
| 18:13:18 | sean-k-mooney | i think its lazy loaded | |
| 18:13:33 | sean-k-mooney | we are not usign the fork modle i belive | |
| 18:13:51 | sean-k-mooney | there are 2 ways to run privsep but i think we are using the privsep helper way | |
| 18:13:58 | sean-k-mooney | vai sudo/rootwrap | |
| 18:14:14 | gibi | I will check tomorrow and report back | |
| 18:14:26 | sean-k-mooney | cool. are you almost done for today | |
| 18:14:49 | sean-k-mooney | im going to be aroudn until the top of the hour but i approved the revert quickly to unblock folks | |
| 18:15:03 | gibi | I'm done for today | |
| 18:15:08 | gibi | see you all tomorrow | |
| 18:15:11 | sean-k-mooney | o/ | |
| 19:07:49 | opendevreview | Merged openstack/nova stable/2023.2: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/897297 | |
| 19:31:22 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| 19:32:59 | sean-k-mooney | dansmith: it looks like the 16g nodepool lables are not aviable so i used the 32G ones ^ i can see about readding the 16G flavors but i just want to see if that is correct first. the vxhost flaovr still exists as we have 16G lables but with bionic | |
| 19:33:11 | sean-k-mooney | so we just need a 16G lable with jammy | |
| 19:35:35 | sean-k-mooney | with that said im not sure but it looks like vexhost might be providing 32G vms by defult now https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.opendev.org.yaml#L147-L187 | |
| 20:06:49 | opendevreview | Merged openstack/nova stable/2023.1: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/904373 | |
| 21:51:58 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| #openstack-nova - 2024-01-16 | |||
| 01:26:02 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| 06:11:14 | auniyal | Hi sean-k-mooney, gibi, bauzas, can you please llok at this change its a simple and important change which can help many tests. | |
| 06:11:15 | auniyal | https://review.opendev.org/c/openstack/nova/+/893584 | |
| 08:03:18 | gibi | elodilles: we can land the zed backport of https://review.opendev.org/q/topic:%22bug/2025480%22 now | |
| 08:06:32 | gibi | auniyal: done | |
| 08:07:34 | auniyal | gibi, thanks there are 2 more patches with same topic - can you please review them as well https://review.opendev.org/q/topic:%22refactor-volumeAttachment-calls%22 | |
| 08:31:26 | elodilles | gibi: ACK, thanks! +2+W'd \o/ | |
| 09:13:15 | opendevreview | Merged openstack/nova stable/zed: Reproduce bug #2025480 in a functional test https://review.opendev.org/c/openstack/nova/+/904374 | |
| 09:23:36 | gibi | sean-k-mooney bauzas: re: powermgmt; it seems that during compute startup when power management offlines unused pcpus the privsep daemon is not spawned automatically. During VM boot the privsep daemon spawned automatically | |
| 09:43:46 | gibi | it seems the privsep decorator fails to apply | |
| 09:43:47 | gibi | https://paste.opendev.org/show/b3m4lKm2z7c9yP2psvSJ/ | |
| 09:45:15 | Uggla | Hello, can someone clarify this: This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo). I mean in which case it is a FQDN or not ? | |
| 09:46:04 | Uggla | And do you know if the behavior of libvirt changed recently ? | |
| 09:54:54 | gibi | bauzas: sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/885293 it is not backported to 2023.1 :/ | |
| 09:55:33 | gibi | bauzas: do you happen to have a list of bugfixes you implemented for the powermgmt feature. I want to check all to see what is not backported to 2023.1 | |
| 09:56:20 | bauzas | gibi: I don't have a lot of bugfixes for this | |
| 09:56:27 | bauzas | I think you found the only one missing :( | |
| 09:56:37 | bauzas | (sorry, was in meeting) | |
| 09:57:37 | gibi | OK | |
| 09:57:40 | gibi | I will backport it | |
| 09:58:54 | bauzas | thanks | |
| 10:02:16 | bauzas | Uggla: are you talking of hypervisor_hostname field ? | |
| 10:03:06 | bauzas | if so, this is virt-dependent | |
| 10:04:15 | Uggla | yep, in fact it seems we have users blocked by the patched introduced by Dan to check rename compute. They registered their nodes with the short name and libvirt seems to provide the FQDN and thus making the comparison to fail and block the compute startup. | |
| 10:05:35 | bauzas | Uggla: libvirt populates this field by https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9611 | |
| 10:06:21 | bauzas | which itself calls libvirt with https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1065 | |
| 10:06:53 | Uggla | @bauzas, yep I looked at the methods doc --> This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo) | |
| 10:07:49 | Uggla | but I guess that's not libvirt which changed it's behavior. | |
| 10:08:08 | bauzas | yeah, just checked the libvirt release notes | |
| 10:08:22 | bauzas | this is possibly something that the operator did which broke | |
| 10:08:37 | Uggla | The question was more personal to understand what libvirt is doing. | |
| 10:08:53 | bauzas | and what dansmith writes actually prevents that, which is definitely what :) | |
| 10:08:56 | bauzas | we want :) | |
| 10:09:21 | bauzas | accidental hostname rewrites break placement and many other things | |
| 10:09:49 | Uggla | It seams the operator upgraded to 2023.1 and he is blocked by the check. | |
| 10:10:25 | Uggla | because short name != fqdn | |
| 10:10:57 | bauzas | that's good that they're blocked then | |
| 10:11:27 | Uggla | I guess it should not be blocked if the compute were registered wit the fqdn previously. | |
| 10:12:15 | Uggla | but I'm not sure we have guideline that enforce this ? | |
| 10:12:46 | Uggla | And how the operator ccan get out this issue without db surgery ? | |
| 10:13:35 | bauzas | nova only registers the nodename | |
| 10:13:46 | bauzas | that means that *something* changed it | |
| 10:13:52 | gibi | bauzas: there is another non backported fix for power mgmt: https://review.opendev.org/c/openstack/nova/+/885352 you implemented this on master but not backported. Therefore I see the bug in 2023.1 but did not noticed the master fix and implemented another fix that is now backported https://review.opendev.org/c/openstack/nova/+/903169 . Unfortunately | |
| 10:13:55 | bauzas | and now nova refuses to start | |
| 10:13:59 | gibi | https://review.opendev.org/c/openstack/nova/+/903169 has a bug in it as it expecting a wrong exception type | |
| 10:14:18 | gibi | bauzas: I will do some reverts and backports to clean this up | |
| 10:14:23 | bauzas | doh | |
| 10:14:40 | bauzas | gibi: apologies for the mess | |
| 10:15:08 | gibi | no worries I made my one fair share of mess in it | |
| 10:15:08 | bauzas | I should create bugfixes with the same gerrit topic than the feature | |
| 10:15:26 | bauzas | instead of using the bug number topic | |
| 10:15:40 | gibi | I prefer bug number topics | |
| 10:15:57 | bauzas | yeah I understand | |
| 10:16:02 | bauzas | this is a tradeoff | |
| 10:16:18 | bauzas | then I should mention some changeid in the commit msg, so gerrit could possibly link it | |
| 10:17:02 | bauzas | tbc, I don't trust my memory and that one is another evidence | |
| 10:17:38 | bauzas | the series was small and there were only two bugfixes, despite that I failed to remember about that | |
| 10:17:52 | bauzas | my brain is so fcked | |
| 10:19:43 | opendevreview | Merged openstack/nova stable/zed: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/904375 | |
| 10:24:40 | gibi | bauzas: I don't know how to avoid this via automation | |