| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-15 | |||
| 18:15:08 | gibi | see you all tomorrow | |
| 18:15:11 | sean-k-mooney | o/ | |
| 19:07:49 | opendevreview | Merged openstack/nova stable/2023.2: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/897297 | |
| 19:31:22 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| 19:32:59 | sean-k-mooney | dansmith: it looks like the 16g nodepool lables are not aviable so i used the 32G ones ^ i can see about readding the 16G flavors but i just want to see if that is correct first. the vxhost flaovr still exists as we have 16G lables but with bionic | |
| 19:33:11 | sean-k-mooney | so we just need a 16G lable with jammy | |
| 19:35:35 | sean-k-mooney | with that said im not sure but it looks like vexhost might be providing 32G vms by defult now https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.opendev.org.yaml#L147-L187 | |
| 20:06:49 | opendevreview | Merged openstack/nova stable/2023.1: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/904373 | |
| 21:51:58 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| #openstack-nova - 2024-01-16 | |||
| 01:26:02 | opendevreview | sean mooney proposed openstack/nova master: [DNM] debug kernel paincs https://review.opendev.org/c/openstack/nova/+/905628 | |
| 06:11:14 | auniyal | Hi sean-k-mooney, gibi, bauzas, can you please llok at this change its a simple and important change which can help many tests. | |
| 06:11:15 | auniyal | https://review.opendev.org/c/openstack/nova/+/893584 | |
| 08:03:18 | gibi | elodilles: we can land the zed backport of https://review.opendev.org/q/topic:%22bug/2025480%22 now | |
| 08:06:32 | gibi | auniyal: done | |
| 08:07:34 | auniyal | gibi, thanks there are 2 more patches with same topic - can you please review them as well https://review.opendev.org/q/topic:%22refactor-volumeAttachment-calls%22 | |
| 08:31:26 | elodilles | gibi: ACK, thanks! +2+W'd \o/ | |
| 09:13:15 | opendevreview | Merged openstack/nova stable/zed: Reproduce bug #2025480 in a functional test https://review.opendev.org/c/openstack/nova/+/904374 | |
| 09:23:36 | gibi | sean-k-mooney bauzas: re: powermgmt; it seems that during compute startup when power management offlines unused pcpus the privsep daemon is not spawned automatically. During VM boot the privsep daemon spawned automatically | |
| 09:43:46 | gibi | it seems the privsep decorator fails to apply | |
| 09:43:47 | gibi | https://paste.opendev.org/show/b3m4lKm2z7c9yP2psvSJ/ | |
| 09:45:15 | Uggla | Hello, can someone clarify this: This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo). I mean in which case it is a FQDN or not ? | |
| 09:46:04 | Uggla | And do you know if the behavior of libvirt changed recently ? | |
| 09:54:54 | gibi | bauzas: sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/885293 it is not backported to 2023.1 :/ | |
| 09:55:33 | gibi | bauzas: do you happen to have a list of bugfixes you implemented for the powermgmt feature. I want to check all to see what is not backported to 2023.1 | |
| 09:56:20 | bauzas | gibi: I don't have a lot of bugfixes for this | |
| 09:56:27 | bauzas | I think you found the only one missing :( | |
| 09:56:37 | bauzas | (sorry, was in meeting) | |
| 09:57:37 | gibi | OK | |
| 09:57:40 | gibi | I will backport it | |
| 09:58:54 | bauzas | thanks | |
| 10:02:16 | bauzas | Uggla: are you talking of hypervisor_hostname field ? | |
| 10:03:06 | bauzas | if so, this is virt-dependent | |
| 10:04:15 | Uggla | yep, in fact it seems we have users blocked by the patched introduced by Dan to check rename compute. They registered their nodes with the short name and libvirt seems to provide the FQDN and thus making the comparison to fail and block the compute startup. | |
| 10:05:35 | bauzas | Uggla: libvirt populates this field by https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9611 | |
| 10:06:21 | bauzas | which itself calls libvirt with https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1065 | |
| 10:06:53 | Uggla | @bauzas, yep I looked at the methods doc --> This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo) | |
| 10:07:49 | Uggla | but I guess that's not libvirt which changed it's behavior. | |
| 10:08:08 | bauzas | yeah, just checked the libvirt release notes | |
| 10:08:22 | bauzas | this is possibly something that the operator did which broke | |
| 10:08:37 | Uggla | The question was more personal to understand what libvirt is doing. | |
| 10:08:53 | bauzas | and what dansmith writes actually prevents that, which is definitely what :) | |
| 10:08:56 | bauzas | we want :) | |
| 10:09:21 | bauzas | accidental hostname rewrites break placement and many other things | |
| 10:09:49 | Uggla | It seams the operator upgraded to 2023.1 and he is blocked by the check. | |
| 10:10:25 | Uggla | because short name != fqdn | |
| 10:10:57 | bauzas | that's good that they're blocked then | |
| 10:11:27 | Uggla | I guess it should not be blocked if the compute were registered wit the fqdn previously. | |
| 10:12:15 | Uggla | but I'm not sure we have guideline that enforce this ? | |
| 10:12:46 | Uggla | And how the operator ccan get out this issue without db surgery ? | |
| 10:13:35 | bauzas | nova only registers the nodename | |
| 10:13:46 | bauzas | that means that *something* changed it | |
| 10:13:52 | gibi | bauzas: there is another non backported fix for power mgmt: https://review.opendev.org/c/openstack/nova/+/885352 you implemented this on master but not backported. Therefore I see the bug in 2023.1 but did not noticed the master fix and implemented another fix that is now backported https://review.opendev.org/c/openstack/nova/+/903169 . Unfortunately | |
| 10:13:55 | bauzas | and now nova refuses to start | |
| 10:13:59 | gibi | https://review.opendev.org/c/openstack/nova/+/903169 has a bug in it as it expecting a wrong exception type | |
| 10:14:18 | gibi | bauzas: I will do some reverts and backports to clean this up | |
| 10:14:23 | bauzas | doh | |
| 10:14:40 | bauzas | gibi: apologies for the mess | |
| 10:15:08 | gibi | no worries I made my one fair share of mess in it | |
| 10:15:08 | bauzas | I should create bugfixes with the same gerrit topic than the feature | |
| 10:15:26 | bauzas | instead of using the bug number topic | |
| 10:15:40 | gibi | I prefer bug number topics | |
| 10:15:57 | bauzas | yeah I understand | |
| 10:16:02 | bauzas | this is a tradeoff | |
| 10:16:18 | bauzas | then I should mention some changeid in the commit msg, so gerrit could possibly link it | |
| 10:17:02 | bauzas | tbc, I don't trust my memory and that one is another evidence | |
| 10:17:38 | bauzas | the series was small and there were only two bugfixes, despite that I failed to remember about that | |
| 10:17:52 | bauzas | my brain is so fcked | |
| 10:19:43 | opendevreview | Merged openstack/nova stable/zed: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/904375 | |
| 10:24:40 | gibi | bauzas: I don't know how to avoid this via automation | |
| 10:25:22 | gibi | elodilles: do I need to backport a revert, or I can do a revert on the stable branch independently of a revert on master? | |
| 10:26:55 | gibi | elodilles: i.e can I do git revert on stable, or do I need to do git cherry-pick -x on the revert commit from master to stable? | |
| 10:32:58 | gibi | elodilles: I will go with the cherry-pick as I affraid of the stable backport script will not allow an independent revert :/ | |
| 10:33:30 | Uggla | @bauzas you said we store only the shortname for compute name in the db right ? | |
| 10:33:37 | bauzas | nope | |
| 10:33:49 | bauzas | I said we store what libvirt gives us | |
| 10:34:24 | bauzas | if after an upgrade libvirt is giving us something else, then now with 2023.1 we fail | |
| 10:34:56 | bauzas | previously, we were blindly recreating duplicate resource providers leading to inconsistencies | |
| 10:35:43 | Uggla | yep it means that when the compute was registered he got the shortname and after upgrade or something else libvirt supply the fqdn. | |
| 10:36:21 | Uggla | @bauzas, note I understand the patch goal. | |
| 10:47:33 | elodilles | gibi: we usually simply backport reverts if it is needed all the way from master branch to multiple stable branches. if only needed for some older branches, then [stable-only] is fine. and of course if the bug is superimportant highly critical ultimate fix is needed, then also mark them as [stable-only] and merge them parallel in all stable branches :) | |
| 10:48:17 | elodilles | these are my memories from the past ^^^ :) | |
| 10:49:56 | gibi | not super critical | |
| 10:50:09 | gibi | going with the cherry-picks | |
| 10:50:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: Revert "[pwmgmt]ignore missin governor when cpu_state used" https://review.opendev.org/c/openstack/nova/+/905671 | |
| 10:51:02 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.2: Revert "[pwmgmt]ignore missin governor when cpu_state used" https://review.opendev.org/c/openstack/nova/+/905672 | |
| 10:51:03 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.2: cpu: make governors to be optional https://review.opendev.org/c/openstack/nova/+/905673 | |
| 10:51:16 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.1: Revert "[pwmgmt]ignore missin governor when cpu_state used" https://review.opendev.org/c/openstack/nova/+/905674 | |
| 10:51:17 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.1: cpu: fix the privsep issue when offlining the cpu https://review.opendev.org/c/openstack/nova/+/905675 | |
| 10:51:18 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.1: cpu: make governors to be optional https://review.opendev.org/c/openstack/nova/+/905676 | |
| 10:51:36 | gibi | elodilles, bauzas, sean-k-mooney ^^ | |
| 10:52:52 | gibi | it is complicated as on master we only need to revert, on 2023.2 we revert and backport the proper fix from master, on 2023.1 we revert, backport a related fix from 2023.2 and then backport the original good fix from master->2023.2 | |
| 10:52:57 | bauzas | gibi: ++ | |
| 11:06:59 | Uggla | do we agree that there is no "proper" way to rename a compute node. And that we need to stop the compute service, remove the compute from the configuration, and restart the compute to have it registered with the new name ? | |
| 11:07:00 | zigo | kashyap: For Ubuntu stuff, please ask jamespage or coreycb_ please, I have no idea about #ubuntu-kernel on Libera at all, and I honestly don't know anything about the Ubuntu kernel in general. | |
| 11:07:46 | kashyap | zigo: Thans for the contacts :) In the past I had a couple of chats on #ubuntu-kernel, while it used to be still on FN :) | |
| 11:12:15 | jamespage | kashyap: lacking some context - just reading the etherpad linked above somewhere | |
| 11:12:59 | kashyap | jamespage: Hi, I didn't ping you with full context yet. :) | |
| 11:13:31 | kashyap | jamespage: It's about a certain class of kernel panics we're seeing in the upstream CI. An example: https://bugs.launchpad.net/nova/+bug/2018612 | |
| 11:16:07 | kashyap | Admittedly (a) these are difficult to reproduce locally; and (b) distro kernel maintainers (understandably) might not have time to debug "unsupported distros" (as evidenced above). | |
| 11:16:30 | kashyap | I need to step out to get some lunch; need to drop here. (I don't have an IRC bouncer anymore) | |