| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-08 | |||
| 07:57:08 | opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | |
| 08:34:28 | opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | |
| 09:18:35 | gibi | melwitt: dansmith: sean-k-mooney: here is the result for a 13 measurement on the PCI device tpool proxy situation https://paste.openstack.org/show/bKHhndSgMIVJtRn1HQLe/ | |
| 09:19:06 | gibi | I see some runtime increase but the timing is pretty noisy | |
| 09:19:50 | gibi | https://www.pasteboard.co/w-LCl8LzsgIk.png | |
| 09:23:49 | gibi | the minimum run times are increasing I would say | |
| 09:36:03 | MaxLamprecht[m] | gibi: can you share the libvirt version you have running? | |
| 09:57:04 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 09:59:43 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 10:01:30 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 10:30:22 | gibi | MaxLamprecht[m]: 10.0.0-2ubuntu8.12 | |
| 10:48:46 | MaxLamprecht[m] | The increase of the minimum run time sounds similar to my reported finding. I assume that if you are updating to libvirt > 10.5 this constant increase is gone. | |
| 11:01:05 | gibi | hm, interesting. That would suggest that the actual increase is due to libvirt and not due to nova itself | |
| 11:08:30 | gibi | so if that is a theory then that suggests something accumlates on the libvirt side and a libvirt restart without a compute restart could clear it | |
| 11:09:14 | gibi | MaxLamprecht[m]: do you have a link to the bug / bugfix in libvirt 10.5 that supposed to help? | |
| 11:10:10 | gibi | sorry I saw it in the bug | |
| 11:10:16 | gibi | thanks for adding it there | |
| 11:10:36 | gibi | it seems we both wrote a comment to that bug around the same time | |
| 11:10:41 | gibi | hence I missed your comment | |
| 11:10:42 | gibi | until now | |
| 11:11:50 | gibi | meanwhile I tried what happens when libvirt is restarted in my env while keep the nova-compute running | |
| 11:12:40 | gibi | indeed after the libvirt restart the timing of the _get_pci_passthrough_devices call is back to the value it was at the start of my measurement run | |
| 11:13:25 | MaxLamprecht[m] | gibi: thx for testing :) I guess that confirms it | |
| 11:20:10 | gibi | I will try to pull in libvirt > 10.5 to see if that also proves it | |
| 11:20:32 | sean-k-mooney | gibi: well libvir tdoes have a nodedevd which si listening to udev events and has a chache compoent | |
| 11:20:49 | MaxLamprecht[m] | We observed this half a year ago and had massive freezes of nova-compute after having libvirt running for multiple days/months. I thought this is also related to this bug https://bugs.launchpad.net/nova/+bug/2091033. Thats also why I referenced it in my caching MR und mentioned the libvirt issue in the releasenote of it. | |
| 11:21:03 | sean-k-mooney | so maybe that was buggy in older version and degraged over tiem with repeated calls | |
| 11:22:06 | sean-k-mooney | gibi: the system i have had up for almost 4 months is runing 11.3.0 for what it worth | |
| 11:22:39 | sean-k-mooney | and the perodic tiems were ~5seconds total | |
| 11:24:52 | sean-k-mooney | sorry nova-compute as up 3 weeks 6 days ago and libvirt 2 months 13 days ago | |
| 11:25:25 | sean-k-mooney | so with newer libvirt maybe it does not happen | |
| 11:25:56 | sean-k-mooney | witht that said Memory: 14.1G (peak: 28.6G) for libvirtd seams... high | |
| 11:37:54 | sean-k-mooney | PID: 729853 Shared: 22496K Name: libvirtd | |
| 11:37:56 | sean-k-mooney | PID VSZ RSS PSS COMMAND | |
| 11:37:58 | sean-k-mooney | 729853 1752908 48344 0 libvirtd | |
| 11:38:22 | sean-k-mooney | i guess that actully ok | |
| 11:38:36 | sean-k-mooney | that only 22MB of shared memory not 22G | |
| 11:39:35 | sean-k-mooney | gibi: can you check https://paste.opendev.org/show/bnD6HOACPqVczxGCqYBi/ with your libvirts pid if you have not already replaced it | |
| 11:39:47 | sean-k-mooney | i wonder if your seeing high memory usage form it or not | |
| 11:40:47 | gibi | Apr 08 11:06:56 aio systemd[1]: libvirtd.service: Consumed 12min 42.721s CPU time, 1.7G memory peak, 24.0K memory swap peak. | |
| 11:41:50 | gibi | this was the libvirt restart after more than 13 hours of usage | |
| 11:42:25 | gibi | after 40 minutes passed since and the now it is on | |
| 11:42:26 | gibi | Memory: 205.9M (peak: 206.7M) | |
| 11:42:32 | gibi | same libvirt version still | |
| 11:44:49 | sean-k-mooney | ack, i was conderd because this is what min shows https://paste.opendev.org/show/bxt9j6Mtzs4Is99u9nbv/ | |
| 11:46:22 | sean-k-mooney | i guess it might be countign all the shile thread/processes | |
| 11:46:32 | sean-k-mooney | https://paste.opendev.org/show/bEi3uoHoWJ7JNjcYXSA7/ | |
| 11:48:02 | sean-k-mooney | bug given free tell me im using 16G total i dont think 14 of that is used by libvirt so im unsure how much we can really trust that systemd is reprotign in this case | |
| 11:48:19 | sean-k-mooney | that proably not relvent to this issue however | |
| 12:52:11 | sean-k-mooney | Uggla: would tuesday 13:00 utc work for you for the nova/cyborg cross project time? | |
| 12:52:25 | sean-k-mooney | its the hour before the nova sessiosn start and we coudl have it in the cyborg room | |
| 12:53:26 | sean-k-mooney | that means we can wrap the cyborg session in the following hour when nova is startign up and i can then proably join the nova room for the rest of the day | |
| 12:53:57 | sean-k-mooney | since watcher is not starting its ptg until wednesday | |
| 12:55:00 | sean-k-mooney | for wednesday and thrusday i do not have a conflict with wathcer/nova on the last hour of the windows as wather is tartign and fisniing 1 hour early then nova each day | |
| 12:55:20 | sean-k-mooney | so that would be the other option unless you prefered monday? | |
| 16:00:57 | opendevreview | Victor proposed openstack/nova master: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/983753 | |
| 17:48:11 | opendevreview | Victor proposed openstack/nova master: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/983753 | |
| 21:17:43 | opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add spec for virtiofs live and cold migration https://review.opendev.org/c/openstack/nova-specs/+/983801 | |
| #openstack-nova - 2026-04-09 | |||
| 04:50:39 | opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add spec for virtiofs live and cold migration https://review.opendev.org/c/openstack/nova-specs/+/983801 | |
| 05:15:26 | opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add spec for virtiofs hot attach and detach https://review.opendev.org/c/openstack/nova-specs/+/983816 | |
| 06:42:15 | gibi | after 10h run with libvirt 11.3 and the same PCI / nova / devstack setup I see no similar runtime increase. So I believe MaxLamprecht[m] is right, this slowdown is due to the libvirt bug linked to the bug now. If I have to guess our tpoolproxy makes the libvirt bug more visible somehow. Removing the proxy make everything faster therefore the increase does not show up that quickly from the noise | |
| 06:42:21 | gibi | I will add data to the bug | |
| 06:44:05 | gibi | this is the libvirt 11.3 result https://www.pasteboard.co/3rrS0RqeQwfn.png | |
| 06:44:30 | gibi | compare that with 10.0 result https://www.pasteboard.co/w-LCl8LzsgIk.png | |
| 08:11:45 | opendevreview | Robbert Nijgh proposed openstack/nova master: Cache wrapped device proxies to prevent accumulation https://review.opendev.org/c/openstack/nova/+/983828 | |
| 09:45:37 | MaxLamprecht[m] | gibi: thx for the detailed reproduction :) https://review.opendev.org/c/openstack/nova/+/922855 - Do you think that we should still tackle the caching that I introduced in this MR. I guess the releasenote with the information about the libvirt bug could also make sense for operators. | |
| 09:57:33 | sean-k-mooney | gibi: do you have th elink to the bug | |
| 09:58:17 | sean-k-mooney | hum intersting timeing on https://review.opendev.org/c/openstack/nova/+/983828 | |
| 10:02:17 | sean-k-mooney | also found the link and comment https://bugs.launchpad.net/nova/+bug/2147424 | |
| 10:26:25 | sean-k-mooney | MaxLamprecht[m]: minor -1 on https://review.opendev.org/c/openstack/nova/+/922855 | |
| 10:27:21 | sean-k-mooney | the cache currently could leak info between test cases so every tiem we cache somethign like that we need to explcity clear it in the base test case | |
| 10:27:32 | sean-k-mooney | otherwise its mostly good form my perspect | |
| 12:34:10 | gibi | https://bugs.launchpad.net/nova/+bug/2147424 | |
| 12:35:07 | gibi | MaxLamprecht[m]: yeah I think it make sense to apply some caching and optimizing the codepath to create less queries to libvirt. | |
| 12:35:24 | gibi | due to launchpad being not accessible to me I only posted my comment to the bug now | |
| 12:37:30 | gibi | btw gmaan I saw some stack traces during my trials yesterday that points to a long race window between nova-compute startup and graceful shutdown so I filed a bug for it https://bugs.launchpad.net/nova/+bug/2147554 | |
| 13:53:28 | opendevreview | Merged openstack/nova master: fix: leftover volume_attachment on instance delete https://review.opendev.org/c/openstack/nova/+/934984 | |
| 14:05:33 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 14:15:22 | tafkamax | ccccccngfgjkjlubkfdrfhbkgtnccgjijvtgvkvbelkj | |
| 14:15:25 | tafkamax | ccccccngfgjklventuvettkrjniiljujfdrkejhejnji | |
| 14:15:29 | tafkamax | ccccccngfgjktfiebfbhrrdjirujgdvteljcgknifnvb | |
| 14:15:38 | tafkamax | ccccccngfgjkfrrfeugfijclrdedkfdhdkdenirblbvv | |
| 14:15:59 | samuelkunkel[m] | playing vimgolf? | |
| 14:21:18 | gibi | feels like some one time password :) | |
| 14:23:29 | opendevreview | Max proposed openstack/nova master: performance: reduce calls to libvirt / add caching https://review.opendev.org/c/openstack/nova/+/922855 | |
| 14:24:47 | tafkamax | sry | |
| 16:33:57 | melwitt | bauzas: two hopefully easy reviews for vtpm - a follow up patch and a generic key manager error handling patch if you might be able to take a look https://review.opendev.org/c/openstack/nova/+/978836 and https://review.opendev.org/c/openstack/nova/+/983504 | |
| 16:35:24 | gmaan | gibi: ack, i will check | |
| 17:45:32 | opendevreview | Victor proposed openstack/nova master: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/983753 | |
| #openstack-nova - 2026-04-10 | |||
| 08:02:19 | opendevreview | Hiroki Narukawa proposed openstack/nova master: SEV-SNP PoC: Add SEV_SNP to MemEncryptionModel object https://review.opendev.org/c/openstack/nova/+/983974 | |
| 08:02:19 | opendevreview | Hiroki Narukawa proposed openstack/nova master: SEV-SNP PoC: scheduler translates SEV-SNP request https://review.opendev.org/c/openstack/nova/+/983975 | |
| 08:02:20 | opendevreview | Hiroki Narukawa proposed openstack/nova master: SEV-SNP PoC: libvirt host to check SEV-SNP availability https://review.opendev.org/c/openstack/nova/+/983976 | |
| 08:02:20 | opendevreview | Hiroki Narukawa proposed openstack/nova master: SEV-SNP PoC: split libvirt config SEV-ES as another class https://review.opendev.org/c/openstack/nova/+/983977 | |
| 08:02:21 | opendevreview | Hiroki Narukawa proposed openstack/nova master: SEV-SNP PoC: add SEV-SNP launch_security to libvirt driver https://review.opendev.org/c/openstack/nova/+/983978 | |
| 08:02:21 | opendevreview | Hiroki Narukawa proposed openstack/nova master: SEV-SNP PoC: libvirt driver to report SEV-SNP trait in sev_es RP https://review.opendev.org/c/openstack/nova/+/983979 | |
| 08:02:22 | opendevreview | Hiroki Narukawa proposed openstack/nova master: SEV-SNP PoC: os_loader_type 'rom' instead of 'pflash' if SEV-SNP https://review.opendev.org/c/openstack/nova/+/983980 | |
| 08:48:38 | opendevreview | Bartosz Bezak proposed openstack/nova stable/2026.1: fix: device_by_alias should respect config type https://review.opendev.org/c/openstack/nova/+/983989 | |
| 09:45:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: minimal change to show sqlite behavior change https://review.opendev.org/c/openstack/nova/+/983995 | |
| 10:43:18 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Mention socket in pci numa affinity extra spec help https://review.opendev.org/c/openstack/nova/+/961634 | |
| 10:48:26 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Fix num_instances_weight_multiplier option example https://review.opendev.org/c/openstack/nova/+/959866 | |