| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-07 | |||
| 21:51:25 | sean-k-mooney | melwitt: well if its a new policy we can recommend that you resize to delegated for better security or something | |
| 21:51:54 | sean-k-mooney | where deploymene i "nova owned secte" adn delegated is "user owned secreate with nova acl" | |
| 21:52:08 | melwitt | yeah. that would be much less pain | |
| 21:52:19 | melwitt | ++ | |
| 21:52:23 | sean-k-mooney | in anycase i think we shoudl finsih the deploymetn live migration supprot first | |
| 21:52:35 | melwitt | yeah, I agree | |
| 21:52:39 | sean-k-mooney | then maybe tack this on if folks agree | |
| 21:52:58 | melwitt | the way I have it is I put 'user' ahead of 'deployment' since 'user' support is a subset of 'deployment' | |
| 21:53:38 | sean-k-mooney | ya host was the most invasive one | |
| 21:53:43 | melwitt | much smaller change than 'deployment' bc 'deployment' ultimately needs to be able to do secret conversions for resizes | |
| 21:53:47 | sean-k-mooney | as all the resed didnt need rpc/obejct changes | |
| 21:54:18 | sean-k-mooney | right ya i see | |
| 21:54:46 | sean-k-mooney | its not alot diffent for the acl approch as you woudl still need to add/remove it when you change form user | |
| 21:54:52 | sean-k-mooney | but that a singel api call | |
| 21:55:04 | sean-k-mooney | no need to copy the data and reupload it | |
| 21:55:05 | melwitt | yeah, I don't think it would be much to add | |
| 21:55:57 | sean-k-mooney | all fo this falls apart if acls dont work that way but that seams to be what those spec imply | |
| 21:57:06 | sean-k-mooney | kilo era features with "This ACL documentation is work in progress and may change in near future." | |
| 21:57:38 | sean-k-mooney | make me causiously optimistic | |
| 21:57:47 | melwitt | they seem to work how we think, at least in my unexpert testing in devstack. I also used the acl in the 'user' live migration tempest tests I wrote and that works too | |
| 22:00:21 | sean-k-mooney | ok im going to call it a night o/ | |
| 22:00:52 | melwitt | seeya o/ | |
| 22:01:55 | opendevreview | sean mooney proposed openstack/nova master: Add regression test for unified limits resource bug https://review.opendev.org/c/openstack/nova/+/975859 | |
| 22:01:55 | opendevreview | sean mooney proposed openstack/nova master: Fix unified limits to include all resource types https://review.opendev.org/c/openstack/nova/+/975872 | |
| 22:03:14 | sean-k-mooney | i dont know if i resoved all the comments on that but i just pushed what i had ill take a look at it again tomorrow | |
| 22:08:10 | melwitt | ack | |
| #openstack-nova - 2026-04-08 | |||
| 00:04:23 | opendevreview | Merged openstack/nova master: db: add indexes for source/dest migrations https://review.opendev.org/c/openstack/nova/+/958556 | |
| 07:20:59 | opendevreview | ChungWon Lee proposed openstack/nova master: Fix port remain while error on update port with multi network https://review.opendev.org/c/openstack/nova/+/983672 | |
| 07:57:08 | opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | |
| 08:34:28 | opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | |
| 09:18:35 | gibi | melwitt: dansmith: sean-k-mooney: here is the result for a 13 measurement on the PCI device tpool proxy situation https://paste.openstack.org/show/bKHhndSgMIVJtRn1HQLe/ | |
| 09:19:06 | gibi | I see some runtime increase but the timing is pretty noisy | |
| 09:19:50 | gibi | https://www.pasteboard.co/w-LCl8LzsgIk.png | |
| 09:23:49 | gibi | the minimum run times are increasing I would say | |
| 09:36:03 | MaxLamprecht[m] | gibi: can you share the libvirt version you have running? | |
| 09:57:04 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 09:59:43 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 10:01:30 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 10:30:22 | gibi | MaxLamprecht[m]: 10.0.0-2ubuntu8.12 | |
| 10:48:46 | MaxLamprecht[m] | The increase of the minimum run time sounds similar to my reported finding. I assume that if you are updating to libvirt > 10.5 this constant increase is gone. | |
| 11:01:05 | gibi | hm, interesting. That would suggest that the actual increase is due to libvirt and not due to nova itself | |
| 11:08:30 | gibi | so if that is a theory then that suggests something accumlates on the libvirt side and a libvirt restart without a compute restart could clear it | |
| 11:09:14 | gibi | MaxLamprecht[m]: do you have a link to the bug / bugfix in libvirt 10.5 that supposed to help? | |
| 11:10:10 | gibi | sorry I saw it in the bug | |
| 11:10:16 | gibi | thanks for adding it there | |
| 11:10:36 | gibi | it seems we both wrote a comment to that bug around the same time | |
| 11:10:41 | gibi | hence I missed your comment | |
| 11:10:42 | gibi | until now | |
| 11:11:50 | gibi | meanwhile I tried what happens when libvirt is restarted in my env while keep the nova-compute running | |
| 11:12:40 | gibi | indeed after the libvirt restart the timing of the _get_pci_passthrough_devices call is back to the value it was at the start of my measurement run | |
| 11:13:25 | MaxLamprecht[m] | gibi: thx for testing :) I guess that confirms it | |
| 11:20:10 | gibi | I will try to pull in libvirt > 10.5 to see if that also proves it | |
| 11:20:32 | sean-k-mooney | gibi: well libvir tdoes have a nodedevd which si listening to udev events and has a chache compoent | |
| 11:20:49 | MaxLamprecht[m] | We observed this half a year ago and had massive freezes of nova-compute after having libvirt running for multiple days/months. I thought this is also related to this bug https://bugs.launchpad.net/nova/+bug/2091033. Thats also why I referenced it in my caching MR und mentioned the libvirt issue in the releasenote of it. | |
| 11:21:03 | sean-k-mooney | so maybe that was buggy in older version and degraged over tiem with repeated calls | |
| 11:22:06 | sean-k-mooney | gibi: the system i have had up for almost 4 months is runing 11.3.0 for what it worth | |
| 11:22:39 | sean-k-mooney | and the perodic tiems were ~5seconds total | |
| 11:24:52 | sean-k-mooney | sorry nova-compute as up 3 weeks 6 days ago and libvirt 2 months 13 days ago | |
| 11:25:25 | sean-k-mooney | so with newer libvirt maybe it does not happen | |
| 11:25:56 | sean-k-mooney | witht that said Memory: 14.1G (peak: 28.6G) for libvirtd seams... high | |
| 11:37:54 | sean-k-mooney | PID: 729853 Shared: 22496K Name: libvirtd | |
| 11:37:56 | sean-k-mooney | PID VSZ RSS PSS COMMAND | |
| 11:37:58 | sean-k-mooney | 729853 1752908 48344 0 libvirtd | |
| 11:38:22 | sean-k-mooney | i guess that actully ok | |
| 11:38:36 | sean-k-mooney | that only 22MB of shared memory not 22G | |
| 11:39:35 | sean-k-mooney | gibi: can you check https://paste.opendev.org/show/bnD6HOACPqVczxGCqYBi/ with your libvirts pid if you have not already replaced it | |
| 11:39:47 | sean-k-mooney | i wonder if your seeing high memory usage form it or not | |
| 11:40:47 | gibi | Apr 08 11:06:56 aio systemd[1]: libvirtd.service: Consumed 12min 42.721s CPU time, 1.7G memory peak, 24.0K memory swap peak. | |
| 11:41:50 | gibi | this was the libvirt restart after more than 13 hours of usage | |
| 11:42:25 | gibi | after 40 minutes passed since and the now it is on | |
| 11:42:26 | gibi | Memory: 205.9M (peak: 206.7M) | |
| 11:42:32 | gibi | same libvirt version still | |
| 11:44:49 | sean-k-mooney | ack, i was conderd because this is what min shows https://paste.opendev.org/show/bxt9j6Mtzs4Is99u9nbv/ | |
| 11:46:22 | sean-k-mooney | i guess it might be countign all the shile thread/processes | |
| 11:46:32 | sean-k-mooney | https://paste.opendev.org/show/bEi3uoHoWJ7JNjcYXSA7/ | |
| 11:48:02 | sean-k-mooney | bug given free tell me im using 16G total i dont think 14 of that is used by libvirt so im unsure how much we can really trust that systemd is reprotign in this case | |
| 11:48:19 | sean-k-mooney | that proably not relvent to this issue however | |
| 12:52:11 | sean-k-mooney | Uggla: would tuesday 13:00 utc work for you for the nova/cyborg cross project time? | |
| 12:52:25 | sean-k-mooney | its the hour before the nova sessiosn start and we coudl have it in the cyborg room | |
| 12:53:26 | sean-k-mooney | that means we can wrap the cyborg session in the following hour when nova is startign up and i can then proably join the nova room for the rest of the day | |
| 12:53:57 | sean-k-mooney | since watcher is not starting its ptg until wednesday | |
| 12:55:00 | sean-k-mooney | for wednesday and thrusday i do not have a conflict with wathcer/nova on the last hour of the windows as wather is tartign and fisniing 1 hour early then nova each day | |
| 12:55:20 | sean-k-mooney | so that would be the other option unless you prefered monday? | |
| 16:00:57 | opendevreview | Victor proposed openstack/nova master: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/983753 | |
| 17:48:11 | opendevreview | Victor proposed openstack/nova master: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/983753 | |
| 21:17:43 | opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add spec for virtiofs live and cold migration https://review.opendev.org/c/openstack/nova-specs/+/983801 | |
| #openstack-nova - 2026-04-09 | |||
| 04:50:39 | opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add spec for virtiofs live and cold migration https://review.opendev.org/c/openstack/nova-specs/+/983801 | |
| 05:15:26 | opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add spec for virtiofs hot attach and detach https://review.opendev.org/c/openstack/nova-specs/+/983816 | |
| 06:42:15 | gibi | after 10h run with libvirt 11.3 and the same PCI / nova / devstack setup I see no similar runtime increase. So I believe MaxLamprecht[m] is right, this slowdown is due to the libvirt bug linked to the bug now. If I have to guess our tpoolproxy makes the libvirt bug more visible somehow. Removing the proxy make everything faster therefore the increase does not show up that quickly from the noise | |
| 06:42:21 | gibi | I will add data to the bug | |
| 06:44:05 | gibi | this is the libvirt 11.3 result https://www.pasteboard.co/3rrS0RqeQwfn.png | |
| 06:44:30 | gibi | compare that with 10.0 result https://www.pasteboard.co/w-LCl8LzsgIk.png | |
| 08:11:45 | opendevreview | Robbert Nijgh proposed openstack/nova master: Cache wrapped device proxies to prevent accumulation https://review.opendev.org/c/openstack/nova/+/983828 | |
| 09:45:37 | MaxLamprecht[m] | gibi: thx for the detailed reproduction :) https://review.opendev.org/c/openstack/nova/+/922855 - Do you think that we should still tackle the caching that I introduced in this MR. I guess the releasenote with the information about the libvirt bug could also make sense for operators. | |
| 09:57:33 | sean-k-mooney | gibi: do you have th elink to the bug | |
| 09:58:17 | sean-k-mooney | hum intersting timeing on https://review.opendev.org/c/openstack/nova/+/983828 | |
| 10:02:17 | sean-k-mooney | also found the link and comment https://bugs.launchpad.net/nova/+bug/2147424 | |
| 10:26:25 | sean-k-mooney | MaxLamprecht[m]: minor -1 on https://review.opendev.org/c/openstack/nova/+/922855 | |
| 10:27:21 | sean-k-mooney | the cache currently could leak info between test cases so every tiem we cache somethign like that we need to explcity clear it in the base test case | |
| 10:27:32 | sean-k-mooney | otherwise its mostly good form my perspect | |