| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-07 | |||
| 21:47:01 | sean-k-mooney | well you dont have to move them | |
| 21:47:20 | sean-k-mooney | that the nice thign about deployment you jsut get them form barbican in pre live migrate | |
| 21:47:27 | sean-k-mooney | there is no copyign right. | |
| 21:47:28 | melwitt | yes you can with acl also. a bit more code is needed to enable 'user' live migration and I have them proposed | |
| 21:47:59 | sean-k-mooney | by migrate deployment secret did you mean add the acl? | |
| 21:48:08 | sean-k-mooney | or did you mean move them during live migrate? | |
| 21:48:21 | sean-k-mooney | or i guess copy them to a new secret | |
| 21:48:31 | sean-k-mooney | but that woudl be on resize right | |
| 21:48:45 | melwitt | yeah but I mean if we wanted to do the user + acl for nova user thing, taking the nova user secret and creating a user secret and copying the passphrase over and adding the acl, I dunno. mess | |
| 21:49:13 | sean-k-mooney | oh your sayign supprotign the 4th policy | |
| 21:49:19 | sean-k-mooney | fo ading the nova user by acl | |
| 21:49:20 | melwitt | for potentially already existing 'deployment' | |
| 21:49:29 | melwitt | secrets | |
| 21:49:39 | sean-k-mooney | ah well we dont have that yet do we? | |
| 21:49:46 | sean-k-mooney | i tought that was not merged | |
| 21:49:55 | melwitt | we do have the secret generation but not the live migration | |
| 21:50:01 | sean-k-mooney | ah ok | |
| 21:50:14 | sean-k-mooney | well if we atned to suprpot it its feeld like a 4th policy | |
| 21:50:42 | sean-k-mooney | but i htink as you suggested talking to the barbican folks at the ptg would be a good next step | |
| 21:50:48 | melwitt | yeah.. that would be easier to not have to deal with it. and then let it stay open to the service project? I dunno. just rambling | |
| 21:50:53 | sean-k-mooney | to confirm that the acls are actully workign as we think they do | |
| 21:51:25 | sean-k-mooney | melwitt: well if its a new policy we can recommend that you resize to delegated for better security or something | |
| 21:51:54 | sean-k-mooney | where deploymene i "nova owned secte" adn delegated is "user owned secreate with nova acl" | |
| 21:52:08 | melwitt | yeah. that would be much less pain | |
| 21:52:19 | melwitt | ++ | |
| 21:52:23 | sean-k-mooney | in anycase i think we shoudl finsih the deploymetn live migration supprot first | |
| 21:52:35 | melwitt | yeah, I agree | |
| 21:52:39 | sean-k-mooney | then maybe tack this on if folks agree | |
| 21:52:58 | melwitt | the way I have it is I put 'user' ahead of 'deployment' since 'user' support is a subset of 'deployment' | |
| 21:53:38 | sean-k-mooney | ya host was the most invasive one | |
| 21:53:43 | melwitt | much smaller change than 'deployment' bc 'deployment' ultimately needs to be able to do secret conversions for resizes | |
| 21:53:47 | sean-k-mooney | as all the resed didnt need rpc/obejct changes | |
| 21:54:18 | sean-k-mooney | right ya i see | |
| 21:54:46 | sean-k-mooney | its not alot diffent for the acl approch as you woudl still need to add/remove it when you change form user | |
| 21:54:52 | sean-k-mooney | but that a singel api call | |
| 21:55:04 | sean-k-mooney | no need to copy the data and reupload it | |
| 21:55:05 | melwitt | yeah, I don't think it would be much to add | |
| 21:55:57 | sean-k-mooney | all fo this falls apart if acls dont work that way but that seams to be what those spec imply | |
| 21:57:06 | sean-k-mooney | kilo era features with "This ACL documentation is work in progress and may change in near future." | |
| 21:57:38 | sean-k-mooney | make me causiously optimistic | |
| 21:57:47 | melwitt | they seem to work how we think, at least in my unexpert testing in devstack. I also used the acl in the 'user' live migration tempest tests I wrote and that works too | |
| 22:00:21 | sean-k-mooney | ok im going to call it a night o/ | |
| 22:00:52 | melwitt | seeya o/ | |
| 22:01:55 | opendevreview | sean mooney proposed openstack/nova master: Add regression test for unified limits resource bug https://review.opendev.org/c/openstack/nova/+/975859 | |
| 22:01:55 | opendevreview | sean mooney proposed openstack/nova master: Fix unified limits to include all resource types https://review.opendev.org/c/openstack/nova/+/975872 | |
| 22:03:14 | sean-k-mooney | i dont know if i resoved all the comments on that but i just pushed what i had ill take a look at it again tomorrow | |
| 22:08:10 | melwitt | ack | |
| #openstack-nova - 2026-04-08 | |||
| 00:04:23 | opendevreview | Merged openstack/nova master: db: add indexes for source/dest migrations https://review.opendev.org/c/openstack/nova/+/958556 | |
| 07:20:59 | opendevreview | ChungWon Lee proposed openstack/nova master: Fix port remain while error on update port with multi network https://review.opendev.org/c/openstack/nova/+/983672 | |
| 07:57:08 | opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | |
| 08:34:28 | opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | |
| 09:18:35 | gibi | melwitt: dansmith: sean-k-mooney: here is the result for a 13 measurement on the PCI device tpool proxy situation https://paste.openstack.org/show/bKHhndSgMIVJtRn1HQLe/ | |
| 09:19:06 | gibi | I see some runtime increase but the timing is pretty noisy | |
| 09:19:50 | gibi | https://www.pasteboard.co/w-LCl8LzsgIk.png | |
| 09:23:49 | gibi | the minimum run times are increasing I would say | |
| 09:36:03 | MaxLamprecht[m] | gibi: can you share the libvirt version you have running? | |
| 09:57:04 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 09:59:43 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 10:01:30 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 10:30:22 | gibi | MaxLamprecht[m]: 10.0.0-2ubuntu8.12 | |
| 10:48:46 | MaxLamprecht[m] | The increase of the minimum run time sounds similar to my reported finding. I assume that if you are updating to libvirt > 10.5 this constant increase is gone. | |
| 11:01:05 | gibi | hm, interesting. That would suggest that the actual increase is due to libvirt and not due to nova itself | |
| 11:08:30 | gibi | so if that is a theory then that suggests something accumlates on the libvirt side and a libvirt restart without a compute restart could clear it | |
| 11:09:14 | gibi | MaxLamprecht[m]: do you have a link to the bug / bugfix in libvirt 10.5 that supposed to help? | |
| 11:10:10 | gibi | sorry I saw it in the bug | |
| 11:10:16 | gibi | thanks for adding it there | |
| 11:10:36 | gibi | it seems we both wrote a comment to that bug around the same time | |
| 11:10:41 | gibi | hence I missed your comment | |
| 11:10:42 | gibi | until now | |
| 11:11:50 | gibi | meanwhile I tried what happens when libvirt is restarted in my env while keep the nova-compute running | |
| 11:12:40 | gibi | indeed after the libvirt restart the timing of the _get_pci_passthrough_devices call is back to the value it was at the start of my measurement run | |
| 11:13:25 | MaxLamprecht[m] | gibi: thx for testing :) I guess that confirms it | |
| 11:20:10 | gibi | I will try to pull in libvirt > 10.5 to see if that also proves it | |
| 11:20:32 | sean-k-mooney | gibi: well libvir tdoes have a nodedevd which si listening to udev events and has a chache compoent | |
| 11:20:49 | MaxLamprecht[m] | We observed this half a year ago and had massive freezes of nova-compute after having libvirt running for multiple days/months. I thought this is also related to this bug https://bugs.launchpad.net/nova/+bug/2091033. Thats also why I referenced it in my caching MR und mentioned the libvirt issue in the releasenote of it. | |
| 11:21:03 | sean-k-mooney | so maybe that was buggy in older version and degraged over tiem with repeated calls | |
| 11:22:06 | sean-k-mooney | gibi: the system i have had up for almost 4 months is runing 11.3.0 for what it worth | |
| 11:22:39 | sean-k-mooney | and the perodic tiems were ~5seconds total | |
| 11:24:52 | sean-k-mooney | sorry nova-compute as up 3 weeks 6 days ago and libvirt 2 months 13 days ago | |
| 11:25:25 | sean-k-mooney | so with newer libvirt maybe it does not happen | |
| 11:25:56 | sean-k-mooney | witht that said Memory: 14.1G (peak: 28.6G) for libvirtd seams... high | |
| 11:37:54 | sean-k-mooney | PID: 729853 Shared: 22496K Name: libvirtd | |
| 11:37:56 | sean-k-mooney | PID VSZ RSS PSS COMMAND | |
| 11:37:58 | sean-k-mooney | 729853 1752908 48344 0 libvirtd | |
| 11:38:22 | sean-k-mooney | i guess that actully ok | |
| 11:38:36 | sean-k-mooney | that only 22MB of shared memory not 22G | |
| 11:39:35 | sean-k-mooney | gibi: can you check https://paste.opendev.org/show/bnD6HOACPqVczxGCqYBi/ with your libvirts pid if you have not already replaced it | |
| 11:39:47 | sean-k-mooney | i wonder if your seeing high memory usage form it or not | |
| 11:40:47 | gibi | Apr 08 11:06:56 aio systemd[1]: libvirtd.service: Consumed 12min 42.721s CPU time, 1.7G memory peak, 24.0K memory swap peak. | |
| 11:41:50 | gibi | this was the libvirt restart after more than 13 hours of usage | |
| 11:42:25 | gibi | after 40 minutes passed since and the now it is on | |
| 11:42:26 | gibi | Memory: 205.9M (peak: 206.7M) | |
| 11:42:32 | gibi | same libvirt version still | |
| 11:44:49 | sean-k-mooney | ack, i was conderd because this is what min shows https://paste.opendev.org/show/bxt9j6Mtzs4Is99u9nbv/ | |
| 11:46:22 | sean-k-mooney | i guess it might be countign all the shile thread/processes | |
| 11:46:32 | sean-k-mooney | https://paste.opendev.org/show/bEi3uoHoWJ7JNjcYXSA7/ | |
| 11:48:02 | sean-k-mooney | bug given free tell me im using 16G total i dont think 14 of that is used by libvirt so im unsure how much we can really trust that systemd is reprotign in this case | |
| 11:48:19 | sean-k-mooney | that proably not relvent to this issue however | |
| 12:52:11 | sean-k-mooney | Uggla: would tuesday 13:00 utc work for you for the nova/cyborg cross project time? | |
| 12:52:25 | sean-k-mooney | its the hour before the nova sessiosn start and we coudl have it in the cyborg room | |