| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-07 | |||
| 17:11:28 | sean-k-mooney | oh that keeps going https://github.com/openstack/nova/blob/b84864d9781ece8387daa967be5208b859e20d97/nova/tests/functional/test_servers.py#L3453-L3558 | |
| 17:11:35 | melwitt | I have only seen the 3 variant. and I think gmaan did something to improve it but I don't remember. I need to look at the git blame | |
| 17:14:19 | gibi | initial results: 142 devs returned from libvirt (10.0), no instances on the host. Over 10 periodic runs I don't see any trends. But I confirm that eventlet mode is slower overall. | |
| 17:14:20 | sean-k-mooney | is that on master | |
| 17:14:43 | gibi | no proxy: _get_pci_passthrough_devices took 0.137605510999947 secs for 142 libvirt devices | |
| 17:14:54 | gibi | with proxy: periodic update_available_resource took 3.2481384640000215 secs | |
| 17:14:57 | gibi | sorry | |
| 17:15:07 | gibi | with proxy: _get_pci_passthrough_devices took 0.5433623499998248 secs for 142 libvirt devices | |
| 17:15:18 | sean-k-mooney | i asume you just seting the env flag on master? | |
| 17:15:22 | gibi | yepp | |
| 17:15:31 | sean-k-mooney | its nice we can do that :) | |
| 17:15:35 | gibi | :) | |
| 17:15:50 | gibi | I will start adding VMs | |
| 17:16:02 | gibi | and taking a longer measurement window | |
| 17:16:51 | gibi | this is the current data I have https://paste.openstack.org/show/bNCNkXgjaGe1EXOjAxNf/ | |
| 17:16:59 | sean-k-mooney | they said 60 repocues it but if we could get 4-8 it should still be visible it just might take longer to notice but runniing the perodic like ever 10 second should offset that | |
| 17:17:41 | gibi | yepp | |
| 17:18:04 | sean-k-mooney | i guess those are new logs | |
| 17:18:22 | sean-k-mooney | let me check my sriov dev env for the overall perod time quickkly | |
| 17:18:55 | gibi | yeah I needed new logs otherwise it is not clear how long things take | |
| 17:19:14 | sean-k-mooney | i think in debug mode we print the total perodic time | |
| 17:21:41 | sean-k-mooney | Lock "compute_resources" "released" by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 2.483s | |
| 17:21:49 | sean-k-mooney | wo we have thsoe logs | |
| 17:22:08 | sean-k-mooney | but im not sure if the call is in that loc | |
| 17:22:21 | sean-k-mooney | it ideally is not | |
| 17:24:27 | sean-k-mooney | ok no we are computing the resouce view well before that | |
| 17:25:08 | gibi | I'm wondering that the "scaling with the number of VMs" is an artifact of the compute node being more and more under load by those VMs. It is definitely the case for my compute by default. I will measure this with VMs in shutoff state to limit that effect | |
| 17:25:35 | sean-k-mooney | even so the pull perodic time for me locally is 12 seconds | |
| 17:25:38 | sean-k-mooney | *full | |
| 17:26:01 | sean-k-mooney | well there will be more contention for the compute_resouces lock | |
| 17:26:09 | sean-k-mooney | in teh resouce tracker formother perodics | |
| 17:27:10 | sean-k-mooney | we do print the wait time ock "compute_resources" acquired by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: waited 0.001s so mabe that a factor | |
| 17:28:40 | sean-k-mooney | although locally the other thing that uses it is Lock "compute_resources" acquired by "nova.compute.resource_tracker.ResourceTracker.clean_compute_node_cache" | |
| 17:29:03 | sean-k-mooney | i think we aquire it when bootign vms as well | |
| 17:29:19 | sean-k-mooney | but in terms of perodic that contenion does nto seam very high | |
| 17:31:49 | sean-k-mooney | Active: active (running) since Wed 2026-03-11 17:16:44 UTC; 3 weeks 6 days ago | |
| 17:31:58 | sean-k-mooney | so i do not currently have instnace on this env | |
| 17:32:26 | sean-k-mooney | but i ahve had them runign with pci device in the past so it does seam to return to a sane baseline at least on an empty host | |
| 17:34:04 | sean-k-mooney | gibi i guess the next best thign to do is leave it run over night in your env and see if it repoduces when there are isntances | |
| 17:34:59 | gibi | 10 VM that are not using PCI devs on the host and are in shutoff state does not increase the time of the _get_pci_passthrough_devices call. It does increase the time of th whole periodic obviously. I don't yet see any treands in either of the times over time | |
| 17:35:23 | gibi | sean-k-mooney: yeah I will, I just try to see what setup make sense to use | |
| 17:35:54 | gibi | next I will try with PCI user VMs | |
| 17:37:46 | gibi | I have 6+6 VFs (2 PFs) so each VM can take a VF | |
| 17:37:48 | sean-k-mooney | ack im currnetly comparing in my cybrog/pci groupign env. ill leave it alone for not but if you dong see anygint i can maybe run ~50-80 vms | |
| 17:39:00 | sean-k-mooney | i belive i have 2 sriov capable nic on this host and i shoudl barely have enough disk/ram but i havent really pushed this host to find its limits | |
| 17:44:42 | gibi | I don't see any timing data difference between 10VMs without PCI usage and 10 VMs with one VF usage each (both cases the VMs are in shutoff) | |
| 17:45:31 | gibi | so what I will do I will keep the 10 VMs with one VF each,take a GMR and then let the periodic run every 10 seconds during the night | |
| 17:47:59 | sean-k-mooney | hum im not sure what runing vs not would cause in this context | |
| 17:48:34 | sean-k-mooney | techinaly that does change the retivial of mac adresses for VFs | |
| 17:49:01 | sean-k-mooney | i.e. if livbirt is restart after the vf is passed to the vm it may not see the mac | |
| 17:49:34 | sean-k-mooney | but did the bug say it was realted to runign/active vms or are you just leaving them stop to reduces load | |
| 17:49:37 | gibi | I simply want to avoid the cpu usage noise of the 10 VMs running | |
| 17:49:44 | sean-k-mooney | ah ok | |
| 17:49:55 | sean-k-mooney | you could start them and then pause them | |
| 17:49:57 | gibi | it is non 0 even with cirros | |
| 17:50:01 | gibi | sure | |
| 17:50:03 | gibi | that is better | |
| 17:50:25 | sean-k-mooney | but i suspect we wont see much of a delta on the perodic althoug hif we do it would be an interesting datapoint i guess | |
| 18:17:33 | gibi | a quick look in two GMRs ~5 minutes distance I don't see increasing number of Green Threads or Threads reported. | |
| 18:25:59 | gibi | OK. This is the starting state of my measurement setup for the night https://paste.openstack.org/show/b96mgBQpguw6KUseBAgZ/ | |
| 18:34:26 | melwitt | awesome gibi | |
| 20:10:24 | melwitt | dansmith: fyi potential cross-project PTG topic re: barbican secrets https://etherpad.opendev.org/p/nova-2026.2-ptg#L90 cc Uggla sorry if I put the text in the wrong location in the etherpad, feel free to move it | |
| 20:28:00 | dansmith | melwitt: ack, not sure that needs to be a cross-project session really, but cool.. I also didn't realize you were worried about the deployment side of the ownership thing, more the user one | |
| 20:28:11 | dansmith | but good point that neutron would have access to those keys | |
| 20:28:19 | dansmith | unless nova could change the ACL itself | |
| 20:29:20 | melwitt | dansmith: my brain started on user and then the realization about service hit. but yeah nova can change the ACL itself as far as I know | |
| 20:30:02 | melwitt | I thought you had suggested we talk to ppl who know more about barbican so I put it cross-project | |
| 20:30:20 | melwitt | it does suck that there's no way to do ACLs through castellan. not sure if that's by design | |
| 20:31:12 | melwitt | I guess maybe not all key managers support ACLs | |
| 20:33:31 | dansmith | ack | |
| 20:56:44 | sean-k-mooney | melwitt: by the way i belive there as a specic commit possibel as part fo there srbac work that change teh secrte to be owned by the proejct | |
| 20:57:36 | sean-k-mooney | melwitt: i dont exactly know why i feel tha tway but i think we talksed about it before | |
| 20:58:22 | melwitt | sean-k-mooney: I wondered if that could be a possibility. I'll take another gander | |
| 20:58:30 | sean-k-mooney | i.e. the fact that is not only accable by the user is someitng that i was vaugly aware of an that it was not alwasy that way | |
| 21:00:40 | melwitt | as far as I could find it goes down to the model and object level, so always been that way | |
| 21:01:15 | melwitt | (this was just me looking in the code) | |
| 21:01:30 | dansmith | yeah I really thought we were told that it was specific to the user and not the project | |
| 21:02:12 | sean-k-mooney | melwitt: maybe https://opendev.org/openstack/barbican/commit/cd9b0ba9f0b4606193858d3cf05acd413a66ba1e | |
| 21:02:34 | sean-k-mooney | they use to enforce _SECRET_CREATOR = "user_id:%(target.secret.creator_id)s" | |
| 21:03:09 | sean-k-mooney | ... storybaord 2010235 has more info | |
| 21:03:38 | sean-k-mooney | huh its actully working https://storyboard.openstack.org/#!/story/2010235 | |
| 21:03:47 | sean-k-mooney | i guess the bots doen know about story board | |
| 21:05:37 | sean-k-mooney | ya so in https://opendev.org/openstack/barbican/commit/cd9b0ba9f0b4606193858d3cf05acd413a66ba1e the allowed project member not jsut teh orgial secreat creator to decrpte the secrete adn basiclly mange it | |
| 21:06:24 | melwitt | lol storyboard | |
| 21:06:30 | sean-k-mooney | without the need for acls so that proably shoudl ahve at least had a release note... or a doc update.... | |
| 21:07:20 | sean-k-mooney | https://review.opendev.org/c/openstack/barbican/+/853680 apprently that was aslo cherry-picked which is qustionable | |
| 21:07:24 | melwitt | yeah that's a pretty major change | |
| 21:07:34 | sean-k-mooney | although i coudl eb misreading those polices but i dont think i am | |
| 21:08:00 | dansmith | but these are API policies - doesn't barbican also have a more robust internal ACL system? | |
| 21:08:03 | dansmith | I guess I thought that | |
| 21:08:12 | dansmith | is what we were talking about here and not just API policy | |
| 21:08:27 | melwitt | I guess the old rule had rule:secret_project_admin which allows project admin also | |
| 21:08:34 | sean-k-mooney | hum actully on gerrit i think i might be https://review.opendev.org/c/openstack/barbican/+/853680/9/barbican/common/policies/secrets.py#72 | |
| 21:09:15 | melwitt | secrets are not as secret as I had thought :P | |
| 21:09:26 | sean-k-mooney | im not familar with rule:secret_is_not_private | |
| 21:09:59 | melwitt | that's the project_access thing. project_access=false=private. at least that's how it is today | |
| 21:10:09 | sean-k-mooney | ah | |
| 21:10:33 | sean-k-mooney | so seperatly form the acl you can mark it as prive to a user or public to the project | |
| 21:11:05 | sean-k-mooney | i assume acls were an evoltion of that with more granualatiy or someitng in later release | |
| 21:11:08 | melwitt | no that is the ACL. i.e. if you want to change it, it's a PUT to the /acl API | |