Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-04-06
17:49:12 dansmith well, I thought ade told us the opposite a long while back, but fair enough.. not sure that "owned by the user" would mean anything other than "...and the project" to most readers since that's the default for most resources,
17:49:19 dansmith unless we say "and NOT the project" somewhere
17:49:32 dansmith but not opposed to clarifying in the docs (of course) either way
17:49:45 dansmith where is that default set or overridden?
17:50:06 dansmith like, barbican-side or us as the client?
19:00:57 melwitt dansmith: I saw the doc here while I was writing a tempest test for 'user' that adds the admin via ACL https://docs.openstack.org/api-guide/key-manager/acls.html#default-acl but I will go see if I can find where that is actually defaulted
19:10:24 melwitt looks like it is defaulted to true in the ACL model https://opendev.org/openstack/barbican/src/branch/master/barbican/model/models.py#L1243 and the ACL object https://opendev.org/openstack/barbican/src/branch/master/barbican/objects/secret_acl.py#L32 so I guess the question is, does every created secret get an ACL record automatically or does that only happen if the ACL API is called
19:14:42 melwitt I also don't see any way to create a secret and specify private from the get go https://docs.openstack.org/barbican/latest/api/reference/secrets.html so I guess in order to avoid a window where the ACL is open to the project, you would have to 1) create a metadata-only secret 2) create an ACL with project_access=false 3) PUT the payload to the metadata-only secret
19:14:50 melwitt maybe I'm not understanding this right
19:17:48 dansmith hmm, well, (a) if it's server-side we should probably avoid any statements in our docs about "how it is" and (b) maybe we should ask just to make sure
19:18:01 dansmith I was thinking there was also a "nobody (not even the owner) can see it until it has an ACL of some kind"
19:18:16 dansmith like maybe the owner can always ACL it, but can't read it until they give themselves that permission?
19:18:39 dansmith kinda like unix ownership.. I can remove my own read access, but I can always give it back to myself if I own it
19:18:46 melwitt yeah I dunno :/
19:20:19 melwitt yeah I agree, "how it is" could change. also I guess we could do whatever explicitly but that's not been the behavior up to now, so I would hesitate to change it if it's operating in a way that vtpm users are fine with
19:20:47 melwitt and yeah, it would be nice to ask if there is someone who could explain all the details
#openstack-nova - 2026-04-07
03:33:34 opendevreview melanie witt proposed openstack/nova master: TPM: fixup for bump service version documentation https://review.opendev.org/c/openstack/nova/+/978836
03:33:34 opendevreview melanie witt proposed openstack/nova master: TPM: support live migration of `user` secret security https://review.opendev.org/c/openstack/nova/+/983333
03:33:35 opendevreview melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629
03:33:35 opendevreview melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771
03:33:36 opendevreview melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975
03:33:36 opendevreview melanie witt proposed openstack/nova master: TPM: enable conversion of secret security modes via resize https://review.opendev.org/c/openstack/nova/+/962052
03:33:37 opendevreview melanie witt proposed openstack/nova master: TPM: clean up orphaned libvirt secret on guest creation failure https://review.opendev.org/c/openstack/nova/+/983505
03:33:37 opendevreview melanie witt proposed openstack/nova master: TPM: handle key manager Forbidden errors consistently https://review.opendev.org/c/openstack/nova/+/983504
10:52:03 opendevreview Robbert Nijgh proposed openstack/nova master: Remove proxy wrapping in libvirt device listing, slowing down update_available_resource https://review.opendev.org/c/openstack/nova/+/983529
13:27:51 MaxLamprecht[m] sean-k-mooney: Can you maybe have another look at this https://review.opendev.org/c/openstack/nova/+/934984. We merged already the reproducer a while ago
13:29:16 MaxLamprecht[m] Would be great to also get https://review.opendev.org/c/openstack/nova/+/958556 reviewed/merged.
13:32:06 sean-k-mooney MaxLamprecht[m]: sure. ill see if i can do it before my next meeting at the top of the hour but i also need to prepare some contet for that
13:32:58 sean-k-mooney it looks like dan is now happy with it and there has only been 1 revision since i last looked so i suspect this will be good to proceed
13:37:46 dansmith I wasn't ever unhappy, I just thought we should wait until after the release
13:42:04 sean-k-mooney ack, i just fisnihed reviewin git i think its ok to proceed with although i wont have tiem to look at the other 2 related repoduces/fixes today
13:45:46 sean-k-mooney im not agaisnt fixing the correct handeling fo build, rescheduler and migate seperatly so that they are indepently backportable bu i am wondering if we need a better long term fix for this in general. i have not spent any time thinkign what that could look like however. im just concerned that unshelve or resize will be next
13:53:13 MaxLamprecht[m] thx, as far what I can tell, these fixes are pretty good for the nova side. We have all of them running since the beginning of this year and our cinder/nova out of sync problems reduced a lot. At least I can't even report a single new case that is nova related currently. I have 2 more race conditions on cinder side that I will submit there.
14:01:16 sean-k-mooney MaxLamprecht[m]: that good context. so you do not currently expect to need to adress simialr failures in other nova codepaths out side of the 2 other you have already sumbited bugs/fixes for?
14:01:32 sean-k-mooney at least based on your current workloads/usage patterens
14:11:53 dansmith sean-k-mooney: oh I was talking about this:
14:11:54 dansmith https://review.opendev.org/c/openstack/nova/+/958556
14:12:04 dansmith this ^ should be a slam dunk early in the cycle I think
14:13:46 sean-k-mooney dansmith: oh ok and ya i think we can proceed with that ill loop back to it whne im finished with the cybrog team meeting
14:34:47 MaxLamprecht[m] sean-k-mooney: Exactly!
15:45:19 melwitt someone has reported a bug related to the tpool.Proxy wrapping of virNodeDevice of PCI devices during update_available_resource() https://bugs.launchpad.net/nova/+bug/2147424 saying that it causes update_available_resource() to increase time taken to complete over time. does anyone have any idea why this would be? if so, comments on the bug would be appreciated
15:48:41 sean-k-mooney did we fix that alreday
15:48:55 sean-k-mooney melwitt: that feels like something we did before
15:50:54 melwitt sean-k-mooney: they are saying that fix caused a different problem, that now update_available_resource() run time progressively increases over time. whereas the problem before the fix was that update_available_resource() could block other nova-compute eventlet threads. they are proposing a revert but I don't see how it could be that simple?
15:52:30 melwitt (i.e. wouldn't that return things to the blocked threads problem)
15:53:23 gibi clearly reverting until we know a bit more about the root cause of the increasing runtime feel premature to me
15:53:29 melwitt I don't really get how tpool.Proxy wrapping could result in a run time increase for the periodic task
15:53:47 dansmith ...if it increases contention
15:53:58 gibi if it is increases for each periodict then we probably leaking something somewhere that accumlates
15:55:10 melwitt yeah. they did test with and without the wrapping and said without the wrapping the run time does not increase
15:56:26 gibi I think we can try to reproduce the degradation. We just need to set the periodic delay to a short enough number and wait
15:58:55 melwitt yeah. I wonder would we need real PCI devices or can this be faked somehow?
15:59:51 dansmith I doubt it requires real PCI devices.. I don't think anything is actually touching the devices there, just getting the list result from libvirt
16:00:36 gibi we do filter for certain devices in that call
16:01:37 gibi https://github.com/openstack/nova/blob/b84864d9781ece8387daa967be5208b859e20d97/nova/virt/libvirt/driver.py#L8905-L8913
16:02:15 gibi if the number of devs are not increasing over time in their env then that call should not increase its runtime over time either
16:07:49 gibi as crawlers killing our infra I cannot set up a new devstack at the moment to poke this
16:08:01 sean-k-mooney it shoudl in theory be repoducable in the functionalt tests
16:08:12 gibi but I added a comment to the bug after couple of retries
16:08:36 sean-k-mooney gibi: so we cache the actual devices form libvirt in that code path
16:08:45 sean-k-mooney so after the first call to libvirt to get the pci devices
16:08:53 sean-k-mooney we should be using it form memory
16:10:00 gibi I don't see the cache, but it is getting late here so...
16:10:01 sean-k-mooney but we might want ot dbouble check that we are not creating new cached copies due to self or similar breaking the caching
16:11:04 melwitt hm I wonder could the cache be making the tpool.Proxy threads stay around and not be cleaned up or something like that?
16:12:24 sean-k-mooney perhaps im trying to find where we do it today to confirm exactly how we are doing that
16:16:51 dansmith gibi: s/our/all/
16:17:30 sean-k-mooney melwitt: im wondering if i am misser remembering and perhasp wrote a caching patch that we have not merged?
16:17:54 gibi dansmith: indeed
16:19:35 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/922855/6/nova/virt/libvirt/driver.py
16:21:09 melwitt oh ok so not yet a factor
16:21:37 sean-k-mooney ya
16:21:51 sean-k-mooney that is not the first time we have looket at that for waht its owrth
16:22:31 sean-k-mooney there are some other optimistaion patches like https://review.opendev.org/c/openstack/nova/+/925382 but i gouth we had merge a ttleast one o the cachign ones before
16:22:38 sean-k-mooney for now that not a factor i guess
16:23:04 melwitt I think the thing I keep focusing on is, why with tpool.Proxy only? seems the only way would be contention like Dan said
16:23:20 sean-k-mooney melwitt: do you know if they are seeing an increase in memory usage as well as time
16:23:39 melwitt I don't really know the details about how tpool.Proxy works. I pulled up the eventlet doc about it but it doesn't say much
16:24:05 melwitt sean-k-mooney: they did not mention memory usage but that would be good to know
16:24:37 sean-k-mooney if we were keeping more nad more info in memoyr over time we shoudl see htat increase.
16:24:39 dansmith "how it works" is "terribly"
16:24:53 dansmith I mean it does what it says but it's really a gross hack we should avoid whenever we can
16:25:08 melwitt well, that's not reassuring
16:25:12 melwitt (haha)
16:26:17 dansmith I can also see it just straight up hurting performance if we wrap an object and then have some really terrible code that hits a bunch of its properties/functions a thousand times, like in a sort routine or something
16:26:24 sean-k-mooney i did look tat the impl a long time ago but i dont rememebr the details
16:26:40 gibi ... OK it seems I got lucky and have at least an aio devstack built with PCI devices now. So I can start experimenting with the periodic
16:26:43 dansmith each of those would be a few instructions for regular code and with the proxy would be a spawn..wait..collect..return instead
16:27:24 sean-k-mooney ya so we are actully collecting the pci device 3 times per call today
16:27:28 dansmith unless we need the actual object to be wrapped, we should wrap the call to libvirt and return the objects, or wrap a wrapper that does the call and copies the objects into like NamedTuples or something
16:27:39 sean-k-mooney that is what https://review.opendev.org/c/openstack/nova/+/925382 fixes
16:28:20 sean-k-mooney but we also dont supprot adding/removing devices at runtime anyway without a agent restart
16:28:38 sean-k-mooney so we can cache this once on the first invocatoin safely today
16:28:48 melwitt it's that the call of virNodeDevice.listCaps() needs to be wrapped somehow to not block stuff. so my thinking was wrap the object and thus get the call wrapped?
16:29:33 sean-k-mooney thats in host.py right
16:30:21 dansmith melwitt: so we need to call listCaps() later and that makes another call into libvirt?
16:30:22 melwitt yes
16:30:48 dansmith ...and do we do that for all the returned devices?
16:31:14 melwitt all of the devices call their listCaps() method so I had thought so. I might be wrong
16:31:46 dansmith so that means each one of those listCaps() calls will be a thread spawn, wait, return with moving some data around for communication

Earlier   Later