Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-03
20:32:04 dansmith this is why I'm asking
20:32:32 dansmith because there's not much visibility into when and by what deterministic process this would clean up
20:33:09 gouthamr on the manila side?
20:33:45 gouthamr JayF: yeah, i plan to write it out.. nova-manage isn't needed here because no cruft is left on nova
20:33:45 JayF gouthamr: here's the key question: in order for the operator to see security benefit, they have to drop the shared "nova" credential. In a large, public-cloud-sized nova deployment, are we giving folks the tools they need to determine they *can* do that?\
20:33:55 dansmith so I want to make sure that (a) we're not going to break existing deployments that use the old key (doesn't sound like we will) but that (b) we're not going to create a thing that will only be orphaned in five years when the last thing stops using it, and the operator won't be able to revoke between now and then nor set their alarm for when they can safely clean it up
20:34:15 dansmith JayF: exactamundo
20:34:38 dansmith gouthamr: the reference is stored in your object is it not?
20:35:16 gouthamr (b) is genuinely a worry - you can lose track.. that can happen today.. there's no real way of saying "are you using this credential" (its a universal "problem" with network-attached-storage if you want to generalize)
20:35:57 gouthamr dansmith: no we're actually not storing any metadata specific to an instance on manila.. we're locking the share from deletion, but the lock just says: "locked by nova"
20:36:21 gouthamr i actually think it'd be fine for the lock to say "locked by nova for instance: UUID"
20:36:39 gouthamr but i don't know if Uggla gibi thought that wasn't necessary when they wrote this up
20:36:42 dansmith gouthamr: it's stored in your access_to field in the nova object we store in the DB?
20:37:16 dansmith s/?/right?/
20:37:59 gouthamr dansmith: yeah, manila knows what "access_to" is.. and today there's a shared hardcoded access_to: "nova".. you can filter on this stuff and isolate what all shares have this access
20:38:53 dansmith gouthamr: I'm saying.. there _is_ cruft left in nova (you asserted not above) in the form of the access_to in the object, _and_ that we _could_ provide what JayF and I are asking, which is a roadmap/status display of what has/has not migrated
20:40:06 dansmith right exactly
20:40:09 dansmith and that goes to my other question/concern,
20:40:38 dansmith which is that we're basically giving $specificName a "random" (or at least obfuscated) value which also makes it hard to track.. which is why I asked if it was secret
20:40:46 dansmith as the obfuscation makes it harder to roll your own toolage
20:40:48 gouthamr dansmith: where is this cruft? ShareMapping has no "access_to"... It was hardcoded in the code: https://github.com/openstack/nova/blob/ad5e2ea26e3bd80a84e0ab205ceef4200906ded1/nova/db/main/models.py#L756-L797
20:41:52 gouthamr dansmith: is it hard though? the HOST field is known, instance IDs are known, you can query manila and get all the rules that map with this combination and the obfuscation logic
20:41:53 dansmith gouthamr: I hadn't looked at the lower layers.. so your object has the access_to field but you never persist it?
20:42:15 gouthamr dansmith: by my object, you mean manila's database? :)
20:42:28 dansmith gouthamr: you have to start with every host and instance and synthesize all the hashes and then subtract from that global list the ones you see in use
20:42:39 dansmith gouthamr: no, I mean the nova/objects/ object
20:42:47 dansmith gouthamr: this: https://review.opendev.org/c/openstack/nova/+/1001900/6/nova/objects/share_mapping.py#41
20:43:03 JayF dansmith: LN 58 in that file
20:43:13 JayF dansmith: specifically indicates the access_* field is ephemeral
20:43:26 dansmith ack indeed
20:44:15 dansmith okay so no stored cruft in nova.. that's good.. but it would also be (have been) the easiest way to harvest the "how close am I to done?" information from a database I think
20:47:11 dansmith gouthamr: if it's never persisted does that mean we either (a) can get it back from manila or (b) never actually need it again? meaning we can clean up the access rules without the access_to information?
20:47:41 gouthamr imv, the obfuscation is deterministic, and harmless.. solves problems for me not to validate what goes into CONF.host and yet retain something that determines the host identity..
20:48:10 dansmith yeah the "host could really be anything" argument holds weight no doubt
20:48:28 gouthamr dansmith: yes, nova still doesn't need it to be stored; the recomputation thingy kicks in with each grant/deny
20:48:44 gouthamr dansmith: a human would have to pay the cost, but do they really need to?
20:48:45 dansmith gouthamr: okay but that's my question
20:49:14 dansmith gouthamr: if we have an instance today, we apply this patch, then we go to delete the access rules, we can't just recompute access_to right?
20:49:48 gouthamr dansmith: yes we can.. we're doing that
20:50:12 gouthamr dansmith: that recomputing helper is called on both grant/deny
20:51:13 dansmith it is, but I don't understand how it can know
20:52:07 dansmith can we gmeet on this so we can resolve quicker? if not I need to punt I think
20:52:47 dansmith just a few minutes I'm sure you can explain to me what I'm missing
20:53:34 dansmith meet.google.com/efg-wrjv-ivw
21:10:13 dansmith sorry
21:10:16 dansmith give me 10 mins
#openstack-nova - 2026-09-04
03:22:03 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900
03:22:04 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633
03:22:04 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349
03:22:05 opendevreview Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916
05:34:19 opendevreview Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
05:53:08 opendevreview Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910
09:43:16 Uggla gouthamr "but i don't know if Uggla gibi thought that wasn't necessary when they wrote this up", If I remember well nothing was really specified about it. I set something just to highlight the share was somewhere managed by nova.
12:26:44 opendevreview Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410
13:17:00 opendevreview ribaudr proposed openstack/nova master: Add regression test for bug 2162835 https://review.opendev.org/c/openstack/nova/+/999977
13:17:01 opendevreview ribaudr proposed openstack/nova master: Fix CPU pinning reset during evacuate periodic race https://review.opendev.org/c/openstack/nova/+/999978
13:19:01 opendevreview Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410
15:17:17 opendevreview Balazs Gibizer proposed openstack/nova master: Fix parallel evacuations pushing instance to ERROR https://review.opendev.org/c/openstack/nova/+/1003994
15:17:25 gibi this is a nasty one ^^
15:17:53 gibi (all hail the automatic evacuation engines hammering the nova-api with parallel evac requests)
15:30:11 opendevreview Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910
15:30:23 opendevreview Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
15:36:36 gmaan gibi: regarding the test_migrate_disk_and_power_off_crash_finish_revert_migration failure, I think I am going in same direction as you commented on the ashish patch. consider the graceful_shutdown_timeout in polling the instance state and if execution is more for this test then we can reduce the graceful_shutdown_timeout for this test to make graceful shutdown fast.
15:37:20 gmaan I am testing the polliing fix first and try to reduce the timeout.
15:38:19 gmaan i tried to find the ashish change but could not due to its topic, i tried to check if any effort to run this test in threading mode and only searched eventlet-removal gerrit topic
15:43:22 gmaan ashigupt: ^^ and updated the topic for your change
15:53:17 gmaan dansmith: there is one bug for the task tracking leaking the live migration task on destination, its fix it ready if you hav time to look :https://review.opendev.org/c/openstack/nova/+/1003386/1
15:54:21 dansmith gmaan: ack I'm trying to focus on the manila FFE today since we can fix bugs later
15:54:34 gmaan sure
16:32:07 opendevreview Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910
16:36:29 ashigupt gmaan, ack thankyou
16:37:09 gmaan ashigupt: let me test the fix on my change and then I can update you, something you can check on monday
16:41:05 ashigupt gmaan: sure thank you
16:51:51 opendevreview Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266
18:01:52 dansmith melwitt: gouthamr I'm a bit unahppy with the double-nested RPC-to-DB calls in the reconcile patch
18:03:32 dansmith we can get a list of all the instances on the current host without doing the list all by host.. and can't we add another query to the object to get all mappings for a list of uuids? that would turn it from n^2 queries to 2
18:05:25 dansmith because we're also doing a share object lookup for every migration we find, even old long-since-confirmed migrations forever and ever it seems
18:06:19 dansmith if we had one "get all share mappings for list-of-uuids" call we could do that once with all the uuids we're looking for, from both the instance list and migration list
18:11:14 dansmith like, let's say we have 500 instances on this host, none of which use shares currently.. and 10k migrations over the last several years, few of which are still on this host and none of which used shares.. we're going to do 10500 RPC calls to DB queries at startup and every hour until the cows come home right?
18:12:35 dansmith for operators that do slide puzzle migrations, every instance running in the system will have gone through every host eventually and those will all have confirmed migration records for any host they've ever been on AFAIK
18:14:06 dansmith (sorry n^2 queries above is not right of course, it's just n^2 iterations)
18:14:26 gouthamr dansmith: ack, yeah no this falls on that scale - although RPC can handle it, its inefficient.. so, get all instance UUIDs on this host, union in the instance UUIDs from the migration list, then one ShareMappingList.get_by_instance_uuids for that whole set... and iterate that instead...
18:14:49 dansmith cc Uggla since you reviewed too ^
18:14:56 sean-k-mooney on startup we pass a list of all the isntance to the driver via a new process instnace on startup up method after thehost start is done
18:15:21 sean-k-mooney *after the host startup stable uuid check is done
18:15:26 dansmith sean-k-mooney: right we already have the list of instances but this adds another
18:16:00 sean-k-mooney rigfht i jsut removed one duplciate list all isntance on host on startup this cycle
18:16:08 sean-k-mooney and got down to doing it only once
18:16:20 sean-k-mooney so if we can not go back to 2 that would be nice.
18:16:38 dansmith gouthamr: I'm not sure RPC can handle it TBH.. if we have 500 nodes in a cell and all of them are hammering the MQ conductors and DB with over 10k requests every time the periodic runs, or storming them at cluster restart.. that's not going to go well
18:17:30 sean-k-mooney im not really followign the full converstion i just saw scary numbers
18:18:26 dansmith oh actually it's worse
18:18:49 dansmith because then for each share we again query all the instances using that share without a host filter I guess
18:18:51 gouthamr let me look, yeah - i guess i can piggy back on "get_uuids_by_host" - pass it down to the helper that'll get invoked at init, and periodically
18:19:20 sean-k-mooney gouthamr: what are you actully trying to do
18:19:32 sean-k-mooney is this for https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/compute/share_management.py#588
18:19:35 gmaan ditto, even in native threading mode, RPC is not so cheap and can put other RPC requests in queue. these numbers are really scary :)
18:20:03 dansmith sean-k-mooney: yes
18:20:07 dansmith gmaan: agree
18:20:12 gouthamr sean-k-mooney: yeah, i'm trying to identify migrations that have leaked manila access rules, find them and reap them periodically

Earlier   Later