| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-04 | |||
| 18:14:56 | sean-k-mooney | on startup we pass a list of all the isntance to the driver via a new process instnace on startup up method after thehost start is done | |
| 18:15:21 | sean-k-mooney | *after the host startup stable uuid check is done | |
| 18:15:26 | dansmith | sean-k-mooney: right we already have the list of instances but this adds another | |
| 18:16:00 | sean-k-mooney | rigfht i jsut removed one duplciate list all isntance on host on startup this cycle | |
| 18:16:08 | sean-k-mooney | and got down to doing it only once | |
| 18:16:20 | sean-k-mooney | so if we can not go back to 2 that would be nice. | |
| 18:16:38 | dansmith | gouthamr: I'm not sure RPC can handle it TBH.. if we have 500 nodes in a cell and all of them are hammering the MQ conductors and DB with over 10k requests every time the periodic runs, or storming them at cluster restart.. that's not going to go well | |
| 18:17:30 | sean-k-mooney | im not really followign the full converstion i just saw scary numbers | |
| 18:18:26 | dansmith | oh actually it's worse | |
| 18:18:49 | dansmith | because then for each share we again query all the instances using that share without a host filter I guess | |
| 18:18:51 | gouthamr | let me look, yeah - i guess i can piggy back on "get_uuids_by_host" - pass it down to the helper that'll get invoked at init, and periodically | |
| 18:19:20 | sean-k-mooney | gouthamr: what are you actully trying to do | |
| 18:19:32 | sean-k-mooney | is this for https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/compute/share_management.py#588 | |
| 18:19:35 | gmaan | ditto, even in native threading mode, RPC is not so cheap and can put other RPC requests in queue. these numbers are really scary :) | |
| 18:20:03 | dansmith | sean-k-mooney: yes | |
| 18:20:07 | dansmith | gmaan: agree | |
| 18:20:12 | gouthamr | sean-k-mooney: yeah, i'm trying to identify migrations that have leaked manila access rules, find them and reap them periodically | |
| 18:20:20 | dansmith | gmaan: native threading will actually let n-cpu hammer the other services even harder | |
| 18:20:29 | gmaan | yeah | |
| 18:21:17 | sean-k-mooney | ok so 1 call to manilla for all share access rules for the current host 1 2 calls to the db 1 for the isntance on this host and one for migrations? | |
| 18:21:39 | dansmith | sean-k-mooney: that's what I suggested but apparently we can't do that? | |
| 18:22:55 | sean-k-mooney | oh ok | |
| 18:23:12 | sean-k-mooney | so to that atht then we woudl neeed to reconstcut it ofrm the nova side | |
| 18:23:39 | sean-k-mooney | what about not makign it a perodic and provide a heal share allocations command liek we did for placement allcoaitns in nova-manage | |
| 18:23:56 | sean-k-mooney | this repair shoudl nto be needed constantly right | |
| 18:24:15 | gouthamr | yeah unfortunately, there's no "give me all access rules for this client" sorta call available in manila.. it's "give me all access rules for this share".. | |
| 18:24:21 | sean-k-mooney | so if we provide a nova-manage command for it operators can use it in the event of a bug | |
| 18:24:52 | sean-k-mooney | and there is no give me all the shares with assess rules for this host command | |
| 18:26:10 | gouthamr | yes | |
| 18:27:16 | sean-k-mooney | ok so host on the share is the hostnaem of the thing exproting the share | |
| 18:27:27 | gouthamr | yes | |
| 18:27:29 | sean-k-mooney | just looking at https://docs.openstack.org/api-ref/shared-file-system/#id17 | |
| 18:27:31 | sean-k-mooney | ack | |
| 18:27:34 | gouthamr | (not compute host) | |
| 18:28:17 | dansmith | sean-k-mooney: weren't you trying to get us to stop doing things like reading proc and sysfs directly in code? | |
| 18:28:31 | dansmith | using nova/filesystem? | |
| 18:28:40 | sean-k-mooney | yes ideally | |
| 18:28:54 | sean-k-mooney | manily ot have a central place that handels the resouce busy issues | |
| 18:28:58 | sean-k-mooney | and make it more testable | |
| 18:29:25 | dansmith | looks like no such handler for proc in there, but this code reads /proc/mounts directly, so probably should go there.. although I'm surprised there's no os module for that | |
| 18:30:16 | sean-k-mooney | the place we had the speic issue that bit us was reading the online state fo cpu cores | |
| 18:30:43 | sean-k-mooney | that was randomly returnign a device busy sortly before/after onlining a core | |
| 18:30:53 | sean-k-mooney | as it was still goting though the state machine in the kernel | |
| 18:31:11 | sean-k-mooney | im not sure if proc would have the same issue | |
| 18:31:18 | dansmith | sean-k-mooney: right this code would not run on macos and so if it's naked in tests will break that compatibility we just achieved | |
| 18:31:22 | sean-k-mooney | since its not actully reading form hardware | |
| 18:31:43 | gouthamr | sean-k-mooney: on the self-healing at the host level vs giving the operator means.. we thought about this earlier.. draining a access needs host-local state (my_shared_fs_storage_ip/CONF.host + "is the share still hard mounted here").. operators could disable this periodic check though and let this only occur at startup (like other optional periodic tasks) | |
| 18:31:52 | sean-k-mooney | oh well yes that too | |
| 18:32:10 | dansmith | yeah making this not self-heal is bad, IMHO | |
| 18:32:21 | dansmith | and requiring cron running nova-manage on compute nodes is no bueno | |
| 18:32:50 | sean-k-mooney | im not agaisnt self healing i just suggested providing a command because we evnetull moved form healign the allcoation automaticly to a commnd | |
| 18:33:10 | sean-k-mooney | but ya i assume its like cider where we neded lcoal state to do that form brick | |
| 18:33:12 | dansmith | yeah but the command has to be run on each compute node, not centrally | |
| 18:33:13 | gouthamr | technically, on macos, you're only running unit tests? | |
| 18:33:28 | sean-k-mooney | dansmith: yes like with volumes | |
| 18:33:56 | sean-k-mooney | wich i also agree is kind of a pain so we can likely park that | |
| 18:34:07 | sean-k-mooney | so the prople is how to do that without melting the rpc bus | |
| 18:34:18 | sean-k-mooney | *problem | |
| 18:34:57 | gouthamr | (i mocked the /proc/mounts calls .. i run unit tests locally on mac, and yes, this would _have_ to be mocked regardless) | |
| 18:36:01 | sean-k-mooney | it sound like a direct remotable obejct method on InstanceList that retuns the list of instnace that have attachment on a given host with the atachmetn info is going to be needed | |
| 18:36:12 | gouthamr | (ese: https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/tests/unit/compute/test_compute_mgr.py#4435) | |
| 18:36:30 | sean-k-mooney | so one rpc that give you only the instnace for a given host that have attachmets with the attachment info | |
| 18:36:53 | sean-k-mooney | dansmith: was the concer only with instnace the were moved sepcificly? | |
| 18:38:57 | sean-k-mooney | doing that filtering on the python side seam like a bad time but we shoudl be able to do it semi effeictly in sql | |
| 18:39:38 | dansmith | gouthamr: mm, nope functional should be working (I'm running now to see if something has regressed) | |
| 18:40:09 | sean-k-mooney | we would need to join on the migration table compute nodes tabel and isntance table and isntance sares table but we shoudl be abel to say give me all instnace that migrated to/from this host that had share attachmetns | |
| 18:40:16 | dansmith | sean-k-mooney: and yeah sounds like maybe a single query method that returns "any share mapping we might care about" would be the best option.. we have a ShareMappingList already which is the right place to put it I thinjk | |
| 18:40:17 | gouthamr | ack, i ran functional too on a mac.. /me is running it again after collapsing the queries | |
| 18:40:50 | sean-k-mooney | ya if its the mappitn we care about that sound better to me as well | |
| 18:41:01 | sean-k-mooney | i was not sure if we wanted the mappitns or the instnace or both | |
| 18:41:12 | dansmith | sean-k-mooney: I'm going to commit comments on the concerns.. would be nice to have your weight applied | |
| 18:41:16 | dansmith | sean-k-mooney: mappings AFAIK | |
| 18:41:26 | gouthamr | yes, mappings | |
| 18:42:13 | sean-k-mooney | sure but as the questions i have been asking imply i have not been folloing the overall work closely | |
| 18:42:18 | melwitt | yeah so it would be get all share mappings for a list of instance uuids rather than doing one by one right | |
| 18:42:26 | sean-k-mooney | but i agree the scalaiblity of the current approch is concerning | |
| 18:42:29 | gouthamr | yeah.. the data model was forcing me with some of these calls in the code.. I'm working on your suggestions to stitch (InstanceList + MigrationList + ShareMappingList) and get a list to parse... one remotable RPC that does all this would help cleanly for sure. but, can i do that in Indri? or would you prefer to have it done now? | |
| 18:45:36 | melwitt | adding a ShareMappingList.get_by_instance_uuids() would be simple enough. do we need more than that? | |
| 18:45:52 | dansmith | gouthamr: what you just described is what I just said we should do in my comment just now | |
| 18:46:15 | sean-k-mooney | or ShareMappingList.get_by_migration_host ? | |
| 18:46:46 | dansmith | gouthamr: I don't think we can/should merge this the way it is, but it would suck to have to punt it all to indri as well.. since you'll have to add RPC changes it won't be backportable either | |
| 18:47:00 | dansmith | melwitt: instance uuids is not enough and it's not even the likely larger list | |
| 18:47:02 | gouthamr | yeah no.. i wasn't looking to backport this stuff | |
| 18:47:48 | gouthamr | its super important for me though to make forward progress on the cold migration :( it makes all of this partially usable before we fix up the hard block on other instance actions that you folks pointed out at the PTG | |
| 18:48:20 | gouthamr | the backport was just going to be for: https://review.opendev.org/c/openstack/nova/+/1001900/ | |
| 18:48:26 | dansmith | yup | |
| 18:48:31 | gouthamr | s/backport/backport request :) | |
| 18:49:41 | gouthamr | ty for this discussion and the comments! | |
| 18:53:15 | sean-k-mooney | gouthamr: i have not looked at https://review.opendev.org/c/openstack/nova/+/1001900 in detail but just looking at the tile im not sure that is backportable | |
| 18:53:56 | gouthamr | sean-k-mooney: oh, read on :P yes, we're moving auth, but it doesn't break anything running until you choose | |
| 18:54:11 | sean-k-mooney | it might be but we have to supprot n-2 compatiblity | |
| 18:54:15 | gouthamr | sean-k-mooney: this will be supplemented with an OSSN/operator guidance on how the move will occur | |
| 18:55:16 | sean-k-mooney | so we need atuth to work with a patched souce node and unpatch dest | |
| 18:55:20 | sean-k-mooney | as an example | |
| 18:55:37 | sean-k-mooney | its too late on a friday for me to really ingest the details | |
| 18:55:56 | gouthamr | sean-k-mooney: it will; on the unpatched destination, we'll use a "shared credential" until that's patched some day | |
| 18:56:32 | gouthamr | sean-k-mooney: yes :) ty for the brainstorm! have a great weekend | |
| 18:57:44 | dansmith | sean-k-mooney: it's really not ideal, I've already raised the concerns | |
| 18:58:13 | sean-k-mooney | ack, im not really sure that shareing a cephx id is a problme | |
| 18:58:17 | dansmith | sean-k-mooney: the virtiofs implementation in the tree is sort of "so leaky that it doesn't break upgrades" which is not good but also means that we _can't_ do automatic cleanup there anyway | |