| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-04 | |||
| 09:43:16 | Uggla | gouthamr "but i don't know if Uggla gibi thought that wasn't necessary when they wrote this up", If I remember well nothing was really specified about it. I set something just to highlight the share was somewhere managed by nova. | |
| 12:26:44 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | |
| 13:17:00 | opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug 2162835 https://review.opendev.org/c/openstack/nova/+/999977 | |
| 13:17:01 | opendevreview | ribaudr proposed openstack/nova master: Fix CPU pinning reset during evacuate periodic race https://review.opendev.org/c/openstack/nova/+/999978 | |
| 13:19:01 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | |
| 15:17:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix parallel evacuations pushing instance to ERROR https://review.opendev.org/c/openstack/nova/+/1003994 | |
| 15:17:25 | gibi | this is a nasty one ^^ | |
| 15:17:53 | gibi | (all hail the automatic evacuation engines hammering the nova-api with parallel evac requests) | |
| 15:30:11 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910 | |
| 15:30:23 | opendevreview | Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | |
| 15:36:36 | gmaan | gibi: regarding the test_migrate_disk_and_power_off_crash_finish_revert_migration failure, I think I am going in same direction as you commented on the ashish patch. consider the graceful_shutdown_timeout in polling the instance state and if execution is more for this test then we can reduce the graceful_shutdown_timeout for this test to make graceful shutdown fast. | |
| 15:37:20 | gmaan | I am testing the polliing fix first and try to reduce the timeout. | |
| 15:38:19 | gmaan | i tried to find the ashish change but could not due to its topic, i tried to check if any effort to run this test in threading mode and only searched eventlet-removal gerrit topic | |
| 15:43:22 | gmaan | ashigupt: ^^ and updated the topic for your change | |
| 15:53:17 | gmaan | dansmith: there is one bug for the task tracking leaking the live migration task on destination, its fix it ready if you hav time to look :https://review.opendev.org/c/openstack/nova/+/1003386/1 | |
| 15:54:21 | dansmith | gmaan: ack I'm trying to focus on the manila FFE today since we can fix bugs later | |
| 15:54:34 | gmaan | sure | |
| 16:32:07 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910 | |
| 16:36:29 | ashigupt | gmaan, ack thankyou | |
| 16:37:09 | gmaan | ashigupt: let me test the fix on my change and then I can update you, something you can check on monday | |
| 16:41:05 | ashigupt | gmaan: sure thank you | |
| 16:51:51 | opendevreview | Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | |
| 18:01:52 | dansmith | melwitt: gouthamr I'm a bit unahppy with the double-nested RPC-to-DB calls in the reconcile patch | |
| 18:03:32 | dansmith | we can get a list of all the instances on the current host without doing the list all by host.. and can't we add another query to the object to get all mappings for a list of uuids? that would turn it from n^2 queries to 2 | |
| 18:05:25 | dansmith | because we're also doing a share object lookup for every migration we find, even old long-since-confirmed migrations forever and ever it seems | |
| 18:06:19 | dansmith | if we had one "get all share mappings for list-of-uuids" call we could do that once with all the uuids we're looking for, from both the instance list and migration list | |
| 18:11:14 | dansmith | like, let's say we have 500 instances on this host, none of which use shares currently.. and 10k migrations over the last several years, few of which are still on this host and none of which used shares.. we're going to do 10500 RPC calls to DB queries at startup and every hour until the cows come home right? | |
| 18:12:35 | dansmith | for operators that do slide puzzle migrations, every instance running in the system will have gone through every host eventually and those will all have confirmed migration records for any host they've ever been on AFAIK | |
| 18:14:06 | dansmith | (sorry n^2 queries above is not right of course, it's just n^2 iterations) | |
| 18:14:26 | gouthamr | dansmith: ack, yeah no this falls on that scale - although RPC can handle it, its inefficient.. so, get all instance UUIDs on this host, union in the instance UUIDs from the migration list, then one ShareMappingList.get_by_instance_uuids for that whole set... and iterate that instead... | |
| 18:14:49 | dansmith | cc Uggla since you reviewed too ^ | |
| 18:14:56 | sean-k-mooney | on startup we pass a list of all the isntance to the driver via a new process instnace on startup up method after thehost start is done | |
| 18:15:21 | sean-k-mooney | *after the host startup stable uuid check is done | |
| 18:15:26 | dansmith | sean-k-mooney: right we already have the list of instances but this adds another | |
| 18:16:00 | sean-k-mooney | rigfht i jsut removed one duplciate list all isntance on host on startup this cycle | |
| 18:16:08 | sean-k-mooney | and got down to doing it only once | |
| 18:16:20 | sean-k-mooney | so if we can not go back to 2 that would be nice. | |
| 18:16:38 | dansmith | gouthamr: I'm not sure RPC can handle it TBH.. if we have 500 nodes in a cell and all of them are hammering the MQ conductors and DB with over 10k requests every time the periodic runs, or storming them at cluster restart.. that's not going to go well | |
| 18:17:30 | sean-k-mooney | im not really followign the full converstion i just saw scary numbers | |
| 18:18:26 | dansmith | oh actually it's worse | |
| 18:18:49 | dansmith | because then for each share we again query all the instances using that share without a host filter I guess | |
| 18:18:51 | gouthamr | let me look, yeah - i guess i can piggy back on "get_uuids_by_host" - pass it down to the helper that'll get invoked at init, and periodically | |
| 18:19:20 | sean-k-mooney | gouthamr: what are you actully trying to do | |
| 18:19:32 | sean-k-mooney | is this for https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/compute/share_management.py#588 | |
| 18:19:35 | gmaan | ditto, even in native threading mode, RPC is not so cheap and can put other RPC requests in queue. these numbers are really scary :) | |
| 18:20:03 | dansmith | sean-k-mooney: yes | |
| 18:20:07 | dansmith | gmaan: agree | |
| 18:20:12 | gouthamr | sean-k-mooney: yeah, i'm trying to identify migrations that have leaked manila access rules, find them and reap them periodically | |
| 18:20:20 | dansmith | gmaan: native threading will actually let n-cpu hammer the other services even harder | |
| 18:20:29 | gmaan | yeah | |
| 18:21:17 | sean-k-mooney | ok so 1 call to manilla for all share access rules for the current host 1 2 calls to the db 1 for the isntance on this host and one for migrations? | |
| 18:21:39 | dansmith | sean-k-mooney: that's what I suggested but apparently we can't do that? | |
| 18:22:55 | sean-k-mooney | oh ok | |
| 18:23:12 | sean-k-mooney | so to that atht then we woudl neeed to reconstcut it ofrm the nova side | |
| 18:23:39 | sean-k-mooney | what about not makign it a perodic and provide a heal share allocations command liek we did for placement allcoaitns in nova-manage | |
| 18:23:56 | sean-k-mooney | this repair shoudl nto be needed constantly right | |
| 18:24:15 | gouthamr | yeah unfortunately, there's no "give me all access rules for this client" sorta call available in manila.. it's "give me all access rules for this share".. | |
| 18:24:21 | sean-k-mooney | so if we provide a nova-manage command for it operators can use it in the event of a bug | |
| 18:24:52 | sean-k-mooney | and there is no give me all the shares with assess rules for this host command | |
| 18:26:10 | gouthamr | yes | |
| 18:27:16 | sean-k-mooney | ok so host on the share is the hostnaem of the thing exproting the share | |
| 18:27:27 | gouthamr | yes | |
| 18:27:29 | sean-k-mooney | just looking at https://docs.openstack.org/api-ref/shared-file-system/#id17 | |
| 18:27:31 | sean-k-mooney | ack | |
| 18:27:34 | gouthamr | (not compute host) | |
| 18:28:17 | dansmith | sean-k-mooney: weren't you trying to get us to stop doing things like reading proc and sysfs directly in code? | |
| 18:28:31 | dansmith | using nova/filesystem? | |
| 18:28:40 | sean-k-mooney | yes ideally | |
| 18:28:54 | sean-k-mooney | manily ot have a central place that handels the resouce busy issues | |
| 18:28:58 | sean-k-mooney | and make it more testable | |
| 18:29:25 | dansmith | looks like no such handler for proc in there, but this code reads /proc/mounts directly, so probably should go there.. although I'm surprised there's no os module for that | |
| 18:30:16 | sean-k-mooney | the place we had the speic issue that bit us was reading the online state fo cpu cores | |
| 18:30:43 | sean-k-mooney | that was randomly returnign a device busy sortly before/after onlining a core | |
| 18:30:53 | sean-k-mooney | as it was still goting though the state machine in the kernel | |
| 18:31:11 | sean-k-mooney | im not sure if proc would have the same issue | |
| 18:31:18 | dansmith | sean-k-mooney: right this code would not run on macos and so if it's naked in tests will break that compatibility we just achieved | |
| 18:31:22 | sean-k-mooney | since its not actully reading form hardware | |
| 18:31:43 | gouthamr | sean-k-mooney: on the self-healing at the host level vs giving the operator means.. we thought about this earlier.. draining a access needs host-local state (my_shared_fs_storage_ip/CONF.host + "is the share still hard mounted here").. operators could disable this periodic check though and let this only occur at startup (like other optional periodic tasks) | |
| 18:31:52 | sean-k-mooney | oh well yes that too | |
| 18:32:10 | dansmith | yeah making this not self-heal is bad, IMHO | |
| 18:32:21 | dansmith | and requiring cron running nova-manage on compute nodes is no bueno | |
| 18:32:50 | sean-k-mooney | im not agaisnt self healing i just suggested providing a command because we evnetull moved form healign the allcoation automaticly to a commnd | |
| 18:33:10 | sean-k-mooney | but ya i assume its like cider where we neded lcoal state to do that form brick | |
| 18:33:12 | dansmith | yeah but the command has to be run on each compute node, not centrally | |
| 18:33:13 | gouthamr | technically, on macos, you're only running unit tests? | |
| 18:33:28 | sean-k-mooney | dansmith: yes like with volumes | |
| 18:33:56 | sean-k-mooney | wich i also agree is kind of a pain so we can likely park that | |
| 18:34:07 | sean-k-mooney | so the prople is how to do that without melting the rpc bus | |
| 18:34:18 | sean-k-mooney | *problem | |
| 18:34:57 | gouthamr | (i mocked the /proc/mounts calls .. i run unit tests locally on mac, and yes, this would _have_ to be mocked regardless) | |
| 18:36:01 | sean-k-mooney | it sound like a direct remotable obejct method on InstanceList that retuns the list of instnace that have attachment on a given host with the atachmetn info is going to be needed | |
| 18:36:12 | gouthamr | (ese: https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/tests/unit/compute/test_compute_mgr.py#4435) | |
| 18:36:30 | sean-k-mooney | so one rpc that give you only the instnace for a given host that have attachmets with the attachment info | |
| 18:36:53 | sean-k-mooney | dansmith: was the concer only with instnace the were moved sepcificly? | |
| 18:38:57 | sean-k-mooney | doing that filtering on the python side seam like a bad time but we shoudl be able to do it semi effeictly in sql | |
| 18:39:38 | dansmith | gouthamr: mm, nope functional should be working (I'm running now to see if something has regressed) | |
| 18:40:09 | sean-k-mooney | we would need to join on the migration table compute nodes tabel and isntance table and isntance sares table but we shoudl be abel to say give me all instnace that migrated to/from this host that had share attachmetns | |
| 18:40:16 | dansmith | sean-k-mooney: and yeah sounds like maybe a single query method that returns "any share mapping we might care about" would be the best option.. we have a ShareMappingList already which is the right place to put it I thinjk | |
| 18:40:17 | gouthamr | ack, i ran functional too on a mac.. /me is running it again after collapsing the queries | |
| 18:40:50 | sean-k-mooney | ya if its the mappitn we care about that sound better to me as well | |