| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-sdks - 2026-09-23 | |||
| 13:53:10 | andreykurilin | The problem - Horizon of 2026.1 started to leak memory. Quite a lot in comparison to 2025.1 . After digging into an affected process, I found ~3,5k of openstacksdk objects in the memory. Each one stores a connection pool with a complete chain of other objects. Horizon never calls close for most of openstack connection it creates, which means that all of them (as far as I understand) keeps to be attached to uwsgi process due to | |
| 13:53:10 | andreykurilin | atexit.register | |
| 13:56:10 | andreykurilin | https://github.com/psf/requests/pull/6767 changed the behavior of default ssl context. Every pooled connection now owns a CA-loaded SSLContext which makes the difference in memory consumption by leaked objects | |
| 14:38:25 | stephenfin | andreykurilin: What do you think is the issue from the SDK side? | |
| 14:41:03 | andreykurilin | `atexit.register` call which makes openstack conn “immortal” :) I’m doing some experiments right now, it seems like switching to weakref.finalize should solve the problem - it would fire close “as soon” as openstacksdk object is no longer used | |
| 14:43:52 | andreykurilin | @stephenfin https://github.com/openstack/openstacksdk/blob/stable/2026.1/openstack/cloud/openstackcloud.py#L279-L280 this part I’m talking about | |
| 14:45:05 | stephenfin | as you likely know, the whole point of that is to ensure things are correctly closed before the Python process exits. gtema is out this week (I think) but he added that some time back https://github.com/openstack/openstacksdk/commit/2b4aeff6d3972761da9c4a6a8b696103664a758b | |
| 14:46:01 | andreykurilin | Yes, it has been there since 2023. I believe it has not caused much problems as leaked requests connections were cheap | |
| 14:46:56 | andreykurilin | Having thousands of connections in memory was taking like couple megabytes as they were sharing the same ssl context | |
| 14:49:16 | andreykurilin | Horizon (uwsgi) is long running process which means that atexit may never fire until redeploy or OOM | |
| 14:51:24 | stephenfin | Right. My reason for pointing that out wasn't to say that this isn't an issue because it's old code. Rather, it was merely to point out that the code is there for a reason | |
| 14:52:32 | stephenfin | I wonder how practical it would be to allow Connection to be used as a context manager and to use that in-tree in Horizon. Or just have horizon ensure it calls `close()` when it's done? | |
| 14:53:01 | andreykurilin | Considering the review speed of horizon changes - near impossible | |
| 14:54:35 | andreykurilin | + it is hard to ensure that all horizon dashboards follow this. For example, octavia-dashboard may create dozens openstacksdk objs for a single user request, no memorize usage | |
| 14:54:46 | stephenfin | fair points | |
| 14:56:29 | stephenfin | weakref.finalize sounds like a good shout. I recall us using that in one of the oslo projects at some point for a similar issue. Neutron uses it too, iirc (?) | |
| 14:56:50 | stephenfin | codesearch.o.o tells we oslo.rootwrap is the lib I was thinking of | |
| 14:56:53 | stephenfin | *me | |
| 15:05:07 | andreykurilin | Ok, I can submit a change for that. Probably, I can try to run it for sometime in our prod to ensure that it fixes horizon memory leakage (5 uwsgi processes ate 30Gb of ram, ridiculous) | |
| 15:26:20 | opendevreview | Andriy Kurilin proposed openstack/openstacksdk master: Replace atexit with weakref.finalize https://review.opendev.org/c/openstack/openstacksdk/+/1006972 | |
| 16:03:29 | stephenfin | andreykurilin: Change looks fine. Couple of small requests though assuming we want to backport that | |
| 16:10:37 | rm_work | stephenfin: do you know if danchild is here? or somewhere to chat with? | |
| 16:11:07 | stephenfin | I don't think so. He should be on k8s Slack though | |
| 16:11:38 | stephenfin | (btw, I have seen your comments on #openstack-oslo. I just haven't gotten back to that series yet) | |
| 16:24:53 | rm_work | yeah no worries | |
| 16:25:01 | rm_work | I'm back to working on the oslo.messaging side | |
| 16:26:21 | rm_work | has been a lot of good collab with Furukawa-san on the LINE side there (and they are the ones that actually use Consul) so I didn't want to change TOO much from what he had last tested and confirmed, but all of those changes were small enough (and the split was identical at the end of the chain to the original) so that was fine | |
| 16:28:00 | rm_work | we've both done pretty extensive testing on the functionality, the code itself though is ... mostly copied from the original driver and modified/extended by AI, so ... yeah, trying not to give you giant slop bomb to review, but it definitely falls into the category of "I have personally reviewed the shape and the behavior but not every line" T_T | |
| 16:28:23 | rm_work | same with gophercloud | |
| 16:29:12 | rm_work | my testing has been pretty exhaustive, and the code is live and in use actively (for both gopher and rpc actually) in my production cloud deployments, so that is the confidence level I have at least 😄 | |
| 16:58:53 | opendevreview | Andriy Kurilin proposed openstack/openstacksdk master: Replace atexit with weakref.finalize https://review.opendev.org/c/openstack/openstacksdk/+/1006972 | |
| 17:34:33 | opendevreview | Brian Haley proposed openstack/openstacksdk master: Make openstacksdk-functional-devstack-networking-ext OVN-based https://review.opendev.org/c/openstack/openstacksdk/+/1006592 | |
| 19:24:28 | opendevreview | Merged openstack/python-openstackclient master: Migrate accelerator commands from python-cyborgclient https://review.opendev.org/c/openstack/python-openstackclient/+/1004707 | |
| 19:24:30 | opendevreview | Merged openstack/python-openstackclient master: Add functional tests and Zuul job for accelerator commands https://review.opendev.org/c/openstack/python-openstackclient/+/996698 | |
| 21:09:34 | opendevreview | melanie witt proposed openstack/openstacksdk master: accelerator: Add device status field and enable/disable support https://review.opendev.org/c/openstack/openstacksdk/+/1004720 | |
| 22:00:41 | opendevreview | melanie witt proposed openstack/openstacksdk master: accelerator: Add device status field and enable/disable support https://review.opendev.org/c/openstack/openstacksdk/+/1004720 | |
| #openstack-sdks - 2026-09-24 | |||
| 08:54:13 | opendevreview | Silvia Wachira proposed openstack/python-openstackclient master: share: Add share server list, show, delete commands https://review.opendev.org/c/openstack/python-openstackclient/+/990746 | |
| 08:54:13 | opendevreview | Silvia Wachira proposed openstack/python-openstackclient master: share: Add share network commands https://review.opendev.org/c/openstack/python-openstackclient/+/990817 | |
| 08:54:14 | opendevreview | Silvia Wachira proposed openstack/python-openstackclient master: share: Fix IGNORED_MODULES entry to load new share module https://review.opendev.org/c/openstack/python-openstackclient/+/1000241 | |
| 08:54:14 | opendevreview | Silvia Wachira proposed openstack/python-openstackclient master: zuul: Enable Manila devstack plugin for functional tests https://review.opendev.org/c/openstack/python-openstackclient/+/1000266 | |
| 10:16:48 | opendevreview | Grzegorz Grasza proposed openstack/openstacksdk master: Add OpenStack-MFA-Passcode header support for identity https://review.opendev.org/c/openstack/openstacksdk/+/1007125 | |
| 10:17:07 | opendevreview | Grzegorz Grasza proposed openstack/python-openstackclient master: Add --mfa-passcode for MFA-sensitive identity commands https://review.opendev.org/c/openstack/python-openstackclient/+/1007127 | |
| 12:02:47 | opendevreview | Merged openstack/openstacksdk master: Replace atexit with weakref.finalize https://review.opendev.org/c/openstack/openstacksdk/+/1006972 | |
| 12:13:27 | opendevreview | Stephen Finucane proposed openstack/openstacksdk master: Clean up connection resources upon garbage collection https://review.opendev.org/c/openstack/openstacksdk/+/1007141 | |
| 12:38:25 | opendevreview | Merged openstack/openstacksdk master: accelerator: Add device status field and enable/disable support https://review.opendev.org/c/openstack/openstacksdk/+/1004720 | |
| 14:39:36 | opendevreview | Silvia Wachira proposed openstack/python-openstackclient master: zuul: Enable Manila devstack plugin for functional tests https://review.opendev.org/c/openstack/python-openstackclient/+/1000266 | |
| 14:39:37 | opendevreview | Silvia Wachira proposed openstack/python-openstackclient master: share: Add share network commands https://review.opendev.org/c/openstack/python-openstackclient/+/990817 | |
| 14:39:37 | opendevreview | Silvia Wachira proposed openstack/python-openstackclient master: share: Fix IGNORED_MODULES entry to load new share module https://review.opendev.org/c/openstack/python-openstackclient/+/1000241 | |
| #openstack-sdks - 2026-09-25 | |||
| 06:25:54 | opendevreview | Pranali Deore proposed openstack/openstacksdk master: image: Parse hw_vif_multiqueue_enabled as BoolStr https://review.opendev.org/c/openstack/openstacksdk/+/1006895 | |
| 15:34:04 | opendevreview | Grzegorz Grasza proposed openstack/keystoneauth master: Mark OAuth1 consumer and access secrets as sensitive opts https://review.opendev.org/c/openstack/keystoneauth/+/1005684 | |
| 18:43:33 | opendevreview | Doug Goldstein proposed openstack/openstacksdk master: Support multiple named scopes per cloud in clouds.yaml https://review.opendev.org/c/openstack/openstacksdk/+/1007536 | |
| 18:53:23 | cardoe | stephenfin: ^ isn't a real patch... it's a conversation starter | |
| 20:23:29 | opendevreview | Merged openstack/openstacksdk master: Clean up connection resources upon garbage collection https://review.opendev.org/c/openstack/openstacksdk/+/1007141 | |
| #openstack-sdks - 2026-09-26 | |||
| 19:15:08 | opendevreview | taha hosseini proposed openstack/cliff master: Normalize sort column names https://review.opendev.org/c/openstack/cliff/+/1007599 | |
| 19:17:15 | opendevreview | taha hosseini proposed openstack/python-openstackclient master: Normalize additional server list columns https://review.opendev.org/c/openstack/python-openstackclient/+/1007600 | |
| #openstack-sdks - 2026-09-28 | |||
| 13:39:20 | opendevreview | Merged openstack/python-openstackclient master: Normalize additional server list columns https://review.opendev.org/c/openstack/python-openstackclient/+/1007600 | |
| 18:10:29 | opendevreview | Merged openstack/openstacksdk master: image: Parse hw_vif_multiqueue_enabled as BoolStr https://review.opendev.org/c/openstack/openstacksdk/+/1006895 | |
| #openstack-sdks - 2026-09-29 | |||
| 06:51:31 | opendevreview | taha hosseini proposed openstack/cliff master: Normalize sort column names https://review.opendev.org/c/openstack/cliff/+/1007599 | |
| 11:33:57 | opendevreview | Merged openstack/cliff master: Normalize sort column names https://review.opendev.org/c/openstack/cliff/+/1007599 | |
| #openstack-sdks - 2026-09-30 | |||
| 13:37:13 | opendevreview | Quentin GROLLEAU proposed openstack/codegenerator master: fix(openapi): Add block-storage version discovery route https://review.opendev.org/c/openstack/codegenerator/+/1008061 | |
| 15:03:39 | melwitt | stephenfin: hey I dunno if you saw my replies to your questions on https://review.opendev.org/c/openstack/python-openstackclient/+/1006597 | |
| 15:05:48 | stephenfin | I had not. Let me reply there | |
| 15:05:54 | melwitt | cool thanks | |
| 15:20:27 | stephenfin | melwitt: a'ight, replied. All your comments make sense. I'd like to see a few TODOs added and one comment updated. lmk if you'd rather do that in that patch or in a follow-up | |
| 15:23:33 | melwitt | stephenfin: super, thanks. I can update this patch, np | |
| 15:53:57 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Implements volume attachments list query options. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| #openstack-sdks - 2026-10-01 | |||
| 07:06:57 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Implements volume attachments list query options. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 08:27:31 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Implements volume attachments list query options. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 08:39:53 | opendevreview | Thomas Goirand proposed openstack/python-openstackclient master: Fix broken backward compat https://review.opendev.org/c/openstack/python-openstackclient/+/1008216 | |
| 09:17:14 | opendevreview | Thomas Goirand proposed openstack/python-openstackclient master: Fix broken backward compat https://review.opendev.org/c/openstack/python-openstackclient/+/1008216 | |
| 09:42:07 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Implements volume attachments list query options. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 10:39:08 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Implements volume attachments list query options. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 13:12:51 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Implements volume attachments list query options. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 14:16:35 | opendevreview | Rico Lin proposed openstack/ansible-collections-openstack master: Fix quota updates with older openstacksdk releases https://review.opendev.org/c/openstack/ansible-collections-openstack/+/1008275 | |
| 16:42:14 | opendevreview | melanie witt proposed openstack/python-openstackclient master: Rename and deprecate legacy options in accelerator commands https://review.opendev.org/c/openstack/python-openstackclient/+/1006597 | |
| 16:45:53 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Implements volume attachments list query options. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| #openstack-sdks - 2026-10-02 | |||
| 05:54:00 | opendevreview | Tore Anderson proposed openstack/openstacksdk master: image: Parse boolean string properties leniently https://review.opendev.org/c/openstack/openstacksdk/+/1008410 | |
| 13:05:24 | opendevreview | Rodolfo Alonso proposed openstack/python-openstackclient master: network: Drop address scope ``shared`` CLI options https://review.opendev.org/c/openstack/python-openstackclient/+/1008456 | |
| 13:05:41 | opendevreview | Rodolfo Alonso proposed openstack/openstacksdk master: network: Drop ``AddressScope`` ``shared`` field support https://review.opendev.org/c/openstack/openstacksdk/+/1008457 | |
| 13:38:18 | opendevreview | Merged openstack/openstacksdk master: image: Parse boolean string properties leniently https://review.opendev.org/c/openstack/openstacksdk/+/1008410 | |
| 16:04:06 | opendevreview | Stephen Finucane proposed openstack/python-openstackclient master: network: Drop address scope ``shared`` CLI options https://review.opendev.org/c/openstack/python-openstackclient/+/1008456 | |
| 16:04:06 | opendevreview | Stephen Finucane proposed openstack/python-openstackclient master: tests: Drop testing for address scope shared attribute https://review.opendev.org/c/openstack/python-openstackclient/+/1008496 | |
| 18:32:00 | opendevreview | Ade Lee proposed openstack/openstacksdk master: Add Barbican secret store migrate helper https://review.opendev.org/c/openstack/openstacksdk/+/1002400 | |
| 19:01:16 | opendevreview | Merged openstack/python-openstackclient master: tests: Drop testing for address scope shared attribute https://review.opendev.org/c/openstack/python-openstackclient/+/1008496 | |
| #openstack-sdks - 2026-10-03 | |||
| 09:06:42 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: Corrects volume attachments list query options handling. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 09:07:56 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: volume: fix volume attachments list query options handling. https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 09:08:30 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: volume: Fix volume attachments list query options handling https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 09:09:12 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: volume: Fix volume attachments list query options handling https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 09:11:28 | opendevreview | Valery Tschopp proposed openstack/python-openstackclient master: volume: Fix volume attachments list query options handling https://review.opendev.org/c/openstack/python-openstackclient/+/1008088 | |
| 17:29:14 | opendevreview | Takashi Kajinami proposed openstack/osc-lib master: Fix cover target https://review.opendev.org/c/openstack/osc-lib/+/1008606 | |
| #openstack-sdks - 2026-10-04 | |||
| 16:48:04 | opendevreview | Takashi Kajinami proposed openstack/microversion-parse master: Resolve deprecation warnings from setuptools https://review.opendev.org/c/openstack/microversion-parse/+/1008719 | |
| 16:48:05 | opendevreview | Takashi Kajinami proposed openstack/microversion-parse master: tox: Use constraints option https://review.opendev.org/c/openstack/microversion-parse/+/1008720 | |
| 16:48:06 | opendevreview | Takashi Kajinami proposed openstack/microversion-parse master: Use pyproject.toml to maintain coverage options https://review.opendev.org/c/openstack/microversion-parse/+/1008721 | |
| 16:51:46 | opendevreview | Takashi Kajinami proposed openstack/microversion-parse master: Declare Python 3.15 support https://review.opendev.org/c/openstack/microversion-parse/+/1008722 | |
| 16:56:13 | opendevreview | Takashi Kajinami proposed openstack/os-api-ref master: Declare Python 3.15 support https://review.opendev.org/c/openstack/os-api-ref/+/1008724 | |
| 17:16:21 | opendevreview | Merged openstack/microversion-parse master: Resolve deprecation warnings from setuptools https://review.opendev.org/c/openstack/microversion-parse/+/1008719 | |
| 17:16:22 | opendevreview | Merged openstack/microversion-parse master: tox: Use constraints option https://review.opendev.org/c/openstack/microversion-parse/+/1008720 | |
| 17:16:23 | opendevreview | Merged openstack/microversion-parse master: Use pyproject.toml to maintain coverage options https://review.opendev.org/c/openstack/microversion-parse/+/1008721 | |
| 17:16:24 | opendevreview | Merged openstack/microversion-parse master: Declare Python 3.15 support https://review.opendev.org/c/openstack/microversion-parse/+/1008722 | |
| 17:31:07 | opendevreview | Merged openstack/os-api-ref master: Declare Python 3.15 support https://review.opendev.org/c/openstack/os-api-ref/+/1008724 | |