| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-15 | |||
| 16:19:11 | gibi | native threading case, really: https://paste.openstack.org/show/bkFNKzsf5fBmonSNMy9g/ | |
| 16:24:45 | gibi | (in native threading I think the two extra threads are workers in our default or scatter gather pool I can prove that if needed) | |
| 16:27:40 | dansmith | gibi: so each individual transaction is really blocking our entire process? | |
| 16:32:03 | gibi | I have to prove that | |
| 16:34:05 | gibi | maybe I misunderstand something. But with uwsgi (devstack) with a single process, nova-api can only handle one API request at a time as the wsgi stack does not know how to pass multiple requests to that single process. So it does not matter if that single process while handling that single request is actually blocking on DB transcations | |
| 16:34:53 | gibi | this is different in conductor where multiple RPC request can be handled in parallel by each eventlet in the RPC pool | |
| 16:35:15 | gibi | *by each ~ via one | |
| 16:40:38 | dansmith | hmm, yeah maybe the move of the API to native wsgi collapsed this in a way that makes it moot at this point | |
| 16:42:01 | dansmith | but in conductor, we're talking about a similar thing as the tests, where we have a single process with multiple threads that can all be trying to access the database right? | |
| 16:42:32 | gibi | right | |
| 16:42:45 | gibi | I have to switch my investigation to the conductor | |
| 16:42:58 | dansmith | ack, I'm in another call so I can't really focus at the moment | |
| 16:44:20 | gibi | no worries | |
| 16:44:27 | gibi | I need to stop for today soonish anyhw | |
| 16:58:21 | gibi | there is 73 native threads in our cell1 conductor in a fresh devstack. I have to assume 64 is the RPC worker pool. But still it is scarry. Each of the looks exactly the same when idle. So I have to start tracing the thread creation to pick out if any belongs to some DB logic | |
| 16:58:46 | gibi | but that is not for today | |
| 16:58:49 | gibi | o/ | |
| 17:24:15 | melwitt | gibi: I think the additional threads could be the DB connection pools https://docs.sqlalchemy.org/en/21/core/pooling.html https://docs.openstack.org/oslo.db/latest/reference/opts.html#database.max_pool_size it would be one pool per cached cell DB connection info. so with 2 cells, 5 + 5 = 10 maximum. I think tracing it would be good to do, this is just a possibility | |
| 17:36:29 | opendevreview | ribaudr proposed openstack/nova master: Reproduce bug #2108974: keypairs lost during cross-cell resize https://review.opendev.org/c/openstack/nova/+/981557 | |
| 17:36:30 | opendevreview | ribaudr proposed openstack/nova master: Fix keypairs lost during cross-cell resize https://review.opendev.org/c/openstack/nova/+/981558 | |
| 17:38:07 | Uggla | dansmith, melwitt if you have a chance to review ^ (no hurry) | |
| 17:38:47 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 17:40:06 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 17:57:23 | bauzas | Uggla: I won't be able to attend sessions between 1pm and 3pm UTC on Wednesday, so I'll miss all the confidential computing sessions and eventlet removal discussions. I don't particularly want you to reschedule all that juicy meat but please be aware that I won't be there | |
| 18:58:57 | opendevreview | Merged openstack/nova master: Speed up RetryDecorator in unit test https://review.opendev.org/c/openstack/nova/+/975443 | |
| 19:12:26 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 19:29:09 | melwitt | gibi: update - I was wrong about DB connection pools in that they are not threads, they are just connection objects. so going back to the drawing board and getting ideas with claude, for cell conductor other than the 64 from RPC it pointed out cell_worker_thread_pool_size which defaults to 5 and the rest beyond that being things like main thread and periodic tasks. this seems to make sense | |
| 19:47:23 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 19:52:01 | opendevreview | melanie witt proposed openstack/nova master: Use tempest_concurrency=1 for nova-vtpm job https://review.opendev.org/c/openstack/nova/+/984864 | |
| 19:53:34 | melwitt | cc gibi ^ | |
| 20:49:29 | opendevreview | ribaudr proposed openstack/nova master: Reproduce bug #2132147: AZ not reverted after failed live migration https://review.opendev.org/c/openstack/nova/+/984874 | |
| 20:49:30 | opendevreview | ribaudr proposed openstack/nova master: Fix instance AZ not reverted after failed live migration https://review.opendev.org/c/openstack/nova/+/984875 | |
| 21:01:38 | Uggla | Thanks bauzas. I'm aware of your constraints this Wednesday, so I've picked subjects you're comfortable with to ensure you can catch up quickly. I hope that helps! | |
| 23:40:51 | opendevreview | Merged openstack/nova stable/2024.2: Reproducer for bug 2114951 https://review.opendev.org/c/openstack/nova/+/959525 | |
| #openstack-nova - 2026-04-16 | |||
| 00:07:21 | opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: adjust requirement checks for mem_encryption guests https://review.opendev.org/c/openstack/nova/+/967974 | |
| 00:07:21 | opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: address minor issues pointed out in the review https://review.opendev.org/c/openstack/nova/+/983278 | |
| 00:07:22 | opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: introduce a check between mem_encryption and locked_memory https://review.opendev.org/c/openstack/nova/+/971300 | |
| 00:48:02 | opendevreview | Merged openstack/nova stable/2024.2: Fix bug 2114951 https://review.opendev.org/c/openstack/nova/+/959526 | |
| 05:49:33 | opendevreview | Esra Ozkan proposed openstack/nova master: Check volume status before live migration https://review.opendev.org/c/openstack/nova/+/973750 | |
| 06:39:21 | gibi | melwitt: re DB: thanks. I will dig forward to understand how we approach parallelism regarding DB interactions. dansmith had valid questions about the problem we see in the func test env (the transaction within transaction issue) if it affects production code as well or not. | |
| 06:40:11 | gibi | melwitt: re vtpm job: Yeah reducing parallelism in the job make sense to me. Unfortunately the patch faild the vtpm job now with a live migration failure, so we still have something fishy there | |
| 06:42:35 | gibi | I rechecked it to see if it is an intermittent network issue between the host during live migration or something deeper | |
| 06:48:06 | opendevreview | ChungWon Lee proposed openstack/nova master: Add regression test for bug #2088066 https://review.opendev.org/c/openstack/nova/+/984900 | |
| 06:49:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 06:52:34 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 06:55:07 | gibi | melwitt: dansmith: re DB parallelism: in parallel (sic) with my manual tracing I try to trigger the transaction within transaction issue in devstack with mysql in my rally fake virt job https://review.opendev.org/c/openstack/nova/+/960130 that has a lot bigger concurrency on conductor operations that the rest of our jobs due to having many (fake virt) computes and may VM boots in parallel. | |
| 06:55:49 | gibi | last time I run that rally the conductor had no native thrading support but we can have result now with native conductor | |
| 07:13:00 | opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | |
| 07:45:03 | opendevreview | ChungWon Lee proposed openstack/nova master: Add regression test for bug #2088066 https://review.opendev.org/c/openstack/nova/+/984900 | |
| 07:57:04 | opendevreview | Kamil Sambor proposed openstack/nova master: Test nova CLI commands with native threading https://review.opendev.org/c/openstack/nova/+/984036 | |
| 09:31:43 | opendevreview | ChungWon Lee proposed openstack/nova master: Add regression test for bug #2088066 https://review.opendev.org/c/openstack/nova/+/984900 | |
| 10:26:48 | opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | |
| 10:26:48 | opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to functional test (DB, RPC, test case ID) https://review.opendev.org/c/openstack/nova/+/980179 | |
| 11:14:20 | opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | |
| 11:17:33 | r-taketn | gibi: bauzas: Hello. Could you please check the remaining two sev refactor patches again? https://review.opendev.org/q/topic:%22bp/generalize-sev-code%22 | |
| 11:17:41 | bauzas | I'll today | |
| 11:18:07 | r-taketn | bauzas: Thank you | |
| 11:18:20 | r-taketn | Regarding the last patch, I've set it to WIP. I've also added it as an PTG item. I'd like to discuss the approach/direction. | |
| 11:20:53 | gibi | r-taketn: looks good to me | |
| 11:23:19 | r-taketn | gibi: Thank you for your quick review. | |
| 11:43:47 | opendevreview | Stephen Finucane proposed openstack/nova master: tox: Remove override of install_command https://review.opendev.org/c/openstack/nova/+/984918 | |
| 11:46:00 | stephenfin | gibi: sean-k-mooney: bauzas: Need to wait on CI, but I'd appreciate if we could get ^ in sooner rather than later. You can't currently run any tox commands in nova if tox-uv is installed (which is that case for me since I installed the python3-tox-uv package) | |
| 11:46:50 | stephenfin | Coincidentally, tox-uv brings a nice little speed increase during package installation, particularly for things like nova with lots of dependencies | |
| 12:00:27 | sean-k-mooney | i have one comment in the commit message that ill push shortly | |
| 12:00:33 | sean-k-mooney | just testign your patch locally | |
| 12:20:41 | gibi | melwitt: dansmith: with native threadig in conductor I does not see any threadpooling for DB. There are 64 RPC threads and couple of other threads, for RPC hearthbeat, cotyledon management, oslo looping call, and our own executor pools. If I configure all our pool to the smallest possible then the list of threads are easy to review https://paste.openstack.org/show/bmpzWGa9YXAjYF4SXxHS/ non of them | |
| 12:20:47 | gibi | seems to be DB related. I will check this with eventlet as well | |
| 12:23:31 | gibi | but I more or less assume that the DB transaction is running in the GreenThread that is used to receive the RPC request | |
| 12:24:31 | gibi | sean-k-mooney: if you wondering why I'm writing the above then: dansmith asked the question if our overlapping transaction issue we see in the functional test effects the production code as well or not. | |
| 12:25:01 | opendevreview | Bartosz Bezak proposed openstack/nova stable/2025.2: fix: device_by_alias should respect config type https://review.opendev.org/c/openstack/nova/+/984920 | |
| 12:25:15 | opendevreview | Bartosz Bezak proposed openstack/nova stable/2025.1: fix: device_by_alias should respect config type https://review.opendev.org/c/openstack/nova/+/984921 | |
| 12:28:17 | sean-k-mooney | gibi: so my understandign is there are mutlipel driver impleaiton in sqlachmey for mariadb | |
| 12:28:30 | sean-k-mooney | and not all fo them use a conenction pool inthe same way | |
| 12:28:47 | sean-k-mooney | so i dont know if that is expected or now with how devstack deploy by defualt | |
| 12:29:16 | sean-k-mooney | gibi: maybe zeeck? can say quickly if we ping them | |
| 12:30:07 | sean-k-mooney | but ya i dont know if that affect productio or not either we would have to dige deeper | |
| 12:31:25 | gibi | I think connection pool and worker pool is different. We can have N connection object and not have any extra worker threads that doing the transaction with those connection objects | |
| 12:32:22 | gibi | I printed the pool of the DB engine before we use QueuePool there in devstack for native threading. I can print it for eventlet too | |
| 12:44:49 | gibi | yepp in eventlet mode it we also use a QueuePool for DB connectons | |
| 12:44:54 | sean-k-mooney | right so conenction pooling shoudl jsut eb reusing tcp conntion not nessiarly requirign sperate threads | |
| 12:45:21 | gibi | yepp | |
| 12:45:24 | gibi | that is what I see | |
| 12:45:52 | sean-k-mooney | yep i think that is the upsteam sqlachemy behvior for mysql | |
| 12:46:09 | sean-k-mooney | so i dont think this is a regression in behvior | |
| 12:46:21 | gibi | so when an RPC GreenThread executes a long transaction it either yields and then other GreenThreads can also start another transaction. Or it does not yield and then the no other RPC can run | |
| 12:46:29 | sean-k-mooney | but when we use in memroy dbs with sqlight i think we are suign the NULLPOOL | |
| 12:46:57 | gibi | yes, in functest we have some static pooling because the in memory DB needs a single connection to keep the DB aliave | |
| 12:47:21 | gibi | so maybe the overlapping transaction happens across that single static connection | |
| 12:47:47 | gibi | OK that is something I can try to trace | |
| 12:48:53 | gibi | if there is one static connection for the in memory DB then that can be the point of the race. And I imagine that it is very different from the DB perspective if one connection starts a transaction within a transaction or two independent connection runs two independent but timewise overlapping transactions | |
| 12:49:49 | gibi | dansmith: ^^ | |
| 12:54:49 | gibi | sqlalchemy.pool.impl.StaticPool this is the pool in the functional test by default | |
| 12:55:43 | gibi | https://github.com/sqlalchemy/sqlalchemy/blob/7dce9fec9d2071351d0c003a607955eca5dcf9ae/lib/sqlalchemy/pool/impl.py#L451 | |
| 12:57:39 | sean-k-mooney | isnt that why oslo.db is overreidng | |
| 13:05:21 | gibi | when the SQLite DB is in memory then we use StaticPool when we switch to file based (for the api DB) then that file based DB uses sqlalchemy.pool.impl.NullPool | |
| 13:05:43 | gibi | this also points to that maybe the source of the race is the StaticPool | |
| 13:06:20 | gibi | null pool basically always opens a new connection | |
| 13:06:23 | gibi | so it cannot race | |
| 13:06:53 | gibi | I'm not sure what oslo_db overrides in this regards | |
| 13:13:19 | bauzas | stephenfin: silly question but I was thinking we were still supporting tox 2.x, am I wrong ? | |