| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-14 | |||
| 18:16:50 | sean-k-mooney | its honestly a pretty small chagne to make | |
| 18:41:23 | opendevreview | melanie witt proposed openstack/nova master: Fix flaky monitor metric tests due to timestamp format mismatch https://review.opendev.org/c/openstack/nova/+/984617 | |
| 18:56:21 | opendevreview | melanie witt proposed openstack/nova master: DNM test ceph ipv6 https://review.opendev.org/c/openstack/nova/+/984623 | |
| 19:01:40 | aravindh | Hey I have a query with respect to multipath and iscsid, I have 2 interfaces that connects to storage, I have setup 2 paths in iscsiadm, and iscsid is set to start automatically with discovery on hypervisor boot. When it boots, every multipath list shows me 4 paths. But when I start a VM from nova that creates only 2 paths rather than 4. If i rescan the iscsi bus and restart multipathd, that 2 paths move to 4 paths. Is t | |
| 19:05:41 | opendevreview | melanie witt proposed openstack/nova master: DNM test ceph ipv6 https://review.opendev.org/c/openstack/nova/+/984623 | |
| 21:45:46 | melwitt | gmaan: easy flaky unit test fix if you might be interested https://review.opendev.org/c/openstack/nova/+/984617 | |
| 21:47:55 | gmaan | checking | |
| 21:51:25 | gmaan | melwitt: thanks for fixing, lgtm | |
| 21:51:37 | melwitt | thanks gmaan | |
| #openstack-nova - 2026-04-15 | |||
| 07:28:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Speed up RetryDecorator in unit test https://review.opendev.org/c/openstack/nova/+/975443 | |
| 07:28:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: [py313-threading]Reenable last scatter-gather unit test https://review.opendev.org/c/openstack/nova/+/984349 | |
| 07:28:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: [py313-threading]Reenable test_show_simple_tenant_usage_policy https://review.opendev.org/c/openstack/nova/+/984606 | |
| 07:28:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Delete duplicate unit test https://review.opendev.org/c/openstack/nova/+/984607 | |
| 07:28:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: [py313-threading]Skip eventlet specific tests https://review.opendev.org/c/openstack/nova/+/984608 | |
| 08:01:10 | opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | |
| 08:15:26 | Uggla | Hi gibi, I think you missed my question from yesterday about eventlet. I would like also to know about the duration. 30mn ? 1h ? | |
| 08:20:24 | opendevreview | Merged openstack/nova master: Fix flaky monitor metric tests due to timestamp format mismatch https://review.opendev.org/c/openstack/nova/+/984617 | |
| 08:49:42 | gibi | Uggla: indeed I missed the question. Thanks for pinging again. | |
| 08:49:46 | gibi | 30m is enough | |
| 10:35:15 | opendevreview | Elod Illes proposed openstack/nova master: [doc] Rephrase bfv rescue note https://review.opendev.org/c/openstack/nova/+/984713 | |
| 10:58:03 | opendevreview | Kamil Sambor proposed openstack/nova master: Test nova CLI commands with native threading https://review.opendev.org/c/openstack/nova/+/984036 | |
| 11:43:17 | Uggla | gibi, thx any pb if it is before the "global" eventlet session ? | |
| 11:44:17 | Uggla | sean-k-mooney regarding https://blueprints.launchpad.net/nova/+spec/adopt-port-trusted-vif-extension something to discuss at ptg ? | |
| 11:52:52 | sean-k-mooney | Uggla: we coudl or in a team meeting | |
| 11:53:09 | sean-k-mooney | its honestly something that i could have an agent do in like an hour | |
| 11:53:26 | sean-k-mooney | when i say the chagne is small i really mean that | |
| 11:53:44 | sean-k-mooney | we just need to read a new atibute if its there and pass that along | |
| 11:59:38 | opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | |
| 12:06:38 | opendevreview | Elod Illes proposed openstack/nova master: [doc] Rephrase bfv rescue note https://review.opendev.org/c/openstack/nova/+/984713 | |
| 12:33:13 | LarsErikP | sean-k-mooney: hi! I was just wondering.. What's the status here? (A) https://review.opendev.org/c/openstack/nova/+/916409 | |
| 12:39:33 | opendevreview | Merged openstack/nova master: mem-enc: make RP creation independent of specific encryption models https://review.opendev.org/c/openstack/nova/+/967972 | |
| 12:42:00 | sean-k-mooney | LarsErikP: good quition i have not touched it since the last rebase and its still waiting on other cores to review | |
| 12:42:13 | sean-k-mooney | no one has looked at https://review.opendev.org/c/openstack/nova/+/976654/1 | |
| 12:43:06 | LarsErikP | aha..! anyone someone could poke? :P | |
| 12:43:18 | sean-k-mooney | i can rebase them again quickly | |
| 12:43:49 | LarsErikP | that could potentially wake someone up? | |
| 12:46:31 | gibi | Uggla: no problem if it is before the global one | |
| 12:47:36 | Uggla | cool, thx | |
| 13:01:26 | gibi | melwitt: when you are up. I noticed a fauilure in tn the new vtpm job https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9a1/openstack/9a13b97fb5204ac6a8f4c0d726431f53/testr_results.html Maybe we have some instabilities?/ | |
| 13:02:05 | gibi | I did not dif into the reason of th NoValidHost there | |
| 13:03:08 | opendevreview | Merged openstack/nova master: mem-enc: refactor _guest_configure_mem_encryption() for extensibility https://review.opendev.org/c/openstack/nova/+/967973 | |
| 13:03:09 | gibi | bauzas: as you asked to ping you again today, here starts a series with small unit test reenablement patches https://review.opendev.org/c/openstack/nova/+/975443/7 | |
| 13:24:12 | bauzas | ack will look today | |
| 13:24:19 | gibi | thanks | |
| 13:57:03 | gibi | melwitt: when you are up, can we discuss your cell db fixuter fix. I have a question about how this works https://review.opendev.org/c/openstack/nova/+/984480/comment/b5ef5f95_c8487736/ | |
| 14:24:47 | Uggla | https://etherpad.opendev.org/p/nova-2026.2-ptg is almost ready, at least in a descent shape. Please ping me if you notice something wrong. | |
| 14:50:51 | bauzas | I'll look at them | |
| 15:08:13 | melwitt | gibi: I will look thanks. and I will also look into the vtpm failure to see what happened, it is possible there could be instability | |
| 15:15:52 | gibi | melwitt: thanks | |
| 15:24:01 | opendevreview | ribaudr proposed openstack/nova master: Add reproducer for bug #2146675 https://review.opendev.org/c/openstack/nova/+/984801 | |
| 15:24:01 | opendevreview | ribaudr proposed openstack/nova master: Set instance to ERROR state on start_instance power_on failure https://review.opendev.org/c/openstack/nova/+/984802 | |
| 16:18:39 | gibi | dansmith: re how database connection is hanlded vs eventlet yielding. I tried to get a picture by collecting GMR from nova-api after it already accessed the nova_api DB at least once. I see nothing in the GMR that points to a worker pool | |
| 16:18:47 | gibi | eventlet case: https://paste.openstack.org/show/bqGu0wQfQPTAiyZQXvoA/ | |
| 16:18:58 | gibi | native threading case: https://paste.openstack.org/show/bqGu0wQfQPTAiyZQXvoA/ | |
| 16:19:11 | gibi | native threading case, really: https://paste.openstack.org/show/bkFNKzsf5fBmonSNMy9g/ | |
| 16:24:45 | gibi | (in native threading I think the two extra threads are workers in our default or scatter gather pool I can prove that if needed) | |
| 16:27:40 | dansmith | gibi: so each individual transaction is really blocking our entire process? | |
| 16:32:03 | gibi | I have to prove that | |
| 16:34:05 | gibi | maybe I misunderstand something. But with uwsgi (devstack) with a single process, nova-api can only handle one API request at a time as the wsgi stack does not know how to pass multiple requests to that single process. So it does not matter if that single process while handling that single request is actually blocking on DB transcations | |
| 16:34:53 | gibi | this is different in conductor where multiple RPC request can be handled in parallel by each eventlet in the RPC pool | |
| 16:35:15 | gibi | *by each ~ via one | |
| 16:40:38 | dansmith | hmm, yeah maybe the move of the API to native wsgi collapsed this in a way that makes it moot at this point | |
| 16:42:01 | dansmith | but in conductor, we're talking about a similar thing as the tests, where we have a single process with multiple threads that can all be trying to access the database right? | |
| 16:42:32 | gibi | right | |
| 16:42:45 | gibi | I have to switch my investigation to the conductor | |
| 16:42:58 | dansmith | ack, I'm in another call so I can't really focus at the moment | |
| 16:44:20 | gibi | no worries | |
| 16:44:27 | gibi | I need to stop for today soonish anyhw | |
| 16:58:21 | gibi | there is 73 native threads in our cell1 conductor in a fresh devstack. I have to assume 64 is the RPC worker pool. But still it is scarry. Each of the looks exactly the same when idle. So I have to start tracing the thread creation to pick out if any belongs to some DB logic | |
| 16:58:46 | gibi | but that is not for today | |
| 16:58:49 | gibi | o/ | |
| 17:24:15 | melwitt | gibi: I think the additional threads could be the DB connection pools https://docs.sqlalchemy.org/en/21/core/pooling.html https://docs.openstack.org/oslo.db/latest/reference/opts.html#database.max_pool_size it would be one pool per cached cell DB connection info. so with 2 cells, 5 + 5 = 10 maximum. I think tracing it would be good to do, this is just a possibility | |
| 17:36:29 | opendevreview | ribaudr proposed openstack/nova master: Reproduce bug #2108974: keypairs lost during cross-cell resize https://review.opendev.org/c/openstack/nova/+/981557 | |
| 17:36:30 | opendevreview | ribaudr proposed openstack/nova master: Fix keypairs lost during cross-cell resize https://review.opendev.org/c/openstack/nova/+/981558 | |
| 17:38:07 | Uggla | dansmith, melwitt if you have a chance to review ^ (no hurry) | |
| 17:38:47 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 17:40:06 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 17:57:23 | bauzas | Uggla: I won't be able to attend sessions between 1pm and 3pm UTC on Wednesday, so I'll miss all the confidential computing sessions and eventlet removal discussions. I don't particularly want you to reschedule all that juicy meat but please be aware that I won't be there | |
| 18:58:57 | opendevreview | Merged openstack/nova master: Speed up RetryDecorator in unit test https://review.opendev.org/c/openstack/nova/+/975443 | |
| 19:12:26 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 19:29:09 | melwitt | gibi: update - I was wrong about DB connection pools in that they are not threads, they are just connection objects. so going back to the drawing board and getting ideas with claude, for cell conductor other than the 64 from RPC it pointed out cell_worker_thread_pool_size which defaults to 5 and the rest beyond that being things like main thread and periodic tasks. this seems to make sense | |
| 19:47:23 | opendevreview | sean mooney proposed openstack/nova master: add agentic coding scaffold and contributor docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 19:52:01 | opendevreview | melanie witt proposed openstack/nova master: Use tempest_concurrency=1 for nova-vtpm job https://review.opendev.org/c/openstack/nova/+/984864 | |
| 19:53:34 | melwitt | cc gibi ^ | |
| 20:49:29 | opendevreview | ribaudr proposed openstack/nova master: Reproduce bug #2132147: AZ not reverted after failed live migration https://review.opendev.org/c/openstack/nova/+/984874 | |
| 20:49:30 | opendevreview | ribaudr proposed openstack/nova master: Fix instance AZ not reverted after failed live migration https://review.opendev.org/c/openstack/nova/+/984875 | |
| 21:01:38 | Uggla | Thanks bauzas. I'm aware of your constraints this Wednesday, so I've picked subjects you're comfortable with to ensure you can catch up quickly. I hope that helps! | |
| 23:40:51 | opendevreview | Merged openstack/nova stable/2024.2: Reproducer for bug 2114951 https://review.opendev.org/c/openstack/nova/+/959525 | |
| #openstack-nova - 2026-04-16 | |||
| 00:07:21 | opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: adjust requirement checks for mem_encryption guests https://review.opendev.org/c/openstack/nova/+/967974 | |
| 00:07:21 | opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: address minor issues pointed out in the review https://review.opendev.org/c/openstack/nova/+/983278 | |
| 00:07:22 | opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: introduce a check between mem_encryption and locked_memory https://review.opendev.org/c/openstack/nova/+/971300 | |
| 00:48:02 | opendevreview | Merged openstack/nova stable/2024.2: Fix bug 2114951 https://review.opendev.org/c/openstack/nova/+/959526 | |
| 05:49:33 | opendevreview | Esra Ozkan proposed openstack/nova master: Check volume status before live migration https://review.opendev.org/c/openstack/nova/+/973750 | |
| 06:39:21 | gibi | melwitt: re DB: thanks. I will dig forward to understand how we approach parallelism regarding DB interactions. dansmith had valid questions about the problem we see in the func test env (the transaction within transaction issue) if it affects production code as well or not. | |
| 06:40:11 | gibi | melwitt: re vtpm job: Yeah reducing parallelism in the job make sense to me. Unfortunately the patch faild the vtpm job now with a live migration failure, so we still have something fishy there | |
| 06:42:35 | gibi | I rechecked it to see if it is an intermittent network issue between the host during live migration or something deeper | |
| 06:48:06 | opendevreview | ChungWon Lee proposed openstack/nova master: Add regression test for bug #2088066 https://review.opendev.org/c/openstack/nova/+/984900 | |
| 06:49:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 06:52:34 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 06:55:07 | gibi | melwitt: dansmith: re DB parallelism: in parallel (sic) with my manual tracing I try to trigger the transaction within transaction issue in devstack with mysql in my rally fake virt job https://review.opendev.org/c/openstack/nova/+/960130 that has a lot bigger concurrency on conductor operations that the rest of our jobs due to having many (fake virt) computes and may VM boots in parallel. | |