| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-28 | |||
| 15:15:14 | sean-k-mooney | if you have time to test it and give feeback please do | |
| 15:16:12 | amorin | yup, will try to test this here, I have a lab that could maybe be used for this | |
| 15:16:18 | amorin | thanks for the heads up | |
| 15:34:04 | gibi | upstream eventlet meeting https://meet.google.com/bcy-uqoz-hje | |
| 16:35:30 | Uggla | lajoskatona I have seen your request above. I will ask during next upstream meeting. In the meantime I will try to review them. I know a little about it because I did it for manila. | |
| 16:37:07 | lajoskatona | Uggla: thanks, I can join the meeting (next Monday?) | |
| 16:38:05 | Uggla | lajoskatona if you wish yes. Monday 16:00 UTC. | |
| 16:39:27 | lajoskatona | Uggla: thanks | |
| 17:11:09 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 17:17:38 | stephenfin | gmaan: jfyi, we're only missing your review on the patches for the servers and server shares APIs to close out the openapi series. It would be great if you could get to it at some point in the near future | |
| 17:40:02 | gmaan | stephenfin: yeah, its in my list. I will check those sometime early next week | |
| 22:55:12 | opendevreview | melanie witt proposed openstack/nova master: Add vtpm_secret_(uuid|value) to LibvirtLiveMigrateData https://review.opendev.org/c/openstack/nova/+/952628 | |
| 22:55:13 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/941483 | |
| 22:55:13 | opendevreview | melanie witt proposed openstack/nova master: TPM: prepare to bump service version for live migration https://review.opendev.org/c/openstack/nova/+/962051 | |
| 22:55:14 | opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | |
| 22:55:14 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | |
| 22:55:15 | opendevreview | melanie witt proposed openstack/nova master: TPM: opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 22:55:15 | opendevreview | melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975 | |
| 22:55:16 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| #openstack-nova - 2026-01-29 | |||
| 00:44:04 | opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | |
| 00:48:23 | opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | |
| 01:27:37 | opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | |
| 07:23:30 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 07:27:03 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 08:07:25 | opendevreview | huanhongda proposed openstack/nova master: numa: Respect allocation ratios for non-pinned NUMA instances https://review.opendev.org/c/openstack/nova/+/971409 | |
| 10:27:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:27:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 10:27:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:27:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:35:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:35:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:35:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 10:35:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:36:41 | gibi | bauzas: now we have the executor shutdown at compute service stop ^^ | |
| 10:37:17 | bauzas | all cool, doing some backport work atm but I'll try to look at it today | |
| 10:40:57 | gibi | OK thanks | |
| 10:56:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: SubclassSignatureTestCase to use NoDBTestCase as base https://review.opendev.org/c/openstack/nova/+/974861 | |
| 10:56:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | |
| 10:56:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:56:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable mypy on nova/utils.py https://review.opendev.org/c/openstack/nova/+/969936 | |
| 10:56:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:56:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:56:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 10:57:13 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable mypy on nova/utils.py https://review.opendev.org/c/openstack/nova/+/969936 | |
| 10:57:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:57:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:57:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:57:32 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 13:43:51 | gibi | fwiesel, sean-k-mooney : I see the issue behind the stack trace in the oslo.vmware native threaded run. This is probably a bug with the libvirt driver too. I will do the needful | |
| 14:07:11 | sean-k-mooney | gibi: ack | |
| 14:07:13 | sean-k-mooney | what was it | |
| 14:27:26 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975162 | |
| 14:27:27 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975163 | |
| 14:29:20 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975164 | |
| 14:29:21 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975165 | |
| 14:32:51 | gibi | sean-k-mooney: https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/utils.py#L605 this should be conditional as not all executors has a name | |
| 14:33:27 | gibi | yet another refactoring (for the future) to ensure all our executors are named and created in the same way | |
| 14:33:30 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975166 | |
| 14:33:31 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975167 | |
| 14:33:41 | sean-k-mooney | yep i was just going to say | |
| 14:33:54 | sean-k-mooney | that or well we shoudl jsut not depend on it for the warning | |
| 15:24:02 | opendevreview | sean mooney proposed openstack/nova master: Add regression test to repoduce bug 2139351 https://review.opendev.org/c/openstack/nova/+/975171 | |
| 15:39:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 15:39:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | |
| 15:39:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 15:44:34 | gibi | sean-k-mooney: btw there are bigger issues in the vmware logs than that stack trace. | |
| 15:44:40 | gibi | [10197.783268] env[89459]: WARNING nova.utils [None req-08c5913a-0cad-45f4-991c-76f091773eb3 tempest-ServersTestJSON-388458959 tempest-ServersTestJSON-388458959-project-member] The MainProcess.default pool does not have free threads so the task <function ComputeManager.build_and_run_instance.<locals>._locked_do_build_and_run_instance at 0x7647d040cee0> will be queued. If this happens repeatedly then | |
| 15:44:46 | gibi | the size of the pool is too small for the load or there are stuck threads filling the pool. | |
| 15:45:11 | gibi | this is the reason why VM builds are failing. Something is using up our thread pool | |
| 15:45:30 | sean-k-mooney | gibi: i notice a few other tracebaces ya but i was nto sure wht are "expected error" vs related to event let liek the "expected" os-brick errors | |
| 15:46:31 | sean-k-mooney | so to get the message that means somting is callign spawn right? | |
| 15:46:51 | sean-k-mooney | as in it can only get to that executor by going thorugh the nova.utils metods | |
| 15:47:24 | sean-k-mooney | which means it has to be in hte nova side fo the vmware driver rather then oslo.vmware | |
| 15:47:48 | gibi | yepp _locked_do_build_and_run_instance calling the spawn. That is normal, but somewho we run out of workers in the pool so somethign is hanging | |
| 15:48:38 | gibi | https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/compute/manager.py#L2455 | |
| 15:49:51 | gibi | we need a run with thread_pool_statistic_period=0 to get all the places that puts work into that pool to see when we start running out of workers | |
| 15:50:39 | gibi | I can look at differently to see if all VM boots are failing or not | |
| 15:51:13 | gibi | to try to get what was the first failed boot on the compute and see previous successful boot (if any) doing something that does not return a worker | |
| 15:53:48 | sean-k-mooney | sounds like a resonable plan of attack | |
| 15:54:11 | sean-k-mooney | you could jsut add a dnm to make 0 the default for testing | |
| 16:04:38 | opendevreview | Merged openstack/os-vif master: Add TAP device pre-creation support for OVS/OVN https://review.opendev.org/c/openstack/os-vif/+/971231 | |
| 18:33:16 | opendevreview | Merged openstack/os-vif master: Fixed bridge name when per_port_bridge is used https://review.opendev.org/c/openstack/os-vif/+/966410 | |
| 22:36:35 | gmaan | sean-k-mooney: thanks for reply on Fair lock comments, I am good on that except one new comment https://review.opendev.org/c/openstack/nova/+/973438/comment/ff4bb8ea_53533e58/ | |
| 22:37:21 | gmaan | sean-k-mooney: if that comment make sense then i am ok to fix that in follow up because current change is in mergable state. let me know. | |
| #openstack-nova - 2026-01-30 | |||
| 03:51:34 | opendevreview | Ghanshyam proposed openstack/nova master: Add a new field 'topic_alt' in service: object, DB, notification payload https://review.opendev.org/c/openstack/nova/+/975240 | |
| 03:54:35 | opendevreview | Ghanshyam proposed openstack/nova master: Add a new field 'topic_alt' in service: object, DB, notification payload https://review.opendev.org/c/openstack/nova/+/975240 | |
| 04:27:08 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| 04:28:21 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| 04:41:10 | gmaan | gibi: bauzas: these two change (adding 2nd RPC server) for graceful shutdown series are ready for review: https://review.opendev.org/c/openstack/nova/+/975240 https://review.opendev.org/c/openstack/nova/+/975242 | |
| 06:40:12 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 06:52:05 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 08:43:32 | bauzas | gmaan: saw those, will try to do review them today but my day is already packed with review needs | |
| 09:39:37 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 10:36:11 | gibi | gmaan: did my first review round on the second RPC patches and left comments | |
| 15:02:52 | gibi | sean-k-mooney: heh I think I see how the oslo.vmware job could run out of threads in the default pool. If there are at least two differnt type of task is using the default pool during a single VM boot like buil_and_run_instance and then later that task spawns _allocate_network_async then if the pool size is e.g. 5 and I send in 10 parallel VM boot requests. Then the first 5 VM boot fills the pool and | |
| 15:02:58 | gibi | then none of them can move forward as the _allocate_network_async task for each of them is also queued up waiting for executor. | |
| 15:04:02 | sean-k-mooney | ah i see | |
| 15:04:56 | sean-k-mooney | i guess im not sure why buil_and_run_instance or _allocate_network_async woudl be dispatced to the defautl thread pool in the first place | |
| 15:05:17 | sean-k-mooney | the should both be on the rpc trhead pool. | |
| 15:06:11 | sean-k-mooney | if we had to use utils.spawn inside if for some async work i can see it having contention but it shoudl not deadlock | |