| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-29 | |||
| 10:27:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:27:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 10:27:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:27:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:35:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:35:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:35:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 10:35:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:36:41 | gibi | bauzas: now we have the executor shutdown at compute service stop ^^ | |
| 10:37:17 | bauzas | all cool, doing some backport work atm but I'll try to look at it today | |
| 10:40:57 | gibi | OK thanks | |
| 10:56:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: SubclassSignatureTestCase to use NoDBTestCase as base https://review.opendev.org/c/openstack/nova/+/974861 | |
| 10:56:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | |
| 10:56:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:56:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable mypy on nova/utils.py https://review.opendev.org/c/openstack/nova/+/969936 | |
| 10:56:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:56:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:56:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 10:57:13 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable mypy on nova/utils.py https://review.opendev.org/c/openstack/nova/+/969936 | |
| 10:57:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 10:57:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 10:57:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 10:57:32 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 13:43:51 | gibi | fwiesel, sean-k-mooney : I see the issue behind the stack trace in the oslo.vmware native threaded run. This is probably a bug with the libvirt driver too. I will do the needful | |
| 14:07:11 | sean-k-mooney | gibi: ack | |
| 14:07:13 | sean-k-mooney | what was it | |
| 14:27:26 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975162 | |
| 14:27:27 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975163 | |
| 14:29:20 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975164 | |
| 14:29:21 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975165 | |
| 14:32:51 | gibi | sean-k-mooney: https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/utils.py#L605 this should be conditional as not all executors has a name | |
| 14:33:27 | gibi | yet another refactoring (for the future) to ensure all our executors are named and created in the same way | |
| 14:33:30 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975166 | |
| 14:33:31 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975167 | |
| 14:33:41 | sean-k-mooney | yep i was just going to say | |
| 14:33:54 | sean-k-mooney | that or well we shoudl jsut not depend on it for the warning | |
| 15:24:02 | opendevreview | sean mooney proposed openstack/nova master: Add regression test to repoduce bug 2139351 https://review.opendev.org/c/openstack/nova/+/975171 | |
| 15:39:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 15:39:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | |
| 15:39:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 15:44:34 | gibi | sean-k-mooney: btw there are bigger issues in the vmware logs than that stack trace. | |
| 15:44:40 | gibi | [10197.783268] env[89459]: WARNING nova.utils [None req-08c5913a-0cad-45f4-991c-76f091773eb3 tempest-ServersTestJSON-388458959 tempest-ServersTestJSON-388458959-project-member] The MainProcess.default pool does not have free threads so the task <function ComputeManager.build_and_run_instance.<locals>._locked_do_build_and_run_instance at 0x7647d040cee0> will be queued. If this happens repeatedly then | |
| 15:44:46 | gibi | the size of the pool is too small for the load or there are stuck threads filling the pool. | |
| 15:45:11 | gibi | this is the reason why VM builds are failing. Something is using up our thread pool | |
| 15:45:30 | sean-k-mooney | gibi: i notice a few other tracebaces ya but i was nto sure wht are "expected error" vs related to event let liek the "expected" os-brick errors | |
| 15:46:31 | sean-k-mooney | so to get the message that means somting is callign spawn right? | |
| 15:46:51 | sean-k-mooney | as in it can only get to that executor by going thorugh the nova.utils metods | |
| 15:47:24 | sean-k-mooney | which means it has to be in hte nova side fo the vmware driver rather then oslo.vmware | |
| 15:47:48 | gibi | yepp _locked_do_build_and_run_instance calling the spawn. That is normal, but somewho we run out of workers in the pool so somethign is hanging | |
| 15:48:38 | gibi | https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/compute/manager.py#L2455 | |
| 15:49:51 | gibi | we need a run with thread_pool_statistic_period=0 to get all the places that puts work into that pool to see when we start running out of workers | |
| 15:50:39 | gibi | I can look at differently to see if all VM boots are failing or not | |
| 15:51:13 | gibi | to try to get what was the first failed boot on the compute and see previous successful boot (if any) doing something that does not return a worker | |
| 15:53:48 | sean-k-mooney | sounds like a resonable plan of attack | |
| 15:54:11 | sean-k-mooney | you could jsut add a dnm to make 0 the default for testing | |
| 16:04:38 | opendevreview | Merged openstack/os-vif master: Add TAP device pre-creation support for OVS/OVN https://review.opendev.org/c/openstack/os-vif/+/971231 | |
| 18:33:16 | opendevreview | Merged openstack/os-vif master: Fixed bridge name when per_port_bridge is used https://review.opendev.org/c/openstack/os-vif/+/966410 | |
| 22:36:35 | gmaan | sean-k-mooney: thanks for reply on Fair lock comments, I am good on that except one new comment https://review.opendev.org/c/openstack/nova/+/973438/comment/ff4bb8ea_53533e58/ | |
| 22:37:21 | gmaan | sean-k-mooney: if that comment make sense then i am ok to fix that in follow up because current change is in mergable state. let me know. | |
| #openstack-nova - 2026-01-30 | |||
| 03:51:34 | opendevreview | Ghanshyam proposed openstack/nova master: Add a new field 'topic_alt' in service: object, DB, notification payload https://review.opendev.org/c/openstack/nova/+/975240 | |
| 03:54:35 | opendevreview | Ghanshyam proposed openstack/nova master: Add a new field 'topic_alt' in service: object, DB, notification payload https://review.opendev.org/c/openstack/nova/+/975240 | |
| 04:27:08 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| 04:28:21 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| 04:41:10 | gmaan | gibi: bauzas: these two change (adding 2nd RPC server) for graceful shutdown series are ready for review: https://review.opendev.org/c/openstack/nova/+/975240 https://review.opendev.org/c/openstack/nova/+/975242 | |
| 06:40:12 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 06:52:05 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 08:43:32 | bauzas | gmaan: saw those, will try to do review them today but my day is already packed with review needs | |
| 09:39:37 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 10:36:11 | gibi | gmaan: did my first review round on the second RPC patches and left comments | |
| 15:02:52 | gibi | sean-k-mooney: heh I think I see how the oslo.vmware job could run out of threads in the default pool. If there are at least two differnt type of task is using the default pool during a single VM boot like buil_and_run_instance and then later that task spawns _allocate_network_async then if the pool size is e.g. 5 and I send in 10 parallel VM boot requests. Then the first 5 VM boot fills the pool and | |
| 15:02:58 | gibi | then none of them can move forward as the _allocate_network_async task for each of them is also queued up waiting for executor. | |
| 15:04:02 | sean-k-mooney | ah i see | |
| 15:04:56 | sean-k-mooney | i guess im not sure why buil_and_run_instance or _allocate_network_async woudl be dispatced to the defautl thread pool in the first place | |
| 15:05:17 | sean-k-mooney | the should both be on the rpc trhead pool. | |
| 15:06:11 | sean-k-mooney | if we had to use utils.spawn inside if for some async work i can see it having contention but it shoudl not deadlock | |
| 15:06:35 | sean-k-mooney | oh | |
| 15:06:48 | sean-k-mooney | no it could | |
| 15:07:04 | gibi | for some historical reasons build_and_run is moved to the default pool from the RPC pool. And then that task utils.spawns the network task | |
| 15:07:12 | sean-k-mooney | i guess a task that is on a executor shoudl never dispatch an other task to the same executor | |
| 15:07:21 | sean-k-mooney | if it does it could alwasy starve | |
| 15:07:26 | gibi | yeah | |
| 15:07:57 | sean-k-mooney | ok wll i think the network task shoudl jsut move to the io thread pool | |
| 15:08:03 | sean-k-mooney | at least for the short term | |
| 15:08:08 | sean-k-mooney | rather the the default | |
| 15:08:44 | gibi | yeah. Long term I need to think how can we prevent a new task being introduced and spawned to the same pool as the parent task | |
| 15:09:30 | sean-k-mooney | so we are not seeing this in the libvirt driver because this behvior is vmware dirver specific? | |
| 15:10:08 | gibi | nope I see it in the libvirt dirver too | |
| 15:10:10 | gibi | now that I know what to look for | |
| 15:10:25 | gibi | both task are from the compute manager so the problem is virt driver agnostic | |
| 15:11:03 | gibi | it seems tempest in our CI never tried to spawn more than 2-3 VMs in parallel but oslo.vmaware CI does so | |
| 15:11:08 | sean-k-mooney | ack in that case for now we coudl use the io thread pool for all network or file operations and the default for normal function calls? | |
| 15:11:35 | gibi | I will take a closer look at all utils.spawn call to see how much contention we have | |
| 15:11:49 | gibi | but yes in general we need to separate out these tasks to different pools | |
| 15:12:53 | sean-k-mooney | not a now descison but we might actully want a "packaged task" construct in the future where we can package up a closure of the task to be executeded (with cansletion if need) and metadata to tack/prevent this | |
| 15:13:36 | sean-k-mooney | i.e. so we know what executor we are on and on what executor we can spawn the task | |
| 15:14:16 | gibi | yeah a generic solution probably needs some tracking metadata | |
| 15:14:17 | sean-k-mooney | for now we could add a simple execotre fild to the nova context object which is None by defualt but set to a link to the execotre by spawn | |
| 15:14:39 | sean-k-mooney | then spawn could check if we are ever tying to spawn a task on teh same executor and raise | |
| 15:14:49 | gibi | yeah context is not a bad idea | |
| 15:14:50 | gibi | thanks | |