| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-29 | |||
| 13:43:51 | gibi | fwiesel, sean-k-mooney : I see the issue behind the stack trace in the oslo.vmware native threaded run. This is probably a bug with the libvirt driver too. I will do the needful | |
| 14:07:11 | sean-k-mooney | gibi: ack | |
| 14:07:13 | sean-k-mooney | what was it | |
| 14:27:26 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975162 | |
| 14:27:27 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975163 | |
| 14:29:20 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975164 | |
| 14:29:21 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975165 | |
| 14:32:51 | gibi | sean-k-mooney: https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/utils.py#L605 this should be conditional as not all executors has a name | |
| 14:33:27 | gibi | yet another refactoring (for the future) to ensure all our executors are named and created in the same way | |
| 14:33:30 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975166 | |
| 14:33:31 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975167 | |
| 14:33:41 | sean-k-mooney | yep i was just going to say | |
| 14:33:54 | sean-k-mooney | that or well we shoudl jsut not depend on it for the warning | |
| 15:24:02 | opendevreview | sean mooney proposed openstack/nova master: Add regression test to repoduce bug 2139351 https://review.opendev.org/c/openstack/nova/+/975171 | |
| 15:39:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 15:39:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | |
| 15:39:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 15:44:34 | gibi | sean-k-mooney: btw there are bigger issues in the vmware logs than that stack trace. | |
| 15:44:40 | gibi | [10197.783268] env[89459]: WARNING nova.utils [None req-08c5913a-0cad-45f4-991c-76f091773eb3 tempest-ServersTestJSON-388458959 tempest-ServersTestJSON-388458959-project-member] The MainProcess.default pool does not have free threads so the task <function ComputeManager.build_and_run_instance.<locals>._locked_do_build_and_run_instance at 0x7647d040cee0> will be queued. If this happens repeatedly then | |
| 15:44:46 | gibi | the size of the pool is too small for the load or there are stuck threads filling the pool. | |
| 15:45:11 | gibi | this is the reason why VM builds are failing. Something is using up our thread pool | |
| 15:45:30 | sean-k-mooney | gibi: i notice a few other tracebaces ya but i was nto sure wht are "expected error" vs related to event let liek the "expected" os-brick errors | |
| 15:46:31 | sean-k-mooney | so to get the message that means somting is callign spawn right? | |
| 15:46:51 | sean-k-mooney | as in it can only get to that executor by going thorugh the nova.utils metods | |
| 15:47:24 | sean-k-mooney | which means it has to be in hte nova side fo the vmware driver rather then oslo.vmware | |
| 15:47:48 | gibi | yepp _locked_do_build_and_run_instance calling the spawn. That is normal, but somewho we run out of workers in the pool so somethign is hanging | |
| 15:48:38 | gibi | https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/compute/manager.py#L2455 | |
| 15:49:51 | gibi | we need a run with thread_pool_statistic_period=0 to get all the places that puts work into that pool to see when we start running out of workers | |
| 15:50:39 | gibi | I can look at differently to see if all VM boots are failing or not | |
| 15:51:13 | gibi | to try to get what was the first failed boot on the compute and see previous successful boot (if any) doing something that does not return a worker | |
| 15:53:48 | sean-k-mooney | sounds like a resonable plan of attack | |
| 15:54:11 | sean-k-mooney | you could jsut add a dnm to make 0 the default for testing | |
| 16:04:38 | opendevreview | Merged openstack/os-vif master: Add TAP device pre-creation support for OVS/OVN https://review.opendev.org/c/openstack/os-vif/+/971231 | |
| 18:33:16 | opendevreview | Merged openstack/os-vif master: Fixed bridge name when per_port_bridge is used https://review.opendev.org/c/openstack/os-vif/+/966410 | |
| 22:36:35 | gmaan | sean-k-mooney: thanks for reply on Fair lock comments, I am good on that except one new comment https://review.opendev.org/c/openstack/nova/+/973438/comment/ff4bb8ea_53533e58/ | |
| 22:37:21 | gmaan | sean-k-mooney: if that comment make sense then i am ok to fix that in follow up because current change is in mergable state. let me know. | |
| #openstack-nova - 2026-01-30 | |||
| 03:51:34 | opendevreview | Ghanshyam proposed openstack/nova master: Add a new field 'topic_alt' in service: object, DB, notification payload https://review.opendev.org/c/openstack/nova/+/975240 | |
| 03:54:35 | opendevreview | Ghanshyam proposed openstack/nova master: Add a new field 'topic_alt' in service: object, DB, notification payload https://review.opendev.org/c/openstack/nova/+/975240 | |
| 04:27:08 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| 04:28:21 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| 04:41:10 | gmaan | gibi: bauzas: these two change (adding 2nd RPC server) for graceful shutdown series are ready for review: https://review.opendev.org/c/openstack/nova/+/975240 https://review.opendev.org/c/openstack/nova/+/975242 | |
| 06:40:12 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 06:52:05 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 08:43:32 | bauzas | gmaan: saw those, will try to do review them today but my day is already packed with review needs | |
| 09:39:37 | opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | |
| 10:36:11 | gibi | gmaan: did my first review round on the second RPC patches and left comments | |
| 15:02:52 | gibi | sean-k-mooney: heh I think I see how the oslo.vmware job could run out of threads in the default pool. If there are at least two differnt type of task is using the default pool during a single VM boot like buil_and_run_instance and then later that task spawns _allocate_network_async then if the pool size is e.g. 5 and I send in 10 parallel VM boot requests. Then the first 5 VM boot fills the pool and | |
| 15:02:58 | gibi | then none of them can move forward as the _allocate_network_async task for each of them is also queued up waiting for executor. | |
| 15:04:02 | sean-k-mooney | ah i see | |
| 15:04:56 | sean-k-mooney | i guess im not sure why buil_and_run_instance or _allocate_network_async woudl be dispatced to the defautl thread pool in the first place | |
| 15:05:17 | sean-k-mooney | the should both be on the rpc trhead pool. | |
| 15:06:11 | sean-k-mooney | if we had to use utils.spawn inside if for some async work i can see it having contention but it shoudl not deadlock | |
| 15:06:35 | sean-k-mooney | oh | |
| 15:06:48 | sean-k-mooney | no it could | |
| 15:07:04 | gibi | for some historical reasons build_and_run is moved to the default pool from the RPC pool. And then that task utils.spawns the network task | |
| 15:07:12 | sean-k-mooney | i guess a task that is on a executor shoudl never dispatch an other task to the same executor | |
| 15:07:21 | sean-k-mooney | if it does it could alwasy starve | |
| 15:07:26 | gibi | yeah | |
| 15:07:57 | sean-k-mooney | ok wll i think the network task shoudl jsut move to the io thread pool | |
| 15:08:03 | sean-k-mooney | at least for the short term | |
| 15:08:08 | sean-k-mooney | rather the the default | |
| 15:08:44 | gibi | yeah. Long term I need to think how can we prevent a new task being introduced and spawned to the same pool as the parent task | |
| 15:09:30 | sean-k-mooney | so we are not seeing this in the libvirt driver because this behvior is vmware dirver specific? | |
| 15:10:08 | gibi | nope I see it in the libvirt dirver too | |
| 15:10:10 | gibi | now that I know what to look for | |
| 15:10:25 | gibi | both task are from the compute manager so the problem is virt driver agnostic | |
| 15:11:03 | gibi | it seems tempest in our CI never tried to spawn more than 2-3 VMs in parallel but oslo.vmaware CI does so | |
| 15:11:08 | sean-k-mooney | ack in that case for now we coudl use the io thread pool for all network or file operations and the default for normal function calls? | |
| 15:11:35 | gibi | I will take a closer look at all utils.spawn call to see how much contention we have | |
| 15:11:49 | gibi | but yes in general we need to separate out these tasks to different pools | |
| 15:12:53 | sean-k-mooney | not a now descison but we might actully want a "packaged task" construct in the future where we can package up a closure of the task to be executeded (with cansletion if need) and metadata to tack/prevent this | |
| 15:13:36 | sean-k-mooney | i.e. so we know what executor we are on and on what executor we can spawn the task | |
| 15:14:16 | gibi | yeah a generic solution probably needs some tracking metadata | |
| 15:14:17 | sean-k-mooney | for now we could add a simple execotre fild to the nova context object which is None by defualt but set to a link to the execotre by spawn | |
| 15:14:39 | sean-k-mooney | then spawn could check if we are ever tying to spawn a task on teh same executor and raise | |
| 15:14:49 | gibi | yeah context is not a bad idea | |
| 15:14:50 | gibi | thanks | |
| 15:15:29 | sean-k-mooney | the reason i said we might need a packaged task in the future is i dont know if we have the context as an arguemt every where we are usign spawn but we might | |
| 15:16:12 | sean-k-mooney | i.e. if we are currently spawning a fucntion that does not take the context as its first argument then we woudl either need to add it or find a diffent solution | |
| 15:16:18 | sean-k-mooney | but thats what i woudl try first | |
| 15:16:46 | gibi | context should be in a thread local store, I can check if we can grab it from there | |
| 15:18:00 | sean-k-mooney | oh you might be right i vagely remoemebr ther ebeing a psudo signelton function to get it at module scope | |
| 15:18:13 | gibi | btw, this logic to move the build_and_run_instance from the RPC to the default pool might not be a valid strategy any more for native threading. In eventlet we had 60 RPC worker but potentially unlimited worker in the default pool. So the default pool was bigger. But in native threading the default pool is smaller than the RPC worker pool. | |
| 15:20:02 | gibi | (by default) | |
| 15:20:09 | sean-k-mooney | ya im not sure it really requried (or why it was done) | |
| 15:20:29 | sean-k-mooney | just runign it on the rpc thread i think should be ok | |
| 15:21:29 | sean-k-mooney | even if it was the same size i dont think we shoudl design the system so that with the incorrect config value you can deadlock | |
| 15:21:58 | sean-k-mooney | so threading may be highlighting the problem but i would say it was still a bug even with eventlet | |
| 15:23:03 | gibi | I think it originally was done to avoid build requests to hold up the RPC pool and prevent other lifecycle operations to progress in the meantime | |
| 15:23:58 | gibi | https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/compute/manager.py#L2452-L2460 | |
| 15:24:20 | gibi | maybe we need a separete build_and_run_instance pool then | |
| 15:24:30 | gibi | to limit the parallel boot requests | |
| 15:24:40 | sean-k-mooney | well we have a lock for that | |
| 15:24:44 | sean-k-mooney | btu we could | |
| 15:25:03 | sean-k-mooney | its technially the build semephore not a lock | |
| 15:25:29 | sean-k-mooney | but we could rate limit by the executor worker count instead | |
| 15:25:33 | gibi | yeah we can move that semaphor to be implemented by the pool size | |
| 15:26:06 | gibi | we discussed before that we want this as this allow shutting down the pool during graceful shutdown | |
| 15:26:26 | gibi | then we went to shutting down the whole RPC pool instead | |
| 15:27:13 | sean-k-mooney | so you would shutdown the build pool which would block until done but now allow enquing new builds | |