Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-01-30
15:08:03 sean-k-mooney at least for the short term
15:08:08 sean-k-mooney rather the the default
15:08:44 gibi yeah. Long term I need to think how can we prevent a new task being introduced and spawned to the same pool as the parent task
15:09:30 sean-k-mooney so we are not seeing this in the libvirt driver because this behvior is vmware dirver specific?
15:10:08 gibi nope I see it in the libvirt dirver too
15:10:10 gibi now that I know what to look for
15:10:25 gibi both task are from the compute manager so the problem is virt driver agnostic
15:11:03 gibi it seems tempest in our CI never tried to spawn more than 2-3 VMs in parallel but oslo.vmaware CI does so
15:11:08 sean-k-mooney ack in that case for now we coudl use the io thread pool for all network or file operations and the default for normal function calls?
15:11:35 gibi I will take a closer look at all utils.spawn call to see how much contention we have
15:11:49 gibi but yes in general we need to separate out these tasks to different pools
15:12:53 sean-k-mooney not a now descison but we might actully want a "packaged task" construct in the future where we can package up a closure of the task to be executeded (with cansletion if need) and metadata to tack/prevent this
15:13:36 sean-k-mooney i.e. so we know what executor we are on and on what executor we can spawn the task
15:14:16 gibi yeah a generic solution probably needs some tracking metadata
15:14:17 sean-k-mooney for now we could add a simple execotre fild to the nova context object which is None by defualt but set to a link to the execotre by spawn
15:14:39 sean-k-mooney then spawn could check if we are ever tying to spawn a task on teh same executor and raise
15:14:49 gibi yeah context is not a bad idea
15:14:50 gibi thanks
15:15:29 sean-k-mooney the reason i said we might need a packaged task in the future is i dont know if we have the context as an arguemt every where we are usign spawn but we might
15:16:12 sean-k-mooney i.e. if we are currently spawning a fucntion that does not take the context as its first argument then we woudl either need to add it or find a diffent solution
15:16:18 sean-k-mooney but thats what i woudl try first
15:16:46 gibi context should be in a thread local store, I can check if we can grab it from there
15:18:00 sean-k-mooney oh you might be right i vagely remoemebr ther ebeing a psudo signelton function to get it at module scope
15:18:13 gibi btw, this logic to move the build_and_run_instance from the RPC to the default pool might not be a valid strategy any more for native threading. In eventlet we had 60 RPC worker but potentially unlimited worker in the default pool. So the default pool was bigger. But in native threading the default pool is smaller than the RPC worker pool.
15:20:02 gibi (by default)
15:20:09 sean-k-mooney ya im not sure it really requried (or why it was done)
15:20:29 sean-k-mooney just runign it on the rpc thread i think should be ok
15:21:29 sean-k-mooney even if it was the same size i dont think we shoudl design the system so that with the incorrect config value you can deadlock
15:21:58 sean-k-mooney so threading may be highlighting the problem but i would say it was still a bug even with eventlet
15:23:03 gibi I think it originally was done to avoid build requests to hold up the RPC pool and prevent other lifecycle operations to progress in the meantime
15:23:58 gibi https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/compute/manager.py#L2452-L2460
15:24:20 gibi maybe we need a separete build_and_run_instance pool then
15:24:30 gibi to limit the parallel boot requests
15:24:40 sean-k-mooney well we have a lock for that
15:24:44 sean-k-mooney btu we could
15:25:03 sean-k-mooney its technially the build semephore not a lock
15:25:29 sean-k-mooney but we could rate limit by the executor worker count instead
15:25:33 gibi yeah we can move that semaphor to be implemented by the pool size
15:26:06 gibi we discussed before that we want this as this allow shutting down the pool during graceful shutdown
15:26:26 gibi then we went to shutting down the whole RPC pool instead
15:27:13 sean-k-mooney so you would shutdown the build pool which would block until done but now allow enquing new builds
15:27:19 gibi yes
15:27:34 sean-k-mooney i assume calling submit on an executore that you have called shutdown on will raise?
15:27:39 gibi yepp
15:28:10 gibi it is a bit moot point as with the current graceful shutdown plans we will shut down the whole RPC topic so no extra request will come in anyhow
15:28:11 sean-k-mooney ok so that could allow us to catch that and gracefully raise a buidl exepction allowing a requeue
15:29:47 sean-k-mooney i need to think about this a bit but it almost sound like what we really want instead of multipe execotres is an execotr that under stand task priorities
15:30:32 sean-k-mooney i.e. submit the build with the lowest peririty then if that submit a new request iw would submit it with context.priority+1
15:31:03 sean-k-mooney well no
15:31:19 sean-k-mooney that only works with corutiens that can be suspended
15:31:37 sean-k-mooney because we need to be able to premet the lower priority task
15:32:21 gibi yeah I vote for the thinking more approach :) it is friday afternoon
15:32:27 sean-k-mooney gibi: so im not really concerd with the idea of having 1 or 2 or even 3 pools (default, io and rpc)
15:32:44 sean-k-mooney but if we need like 5 pools i woudl be a bit concerned
15:32:55 gibi yeah I understand that fear
15:33:30 sean-k-mooney am i didnt get a chance to look at your event handeling patch by the way
15:33:45 sean-k-mooney its a public holdiay on monday so it will likely be tuesday before i have time
15:33:51 gibi no worries
15:34:01 gibi I have now plenty of thinking work
15:34:13 gibi enjoy your day off
15:48:32 gibi I summarized this discussion back to gerrit https://review.opendev.org/c/openstack/nova/+/965467/44#message-191254e8132d5162ced2217faeb06910e85527d5
15:51:56 gmaan bauzas: gibi thnaks
18:25:03 sean-k-mooney gibi: your probaly done for today but https://review.opendev.org/c/openstack/nova/+/975171/1 is the repoducer for the iothread live migration issue if you have time or anyone else. sylvain is +2 and the fix patch is approved
18:25:55 sean-k-mooney gmaan: if you want me to rework https://review.opendev.org/c/openstack/nova/+/973438/comment/ff4bb8ea_53533e58/ i can but it will be a tuesday task
18:26:12 sean-k-mooney im going to wrap up for the long weekend shortly
18:28:46 gmaan sean-k-mooney: checking your reply
18:42:24 gmaan sean-k-mooney: I think catching exception and reseting the internal state is more safe and better
18:43:37 gmaan i know release_write_lock does not raise exception other then other writter try to unlock them but 1. its 3rd party so changes can happen there 2. resting internal state before actually releasing lock seems odd to m,e
18:46:40 gmaan gibi: let's discuss on Monday about the DB/object changes for new RPC topic. I was thinking the same as you commented but proposed the changes because old field 'topic' was in DB/object. I will comment in review too.
18:49:00 gmaan dansmith: maybe you remember if any rational or it is from starting, we have RPC 'topic' field in DB/Object but it is not user facing or changable things - https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/objects/service.py#L328
18:49:27 gmaan as I am adding a new RPC server for graceful shutdown, I proposed to add that new topic field 'topic_alt' in DB/object - https://review.opendev.org/c/openstack/nova/+/975240/2
18:50:01 sean-k-mooney gmaan: ok ill do that quickly then before i leave
18:50:17 dansmith I don't remember specifically, I suspect it's a holdover from much longer ago where "everything was pluggable"
18:50:48 gmaan ohk, In that case, we can leave this old field but not add the new topic field as no use case as of now
18:50:51 gmaan sean-k-mooney: ack
18:51:27 dansmith yeah doesn't appear to be used anywhere
18:52:20 gmaan k, will abandon my change, if we need it can be added in future anyways
18:52:31 dansmith ah, looks like it may have been a nova-network thing
18:52:43 dansmith like, to get the network manager for a host, which may be shared
18:53:29 dansmith gmaan: https://github.com/openstack/nova/blob/icehouse-eol/nova/network/manager.py#L249
18:53:34 dansmith that's a deep cut :)
18:55:12 gmaan ohk, I see that. I tried to find history in but did not go that old. :) thanks
18:55:54 dansmith objects were maturing around icehouse so I usually use that as the point to look for pre-object histopry :)
18:56:44 gmaan i remember icehouse that was the release I started my upstream work :)
18:56:58 dansmith you're such a newb ;P
18:57:22 gmaan :)
19:19:40 opendevreview sean mooney proposed openstack/nova master: FairLockGuard: Support cross-thread sharing and nesting https://review.opendev.org/c/openstack/nova/+/973438
19:41:58 opendevreview Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242
19:43:35 opendevreview Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242
#openstack-nova - 2026-01-31
00:36:25 opendevreview melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106
03:02:21 opendevreview melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106
03:02:21 opendevreview melanie witt proposed openstack/nova master: testing: Run functional tests under [testenv:cover] https://review.opendev.org/c/openstack/nova/+/975346
#openstack-nova - 2026-02-02
08:02:31 obre Hi! There is a BP (add-amx-traits) thats been lying around since 2023, which describes a feature I would like to see in nova, and that I guess Im able to implement. The BP is not approved, and posted by someone else. Should I in some way update the BP and assign it to myself, or should I create a new one to get it approved and then Implemented?
08:17:39 mkuroha gibi: Please take another look when you have time. I’ve addressed your comments. https://review.opendev.org/c/openstack/nova/+/948304
08:41:21 ykarel bauzas, auniyal can you check os-vif release patch https://review.opendev.org/c/openstack/releases/+/975181
10:03:22 elodilles auniyal bauzas Uggla : hi, sorry for pinging, but there are a couple of release patches that would require a release liaison review o:) if you have some spare time, then please take a look: https://review.opendev.org/q/project:openstack/releases+reviewer:self+is:open
11:15:59 gibi gmaan: thanks for the review on the eventlet removal side. I see you went back and forth about waiting for the deadline of the currently waited for task. I think I have a way out of that debate. Currently the libvirt driver will cancel() all the outstanding tasks on the executor then shut it down. So at shutdown the executor can be awaken, check if the current task is cancelled or not then decide
11:16:05 gibi what to do. This way a cancelled task's deadline does not need to be waited for
12:01:16 opendevreview Merged openstack/nova stable/2025.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975162
12:48:06 opendevreview Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445

Earlier   Later