| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-30 | |||
| 15:18:13 | gibi | btw, this logic to move the build_and_run_instance from the RPC to the default pool might not be a valid strategy any more for native threading. In eventlet we had 60 RPC worker but potentially unlimited worker in the default pool. So the default pool was bigger. But in native threading the default pool is smaller than the RPC worker pool. | |
| 15:20:02 | gibi | (by default) | |
| 15:20:09 | sean-k-mooney | ya im not sure it really requried (or why it was done) | |
| 15:20:29 | sean-k-mooney | just runign it on the rpc thread i think should be ok | |
| 15:21:29 | sean-k-mooney | even if it was the same size i dont think we shoudl design the system so that with the incorrect config value you can deadlock | |
| 15:21:58 | sean-k-mooney | so threading may be highlighting the problem but i would say it was still a bug even with eventlet | |
| 15:23:03 | gibi | I think it originally was done to avoid build requests to hold up the RPC pool and prevent other lifecycle operations to progress in the meantime | |
| 15:23:58 | gibi | https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/compute/manager.py#L2452-L2460 | |
| 15:24:20 | gibi | maybe we need a separete build_and_run_instance pool then | |
| 15:24:30 | gibi | to limit the parallel boot requests | |
| 15:24:40 | sean-k-mooney | well we have a lock for that | |
| 15:24:44 | sean-k-mooney | btu we could | |
| 15:25:03 | sean-k-mooney | its technially the build semephore not a lock | |
| 15:25:29 | sean-k-mooney | but we could rate limit by the executor worker count instead | |
| 15:25:33 | gibi | yeah we can move that semaphor to be implemented by the pool size | |
| 15:26:06 | gibi | we discussed before that we want this as this allow shutting down the pool during graceful shutdown | |
| 15:26:26 | gibi | then we went to shutting down the whole RPC pool instead | |
| 15:27:13 | sean-k-mooney | so you would shutdown the build pool which would block until done but now allow enquing new builds | |
| 15:27:19 | gibi | yes | |
| 15:27:34 | sean-k-mooney | i assume calling submit on an executore that you have called shutdown on will raise? | |
| 15:27:39 | gibi | yepp | |
| 15:28:10 | gibi | it is a bit moot point as with the current graceful shutdown plans we will shut down the whole RPC topic so no extra request will come in anyhow | |
| 15:28:11 | sean-k-mooney | ok so that could allow us to catch that and gracefully raise a buidl exepction allowing a requeue | |
| 15:29:47 | sean-k-mooney | i need to think about this a bit but it almost sound like what we really want instead of multipe execotres is an execotr that under stand task priorities | |
| 15:30:32 | sean-k-mooney | i.e. submit the build with the lowest peririty then if that submit a new request iw would submit it with context.priority+1 | |
| 15:31:03 | sean-k-mooney | well no | |
| 15:31:19 | sean-k-mooney | that only works with corutiens that can be suspended | |
| 15:31:37 | sean-k-mooney | because we need to be able to premet the lower priority task | |
| 15:32:21 | gibi | yeah I vote for the thinking more approach :) it is friday afternoon | |
| 15:32:27 | sean-k-mooney | gibi: so im not really concerd with the idea of having 1 or 2 or even 3 pools (default, io and rpc) | |
| 15:32:44 | sean-k-mooney | but if we need like 5 pools i woudl be a bit concerned | |
| 15:32:55 | gibi | yeah I understand that fear | |
| 15:33:30 | sean-k-mooney | am i didnt get a chance to look at your event handeling patch by the way | |
| 15:33:45 | sean-k-mooney | its a public holdiay on monday so it will likely be tuesday before i have time | |
| 15:33:51 | gibi | no worries | |
| 15:34:01 | gibi | I have now plenty of thinking work | |
| 15:34:13 | gibi | enjoy your day off | |
| 15:48:32 | gibi | I summarized this discussion back to gerrit https://review.opendev.org/c/openstack/nova/+/965467/44#message-191254e8132d5162ced2217faeb06910e85527d5 | |
| 15:51:56 | gmaan | bauzas: gibi thnaks | |
| 18:25:03 | sean-k-mooney | gibi: your probaly done for today but https://review.opendev.org/c/openstack/nova/+/975171/1 is the repoducer for the iothread live migration issue if you have time or anyone else. sylvain is +2 and the fix patch is approved | |
| 18:25:55 | sean-k-mooney | gmaan: if you want me to rework https://review.opendev.org/c/openstack/nova/+/973438/comment/ff4bb8ea_53533e58/ i can but it will be a tuesday task | |
| 18:26:12 | sean-k-mooney | im going to wrap up for the long weekend shortly | |
| 18:28:46 | gmaan | sean-k-mooney: checking your reply | |
| 18:42:24 | gmaan | sean-k-mooney: I think catching exception and reseting the internal state is more safe and better | |
| 18:43:37 | gmaan | i know release_write_lock does not raise exception other then other writter try to unlock them but 1. its 3rd party so changes can happen there 2. resting internal state before actually releasing lock seems odd to m,e | |
| 18:46:40 | gmaan | gibi: let's discuss on Monday about the DB/object changes for new RPC topic. I was thinking the same as you commented but proposed the changes because old field 'topic' was in DB/object. I will comment in review too. | |
| 18:49:00 | gmaan | dansmith: maybe you remember if any rational or it is from starting, we have RPC 'topic' field in DB/Object but it is not user facing or changable things - https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/objects/service.py#L328 | |
| 18:49:27 | gmaan | as I am adding a new RPC server for graceful shutdown, I proposed to add that new topic field 'topic_alt' in DB/object - https://review.opendev.org/c/openstack/nova/+/975240/2 | |
| 18:50:01 | sean-k-mooney | gmaan: ok ill do that quickly then before i leave | |
| 18:50:17 | dansmith | I don't remember specifically, I suspect it's a holdover from much longer ago where "everything was pluggable" | |
| 18:50:48 | gmaan | ohk, In that case, we can leave this old field but not add the new topic field as no use case as of now | |
| 18:50:51 | gmaan | sean-k-mooney: ack | |
| 18:51:27 | dansmith | yeah doesn't appear to be used anywhere | |
| 18:52:20 | gmaan | k, will abandon my change, if we need it can be added in future anyways | |
| 18:52:31 | dansmith | ah, looks like it may have been a nova-network thing | |
| 18:52:43 | dansmith | like, to get the network manager for a host, which may be shared | |
| 18:53:29 | dansmith | gmaan: https://github.com/openstack/nova/blob/icehouse-eol/nova/network/manager.py#L249 | |
| 18:53:34 | dansmith | that's a deep cut :) | |
| 18:55:12 | gmaan | ohk, I see that. I tried to find history in but did not go that old. :) thanks | |
| 18:55:54 | dansmith | objects were maturing around icehouse so I usually use that as the point to look for pre-object histopry :) | |
| 18:56:44 | gmaan | i remember icehouse that was the release I started my upstream work :) | |
| 18:56:58 | dansmith | you're such a newb ;P | |
| 18:57:22 | gmaan | :) | |
| 19:19:40 | opendevreview | sean mooney proposed openstack/nova master: FairLockGuard: Support cross-thread sharing and nesting https://review.opendev.org/c/openstack/nova/+/973438 | |
| 19:41:58 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| 19:43:35 | opendevreview | Ghanshyam proposed openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242 | |
| #openstack-nova - 2026-01-31 | |||
| 00:36:25 | opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | |
| 03:02:21 | opendevreview | melanie witt proposed openstack/nova master: testing: Run functional tests under [testenv:cover] https://review.opendev.org/c/openstack/nova/+/975346 | |
| 03:02:21 | opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | |
| #openstack-nova - 2026-02-02 | |||
| 08:02:31 | obre | Hi! There is a BP (add-amx-traits) thats been lying around since 2023, which describes a feature I would like to see in nova, and that I guess Im able to implement. The BP is not approved, and posted by someone else. Should I in some way update the BP and assign it to myself, or should I create a new one to get it approved and then Implemented? | |
| 08:17:39 | mkuroha | gibi: Please take another look when you have time. I’ve addressed your comments. https://review.opendev.org/c/openstack/nova/+/948304 | |
| 08:41:21 | ykarel | bauzas, auniyal can you check os-vif release patch https://review.opendev.org/c/openstack/releases/+/975181 | |
| 10:03:22 | elodilles | auniyal bauzas Uggla : hi, sorry for pinging, but there are a couple of release patches that would require a release liaison review o:) if you have some spare time, then please take a look: https://review.opendev.org/q/project:openstack/releases+reviewer:self+is:open | |
| 11:15:59 | gibi | gmaan: thanks for the review on the eventlet removal side. I see you went back and forth about waiting for the deadline of the currently waited for task. I think I have a way out of that debate. Currently the libvirt driver will cancel() all the outstanding tasks on the executor then shut it down. So at shutdown the executor can be awaken, check if the current task is cancelled or not then decide | |
| 11:16:05 | gibi | what to do. This way a cancelled task's deadline does not need to be waited for | |
| 12:01:16 | opendevreview | Merged openstack/nova stable/2025.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975162 | |
| 12:48:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 12:48:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 12:48:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 12:48:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | |
| 12:48:08 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 12:49:05 | gibi | gmaan: ^^ improvement is up. I will still need to do a cleanup round on the patches (e.g. moving prints to log) | |
| 13:58:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 13:58:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 13:58:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 13:58:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | |
| 13:58:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 14:05:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | |
| 14:05:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | |
| 14:05:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 14:05:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 14:05:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove spawn_after https://review.opendev.org/c/openstack/nova/+/975396 | |
| 14:06:57 | gibi | gmaan: and now the cleanups are also done | |
| 14:20:55 | gibi | gmaan: hold on I'm seeing issuess | |
| 14:32:52 | dansmith | gibi: did you see my response on the async detach patch and asking you for clarification? | |
| 14:33:20 | gibi | dansmith: not yet, but I will check now... | |
| 14:45:18 | gibi | dansmith: I replied | |
| 14:46:09 | dansmith | ack | |
| 14:48:21 | gibi | you basically answered my question, we have no issue here, I only have a minor request to translate 500 to 503 | |
| 14:50:48 | opendevreview | Merged openstack/nova stable/2025.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975163 | |