| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-07-30 | |||
| 16:49:31 | gibi | we can land https://review.opendev.org/c/openstack/nova/+/992862 and see if it helps | |
| 16:49:31 | sean-k-mooney | this was in the eventlet based job | |
| 16:50:17 | sean-k-mooney | i mena i can go review that now in either case | |
| 16:50:28 | gibi | we could add some tracing to the lock handling and see if that shows that we somehow introduced the lock to the eventlet job as well | |
| 16:50:30 | sean-k-mooney | runing jgilaber's patch locally its not repoduceing for me | |
| 16:51:35 | sean-k-mooney | hum | |
| 16:52:03 | sean-k-mooney | gibi: so this woudl be a good case to use the FiarLockGuarde i wrote | |
| 16:52:49 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/utils.py#L1149 | |
| 16:59:48 | gibi | I'm not following how | |
| 17:00:12 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/992862/comment/6022582b_82867746/ | |
| 17:01:06 | gibi | but fairlockguard is to handle when multiple locks is being taken in an order | |
| 17:01:13 | gibi | here we only take one lock | |
| 17:01:16 | sean-k-mooney | no | |
| 17:01:22 | sean-k-mooney | that a usecase | |
| 17:01:38 | sean-k-mooney | it was intened to be used anytime you need to aquire the same named lock in diffent context | |
| 17:01:46 | sean-k-mooney | not just multipel locks | |
| 17:03:42 | gibi | I feel like here we have a simpler use case than what the fairlockguard is created for. Sure fairlockguard is also capable of doing this but to do this we don't necessary need the complexity of the fairlockguard | |
| 17:04:04 | sean-k-mooney | right but i wnated ot creat that so we would stop using one of version of it | |
| 17:04:24 | sean-k-mooney | this is reimpelnted a new lock guard | |
| 17:05:36 | sean-k-mooney | we dont have to use it i think it woudl be good to use it when we need soemthing like this in the futre | |
| 17:06:40 | gibi | it was a long day so I might not capable of seeing the whole picture any more but fairlockguard could be used in any place where a single lock is taken, but we don't want to replace all the locks in our code with a fairlockguard as that would be overkill | |
| 17:07:20 | gibi | fairlockguard shines where you have a list of lock to be taken | |
| 17:07:26 | gibi | and you want to do that safely | |
| 17:07:50 | gibi | if you have just a single lock to take at a time then it is overkill in my eyes | |
| 17:08:23 | sean-k-mooney | the bit i dont like is the fact your creating collections.defaultdict(threading.RLock) | |
| 17:08:31 | sean-k-mooney | and locking that collection access | |
| 17:08:48 | sean-k-mooney | that changed it form a singel lock to a set of related locks | |
| 17:09:40 | sean-k-mooney | even if any one invoction of the _locked_scope context manager is only useing one lock | |
| 17:12:40 | gibi | we could monkey patch the lock object into transaction_context_manager._root for each DB then the DatabaseWriteLock does not need to look up the lock in a dict that maps the context manager to a lock | |
| 17:13:15 | sean-k-mooney | we could im not aginst merging the patch as it is | |
| 17:13:24 | gibi | but as we already hooking into sqla here it would be extra complexity to hook this deep into it again at DB creation | |
| 17:13:47 | sean-k-mooney | but we spent a lot of effort addign nesting suprot and other infra in the fairlockgurard and testing to verify it for complex lockign cases like this | |
| 17:15:31 | gibi | but exactly those things are not needed in this case | |
| 17:15:37 | gibi | (I think) | |
| 17:15:59 | gibi | it is not such a complex case | |
| 17:16:00 | sean-k-mooney | you have muyltiple related locks, nesting / rentrancy | |
| 17:16:11 | sean-k-mooney | hence the callout why you needed to RLOCKS | |
| 17:16:16 | gibi | these locks are not related each is for its own DB | |
| 17:16:59 | gibi | they are not guarding the same shared resouce it is one lock - one DB | |
| 17:17:07 | sean-k-mooney | i know | |
| 17:17:28 | sean-k-mooney | but you all out thing like "An RLock is used per database because a single thread may nest writer | |
| 17:17:30 | sean-k-mooney | transactions against the same engine facade." | |
| 17:18:20 | gibi | and that is exactly what an RLock is for :) | |
| 17:18:48 | sean-k-mooney | and its also one of the thing build into the FairlockGurads constaitns | |
| 17:18:54 | sean-k-mooney | ill upgarde to a +2 for now | |
| 17:19:19 | sean-k-mooney | but if this proves unstabel i would prefrer ot adotp the FairLockGuard instead fo theying ot fix this implemetion | |
| 17:19:28 | sean-k-mooney | gibi: does that work for you | |
| 17:20:54 | gibi | if the instability is due to a problem that is solved by the fairlockguard then yes it is good for me | |
| 17:22:28 | sean-k-mooney | patchset 8 was created 21 days ago | |
| 17:22:43 | sean-k-mooney | so that before your shutdown speedup patch landed yes | |
| 17:23:11 | sean-k-mooney | so we woudl execte a new run to also be faster | |
| 17:23:47 | sean-k-mooney | or rather the thereaded and eventet version shoudl be closer now | |
| 17:41:23 | jgilaber | sean-k-mooney, so after clearing the cache the functional tests are green again https://zuul.opendev.org/t/openstack/buildset/6a7d3edc4ae84ba7b33487e037f1acc5 | |
| 17:41:50 | sean-k-mooney | jgilaber: with an without that change i was not able to cause it to fial locally | |
| 17:41:56 | sean-k-mooney | i still have it runingin the backgorund | |
| 17:42:20 | sean-k-mooney | but if ci is happy i guess that is progress | |
| 17:43:31 | jgilaber | yeah, I don't like that I don't know what was wrong there, but good to advance | |
| 17:44:15 | jgilaber | I guess CI harware is slower than our local machines and caused the problem to surface somehow | |
| 17:44:58 | sean-k-mooney | yes that happens | |
| 17:45:44 | sean-k-mooney | you can sort of emulatie that by messign with some of the test teimouts | |
| 17:46:03 | sean-k-mooney | but then its hard to tell if tis a real failure or not | |
| 17:47:59 | gmaan | gibi: you have time to talk about unify the executors get/destroy comment https://review.opendev.org/c/openstack/nova/+/998571/comment/23f68a57_21843d10/ | |
| 17:48:16 | gmaan | if you are done for the day, I can add comment on gerrit and we can discuss there or tomorrow | |
| 18:01:01 | opendevreview | Merged openstack/nova master: tests: Use per-database write locks instead of global lock https://review.opendev.org/c/openstack/nova/+/992862 | |
| 19:22:58 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 19:22:58 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Fix deny_share host locality for NFS https://review.opendev.org/c/openstack/nova/+/998752 | |
| 19:54:06 | Zhan[m] | Hi friends, recently I found a somewhat tricky bug with live migration: https://bugs.launchpad.net/nova/+bug/2160728. Wondering if I can get some feedbacks on it and I can work on the fix too, thanks! | |
| 22:47:33 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Allow rebuild for instances with virtiofs shares https://review.opendev.org/c/openstack/nova/+/999048 | |
| #openstack-nova - 2026-07-31 | |||
| 01:45:38 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 01:45:39 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 02:02:22 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 02:02:23 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 07:35:17 | gibi | gmaan: sorry I was already done for the day at that point yesterday. | |
| 08:03:40 | opendevreview | Roman Hros proposed openstack/nova master: Delete inactive port bindings during live migration https://review.opendev.org/c/openstack/nova/+/999374 | |
| 08:25:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: check tempest for instability https://review.opendev.org/c/openstack/nova/+/999378 | |
| 09:30:53 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 09:30:53 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 09:34:32 | opendevreview | Roman Hros proposed openstack/nova master: Delete inactive port bindings during live migration https://review.opendev.org/c/openstack/nova/+/999374 | |
| 10:20:32 | opendevreview | Roman Hros proposed openstack/nova master: Delete inactive port bindings during live migration https://review.opendev.org/c/openstack/nova/+/999374 | |
| 10:34:15 | ralonsoh | sean-k-mooney, hi! will you have some time to talk about https://bugs.launchpad.net/nova/+bug/2161894? | |
| 10:34:38 | ralonsoh | In particular about `create_tap`!=`ovs_create_tap` | |
| 10:38:21 | sean-k-mooney | sure am are you free now ish, or at the top of the hour? | |
| 10:38:34 | sean-k-mooney | i was going to go grab a coffee quickly | |
| 10:38:56 | sean-k-mooney | but we can chat in say 5 mins | |
| 10:39:41 | ralonsoh | perfect | |
| 10:51:51 | sean-k-mooney | those https://meet.google.com/nbj-ernx-ymb work for you other are fee to join to we will be talking about https://bugs.launchpad.net/nova/+bug/2161894 a live migration instablity with ml2/ovs | |
| 12:51:17 | opendevreview | Roman Hros proposed openstack/nova master: Delete inactive port bindings during live migration https://review.opendev.org/c/openstack/nova/+/999374 | |
| 17:21:53 | gmaan | gibi: no issue, I commented on the change, I think doing option 3 (moving all thread/executors things in a separate file) will make it easy to maintain in long term | |
| 17:22:18 | gmaan | anyways, will wait for your feedback before I start doing the modification | |
| 18:29:18 | gmaan | dansmith: thanks for review graceful shutdown change, I replied please check, mainly for this comment if the approach i replied is ok https://review.opendev.org/c/openstack/nova/+/996299/comment/5c3a3d23_ff206cc3/ | |
| 18:30:51 | gmaan | for revert_resize I think making dest revert_resize idempotent will work, but I will handle that in separate change with functional reproducer test | |
| 18:36:29 | dansmith | gmaan: reading but just realizing I forgot to delete half a sentence in there :/ | |
| 18:46:54 | dansmith | gmaan: replied | |
| 18:58:00 | gmaan | dansmith: thanks, for task end. I end task for both success and failure case. do you think recording the status also can be useful for shutdown case? we will be returning the erorr to users in that case but in case that is missed as we are shutting down, having tasks failed can be useful for users to trigger them again? | |
| 18:58:50 | dansmith | gmaan: you mean logging what tasks failed during shutdown right? | |
| 18:58:57 | gmaan | yeah | |
| 18:59:07 | dansmith | yeah, definitely seems good to log that stuff | |
| 18:59:31 | gmaan | ok, will add that | |
| 18:59:58 | dansmith | on the multiple end thing, I'm really just saying that if we expect one start and one end for each task, but we get a second end it might mean we raced or lost track of something and so it'd be good to flag it .. if it is expected for some reason then no need | |
| 19:02:08 | gmaan | ah i see yeah missed to record start task is good point. | |