| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-07-14 | |||
| 02:50:03 | opendevreview | yaguang proposed openstack/nova master: libvirt: enable Hyper-V evmcs on Intel VMX hosts https://review.opendev.org/c/openstack/nova/+/995620 | |
| 04:20:18 | opendevreview | Masahito Muroi proposed openstack/nova master: Add bug 2106085 reproducer test case https://review.opendev.org/c/openstack/nova/+/971153 | |
| 04:20:18 | opendevreview | Masahito Muroi proposed openstack/nova master: Add check_instance_state validation to the four action API https://review.opendev.org/c/openstack/nova/+/946223 | |
| 04:37:36 | opendevreview | Bhavana proposed openstack/nova master: Add custom GPU PCI weigher for scheduler optimization Add Minimal Displacement Solver for MIG GPU Repartitioning https://review.opendev.org/c/openstack/nova/+/996058 | |
| 07:26:59 | ralonsoh | hi folks, if you have a few minutes, please check https://review.opendev.org/c/openstack/os-vif/+/995933 | |
| 07:27:14 | ralonsoh | dansmith, ^ thanks in advance | |
| 07:58:36 | gibi | gmaan: thanks I will check your feedback | |
| 09:03:19 | gibi | sean-k-mooney: elodilles: the oslo.concurrency 7.6.0 cross test failure turns out to be a real nova bug we now see but was hidden before https://bugs.launchpad.net/nova/+bug/2160596 Two VMs on the host using the same multi attach volume being deleted in parallel can result in danging volume attachments | |
| 09:03:34 | gibi | as we are not serializing the deletion by volume id just by VM id | |
| 09:04:32 | gibi | as the attachment delete (in the manager) is far from the volume disconnect check (in the driver) it is not easy to just add a new lock. | |
| 09:17:23 | gibi | not dangling volume attachements, but dangling volume host mounts | |
| 09:36:48 | elodilles | gibi: ah, ACK, thanks for the information | |
| 09:46:41 | sean-k-mooney | gibi: hum i really tought i had properly fixed that | |
| 09:46:50 | sean-k-mooney | gibi: did we regress it | |
| 09:47:36 | sean-k-mooney | gibi: that what i was fixing in https://bugs.launchpad.net/nova/+bug/2048837 | |
| 09:48:26 | sean-k-mooney | gibi: if this only happens with the new oslo.concurrency then its a but in oslo | |
| 09:49:38 | sean-k-mooney | gibi: i added per volume locking in https://opendev.org/openstack/nova/commit/72132f89ee4ca7367a6a5639f763f9b20d2c441a | |
| 09:50:19 | sean-k-mooney | sorry in https://opendev.org/openstack/nova/commit/22012360c40045ac1bf6dd0dc95aea3e15fed0cd | |
| 09:52:18 | gibi | I have to look deeper as my simple reproduction does not work without the oslo change so something important changed in oslo | |
| 10:20:20 | gibi | the only real change is oslo.concurrency is eventlet related | |
| 10:23:46 | gibi | reverting https://review.opendev.org/c/openstack/oslo.concurrency/+/988652 stops my reproduction to recreate the issue | |
| 10:24:53 | sean-k-mooney | hum i looked at that and discounted it | |
| 10:25:13 | sean-k-mooney | i kwno that eventlet has import order dependent behvior | |
| 10:25:59 | sean-k-mooney | i dont know why that would break locking | |
| 10:26:11 | gibi | yeah me neither but I will dig further | |
| 10:26:14 | sean-k-mooney | espically when we are not chanign the patching | |
| 10:28:01 | sean-k-mooney | gibi: does the cross job run wiht our without monkey patching? | |
| 10:28:20 | sean-k-mooney | i assume with eventlet for now | |
| 10:28:30 | sean-k-mooney | since that stil technially our default | |
| 10:30:04 | sean-k-mooney | i think this this early out https://review.opendev.org/c/openstack/oslo.concurrency/+/988652/1/oslo_concurrency/lockutils.py#111 | |
| 10:30:25 | sean-k-mooney | that is shared across all locks | |
| 10:30:35 | sean-k-mooney | but https://review.opendev.org/c/openstack/oslo.concurrency/+/988652/1/oslo_concurrency/lockutils.py#131 | |
| 10:30:54 | sean-k-mooney | need to run for all rwlocks when we are patched | |
| 10:31:07 | sean-k-mooney | withihs chagne it will hapen for the first lock but not the rest | |
| 10:31:26 | sean-k-mooney | so _EVENTLET_CHECKED should just be removed | |
| 10:31:37 | gibi | it is eventlet for now | |
| 10:32:06 | sean-k-mooney | the refactor i think is fine it the _EVENTLET_CHECKED caching that i think broke things | |
| 10:32:46 | gibi | you mean the current thread thing breaks our multi threahdead testing | |
| 10:32:56 | gibi | *do you mean? | |
| 10:33:12 | sean-k-mooney | the _EVENTLET_CHECKED early out reintoduced https://github.com/eventlet/eventlet/issues/731 | |
| 10:33:39 | gibi | ack, testing it... | |
| 10:38:22 | gibi | yepp commenting out https://review.opendev.org/c/openstack/oslo.concurrency/+/988652/1/oslo_concurrency/lockutils.py#135 makes it not possible the reproduce the problem | |
| 10:38:36 | gibi | switching the bug to oslo.concurrency... | |
| 10:39:06 | sean-k-mooney | ya so the early out i thinkwas an attempt to not import it over and over | |
| 10:39:20 | sean-k-mooney | but python caches import so its cheap | |
| 10:39:42 | sean-k-mooney | if we want to cache somethin we coudl cache the result of eventlet_patcher.is_monkey_patched('thread') | |
| 10:39:57 | sean-k-mooney | but i would just delete it | |
| 10:40:18 | sean-k-mooney | we dont need the extra complications of _EVENTLET_CHECKED | |
| 10:40:47 | sean-k-mooney | brb coffee | |
| 10:44:42 | gibi | I need to craft a oslo.concurrency level of test for this to avoid future regressions | |
| 11:05:58 | sean-k-mooney | in this case just create 2 rwlocks when eventlet is monky patched and assert both have the same method fo self._current_thread | |
| 11:07:49 | sean-k-mooney | a better test might be to test the side effect | |
| 11:08:25 | sean-k-mooney | i.e. that they both agree on the current thread id | |
| 11:15:36 | gibi | yeah I want the latter as that does not depend on the implementation | |
| 11:15:43 | gibi | hm | |
| 11:15:45 | gibi | not event that | |
| 11:15:56 | gibi | I want a test that shows the mutual exlusiveness of a write lock | |
| 11:16:17 | sean-k-mooney | hum ya that is higher level and less coupled | |
| 11:16:41 | sean-k-mooney | well | |
| 11:17:02 | sean-k-mooney | 2 locks are indpemendent | |
| 11:17:32 | sean-k-mooney | so its not as simple as aquire lock a and lock b and try to access somethign shared | |
| 11:18:02 | sean-k-mooney | you need to create 2 locks | |
| 11:18:41 | sean-k-mooney | and aquire both in 2 diffent eventsletlest or somthing | |
| 11:18:53 | sean-k-mooney | ideally with less infram then nova test case | |
| 11:19:44 | gibi | hm. isn't it the second lock I create will not work if used from two different greenlet? | |
| 11:20:13 | sean-k-mooney | its not really that you need two locks | |
| 11:20:33 | sean-k-mooney | in this case it that the second time any RW lock was inited the relevent code was skipped | |
| 11:20:41 | gibi | yepp | |
| 11:20:50 | gibi | so the second lock is broken when shared across greenlets | |
| 11:20:53 | gibi | that is my theory | |
| 11:21:01 | sean-k-mooney | so for the workers you can just use 1 lock and a thread.event | |
| 11:21:17 | sean-k-mooney | gibi: yes thre is a rentrency issue | |
| 11:21:31 | sean-k-mooney | in that when shared the second lock seas the real treadid | |
| 11:21:42 | sean-k-mooney | not the evnetlet greenthread id | |
| 11:22:04 | sean-k-mooney | so it goes one im on the same trhead where i aquired the lock and it races | |
| 11:23:03 | sean-k-mooney | i think its breaking https://github.com/openstack/nova/blob/master/nova/utils.py#L1233 | |
| 11:24:00 | sean-k-mooney | hum although maybe not | |
| 11:24:08 | sean-k-mooney | sinc ei have test for that and those pass | |
| 11:24:23 | sean-k-mooney | it was only one test that actully failed right | |
| 11:24:44 | sean-k-mooney | so its not any fo the simple test case i am already testing | |
| 11:26:20 | sean-k-mooney | im going to stop guessing if you come up with a test that works let me know and i can take a look | |
| 11:31:21 | gibi | sure I'm working on it | |
| 11:31:59 | gibi | it is a bit more complicated due to https://github.com/openstack/oslo.concurrency/blob/9aff32494fb97b7fc7508269224353aceb9ceef6/oslo_concurrency/lockutils.py#L101-L103 | |
| 11:32:23 | gibi | so they are also passing threading.current_thread to the parent | |
| 11:32:49 | gibi | sorry I ment https://github.com/openstack/oslo.concurrency/blob/9aff32494fb97b7fc7508269224353aceb9ceef6/oslo_concurrency/lockutils.py#L101-L108 | |
| 11:33:30 | sean-k-mooney | ya i was looking at that functor | |
| 11:34:19 | sean-k-mooney | threading.current_thread should be monkeypatch to eventlet.getcurrent in general before we hit this code | |
| 11:34:34 | sean-k-mooney | if we are usign eventlet | |
| 11:35:44 | sean-k-mooney | https://github.com/openstack/oslo.concurrency/commit/02dee958453513bb8a5aaae905312d306c2d9083#diff-60d0bce7c34c2c70cf4e3ac63060779235ede2a247bd39d1188f8fedd6306b38R109 | |
| 11:35:54 | sean-k-mooney | so that defaulting was added in the typeing patch | |
| 11:36:27 | gibi | ahh I see it, I need to use spawn_n not spawn to reproduce it | |
| 11:36:28 | sean-k-mooney | and it proably should not be | |
| 11:36:40 | sean-k-mooney | oh | |
| 11:36:40 | gibi | the original eventlet issue is only affecting spawn_n | |
| 11:36:46 | gibi | not spawn | |
| 11:36:54 | gibi | thanks melwitt for the good bug report | |
| 11:37:12 | sean-k-mooney | i know that changes between a raw greenlet and a greenthread | |
| 11:37:19 | sean-k-mooney | is that really the level where it broke | |
| 11:38:07 | gibi | yepp | |
| 11:38:09 | sean-k-mooney | i tought i had a patch to remove all useage of spawn_n at one point | |