Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-01-13
16:49:31 gmaan ++
17:05:59 sean-k-mooney gibi: i can take a look. its possibel that we are leakign locks between tests althoguh im not entirly sure how that woudl happen in this case.
17:07:09 sean-k-mooney nova.tests.unit.test_utils.TestFairLockGuard.test_context_manager_shared_between_threads
17:07:39 sean-k-mooney ok os it sthe test case that is testign the cross treadh sharing which is something i explictly did not want to supprot
17:08:30 sean-k-mooney well i need to look at the test code to see what that is testing
17:09:25 gibi sean-k-mooney: thanks for looking. I'm still trying to catch up after my PTO
17:09:48 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/tests/unit/test_utils.py#L1893-L1917 ya it is
17:10:00 sean-k-mooney gibi: no wories did you enjoy the break
17:10:27 sean-k-mooney i have not tested that in threaded mode so ill run that in a loop and see if it repoduces locally
17:10:38 gibi there were parts I enjoyed very much :)
17:15:11 sean-k-mooney ok ya so that test is not stable
17:15:43 sean-k-mooney it cant work relyably because i expclity detect the case wehre we context switch betwen thread 1 and thread 2. https://github.com/openstack/nova/commit/22012360c40045ac1bf6dd0dc95aea3e15fed0cd#diff-d6d688e548906bcae5669480a03749c0a52f32634762e23adfbcd8de88b9d980R1236-R1240
17:16:27 sean-k-mooney biscly the exectpiotn that is beign raised happens if the first worker thread enters the context manager while the other worker thread is also in the context manger
17:16:47 sean-k-mooney if the kernel runs them serially the test will pass
17:16:56 sean-k-mooney if it decied to interleave them it wont
17:17:44 sean-k-mooney we have 3 options. i can remove the raise which was orginally there to prevent this type of sharing
17:17:52 sean-k-mooney i can remove the test
17:18:09 sean-k-mooney or i can alter the test to prevent the interleving but that not very useful
17:18:49 sean-k-mooney since the interation of the patch where i added that raise i added a buch of addtion lockign to make it safe to share the lockgrarud between threads
17:19:37 sean-k-mooney so removing the raise is proably the correct thing to do and just make it a debug log or remove that if and the self._active varible entirly
17:21:00 sean-k-mooney that raise only exists to provide a singal to use that
17:21:03 sean-k-mooney thread1 = threading.Thread(target=worker, args=(1, lock_guard))
17:21:13 sean-k-mooney thread2 = threading.Thread(target=worker, args=(2, lock_guard))
17:21:25 sean-k-mooney is a programing logic error
17:22:01 sean-k-mooney with the extra lockign i have added that shoudl now be safe even if i dislike it so the extra protection is not required
17:22:07 sean-k-mooney any prefernce on how to fix it
17:23:05 sean-k-mooney we can just say the comments are enouch to discourge that useage and delete the check, we can add a debug log or i can remove the test.
17:23:06 gibi I don't know at the moment. Sorry.
17:23:19 gibi I need to get back to this
17:23:30 sean-k-mooney its fine ill create a quick patch to remvoe the raise and ill ask this quetion on the review
17:23:42 sean-k-mooney and you can leave feedback when you have time
17:24:02 gibi cool. Thanks!
17:24:59 sean-k-mooney was there a bug for this if not ill just do "Related-Bug: #2048837" to tie it to the orgianal bug fix
17:25:33 sean-k-mooney tkajinam: ^
17:46:19 opendevreview Takashi Kajinami proposed openstack/nova master: libvirt: Add capability to load loader and nvram from xml https://review.opendev.org/c/openstack/nova/+/969086
18:17:04 opendevreview Takashi Kajinami proposed openstack/nova master: libvirt: Add capability to load ssm feature from existing xml https://review.opendev.org/c/openstack/nova/+/969131
22:59:25 JayF gibi: finally got back around to this post-holidays; no code change was needed and I have an explanation: https://review.opendev.org/c/openstack/nova/+/969321
#openstack-nova - 2026-01-14
05:23:29 opendevreview Merged openstack/nova master: Do not fork compute workers in native threading mode https://review.opendev.org/c/openstack/nova/+/965466
07:17:41 ralonsoh gibi, hello! I've tested pyroute2 0.9.5 in a /requirements patch
07:17:43 ralonsoh https://review.opendev.org/c/openstack/requirements/+/973210
07:18:17 ralonsoh do you mind if we bump to this version (latest) right now? If we detect any other issue with os-vif, we can revert it again
07:39:02 ralonsoh just for reference: https://zuul.opendev.org/t/openstack/buildset/1b150188643d435e8813c9d8be2ea184. I added nova-ceph-multistore and tempest-integrated-compute jobs
08:08:33 opendevreview Merged openstack/os-vif master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/os-vif/+/965493
10:21:16 gokhan Hi everyone,
10:21:16 gokhan We are currently running OpenStack Caracal and our security audit team has raised several critical concerns regarding VNC console access via nova-novncproxy. I would like to get your insights on the best practices or if there are any missing configurations on our side.
10:21:33 gokhan Lack of Session Timeout: Once a VNC connection is established, it remains open indefinitely (or for very long periods), which is seen as a major risk if a workstation is left unattended.
10:21:51 gokhan Missing Rate Limiting: There is no built-in mechanism to limit the number of console URLs a single user can request in a short period, potentially leading to resource exhaustion or brute-force attempts
10:22:10 gokhan URL Hijacking Risk: If a console URL is intercepted, it allows immediate access to the VM without further authentication.
10:23:35 gokhan we are planning to implement: Setting [console] enforce_session_timeout = True and adjusting [consoleauth] token_ttl to enforce a hard disconnect (e.g., after 30-60 minutes), Implementing rate limiting at the HAProxy level using stick-tables based on source IP for the /remote-consoles endpoint, Removing none from auth_schemes to enforce vencrypt only.
10:24:25 gokhan Does enforce_session_timeout in Caracal fully address the "infinite session" issue by terminating the WebSocket proxy connection on the server side once the TTL expires?
10:24:34 gokhan Are there any known side effects of setting a very low token_ttl (e.g., 30s) while enforce_session_timeout is active?
10:25:06 gokhan Regarding the password authentication spec, what is the current stability status for KVM/Libvirt in Caracal? https://specs.openstack.org/openstack/nova-specs/specs/wallaby/approved/nova-support-webvnc-with-password-authentication.html
10:25:45 gokhan I would appreciate any guidance or best practices from the you, thanks :)
10:30:10 gibi gokhan: about vnc timeout we have https://specs.openstack.org/openstack/nova-specs/specs/2024.1/implemented/enforce-remote-console-session-timeout.html
10:35:15 gibi > Does enforce_session_timeout in Caracal fully address the "infinite session" issue by terminating the WebSocket proxy connection on the server side once the TTL expires?
10:35:20 gibi I believe so
10:35:37 gibi > Are there any known side effects of setting a very low token_ttl (e.g., 30s) while enforce_session_timeout is active?
10:37:21 gibi as far as I understand that would mean that the VNC session would disconnect after 30secs, if that is OK for you as a user experience then it is fine from nova perspective to configure such short ttl
10:40:55 gibi regarding password auth, it was never implemented. I think the last attempt was https://review.opendev.org/c/openstack/nova/+/622336
10:42:08 gibi it had an approved spec so I think the nova core team was OK with that feature proposal. So if you are interested it can be revived.
10:43:53 gibi if you are new to openstack then here are our contributor guide https://docs.openstack.org/nova/latest/contributor/index.html
10:48:37 gibi JayF: +2+A thanks for the explanation
10:56:58 gokhan Regarding the password authentication; it’s a bit disappointing that it wasn’t fully implemented, but knowing its current status helps us explain the situation to our security audit team. We will evaluate the possibility of developing and reviving this feature with our team.
10:56:58 gokhan Thanks a lot for the clarification, Gibi!
10:56:58 gokhan It is very helpful to know that enforce_session_timeout is the server-side solution we were looking for. We now understand that token_ttl serves as both the link validity period and the maximum session duration when this feature is enabled. We will set a reasonable TTL (like 3600s) to balance security and user experience.
10:58:46 gibi ralonsoh: re pyroute2 0.9.5 the job you run actaully shows the same error about the event loop https://zuul.opendev.org/t/openstack/build/9e54327cbaa94a4b8ecf9c5472934c97/log/controller/logs/screen-n-cpu.txt#48564-48581
10:59:15 gibi gokhan: cool. Happy to help
11:00:51 ralonsoh gibi, ok I thought that was fixed
11:04:56 gokhan gibi, I have also created a bug : Live Migration Rollback Corrupts Block Device Mapping of Unrelated VM : https://bugs.launchpad.net/nova/+bug/2133501 do you have any insgihts about it ? we will send a patch for this. this worked for us but I want to be sure I don't ignore critical things > https://bugs.launchpad.net/nova/+bug/2133501/comments/10
12:11:35 sean-k-mooney gokhan: passward auth was not impletned because it very insecure
12:12:03 sean-k-mooney gokhan: we felt adding ti was mroe harmful as it woudl lead operator to think that it was a sufficnet security mechanium to use in production
12:12:58 sean-k-mooney gokhan: vnc's password auth can have a maxium of i belive 7 ascii pritnabale charters as the password so the password space is very small
12:13:26 sean-k-mooney gokhan: we implented vencypt to do cert based auth between the proxy and the qemu instance instead
12:13:51 sean-k-mooney and then expect the guest passwoard auth to be used on the guest
12:15:11 sean-k-mooney gokhan:... any corss instece behiver is generaly treated as a possible secuirty bug so it shoudl not be filed publicly but its too late to change that now
12:18:10 sean-k-mooney hum i see that proably should have been set as public security rahter hten just public
12:18:20 sean-k-mooney as there are still security and multi tenant aspect to this
12:29:37 opendevreview Balazs Gibizer proposed openstack/nova master: Upgrade note for concurrency mode default change https://review.opendev.org/c/openstack/nova/+/969888
12:39:15 opendevreview Balazs Gibizer proposed openstack/nova master: Remove eventlet from CacheConcurrencyTestCase https://review.opendev.org/c/openstack/nova/+/970069
12:39:15 opendevreview Balazs Gibizer proposed openstack/nova master: Remove eventlet from libvirt/test_driver https://review.opendev.org/c/openstack/nova/+/970070
12:39:16 opendevreview Balazs Gibizer proposed openstack/nova master: Remove eventlet from libvirt/volume/test_mount https://review.opendev.org/c/openstack/nova/+/970071
12:42:35 opendevreview Balazs Gibizer proposed openstack/nova master: Remove eventlet from libvirt/volume/test_mount https://review.opendev.org/c/openstack/nova/+/970071
12:55:35 opendevreview Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2026.2 Hibiscus https://review.opendev.org/c/openstack/nova-specs/+/973373
12:57:30 opendevreview Merged openstack/nova master: [ironic] Use constants from Ironic, test w/ddt https://review.opendev.org/c/openstack/nova/+/969321
12:58:39 gokhan Thanks for the warning and the detailed explanation, Sean.
12:58:40 gokhan I shared this publicly because I saw the "approved" status on the Wallaby spec and assumed it was a standard feature discussion rather than a zero-day vulnerability. Interestingly, even that spec mentions: "the VNC password is not a very strong security mechanism" which aligns perfectly with what you said.
12:58:40 gokhan Following your advice, we will rely on vEncrypt for proxy-to-instance security and utilize enforce_session_timeout (implemented in Caracal) to mitigate the risks raised by our security team.
12:58:40 gokhan I now understand the rationale behind not implementing the 7-8 character weak password auth. We will focus on hardening the session management side instead. Thanks again for your time and the heads-up!
12:59:04 gokhan sean-k-mooney, ^^
13:01:27 sean-k-mooney no worries. when we were eveualting the password auth we had some converstation with the qemu and libvirt maintainer and it was there advice to not implement it in nova and pivot to the vencypt approch orginally
13:02:19 sean-k-mooney the enforce_session_timeout was also part of the dicsussion on general hardening of the capablities
13:04:30 opendevreview Takashi Kajinami proposed openstack/nova master: libvirt: Use firmware auto-selection by libvirt https://review.opendev.org/c/openstack/nova/+/969132
13:12:30 gokhan Thanks for the insights on vEncrypt and the 8-character limit of VNC passwords. sean-k-mooney. we have a specific "Session Hijacking" concern. When a legitimate user requests a console URL, logs into the Guest OS, and then someone else copies that exact URL (with the token) to another browser/machine, they find themselves already logged into the session without needing any credentials.
13:12:59 gokhan Is there a way to enforce "One-Time Use" for console tokens, so that once a WebSocket connection is established, the token becomes invalid for any subsequent connection attempts?
13:13:05 sean-k-mooney gokhan: so the mitigation for that is intened to be using ssl
13:13:30 sean-k-mooney i.e. so that the console token cannot eb intersepted over the wire
13:13:45 sean-k-mooney if your worred about client side hijacts in the browser then that diffent
13:13:54 sean-k-mooney and now we do not have the consept of one time use tokens
13:14:49 sean-k-mooney one time use tokes woudl break browser reresh or when you pop the console out into its own tab in horizong but it could eb something we consider supportign eventually
13:15:31 sean-k-mooney it would not be soemthign we could enabel by default IMO based on the end user ux regression it woudl bring

Earlier   Later