| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-22 | |||
| 09:07:52 | gibi | BUT | |
| 09:08:22 | gibi | in our env the envtlet monkey patching is changing python's select.select | |
| 09:08:25 | gibi | /usr/local/lib/python3.10/dist-packages/eventlet/green/select.py:80 in select | |
| 09:08:46 | gibi | and redirects it to implement the envtlet switching mechanism | |
| 09:11:16 | gibi | https://github.com/eventlet/eventlet/blob/88ec603404b2ed25c610dead75d4693c7b3e8072/eventlet/green/select.py#L30-L80C32 | |
| 09:12:34 | gibi | looking at that code it seems enventlet sets a timer with the timeout value | |
| 09:12:45 | gibi | via hub.schedule_call_global | |
| 09:17:05 | gibi | here I'm getting lost in the eventlet code but I assume sheduling a timer with 0.0 timeout in eventlet can be racy | |
| 09:17:31 | gibi | based on the comment in https://github.com/eventlet/eventlet/blob/88ec603404b2ed25c610dead75d4693c7b3e8072/eventlet/green/select.py#L62-L69 | |
| 09:21:38 | gibi | one could argue that what we see is an eventlet bug as select.select with timeout=0.0 should not ever block but it does block in our case. | |
| 09:28:48 | opendevreview | suzhengwei proposed openstack/nova master: rename 'recreate' to 'evacuate' https://review.opendev.org/c/openstack/nova/+/883810 | |
| 10:24:19 | ykarel | Thanks gibi for checking, anyway the issue can be fixed/worked around on nova side? | |
| 10:25:38 | gibi | I'm trying to open an issue on eventlet and see if the maintainer agrees with my analysis or not. I don't see now any easy workaround. Maybe sean-k-mooney or melwitt can see some | |
| 10:25:59 | gibi | ykarel: I will update the launchpad bug | |
| 10:26:05 | ykarel | Thanks gibi | |
| 10:28:04 | sean-k-mooney | gibi: sorry i missed the start of this what is the issue | |
| 10:29:47 | gibi | nova-api using neutron client to call neutron API but get stuck for ever | |
| 10:30:12 | gibi | based on the stack trace it stuck checking if the previous connection is still usable | |
| 10:30:34 | gibi | we end up in eventlet monkeypatched select.select on a socket | |
| 10:30:40 | gibi | with a timeout 0.0 | |
| 10:31:09 | gibi | based on the stdlib doc timeout 0.0 means non blocking but we still block | |
| 10:31:21 | gibi | so I assume eventlet not properly handles timeout 0.0 in the eventlet select impl | |
| 10:31:22 | sean-k-mooney | i see | |
| 10:31:34 | sean-k-mooney | that or its python version specirif | |
| 10:31:46 | sean-k-mooney | but ya sound like a api compaitblity bug | |
| 10:31:54 | gibi | details are here https://bugs.launchpad.net/neutron/+bug/2015065 | |
| 10:36:40 | sean-k-mooney | Changed in version 3.7: The method no longer toggles SOCK_NONBLOCK flag on socket.type. | |
| 10:36:46 | sean-k-mooney | https://docs.python.org/3/library/socket.html#socket.socket.settimeout | |
| 10:37:28 | sean-k-mooney | ykarel: gibi: it looks like we shoudl not be using 0.0 to make it non-blocking after 3.7 | |
| 10:38:00 | sean-k-mooney | we should be using socket.setblocking(false) | |
| 10:38:03 | gibi | sean-k-mooney: https://docs.python.org/3.10/library/select.html#select.select for select.select timeout=0.0 still means | |
| 10:38:06 | gibi | non blocking | |
| 10:38:45 | gibi | so it is not the socket that is set to non blocking mode, it is select called with non blocking timeout | |
| 10:40:17 | sean-k-mooney | i feel like this is still a logic but on our part | |
| 10:40:38 | sean-k-mooney | we likely shoudl be settign timeout to a non zeor value and setting the socket to non-blocking | |
| 10:40:49 | gibi | this is urllib3, not our code | |
| 10:42:38 | sean-k-mooney | fun | |
| 10:44:00 | sean-k-mooney | gibi: as far as i can tell eventlet supprot for 3.10 is still not fully complete | |
| 10:53:38 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: [wip]network: convert usage of neutronclient to openstacksdk https://review.opendev.org/c/openstack/nova/+/882714 | |
| 10:53:42 | sean-k-mooney | gibi: so it does look like its alwasy a blockign operation https://github.com/eventlet/eventlet/blob/master/eventlet/green/select.py#L30-L36 | |
| 10:54:17 | gibi | filed https://github.com/eventlet/eventlet/issues/798 | |
| 10:55:05 | gibi | sean-k-mooney: as far as I understand the eventlet code they do block but set up a timer to wake up after the given timeout | |
| 10:55:44 | sean-k-mooney | if timeout is not None: | |
| 10:55:45 | gibi | so I think that can race and the code can miss the weakup as it is not reached hub.switch() yet | |
| 10:55:46 | sean-k-mooney | timers.append(hub.schedule_call_global(timeout, on_timeout)) | |
| 10:55:48 | sean-k-mooney | so yes | |
| 10:56:02 | sean-k-mooney | but they proably need to instead spwan this in a seperate greenthread | |
| 10:56:51 | sean-k-mooney | although based on the assert | |
| 10:56:56 | sean-k-mooney | https://github.com/eventlet/eventlet/blob/master/eventlet/green/select.py#L40 | |
| 10:57:17 | sean-k-mooney | they are expecting you to be spanwign this in a thread pool or similar i think | |
| 10:57:50 | sean-k-mooney | not that we have contol over this really | |
| 10:58:04 | gibi | yeah | |
| 10:58:10 | sean-k-mooney | i guess we will see what they say | |
| 10:58:27 | gibi | also I don't think I fully understand the logic of the double timer in https://github.com/eventlet/eventlet/blob/88ec603404b2ed25c610dead75d4693c7b3e8072/eventlet/green/select.py#L59-L72 | |
| 10:58:27 | sean-k-mooney | for what its worth this is happening in tempest right | |
| 10:58:34 | gibi | but it feel scarry | |
| 10:58:50 | sean-k-mooney | im surpised we are using eventlet in tempest | |
| 10:59:41 | gibi | we need eventlet in nova-api for the scatter-gather, do we? | |
| 10:59:47 | gibi | don't we? | |
| 11:00:19 | sean-k-mooney | oh sorry i tought this was the tempest client that was blocking | |
| 11:00:33 | sean-k-mooney | not in nova | |
| 11:00:55 | gibi | it is nova-api using neutron client to call neutron API | |
| 11:01:04 | gibi | then nova-api blocks | |
| 11:01:05 | sean-k-mooney | i see | |
| 11:01:14 | gibi | and therefore tempest call to nova-api timeouts too | |
| 11:01:25 | sean-k-mooney | yep yep yep | |
| 11:01:28 | gibi | OK | |
| 11:01:28 | sean-k-mooney | following now | |
| 11:01:39 | sean-k-mooney | ya thats not good | |
| 11:01:42 | gibi | I will stop here now and move over to golang for another set of challenges :) | |
| 11:01:57 | gibi | let's see if eventlet maintainers has some ideas | |
| 11:02:25 | gibi | another way would be to ask urllib3 maintainers to change their side | |
| 11:02:44 | sean-k-mooney | well eventlet is monkeytpatching urllib3 | |
| 11:03:03 | sean-k-mooney | https://github.com/eventlet/eventlet/blob/master/eventlet/green/urllib/request.py | |
| 11:03:07 | gibi | (or we can rip out eventlet from nova and move to pure threads in the next coupe of years :)) | |
| 11:03:42 | sean-k-mooney | well nova-api didnt have any direct depency on eventlet until we added scater gateher | |
| 11:03:55 | sean-k-mooney | so if it was just that we could rip that out | |
| 11:04:15 | sean-k-mooney | but this could obvioulys happen for all teh other client invocations | |
| 11:04:20 | sean-k-mooney | in the other services so ya... | |
| 11:05:08 | gibi | hm, that seems like only monkey patching urllib but not urllib3 | |
| 11:05:44 | sean-k-mooney | maybe i didn't look too closely i assumed it was the same | |
| 11:05:48 | gibi | it seem for urllib3 the socket and select monkey considered enough | |
| 11:07:05 | opendevreview | Merged openstack/os-vif stable/wallaby: Use TCP keepalives for ovsdb connections https://review.opendev.org/c/openstack/os-vif/+/841773 | |
| 11:07:07 | opendevreview | Merged openstack/os-vif stable/wallaby: only register tables used by os-vif https://review.opendev.org/c/openstack/os-vif/+/841774 | |
| 11:08:50 | sean-k-mooney | oh urllib is the standard libary | |
| 11:09:06 | sean-k-mooney | https://docs.python.org/3/library/urllib.html?highlight=urllib#module-urllib | |
| 11:32:46 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284 | |
| 11:32:46 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Reproducer for dangling volumes https://review.opendev.org/c/openstack/nova/+/881457 | |
| 13:37:29 | opendevreview | Dan Smith proposed openstack/nova master: Add compute_id columns to instances, migrations https://review.opendev.org/c/openstack/nova/+/879499 | |
| 13:37:29 | opendevreview | Dan Smith proposed openstack/nova master: Populate ComputeNode.service_id https://review.opendev.org/c/openstack/nova/+/879904 | |
| 13:37:30 | opendevreview | Dan Smith proposed openstack/nova master: Add compute_id to Instance object https://review.opendev.org/c/openstack/nova/+/879500 | |
| 13:37:30 | opendevreview | Dan Smith proposed openstack/nova master: Add dest_compute_id to Migration object https://review.opendev.org/c/openstack/nova/+/879682 | |
| 13:37:31 | opendevreview | Dan Smith proposed openstack/nova master: Online migrate missing Instance.compute_id fields https://review.opendev.org/c/openstack/nova/+/879905 | |
| 15:39:39 | opendevreview | Sylvain Bauza proposed openstack/nova stable/zed: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883723 | |
| 15:40:02 | opendevreview | Sylvain Bauza proposed openstack/nova stable/yoga: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883724 | |
| 15:41:08 | opendevreview | Sylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883725 | |
| 15:42:29 | opendevreview | Sylvain Bauza proposed openstack/nova stable/wallaby: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883726 | |
| 15:50:58 | sean-k-mooney | o/ | |
| 15:53:33 | sean-k-mooney | melwitt: bauzas dansmith could ye take a look at https://review.opendev.org/c/openstack/nova/+/853269/1 and the follow ups | |
| 15:54:11 | melwitt | sure | |