| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-22 | |||
| 10:24:19 | ykarel | Thanks gibi for checking, anyway the issue can be fixed/worked around on nova side? | |
| 10:25:38 | gibi | I'm trying to open an issue on eventlet and see if the maintainer agrees with my analysis or not. I don't see now any easy workaround. Maybe sean-k-mooney or melwitt can see some | |
| 10:25:59 | gibi | ykarel: I will update the launchpad bug | |
| 10:26:05 | ykarel | Thanks gibi | |
| 10:28:04 | sean-k-mooney | gibi: sorry i missed the start of this what is the issue | |
| 10:29:47 | gibi | nova-api using neutron client to call neutron API but get stuck for ever | |
| 10:30:12 | gibi | based on the stack trace it stuck checking if the previous connection is still usable | |
| 10:30:34 | gibi | we end up in eventlet monkeypatched select.select on a socket | |
| 10:30:40 | gibi | with a timeout 0.0 | |
| 10:31:09 | gibi | based on the stdlib doc timeout 0.0 means non blocking but we still block | |
| 10:31:21 | gibi | so I assume eventlet not properly handles timeout 0.0 in the eventlet select impl | |
| 10:31:22 | sean-k-mooney | i see | |
| 10:31:34 | sean-k-mooney | that or its python version specirif | |
| 10:31:46 | sean-k-mooney | but ya sound like a api compaitblity bug | |
| 10:31:54 | gibi | details are here https://bugs.launchpad.net/neutron/+bug/2015065 | |
| 10:36:40 | sean-k-mooney | Changed in version 3.7: The method no longer toggles SOCK_NONBLOCK flag on socket.type. | |
| 10:36:46 | sean-k-mooney | https://docs.python.org/3/library/socket.html#socket.socket.settimeout | |
| 10:37:28 | sean-k-mooney | ykarel: gibi: it looks like we shoudl not be using 0.0 to make it non-blocking after 3.7 | |
| 10:38:00 | sean-k-mooney | we should be using socket.setblocking(false) | |
| 10:38:03 | gibi | sean-k-mooney: https://docs.python.org/3.10/library/select.html#select.select for select.select timeout=0.0 still means | |
| 10:38:06 | gibi | non blocking | |
| 10:38:45 | gibi | so it is not the socket that is set to non blocking mode, it is select called with non blocking timeout | |
| 10:40:17 | sean-k-mooney | i feel like this is still a logic but on our part | |
| 10:40:38 | sean-k-mooney | we likely shoudl be settign timeout to a non zeor value and setting the socket to non-blocking | |
| 10:40:49 | gibi | this is urllib3, not our code | |
| 10:42:38 | sean-k-mooney | fun | |
| 10:44:00 | sean-k-mooney | gibi: as far as i can tell eventlet supprot for 3.10 is still not fully complete | |
| 10:53:38 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: [wip]network: convert usage of neutronclient to openstacksdk https://review.opendev.org/c/openstack/nova/+/882714 | |
| 10:53:42 | sean-k-mooney | gibi: so it does look like its alwasy a blockign operation https://github.com/eventlet/eventlet/blob/master/eventlet/green/select.py#L30-L36 | |
| 10:54:17 | gibi | filed https://github.com/eventlet/eventlet/issues/798 | |
| 10:55:05 | gibi | sean-k-mooney: as far as I understand the eventlet code they do block but set up a timer to wake up after the given timeout | |
| 10:55:44 | sean-k-mooney | if timeout is not None: | |
| 10:55:45 | gibi | so I think that can race and the code can miss the weakup as it is not reached hub.switch() yet | |
| 10:55:46 | sean-k-mooney | timers.append(hub.schedule_call_global(timeout, on_timeout)) | |
| 10:55:48 | sean-k-mooney | so yes | |
| 10:56:02 | sean-k-mooney | but they proably need to instead spwan this in a seperate greenthread | |
| 10:56:51 | sean-k-mooney | although based on the assert | |
| 10:56:56 | sean-k-mooney | https://github.com/eventlet/eventlet/blob/master/eventlet/green/select.py#L40 | |
| 10:57:17 | sean-k-mooney | they are expecting you to be spanwign this in a thread pool or similar i think | |
| 10:57:50 | sean-k-mooney | not that we have contol over this really | |
| 10:58:04 | gibi | yeah | |
| 10:58:10 | sean-k-mooney | i guess we will see what they say | |
| 10:58:27 | gibi | also I don't think I fully understand the logic of the double timer in https://github.com/eventlet/eventlet/blob/88ec603404b2ed25c610dead75d4693c7b3e8072/eventlet/green/select.py#L59-L72 | |
| 10:58:27 | sean-k-mooney | for what its worth this is happening in tempest right | |
| 10:58:34 | gibi | but it feel scarry | |
| 10:58:50 | sean-k-mooney | im surpised we are using eventlet in tempest | |
| 10:59:41 | gibi | we need eventlet in nova-api for the scatter-gather, do we? | |
| 10:59:47 | gibi | don't we? | |
| 11:00:19 | sean-k-mooney | oh sorry i tought this was the tempest client that was blocking | |
| 11:00:33 | sean-k-mooney | not in nova | |
| 11:00:55 | gibi | it is nova-api using neutron client to call neutron API | |
| 11:01:04 | gibi | then nova-api blocks | |
| 11:01:05 | sean-k-mooney | i see | |
| 11:01:14 | gibi | and therefore tempest call to nova-api timeouts too | |
| 11:01:25 | sean-k-mooney | yep yep yep | |
| 11:01:28 | gibi | OK | |
| 11:01:28 | sean-k-mooney | following now | |
| 11:01:39 | sean-k-mooney | ya thats not good | |
| 11:01:42 | gibi | I will stop here now and move over to golang for another set of challenges :) | |
| 11:01:57 | gibi | let's see if eventlet maintainers has some ideas | |
| 11:02:25 | gibi | another way would be to ask urllib3 maintainers to change their side | |
| 11:02:44 | sean-k-mooney | well eventlet is monkeytpatching urllib3 | |
| 11:03:03 | sean-k-mooney | https://github.com/eventlet/eventlet/blob/master/eventlet/green/urllib/request.py | |
| 11:03:07 | gibi | (or we can rip out eventlet from nova and move to pure threads in the next coupe of years :)) | |
| 11:03:42 | sean-k-mooney | well nova-api didnt have any direct depency on eventlet until we added scater gateher | |
| 11:03:55 | sean-k-mooney | so if it was just that we could rip that out | |
| 11:04:15 | sean-k-mooney | but this could obvioulys happen for all teh other client invocations | |
| 11:04:20 | sean-k-mooney | in the other services so ya... | |
| 11:05:08 | gibi | hm, that seems like only monkey patching urllib but not urllib3 | |
| 11:05:44 | sean-k-mooney | maybe i didn't look too closely i assumed it was the same | |
| 11:05:48 | gibi | it seem for urllib3 the socket and select monkey considered enough | |
| 11:07:05 | opendevreview | Merged openstack/os-vif stable/wallaby: Use TCP keepalives for ovsdb connections https://review.opendev.org/c/openstack/os-vif/+/841773 | |
| 11:07:07 | opendevreview | Merged openstack/os-vif stable/wallaby: only register tables used by os-vif https://review.opendev.org/c/openstack/os-vif/+/841774 | |
| 11:08:50 | sean-k-mooney | oh urllib is the standard libary | |
| 11:09:06 | sean-k-mooney | https://docs.python.org/3/library/urllib.html?highlight=urllib#module-urllib | |
| 11:32:46 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284 | |
| 11:32:46 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Reproducer for dangling volumes https://review.opendev.org/c/openstack/nova/+/881457 | |
| 13:37:29 | opendevreview | Dan Smith proposed openstack/nova master: Add compute_id columns to instances, migrations https://review.opendev.org/c/openstack/nova/+/879499 | |
| 13:37:29 | opendevreview | Dan Smith proposed openstack/nova master: Populate ComputeNode.service_id https://review.opendev.org/c/openstack/nova/+/879904 | |
| 13:37:30 | opendevreview | Dan Smith proposed openstack/nova master: Add compute_id to Instance object https://review.opendev.org/c/openstack/nova/+/879500 | |
| 13:37:30 | opendevreview | Dan Smith proposed openstack/nova master: Add dest_compute_id to Migration object https://review.opendev.org/c/openstack/nova/+/879682 | |
| 13:37:31 | opendevreview | Dan Smith proposed openstack/nova master: Online migrate missing Instance.compute_id fields https://review.opendev.org/c/openstack/nova/+/879905 | |
| 15:39:39 | opendevreview | Sylvain Bauza proposed openstack/nova stable/zed: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883723 | |
| 15:40:02 | opendevreview | Sylvain Bauza proposed openstack/nova stable/yoga: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883724 | |
| 15:41:08 | opendevreview | Sylvain Bauza proposed openstack/nova stable/xena: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883725 | |
| 15:42:29 | opendevreview | Sylvain Bauza proposed openstack/nova stable/wallaby: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883726 | |
| 15:50:58 | sean-k-mooney | o/ | |
| 15:53:33 | sean-k-mooney | melwitt: bauzas dansmith could ye take a look at https://review.opendev.org/c/openstack/nova/+/853269/1 and the follow ups | |
| 15:54:11 | melwitt | sure | |
| 15:54:34 | bauzas | done | |
| 15:55:16 | sean-k-mooney | thanks | |
| 15:55:20 | melwitt | sweet | |
| 16:20:07 | dansmith | so if powerkvm is gone, why do we still have its CI commenting on all our patches? Looks to me like it can't even devstack anymore | |
| 16:20:28 | dansmith | maybe it's just a bot in a cloud somewhere that someone forgot to turn off? | |
| 16:23:38 | opendevreview | Merged openstack/nova stable/xena: Reproducer for bug 1983753 https://review.opendev.org/c/openstack/nova/+/853269 | |
| 16:30:24 | clarkb | dansmith: there should be a contact email in the account as well as on the wiki for the third party ci. Asking them to stop would be the first step and if they don't then we (a gerrit admin) can disable the account | |
| 16:33:29 | dansmith | okay, the "lightbits" CI also seems superfluous, but I think we've asked before and nobody knows what it is? | |
| 16:33:44 | sean-k-mooney | i do | |
| 16:33:56 | sean-k-mooney | its a ci that test tehre os-brick integration | |
| 16:34:34 | sean-k-mooney | its testing changes to https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/lightos.py | |