| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-04 | |||
| 13:18:27 | sean-k-mooney | tiem outs where | |
| 13:18:35 | sean-k-mooney | i dont see them in the zuul jobs | |
| 13:18:39 | sean-k-mooney | you mean in a test | |
| 13:19:31 | bauzas | yes, see my comments | |
| 13:19:37 | bauzas | you'll see the stacktrace | |
| 13:19:54 | bauzas | I haven't looked yet at Neutron APIs | |
| 13:20:07 | sean-k-mooney | urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='173.231.255.102', port=80): Read timed out. (read timeout=60) | |
| 13:20:28 | sean-k-mooney | that could be related to memory pressure or a slow node | |
| 13:21:06 | bauzas | yes, or no workers available | |
| 13:21:35 | sean-k-mooney | looking at the memory tracker the low point was 1342048 | |
| 13:21:58 | sean-k-mooney | so i dont think that is the issue 130MB is not much but its not terible for these jobs | |
| 13:22:09 | bauzas | read timeouts can happen on a synchronous call if the call takes more than 60 secs to return | |
| 13:22:28 | bauzas | so this could be something else but just a memory issue | |
| 13:22:34 | sean-k-mooney | ya im wondering if the neutron server was deadlocked on something | |
| 13:22:35 | bauzas | like a DB lock | |
| 13:23:17 | sean-k-mooney | ya i know they had issue with db deadlocks last year | |
| 13:24:56 | sean-k-mooney | im seeing quite a few trace backs | |
| 13:25:08 | sean-k-mooney | realed to security gorups that apprently dont exist | |
| 13:27:14 | sean-k-mooney | https://paste.opendev.org/show/bxCPO7Jo1jSzEoRNwHKS/ | |
| 13:30:48 | sean-k-mooney | this seams to be the relevent logs https://paste.opendev.org/show/bMFDSLaKg5TZVpRa3OXD/ | |
| 13:31:44 | sean-k-mooney | there are tracbacks before and after that | |
| 13:32:28 | sean-k-mooney | bauzas: if you care its roughly here https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d/log/controller/logs/screen-q-svc.txt#53526 | |
| 13:33:45 | bauzas | I'll check whether there is an upstream gate report on it | |
| 13:33:50 | bauzas | bug report* | |
| 13:34:20 | sean-k-mooney | neutron does seam to be respondign to the get request May 04 12:02:50.886890 np0033942429 neutron-server[162430]: INFO neutron.wsgi [req-49c6f58c-b29d-4c83-8ad1-5a588383533f req-bc14944a-4c80-4107-9ce5-de32f17088f8 tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] 173.231.255.102 "GET | |
| 13:34:22 | sean-k-mooney | /v2.0/security-groups/b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76 HTTP/1.1" status: 200 len: 1820 time: 0.1247852 | |
| 13:35:43 | sean-k-mooney | but i dont knwo when nova actully made it | |
| 13:36:06 | sean-k-mooney | it could have been sitting in apache waiting for other requests for a while | |
| 13:37:07 | opendevreview | Merged openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 13:38:13 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2023.1: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882293 | |
| 13:41:51 | bauzas | lajoskatona: slaweq: ralonsoh: not sure you already know but it seems we have some Neutron API deadlock in the gate ^ | |
| 13:42:21 | bauzas | and I wasn't able to find an exiting neutron bug report that relates to it | |
| 13:44:57 | ralonsoh | bauzas, but I'm not sure about this method, I don't see in the "list_subnets" method where the neutron client is returning the segment_id | |
| 13:45:08 | ralonsoh | I see that in "show_subnet" | |
| 13:45:28 | bauzas | ralonsoh: ah sorry, you looked at the wrong thing due to my lack of explanation :) | |
| 13:46:16 | bauzas | ralonsoh: see rather the discussion we had between sean-k-mooney and I, the context is https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d | |
| 13:46:36 | ralonsoh | yes, we have the same problem | |
| 13:46:43 | ralonsoh | you discussed that with lajoskatona | |
| 13:46:49 | bauzas | and https://zuul.opendev.org/t/openstack/build/de4b1d075b0745aab2bdec9bc8319877 | |
| 13:47:20 | bauzas | then I forgot about it | |
| 13:48:24 | ralonsoh | bauzas, if recall correctly (maybe I'm wrong), the issue with create_security_group_rule was that we where calling the nova client | |
| 13:52:49 | ralonsoh | sean-k-mooney, I'm checking https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e0/882052/1/gate/grenade-skip-level-always/6e0e087/testr_results.html | |
| 13:53:00 | ralonsoh | and I don't see where Neutron is failing | |
| 13:55:17 | bauzas | ralonsoh: I can be wrong but in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e0/882052/1/gate/grenade-skip-level-always/6e0e087/testr_results.html we have test_security_group_rules_create[id-850795d7-d4d3-4e55-b527-a774c0123d3a,smoke] failing on File "/opt/stack/new/tempest/tempest/lib/services/compute/security_group_rules_client.py", line 35, in create_security_group_rule | |
| 13:55:47 | bauzas | this is an API call to the os-security-groups API which belongs to Nova | |
| 13:56:01 | bauzas | but this actually is a proxy call to Neutron | |
| 13:56:36 | ralonsoh | yes and I'm reviewing the logs and I can't find any error | |
| 13:56:48 | ralonsoh | I'm checking the same call in other tempest runs | |
| 14:00:19 | bauzas | that's unfortunate that given the timeout, we don't have the request-id | |
| 14:03:25 | ralonsoh | bauzas, but what call is timing out? because is not the SG creation | |
| 14:06:20 | bauzas | ralonsoh: this is failing on https://github.com/openstack/tempest/blob/master/tempest/api/compute/security_groups/test_security_group_rules.py#L67 | |
| 14:06:53 | bauzas | I think I can trace the nova-api call with May 04 12:02:50.750575 np0033942429 devstack@n-api.service[165740]: DEBUG nova.api.openstack.wsgi [None req-49c6f58c-b29d-4c83-8ad1-5a588383533f tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] Action: 'create', calling method: <function Controller.__getattribute__.<locals>.version_select at 0x7f7d37c793f0>, body: {"security_group_ | |
| 14:06:53 | bauzas | rule": {"parent_group_id": "b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76", "ip_protocol": "tcp", "from_port": 22, "to_port": 22}} {{(pid=165740) _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:511}} | |
| 14:08:01 | bauzas | so now I'm trying to find logs from req-49c6f58c-b29d-4c83-8ad1-5a588383533f | |
| 14:10:17 | bauzas | look, I can see the req-id in neutron May 04 12:02:50.886890 np0033942429 neutron-server[162430]: INFO neutron.wsgi [req-49c6f58c-b29d-4c83-8ad1-5a588383533f req-bc14944a-4c80-4107-9ce5-de32f17088f8 tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] 173.231.255.102 "GET /v2.0/security-groups/b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76 HTTP/1.1" status: 200 len: 1820 time: 0.1247852 | |
| 14:12:00 | ralonsoh | but this is a get call, not the SG rule creation | |
| 14:12:48 | ralonsoh | well, actually this is the req-id of the SG rule creation | |
| 14:12:48 | lajoskatona | bauzas, ralonsoh: Hi, this is related to https://bugs.launchpad.net/neutron/+bug/2015065 as I see (sorry perhaps you already linked the lp link) | |
| 14:13:31 | bauzas | lajoskatona: excellent, will mention it then | |
| 14:13:56 | bauzas | lajoskatona: fwiw, this is not only on neutron-ovs-grenade-dvr-multinode but also on our jobs too :) | |
| 14:15:26 | lajoskatona | bauzas: yeah, with rodolfo we added later that as comment with opensearch links and examples also from tempest | |
| 14:15:36 | bauzas | cool ++ | |
| 14:16:56 | ralonsoh | bauzas, this is using neutronclient, right? | |
| 14:17:09 | bauzas | good question | |
| 14:17:20 | bauzas | afaik yes, but recently we wanted to use the sdk | |
| 14:17:38 | bauzas | I don't think we merged any change yet, but lemme doublecheck | |
| 14:17:41 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882313 | |
| 14:17:42 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882314 | |
| 14:18:01 | ralonsoh | I don't see any change in neutronclient nor nova | |
| 14:18:29 | opendevreview | Artom Lifshitz proposed openstack/nova stable/zed: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882315 | |
| 14:18:30 | opendevreview | Artom Lifshitz proposed openstack/nova stable/zed: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882316 | |
| 14:18:58 | opendevreview | Artom Lifshitz proposed openstack/nova stable/yoga: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882317 | |
| 14:18:59 | opendevreview | Artom Lifshitz proposed openstack/nova stable/yoga: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882318 | |
| 14:19:24 | bauzas | ralonsoh: you're correct, still neutronclient https://github.com/openstack/nova/blob/master/nova/network/security_group_api.py#L379 | |
| 14:19:37 | ralonsoh | yes | |
| 14:19:54 | bauzas | anyway, I need to go get my daughter from school, bbiab (~15 mins) | |
| 14:20:36 | opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882319 | |
| 14:20:37 | opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882320 | |
| 14:21:07 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882321 | |
| 14:21:08 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882322 | |
| 14:42:14 | dvo-plv_ | gibi: we have finished with os-traits patch. Could you please propose the release | |
| 14:56:15 | bauzas | dvo-plv_: I can do it | |
| 14:58:50 | dvo-plv_ | great, thank you | |
| 15:01:08 | gibi | bauzas: if you have cycles right now then thanks for proposing it | |
| 15:01:25 | bauzas | just doing it now | |
| 15:02:39 | gibi | bauzas++ | |
| 15:09:30 | bauzas | Uggla: your approval is nice for https://review.opendev.org/c/openstack/releases/+/882325 | |
| 15:09:36 | bauzas | gibi: dvo-plv_: ^ | |
| 15:10:02 | Uggla | bauzas, I will have a look | |
| 15:11:44 | sean-k-mooney | dvo-plv_: when that is released plese ensure you bump the min requirement for os-traits in the nova patch that uses it | |
| 15:17:07 | bauzas | sean-k-mooney: he'll need to wait for the upper-constraints bot patch to be generated first if I'm not wrong | |
| 15:17:43 | sean-k-mooney | for it to pass ci | |
| 15:17:49 | sean-k-mooney | but they can do the bump | |
| 15:22:16 | bauzas | correct, but dvo-plv_'s main concern was that zuul wasn't happy with its patch, hence his request to super-fast-approve traits | |
| 15:22:36 | bauzas | so I'm just explaining that the release is only half of the definition of done | |
| 15:25:27 | sean-k-mooney | dvo-plv_: for what its worth this should all be resolved early next week | |
| 15:25:58 | sean-k-mooney | i would expect the release to hapeen todya or tomorrow and the uper constraits bump should land shortly after | |
| 15:27:29 | bauzas | well, there are humans behind the releases approval and the upper-constraints patch approval too, so I'd give them a few more days | |
| 15:27:45 | bauzas | but yeah, early next week seems a reasonable ETA | |
| 15:32:56 | gibi | I can cosy up to elodilles to get attention on https://review.opendev.org/c/openstack/releases/+/882325 :) | |