| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-03 | |||
| 23:25:18 | melwitt | I saw one yesterday | |
| 23:58:33 | dansmith | maybe sean-k-mooney could take a look in the morning | |
| #openstack-nova - 2023-05-04 | |||
| 01:29:08 | opendevreview | Merged openstack/nova master: add hypervisor version weigher https://review.opendev.org/c/openstack/nova/+/880231 | |
| 08:06:32 | dvo-plv_ | gibi, sean-k-monney: Hello, could you pelase verify os-trait patch: https://review.opendev.org/c/openstack/os-traits/+/876069 to unblock zuul verification for nova patch, cause it fails according to the deendencies | |
| 08:07:12 | dvo-plv_ | sean-k-mooney: sorry, I have made a mistake in your nick | |
| 08:23:19 | gibi | dvo-plv_: the os-traits patch looks good to me. After that merges, we need to propose a new os-traits release, so that you can depend on that release of os-traits in your nova patch | |
| 08:23:24 | bauzas | dang, it took me a while to figure out we were mocking the default policies by a fixture... | |
| 08:23:56 | bauzas | context : https://39770df410ace4902fc0-3eec3a080da8356877afe7c3a8a6cc53.ssl.cf5.rackcdn.com/881562/1/check/openstack-tox-py39/c658228/testr_results.html | |
| 08:25:15 | bauzas | now, I know, we also have a fake policy data : https://github.com/openstack/nova/blob/master/nova/tests/unit/fake_policy.py#L16 | |
| 08:25:19 | bauzas | TIL. | |
| 08:34:40 | dvo-plv_ | gibi: I have only update requirements.txt for nova, or there is some additional doc files what should be updated too | |
| 08:48:12 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a new policy for cold-migrate with host https://review.opendev.org/c/openstack/nova/+/881562 | |
| 09:00:18 | sahid | o/ | |
| 09:09:53 | gibi | dvo-plv_: you only need to updat the requirements.txt on the nova side. | |
| 09:11:03 | bauzas | gibi: but before doing it, we need to have an upper-constraints change, right ? :) | |
| 09:11:18 | gibi | bauzas: right, I above mentioned to dvo-plv_ that we need an os-traits release | |
| 09:11:35 | gibi | but then there will be either an automatic upper constraints bump | |
| 09:11:44 | gibi | or do a manual one in the requirements repo | |
| 09:11:54 | gibi | then the requirements.txt on the nova side can be bumped | |
| 09:12:06 | gibi | in the meantime we should not forget to bump the min version in placement too | |
| 09:13:24 | bauzas | dvo-plv_: fwiw, I +Wd https://review.opendev.org/c/openstack/os-traits/+/876069 with a comment | |
| 09:22:00 | opendevreview | Merged openstack/os-traits master: Add 'COMPUTE_NET_VIRTIO_PACKED' https://review.opendev.org/c/openstack/os-traits/+/876069 | |
| 09:32:27 | dvo-plv_ | bauzas: Could I update patch with comment, ot it should be done in some another way. because it alredy merged? | |
| 09:32:45 | bauzas | dvo-plv_: just add another patch as a follow-up :) | |
| 09:34:20 | dvo-plv_ | okay, sure | |
| 09:35:23 | kashyap | Can anyone remind me: for picking the min libvirt/QEMU versions what OSes do we care? I'm guessing the "usual": Debian, Ubuntu, Fedora, and RHEL | |
| 09:36:06 | kashyap | In the past we've also considered openSUSE / SLES. Should they be considered too? | |
| 09:36:26 | dvo-plv_ | gibi: Does os-trait release has some estimates? I would like to plan our next activities according to them | |
| 09:37:36 | gibi | dvo-plv_: if you ping me after the os-trait patch lands I can propose the release quickly and it should not take more than couple day to get that release proposal landed | |
| 09:49:51 | kashyap | bauzas: Have we got a name after Bobcat? | |
| 09:54:29 | bauzas | kashyap: nope, afaik not yet | |
| 09:54:42 | opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 10:10:22 | dvo-plv_ | bauzas: I added comment ot the trait: https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 10:27:26 | opendevreview | Merged openstack/nova master: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/862964 | |
| 10:27:34 | opendevreview | Merged openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | |
| 10:31:27 | opendevreview | Merged openstack/nova master: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882160 | |
| 11:15:51 | sean-k-mooney | bauzas: i wont get to it for a week or two but im going to work on a backlog spec for AZ enhancements. i feel liek ther eare enough rough edges and pain point that collecting them in one docs would make sense but while i thik i know how to adrss some of them i dont know how to adress all fo them so im hoping that if we aggreated the pain point some cominalities will emerge | |
| 11:15:53 | sean-k-mooney | that can then be adress in the C cycle after some tought | |
| 11:20:44 | opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 12:45:07 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Selete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | |
| 13:00:10 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Delete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | |
| 13:17:16 | bauzas | tell me I'm wrong but os-port-interfaces and os-security-groups API calls proxy to Neutron, right? | |
| 13:17:30 | sean-k-mooney | yes | |
| 13:17:31 | bauzas | if so, looks like we have a performance regression in the gate https://review.opendev.org/c/openstack/nova/+/882052 | |
| 13:17:47 | bauzas | I got two timeouts | |
| 13:18:27 | sean-k-mooney | tiem outs where | |
| 13:18:35 | sean-k-mooney | i dont see them in the zuul jobs | |
| 13:18:39 | sean-k-mooney | you mean in a test | |
| 13:19:31 | bauzas | yes, see my comments | |
| 13:19:37 | bauzas | you'll see the stacktrace | |
| 13:19:54 | bauzas | I haven't looked yet at Neutron APIs | |
| 13:20:07 | sean-k-mooney | urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='173.231.255.102', port=80): Read timed out. (read timeout=60) | |
| 13:20:28 | sean-k-mooney | that could be related to memory pressure or a slow node | |
| 13:21:06 | bauzas | yes, or no workers available | |
| 13:21:35 | sean-k-mooney | looking at the memory tracker the low point was 1342048 | |
| 13:21:58 | sean-k-mooney | so i dont think that is the issue 130MB is not much but its not terible for these jobs | |
| 13:22:09 | bauzas | read timeouts can happen on a synchronous call if the call takes more than 60 secs to return | |
| 13:22:28 | bauzas | so this could be something else but just a memory issue | |
| 13:22:34 | sean-k-mooney | ya im wondering if the neutron server was deadlocked on something | |
| 13:22:35 | bauzas | like a DB lock | |
| 13:23:17 | sean-k-mooney | ya i know they had issue with db deadlocks last year | |
| 13:24:56 | sean-k-mooney | im seeing quite a few trace backs | |
| 13:25:08 | sean-k-mooney | realed to security gorups that apprently dont exist | |
| 13:27:14 | sean-k-mooney | https://paste.opendev.org/show/bxCPO7Jo1jSzEoRNwHKS/ | |
| 13:30:48 | sean-k-mooney | this seams to be the relevent logs https://paste.opendev.org/show/bMFDSLaKg5TZVpRa3OXD/ | |
| 13:31:44 | sean-k-mooney | there are tracbacks before and after that | |
| 13:32:28 | sean-k-mooney | bauzas: if you care its roughly here https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d/log/controller/logs/screen-q-svc.txt#53526 | |
| 13:33:45 | bauzas | I'll check whether there is an upstream gate report on it | |
| 13:33:50 | bauzas | bug report* | |
| 13:34:20 | sean-k-mooney | neutron does seam to be respondign to the get request May 04 12:02:50.886890 np0033942429 neutron-server[162430]: INFO neutron.wsgi [req-49c6f58c-b29d-4c83-8ad1-5a588383533f req-bc14944a-4c80-4107-9ce5-de32f17088f8 tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] 173.231.255.102 "GET | |
| 13:34:22 | sean-k-mooney | /v2.0/security-groups/b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76 HTTP/1.1" status: 200 len: 1820 time: 0.1247852 | |
| 13:35:43 | sean-k-mooney | but i dont knwo when nova actully made it | |
| 13:36:06 | sean-k-mooney | it could have been sitting in apache waiting for other requests for a while | |
| 13:37:07 | opendevreview | Merged openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 13:38:13 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2023.1: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882293 | |
| 13:41:51 | bauzas | lajoskatona: slaweq: ralonsoh: not sure you already know but it seems we have some Neutron API deadlock in the gate ^ | |
| 13:42:21 | bauzas | and I wasn't able to find an exiting neutron bug report that relates to it | |
| 13:44:57 | ralonsoh | bauzas, but I'm not sure about this method, I don't see in the "list_subnets" method where the neutron client is returning the segment_id | |
| 13:45:08 | ralonsoh | I see that in "show_subnet" | |
| 13:45:28 | bauzas | ralonsoh: ah sorry, you looked at the wrong thing due to my lack of explanation :) | |
| 13:46:16 | bauzas | ralonsoh: see rather the discussion we had between sean-k-mooney and I, the context is https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d | |
| 13:46:36 | ralonsoh | yes, we have the same problem | |
| 13:46:43 | ralonsoh | you discussed that with lajoskatona | |
| 13:46:49 | bauzas | and https://zuul.opendev.org/t/openstack/build/de4b1d075b0745aab2bdec9bc8319877 | |
| 13:47:20 | bauzas | then I forgot about it | |
| 13:48:24 | ralonsoh | bauzas, if recall correctly (maybe I'm wrong), the issue with create_security_group_rule was that we where calling the nova client | |
| 13:52:49 | ralonsoh | sean-k-mooney, I'm checking https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e0/882052/1/gate/grenade-skip-level-always/6e0e087/testr_results.html | |
| 13:53:00 | ralonsoh | and I don't see where Neutron is failing | |
| 13:55:17 | bauzas | ralonsoh: I can be wrong but in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e0/882052/1/gate/grenade-skip-level-always/6e0e087/testr_results.html we have test_security_group_rules_create[id-850795d7-d4d3-4e55-b527-a774c0123d3a,smoke] failing on File "/opt/stack/new/tempest/tempest/lib/services/compute/security_group_rules_client.py", line 35, in create_security_group_rule | |
| 13:55:47 | bauzas | this is an API call to the os-security-groups API which belongs to Nova | |
| 13:56:01 | bauzas | but this actually is a proxy call to Neutron | |
| 13:56:36 | ralonsoh | yes and I'm reviewing the logs and I can't find any error | |
| 13:56:48 | ralonsoh | I'm checking the same call in other tempest runs | |
| 14:00:19 | bauzas | that's unfortunate that given the timeout, we don't have the request-id | |
| 14:03:25 | ralonsoh | bauzas, but what call is timing out? because is not the SG creation | |
| 14:06:20 | bauzas | ralonsoh: this is failing on https://github.com/openstack/tempest/blob/master/tempest/api/compute/security_groups/test_security_group_rules.py#L67 | |
| 14:06:53 | bauzas | I think I can trace the nova-api call with May 04 12:02:50.750575 np0033942429 devstack@n-api.service[165740]: DEBUG nova.api.openstack.wsgi [None req-49c6f58c-b29d-4c83-8ad1-5a588383533f tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] Action: 'create', calling method: <function Controller.__getattribute__.<locals>.version_select at 0x7f7d37c793f0>, body: {"security_group_ | |
| 14:06:53 | bauzas | rule": {"parent_group_id": "b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76", "ip_protocol": "tcp", "from_port": 22, "to_port": 22}} {{(pid=165740) _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:511}} | |
| 14:08:01 | bauzas | so now I'm trying to find logs from req-49c6f58c-b29d-4c83-8ad1-5a588383533f | |
| 14:10:17 | bauzas | look, I can see the req-id in neutron May 04 12:02:50.886890 np0033942429 neutron-server[162430]: INFO neutron.wsgi [req-49c6f58c-b29d-4c83-8ad1-5a588383533f req-bc14944a-4c80-4107-9ce5-de32f17088f8 tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] 173.231.255.102 "GET /v2.0/security-groups/b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76 HTTP/1.1" status: 200 len: 1820 time: 0.1247852 | |