| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-03 | |||
| 18:47:17 | dansmith | I can do the removal and refactoring there, I just don't want to change so much, especially in a patch that was supposed to be a revert :) | |
| 18:48:21 | dansmith | I just want to get this landed so we can start getting more runs on the new stuff and not keep refactoring this, blocking that process | |
| 18:48:38 | dansmith | and since I have it working in this form I just hesitate to move too much at once | |
| 18:56:26 | gouthamr | ack; this makes sense dansmith.. | |
| 18:57:27 | dansmith | gouthamr: okay I just pushed that up let me know if that looks okay | |
| 18:59:53 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test new ceph job configuration with nova https://review.opendev.org/c/openstack/nova/+/881585 | |
| 19:03:23 | gouthamr | dansmith: a couple of comments | |
| 19:04:32 | dansmith | gouthamr: oh gate, right thanks :) | |
| 19:04:41 | dansmith | gouthamr: "gate" has seemed so far off for a long time :D | |
| 19:04:52 | dansmith | gouthamr: I don't know what is needed for rpm | |
| 19:05:07 | dansmith | gouthamr: but since these jobs don't run on there, I think we can defer that | |
| 19:06:46 | gouthamr | dansmith: "devstack-plugin-ceph-cephfs-nfs" is passing; although it doesn't do cephadm, it runs on centos-stream-9 | |
| 19:07:19 | dansmith | right, and those packages are only needed for cephadm | |
| 19:07:22 | gouthamr | dansmith: if we bump that to use cephadm soon, and we hit any issues, we'll get that covered :) | |
| 19:07:28 | dansmith | ack, cool | |
| 20:46:59 | opendevreview | sean mooney proposed openstack/os-vif master: [WIP] set default qos policy https://review.opendev.org/c/openstack/os-vif/+/881751 | |
| 22:04:03 | opendevreview | Dan Smith proposed openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | |
| 22:37:04 | dansmith | I have seen a lot of port binding related failures today, many of them on the grenade multinode job: https://c68ef44fab0ad498c042-f24a7834eba09db97966c05c7e428413.ssl.cf1.rackcdn.com/881585/8/check/nova-grenade-multinode/cbd5c81/testr_results.html | |
| 23:25:18 | melwitt | I saw one yesterday | |
| 23:58:33 | dansmith | maybe sean-k-mooney could take a look in the morning | |
| #openstack-nova - 2023-05-04 | |||
| 01:29:08 | opendevreview | Merged openstack/nova master: add hypervisor version weigher https://review.opendev.org/c/openstack/nova/+/880231 | |
| 08:06:32 | dvo-plv_ | gibi, sean-k-monney: Hello, could you pelase verify os-trait patch: https://review.opendev.org/c/openstack/os-traits/+/876069 to unblock zuul verification for nova patch, cause it fails according to the deendencies | |
| 08:07:12 | dvo-plv_ | sean-k-mooney: sorry, I have made a mistake in your nick | |
| 08:23:19 | gibi | dvo-plv_: the os-traits patch looks good to me. After that merges, we need to propose a new os-traits release, so that you can depend on that release of os-traits in your nova patch | |
| 08:23:24 | bauzas | dang, it took me a while to figure out we were mocking the default policies by a fixture... | |
| 08:23:56 | bauzas | context : https://39770df410ace4902fc0-3eec3a080da8356877afe7c3a8a6cc53.ssl.cf5.rackcdn.com/881562/1/check/openstack-tox-py39/c658228/testr_results.html | |
| 08:25:15 | bauzas | now, I know, we also have a fake policy data : https://github.com/openstack/nova/blob/master/nova/tests/unit/fake_policy.py#L16 | |
| 08:25:19 | bauzas | TIL. | |
| 08:34:40 | dvo-plv_ | gibi: I have only update requirements.txt for nova, or there is some additional doc files what should be updated too | |
| 08:48:12 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a new policy for cold-migrate with host https://review.opendev.org/c/openstack/nova/+/881562 | |
| 09:00:18 | sahid | o/ | |
| 09:09:53 | gibi | dvo-plv_: you only need to updat the requirements.txt on the nova side. | |
| 09:11:03 | bauzas | gibi: but before doing it, we need to have an upper-constraints change, right ? :) | |
| 09:11:18 | gibi | bauzas: right, I above mentioned to dvo-plv_ that we need an os-traits release | |
| 09:11:35 | gibi | but then there will be either an automatic upper constraints bump | |
| 09:11:44 | gibi | or do a manual one in the requirements repo | |
| 09:11:54 | gibi | then the requirements.txt on the nova side can be bumped | |
| 09:12:06 | gibi | in the meantime we should not forget to bump the min version in placement too | |
| 09:13:24 | bauzas | dvo-plv_: fwiw, I +Wd https://review.opendev.org/c/openstack/os-traits/+/876069 with a comment | |
| 09:22:00 | opendevreview | Merged openstack/os-traits master: Add 'COMPUTE_NET_VIRTIO_PACKED' https://review.opendev.org/c/openstack/os-traits/+/876069 | |
| 09:32:27 | dvo-plv_ | bauzas: Could I update patch with comment, ot it should be done in some another way. because it alredy merged? | |
| 09:32:45 | bauzas | dvo-plv_: just add another patch as a follow-up :) | |
| 09:34:20 | dvo-plv_ | okay, sure | |
| 09:35:23 | kashyap | Can anyone remind me: for picking the min libvirt/QEMU versions what OSes do we care? I'm guessing the "usual": Debian, Ubuntu, Fedora, and RHEL | |
| 09:36:06 | kashyap | In the past we've also considered openSUSE / SLES. Should they be considered too? | |
| 09:36:26 | dvo-plv_ | gibi: Does os-trait release has some estimates? I would like to plan our next activities according to them | |
| 09:37:36 | gibi | dvo-plv_: if you ping me after the os-trait patch lands I can propose the release quickly and it should not take more than couple day to get that release proposal landed | |
| 09:49:51 | kashyap | bauzas: Have we got a name after Bobcat? | |
| 09:54:29 | bauzas | kashyap: nope, afaik not yet | |
| 09:54:42 | opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 10:10:22 | dvo-plv_ | bauzas: I added comment ot the trait: https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 10:27:26 | opendevreview | Merged openstack/nova master: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/862964 | |
| 10:27:34 | opendevreview | Merged openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | |
| 10:31:27 | opendevreview | Merged openstack/nova master: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882160 | |
| 11:15:51 | sean-k-mooney | bauzas: i wont get to it for a week or two but im going to work on a backlog spec for AZ enhancements. i feel liek ther eare enough rough edges and pain point that collecting them in one docs would make sense but while i thik i know how to adrss some of them i dont know how to adress all fo them so im hoping that if we aggreated the pain point some cominalities will emerge | |
| 11:15:53 | sean-k-mooney | that can then be adress in the C cycle after some tought | |
| 11:20:44 | opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 12:45:07 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Selete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | |
| 13:00:10 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Delete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | |
| 13:17:16 | bauzas | tell me I'm wrong but os-port-interfaces and os-security-groups API calls proxy to Neutron, right? | |
| 13:17:30 | sean-k-mooney | yes | |
| 13:17:31 | bauzas | if so, looks like we have a performance regression in the gate https://review.opendev.org/c/openstack/nova/+/882052 | |
| 13:17:47 | bauzas | I got two timeouts | |
| 13:18:27 | sean-k-mooney | tiem outs where | |
| 13:18:35 | sean-k-mooney | i dont see them in the zuul jobs | |
| 13:18:39 | sean-k-mooney | you mean in a test | |
| 13:19:31 | bauzas | yes, see my comments | |
| 13:19:37 | bauzas | you'll see the stacktrace | |
| 13:19:54 | bauzas | I haven't looked yet at Neutron APIs | |
| 13:20:07 | sean-k-mooney | urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='173.231.255.102', port=80): Read timed out. (read timeout=60) | |
| 13:20:28 | sean-k-mooney | that could be related to memory pressure or a slow node | |
| 13:21:06 | bauzas | yes, or no workers available | |
| 13:21:35 | sean-k-mooney | looking at the memory tracker the low point was 1342048 | |
| 13:21:58 | sean-k-mooney | so i dont think that is the issue 130MB is not much but its not terible for these jobs | |
| 13:22:09 | bauzas | read timeouts can happen on a synchronous call if the call takes more than 60 secs to return | |
| 13:22:28 | bauzas | so this could be something else but just a memory issue | |
| 13:22:34 | sean-k-mooney | ya im wondering if the neutron server was deadlocked on something | |
| 13:22:35 | bauzas | like a DB lock | |
| 13:23:17 | sean-k-mooney | ya i know they had issue with db deadlocks last year | |
| 13:24:56 | sean-k-mooney | im seeing quite a few trace backs | |
| 13:25:08 | sean-k-mooney | realed to security gorups that apprently dont exist | |
| 13:27:14 | sean-k-mooney | https://paste.opendev.org/show/bxCPO7Jo1jSzEoRNwHKS/ | |
| 13:30:48 | sean-k-mooney | this seams to be the relevent logs https://paste.opendev.org/show/bMFDSLaKg5TZVpRa3OXD/ | |
| 13:31:44 | sean-k-mooney | there are tracbacks before and after that | |
| 13:32:28 | sean-k-mooney | bauzas: if you care its roughly here https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d/log/controller/logs/screen-q-svc.txt#53526 | |
| 13:33:45 | bauzas | I'll check whether there is an upstream gate report on it | |
| 13:33:50 | bauzas | bug report* | |
| 13:34:20 | sean-k-mooney | neutron does seam to be respondign to the get request May 04 12:02:50.886890 np0033942429 neutron-server[162430]: INFO neutron.wsgi [req-49c6f58c-b29d-4c83-8ad1-5a588383533f req-bc14944a-4c80-4107-9ce5-de32f17088f8 tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] 173.231.255.102 "GET | |
| 13:34:22 | sean-k-mooney | /v2.0/security-groups/b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76 HTTP/1.1" status: 200 len: 1820 time: 0.1247852 | |
| 13:35:43 | sean-k-mooney | but i dont knwo when nova actully made it | |
| 13:36:06 | sean-k-mooney | it could have been sitting in apache waiting for other requests for a while | |
| 13:37:07 | opendevreview | Merged openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 13:38:13 | opendevreview | Sylvain Bauza proposed openstack/nova stable/2023.1: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882293 | |
| 13:41:51 | bauzas | lajoskatona: slaweq: ralonsoh: not sure you already know but it seems we have some Neutron API deadlock in the gate ^ | |
| 13:42:21 | bauzas | and I wasn't able to find an exiting neutron bug report that relates to it | |
| 13:44:57 | ralonsoh | bauzas, but I'm not sure about this method, I don't see in the "list_subnets" method where the neutron client is returning the segment_id | |
| 13:45:08 | ralonsoh | I see that in "show_subnet" | |
| 13:45:28 | bauzas | ralonsoh: ah sorry, you looked at the wrong thing due to my lack of explanation :) | |
| 13:46:16 | bauzas | ralonsoh: see rather the discussion we had between sean-k-mooney and I, the context is https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d | |
| 13:46:36 | ralonsoh | yes, we have the same problem | |