| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-03 | |||
| 17:43:29 | dansmith | btw "nothing is failing" feels *amazing* to say :) | |
| 17:45:34 | gmann | :) | |
| 18:13:03 | gouthamr | ++ | |
| 18:16:09 | gmann | dansmith: replied, agree with Eric to bump tempest version there https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/881764/comments/0be5d072_64ceae34 | |
| 18:17:04 | dansmith | gmann: okay I'll update it, but I don't understand why it would time out | |
| 18:17:32 | dansmith | oh is it because with ACTIVE we wait for the instance state to become the wait_until value? | |
| 18:17:46 | gmann | yeah https://github.com/openstack/tempest/blob/27.0.0/tempest/common/compute.py#L236 | |
| 18:18:02 | gmann | it will pass SSHABLE as server status to wait for | |
| 18:18:17 | dansmith | ack, okay I was focused on the SSHABLE special cases up earlier, I see | |
| 18:18:28 | dansmith | just updated, gouthamr ^ | |
| 18:20:49 | gmann | ack | |
| 18:38:43 | opendevreview | Merged openstack/nova stable/yoga: Remove mentions of removed scheduler filters https://review.opendev.org/c/openstack/nova/+/858025 | |
| 18:41:11 | dansmith | gouthamr: I think I better go ahead and do the cephadm->py3 job name change in the devstack plugin, otherwise we'll break nova as soon as that lands before we either rename it there or change nova (et al) to point to the new job | |
| 18:45:14 | gouthamr | dansmith: +1; to clear this in my head and plan further cleanup - in your revert patch, you set cephadm options in "devstack-plugin-ceph-tempest-py3-base", correct? | |
| 18:46:09 | dansmith | gouthamr: no I set the cephadm thing only in the cephadm job (which would be renamed to -py3).. probably no reason not to set it on the base one yet | |
| 18:46:32 | dansmith | i figured I would rename the jobs and then you can remove the old one when you remove the non cephadm support from it | |
| 18:46:57 | dansmith | just leave the distro-based one named -ubuntu, nonvoting | |
| 18:47:17 | dansmith | I can do the removal and refactoring there, I just don't want to change so much, especially in a patch that was supposed to be a revert :) | |
| 18:48:21 | dansmith | I just want to get this landed so we can start getting more runs on the new stuff and not keep refactoring this, blocking that process | |
| 18:48:38 | dansmith | and since I have it working in this form I just hesitate to move too much at once | |
| 18:56:26 | gouthamr | ack; this makes sense dansmith.. | |
| 18:57:27 | dansmith | gouthamr: okay I just pushed that up let me know if that looks okay | |
| 18:59:53 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test new ceph job configuration with nova https://review.opendev.org/c/openstack/nova/+/881585 | |
| 19:03:23 | gouthamr | dansmith: a couple of comments | |
| 19:04:32 | dansmith | gouthamr: oh gate, right thanks :) | |
| 19:04:41 | dansmith | gouthamr: "gate" has seemed so far off for a long time :D | |
| 19:04:52 | dansmith | gouthamr: I don't know what is needed for rpm | |
| 19:05:07 | dansmith | gouthamr: but since these jobs don't run on there, I think we can defer that | |
| 19:06:46 | gouthamr | dansmith: "devstack-plugin-ceph-cephfs-nfs" is passing; although it doesn't do cephadm, it runs on centos-stream-9 | |
| 19:07:19 | dansmith | right, and those packages are only needed for cephadm | |
| 19:07:22 | gouthamr | dansmith: if we bump that to use cephadm soon, and we hit any issues, we'll get that covered :) | |
| 19:07:28 | dansmith | ack, cool | |
| 20:46:59 | opendevreview | sean mooney proposed openstack/os-vif master: [WIP] set default qos policy https://review.opendev.org/c/openstack/os-vif/+/881751 | |
| 22:04:03 | opendevreview | Dan Smith proposed openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | |
| 22:37:04 | dansmith | I have seen a lot of port binding related failures today, many of them on the grenade multinode job: https://c68ef44fab0ad498c042-f24a7834eba09db97966c05c7e428413.ssl.cf1.rackcdn.com/881585/8/check/nova-grenade-multinode/cbd5c81/testr_results.html | |
| 23:25:18 | melwitt | I saw one yesterday | |
| 23:58:33 | dansmith | maybe sean-k-mooney could take a look in the morning | |
| #openstack-nova - 2023-05-04 | |||
| 01:29:08 | opendevreview | Merged openstack/nova master: add hypervisor version weigher https://review.opendev.org/c/openstack/nova/+/880231 | |
| 08:06:32 | dvo-plv_ | gibi, sean-k-monney: Hello, could you pelase verify os-trait patch: https://review.opendev.org/c/openstack/os-traits/+/876069 to unblock zuul verification for nova patch, cause it fails according to the deendencies | |
| 08:07:12 | dvo-plv_ | sean-k-mooney: sorry, I have made a mistake in your nick | |
| 08:23:19 | gibi | dvo-plv_: the os-traits patch looks good to me. After that merges, we need to propose a new os-traits release, so that you can depend on that release of os-traits in your nova patch | |
| 08:23:24 | bauzas | dang, it took me a while to figure out we were mocking the default policies by a fixture... | |
| 08:23:56 | bauzas | context : https://39770df410ace4902fc0-3eec3a080da8356877afe7c3a8a6cc53.ssl.cf5.rackcdn.com/881562/1/check/openstack-tox-py39/c658228/testr_results.html | |
| 08:25:15 | bauzas | now, I know, we also have a fake policy data : https://github.com/openstack/nova/blob/master/nova/tests/unit/fake_policy.py#L16 | |
| 08:25:19 | bauzas | TIL. | |
| 08:34:40 | dvo-plv_ | gibi: I have only update requirements.txt for nova, or there is some additional doc files what should be updated too | |
| 08:48:12 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a new policy for cold-migrate with host https://review.opendev.org/c/openstack/nova/+/881562 | |
| 09:00:18 | sahid | o/ | |
| 09:09:53 | gibi | dvo-plv_: you only need to updat the requirements.txt on the nova side. | |
| 09:11:03 | bauzas | gibi: but before doing it, we need to have an upper-constraints change, right ? :) | |
| 09:11:18 | gibi | bauzas: right, I above mentioned to dvo-plv_ that we need an os-traits release | |
| 09:11:35 | gibi | but then there will be either an automatic upper constraints bump | |
| 09:11:44 | gibi | or do a manual one in the requirements repo | |
| 09:11:54 | gibi | then the requirements.txt on the nova side can be bumped | |
| 09:12:06 | gibi | in the meantime we should not forget to bump the min version in placement too | |
| 09:13:24 | bauzas | dvo-plv_: fwiw, I +Wd https://review.opendev.org/c/openstack/os-traits/+/876069 with a comment | |
| 09:22:00 | opendevreview | Merged openstack/os-traits master: Add 'COMPUTE_NET_VIRTIO_PACKED' https://review.opendev.org/c/openstack/os-traits/+/876069 | |
| 09:32:27 | dvo-plv_ | bauzas: Could I update patch with comment, ot it should be done in some another way. because it alredy merged? | |
| 09:32:45 | bauzas | dvo-plv_: just add another patch as a follow-up :) | |
| 09:34:20 | dvo-plv_ | okay, sure | |
| 09:35:23 | kashyap | Can anyone remind me: for picking the min libvirt/QEMU versions what OSes do we care? I'm guessing the "usual": Debian, Ubuntu, Fedora, and RHEL | |
| 09:36:06 | kashyap | In the past we've also considered openSUSE / SLES. Should they be considered too? | |
| 09:36:26 | dvo-plv_ | gibi: Does os-trait release has some estimates? I would like to plan our next activities according to them | |
| 09:37:36 | gibi | dvo-plv_: if you ping me after the os-trait patch lands I can propose the release quickly and it should not take more than couple day to get that release proposal landed | |
| 09:49:51 | kashyap | bauzas: Have we got a name after Bobcat? | |
| 09:54:29 | bauzas | kashyap: nope, afaik not yet | |
| 09:54:42 | opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 10:10:22 | dvo-plv_ | bauzas: I added comment ot the trait: https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 10:27:26 | opendevreview | Merged openstack/nova master: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/862964 | |
| 10:27:34 | opendevreview | Merged openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | |
| 10:31:27 | opendevreview | Merged openstack/nova master: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882160 | |
| 11:15:51 | sean-k-mooney | bauzas: i wont get to it for a week or two but im going to work on a backlog spec for AZ enhancements. i feel liek ther eare enough rough edges and pain point that collecting them in one docs would make sense but while i thik i know how to adrss some of them i dont know how to adress all fo them so im hoping that if we aggreated the pain point some cominalities will emerge | |
| 11:15:53 | sean-k-mooney | that can then be adress in the C cycle after some tought | |
| 11:20:44 | opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | |
| 12:45:07 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Selete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | |
| 13:00:10 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Delete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | |
| 13:17:16 | bauzas | tell me I'm wrong but os-port-interfaces and os-security-groups API calls proxy to Neutron, right? | |
| 13:17:30 | sean-k-mooney | yes | |
| 13:17:31 | bauzas | if so, looks like we have a performance regression in the gate https://review.opendev.org/c/openstack/nova/+/882052 | |
| 13:17:47 | bauzas | I got two timeouts | |
| 13:18:27 | sean-k-mooney | tiem outs where | |
| 13:18:35 | sean-k-mooney | i dont see them in the zuul jobs | |
| 13:18:39 | sean-k-mooney | you mean in a test | |
| 13:19:31 | bauzas | yes, see my comments | |
| 13:19:37 | bauzas | you'll see the stacktrace | |
| 13:19:54 | bauzas | I haven't looked yet at Neutron APIs | |
| 13:20:07 | sean-k-mooney | urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='173.231.255.102', port=80): Read timed out. (read timeout=60) | |
| 13:20:28 | sean-k-mooney | that could be related to memory pressure or a slow node | |
| 13:21:06 | bauzas | yes, or no workers available | |
| 13:21:35 | sean-k-mooney | looking at the memory tracker the low point was 1342048 | |
| 13:21:58 | sean-k-mooney | so i dont think that is the issue 130MB is not much but its not terible for these jobs | |
| 13:22:09 | bauzas | read timeouts can happen on a synchronous call if the call takes more than 60 secs to return | |
| 13:22:28 | bauzas | so this could be something else but just a memory issue | |
| 13:22:34 | sean-k-mooney | ya im wondering if the neutron server was deadlocked on something | |
| 13:22:35 | bauzas | like a DB lock | |
| 13:23:17 | sean-k-mooney | ya i know they had issue with db deadlocks last year | |
| 13:24:56 | sean-k-mooney | im seeing quite a few trace backs | |
| 13:25:08 | sean-k-mooney | realed to security gorups that apprently dont exist | |
| 13:27:14 | sean-k-mooney | https://paste.opendev.org/show/bxCPO7Jo1jSzEoRNwHKS/ | |
| 13:30:48 | sean-k-mooney | this seams to be the relevent logs https://paste.opendev.org/show/bMFDSLaKg5TZVpRa3OXD/ | |