| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-10-11 | |||
| 16:09:05 | gibi | timeout is a the visible fault but there is deeper reasons why that happnes | |
| 16:09:28 | bauzas | ok, so let's continue to have this bug report then | |
| 16:09:43 | bauzas | and if someone wants to fix it, he/she could duplicate it if needed | |
| 16:09:57 | elodilles | ack | |
| 16:10:59 | bauzas | ok, any other bug report to look at ? | |
| 16:11:11 | elodilles | nothing else from me | |
| 16:12:52 | bauzas | ok, continuing then | |
| 16:13:09 | bauzas | gibi: can you be the bug baton for next week ? | |
| 16:13:15 | gibi | lets see | |
| 16:13:17 | bauzas | even if next week it will be the PTG ? | |
| 16:13:30 | gibi | I will take it | |
| 16:13:32 | bauzas | ok | |
| 16:13:34 | bauzas | thanks | |
| 16:13:41 | bauzas | #info bug baton is being passed to gibi | |
| 16:13:58 | bauzas | (sorry, passing it to you, as I was having the baton for 2 weeks :) ) | |
| 16:14:13 | bauzas | moving on | |
| 16:14:15 | bauzas | #topic Gate status | |
| 16:14:20 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:14:26 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:14:28 | bauzas | heh :) | |
| 16:14:44 | bauzas | as you see, we have a timeout with the centos9-fips job | |
| 16:15:20 | bauzas | but I dunno for how long it was having this timeout | |
| 16:16:20 | elodilles | https://zuul.openstack.org/builds?job_name=tempest-centos9-stream-fips&project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly&skip=0 | |
| 16:16:28 | elodilles | it seems always? | |
| 16:16:33 | bauzas | yes | |
| 16:17:53 | bauzas | I guess the owner of the job was adalee, right?. | |
| 16:18:40 | bauzas | this is bizarre, I don't see where the job run is timing out | |
| 16:20:01 | sean-k-mooney | well the default timeout i think is 2 hours | |
| 16:20:14 | sean-k-mooney | so it might just need a little longer sicne the fips job does a reboot | |
| 16:20:24 | bauzas | yeah but the job seems to have done | |
| 16:20:30 | bauzas | be* done | |
| 16:20:38 | bauzas | anyway, nothing urgent | |
| 16:20:53 | bauzas | this is just the fact we can't get the result | |
| 16:21:00 | bauzas | even if it looks tempest works fine | |
| 16:21:11 | sean-k-mooney | its posible it has addtional test in a post playbook | |
| 16:21:24 | bauzas | I don't know who wrote the job | |
| 16:21:34 | bauzas | but I'll try to find it | |
| 16:21:46 | bauzas | should be easy to find | |
| 16:22:24 | clarkb | https://zuul.openstack.org/build/5f6e6a2f65ee4a5e90754d07142fab9f/log/job-output.txt#23945-23946 shows where it is timing out | |
| 16:23:24 | bauzas | found who added it https://review.opendev.org/c/openstack/nova/+/831844 | |
| 16:23:54 | bauzas | thanks clarkb, will look | |
| 16:24:31 | sean-k-mooney | ok its tempest full | |
| 16:24:47 | sean-k-mooney | so its two tempest runs | |
| 16:25:05 | bauzas | | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tempest/playbooks/devstack-tempest.yaml@master] | |
| 16:25:09 | sean-k-mooney | the first one completed but the second that runs the slow test and senarior tests timed out | |
| 16:25:10 | bauzas | hmpf | |
| 16:25:24 | bauzas | yup | |
| 16:25:45 | sean-k-mooney | so ya it just need anther say 30 mins added to the tiemout to be safe | |
| 16:26:05 | sean-k-mooney | it also could have just been a slow node | |
| 16:26:10 | clarkb | and maybe double check logs to see that something didn't get stuck there due to fips | |
| 16:26:19 | clarkb | there is a bit of a time delta between the timeout and tempest reporting anything | |
| 16:26:38 | clarkb | oh its only 15 seconds nevermind | |
| 16:26:42 | sean-k-mooney | yep | |
| 16:26:47 | sean-k-mooney | there isnt really a break in the logs | |
| 16:26:53 | clarkb | I mathed wrong the first time | |
| 16:27:08 | bauzas | yes, this is just a slow test | |
| 16:27:42 | bauzas | but should we modify the timeout for that job to be larger ? | |
| 16:27:46 | sean-k-mooney | i dont know how much time the reboot adds for fips but 2 hours is proably borderlien | |
| 16:28:01 | bauzas | we can DNM this at least | |
| 16:28:16 | bauzas | and verify if adding more timeout time helps | |
| 16:28:17 | sean-k-mooney | i would add 30mins and monitor it and see roughly how long it takes over the next few weeks | |
| 16:28:23 | bauzas | right | |
| 16:28:46 | bauzas | I can propose a patch | |
| 16:28:51 | sean-k-mooney | ack | |
| 16:29:21 | bauzas | #action bauzas to add a 30min more timeout for the centos9-fips periodic job so we will see whether it fixes the timeout | |
| 16:29:27 | bauzas | moving on | |
| 16:29:38 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:29:51 | sean-k-mooney | bauzas: its here https://opendev.org/openstack/tempest/src/branch/master/zuul.d/integrated-gate.yaml#L325 just incase your wondering where to do that | |
| 16:31:19 | bauzas | ack, thanks but I found it | |
| 16:31:40 | bauzas | hence the DNM patch in nova | |
| 16:31:51 | bauzas | to test it | |
| 16:32:07 | bauzas | with experimental | |
| 16:32:10 | bauzas | anyway | |
| 16:32:36 | bauzas | nothing to tell more about blind rechecks, I haven't looked at the numbers | |
| 16:32:38 | bauzas | moving on | |
| 16:32:48 | bauzas | #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | |
| 16:32:53 | bauzas | #topic Release Planning | |
| 16:32:59 | bauzas | #link https://releases.openstack.org/zed/schedule.html | |
| 16:33:04 | bauzas | #info Zed GA was last week, kudos to the team. | |
| 16:33:10 | bauzas | we're officially done with Zed :) | |
| 16:33:21 | bauzas | so... | |
| 16:33:23 | bauzas | #link https://releases.openstack.org/antelope/schedule.html | |
| 16:33:28 | bauzas | #info Antelope-1 is planned in 5 weeks | |
| 16:33:38 | bauzas | nothing to tell it, the crux is for the next topic | |
| 16:33:50 | bauzas | #topic PTG planning | |
| 16:33:55 | bauzas | #link https://etherpad.opendev.org/p/nova-antelope-ptg Antelope PTG etherpad | |
| 16:34:00 | bauzas | #link https://ptg.opendev.org/ptg.html PTG schedule | |
| 16:34:09 | bauzas | #info Attempted schedule in the etherpad | |
| 16:34:21 | bauzas | yeah I did a bit of a topics scrambling | |
| 16:34:38 | bauzas | but honestly, we don't have a large agendat *yet* | |
| 16:35:04 | bauzas | I'll just move the nova-otherproject topics | |
| 16:35:29 | bauzas | for the moment, we have nova-neutron (Thur) and nova-ironic (reminder : *Monday*) | |
| 16:35:44 | bauzas | do people want to talk with cinder or other projects ? | |
| 16:35:52 | bauzas | I haven't got any ask from the cyborg project | |
| 16:36:05 | bauzas | or oslo or QA ? | |
| 16:36:25 | bauzas | or manila, or glance... :) | |
| 16:36:57 | gibi | nothing on my side | |
| 16:37:04 | sean-k-mooney | i think the ironic topic will fill any space we give it | |
| 16:37:14 | sean-k-mooney | but im not sure if we will be able to make progress remotely | |
| 16:37:57 | bauzas | hah | |