Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-11-29
16:26:45 bauzas #topic Open discussion
16:26:55 bauzas nothing in the wikipage
16:26:58 bauzas so
16:27:04 bauzas anything to discuss here by now ?
16:27:08 gibi -
16:27:17 sean-k-mooney did you merge skipign the failing nova-lvm tests yet
16:27:26 sean-k-mooney or is the master gate still explodingon that
16:27:30 bauzas I think yesterday we said we could discuss during this meeting about the test skips
16:27:46 bauzas but given we merged gmann's patch, the ship has sailed
16:27:56 sean-k-mooney ack
16:28:02 sean-k-mooney so they are disabeled currently
16:28:05 bauzas sean-k-mooney: see my ML thread above ^
16:28:06 sean-k-mooney the failing detach tests
16:28:15 sean-k-mooney ah ok will check after meeting
16:28:19 bauzas sean-k-mooney: you'll get the link to the gerrit change
16:28:20 sean-k-mooney nothing else form me
16:28:29 auniyal hand-raise: zuul frequent timeout issue/fails - this seems to be resource issue, is it possible zuull resource can be increased ?
16:29:09 bauzas sean-k-mooney: tl;dr: yes we skipped the related tests but maybe they are actually not needed as you said
16:29:16 sean-k-mooney auniyal: not really timeout are not that common in our jobs
16:29:24 bauzas auniyal: see what I said above, we had problems with the gate very recently
16:29:26 sean-k-mooney auniyal: do you have an example
16:29:31 auniyal in morning when there are less number of jobs running if we run same, job gets passed
16:29:40 auniyal like less then 20
16:29:47 auniyal right now 60 jobs are running
16:29:52 sean-k-mooney that should not really be a thing
16:30:03 sean-k-mooney unless we have issues with our ci providers
16:30:17 bauzas auniyal: if you speak about job results telling timeouts, agreed with sean-k-mooney, you should tell which ones so we could investigate
16:30:24 sean-k-mooney we ocationally have issues with slow providers but its not normally coralated with the number of runnign jobs
16:30:29 bauzas yup
16:30:35 auniyal ack
16:30:38 bauzas timeouts are generally an infra issue
16:30:43 bauzas from a ci provider
16:30:50 bauzas but "generally"
16:31:04 bauzas which means sometimes we may have a larger problem
16:31:07 sean-k-mooney auniyal: do you have a gerrit link to a change where it happend
16:31:12 dansmith are they fips jobs?
16:31:31 clarkb bauzas: I'm not sure I agree with that statement
16:31:31 sean-k-mooney oh ya it could be that did we add the extra 30 mins ot the job yet
16:31:38 clarkb we have significant amounts of very inefficient test payload
16:31:55 clarkb yes slow providers make that worse, but we have lots of ability to improve things in the jobs just about every time I look
16:32:15 sean-k-mooney clarkb: we dont often see timeouts in the jobs that run on the nova gate
16:32:29 sean-k-mooney we tent to be well within the job timeout interval
16:32:45 sean-k-mooney that is not nessialy the same for other projects
16:32:46 clarkb (it is common for tempets jobs to dig into swap which slows everything down, devstack uses osc which is super slow because it gets a new token for every request and has python spin up time, ansible loops are costly with large numbers of entries and so on)
16:32:58 auniyal sean, I am trying to find a link but its time taking
16:33:04 clarkb sean-k-mooney: yes swap is a common cause for the difference in behaviors and that isn't an infra issue
16:33:15 clarkb sean-k-mooney: and devstack runtime could be ~halved if we stopped using osc
16:33:25 clarkb or improved osc's startup and token acquisition time
16:33:31 sean-k-mooney clarkb: ack
16:33:36 clarkb I just want to avoid the idea its an infra issue so ignore it
16:33:39 sean-k-mooney ya the osc thing is a long runing known issue
16:33:49 clarkb this assertion gets made often then I go looking and there is plenty of job payload that is just slow
16:33:55 sean-k-mooney the parallel improments dansmith did helped indirectly
16:33:55 bauzas clarkb: sorry, I was unclear
16:33:59 auniyal although, I have experinced this alot, if my zuul, is not passing at night time (IST), even after recheck I ran them in morning, then pass
16:34:10 bauzas clarkb: I wasn't advocating about someone else's fault
16:34:52 bauzas clarkb: I was just explaining to some new nova contributor that given the current situation, we only have timeouts with nova jobs due to some ci provider issue
16:35:04 clarkb bauzas: right I disagree with that
16:35:13 bauzas clarkb: but I agree with you on some jobs that are wasting resources
16:35:13 clarkb jobs timeout due to an accumulation of slow steps
16:35:21 clarkb some of those may be due to a slow provider or slow instance
16:35:30 clarkb but, it is extremely rare that this is the only problem
16:35:34 sean-k-mooney clarkb: we tend to be seeing an avgerate runtime at about 75% or less of the job timeout in my experince
16:35:42 clarkb and I know nova tempest jobs have a large number of other slowness problems
16:35:55 clarkb sean-k-mooney: yes, but if a job digs deeply into swap its all downhill from there
16:35:56 bauzas clarkb: that's a fair point
16:35:56 sean-k-mooney we have 2 hour timeouts on our tempest jobs and we rarly go above about 90 mins
16:36:05 clarkb suddenly your 75% typical runtime can balloon to 200%
16:36:08 bauzas except the fips one
16:36:14 sean-k-mooney clarkb: sure but i dont think we are
16:36:25 sean-k-mooney but its somethign we can look at
16:36:40 sean-k-mooney auniyal: the best thing you can do is provide us an example and we can look into it
16:36:46 clarkb ++ to looking at it
16:36:47 sean-k-mooney and then see if there is a trend
16:37:10 auniyal ack
16:38:36 bauzas I actually wonder how we can track the trend
16:38:50 sean-k-mooney https://zuul.openstack.org/builds?project=openstack%2Fnova&result=TIMED_OUT&skip=0
16:38:56 sean-k-mooney that but its currently loading
16:39:17 sean-k-mooney we have a couple every few days
16:39:18 bauzas sure, but you don't have the time a SUCCESS job runs
16:39:28 bauzas which is what we should track
16:39:29 clarkb you can show both success and timeouts in a listing
16:39:35 clarkb (and failures, etc)
16:39:37 sean-k-mooney well we can change the result to filter both
16:39:51 bauzas the duration field, shit, missed it
16:40:01 sean-k-mooney we also hav ento fixed the fips job
16:40:09 sean-k-mooney ill create a patch for that i think
16:40:16 bauzas sean-k-mooney: I said I should do it
16:40:27 sean-k-mooney bauzas: ok please do
16:40:33 bauzas sean-k-mooney: that's simple to do and that's like 4 weeks I promised it
16:41:13 bauzas sean-k-mooney: you know what ? I'll end this meeting by now so everyone can do what they want, including me writing a zuul patch :)
16:41:25 sean-k-mooney :)
16:41:35 bauzas having said it,
16:41:39 bauzas thanks folks
16:41:43 bauzas #endmeeting
16:41:43 opendevmeet Meeting ended Tue Nov 29 16:41:43 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:41:43 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-11-29-16.00.html
16:41:43 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-11-29-16.00.txt
16:41:43 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-11-29-16.00.log.html
16:42:03 gibi o/

Earlier   Later