Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-16
21:54:45 sean-k-mooney there are not patches in the gate for nova. they are all in check. but ill leave it wait till the morning
21:55:08 mriedem idk about that, but zzzeek might be able to tell us what to turn on for debugging
21:55:37 efried sean-k-mooney: right, about 12 of the 15 patches have cleared the check queue at this point. The bottom one is waiting; once it completes, the whole swatch will get shoved into the gate pipeline.
21:55:42 mriedem we're also using some deprecated opportunistic db test fixture stuff from oslo.db and i don't know if moving off the deprecated stuff would help us
21:55:59 efried mriedem: well, couldn't hurt. What are you talking about specifically?
21:56:54 mriedem it's been awhile since i looked, sec
21:57:07 sean-k-mooney ya ok then let hold off untill the patches below it have merged
21:57:23 efried sean-k-mooney: and tbh I'm not super worried about a merge conflict on that 15th patch. We've got at least a dozen rechecks to go before we get there.
21:58:01 sean-k-mooney ... hopefully not but we do need to kill some of the random failures
21:58:21 efried well hopefully we killed two of them today
21:58:42 sean-k-mooney ya and mriedem killed 1?2? yesterday
21:58:52 efried I took a couple of samples this morning and determined that the Nth patch takes ~N rechecks to merge.
21:58:54 sean-k-mooney firday i cant rememebr
21:59:10 sean-k-mooney :(
21:59:35 efried mriedem: test_fixtures.PostgresqlOpportunisticFixture and test_fixtures.MySQLOpportunisticFixture ?
22:00:04 mriedem efried: was fixed already https://review.opendev.org/#/c/609352/
22:01:08 efried okay
22:02:53 sean-k-mooney mriedem: by the way i got an error running the sql migate unit tests http://paste.openstack.org/show/776829/
22:03:21 sean-k-mooney given the rest passed im going to push this soon but shoudl tox not install that
22:03:29 sean-k-mooney i guess it could have moved
22:04:50 sean-k-mooney i assume ibm_db_sa is a sqlalcheme pluging for ibm db backend and its just not installled by tox for some reason
22:05:30 mriedem umm, that should just be skipped
22:06:12 mriedem well, or,
22:06:36 mriedem tox should have pulled it in https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py2.txt#L1
22:06:49 sean-k-mooney the py27 version allso requries me to install the mysql hearder to compile the client...
22:06:53 mriedem https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py3.txt#L1
22:06:54 sean-k-mooney i was runnnig py36]
22:07:19 sean-k-mooney which also should install it
22:07:24 mriedem right
22:07:28 mriedem i don't have that problem locally
22:07:40 mriedem are you hitting some pypi mirror?
22:07:51 sean-k-mooney no
22:08:00 sean-k-mooney but i can recreat the env and see what happens
22:08:41 sean-k-mooney the rest of the test pass fine
22:08:59 sean-k-mooney i assume they just use sqlite
22:10:33 sean-k-mooney thats interesting its not in pip freeze in the tox env
22:11:08 sean-k-mooney oh there is not explcit py36 target
22:11:31 sean-k-mooney and it wont look for requirement-py3.txt by defualt
22:12:13 sean-k-mooney i might update the tox file in a sperate patch.
22:14:47 efried mriedem: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/testr_results.html.gz looks like the root cause is some kind of timeout. If we could figure out where, we could probably extend it a bit and make the problem go away...?
22:15:56 efried mmph, it's a socket read, probably a real deadlock.
22:17:29 mriedem sean-k-mooney: do you have this? https://review.opendev.org/#/c/659388/
22:17:36 efried actually, it's the test timeout
22:17:52 mriedem efried: right,
22:17:55 mriedem that's a red herring
22:18:03 sean-k-mooney mriedem: apparently not but i cloned this on friday
22:18:13 mriedem i think we hit some db failure, switch eventlet context, and then never fail the test outright but it times out
22:18:32 mriedem sean-k-mooney: that migrate patch merged awhile back
22:18:45 sean-k-mooney mriedem: oh i cloned it form github so look like github sync for this is broken
22:18:47 efried the time stamps are pretty funky on the log messages leading up to the exception.
22:18:47 mriedem sean-k-mooney: did you clone from github or opendev?
22:18:54 mriedem efried: yup
22:19:01 mriedem sean-k-mooney: yeah you need opendev
22:19:02 efried nearly 12 minutes elapse
22:19:13 sean-k-mooney ya ok ill rebase
22:19:31 sean-k-mooney am should i still clean up the tox file seperatly
22:19:40 mriedem efried: b/c https://github.com/openstack/nova/blob/master/nova/tests/unit/db/test_migrations.py#L72
22:19:47 sean-k-mooney we done actully still support py 26 or py 33/34 right
22:19:55 sean-k-mooney *don't
22:20:17 mriedem sean-k-mooney: how about you clone from the current repo before asking questoins
22:20:18 efried 4 * 160 = 11m40s, yup, that's right on.
22:20:52 mriedem the timeouts on those were bumped a couple of years ago for a valid reason at the time (slow nodes i think)
22:20:58 mriedem https://github.com/openstack/nova/commit/71d6333f855e139894f497fc120487895a1d66ce
22:21:14 mriedem but the thread switch thing kills us
22:23:20 efried this is always on limestone-regionone??
22:25:09 efried that... seems like a thing
22:25:17 efried is it because limestone has slower nodes consistently?
22:25:26 efried or because of some particular config over there?
22:25:52 sean-k-mooney are the limesotone jobs gernerally slower or is it just this one thing that is failing
22:25:55 efried could we... increase the timeout s'more and see if it goes away? Not the greatest answer, but overall gate time wasted would be less
22:26:08 sean-k-mooney of the over all job is slow then it coudl be that its just slower in general
22:26:09 efried sean-k-mooney: I dunno, before digging into that I was going to see if anyone knew off the top.
22:26:37 efried not sure how I would look into that, other than clicking through a zillion different builds
22:26:40 mriedem yes it's always limestone
22:26:46 mriedem idk if they are slower or what, you'd have to ask in infra
22:27:14 mriedem there is probably some way to find average job time per node provider and nova openstack-tox-py36 job or something
22:27:16 sean-k-mooney infra have many grapha dashborads that track things but i dont know if they yave abuild time per regoin one
22:27:25 sean-k-mooney you could proably figure it out form logstash
22:27:42 mriedem i think awhile back i had wondered if those nodes used some different mysql binary or something
22:28:16 mriedem for everything to be not crazy though it should be the same ubuntu 18.04 image and such regardless of node provider
22:28:23 sean-k-mooney they should all use the nodepool image and then hit the infra mirros which are avaible via afs in all regions right
22:28:26 sean-k-mooney so it shoudl be the same
22:29:17 efried http://zuul.openstack.org/builds?job_name=openstack-tox-py27&branch=master&project=openstack%2Fnova but this table doesn't show the node provider...
22:29:21 sean-k-mooney i think infra has been uploading its own image nightly since the zuul v2 days to not have random things break in diffrent clouds
22:30:16 logan- o/ from limestone
22:30:50 efried this slow bastard happened on rax https://zuul.opendev.org/t/openstack/build/38181c3314284545943b38b52c72f8d7/log/job-output.txt
22:31:54 sean-k-mooney logan- they are trying to debug a test that seam to be hitting a timeout
22:32:16 sean-k-mooney logan-: but only (mostly?) happens in limestone-regionone
22:33:03 alex_xu efried: I'm here
22:33:04 sean-k-mooney logan-: so we were wondering if limesotne is generally slower the other providre and hiting the timeout more often or if it could be a config/could issue
22:33:38 sean-k-mooney logan-: e.g. is there something differnet that could make it fail more often on limestone or it that just a red herring
22:33:45 efried alex_xu: Can you, or Rui, or luyao, or someone please formulate a bullet for vpmem for the train cycle highlights in a followon to https://review.opendev.org/#/c/681943/ ?
22:34:11 alex_xu efried: got it, will do today
22:34:25 efried alex_xu: If you seed the message, I'll scrub the grammar etc. Thanks.
22:34:38 alex_xu efried: thanks
22:35:38 logan- i'm not sure if our testcloud is slower than other nodepool providers. certainly the image would be the same as any other nodepool node. looking thru the log.. i'm curious if the job hangs somewhere specific, or if its just slower in general. and if our nodes are just showing up slower on average i'd like to know so we can figure out if it is h/w or config related.
22:36:23 efried logan-: The thing we're trying to nail down is http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22sqlalchemy.exc.InterfaceError:%20(pymysql.err.InterfaceError)%20(0,%20'')%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s
22:36:48 efried we consistently see the test suite time out on the same 1-3 tests
22:39:22 efried logan-: and if you punch the `node_provider` checkbox on the left-hand side next to the result list, you see it's *always* limestone-regionone
22:40:52 logan- https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/mysql/error.log are any of these messages at the bottom pertinent?

Earlier   Later