| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-16 | |||
| 21:52:56 | sean-k-mooney | shall i pull it down and rebase against master? | |
| 21:53:08 | efried | no | |
| 21:53:12 | sean-k-mooney | ok | |
| 21:53:18 | efried | then we lose all of our +Vs on the rest of the series. | |
| 21:53:41 | efried | Next time a bottom one fails in the gate and -2s the whole pile, maybe we can do that. | |
| 21:53:53 | sean-k-mooney | i was just about to check that | |
| 21:54:10 | efried | mriedem: So should we be getting all those "Aborted connection" things in the first place? | |
| 21:54:20 | mriedem | efried: that i don't know | |
| 21:54:22 | efried | does that mean we're not shutting down the db gracefully? | |
| 21:54:31 | efried | and regardless, should that matter? | |
| 21:54:38 | mriedem | the wacky names are just the random db names | |
| 21:54:45 | efried | is this a job for zzzeek? | |
| 21:54:45 | sean-k-mooney | there are not patches in the gate for nova. they are all in check. but ill leave it wait till the morning | |
| 21:55:08 | mriedem | idk about that, but zzzeek might be able to tell us what to turn on for debugging | |
| 21:55:37 | efried | sean-k-mooney: right, about 12 of the 15 patches have cleared the check queue at this point. The bottom one is waiting; once it completes, the whole swatch will get shoved into the gate pipeline. | |
| 21:55:42 | mriedem | we're also using some deprecated opportunistic db test fixture stuff from oslo.db and i don't know if moving off the deprecated stuff would help us | |
| 21:55:59 | efried | mriedem: well, couldn't hurt. What are you talking about specifically? | |
| 21:56:54 | mriedem | it's been awhile since i looked, sec | |
| 21:57:07 | sean-k-mooney | ya ok then let hold off untill the patches below it have merged | |
| 21:57:23 | efried | sean-k-mooney: and tbh I'm not super worried about a merge conflict on that 15th patch. We've got at least a dozen rechecks to go before we get there. | |
| 21:58:01 | sean-k-mooney | ... hopefully not but we do need to kill some of the random failures | |
| 21:58:21 | efried | well hopefully we killed two of them today | |
| 21:58:42 | sean-k-mooney | ya and mriedem killed 1?2? yesterday | |
| 21:58:52 | efried | I took a couple of samples this morning and determined that the Nth patch takes ~N rechecks to merge. | |
| 21:58:54 | sean-k-mooney | firday i cant rememebr | |
| 21:59:10 | sean-k-mooney | :( | |
| 21:59:35 | efried | mriedem: test_fixtures.PostgresqlOpportunisticFixture and test_fixtures.MySQLOpportunisticFixture ? | |
| 22:00:04 | mriedem | efried: was fixed already https://review.opendev.org/#/c/609352/ | |
| 22:01:08 | efried | okay | |
| 22:02:53 | sean-k-mooney | mriedem: by the way i got an error running the sql migate unit tests http://paste.openstack.org/show/776829/ | |
| 22:03:21 | sean-k-mooney | given the rest passed im going to push this soon but shoudl tox not install that | |
| 22:03:29 | sean-k-mooney | i guess it could have moved | |
| 22:04:50 | sean-k-mooney | i assume ibm_db_sa is a sqlalcheme pluging for ibm db backend and its just not installled by tox for some reason | |
| 22:05:30 | mriedem | umm, that should just be skipped | |
| 22:06:12 | mriedem | well, or, | |
| 22:06:36 | mriedem | tox should have pulled it in https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py2.txt#L1 | |
| 22:06:49 | sean-k-mooney | the py27 version allso requries me to install the mysql hearder to compile the client... | |
| 22:06:53 | mriedem | https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py3.txt#L1 | |
| 22:06:54 | sean-k-mooney | i was runnnig py36] | |
| 22:07:19 | sean-k-mooney | which also should install it | |
| 22:07:24 | mriedem | right | |
| 22:07:28 | mriedem | i don't have that problem locally | |
| 22:07:40 | mriedem | are you hitting some pypi mirror? | |
| 22:07:51 | sean-k-mooney | no | |
| 22:08:00 | sean-k-mooney | but i can recreat the env and see what happens | |
| 22:08:41 | sean-k-mooney | the rest of the test pass fine | |
| 22:08:59 | sean-k-mooney | i assume they just use sqlite | |
| 22:10:33 | sean-k-mooney | thats interesting its not in pip freeze in the tox env | |
| 22:11:08 | sean-k-mooney | oh there is not explcit py36 target | |
| 22:11:31 | sean-k-mooney | and it wont look for requirement-py3.txt by defualt | |
| 22:12:13 | sean-k-mooney | i might update the tox file in a sperate patch. | |
| 22:14:47 | efried | mriedem: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/testr_results.html.gz looks like the root cause is some kind of timeout. If we could figure out where, we could probably extend it a bit and make the problem go away...? | |
| 22:15:56 | efried | mmph, it's a socket read, probably a real deadlock. | |
| 22:17:29 | mriedem | sean-k-mooney: do you have this? https://review.opendev.org/#/c/659388/ | |
| 22:17:36 | efried | actually, it's the test timeout | |
| 22:17:52 | mriedem | efried: right, | |
| 22:17:55 | mriedem | that's a red herring | |
| 22:18:03 | sean-k-mooney | mriedem: apparently not but i cloned this on friday | |
| 22:18:13 | mriedem | i think we hit some db failure, switch eventlet context, and then never fail the test outright but it times out | |
| 22:18:32 | mriedem | sean-k-mooney: that migrate patch merged awhile back | |
| 22:18:45 | sean-k-mooney | mriedem: oh i cloned it form github so look like github sync for this is broken | |
| 22:18:47 | efried | the time stamps are pretty funky on the log messages leading up to the exception. | |
| 22:18:47 | mriedem | sean-k-mooney: did you clone from github or opendev? | |
| 22:18:54 | mriedem | efried: yup | |
| 22:19:01 | mriedem | sean-k-mooney: yeah you need opendev | |
| 22:19:02 | efried | nearly 12 minutes elapse | |
| 22:19:13 | sean-k-mooney | ya ok ill rebase | |
| 22:19:31 | sean-k-mooney | am should i still clean up the tox file seperatly | |
| 22:19:40 | mriedem | efried: b/c https://github.com/openstack/nova/blob/master/nova/tests/unit/db/test_migrations.py#L72 | |
| 22:19:47 | sean-k-mooney | we done actully still support py 26 or py 33/34 right | |
| 22:19:55 | sean-k-mooney | *don't | |
| 22:20:17 | mriedem | sean-k-mooney: how about you clone from the current repo before asking questoins | |
| 22:20:18 | efried | 4 * 160 = 11m40s, yup, that's right on. | |
| 22:20:52 | mriedem | the timeouts on those were bumped a couple of years ago for a valid reason at the time (slow nodes i think) | |
| 22:20:58 | mriedem | https://github.com/openstack/nova/commit/71d6333f855e139894f497fc120487895a1d66ce | |
| 22:21:14 | mriedem | but the thread switch thing kills us | |
| 22:23:20 | efried | this is always on limestone-regionone?? | |
| 22:25:09 | efried | that... seems like a thing | |
| 22:25:17 | efried | is it because limestone has slower nodes consistently? | |
| 22:25:26 | efried | or because of some particular config over there? | |
| 22:25:52 | sean-k-mooney | are the limesotone jobs gernerally slower or is it just this one thing that is failing | |
| 22:25:55 | efried | could we... increase the timeout s'more and see if it goes away? Not the greatest answer, but overall gate time wasted would be less | |
| 22:26:08 | sean-k-mooney | of the over all job is slow then it coudl be that its just slower in general | |
| 22:26:09 | efried | sean-k-mooney: I dunno, before digging into that I was going to see if anyone knew off the top. | |
| 22:26:37 | efried | not sure how I would look into that, other than clicking through a zillion different builds | |
| 22:26:40 | mriedem | yes it's always limestone | |
| 22:26:46 | mriedem | idk if they are slower or what, you'd have to ask in infra | |
| 22:27:14 | mriedem | there is probably some way to find average job time per node provider and nova openstack-tox-py36 job or something | |
| 22:27:16 | sean-k-mooney | infra have many grapha dashborads that track things but i dont know if they yave abuild time per regoin one | |
| 22:27:25 | sean-k-mooney | you could proably figure it out form logstash | |
| 22:27:42 | mriedem | i think awhile back i had wondered if those nodes used some different mysql binary or something | |
| 22:28:16 | mriedem | for everything to be not crazy though it should be the same ubuntu 18.04 image and such regardless of node provider | |
| 22:28:23 | sean-k-mooney | they should all use the nodepool image and then hit the infra mirros which are avaible via afs in all regions right | |
| 22:28:26 | sean-k-mooney | so it shoudl be the same | |
| 22:29:17 | efried | http://zuul.openstack.org/builds?job_name=openstack-tox-py27&branch=master&project=openstack%2Fnova but this table doesn't show the node provider... | |
| 22:29:21 | sean-k-mooney | i think infra has been uploading its own image nightly since the zuul v2 days to not have random things break in diffrent clouds | |
| 22:30:16 | logan- | o/ from limestone | |
| 22:30:50 | efried | this slow bastard happened on rax https://zuul.opendev.org/t/openstack/build/38181c3314284545943b38b52c72f8d7/log/job-output.txt | |
| 22:31:54 | sean-k-mooney | logan- they are trying to debug a test that seam to be hitting a timeout | |
| 22:32:16 | sean-k-mooney | logan-: but only (mostly?) happens in limestone-regionone | |