Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-16
21:27:29 sean-k-mooney but not local.conf
21:27:41 sean-k-mooney so i could not do the config overrides i needed
21:28:20 sean-k-mooney to be fair i dont know if normal greande supports that but it was something i found when i treid to use it
21:37:25 sean-k-mooney mriedem: by the way im going to spend some time this week trying to refine the nfv job into something we can have more permentaly and aslo creating an ovs-dpdk job. is this something we would consider merging before RC1 or shoudl i target U with the hope of maybe backporting.
21:38:04 sean-k-mooney initally i want to run them as periodic jobs/via experimental but if i get multiple provider i would like to consier addign them to check if they seam stable
21:38:45 mriedem i would say don't get the cart (backports) before the horse (actually getting something working on master)
21:39:33 sean-k-mooney well i had a dpdk job working w while ago then it broke because fedora
21:39:35 sean-k-mooney https://review.opendev.org/#/c/656580/
21:40:10 sean-k-mooney and the nfv job works. but i want to get them working properly before going anywher near nova check
21:40:20 sean-k-mooney hence start with a nightly periodic job
21:46:28 efried mriedem: any chance of nailing down test_walk_versions this week?
21:47:49 mriedem efried: i lost track of that
21:48:02 mriedem weren't you trying to pull the mysql logs or something for debug?
21:48:18 efried yeah, I think I eventually succeeded in doing that but they didn't show anything interesting.
21:48:50 efried mriedem: https://review.opendev.org/#/c/678051/
21:49:11 sean-k-mooney look like the pcpu reshap went into merge conflict with somehting
21:50:31 efried and a few more of the final 'Aborted connection' lines
21:50:31 efried mriedem: the bad one has this additional message https://zuul.opendev.org/t/openstack/build/f9b92d66ade145a195b996708cd66c28/log/mysql/error.log#202
21:50:35 efried but otherwise they look the same to me.
21:50:43 efried sean-k-mooney: what makes you say that? The yellow dot?
21:50:57 efried oh
21:51:00 sean-k-mooney no the "Patch in Merge Conflict"
21:51:02 efried well wtf
21:51:18 efried nothing has merged lately
21:51:30 sean-k-mooney we merged a few patches like an hour ago
21:51:40 mriedem i've noticed sometimes gerrit will say something is in merge conflict and then i have a triviall conflict-free local rebase
21:52:08 sean-k-mooney proably the service wide pause thing
21:52:22 sean-k-mooney ya
21:52:56 sean-k-mooney shall i pull it down and rebase against master?
21:53:08 efried no
21:53:12 sean-k-mooney ok
21:53:18 efried then we lose all of our +Vs on the rest of the series.
21:53:41 efried Next time a bottom one fails in the gate and -2s the whole pile, maybe we can do that.
21:53:53 sean-k-mooney i was just about to check that
21:54:10 efried mriedem: So should we be getting all those "Aborted connection" things in the first place?
21:54:20 mriedem efried: that i don't know
21:54:22 efried does that mean we're not shutting down the db gracefully?
21:54:31 efried and regardless, should that matter?
21:54:38 mriedem the wacky names are just the random db names
21:54:45 efried is this a job for zzzeek?
21:54:45 sean-k-mooney there are not patches in the gate for nova. they are all in check. but ill leave it wait till the morning
21:55:08 mriedem idk about that, but zzzeek might be able to tell us what to turn on for debugging
21:55:37 efried sean-k-mooney: right, about 12 of the 15 patches have cleared the check queue at this point. The bottom one is waiting; once it completes, the whole swatch will get shoved into the gate pipeline.
21:55:42 mriedem we're also using some deprecated opportunistic db test fixture stuff from oslo.db and i don't know if moving off the deprecated stuff would help us
21:55:59 efried mriedem: well, couldn't hurt. What are you talking about specifically?
21:56:54 mriedem it's been awhile since i looked, sec
21:57:07 sean-k-mooney ya ok then let hold off untill the patches below it have merged
21:57:23 efried sean-k-mooney: and tbh I'm not super worried about a merge conflict on that 15th patch. We've got at least a dozen rechecks to go before we get there.
21:58:01 sean-k-mooney ... hopefully not but we do need to kill some of the random failures
21:58:21 efried well hopefully we killed two of them today
21:58:42 sean-k-mooney ya and mriedem killed 1?2? yesterday
21:58:52 efried I took a couple of samples this morning and determined that the Nth patch takes ~N rechecks to merge.
21:58:54 sean-k-mooney firday i cant rememebr
21:59:10 sean-k-mooney :(
21:59:35 efried mriedem: test_fixtures.PostgresqlOpportunisticFixture and test_fixtures.MySQLOpportunisticFixture ?
22:00:04 mriedem efried: was fixed already https://review.opendev.org/#/c/609352/
22:01:08 efried okay
22:02:53 sean-k-mooney mriedem: by the way i got an error running the sql migate unit tests http://paste.openstack.org/show/776829/
22:03:21 sean-k-mooney given the rest passed im going to push this soon but shoudl tox not install that
22:03:29 sean-k-mooney i guess it could have moved
22:04:50 sean-k-mooney i assume ibm_db_sa is a sqlalcheme pluging for ibm db backend and its just not installled by tox for some reason
22:05:30 mriedem umm, that should just be skipped
22:06:12 mriedem well, or,
22:06:36 mriedem tox should have pulled it in https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py2.txt#L1
22:06:49 sean-k-mooney the py27 version allso requries me to install the mysql hearder to compile the client...
22:06:53 mriedem https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py3.txt#L1
22:06:54 sean-k-mooney i was runnnig py36]
22:07:19 sean-k-mooney which also should install it
22:07:24 mriedem right
22:07:28 mriedem i don't have that problem locally
22:07:40 mriedem are you hitting some pypi mirror?
22:07:51 sean-k-mooney no
22:08:00 sean-k-mooney but i can recreat the env and see what happens
22:08:41 sean-k-mooney the rest of the test pass fine
22:08:59 sean-k-mooney i assume they just use sqlite
22:10:33 sean-k-mooney thats interesting its not in pip freeze in the tox env
22:11:08 sean-k-mooney oh there is not explcit py36 target
22:11:31 sean-k-mooney and it wont look for requirement-py3.txt by defualt
22:12:13 sean-k-mooney i might update the tox file in a sperate patch.
22:14:47 efried mriedem: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/testr_results.html.gz looks like the root cause is some kind of timeout. If we could figure out where, we could probably extend it a bit and make the problem go away...?
22:15:56 efried mmph, it's a socket read, probably a real deadlock.
22:17:29 mriedem sean-k-mooney: do you have this? https://review.opendev.org/#/c/659388/
22:17:36 efried actually, it's the test timeout
22:17:52 mriedem efried: right,
22:17:55 mriedem that's a red herring
22:18:03 sean-k-mooney mriedem: apparently not but i cloned this on friday
22:18:13 mriedem i think we hit some db failure, switch eventlet context, and then never fail the test outright but it times out
22:18:32 mriedem sean-k-mooney: that migrate patch merged awhile back
22:18:45 sean-k-mooney mriedem: oh i cloned it form github so look like github sync for this is broken
22:18:47 efried the time stamps are pretty funky on the log messages leading up to the exception.
22:18:47 mriedem sean-k-mooney: did you clone from github or opendev?
22:18:54 mriedem efried: yup
22:19:01 mriedem sean-k-mooney: yeah you need opendev
22:19:02 efried nearly 12 minutes elapse
22:19:13 sean-k-mooney ya ok ill rebase
22:19:31 sean-k-mooney am should i still clean up the tox file seperatly
22:19:40 mriedem efried: b/c https://github.com/openstack/nova/blob/master/nova/tests/unit/db/test_migrations.py#L72
22:19:47 sean-k-mooney we done actully still support py 26 or py 33/34 right
22:19:55 sean-k-mooney *don't
22:20:17 mriedem sean-k-mooney: how about you clone from the current repo before asking questoins

Earlier   Later