| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-16 | |||
| 21:24:57 | mriedem | it's all just one | |
| 21:24:58 | sean-k-mooney | to start the conductor in the functional test | |
| 21:25:06 | mriedem | efried: meaning we get away with shit in functional tests that we wouldn't in devstack | |
| 21:25:47 | mriedem | e.g. we do stuff like have the CheatingSerializer so it all looks like one RPC | |
| 21:25:54 | sean-k-mooney | i think devstack has the abiltiy to deploy in the legacy mode too but i dont know if thats tested still so it could be broken | |
| 21:25:58 | mriedem | which has been a problem in my cross-cell series with functional tests | |
| 21:26:10 | mriedem | sean-k-mooney: all grenade jobs are non-superconductor | |
| 21:26:39 | sean-k-mooney | ah ok. is that still true for the new zuulv3 version? | |
| 21:27:07 | mriedem | i would expect so but haven't looked | |
| 21:27:22 | sean-k-mooney | i tried using that to do a greade version of my numa jobs but i found i could not set local.conf diffrently for new/old nodes | |
| 21:27:25 | sean-k-mooney | localrc yes | |
| 21:27:29 | sean-k-mooney | but not local.conf | |
| 21:27:41 | sean-k-mooney | so i could not do the config overrides i needed | |
| 21:28:20 | sean-k-mooney | to be fair i dont know if normal greande supports that but it was something i found when i treid to use it | |
| 21:37:25 | sean-k-mooney | mriedem: by the way im going to spend some time this week trying to refine the nfv job into something we can have more permentaly and aslo creating an ovs-dpdk job. is this something we would consider merging before RC1 or shoudl i target U with the hope of maybe backporting. | |
| 21:38:04 | sean-k-mooney | initally i want to run them as periodic jobs/via experimental but if i get multiple provider i would like to consier addign them to check if they seam stable | |
| 21:38:45 | mriedem | i would say don't get the cart (backports) before the horse (actually getting something working on master) | |
| 21:39:33 | sean-k-mooney | well i had a dpdk job working w while ago then it broke because fedora | |
| 21:39:35 | sean-k-mooney | https://review.opendev.org/#/c/656580/ | |
| 21:40:10 | sean-k-mooney | and the nfv job works. but i want to get them working properly before going anywher near nova check | |
| 21:40:20 | sean-k-mooney | hence start with a nightly periodic job | |
| 21:46:28 | efried | mriedem: any chance of nailing down test_walk_versions this week? | |
| 21:47:49 | mriedem | efried: i lost track of that | |
| 21:48:02 | mriedem | weren't you trying to pull the mysql logs or something for debug? | |
| 21:48:18 | efried | yeah, I think I eventually succeeded in doing that but they didn't show anything interesting. | |
| 21:48:50 | efried | mriedem: https://review.opendev.org/#/c/678051/ | |
| 21:49:11 | sean-k-mooney | look like the pcpu reshap went into merge conflict with somehting | |
| 21:50:31 | efried | and a few more of the final 'Aborted connection' lines | |
| 21:50:31 | efried | mriedem: the bad one has this additional message https://zuul.opendev.org/t/openstack/build/f9b92d66ade145a195b996708cd66c28/log/mysql/error.log#202 | |
| 21:50:35 | efried | but otherwise they look the same to me. | |
| 21:50:43 | efried | sean-k-mooney: what makes you say that? The yellow dot? | |
| 21:50:57 | efried | oh | |
| 21:51:00 | sean-k-mooney | no the "Patch in Merge Conflict" | |
| 21:51:02 | efried | well wtf | |
| 21:51:18 | efried | nothing has merged lately | |
| 21:51:30 | sean-k-mooney | we merged a few patches like an hour ago | |
| 21:51:40 | mriedem | i've noticed sometimes gerrit will say something is in merge conflict and then i have a triviall conflict-free local rebase | |
| 21:52:08 | sean-k-mooney | proably the service wide pause thing | |
| 21:52:22 | sean-k-mooney | ya | |
| 21:52:56 | sean-k-mooney | shall i pull it down and rebase against master? | |
| 21:53:08 | efried | no | |
| 21:53:12 | sean-k-mooney | ok | |
| 21:53:18 | efried | then we lose all of our +Vs on the rest of the series. | |
| 21:53:41 | efried | Next time a bottom one fails in the gate and -2s the whole pile, maybe we can do that. | |
| 21:53:53 | sean-k-mooney | i was just about to check that | |
| 21:54:10 | efried | mriedem: So should we be getting all those "Aborted connection" things in the first place? | |
| 21:54:20 | mriedem | efried: that i don't know | |
| 21:54:22 | efried | does that mean we're not shutting down the db gracefully? | |
| 21:54:31 | efried | and regardless, should that matter? | |
| 21:54:38 | mriedem | the wacky names are just the random db names | |
| 21:54:45 | efried | is this a job for zzzeek? | |
| 21:54:45 | sean-k-mooney | there are not patches in the gate for nova. they are all in check. but ill leave it wait till the morning | |
| 21:55:08 | mriedem | idk about that, but zzzeek might be able to tell us what to turn on for debugging | |
| 21:55:37 | efried | sean-k-mooney: right, about 12 of the 15 patches have cleared the check queue at this point. The bottom one is waiting; once it completes, the whole swatch will get shoved into the gate pipeline. | |
| 21:55:42 | mriedem | we're also using some deprecated opportunistic db test fixture stuff from oslo.db and i don't know if moving off the deprecated stuff would help us | |
| 21:55:59 | efried | mriedem: well, couldn't hurt. What are you talking about specifically? | |
| 21:56:54 | mriedem | it's been awhile since i looked, sec | |
| 21:57:07 | sean-k-mooney | ya ok then let hold off untill the patches below it have merged | |
| 21:57:23 | efried | sean-k-mooney: and tbh I'm not super worried about a merge conflict on that 15th patch. We've got at least a dozen rechecks to go before we get there. | |
| 21:58:01 | sean-k-mooney | ... hopefully not but we do need to kill some of the random failures | |
| 21:58:21 | efried | well hopefully we killed two of them today | |
| 21:58:42 | sean-k-mooney | ya and mriedem killed 1?2? yesterday | |
| 21:58:52 | efried | I took a couple of samples this morning and determined that the Nth patch takes ~N rechecks to merge. | |
| 21:58:54 | sean-k-mooney | firday i cant rememebr | |
| 21:59:10 | sean-k-mooney | :( | |
| 21:59:35 | efried | mriedem: test_fixtures.PostgresqlOpportunisticFixture and test_fixtures.MySQLOpportunisticFixture ? | |
| 22:00:04 | mriedem | efried: was fixed already https://review.opendev.org/#/c/609352/ | |
| 22:01:08 | efried | okay | |
| 22:02:53 | sean-k-mooney | mriedem: by the way i got an error running the sql migate unit tests http://paste.openstack.org/show/776829/ | |
| 22:03:21 | sean-k-mooney | given the rest passed im going to push this soon but shoudl tox not install that | |
| 22:03:29 | sean-k-mooney | i guess it could have moved | |
| 22:04:50 | sean-k-mooney | i assume ibm_db_sa is a sqlalcheme pluging for ibm db backend and its just not installled by tox for some reason | |
| 22:05:30 | mriedem | umm, that should just be skipped | |
| 22:06:12 | mriedem | well, or, | |
| 22:06:36 | mriedem | tox should have pulled it in https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py2.txt#L1 | |
| 22:06:49 | sean-k-mooney | the py27 version allso requries me to install the mysql hearder to compile the client... | |
| 22:06:53 | mriedem | https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py3.txt#L1 | |
| 22:06:54 | sean-k-mooney | i was runnnig py36] | |
| 22:07:19 | sean-k-mooney | which also should install it | |
| 22:07:24 | mriedem | right | |
| 22:07:28 | mriedem | i don't have that problem locally | |
| 22:07:40 | mriedem | are you hitting some pypi mirror? | |
| 22:07:51 | sean-k-mooney | no | |
| 22:08:00 | sean-k-mooney | but i can recreat the env and see what happens | |
| 22:08:41 | sean-k-mooney | the rest of the test pass fine | |
| 22:08:59 | sean-k-mooney | i assume they just use sqlite | |
| 22:10:33 | sean-k-mooney | thats interesting its not in pip freeze in the tox env | |
| 22:11:08 | sean-k-mooney | oh there is not explcit py36 target | |
| 22:11:31 | sean-k-mooney | and it wont look for requirement-py3.txt by defualt | |
| 22:12:13 | sean-k-mooney | i might update the tox file in a sperate patch. | |
| 22:14:47 | efried | mriedem: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/testr_results.html.gz looks like the root cause is some kind of timeout. If we could figure out where, we could probably extend it a bit and make the problem go away...? | |
| 22:15:56 | efried | mmph, it's a socket read, probably a real deadlock. | |
| 22:17:29 | mriedem | sean-k-mooney: do you have this? https://review.opendev.org/#/c/659388/ | |
| 22:17:36 | efried | actually, it's the test timeout | |
| 22:17:52 | mriedem | efried: right, | |
| 22:17:55 | mriedem | that's a red herring | |
| 22:18:03 | sean-k-mooney | mriedem: apparently not but i cloned this on friday | |
| 22:18:13 | mriedem | i think we hit some db failure, switch eventlet context, and then never fail the test outright but it times out | |
| 22:18:32 | mriedem | sean-k-mooney: that migrate patch merged awhile back | |
| 22:18:45 | sean-k-mooney | mriedem: oh i cloned it form github so look like github sync for this is broken | |