Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-09
15:27:22 sean-k-mooney that was the error i endup having
15:27:42 sean-k-mooney its in the rewrite module when its actully trying to patch stuff
15:28:49 sean-k-mooney the point in the code it raise the exception is actully kind of missleading but the final exception you get makes more sense
15:31:18 sean-k-mooney im not really sure why "from oslo_utils.fixture import uuidsentinel as uuids" has to do with placement but im guessing we are messing with some gloabl state and its not happy
15:32:09 cdent look at the functional job defs: you have to require placement explicitly
15:32:23 cdent i've done that and I'm running with some moderate form of success
15:32:30 cdent but it's not done yet
15:32:37 sean-k-mooney how do you mean
15:32:41 mriedem deps =
15:32:41 mriedem -r{toxinidir}/test-requirements.txt
15:32:42 mriedem git+https://opendev.org/openstack/placement#egg=openstack-placement
15:32:44 mriedem good call
15:33:38 sean-k-mooney i have placmenet in the functional-py36 env
15:34:04 cdent we need sean-k-mooney 's 97 million core machine, because this is so slooooow
15:34:05 sean-k-mooney what did you mean by require placement explcitly
15:34:29 mriedem cdent: do you have the c extension?
15:34:36 mriedem coverage --version in the venv should tell you
15:35:18 cdent yes
15:36:21 sean-k-mooney i can reboot that system into linux and test some things if ye like
15:36:38 sean-k-mooney i was playing games on it last night so its still booted into windows
15:36:39 cdent unit tests in RbdTestCase are blocking and timing out
15:37:29 cdent non-linear unit tests are funny
15:37:56 cdent mriedem: my guess is we can make it work but might need to blacklist a few of the "funny" tests
15:38:45 mriedem yeah, i've got it running now too
15:38:57 mriedem using all 8 cores...hammering
15:41:43 cdent libvirt _get_new_connection is another issue
15:41:53 mriedem i wonder if that one is eventlet
15:41:59 mriedem i haven't run with concurrency=eventlet yet
15:42:01 cdent it is
15:42:11 mriedem likely need this in .coveragerc:
15:42:14 mriedem concurrency =
15:42:15 mriedem eventlet
15:42:18 mriedem greenlet
15:42:25 mriedem in the [run] section
15:42:25 cdent it gets chundered up in eventlet tpool proxy call
15:42:40 cdent my read on the docs was that was just for accounting
15:42:47 cdent (and I added the eventlet bit)
15:42:53 cdent but not greenlet
15:43:35 sean-k-mooney technically its gevent under the covers of both right
15:44:33 cdent mriedem: I'm going to restart mine and let it run while I'm away, will post up any findings later
15:44:40 mriedem yup, same
15:44:50 mriedem i could see this being a sinkhole for my day
15:46:35 cdent indeed
15:46:36 cdent bbl
15:55:13 sean-k-mooney for what its worth i just kicked it off on the 24 core server i have. i have turned off hyper treading for reasons
15:58:55 sean-k-mooney hum the cover job seam to only be running the unit test
15:59:05 sean-k-mooney i dont see where that is set in the tox env
15:59:30 sean-k-mooney 89% is not too bad
16:00:18 sean-k-mooney mriedem: cdent care to point out what i need to chagne to get it to run everything
16:02:20 mriedem mine isn't done but this is what i have http://paste.openstack.org/show/754229/
16:02:55 mriedem i've mixed some cleanups into that
16:04:55 sean-k-mooney ok its re running now and ya the functional tests are now running too
16:05:25 sean-k-mooney actully ill add teh concurrancy stuff
16:09:27 sean-k-mooney should be done in another miniute or so just grabing a coffee
16:13:15 sean-k-mooney ya im seeing it stall with timeout near the end so we are not mocking something properly although it appears to be in the unit test which is weird
16:14:01 sean-k-mooney maybe we mess up some global state in the functional tests?
16:15:13 sean-k-mooney i am running these under py37 which also could be related but probably not
16:18:12 sean-k-mooney hum it look liek the libvirt issue might be real
16:19:16 sean-k-mooney it looks like some of our mock dont work under python 3.7 and we actully try to connect to libvirt
16:23:57 mriedem gibi: now that you're leaving or already gone for the day, i've gone through your tests https://review.opendev.org/#/c/637955/34
16:24:54 mriedem sean-k-mooney: i would not be surprised, probably missing the FakeLibvirtFixture somewhere
16:25:11 mriedem because several of the libvirt modules have a global libvirt variable which tests are supposed to stub out to be the fakelibvirt module
16:28:19 mriedem test_cleanup_volumes_pending_resize is mocking the wrong thing...
16:28:30 mriedem RBDVolumeProxy rather than RbdProxy
16:28:31 sean-k-mooney im seeing a whole bunch of placement time out where we call _get_guest_config which goes all the way down to calling _connect in nova.virt.libvirt.host
16:28:44 sean-k-mooney ya i noticed that too
16:29:13 sean-k-mooney im piping the full output to a file and ill through it up on google drive or something
16:29:39 sean-k-mooney its more or less done at this point i think just waiting for all the timeouts to fire
16:30:17 sean-k-mooney all the tox treads are more or less sleeping and then ever so foter a test times out and a bunch suceed
16:33:46 sean-k-mooney test_min_version_file_backed_bad_ram_allocation_ratio apparently modifies teh db witout without using the right fixture
16:34:53 sean-k-mooney so ya there are a whole bunc of not so nice errors showing up
16:35:03 mriedem i don't see _check_file_backed_memory_support hitting the db at all
16:35:30 sean-k-mooney my output could be interleaved
16:37:16 sean-k-mooney i have a couple of things that look like this
16:37:17 sean-k-mooney http://paste.openstack.org/show/754231/
16:38:08 mriedem that would be the libvirt host connection callback stuff that calls _set_host_enabled
16:38:31 sean-k-mooney ya which is happening because our mocking of the connection info is messed up
16:38:59 sean-k-mooney we should be useing the fake libvirt conenction there or just mocking it out entirely sicne it a unit test
16:39:06 mriedem i've had problems in the past with mocks on the libvirt Host stuff leading to weird intermittent failures
16:39:12 mriedem especially with the hasVersion type calls
16:39:35 mriedem like this:
16:39:36 sean-k-mooney ya you mentioned that on my vidoe model patchs
16:39:36 mriedem test_swap_volume_file
16:39:47 mriedem @mock.patch.object(host.Host,
16:39:47 mriedem 'has_min_version', return_value=True)
16:40:00 mriedem rather than mocking the drvr.host.has_min_version directly
16:40:30 mriedem *drvr._host.has_min_version
16:40:40 mriedem there are a billion libvirt driver unit tests so untangling that would be pretty hard
16:41:10 sean-k-mooney well we could do a find an replace
16:41:20 mriedem i don't think it's that easy
16:41:26 sean-k-mooney e.g. if there is one way we know works grep for the broken form
16:41:31 sean-k-mooney proably not
16:41:50 sean-k-mooney but if you know a way that definetly works we can fix the unstable kind as we find them
16:42:12 sean-k-mooney is there a way we could shorten these timeout for the tests by the way?
16:42:18 mriedem yes
16:42:25 sean-k-mooney e.g. globaly somehow
16:42:42 mriedem OS_TEST_TIMEOUT i think
16:42:58 mriedem it's in tox.ini
16:43:10 sean-k-mooney or time out the tests them seleves
16:43:43 sean-k-mooney that might cause issue with some of them but it would be worth trying

Earlier   Later