Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-09
15:35:18 cdent yes
15:36:21 sean-k-mooney i can reboot that system into linux and test some things if ye like
15:36:38 sean-k-mooney i was playing games on it last night so its still booted into windows
15:36:39 cdent unit tests in RbdTestCase are blocking and timing out
15:37:29 cdent non-linear unit tests are funny
15:37:56 cdent mriedem: my guess is we can make it work but might need to blacklist a few of the "funny" tests
15:38:45 mriedem yeah, i've got it running now too
15:38:57 mriedem using all 8 cores...hammering
15:41:43 cdent libvirt _get_new_connection is another issue
15:41:53 mriedem i wonder if that one is eventlet
15:41:59 mriedem i haven't run with concurrency=eventlet yet
15:42:01 cdent it is
15:42:11 mriedem likely need this in .coveragerc:
15:42:14 mriedem concurrency =
15:42:15 mriedem eventlet
15:42:18 mriedem greenlet
15:42:25 mriedem in the [run] section
15:42:25 cdent it gets chundered up in eventlet tpool proxy call
15:42:40 cdent my read on the docs was that was just for accounting
15:42:47 cdent (and I added the eventlet bit)
15:42:53 cdent but not greenlet
15:43:35 sean-k-mooney technically its gevent under the covers of both right
15:44:33 cdent mriedem: I'm going to restart mine and let it run while I'm away, will post up any findings later
15:44:40 mriedem yup, same
15:44:50 mriedem i could see this being a sinkhole for my day
15:46:35 cdent indeed
15:46:36 cdent bbl
15:55:13 sean-k-mooney for what its worth i just kicked it off on the 24 core server i have. i have turned off hyper treading for reasons
15:58:55 sean-k-mooney hum the cover job seam to only be running the unit test
15:59:05 sean-k-mooney i dont see where that is set in the tox env
15:59:30 sean-k-mooney 89% is not too bad
16:00:18 sean-k-mooney mriedem: cdent care to point out what i need to chagne to get it to run everything
16:02:20 mriedem mine isn't done but this is what i have http://paste.openstack.org/show/754229/
16:02:55 mriedem i've mixed some cleanups into that
16:04:55 sean-k-mooney ok its re running now and ya the functional tests are now running too
16:05:25 sean-k-mooney actully ill add teh concurrancy stuff
16:09:27 sean-k-mooney should be done in another miniute or so just grabing a coffee
16:13:15 sean-k-mooney ya im seeing it stall with timeout near the end so we are not mocking something properly although it appears to be in the unit test which is weird
16:14:01 sean-k-mooney maybe we mess up some global state in the functional tests?
16:15:13 sean-k-mooney i am running these under py37 which also could be related but probably not
16:18:12 sean-k-mooney hum it look liek the libvirt issue might be real
16:19:16 sean-k-mooney it looks like some of our mock dont work under python 3.7 and we actully try to connect to libvirt
16:23:57 mriedem gibi: now that you're leaving or already gone for the day, i've gone through your tests https://review.opendev.org/#/c/637955/34
16:24:54 mriedem sean-k-mooney: i would not be surprised, probably missing the FakeLibvirtFixture somewhere
16:25:11 mriedem because several of the libvirt modules have a global libvirt variable which tests are supposed to stub out to be the fakelibvirt module
16:28:19 mriedem test_cleanup_volumes_pending_resize is mocking the wrong thing...
16:28:30 mriedem RBDVolumeProxy rather than RbdProxy
16:28:31 sean-k-mooney im seeing a whole bunch of placement time out where we call _get_guest_config which goes all the way down to calling _connect in nova.virt.libvirt.host
16:28:44 sean-k-mooney ya i noticed that too
16:29:13 sean-k-mooney im piping the full output to a file and ill through it up on google drive or something
16:29:39 sean-k-mooney its more or less done at this point i think just waiting for all the timeouts to fire
16:30:17 sean-k-mooney all the tox treads are more or less sleeping and then ever so foter a test times out and a bunch suceed
16:33:46 sean-k-mooney test_min_version_file_backed_bad_ram_allocation_ratio apparently modifies teh db witout without using the right fixture
16:34:53 sean-k-mooney so ya there are a whole bunc of not so nice errors showing up
16:35:03 mriedem i don't see _check_file_backed_memory_support hitting the db at all
16:35:30 sean-k-mooney my output could be interleaved
16:37:16 sean-k-mooney i have a couple of things that look like this
16:37:17 sean-k-mooney http://paste.openstack.org/show/754231/
16:38:08 mriedem that would be the libvirt host connection callback stuff that calls _set_host_enabled
16:38:31 sean-k-mooney ya which is happening because our mocking of the connection info is messed up
16:38:59 sean-k-mooney we should be useing the fake libvirt conenction there or just mocking it out entirely sicne it a unit test
16:39:06 mriedem i've had problems in the past with mocks on the libvirt Host stuff leading to weird intermittent failures
16:39:12 mriedem especially with the hasVersion type calls
16:39:35 mriedem like this:
16:39:36 sean-k-mooney ya you mentioned that on my vidoe model patchs
16:39:36 mriedem test_swap_volume_file
16:39:47 mriedem @mock.patch.object(host.Host,
16:39:47 mriedem 'has_min_version', return_value=True)
16:40:00 mriedem rather than mocking the drvr.host.has_min_version directly
16:40:30 mriedem *drvr._host.has_min_version
16:40:40 mriedem there are a billion libvirt driver unit tests so untangling that would be pretty hard
16:41:10 sean-k-mooney well we could do a find an replace
16:41:20 mriedem i don't think it's that easy
16:41:26 sean-k-mooney e.g. if there is one way we know works grep for the broken form
16:41:31 sean-k-mooney proably not
16:41:50 sean-k-mooney but if you know a way that definetly works we can fix the unstable kind as we find them
16:42:12 sean-k-mooney is there a way we could shorten these timeout for the tests by the way?
16:42:18 mriedem yes
16:42:25 sean-k-mooney e.g. globaly somehow
16:42:42 mriedem OS_TEST_TIMEOUT i think
16:42:58 mriedem it's in tox.ini
16:43:10 sean-k-mooney or time out the tests them seleves
16:43:43 sean-k-mooney that might cause issue with some of them but it would be worth trying
16:44:02 mriedem well it would make this monster coverage thing fail faster yes
16:44:13 mriedem dropping OS_TEST_TIMEOUT=60 would help
16:44:14 sean-k-mooney i was thinking more doing set.flags in the base test case
16:44:29 mriedem nova.conf doesn't have anything to do with test execution timeout...
16:44:47 sean-k-mooney i was wondering if we could modify libvirt timeouts in nova.conf
16:45:02 mriedem nope
16:45:11 openstackgerrit Merged openstack/nova master: Implement update_provider_tree https://review.opendev.org/667417
16:45:20 mriedem configuring libvirt is kind of a libvirt.conf thing right?
16:45:50 sean-k-mooney ya i guess i think these timeout are comeing form teh python binding or the c lib rather then libvirt
16:47:03 sean-k-mooney ill give it another 15mins and if its still timing out ten ill run with OS_TEST_TIMEOUT=30 currently its timing out after 160s
16:50:40 sean-k-mooney actully i might make that OS_TEST_TIMEOUT=8 im not seeing any test over about 4.5 seconds
16:51:09 sean-k-mooney oh there is 1 at 13s
16:55:52 dansmith artom: I assume you're working on feedback on the resize revert patches yeah?
17:09:49 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: coverage job changes https://review.opendev.org/669912
17:12:50 openstackgerrit Merged openstack/nova master: xenapi: implement update_provider_tree https://review.opendev.org/667435
17:12:58 openstackgerrit Merged openstack/nova master: Deprecate non-update_provider_tree compat code https://review.opendev.org/667442
17:13:06 Sundar gibi, efried: Re. https://review.opendev.org/#/c/657464/, there was some discussion over naming, which got resolved in the review of the spec https://review.opendev.org/#/c/603955/12/specs/train/approved/nova-cyborg-interaction.rst@307 . Can we resume the review of 657464?

Earlier   Later