| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-09 | |||
| 15:37:29 | cdent | non-linear unit tests are funny | |
| 15:37:56 | cdent | mriedem: my guess is we can make it work but might need to blacklist a few of the "funny" tests | |
| 15:38:45 | mriedem | yeah, i've got it running now too | |
| 15:38:57 | mriedem | using all 8 cores...hammering | |
| 15:41:43 | cdent | libvirt _get_new_connection is another issue | |
| 15:41:53 | mriedem | i wonder if that one is eventlet | |
| 15:41:59 | mriedem | i haven't run with concurrency=eventlet yet | |
| 15:42:01 | cdent | it is | |
| 15:42:11 | mriedem | likely need this in .coveragerc: | |
| 15:42:14 | mriedem | concurrency = | |
| 15:42:15 | mriedem | eventlet | |
| 15:42:18 | mriedem | greenlet | |
| 15:42:25 | mriedem | in the [run] section | |
| 15:42:25 | cdent | it gets chundered up in eventlet tpool proxy call | |
| 15:42:40 | cdent | my read on the docs was that was just for accounting | |
| 15:42:47 | cdent | (and I added the eventlet bit) | |
| 15:42:53 | cdent | but not greenlet | |
| 15:43:35 | sean-k-mooney | technically its gevent under the covers of both right | |
| 15:44:33 | cdent | mriedem: I'm going to restart mine and let it run while I'm away, will post up any findings later | |
| 15:44:40 | mriedem | yup, same | |
| 15:44:50 | mriedem | i could see this being a sinkhole for my day | |
| 15:46:35 | cdent | indeed | |
| 15:46:36 | cdent | bbl | |
| 15:55:13 | sean-k-mooney | for what its worth i just kicked it off on the 24 core server i have. i have turned off hyper treading for reasons | |
| 15:58:55 | sean-k-mooney | hum the cover job seam to only be running the unit test | |
| 15:59:05 | sean-k-mooney | i dont see where that is set in the tox env | |
| 15:59:30 | sean-k-mooney | 89% is not too bad | |
| 16:00:18 | sean-k-mooney | mriedem: cdent care to point out what i need to chagne to get it to run everything | |
| 16:02:20 | mriedem | mine isn't done but this is what i have http://paste.openstack.org/show/754229/ | |
| 16:02:55 | mriedem | i've mixed some cleanups into that | |
| 16:04:55 | sean-k-mooney | ok its re running now and ya the functional tests are now running too | |
| 16:05:25 | sean-k-mooney | actully ill add teh concurrancy stuff | |
| 16:09:27 | sean-k-mooney | should be done in another miniute or so just grabing a coffee | |
| 16:13:15 | sean-k-mooney | ya im seeing it stall with timeout near the end so we are not mocking something properly although it appears to be in the unit test which is weird | |
| 16:14:01 | sean-k-mooney | maybe we mess up some global state in the functional tests? | |
| 16:15:13 | sean-k-mooney | i am running these under py37 which also could be related but probably not | |
| 16:18:12 | sean-k-mooney | hum it look liek the libvirt issue might be real | |
| 16:19:16 | sean-k-mooney | it looks like some of our mock dont work under python 3.7 and we actully try to connect to libvirt | |
| 16:23:57 | mriedem | gibi: now that you're leaving or already gone for the day, i've gone through your tests https://review.opendev.org/#/c/637955/34 | |
| 16:24:54 | mriedem | sean-k-mooney: i would not be surprised, probably missing the FakeLibvirtFixture somewhere | |
| 16:25:11 | mriedem | because several of the libvirt modules have a global libvirt variable which tests are supposed to stub out to be the fakelibvirt module | |
| 16:28:19 | mriedem | test_cleanup_volumes_pending_resize is mocking the wrong thing... | |
| 16:28:30 | mriedem | RBDVolumeProxy rather than RbdProxy | |
| 16:28:31 | sean-k-mooney | im seeing a whole bunch of placement time out where we call _get_guest_config which goes all the way down to calling _connect in nova.virt.libvirt.host | |
| 16:28:44 | sean-k-mooney | ya i noticed that too | |
| 16:29:13 | sean-k-mooney | im piping the full output to a file and ill through it up on google drive or something | |
| 16:29:39 | sean-k-mooney | its more or less done at this point i think just waiting for all the timeouts to fire | |
| 16:30:17 | sean-k-mooney | all the tox treads are more or less sleeping and then ever so foter a test times out and a bunch suceed | |
| 16:33:46 | sean-k-mooney | test_min_version_file_backed_bad_ram_allocation_ratio apparently modifies teh db witout without using the right fixture | |
| 16:34:53 | sean-k-mooney | so ya there are a whole bunc of not so nice errors showing up | |
| 16:35:03 | mriedem | i don't see _check_file_backed_memory_support hitting the db at all | |
| 16:35:30 | sean-k-mooney | my output could be interleaved | |
| 16:37:16 | sean-k-mooney | i have a couple of things that look like this | |
| 16:37:17 | sean-k-mooney | http://paste.openstack.org/show/754231/ | |
| 16:38:08 | mriedem | that would be the libvirt host connection callback stuff that calls _set_host_enabled | |
| 16:38:31 | sean-k-mooney | ya which is happening because our mocking of the connection info is messed up | |
| 16:38:59 | sean-k-mooney | we should be useing the fake libvirt conenction there or just mocking it out entirely sicne it a unit test | |
| 16:39:06 | mriedem | i've had problems in the past with mocks on the libvirt Host stuff leading to weird intermittent failures | |
| 16:39:12 | mriedem | especially with the hasVersion type calls | |
| 16:39:35 | mriedem | like this: | |
| 16:39:36 | sean-k-mooney | ya you mentioned that on my vidoe model patchs | |
| 16:39:36 | mriedem | test_swap_volume_file | |
| 16:39:47 | mriedem | @mock.patch.object(host.Host, | |
| 16:39:47 | mriedem | 'has_min_version', return_value=True) | |
| 16:40:00 | mriedem | rather than mocking the drvr.host.has_min_version directly | |
| 16:40:30 | mriedem | *drvr._host.has_min_version | |
| 16:40:40 | mriedem | there are a billion libvirt driver unit tests so untangling that would be pretty hard | |
| 16:41:10 | sean-k-mooney | well we could do a find an replace | |
| 16:41:20 | mriedem | i don't think it's that easy | |
| 16:41:26 | sean-k-mooney | e.g. if there is one way we know works grep for the broken form | |
| 16:41:31 | sean-k-mooney | proably not | |
| 16:41:50 | sean-k-mooney | but if you know a way that definetly works we can fix the unstable kind as we find them | |
| 16:42:12 | sean-k-mooney | is there a way we could shorten these timeout for the tests by the way? | |
| 16:42:18 | mriedem | yes | |
| 16:42:25 | sean-k-mooney | e.g. globaly somehow | |
| 16:42:42 | mriedem | OS_TEST_TIMEOUT i think | |
| 16:42:58 | mriedem | it's in tox.ini | |
| 16:43:10 | sean-k-mooney | or time out the tests them seleves | |
| 16:43:43 | sean-k-mooney | that might cause issue with some of them but it would be worth trying | |
| 16:44:02 | mriedem | well it would make this monster coverage thing fail faster yes | |
| 16:44:13 | mriedem | dropping OS_TEST_TIMEOUT=60 would help | |
| 16:44:14 | sean-k-mooney | i was thinking more doing set.flags in the base test case | |
| 16:44:29 | mriedem | nova.conf doesn't have anything to do with test execution timeout... | |
| 16:44:47 | sean-k-mooney | i was wondering if we could modify libvirt timeouts in nova.conf | |
| 16:45:02 | mriedem | nope | |
| 16:45:11 | openstackgerrit | Merged openstack/nova master: Implement update_provider_tree https://review.opendev.org/667417 | |
| 16:45:20 | mriedem | configuring libvirt is kind of a libvirt.conf thing right? | |
| 16:45:50 | sean-k-mooney | ya i guess i think these timeout are comeing form teh python binding or the c lib rather then libvirt | |
| 16:47:03 | sean-k-mooney | ill give it another 15mins and if its still timing out ten ill run with OS_TEST_TIMEOUT=30 currently its timing out after 160s | |
| 16:50:40 | sean-k-mooney | actully i might make that OS_TEST_TIMEOUT=8 im not seeing any test over about 4.5 seconds | |
| 16:51:09 | sean-k-mooney | oh there is 1 at 13s | |
| 16:55:52 | dansmith | artom: I assume you're working on feedback on the resize revert patches yeah? | |
| 17:09:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: coverage job changes https://review.opendev.org/669912 | |
| 17:12:50 | openstackgerrit | Merged openstack/nova master: xenapi: implement update_provider_tree https://review.opendev.org/667435 | |
| 17:12:58 | openstackgerrit | Merged openstack/nova master: Deprecate non-update_provider_tree compat code https://review.opendev.org/667442 | |
| 17:13:06 | Sundar | gibi, efried: Re. https://review.opendev.org/#/c/657464/, there was some discussion over naming, which got resolved in the review of the spec https://review.opendev.org/#/c/603955/12/specs/train/approved/nova-cyborg-interaction.rst@307 . Can we resume the review of 657464? | |
| 17:13:07 | openstackgerrit | Merged openstack/nova master: Update COMPUTE_STATUS_DISABLED from set_host_enabled compute call https://review.opendev.org/668743 | |
| 17:14:02 | efried | Sundar: As far as I'm concerned, absolutely. Not much I can do though, since I own the patch :) | |
| 17:31:49 | openstackgerrit | Merged openstack/nova master: Add placement request pre-filter compute_status_filter https://review.opendev.org/668752 | |
| 17:31:56 | openstackgerrit | Merged openstack/nova master: Refactor HostAPI.service_update https://review.opendev.org/668756 | |