| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-09 | |||
| 14:46:31 | sean-k-mooney | mriedem: by the way we talked about actully enfroceing our policy of deprecating and removing virt drivers/backend without ci. have we pushed a patch to mark xen as deprected yet? | |
| 14:47:09 | mriedem | we merged a patch to say xen is experimental but not yet deprecated | |
| 14:47:35 | mriedem | https://github.com/openstack/nova/commit/92c7e0e0b4fcc596e86b493aafb28ee656cb046c#diff-775c4605e52aa43e2ad8c90489216dc8 | |
| 14:48:02 | sean-k-mooney | i still hope to either spend a day and fix the libvirt/lxc driver or push a patch to deprecate it. | |
| 14:48:25 | sean-k-mooney | but ya we will still have them for train in anycase | |
| 14:49:58 | mriedem | if you care about lxc then maybe follow / talk to this guy https://review.opendev.org/#/q/owner:miguel%2540midokura.com+(status:open+OR+status:merged) | |
| 14:50:36 | sean-k-mooney | oh he fixed the bug i was thinking of fixing | |
| 14:50:38 | sean-k-mooney | cool | |
| 14:51:29 | sean-k-mooney | am im not sure how much i care about it but i am somewhat nostalgic in that lxc was second technically container tech i used and it has some usecases still | |
| 14:51:50 | sean-k-mooney | the two thing i was thinking of doing was fixing that bug and trying to add a ci job | |
| 14:52:25 | sean-k-mooney | basically if we still offically support it i would like to ci it but if no one is useing it then i dont mind if its removed. | |
| 14:54:00 | mriedem | i had a non-voting nova-lxc ci job for it long ago and had to blacklist a lot of tempest tests and even then it would fail and the failures would snowball hard on the compute | |
| 14:54:33 | mriedem | nbd mounts and such would get messed up from what i remember | |
| 14:55:08 | mriedem | we don't officially support it btw, | |
| 14:55:16 | mriedem | there is a warning that gets logged on the start of the driver saying it's not tested | |
| 14:55:48 | sean-k-mooney | ah ok well its still there and i think its more useful the uml | |
| 14:56:05 | sean-k-mooney | or some of the other non qmeu/kvm libvirt backends | |
| 14:57:07 | sean-k-mooney | anyway its a low priority | |
| 14:57:55 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.opendev.org/645520 | |
| 14:57:55 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Update AZ admin doc to mention the new way to specify hosts https://review.opendev.org/666767 | |
| 14:59:26 | mriedem | speaking of priorities, how is that numa aware live migration series coming? | |
| 15:04:32 | mriedem | cdent: did i dream this or did you at some point get nova's code coverage job working with unit + functional tests? it only runs unit tests today so we're not reporting all of the coverage. | |
| 15:05:29 | cdent | I think I played with it after I got it working for placement but don't remember if it ever went beyond that. I can play with it again if that seems like a useful thing to do | |
| 15:06:50 | mriedem | i played with it at some point too and seem to remember things hanging or crashing somehow | |
| 15:07:55 | cdent | let's blame eventlet | |
| 15:08:02 | cdent | which might mean things are different now | |
| 15:10:58 | mriedem | right, i think i tried configuring coverage with concurrency=eventlet but that didn't help | |
| 15:11:04 | mriedem | https://coverage.readthedocs.io/en/v4.5.x/config.html#run | |
| 15:14:13 | sean-k-mooney | mriedem: its proably failing due to our use of fixture. one of the oslo libs (oslo.fixtures?) i think calles exec directly with a code object and that breaks my debugger | |
| 15:14:25 | sean-k-mooney | the coverage jobs hooks the debug entry point | |
| 15:14:31 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969 | |
| 15:15:10 | sean-k-mooney | its break on from oslo_utils.fixture import uuidsentinel as uuids | |
| 15:25:14 | mriedem | i'm actually hitting errors with it trying to find the placement fixture | |
| 15:27:06 | sean-k-mooney | yep | |
| 15:27:22 | sean-k-mooney | that was the error i endup having | |
| 15:27:42 | sean-k-mooney | its in the rewrite module when its actully trying to patch stuff | |
| 15:28:49 | sean-k-mooney | the point in the code it raise the exception is actully kind of missleading but the final exception you get makes more sense | |
| 15:31:18 | sean-k-mooney | im not really sure why "from oslo_utils.fixture import uuidsentinel as uuids" has to do with placement but im guessing we are messing with some gloabl state and its not happy | |
| 15:32:09 | cdent | look at the functional job defs: you have to require placement explicitly | |
| 15:32:23 | cdent | i've done that and I'm running with some moderate form of success | |
| 15:32:30 | cdent | but it's not done yet | |
| 15:32:37 | sean-k-mooney | how do you mean | |
| 15:32:41 | mriedem | deps = | |
| 15:32:41 | mriedem | -r{toxinidir}/test-requirements.txt | |
| 15:32:42 | mriedem | git+https://opendev.org/openstack/placement#egg=openstack-placement | |
| 15:32:44 | mriedem | good call | |
| 15:33:38 | sean-k-mooney | i have placmenet in the functional-py36 env | |
| 15:34:04 | cdent | we need sean-k-mooney 's 97 million core machine, because this is so slooooow | |
| 15:34:05 | sean-k-mooney | what did you mean by require placement explcitly | |
| 15:34:29 | mriedem | cdent: do you have the c extension? | |
| 15:34:36 | mriedem | coverage --version in the venv should tell you | |
| 15:35:18 | cdent | yes | |
| 15:36:21 | sean-k-mooney | i can reboot that system into linux and test some things if ye like | |
| 15:36:38 | sean-k-mooney | i was playing games on it last night so its still booted into windows | |
| 15:36:39 | cdent | unit tests in RbdTestCase are blocking and timing out | |
| 15:37:29 | cdent | non-linear unit tests are funny | |
| 15:37:56 | cdent | mriedem: my guess is we can make it work but might need to blacklist a few of the "funny" tests | |
| 15:38:45 | mriedem | yeah, i've got it running now too | |
| 15:38:57 | mriedem | using all 8 cores...hammering | |
| 15:41:43 | cdent | libvirt _get_new_connection is another issue | |
| 15:41:53 | mriedem | i wonder if that one is eventlet | |
| 15:41:59 | mriedem | i haven't run with concurrency=eventlet yet | |
| 15:42:01 | cdent | it is | |
| 15:42:11 | mriedem | likely need this in .coveragerc: | |
| 15:42:14 | mriedem | concurrency = | |
| 15:42:15 | mriedem | eventlet | |
| 15:42:18 | mriedem | greenlet | |
| 15:42:25 | mriedem | in the [run] section | |
| 15:42:25 | cdent | it gets chundered up in eventlet tpool proxy call | |
| 15:42:40 | cdent | my read on the docs was that was just for accounting | |
| 15:42:47 | cdent | (and I added the eventlet bit) | |
| 15:42:53 | cdent | but not greenlet | |
| 15:43:35 | sean-k-mooney | technically its gevent under the covers of both right | |
| 15:44:33 | cdent | mriedem: I'm going to restart mine and let it run while I'm away, will post up any findings later | |
| 15:44:40 | mriedem | yup, same | |
| 15:44:50 | mriedem | i could see this being a sinkhole for my day | |
| 15:46:35 | cdent | indeed | |
| 15:46:36 | cdent | bbl | |
| 15:55:13 | sean-k-mooney | for what its worth i just kicked it off on the 24 core server i have. i have turned off hyper treading for reasons | |
| 15:58:55 | sean-k-mooney | hum the cover job seam to only be running the unit test | |
| 15:59:05 | sean-k-mooney | i dont see where that is set in the tox env | |
| 15:59:30 | sean-k-mooney | 89% is not too bad | |
| 16:00:18 | sean-k-mooney | mriedem: cdent care to point out what i need to chagne to get it to run everything | |
| 16:02:20 | mriedem | mine isn't done but this is what i have http://paste.openstack.org/show/754229/ | |
| 16:02:55 | mriedem | i've mixed some cleanups into that | |
| 16:04:55 | sean-k-mooney | ok its re running now and ya the functional tests are now running too | |
| 16:05:25 | sean-k-mooney | actully ill add teh concurrancy stuff | |
| 16:09:27 | sean-k-mooney | should be done in another miniute or so just grabing a coffee | |
| 16:13:15 | sean-k-mooney | ya im seeing it stall with timeout near the end so we are not mocking something properly although it appears to be in the unit test which is weird | |
| 16:14:01 | sean-k-mooney | maybe we mess up some global state in the functional tests? | |
| 16:15:13 | sean-k-mooney | i am running these under py37 which also could be related but probably not | |
| 16:18:12 | sean-k-mooney | hum it look liek the libvirt issue might be real | |
| 16:19:16 | sean-k-mooney | it looks like some of our mock dont work under python 3.7 and we actully try to connect to libvirt | |
| 16:23:57 | mriedem | gibi: now that you're leaving or already gone for the day, i've gone through your tests https://review.opendev.org/#/c/637955/34 | |
| 16:24:54 | mriedem | sean-k-mooney: i would not be surprised, probably missing the FakeLibvirtFixture somewhere | |
| 16:25:11 | mriedem | because several of the libvirt modules have a global libvirt variable which tests are supposed to stub out to be the fakelibvirt module | |
| 16:28:19 | mriedem | test_cleanup_volumes_pending_resize is mocking the wrong thing... | |
| 16:28:30 | mriedem | RBDVolumeProxy rather than RbdProxy | |
| 16:28:31 | sean-k-mooney | im seeing a whole bunch of placement time out where we call _get_guest_config which goes all the way down to calling _connect in nova.virt.libvirt.host | |
| 16:28:44 | sean-k-mooney | ya i noticed that too | |