| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-04 | |||
| 18:21:06 | jaypipes | cdent: even doing three or four of the bottom ones would be useful. | |
| 18:21:13 | jaypipes | they're pretty smallish patches. | |
| 18:22:07 | jaypipes | cdent: fwiw, the HEAD of that series gets the resource_provider.py file to the point where I can do the root_provider_id/parent_provider_id fields (instead of the root_provider_uuid/parent_provider_uuid fields) that dansmith has asked for on the nested series. | |
| 18:22:33 | jaypipes | that dansmith has requested and that I agree with, btw | |
| 18:23:35 | mriedem | that reminds me, | |
| 18:23:47 | mriedem | andreykurilin: is there a devstack plugin for rally to make it easy to get an env setup? | |
| 18:24:01 | mriedem | oooo there is https://github.com/openstack/rally/tree/master/devstack | |
| 18:24:04 | mriedem | does it work? | |
| 18:24:05 | andreykurilin | mriedem: sure | |
| 18:24:43 | mriedem | what is the default backend when doing that? like do i need to also have ceilometer enabled? | |
| 18:24:44 | andreykurilin | mriedem: yes. it is used in all our gates | |
| 18:25:29 | andreykurilin | hm... ceilometer? rally has nothing related to ceilometer except test plugins for it. the default database backend is sqllite | |
| 18:25:57 | mriedem | ok yeah i was wondering about a db, i don't know much about rally, and i thought there was something to do with configuring it to send metrics collection to ceilometer | |
| 18:26:35 | mriedem | i'm looking for something simple to setup so i can compare test runs before and after some nova patch is applied | |
| 18:26:48 | mriedem | or, comparing scheduling times between ocata and pike | |
| 18:26:49 | mriedem | for example | |
| 18:28:19 | mriedem | i guess i should shut up and just read https://rally.readthedocs.io/en/latest/quick_start/tutorial.html | |
| 18:29:12 | andreykurilin | mriedem: so rally is quite simple and doesn't require any additional services to be installed. All the results are stored in the sql-dabase. One rally installation can check different clusters. The latest clients work perfectly with older releases of openstack. For example we have regular job which launch the scenarios at mitaka release | |
| 18:29:39 | mriedem | ok, i'm mostly interested in scheduling performance between ocata and pike, | |
| 18:30:01 | mriedem | i only have a single node devstack, so i'd have to get creative with mocking out computes | |
| 18:31:08 | mriedem | i know devstack has some flags for mocking out fake compute services when using the fake virt driver, but not sure if that works anymore with the tiered conductor stuff in devstack | |
| 18:31:16 | mriedem | i could run it in singleconductor mode to workaround that | |
| 18:36:04 | openstackgerrit | Merged openstack/nova master: Log consumer uuid when retrying claims in the scheduler https://review.openstack.org/507705 | |
| 18:36:38 | openstackgerrit | Merged openstack/nova master: [placement] gabbi tests for shared custom resource class https://review.openstack.org/485209 | |
| 18:38:38 | andreykurilin | mriedem: btw, I know some guys from huawei who are working on inner performance testing of openstack(as far as I know). I'll contact them about the hardware for such testing as you need | |
| 18:39:13 | mriedem | andreykurilin: are they in toronto? | |
| 18:39:17 | mriedem | hongbin's team? | |
| 18:39:44 | andreykurilin | mriedem: they are from China | |
| 18:39:48 | mriedem | oh | |
| 18:40:10 | mriedem | well, if you can get answers on internal openstack perf testing at huawei faster than i can, kudos to you sir | |
| 18:41:10 | andreykurilin | mriedem: heh. At least I'll try, but it will happen tomorrow, since they are slipping now | |
| 18:41:34 | mriedem | it's a chinese national holiday this week too | |
| 18:41:48 | openstackgerrit | priyaduggirala proposed openstack/nova master: Rename parameters in call() of nova/image/glance.py https://review.openstack.org/508533 | |
| 18:42:00 | melwitt | clarkb: sorry, was on a call. what did you mean by get the order sorted out between the uca package being installed and the pip package being installed? | |
| 18:43:03 | melwitt | mriedem: yeah, there's no ubuntu bug I don't think, because I think this is a unique problem to having devstack install PasteDeploy via pip and then the installation of ceph packages in the devstack plugin installing the uca packages | |
| 18:44:04 | melwitt | I thought the same, that I could probably do something like try uninstalling the uca python-pastedeploy package in the ceph devstack plugin and see if that would work | |
| 18:44:15 | clarkb | melwitt: basically make sure that apt-get happens first, then pip install, then keystone start | |
| 18:45:18 | melwitt | clarkb: ah, okay thanks. I'll see if that's possible in devstack plugin land, if there's a hook point for plugins that occurs before the pip installs that devstack does | |
| 18:46:04 | andreykurilin | mriedem: oh... yes, I forgot about it | |
| 18:46:09 | melwitt | because the devstack ceph plugin is doing the apt-get installs | |
| 18:48:17 | mriedem | melwitt: should be in this phase https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/plugin.sh#L12 | |
| 18:48:41 | mriedem | check the logs and see if that happens before or after the pip install happens | |
| 18:49:12 | melwitt | hm, yeah ... so it should be doing the right thing already. looking | |
| 18:54:07 | melwitt | says "Requirement already satisfied: PasteDeploy===1.5.2 in /usr/lib/python2.7/dist-packages" which means it skipped the pip install | |
| 18:54:13 | melwitt | (I think) | |
| 18:54:17 | melwitt | so how is keystone not finding it | |
| 18:54:24 | openstackgerrit | Merged openstack/nova master: fix nova accepting invalid availability zone name with ':' https://review.openstack.org/490722 | |
| 18:55:01 | melwitt | I had thought that because PasteDeploy was in the pip freeze output, that it had been pip installed too http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/pip2-freeze.txt.gz | |
| 18:55:17 | melwitt | but I guess that's not necessarily true | |
| 18:55:18 | mriedem | not necessarily | |
| 18:55:32 | mriedem | i remember yum installed packages show up in pip freeze output too | |
| 18:55:43 | mriedem | you just generally aren't using pip with rpms because things get wonky | |
| 18:56:03 | melwitt | yeah, huh. | |
| 18:57:25 | mriedem | right so it's setup via deb first http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/devstacklog.txt.gz | |
| 18:57:39 | mriedem | and then pip ignores it http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/devstacklog.txt.gz#_2017-10-04_01_50_59_811 | |
| 18:57:41 | clarkb | hrm pastedeploy 1.5.2 is latest | |
| 18:57:53 | clarkb | so is the ubuntu package just broken and not including all the bits? | |
| 18:57:58 | mriedem | oops first link is http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/devstacklog.txt.gz#_2017-10-04_01_49_51_453 | |
| 18:58:13 | melwitt | do you think it has anything to do with this? wsgi-file = /usr/local/bin/keystone-wsgi-public http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/etc/keystone/keystone-uwsgi-public.ini.txt.gz | |
| 18:58:16 | mriedem | we are using pike uca now | |
| 18:58:17 | mriedem | as of last week | |
| 18:58:41 | melwitt | is the /usr/local/ there causing keystone to only look under /usr/local for paste deploy? | |
| 18:59:51 | melwitt | clarkb: hm, maybe. I hadn't considered the package could be broken | |
| 19:00:10 | cfriesen | is there a way to specify running a single functional test class? | |
| 19:00:27 | clarkb | cfriesen: tox -e functional -- ClassNameHere ? | |
| 19:00:32 | melwitt | mriedem: yeah, the ceph-mgr started pulling in python-pastedeploy starting in the pike uca | |
| 19:00:35 | clarkb | that actually takes a regex (or should) | |
| 19:00:59 | cfriesen | clarkb: will try it...I had done that with the full dotted path to the class and it didn't like it. | |
| 19:01:15 | artom | cfriesen, clarkb's right, it should be a regex | |
| 19:01:22 | artom | So just ClassName should do it | |
| 19:01:30 | artom | Or even assName, because it matches as wlel | |
| 19:01:38 | mriedem | sorry make that monday https://review.openstack.org/#/c/505674/ | |
| 19:02:08 | cfriesen | artom: clarkb: nope. I ran "tox -e functional -- AggregateMigrationTestCase" and it started doing tests from nova.tests.functional.api.openstack.placement.test_placement_api | |
| 19:02:32 | melwitt | cfriesen: do you have an old checkout? there was some issues around the regex functionality that got fixed while we were at the PTG | |
| 19:02:44 | artom | cfriesen, hrmm - try without the --? | |
| 19:02:45 | mriedem | cfriesen: rebuild | |
| 19:02:48 | cfriesen | melwitt: entirely possible | |
| 19:02:49 | mriedem | tox -r -e | |
| 19:03:48 | melwitt | cfriesen: https://github.com/openstack/nova/commits/master/tox.ini I'd make sure you have the latest version of tox.ini | |
| 19:08:26 | cfriesen | combination of a fresh pull and tox -r seems to have done the trick | |
| 19:08:27 | cfriesen | thansk | |
| 19:14:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: use already loaded BDM in instance. |
|
| 19:14:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: use already loaded BDM in instance.create https://review.openstack.org/483969 | |
| 19:14:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: use already loaded BDM in instance. |
|
| 19:18:06 | krypto | hello all i have a strange problem with Mitaka release ,on one compute if i enable debug to true nova-compute is restarting continously with error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 335: ordinal not in range(128)" with out debug everything is working as expected ,any idea? | |
| 19:20:00 | mriedem | krypto: there is likely some debug-only log message hitting something that's not encoded properly | |
| 19:20:08 | mriedem | i probably know what / where it is too | |
| 19:20:34 | mriedem | https://bugs.launchpad.net/nova/+bug/1624277 is similar | |
| 19:20:35 | openstack | Launchpad bug 1624277 in OpenStack Compute (nova) mitaka "nova-scheduler: UnicodeDecodeError in host aggregates handling" [Medium,Confirmed] - Assigned to Nazeema Begum (nazeema123) | |
| 19:21:04 | mriedem | krypto: look at this fix in oslo.log https://review.openstack.org/#/c/371582/ which was never released in the mitaka series | |
| 19:21:19 | cfriesen | mriedem: I tweaked the code slightly to test the host aggregate migration, and confirmed that we don't soft-delete the entries in table 'aggregate_hosts'. Diff is here, with functional test output at bottom: http://paste.openstack.org/show/622702/ This shouldn't affect normal code, but it messed up one of our audits that verified DB consistency. | |
| 19:21:24 | mriedem | looks like oslo.log 3.9.0 has that fix | |
| 19:22:57 | mriedem | krypto: i've also seen one in nova-compute when logging the vif | |
| 19:23:01 | mriedem | i can't find the bug for it though | |
| 19:23:07 | krypto | i see thank you mriedem ,thanks alot i didnt know that turning debug=true could bring down nova-compute :) not sure whats wrong with my compute | |
| 19:24:28 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add alternate hosts https://review.openstack.org/486215 | |
| 19:24:29 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add Selection objects https://review.openstack.org/499239 | |
| 19:24:30 | openstackgerrit | Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854 | |
| 19:25:24 | mriedem | krypto: you would need to capture the stacktrace from the n-cpu logs and put those in a paste to know where the actual problem needs to be fixed, or if it's already fixed and you can cherry pick that fix to your mitaka env, since mitaka is eol upstream | |
| 19:26:02 | mriedem | it might not be in nova, it could be in glanceclient, i see fixed bugs for the same thing in glanceclient from several years ago | |