Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-01
21:32:29 mriedem edleafe: it's true, it reminds me of things i got in trouble for in school
21:32:47 mriedem melwitt: likely means we aren't using six.text_type somewhere
21:33:04 mriedem melwitt: link and stacktrace?
21:33:22 mriedem like, LOG.warning('something %s' % unicode) will do that
21:34:51 melwitt mriedem: http://logs.openstack.org/39/446239/5/check/gate-tempest-dsvm-cells-ubuntu-xenial/7061c58/logs/screen-n-cell-region.txt.gz?#_May_01_20_22_15_324264
21:35:37 mriedem hmm https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1011
21:35:47 cdent edleafe: I guess you didn't run all the functional tests before pushing that can_host change. I can no longer trust you!
21:36:00 mriedem melwitt: i've seen something like that in n-cpu where we log an instance with plugging vifs
21:36:11 mriedem it dumps the entire instance and there must be something in there that has unicode in it
21:36:23 melwitt hm, okay. thanks
21:36:26 cdent melwitt: saw some melwitt I've just started seeing that too
21:36:31 mriedem melwitt: https://bugs.launchpad.net/nova/+bug/1580728
21:36:31 openstack Launchpad bug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged]
21:36:33 melwitt I don't think I did anything to cause that but maybe I did
21:37:41 mriedem in ^ it's unicode in the instance display name
21:37:43 cdent http://logs.openstack.org/13/459413/5/check/gate-tempest-dsvm-cells-ubuntu-xenial/da51ac2/console.html#_2017-05-01_20_33_50_887866
21:37:45 melwitt cdent: ack. good it's not my patch maybe probably
21:38:15 melwitt that's weird, I wonder what made this start happening all of a sudden. implies something recently changed to unicode
21:38:55 mriedem check logstash
21:40:40 mriedem i only see 2 hits in logstash
21:41:11 melwitt doing. I also found earlier today that tempest.api.volume.admin.test_volume_quota_classes.VolumeQuotaClassesTest.test_update_default_quota fails intermittently. I'm guessing it's something to do with parallel tests racing
21:42:55 cdent mriedem: what did you search on?
21:43:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI._find_service https://review.openstack.org/461519
21:43:11 mtreinish melwitt: heh, the default quota tests are an inherent race condition
21:43:11 mriedem cdent: i deleted it :) sec
21:43:36 melwitt message:"UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 413: ordinal not in range(128)"
21:43:38 mriedem melwitt: the default quota compute api tests in tempest are locked to run serially
21:43:46 cdent cuz i've got 259 in the last 6 hours on message:"decode byte 0xc3 in position" which is clearly too general, but...
21:43:47 melwitt is what I searched on and see the same, only two hits
21:43:54 mtreinish but I think we set it to have sane defaults so we don't ever decrease the default quotas beyond where they start (which would be a big problem)
21:44:04 mriedem message:"UnicodeDecodeError" AND message:"instance_update_at_top" AND tags:"screen-n-cell-region.txt"
21:45:58 edleafe cdent: I was meaning to mark it WIP and forgot. I knew it wasn't fully tested.
21:46:15 melwitt mtreinish, mriedem: so they're set to run serially ... the thing I see is e.g. (u'snapshots_tempest-QosSpecsTestJSON-volume-type-1219921907', -1) not in [(u'gigabytes', 1001), (u'backups', 11), (u'snapshots_lvmdriver-1', -1), (u'volumes_lvmdriver-1', -1), (u'volumes', 11), (u'per_volume_gigabytes', -1), (u'backup_gigabytes', 1001), (u'gigabytes_lvmdriver-1', -1), (u'snapshots', 11)]
21:46:21 cdent edleafe: my bad really, I was being far too optimistic
21:47:29 melwitt mtreinish, mriedem: as though some other test had added a quota default and it gets picked up unexpectedly in another
21:48:06 cdent g'night all
21:50:39 mtreinish melwitt: do you have a link to that failure?
21:51:03 melwitt mtreinish: yeah, sorry I didn't include it http://logs.openstack.org/39/446239/5/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0b6acd6/logs/testr_results.html.gz
21:57:11 mtreinish melwitt: that's really weird. It looks like tempest sends a PUT with the new default quota entries, cinder returns 200, and the tempest does a GET to list the default quotas and the stuff it just added isn't there
21:58:41 mtreinish melwitt: oh, I think you might be right about a race
21:58:44 mriedem dansmith: while looking at this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4508 for using uuid rather than id, i realized this could have been a direct lookup if we stored the compute node uuid on the host mapping, but we only store the ComputeNode.host field...
21:59:15 melwitt mtreinish: hm, what I thought is that update_body is returning an additional unexpected item that was added by a different test
22:02:37 mtreinish melwitt: the test does a GET to get all the default quotas, loops over that list increases each quota there by 1 with a big update, then does another list and loops over them to see everything was updated
22:03:26 mtreinish the failure looks like another test class is making a default quota set and cleaning up after itself between the first get and after the update
22:03:48 mtreinish and tempest is failing on the verification that everything it asked to be updated is
22:03:53 mtreinish because a bunch of stuff was deleted
22:04:20 mtreinish at least that's what it looks like
22:04:41 melwitt okay. I'm looking at the code too but highly possible I've not understood it :)
22:05:26 mtreinish I'm having a hard time figuring out where the other default quotas are coming from though. It says volumes_tempest-QosSpecsTestJSON-volume-type-1219921907 which should mean the QosSpecsTestJSON class created it
22:05:34 mtreinish but I don't see where that happens in the code
22:05:39 mtreinish but I could just not be seeing it
22:07:53 melwitt I was focusing on the part where it's asserting (new, old) and I thought it mean there was something unexpected in "new", the return from the PUT
22:08:49 melwitt maybe I got that backward because I'm not familiar with the ContainsAll matcher
22:11:29 mtreinish melwitt: iirc the contains all matcher is basically a for loop that does: for x in arg[0]: self.assertIn(x, arg[1])
22:11:36 mtreinish but I'd have to dig up the code to be sure
22:12:11 melwitt agreed that it should mean the QosSpecsTestJSON created it but all I see is e.g. "cls.create_test_qos_specs(...)" and I dunno what all that entails. if it could possibly include creating a quota default
22:12:33 mtreinish melwitt: but looking at the tempest log: http://logs.openstack.org/39/446239/5/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0b6acd6/logs/tempest.txt.gz#_2017-04-29_03_26_53_000
22:12:54 mtreinish the teardownclass is deleting volume types around the same time the default quotas tests are doing their thing
22:13:05 mtreinish so I think we want to add an external lock between the classes to protect against that
22:16:15 mtreinish melwitt: can you open a bug for this so we can track it with e-r? I can work on a fix for it. It shouldn't be too hard
22:16:47 melwitt mtreinish: sure
22:17:14 mtreinish unless you wanted to work on the patch :)
22:20:38 melwitt mtreinish: no thanks :)
22:22:51 dansmith mriedem: you mean we could have known which cells to check instead of iterating over all of them?
22:23:16 dansmith mriedem: the id isn't unique, so it only buys us not iterating over all the cells in the case where the compute id happens to be uniquer
22:23:28 dansmith but on average, for similar-sized cells, you'll have nearly 100% overlap
22:25:02 openstackgerrit Ed Leafe proposed openstack/nova master: Remove the can_host column https://review.openstack.org/461494
22:28:57 melwitt mtreinish: https://bugs.launchpad.net/tempest/+bug/1687488
22:28:57 openstack Launchpad bug 1687488 in tempest "VolumeQuotaClassesTest.test_update_default_quota fails intermittently" [Undecided,New]
22:29:09 mtreinish melwitt: thanks
22:31:53 mriedem dansmith: right i meant we could just take the compute node uuid to look up a host mapping to find out the cell it's in, but right now we'll have to iterate the cells (like we do if given the primary key id)
22:32:11 mriedem not a huge deal, just something i thought of when seeing your comment in that host api method
22:32:35 dansmith you mean once we have the uuid from the api, then we can use the uuid for the lookup, right?
22:32:42 mriedem yes
22:32:50 dansmith or do you mean if we had uuid on the hostmapping we could avoid the cell scan?
22:32:54 mriedem that's what i'm adding to HostAPI.compute_node_get now, but we still have to iterate the cells
22:32:58 mriedem correct
22:33:22 mriedem dansmith: https://gist.github.com/mriedem/013d1bcb9fd547a61adcccb99b9f0eae
22:33:29 dansmith okay, since hostmapping is 1:1 to compute node that should be okay, I think.. but we don't want to do the same for service I think
22:33:43 mriedem we don't have a ServiceMapping in the api db
22:33:46 mriedem so we can't
22:34:06 dansmith what I meant was, we can't just tack the service uuid onto hostmapping too
22:34:15 mriedem oh, ok yeah agree
22:34:27 mriedem if we did add uuid to hostmappings, we'd have to have an online data migration
22:34:32 mriedem so it becomes less fun
22:34:36 dansmith yeah ;)
22:35:15 mriedem i bet at some point this will be one of those things we wish we had though, like bdm.uuid
22:35:23 mriedem but if we really need it at some point, we could add it later
22:41:00 openstackgerrit Eric Fried proposed openstack/nova master: nova.utils.get_service_url() https://review.openstack.org/458257
23:22:25 efried mordred jaypipes johnthetubaguy FYI ^^ is failing Jenkins because it Depends-On a keystoneauth patch which apparently isn't get picked up. (Are there only certain projects Depends-On works with?)
23:22:43 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532
23:23:51 mriedem efried: depends-on doesn't work if the dependent change is in a library or client,
23:23:58 mriedem unless you have a job that uses LIBS_FROM_GIT
23:24:01 mriedem in devstack
23:24:03 efried mriedem Okay, good to know.
23:24:22 efried FWIW, it live-tests okay ;-)
23:27:05 cfriesen_ for testing purposes is there a helper function to compare dicts where one outputs keys/values as strings and the other as unicode?
23:29:24 efried cfriesen_ map()?
23:30:36 efried Or this? http://stackoverflow.com/questions/1254454/fastest-way-to-convert-a-dicts-keys-values-from-unicode-to-str
23:31:57 cfriesen_ efried: I was wondering if nova (or oslo) already had something

Earlier   Later