Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-01
19:53:09 openstackgerrit Ed Leafe proposed openstack/nova master: Remove the can_host column https://review.openstack.org/461494
19:56:21 melwitt jaypipes: fyi I pushed an update to the quotas series attempting to address your comments https://review.openstack.org/#/c/446239
19:56:29 jaypipes melwitt: yup, reviewing it now :)
19:56:37 jaypipes while waiting for functional tests to complete..
19:57:01 melwitt jaypipes: cool, thanks :)
20:02:24 ildikov jaypipes: around for a meeting on #openstack-meeting-3? :)
20:03:03 jaypipes ildikov: I'm sorry, Ildiko, I'm right in the middle of debugging a nasty little feature and really need to focus on it.
20:03:06 jaypipes sorry :(
20:03:16 jaypipes ildikov: will read back from channel later.
20:03:29 ildikov jaypipes: ok, let's catch up later then
20:04:52 jaypipes thx ildikov
21:10:04 openstackgerrit Eric Fried proposed openstack/nova master: nova.utils.get_service_url() https://review.openstack.org/458257
21:16:54 mriedem oh boy, so my daughter got home and there is a note from the teacher in her backpack about inappropriate behavior,
21:17:09 mriedem e.g. a worksheet of sight words to fille out, every entry has the word "butt" in it
21:17:34 mriedem she at least spelled it correctly...
21:23:48 cdent \o/
21:25:39 dansmith mriedem: sight words?
21:26:08 mriedem dansmith: yeah, like simple single-syllable words that they should recognize immediately
21:26:12 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Debug race in delete-while-booting test https://review.openstack.org/461513
21:26:14 mriedem they, here, were, are, them
21:26:22 mriedem butt
21:26:25 dansmith mriedem: and butt is inappropriate?
21:26:37 dansmith maybe the teacher should read "everybody poops"
21:26:39 imacdonn I'd recognise a butt anywhere!
21:26:40 mriedem well, not only that, but the picture of a butt pooping
21:26:59 dansmith mriedem: I dunno, I think I'd go argue with the teacher
21:28:33 cdent tbachman++
21:28:59 dansmith tbachman: that would be inappropriate (apparently)
21:30:39 edleafe mriedem: the apple doesn't fall far...
21:32:28 melwitt wat is this "UnicodeDecodeError \'ascii\' codec can\'t decode byte 0xc3 in position 3575: ordinal not in range(128)" coming from the cells v1 tempest job
21:32:29 mriedem edleafe: it's true, it reminds me of things i got in trouble for in school
21:32:47 mriedem melwitt: likely means we aren't using six.text_type somewhere
21:33:04 mriedem melwitt: link and stacktrace?
21:33:22 mriedem like, LOG.warning('something %s' % unicode) will do that
21:34:51 melwitt mriedem: http://logs.openstack.org/39/446239/5/check/gate-tempest-dsvm-cells-ubuntu-xenial/7061c58/logs/screen-n-cell-region.txt.gz?#_May_01_20_22_15_324264
21:35:37 mriedem hmm https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1011
21:35:47 cdent edleafe: I guess you didn't run all the functional tests before pushing that can_host change. I can no longer trust you!
21:36:00 mriedem melwitt: i've seen something like that in n-cpu where we log an instance with plugging vifs
21:36:11 mriedem it dumps the entire instance and there must be something in there that has unicode in it
21:36:23 melwitt hm, okay. thanks
21:36:26 cdent melwitt: saw some melwitt I've just started seeing that too
21:36:31 openstack Launchpad bug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged]
21:36:31 mriedem melwitt: https://bugs.launchpad.net/nova/+bug/1580728
21:36:33 melwitt I don't think I did anything to cause that but maybe I did
21:37:41 mriedem in ^ it's unicode in the instance display name
21:37:43 cdent http://logs.openstack.org/13/459413/5/check/gate-tempest-dsvm-cells-ubuntu-xenial/da51ac2/console.html#_2017-05-01_20_33_50_887866
21:37:45 melwitt cdent: ack. good it's not my patch maybe probably
21:38:15 melwitt that's weird, I wonder what made this start happening all of a sudden. implies something recently changed to unicode
21:38:55 mriedem check logstash
21:40:40 mriedem i only see 2 hits in logstash
21:41:11 melwitt doing. I also found earlier today that tempest.api.volume.admin.test_volume_quota_classes.VolumeQuotaClassesTest.test_update_default_quota fails intermittently. I'm guessing it's something to do with parallel tests racing
21:42:55 cdent mriedem: what did you search on?
21:43:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI._find_service https://review.openstack.org/461519
21:43:11 mriedem cdent: i deleted it :) sec
21:43:11 mtreinish melwitt: heh, the default quota tests are an inherent race condition
21:43:36 melwitt message:"UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 413: ordinal not in range(128)"
21:43:38 mriedem melwitt: the default quota compute api tests in tempest are locked to run serially
21:43:46 cdent cuz i've got 259 in the last 6 hours on message:"decode byte 0xc3 in position" which is clearly too general, but...
21:43:47 melwitt is what I searched on and see the same, only two hits
21:43:54 mtreinish but I think we set it to have sane defaults so we don't ever decrease the default quotas beyond where they start (which would be a big problem)
21:44:04 mriedem message:"UnicodeDecodeError" AND message:"instance_update_at_top" AND tags:"screen-n-cell-region.txt"
21:45:58 edleafe cdent: I was meaning to mark it WIP and forgot. I knew it wasn't fully tested.
21:46:15 melwitt mtreinish, mriedem: so they're set to run serially ... the thing I see is e.g. (u'snapshots_tempest-QosSpecsTestJSON-volume-type-1219921907', -1) not in [(u'gigabytes', 1001), (u'backups', 11), (u'snapshots_lvmdriver-1', -1), (u'volumes_lvmdriver-1', -1), (u'volumes', 11), (u'per_volume_gigabytes', -1), (u'backup_gigabytes', 1001), (u'gigabytes_lvmdriver-1', -1), (u'snapshots', 11)]
21:46:21 cdent edleafe: my bad really, I was being far too optimistic
21:47:29 melwitt mtreinish, mriedem: as though some other test had added a quota default and it gets picked up unexpectedly in another
21:48:06 cdent g'night all
21:50:39 mtreinish melwitt: do you have a link to that failure?
21:51:03 melwitt mtreinish: yeah, sorry I didn't include it http://logs.openstack.org/39/446239/5/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0b6acd6/logs/testr_results.html.gz
21:57:11 mtreinish melwitt: that's really weird. It looks like tempest sends a PUT with the new default quota entries, cinder returns 200, and the tempest does a GET to list the default quotas and the stuff it just added isn't there
21:58:41 mtreinish melwitt: oh, I think you might be right about a race
21:58:44 mriedem dansmith: while looking at this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4508 for using uuid rather than id, i realized this could have been a direct lookup if we stored the compute node uuid on the host mapping, but we only store the ComputeNode.host field...
21:59:15 melwitt mtreinish: hm, what I thought is that update_body is returning an additional unexpected item that was added by a different test
22:02:37 mtreinish melwitt: the test does a GET to get all the default quotas, loops over that list increases each quota there by 1 with a big update, then does another list and loops over them to see everything was updated
22:03:26 mtreinish the failure looks like another test class is making a default quota set and cleaning up after itself between the first get and after the update
22:03:48 mtreinish and tempest is failing on the verification that everything it asked to be updated is
22:03:53 mtreinish because a bunch of stuff was deleted
22:04:20 mtreinish at least that's what it looks like
22:04:41 melwitt okay. I'm looking at the code too but highly possible I've not understood it :)
22:05:26 mtreinish I'm having a hard time figuring out where the other default quotas are coming from though. It says volumes_tempest-QosSpecsTestJSON-volume-type-1219921907 which should mean the QosSpecsTestJSON class created it
22:05:34 mtreinish but I don't see where that happens in the code
22:05:39 mtreinish but I could just not be seeing it
22:07:53 melwitt I was focusing on the part where it's asserting (new, old) and I thought it mean there was something unexpected in "new", the return from the PUT
22:08:49 melwitt maybe I got that backward because I'm not familiar with the ContainsAll matcher
22:11:29 mtreinish melwitt: iirc the contains all matcher is basically a for loop that does: for x in arg[0]: self.assertIn(x, arg[1])
22:11:36 mtreinish but I'd have to dig up the code to be sure
22:12:11 melwitt agreed that it should mean the QosSpecsTestJSON created it but all I see is e.g. "cls.create_test_qos_specs(...)" and I dunno what all that entails. if it could possibly include creating a quota default
22:12:33 mtreinish melwitt: but looking at the tempest log: http://logs.openstack.org/39/446239/5/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0b6acd6/logs/tempest.txt.gz#_2017-04-29_03_26_53_000
22:12:54 mtreinish the teardownclass is deleting volume types around the same time the default quotas tests are doing their thing
22:13:05 mtreinish so I think we want to add an external lock between the classes to protect against that
22:16:15 mtreinish melwitt: can you open a bug for this so we can track it with e-r? I can work on a fix for it. It shouldn't be too hard
22:16:47 melwitt mtreinish: sure
22:17:14 mtreinish unless you wanted to work on the patch :)
22:20:38 melwitt mtreinish: no thanks :)
22:22:51 dansmith mriedem: you mean we could have known which cells to check instead of iterating over all of them?
22:23:16 dansmith mriedem: the id isn't unique, so it only buys us not iterating over all the cells in the case where the compute id happens to be uniquer
22:23:28 dansmith but on average, for similar-sized cells, you'll have nearly 100% overlap
22:25:02 openstackgerrit Ed Leafe proposed openstack/nova master: Remove the can_host column https://review.openstack.org/461494
22:28:57 openstack Launchpad bug 1687488 in tempest "VolumeQuotaClassesTest.test_update_default_quota fails intermittently" [Undecided,New]
22:28:57 melwitt mtreinish: https://bugs.launchpad.net/tempest/+bug/1687488
22:29:09 mtreinish melwitt: thanks

Earlier   Later