Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-01
13:15:33 sean-k-mooney cdent: im fine to abandon min in favor of yours as long as one of them is merged
13:18:53 efried They do seem to differ a fair bit in the updates to l-c
13:19:01 sean-k-mooney they do
13:19:09 efried and requirements.txt
13:19:37 sean-k-mooney i tried to make as few chagnes as i coudl initally but ended up pulling in some of the changes cdent had made.
13:19:59 efried sean-k-mooney: You're not thinking to backport this to stein, are you?
13:20:35 sean-k-mooney efried: no. we could but if we did the lower constraitn version change would have to be different
13:20:54 sean-k-mooney well we proably should backport somthing but it cant be exactly the same
13:21:07 efried backporting requirements changes is messy under the best of circumstances; I would think we should avoid doing it at all unless there's a really good reason.
13:21:50 sean-k-mooney efried: well what i was planning to do was backport the tox change only and see if it worked on stable*
13:22:16 sean-k-mooney the older the branch the more likely it will
13:22:26 efried but not stein
13:22:49 sean-k-mooney stein need some of the version bumps or the unitest wont work
13:23:51 sean-k-mooney for example oslo.messaging 7.0.0 is need for the mocking of RPC calls to work properly
13:24:55 sean-k-mooney were as branches that test with ubuntu-xenial dont ned the psycopg2 bump as i think they wont have postgresql 10.3
13:27:22 efried sean-k-mooney: I'm asking thode about it in -requirements.
13:27:28 efried cdent also ^
13:27:37 efried sean-k-mooney: meanwhile, would you please ack https://review.openstack.org/#/c/647512/ for me?
13:27:57 efried mriedem: have you started looking at the lm job yet?
13:28:14 mriedem efried: i didn't see any bug reported for it
13:28:33 efried Yeah, I haven't opened one or looked into it at all, just noticed (on my bug mail on my phone) that it's failing 100% since like Friday.
13:28:40 sean-k-mooney if by ack you mean +1 ill review it now but yes i think this is the more complete fix for the patch i abandoned
13:28:43 efried just didn't want to start digging if you were already in there
13:28:58 mriedem efried: first morning back from vacation so no i'm not on it
13:29:05 efried sean-k-mooney: Yes, that's what I'm looking for, since you had already started to think about the solution previously.
13:30:40 efried mriedem: ack. I could actually use a brief tutorial on how you do the triage here. You seem to have a system for identifying when failures started and registering the signature of the problem somewhere (e-r?).
13:30:42 efried if you have time.
13:30:52 mriedem efried: i'd need a link to a failing job first
13:30:53 efried teach me to fish, in other words
13:31:04 efried mriedem: stand by
13:31:12 efried (but any patch from the past three days)
13:31:19 mriedem ok http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/
13:31:27 efried https://review.openstack.org/#/c/648653/
13:31:46 mriedem looks like this is the problem http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/job-output.txt.gz#_2019-04-01_11_12_47_685033
13:31:51 mriedem Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-6c6a7e2b-6e5a-4ff4-ba6a-a3aaa8d8f83a)
13:32:18 mriedem which probably means https://github.com/openstack-dev/devstack/commit/0d4c9c9fa37c6f47523359044e5f65629321ff6d
13:33:23 mriedem so if i take "Multiple possible networks found, use a Network ID to be more specific" from that nova-live-migration console log and dump it in logstash, i bet it starts around 3/29
13:35:34 efried clearly I don't know how to use logstash
13:35:45 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22nova-live-migration%5C%22&from=7d
13:36:08 mriedem so the message filter is part of the message above, excluding anything unique like a request id
13:36:09 mriedem or timestamp
13:36:19 mriedem the 'tags' is the file the message is in, so 'console' in this case
13:36:40 mriedem and i threw build_name in there, which we don't normally do, but figured i would to avoid noise from falst negatives in functional jobs or something
13:36:47 mriedem *false
13:37:18 efried how did you get it to span the correct time frame?
13:37:32 mriedem anyway, my guess is https://review.openstack.org/#/c/601433/ because (1) devstack creates the networks we're using here and (2) it doesn't fail for tempest because tempest always provides a specific nic when creating a server, but in the evacuate tests in nova-live-migration we don't
13:37:46 mriedem used the dropdown at the top right
13:38:20 mriedem https://wiki.openstack.org/wiki/BootstrappingHour/Debugging_Gate is old but might be relevant still
13:38:27 efried Cool. So now a bug needs to be opened, and then registered in e-r?
13:38:31 mriedem yeah
13:39:19 efried how did you get from the error message to the patch? Did you just happen to know about that devstack change?
13:40:50 mriedem i didn't, but i know that devstack creates the network we use in testing,
13:41:06 mriedem so assumed that because we're now failing on ambiguous network errors, someone started creating a 2nd network in devstack
13:41:10 mriedem lo and behold someone did
13:41:49 efried do you want me to open the bug?
13:41:53 mriedem sure
13:42:11 yonglihe orphan patch was splited: https://review.openstack.org/#/q/topic:bug/1820802+(status:open+OR+status:merged) nova-live-migration test fail, but the test result said all test passed...
13:42:12 efried against nova or against devstack?
13:42:35 mriedem yonglihe: we're talking about that nova-live-migration failure right now
13:42:38 mriedem efried: i guess start with nova
13:42:52 yonglihe ooh, got you.
13:42:53 mriedem it's really a behavior change in devstack, but it looks like it's only breaking nova-live-migration right now
13:43:14 sean-k-mooney efried: one nit inline but yep it looks good to me. i left a +1 to recored the same
13:44:51 mriedem i guess our options are (1) make nova-live-migration pick a network if there are multiple or (2) add a variable to devstack to toggle whether or not it creates a shared network
13:45:17 mriedem #1 is more flexible to env changes, but i can see myself being really annoyed without #2 in local devstack testing if i have to always specify a network now
13:45:33 efried mriedem: https://bugs.launchpad.net/nova/+bug/1822605
13:45:34 openstack Launchpad bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Undecided,New]
13:46:14 mriedem although i guess i always disable tempest in my devstack setup local.conf so i won't hit #2
13:46:56 efried mriedem: okay, so how do I register this with e-r?
13:47:13 mriedem https://docs.openstack.org/infra/elastic-recheck/readme.html
13:48:02 mriedem https://docs.openstack.org/infra/elastic-recheck/readme.html#queries specifically
13:48:10 sean-k-mooney mriedem: isnt there already a tempest config option to specify the network to use
13:48:27 mriedem sean-k-mooney: see https://review.openstack.org/#/c/615425/
13:48:33 mriedem but what's failing isn't tempest
13:53:01 sean-k-mooney mriedem: sorry thats too much of a context switch to figure out if the review you linked helps or not.
13:53:25 sean-k-mooney i need either more or less coffee but not the amount i have currently consumed
13:53:32 mdbooth kaisers: Just found a bug in https://review.openstack.org/#/c/648093/ which just merged. Nothing catastrophic, but definitely worthy of another follow-up.
13:55:44 mriedem efried: i should have a patch up shortly
13:55:46 sean-k-mooney mriedem: #2 sound resonable just in general as a way to auto create an env with a shared network
13:57:00 sean-k-mooney althouhg i often just do that via a local.sh file which you could also do in the gate job
13:58:48 efried mriedem: https://review.openstack.org/#/c/649034/ ?
14:00:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Pass --nic when creating servers in evacuate integration test script https://review.openstack.org/649036
14:02:06 sean-k-mooney https://bugs.launchpad.net/tempest/+bug/1821007 is ment to be fixed right?
14:02:07 openstack Launchpad bug 1821007 in tempest "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'gabbi_tempest_path' is undefined" [Critical,Fix released] - Assigned to Paul Belanger (pabelanger)
14:02:22 mriedem efried: +W
14:02:59 mriedem sean-k-mooney: yes http://status.openstack.org/elastic-recheck/#1821007
14:03:33 sean-k-mooney mriedem: im still hitting it in my thirdparty ci job and just trying to figure out why
14:11:06 sean-k-mooney for some reason it appears to be using a commit form 7 days ago which was the commit just before that merged.
14:11:38 efried sean-k-mooney: prometheanfire doesn't care which l-c fix as long as it passes. I'm going to take cdent's because a) older, b) more comprehensive commit message. Cool?
14:12:08 sean-k-mooney efried: yep that is fine with me
14:12:26 sean-k-mooney efried: can we land it in stien or no?
14:13:04 efried sean-k-mooney: That I don't know. Sounds hairy to me.
14:13:11 efried sean-k-mooney: What would be the motivation to do that?
14:13:15 mriedem we don't need an rc2 for lower-constraints fixes right?
14:13:16 efried Is it going to break distros or something?
14:13:37 mriedem also, how latent is this?
14:13:51 sean-k-mooney mriedem: it has always been broken
14:14:04 sean-k-mooney since we introduced lower-constarits
14:14:12 sean-k-mooney i think
14:14:13 mriedem then it's definitely not an rc2 issue

Earlier   Later