| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-01 | |||
| 13:20:35 | sean-k-mooney | efried: no. we could but if we did the lower constraitn version change would have to be different | |
| 13:20:54 | sean-k-mooney | well we proably should backport somthing but it cant be exactly the same | |
| 13:21:07 | efried | backporting requirements changes is messy under the best of circumstances; I would think we should avoid doing it at all unless there's a really good reason. | |
| 13:21:50 | sean-k-mooney | efried: well what i was planning to do was backport the tox change only and see if it worked on stable* | |
| 13:22:16 | sean-k-mooney | the older the branch the more likely it will | |
| 13:22:26 | efried | but not stein | |
| 13:22:49 | sean-k-mooney | stein need some of the version bumps or the unitest wont work | |
| 13:23:51 | sean-k-mooney | for example oslo.messaging 7.0.0 is need for the mocking of RPC calls to work properly | |
| 13:24:55 | sean-k-mooney | were as branches that test with ubuntu-xenial dont ned the psycopg2 bump as i think they wont have postgresql 10.3 | |
| 13:27:22 | efried | sean-k-mooney: I'm asking thode about it in -requirements. | |
| 13:27:28 | efried | cdent also ^ | |
| 13:27:37 | efried | sean-k-mooney: meanwhile, would you please ack https://review.openstack.org/#/c/647512/ for me? | |
| 13:27:57 | efried | mriedem: have you started looking at the lm job yet? | |
| 13:28:14 | mriedem | efried: i didn't see any bug reported for it | |
| 13:28:33 | efried | Yeah, I haven't opened one or looked into it at all, just noticed (on my bug mail on my phone) that it's failing 100% since like Friday. | |
| 13:28:40 | sean-k-mooney | if by ack you mean +1 ill review it now but yes i think this is the more complete fix for the patch i abandoned | |
| 13:28:43 | efried | just didn't want to start digging if you were already in there | |
| 13:28:58 | mriedem | efried: first morning back from vacation so no i'm not on it | |
| 13:29:05 | efried | sean-k-mooney: Yes, that's what I'm looking for, since you had already started to think about the solution previously. | |
| 13:30:40 | efried | mriedem: ack. I could actually use a brief tutorial on how you do the triage here. You seem to have a system for identifying when failures started and registering the signature of the problem somewhere (e-r?). | |
| 13:30:42 | efried | if you have time. | |
| 13:30:52 | mriedem | efried: i'd need a link to a failing job first | |
| 13:30:53 | efried | teach me to fish, in other words | |
| 13:31:04 | efried | mriedem: stand by | |
| 13:31:12 | efried | (but any patch from the past three days) | |
| 13:31:19 | mriedem | ok http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/ | |
| 13:31:27 | efried | https://review.openstack.org/#/c/648653/ | |
| 13:31:46 | mriedem | looks like this is the problem http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/job-output.txt.gz#_2019-04-01_11_12_47_685033 | |
| 13:31:51 | mriedem | Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-6c6a7e2b-6e5a-4ff4-ba6a-a3aaa8d8f83a) | |
| 13:32:18 | mriedem | which probably means https://github.com/openstack-dev/devstack/commit/0d4c9c9fa37c6f47523359044e5f65629321ff6d | |
| 13:33:23 | mriedem | so if i take "Multiple possible networks found, use a Network ID to be more specific" from that nova-live-migration console log and dump it in logstash, i bet it starts around 3/29 | |
| 13:35:34 | efried | clearly I don't know how to use logstash | |
| 13:35:45 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22nova-live-migration%5C%22&from=7d | |
| 13:36:08 | mriedem | so the message filter is part of the message above, excluding anything unique like a request id | |
| 13:36:09 | mriedem | or timestamp | |
| 13:36:19 | mriedem | the 'tags' is the file the message is in, so 'console' in this case | |
| 13:36:40 | mriedem | and i threw build_name in there, which we don't normally do, but figured i would to avoid noise from falst negatives in functional jobs or something | |
| 13:36:47 | mriedem | *false | |
| 13:37:18 | efried | how did you get it to span the correct time frame? | |
| 13:37:32 | mriedem | anyway, my guess is https://review.openstack.org/#/c/601433/ because (1) devstack creates the networks we're using here and (2) it doesn't fail for tempest because tempest always provides a specific nic when creating a server, but in the evacuate tests in nova-live-migration we don't | |
| 13:37:46 | mriedem | used the dropdown at the top right | |
| 13:38:20 | mriedem | https://wiki.openstack.org/wiki/BootstrappingHour/Debugging_Gate is old but might be relevant still | |
| 13:38:27 | efried | Cool. So now a bug needs to be opened, and then registered in e-r? | |
| 13:38:31 | mriedem | yeah | |
| 13:39:19 | efried | how did you get from the error message to the patch? Did you just happen to know about that devstack change? | |
| 13:40:50 | mriedem | i didn't, but i know that devstack creates the network we use in testing, | |
| 13:41:06 | mriedem | so assumed that because we're now failing on ambiguous network errors, someone started creating a 2nd network in devstack | |
| 13:41:10 | mriedem | lo and behold someone did | |
| 13:41:49 | efried | do you want me to open the bug? | |
| 13:41:53 | mriedem | sure | |
| 13:42:11 | yonglihe | orphan patch was splited: https://review.openstack.org/#/q/topic:bug/1820802+(status:open+OR+status:merged) nova-live-migration test fail, but the test result said all test passed... | |
| 13:42:12 | efried | against nova or against devstack? | |
| 13:42:35 | mriedem | yonglihe: we're talking about that nova-live-migration failure right now | |
| 13:42:38 | mriedem | efried: i guess start with nova | |
| 13:42:52 | yonglihe | ooh, got you. | |
| 13:42:53 | mriedem | it's really a behavior change in devstack, but it looks like it's only breaking nova-live-migration right now | |
| 13:43:14 | sean-k-mooney | efried: one nit inline but yep it looks good to me. i left a +1 to recored the same | |
| 13:44:51 | mriedem | i guess our options are (1) make nova-live-migration pick a network if there are multiple or (2) add a variable to devstack to toggle whether or not it creates a shared network | |
| 13:45:17 | mriedem | #1 is more flexible to env changes, but i can see myself being really annoyed without #2 in local devstack testing if i have to always specify a network now | |
| 13:45:33 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1822605 | |
| 13:45:34 | openstack | Launchpad bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Undecided,New] | |
| 13:46:14 | mriedem | although i guess i always disable tempest in my devstack setup local.conf so i won't hit #2 | |
| 13:46:56 | efried | mriedem: okay, so how do I register this with e-r? | |
| 13:47:13 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html | |
| 13:48:02 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html#queries specifically | |
| 13:48:10 | sean-k-mooney | mriedem: isnt there already a tempest config option to specify the network to use | |
| 13:48:27 | mriedem | sean-k-mooney: see https://review.openstack.org/#/c/615425/ | |
| 13:48:33 | mriedem | but what's failing isn't tempest | |
| 13:53:01 | sean-k-mooney | mriedem: sorry thats too much of a context switch to figure out if the review you linked helps or not. | |
| 13:53:25 | sean-k-mooney | i need either more or less coffee but not the amount i have currently consumed | |
| 13:53:32 | mdbooth | kaisers: Just found a bug in https://review.openstack.org/#/c/648093/ which just merged. Nothing catastrophic, but definitely worthy of another follow-up. | |
| 13:55:44 | mriedem | efried: i should have a patch up shortly | |
| 13:55:46 | sean-k-mooney | mriedem: #2 sound resonable just in general as a way to auto create an env with a shared network | |
| 13:57:00 | sean-k-mooney | althouhg i often just do that via a local.sh file which you could also do in the gate job | |
| 13:58:48 | efried | mriedem: https://review.openstack.org/#/c/649034/ ? | |
| 14:00:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass --nic when creating servers in evacuate integration test script https://review.openstack.org/649036 | |
| 14:02:06 | sean-k-mooney | https://bugs.launchpad.net/tempest/+bug/1821007 is ment to be fixed right? | |
| 14:02:07 | openstack | Launchpad bug 1821007 in tempest "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'gabbi_tempest_path' is undefined" [Critical,Fix released] - Assigned to Paul Belanger (pabelanger) | |
| 14:02:22 | mriedem | efried: +W | |
| 14:02:59 | mriedem | sean-k-mooney: yes http://status.openstack.org/elastic-recheck/#1821007 | |
| 14:03:33 | sean-k-mooney | mriedem: im still hitting it in my thirdparty ci job and just trying to figure out why | |
| 14:11:06 | sean-k-mooney | for some reason it appears to be using a commit form 7 days ago which was the commit just before that merged. | |
| 14:11:38 | efried | sean-k-mooney: prometheanfire doesn't care which l-c fix as long as it passes. I'm going to take cdent's because a) older, b) more comprehensive commit message. Cool? | |
| 14:12:08 | sean-k-mooney | efried: yep that is fine with me | |
| 14:12:26 | sean-k-mooney | efried: can we land it in stien or no? | |
| 14:13:04 | efried | sean-k-mooney: That I don't know. Sounds hairy to me. | |
| 14:13:11 | efried | sean-k-mooney: What would be the motivation to do that? | |
| 14:13:15 | mriedem | we don't need an rc2 for lower-constraints fixes right? | |
| 14:13:16 | efried | Is it going to break distros or something? | |
| 14:13:37 | mriedem | also, how latent is this? | |
| 14:13:51 | sean-k-mooney | mriedem: it has always been broken | |
| 14:14:04 | sean-k-mooney | since we introduced lower-constarits | |
| 14:14:12 | sean-k-mooney | i think | |
| 14:14:13 | mriedem | then it's definitely not an rc2 issue | |
| 14:14:23 | sean-k-mooney | mriedem: correcct | |
| 14:14:36 | sean-k-mooney | i was wondering if we want to backport it after rc2 | |
| 14:14:55 | sean-k-mooney | e.g. do we want to fix lower-constratins on stable branches or not in general | |
| 14:15:26 | sean-k-mooney | if the answer is no we proably shoudl disable the job on the stable branches | |
| 14:27:38 | sean-k-mooney | mriedem: oh i realised i have my own version of devstack-tempest because i cant import jobs from tempest without needing to pull in amost all of openstack into my zuul configruation. so ya i need to copy the same fix. thanks for confiming its fixed upstream | |
| 14:30:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Full stub out os_vif https://review.openstack.org/648748 | |