| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-01 | |||
| 13:28:58 | mriedem | efried: first morning back from vacation so no i'm not on it | |
| 13:29:05 | efried | sean-k-mooney: Yes, that's what I'm looking for, since you had already started to think about the solution previously. | |
| 13:30:40 | efried | mriedem: ack. I could actually use a brief tutorial on how you do the triage here. You seem to have a system for identifying when failures started and registering the signature of the problem somewhere (e-r?). | |
| 13:30:42 | efried | if you have time. | |
| 13:30:52 | mriedem | efried: i'd need a link to a failing job first | |
| 13:30:53 | efried | teach me to fish, in other words | |
| 13:31:04 | efried | mriedem: stand by | |
| 13:31:12 | efried | (but any patch from the past three days) | |
| 13:31:19 | mriedem | ok http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/ | |
| 13:31:27 | efried | https://review.openstack.org/#/c/648653/ | |
| 13:31:46 | mriedem | looks like this is the problem http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/job-output.txt.gz#_2019-04-01_11_12_47_685033 | |
| 13:31:51 | mriedem | Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-6c6a7e2b-6e5a-4ff4-ba6a-a3aaa8d8f83a) | |
| 13:32:18 | mriedem | which probably means https://github.com/openstack-dev/devstack/commit/0d4c9c9fa37c6f47523359044e5f65629321ff6d | |
| 13:33:23 | mriedem | so if i take "Multiple possible networks found, use a Network ID to be more specific" from that nova-live-migration console log and dump it in logstash, i bet it starts around 3/29 | |
| 13:35:34 | efried | clearly I don't know how to use logstash | |
| 13:35:45 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22nova-live-migration%5C%22&from=7d | |
| 13:36:08 | mriedem | so the message filter is part of the message above, excluding anything unique like a request id | |
| 13:36:09 | mriedem | or timestamp | |
| 13:36:19 | mriedem | the 'tags' is the file the message is in, so 'console' in this case | |
| 13:36:40 | mriedem | and i threw build_name in there, which we don't normally do, but figured i would to avoid noise from falst negatives in functional jobs or something | |
| 13:36:47 | mriedem | *false | |
| 13:37:18 | efried | how did you get it to span the correct time frame? | |
| 13:37:32 | mriedem | anyway, my guess is https://review.openstack.org/#/c/601433/ because (1) devstack creates the networks we're using here and (2) it doesn't fail for tempest because tempest always provides a specific nic when creating a server, but in the evacuate tests in nova-live-migration we don't | |
| 13:37:46 | mriedem | used the dropdown at the top right | |
| 13:38:20 | mriedem | https://wiki.openstack.org/wiki/BootstrappingHour/Debugging_Gate is old but might be relevant still | |
| 13:38:27 | efried | Cool. So now a bug needs to be opened, and then registered in e-r? | |
| 13:38:31 | mriedem | yeah | |
| 13:39:19 | efried | how did you get from the error message to the patch? Did you just happen to know about that devstack change? | |
| 13:40:50 | mriedem | i didn't, but i know that devstack creates the network we use in testing, | |
| 13:41:06 | mriedem | so assumed that because we're now failing on ambiguous network errors, someone started creating a 2nd network in devstack | |
| 13:41:10 | mriedem | lo and behold someone did | |
| 13:41:49 | efried | do you want me to open the bug? | |
| 13:41:53 | mriedem | sure | |
| 13:42:11 | yonglihe | orphan patch was splited: https://review.openstack.org/#/q/topic:bug/1820802+(status:open+OR+status:merged) nova-live-migration test fail, but the test result said all test passed... | |
| 13:42:12 | efried | against nova or against devstack? | |
| 13:42:35 | mriedem | yonglihe: we're talking about that nova-live-migration failure right now | |
| 13:42:38 | mriedem | efried: i guess start with nova | |
| 13:42:52 | yonglihe | ooh, got you. | |
| 13:42:53 | mriedem | it's really a behavior change in devstack, but it looks like it's only breaking nova-live-migration right now | |
| 13:43:14 | sean-k-mooney | efried: one nit inline but yep it looks good to me. i left a +1 to recored the same | |
| 13:44:51 | mriedem | i guess our options are (1) make nova-live-migration pick a network if there are multiple or (2) add a variable to devstack to toggle whether or not it creates a shared network | |
| 13:45:17 | mriedem | #1 is more flexible to env changes, but i can see myself being really annoyed without #2 in local devstack testing if i have to always specify a network now | |
| 13:45:33 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1822605 | |
| 13:45:34 | openstack | Launchpad bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Undecided,New] | |
| 13:46:14 | mriedem | although i guess i always disable tempest in my devstack setup local.conf so i won't hit #2 | |
| 13:46:56 | efried | mriedem: okay, so how do I register this with e-r? | |
| 13:47:13 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html | |
| 13:48:02 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html#queries specifically | |
| 13:48:10 | sean-k-mooney | mriedem: isnt there already a tempest config option to specify the network to use | |
| 13:48:27 | mriedem | sean-k-mooney: see https://review.openstack.org/#/c/615425/ | |
| 13:48:33 | mriedem | but what's failing isn't tempest | |
| 13:53:01 | sean-k-mooney | mriedem: sorry thats too much of a context switch to figure out if the review you linked helps or not. | |
| 13:53:25 | sean-k-mooney | i need either more or less coffee but not the amount i have currently consumed | |
| 13:53:32 | mdbooth | kaisers: Just found a bug in https://review.openstack.org/#/c/648093/ which just merged. Nothing catastrophic, but definitely worthy of another follow-up. | |
| 13:55:44 | mriedem | efried: i should have a patch up shortly | |
| 13:55:46 | sean-k-mooney | mriedem: #2 sound resonable just in general as a way to auto create an env with a shared network | |
| 13:57:00 | sean-k-mooney | althouhg i often just do that via a local.sh file which you could also do in the gate job | |
| 13:58:48 | efried | mriedem: https://review.openstack.org/#/c/649034/ ? | |
| 14:00:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass --nic when creating servers in evacuate integration test script https://review.openstack.org/649036 | |
| 14:02:06 | sean-k-mooney | https://bugs.launchpad.net/tempest/+bug/1821007 is ment to be fixed right? | |
| 14:02:07 | openstack | Launchpad bug 1821007 in tempest "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'gabbi_tempest_path' is undefined" [Critical,Fix released] - Assigned to Paul Belanger (pabelanger) | |
| 14:02:22 | mriedem | efried: +W | |
| 14:02:59 | mriedem | sean-k-mooney: yes http://status.openstack.org/elastic-recheck/#1821007 | |
| 14:03:33 | sean-k-mooney | mriedem: im still hitting it in my thirdparty ci job and just trying to figure out why | |
| 14:11:06 | sean-k-mooney | for some reason it appears to be using a commit form 7 days ago which was the commit just before that merged. | |
| 14:11:38 | efried | sean-k-mooney: prometheanfire doesn't care which l-c fix as long as it passes. I'm going to take cdent's because a) older, b) more comprehensive commit message. Cool? | |
| 14:12:08 | sean-k-mooney | efried: yep that is fine with me | |
| 14:12:26 | sean-k-mooney | efried: can we land it in stien or no? | |
| 14:13:04 | efried | sean-k-mooney: That I don't know. Sounds hairy to me. | |
| 14:13:11 | efried | sean-k-mooney: What would be the motivation to do that? | |
| 14:13:15 | mriedem | we don't need an rc2 for lower-constraints fixes right? | |
| 14:13:16 | efried | Is it going to break distros or something? | |
| 14:13:37 | mriedem | also, how latent is this? | |
| 14:13:51 | sean-k-mooney | mriedem: it has always been broken | |
| 14:14:04 | sean-k-mooney | since we introduced lower-constarits | |
| 14:14:12 | sean-k-mooney | i think | |
| 14:14:13 | mriedem | then it's definitely not an rc2 issue | |
| 14:14:23 | sean-k-mooney | mriedem: correcct | |
| 14:14:36 | sean-k-mooney | i was wondering if we want to backport it after rc2 | |
| 14:14:55 | sean-k-mooney | e.g. do we want to fix lower-constratins on stable branches or not in general | |
| 14:15:26 | sean-k-mooney | if the answer is no we proably shoudl disable the job on the stable branches | |
| 14:27:38 | sean-k-mooney | mriedem: oh i realised i have my own version of devstack-tempest because i cant import jobs from tempest without needing to pull in amost all of openstack into my zuul configruation. so ya i need to copy the same fix. thanks for confiming its fixed upstream | |
| 14:30:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Full stub out os_vif https://review.openstack.org/648748 | |
| 14:56:58 | mriedem | efried: looks like https://review.openstack.org/#/c/649036/ fixes the problem, nova-live-migration job is past that issue now, but will need to recheck the change | |
| 14:57:22 | efried | ack | |
| 14:57:50 | efried | mriedem: grenade failure spurious? | |
| 14:58:41 | efried | ssh auth error | |
| 14:59:35 | bauzas | I just feel we have enough consensus with https://review.openstack.org/#/c/647512/4 | |
| 14:59:39 | bauzas | efried: alex_xu: ^ | |
| 15:00:07 | mriedem | efried: yes | |
| 15:00:29 | efried | bauzas: Yes. Let's wait for the gate fix before pushing this into the queue. | |
| 15:01:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/647512 | |
| 15:01:42 | efried | bauzas: I rebased on the gate fix and pushed ^ | |
| 15:02:18 | efried | mriedem: any reason to wait to recheck? | |
| 15:03:08 | mriedem | efried: i'm not sure it will do anything while zuul is still running | |
| 15:03:11 | bauzas | cool, mriedem: planning to propose the backports or want me doing it ? | |
| 15:03:28 | mriedem | bauzas: for the RequestSpec.ignore_hosts thing? i'll handle it. | |
| 15:05:45 | bauzas | mriedem: ok :) | |
| 15:08:58 | efried | answer appears to be no, the recheck doesn't kick the running job out. | |
| 15:09:39 | dansmith | you have to change the patch to kick it out of any queue | |