| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-01 | |||
| 13:35:34 | efried | clearly I don't know how to use logstash | |
| 13:35:45 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22nova-live-migration%5C%22&from=7d | |
| 13:36:08 | mriedem | so the message filter is part of the message above, excluding anything unique like a request id | |
| 13:36:09 | mriedem | or timestamp | |
| 13:36:19 | mriedem | the 'tags' is the file the message is in, so 'console' in this case | |
| 13:36:40 | mriedem | and i threw build_name in there, which we don't normally do, but figured i would to avoid noise from falst negatives in functional jobs or something | |
| 13:36:47 | mriedem | *false | |
| 13:37:18 | efried | how did you get it to span the correct time frame? | |
| 13:37:32 | mriedem | anyway, my guess is https://review.openstack.org/#/c/601433/ because (1) devstack creates the networks we're using here and (2) it doesn't fail for tempest because tempest always provides a specific nic when creating a server, but in the evacuate tests in nova-live-migration we don't | |
| 13:37:46 | mriedem | used the dropdown at the top right | |
| 13:38:20 | mriedem | https://wiki.openstack.org/wiki/BootstrappingHour/Debugging_Gate is old but might be relevant still | |
| 13:38:27 | efried | Cool. So now a bug needs to be opened, and then registered in e-r? | |
| 13:38:31 | mriedem | yeah | |
| 13:39:19 | efried | how did you get from the error message to the patch? Did you just happen to know about that devstack change? | |
| 13:40:50 | mriedem | i didn't, but i know that devstack creates the network we use in testing, | |
| 13:41:06 | mriedem | so assumed that because we're now failing on ambiguous network errors, someone started creating a 2nd network in devstack | |
| 13:41:10 | mriedem | lo and behold someone did | |
| 13:41:49 | efried | do you want me to open the bug? | |
| 13:41:53 | mriedem | sure | |
| 13:42:11 | yonglihe | orphan patch was splited: https://review.openstack.org/#/q/topic:bug/1820802+(status:open+OR+status:merged) nova-live-migration test fail, but the test result said all test passed... | |
| 13:42:12 | efried | against nova or against devstack? | |
| 13:42:35 | mriedem | yonglihe: we're talking about that nova-live-migration failure right now | |
| 13:42:38 | mriedem | efried: i guess start with nova | |
| 13:42:52 | yonglihe | ooh, got you. | |
| 13:42:53 | mriedem | it's really a behavior change in devstack, but it looks like it's only breaking nova-live-migration right now | |
| 13:43:14 | sean-k-mooney | efried: one nit inline but yep it looks good to me. i left a +1 to recored the same | |
| 13:44:51 | mriedem | i guess our options are (1) make nova-live-migration pick a network if there are multiple or (2) add a variable to devstack to toggle whether or not it creates a shared network | |
| 13:45:17 | mriedem | #1 is more flexible to env changes, but i can see myself being really annoyed without #2 in local devstack testing if i have to always specify a network now | |
| 13:45:33 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1822605 | |
| 13:45:34 | openstack | Launchpad bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Undecided,New] | |
| 13:46:14 | mriedem | although i guess i always disable tempest in my devstack setup local.conf so i won't hit #2 | |
| 13:46:56 | efried | mriedem: okay, so how do I register this with e-r? | |
| 13:47:13 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html | |
| 13:48:02 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html#queries specifically | |
| 13:48:10 | sean-k-mooney | mriedem: isnt there already a tempest config option to specify the network to use | |
| 13:48:27 | mriedem | sean-k-mooney: see https://review.openstack.org/#/c/615425/ | |
| 13:48:33 | mriedem | but what's failing isn't tempest | |
| 13:53:01 | sean-k-mooney | mriedem: sorry thats too much of a context switch to figure out if the review you linked helps or not. | |
| 13:53:25 | sean-k-mooney | i need either more or less coffee but not the amount i have currently consumed | |
| 13:53:32 | mdbooth | kaisers: Just found a bug in https://review.openstack.org/#/c/648093/ which just merged. Nothing catastrophic, but definitely worthy of another follow-up. | |
| 13:55:44 | mriedem | efried: i should have a patch up shortly | |
| 13:55:46 | sean-k-mooney | mriedem: #2 sound resonable just in general as a way to auto create an env with a shared network | |
| 13:57:00 | sean-k-mooney | althouhg i often just do that via a local.sh file which you could also do in the gate job | |
| 13:58:48 | efried | mriedem: https://review.openstack.org/#/c/649034/ ? | |
| 14:00:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass --nic when creating servers in evacuate integration test script https://review.openstack.org/649036 | |
| 14:02:06 | sean-k-mooney | https://bugs.launchpad.net/tempest/+bug/1821007 is ment to be fixed right? | |
| 14:02:07 | openstack | Launchpad bug 1821007 in tempest "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'gabbi_tempest_path' is undefined" [Critical,Fix released] - Assigned to Paul Belanger (pabelanger) | |
| 14:02:22 | mriedem | efried: +W | |
| 14:02:59 | mriedem | sean-k-mooney: yes http://status.openstack.org/elastic-recheck/#1821007 | |
| 14:03:33 | sean-k-mooney | mriedem: im still hitting it in my thirdparty ci job and just trying to figure out why | |
| 14:11:06 | sean-k-mooney | for some reason it appears to be using a commit form 7 days ago which was the commit just before that merged. | |
| 14:11:38 | efried | sean-k-mooney: prometheanfire doesn't care which l-c fix as long as it passes. I'm going to take cdent's because a) older, b) more comprehensive commit message. Cool? | |
| 14:12:08 | sean-k-mooney | efried: yep that is fine with me | |
| 14:12:26 | sean-k-mooney | efried: can we land it in stien or no? | |
| 14:13:04 | efried | sean-k-mooney: That I don't know. Sounds hairy to me. | |
| 14:13:11 | efried | sean-k-mooney: What would be the motivation to do that? | |
| 14:13:15 | mriedem | we don't need an rc2 for lower-constraints fixes right? | |
| 14:13:16 | efried | Is it going to break distros or something? | |
| 14:13:37 | mriedem | also, how latent is this? | |
| 14:13:51 | sean-k-mooney | mriedem: it has always been broken | |
| 14:14:04 | sean-k-mooney | since we introduced lower-constarits | |
| 14:14:12 | sean-k-mooney | i think | |
| 14:14:13 | mriedem | then it's definitely not an rc2 issue | |
| 14:14:23 | sean-k-mooney | mriedem: correcct | |
| 14:14:36 | sean-k-mooney | i was wondering if we want to backport it after rc2 | |
| 14:14:55 | sean-k-mooney | e.g. do we want to fix lower-constratins on stable branches or not in general | |
| 14:15:26 | sean-k-mooney | if the answer is no we proably shoudl disable the job on the stable branches | |
| 14:27:38 | sean-k-mooney | mriedem: oh i realised i have my own version of devstack-tempest because i cant import jobs from tempest without needing to pull in amost all of openstack into my zuul configruation. so ya i need to copy the same fix. thanks for confiming its fixed upstream | |
| 14:30:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Full stub out os_vif https://review.openstack.org/648748 | |
| 14:56:58 | mriedem | efried: looks like https://review.openstack.org/#/c/649036/ fixes the problem, nova-live-migration job is past that issue now, but will need to recheck the change | |
| 14:57:22 | efried | ack | |
| 14:57:50 | efried | mriedem: grenade failure spurious? | |
| 14:58:41 | efried | ssh auth error | |
| 14:59:35 | bauzas | I just feel we have enough consensus with https://review.openstack.org/#/c/647512/4 | |
| 14:59:39 | bauzas | efried: alex_xu: ^ | |
| 15:00:07 | mriedem | efried: yes | |
| 15:00:29 | efried | bauzas: Yes. Let's wait for the gate fix before pushing this into the queue. | |
| 15:01:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/647512 | |
| 15:01:42 | efried | bauzas: I rebased on the gate fix and pushed ^ | |
| 15:02:18 | efried | mriedem: any reason to wait to recheck? | |
| 15:03:08 | mriedem | efried: i'm not sure it will do anything while zuul is still running | |
| 15:03:11 | bauzas | cool, mriedem: planning to propose the backports or want me doing it ? | |
| 15:03:28 | mriedem | bauzas: for the RequestSpec.ignore_hosts thing? i'll handle it. | |
| 15:05:45 | bauzas | mriedem: ok :) | |
| 15:08:58 | efried | answer appears to be no, the recheck doesn't kick the running job out. | |
| 15:09:39 | dansmith | you have to change the patch to kick it out of any queue | |
| 15:09:51 | dansmith | no vote or comment will do it | |
| 15:16:35 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Add a script for counting blueprints https://review.openstack.org/581914 | |
| 15:16:36 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Add a process to abandon a spec https://review.openstack.org/648800 | |
| 15:21:14 | openstackgerrit | Merged openstack/nova stable/queens: [Stable Only] hardware: Handle races during pinning https://review.openstack.org/647831 | |
| 15:23:25 | openstackgerrit | Chris Dent proposed openstack/nova master: Add placement as required project to functional py36 and 37 https://review.openstack.org/649068 | |
| 15:31:45 | bauzas | efried: sorry, just saw your question, well, like dansmith said, you need to provide a new PS to put off the check or gate pipeline | |
| 15:32:12 | bauzas | just use the right words, I wasn't understanding your question | |
| 15:32:25 | efried | nod. | |
| 15:33:37 | bauzas | but let's ask jaypipes to provide a follow-up post from http://www.joinfu.com/2014/01/understanding-the-openstack-ci-system/ :p | |
| 15:34:01 | bauzas | or just digest https://zuul-ci.org/docs/zuul/user/gating.html :p | |
| 15:34:49 | bauzas | huh, rather https://zuul-ci.org/docs/zuul/user/concepts.html | |
| 15:57:04 | jaypipes | bauzas: erm... | |
| 15:57:14 | jaypipes | bauzas: yeah, "Just Use Zuul CI" | |
| 16:07:25 | bauzas | jaypipes: FWIW, I give you credits on helping some French folk who was working on a Stackforge project named "Climate" to know why Jenkins was telling him -1 sometimes | |