| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-01 | |||
| 10:36:39 | sean-k-mooney | you could jsut file a new blueprint and update the old one to say its superceed by the new one | |
| 11:05:19 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676 | |
| 11:05:27 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689 | |
| 11:05:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709 | |
| 11:05:47 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.openstack.org/576712 | |
| 11:26:04 | sean-k-mooney | efried: fyi this https://blueprints.launchpad.net/nova/+spec/consistent-use-of-network-events is the blueprint to track teh nova side of the neutron spec https://review.openstack.org/#/c/645173/ | |
| 11:48:15 | openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Support adding the reason behind a server lock https://review.openstack.org/638629 | |
| 12:19:02 | bauzas | stephenfin: I'll take a look | |
| 13:09:00 | openstackgerrit | sean mooney proposed openstack/nova master: make lower-constraints env use lower-constraints https://review.openstack.org/645392 | |
| 13:09:10 | efried | sean-k-mooney: ack, thanks. | |
| 13:09:40 | efried | sean-k-mooney: Are you wanting it targeted for train? | |
| 13:10:26 | sean-k-mooney | the blueprint i think so yes but that will likely depend on the conversation at the PTG. | |
| 13:11:09 | sean-k-mooney | we (nova and neutorn) could decide that it was not a priority for train but it is techdebt i would like to resovle eventually | |
| 13:12:00 | sean-k-mooney | efried: by the way im hoping we can merge either cdents lower constratints fix https://review.openstack.org/#/c/622972 or mine https://review.openstack.org/645392 sooner rather then later | |
| 13:12:18 | efried | sean-k-mooney: I'll mark the bp series goal for train then. | |
| 13:12:31 | cdent | sean-k-mooney: what's the essential difference? | |
| 13:12:39 | cdent | (between mine and yours) | |
| 13:14:06 | sean-k-mooney | cdent: i thihk at this point they are more or less the same although you remove a test i didnt need too | |
| 13:14:59 | sean-k-mooney | cdent: also your's has a better commit message but it does not have the close bug line "Closes-Bug: #1822575" for the bug i opened to track it | |
| 13:15:00 | openstack | bug 1822575 in OpenStack Compute (nova) "lower-constraints are not used in gate job" [Undecided,In progress] https://launchpad.net/bugs/1822575 - Assigned to sean mooney (sean-k-mooney) | |
| 13:15:33 | sean-k-mooney | cdent: im fine to abandon min in favor of yours as long as one of them is merged | |
| 13:18:53 | efried | They do seem to differ a fair bit in the updates to l-c | |
| 13:19:01 | sean-k-mooney | they do | |
| 13:19:09 | efried | and requirements.txt | |
| 13:19:37 | sean-k-mooney | i tried to make as few chagnes as i coudl initally but ended up pulling in some of the changes cdent had made. | |
| 13:19:59 | efried | sean-k-mooney: You're not thinking to backport this to stein, are you? | |
| 13:20:35 | sean-k-mooney | efried: no. we could but if we did the lower constraitn version change would have to be different | |
| 13:20:54 | sean-k-mooney | well we proably should backport somthing but it cant be exactly the same | |
| 13:21:07 | efried | backporting requirements changes is messy under the best of circumstances; I would think we should avoid doing it at all unless there's a really good reason. | |
| 13:21:50 | sean-k-mooney | efried: well what i was planning to do was backport the tox change only and see if it worked on stable* | |
| 13:22:16 | sean-k-mooney | the older the branch the more likely it will | |
| 13:22:26 | efried | but not stein | |
| 13:22:49 | sean-k-mooney | stein need some of the version bumps or the unitest wont work | |
| 13:23:51 | sean-k-mooney | for example oslo.messaging 7.0.0 is need for the mocking of RPC calls to work properly | |
| 13:24:55 | sean-k-mooney | were as branches that test with ubuntu-xenial dont ned the psycopg2 bump as i think they wont have postgresql 10.3 | |
| 13:27:22 | efried | sean-k-mooney: I'm asking thode about it in -requirements. | |
| 13:27:28 | efried | cdent also ^ | |
| 13:27:37 | efried | sean-k-mooney: meanwhile, would you please ack https://review.openstack.org/#/c/647512/ for me? | |
| 13:27:57 | efried | mriedem: have you started looking at the lm job yet? | |
| 13:28:14 | mriedem | efried: i didn't see any bug reported for it | |
| 13:28:33 | efried | Yeah, I haven't opened one or looked into it at all, just noticed (on my bug mail on my phone) that it's failing 100% since like Friday. | |
| 13:28:40 | sean-k-mooney | if by ack you mean +1 ill review it now but yes i think this is the more complete fix for the patch i abandoned | |
| 13:28:43 | efried | just didn't want to start digging if you were already in there | |
| 13:28:58 | mriedem | efried: first morning back from vacation so no i'm not on it | |
| 13:29:05 | efried | sean-k-mooney: Yes, that's what I'm looking for, since you had already started to think about the solution previously. | |
| 13:30:40 | efried | mriedem: ack. I could actually use a brief tutorial on how you do the triage here. You seem to have a system for identifying when failures started and registering the signature of the problem somewhere (e-r?). | |
| 13:30:42 | efried | if you have time. | |
| 13:30:52 | mriedem | efried: i'd need a link to a failing job first | |
| 13:30:53 | efried | teach me to fish, in other words | |
| 13:31:04 | efried | mriedem: stand by | |
| 13:31:12 | efried | (but any patch from the past three days) | |
| 13:31:19 | mriedem | ok http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/ | |
| 13:31:27 | efried | https://review.openstack.org/#/c/648653/ | |
| 13:31:46 | mriedem | looks like this is the problem http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/job-output.txt.gz#_2019-04-01_11_12_47_685033 | |
| 13:31:51 | mriedem | Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-6c6a7e2b-6e5a-4ff4-ba6a-a3aaa8d8f83a) | |
| 13:32:18 | mriedem | which probably means https://github.com/openstack-dev/devstack/commit/0d4c9c9fa37c6f47523359044e5f65629321ff6d | |
| 13:33:23 | mriedem | so if i take "Multiple possible networks found, use a Network ID to be more specific" from that nova-live-migration console log and dump it in logstash, i bet it starts around 3/29 | |
| 13:35:34 | efried | clearly I don't know how to use logstash | |
| 13:35:45 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22nova-live-migration%5C%22&from=7d | |
| 13:36:08 | mriedem | so the message filter is part of the message above, excluding anything unique like a request id | |
| 13:36:09 | mriedem | or timestamp | |
| 13:36:19 | mriedem | the 'tags' is the file the message is in, so 'console' in this case | |
| 13:36:40 | mriedem | and i threw build_name in there, which we don't normally do, but figured i would to avoid noise from falst negatives in functional jobs or something | |
| 13:36:47 | mriedem | *false | |
| 13:37:18 | efried | how did you get it to span the correct time frame? | |
| 13:37:32 | mriedem | anyway, my guess is https://review.openstack.org/#/c/601433/ because (1) devstack creates the networks we're using here and (2) it doesn't fail for tempest because tempest always provides a specific nic when creating a server, but in the evacuate tests in nova-live-migration we don't | |
| 13:37:46 | mriedem | used the dropdown at the top right | |
| 13:38:20 | mriedem | https://wiki.openstack.org/wiki/BootstrappingHour/Debugging_Gate is old but might be relevant still | |
| 13:38:27 | efried | Cool. So now a bug needs to be opened, and then registered in e-r? | |
| 13:38:31 | mriedem | yeah | |
| 13:39:19 | efried | how did you get from the error message to the patch? Did you just happen to know about that devstack change? | |
| 13:40:50 | mriedem | i didn't, but i know that devstack creates the network we use in testing, | |
| 13:41:06 | mriedem | so assumed that because we're now failing on ambiguous network errors, someone started creating a 2nd network in devstack | |
| 13:41:10 | mriedem | lo and behold someone did | |
| 13:41:49 | efried | do you want me to open the bug? | |
| 13:41:53 | mriedem | sure | |
| 13:42:11 | yonglihe | orphan patch was splited: https://review.openstack.org/#/q/topic:bug/1820802+(status:open+OR+status:merged) nova-live-migration test fail, but the test result said all test passed... | |
| 13:42:12 | efried | against nova or against devstack? | |
| 13:42:35 | mriedem | yonglihe: we're talking about that nova-live-migration failure right now | |
| 13:42:38 | mriedem | efried: i guess start with nova | |
| 13:42:52 | yonglihe | ooh, got you. | |
| 13:42:53 | mriedem | it's really a behavior change in devstack, but it looks like it's only breaking nova-live-migration right now | |
| 13:43:14 | sean-k-mooney | efried: one nit inline but yep it looks good to me. i left a +1 to recored the same | |
| 13:44:51 | mriedem | i guess our options are (1) make nova-live-migration pick a network if there are multiple or (2) add a variable to devstack to toggle whether or not it creates a shared network | |
| 13:45:17 | mriedem | #1 is more flexible to env changes, but i can see myself being really annoyed without #2 in local devstack testing if i have to always specify a network now | |
| 13:45:33 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1822605 | |
| 13:45:34 | openstack | Launchpad bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Undecided,New] | |
| 13:46:14 | mriedem | although i guess i always disable tempest in my devstack setup local.conf so i won't hit #2 | |
| 13:46:56 | efried | mriedem: okay, so how do I register this with e-r? | |
| 13:47:13 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html | |
| 13:48:02 | mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html#queries specifically | |
| 13:48:10 | sean-k-mooney | mriedem: isnt there already a tempest config option to specify the network to use | |
| 13:48:27 | mriedem | sean-k-mooney: see https://review.openstack.org/#/c/615425/ | |
| 13:48:33 | mriedem | but what's failing isn't tempest | |
| 13:53:01 | sean-k-mooney | mriedem: sorry thats too much of a context switch to figure out if the review you linked helps or not. | |
| 13:53:25 | sean-k-mooney | i need either more or less coffee but not the amount i have currently consumed | |
| 13:53:32 | mdbooth | kaisers: Just found a bug in https://review.openstack.org/#/c/648093/ which just merged. Nothing catastrophic, but definitely worthy of another follow-up. | |
| 13:55:44 | mriedem | efried: i should have a patch up shortly | |
| 13:55:46 | sean-k-mooney | mriedem: #2 sound resonable just in general as a way to auto create an env with a shared network | |
| 13:57:00 | sean-k-mooney | althouhg i often just do that via a local.sh file which you could also do in the gate job | |