| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 13:49:49 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 13:50:02 | EmilienM | it tries in loop though, but it doesn't work | |
| 13:50:15 | EmilienM | I tried to reproduce it locally but it worked, I'm a bit confused why it fails in CI | |
| 13:50:20 | EmilienM | I can provide logs | |
| 13:50:30 | EmilienM | http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_45_52_920 | |
| 13:50:49 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 13:50:50 | EmilienM | cdent, bauzas ^ just fyi | |
| 13:51:59 | owalsh | EmilienM: hrmm, so it would suggest a timing issue, but it tries in a loop... for how long? | |
| 13:52:26 | EmilienM | owalsh: I don't think it's a timing issue this time but let me check | |
| 13:52:44 | EmilienM | owalsh: if you remember, the first time I reported this issue was because we didn't have this loop than dansmith did | |
| 13:52:54 | EmilienM | but now we have the loop and it still fails to register | |
| 13:53:39 | markus_z | mriedem: Did I clear up your doubts on https://review.openstack.org/#/c/347471/ ? The test hook behaves as expected IMO. | |
| 13:54:14 | EmilienM | owalsh: I would like to find in logs where it tries to register is and the trace of failure | |
| 13:54:22 | EmilienM | owalsh: "Unable to refresh my resource provider record" is not enough | |
| 13:56:26 | EmilienM | owalsh: I'm creating a launchpad bug | |
| 14:00:03 | EmilienM | owalsh: do you see something weird in the nova.conf? https://paste.fedoraproject.org/529703/14847479/ | |
| 14:00:49 | gmann | johnthetubaguy: alex_xu Kevin_Zheng +1 on putting sort_key one last on series | |
| 14:01:04 | openstack | Launchpad bug 1657467 in OpenStack Compute (nova) "placement: unable to refresh compute resource provider record" [Undecided,New] | |
| 14:01:04 | EmilienM | owalsh: https://bugs.launchpad.net/nova/+bug/1657467 | |
| 14:01:23 | owalsh | EmilienM: auth_urls is IPv6 loopback... does that work??? | |
| 14:01:47 | EmilienM | owalsh: of course, why not? | |
| 14:02:04 | alex_xu | gmann: johnthetubaguy I guess for now, the policy one have more thing to do. maybe the policy one is last one | |
| 14:02:34 | EmilienM | owalsh: it works for all services like tha. | |
| 14:02:38 | EmilienM | that* | |
| 14:03:00 | gmann | alex_xu: but, sort_key ignore list seems controversial which might take time | |
| 14:03:13 | johnthetubaguy | alex_xu: yeah, what gmann said | |
| 14:03:55 | gmann | alex_xu: johnthetubaguy need to go to bed, ll check logs and patches tomorrow. | |
| 14:04:02 | alex_xu | gmann: johnthetubaguy ok, got it, that works for me | |
| 14:04:11 | alex_xu | gmann: see you tomorrow | |
| 14:04:32 | owalsh | EmilienM: just fishing for something that might differ from running locally | |
| 14:09:06 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell_and_hosts https://review.openstack.org/421420 | |
| 14:09:06 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_instances https://review.openstack.org/421404 | |
| 14:09:07 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 simple_cell_setup https://review.openstack.org/421333 | |
| 14:09:07 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 verify_instance https://review.openstack.org/421436 | |
| 14:09:08 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 delete_cell https://review.openstack.org/421459 | |
| 14:09:08 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell0 https://review.openstack.org/421384 | |
| 14:09:09 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 list_cells https://review.openstack.org/421458 | |
| 14:09:09 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 discover_hosts https://review.openstack.org/421442 | |
| 14:09:10 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 create_cell https://review.openstack.org/421444 | |
| 14:09:30 | eglynn | cdent: are the placement API logs normally found under /logs/nova ? | |
| 14:10:01 | cdent | eglynn: the logs only show up after it has actually had its first request | |
| 14:10:12 | cdent | so if the resource-tracker can't see the service, it will never make a request | |
| 14:10:16 | cdent | and never make a log entry | |
| 14:10:46 | eglynn | a-ha, /me was wondering why none to be found under http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova | |
| 14:11:13 | cdent | it's one of the side effects of being hosted under mod-wsgi | |
| 14:11:52 | jaypipes | eglynn: heyo, how are you Eoghan? :) | |
| 14:11:55 | takashin | alex_xu: Could you review https://review.openstack.org/#/c/408955/ again? | |
| 14:12:07 | eglynn | jaypipes: all good :) ... and you? | |
| 14:12:11 | EmilienM | eglynn, cdent: yes logs are in /var/log/nova | |
| 14:12:12 | cdent | eglynn, EmilienM : the presenting error "The placement API endpoint not found." suggests either that the rt can't reach keystone, or that keystone is not yet up to date | |
| 14:12:15 | EmilienM | they always were at least | |
| 14:12:24 | jaypipes | eglynn: busy as always, which is good :) | |
| 14:12:28 | EmilienM | cdent: it worked 2 days ago :( | |
| 14:12:31 | eglynn | indeed | |
| 14:12:35 | EmilienM | cdent: I can show you logs, a sec | |
| 14:13:33 | EmilienM | cdent: here are logs from 5 days ago: | |
| 14:13:35 | EmilienM | http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/ | |
| 14:13:43 | EmilienM | you can see nova placement api log, and everything works | |
| 14:14:11 | EmilienM | we haven't changed anything on this topic https://github.com/openstack/puppet-nova/commits/master | |
| 14:15:18 | cdent | EmilienM: http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_20_29_245 | |
| 14:16:00 | EmilienM | cdent: credentials are created by Puppet, we never changed that since December | |
| 14:16:12 | EmilienM | let me show the logs | |
| 14:16:14 | cdent | I don't dispute that | |
| 14:16:31 | cdent | all I'm saying is that the logs you pointed to me as being successful are saying they have bad credentials | |
| 14:16:35 | EmilienM | http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_14 | |
| 14:16:44 | cdent | I don't know if that means anything or not | |
| 14:16:54 | EmilienM | they are created at 21:21:14 | |
| 14:17:02 | EmilienM | and your log is at 21:20 | |
| 14:17:25 | EmilienM | if you look later, it worked: http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_21_32_490 | |
| 14:17:33 | EmilienM | so like I said, it worked on January 13th | |
| 14:17:36 | EmilienM | but not now | |
| 14:17:48 | cdent | I get it, I'm just trying to clarify the pieces | |
| 14:18:08 | EmilienM | something changed in Nova between January 13th and at least 17th (I saw the thing yesterday) | |
| 14:19:31 | EmilienM | cdent: maybe https://github.com/openstack/nova/commit/c17772e1f202c9ffb651adc3299a1990c35766f3 ? | |
| 14:19:59 | openstackgerrit | Zhenyu Zheng proposed openstack/nova: Strict pattern match query parameters https://review.openstack.org/420494 | |
| 14:20:29 | cdent | EmilienM no impact on the presence of the service. I think the problem is most likely on the keystone side, give me a chance to read through the logs more closely | |
| 14:20:58 | EmilienM | cdent: sure, and thanks for helping | |
| 14:21:57 | eglynn | the timing of the credentials creation shouldn't be a show-stopper, right, as the report will be retried since https://review.openstack.org/#/c/418590/4/nova/scheduler/client/report.py landed on Jan 11th | |
| 14:22:48 | Kevin_Zheng | johnthetubaguy: https://review.openstack.org/420494 rebased to the fitler whitelist patch, so it will be the 2nd of the series, please have a check when you got a sec, hope we can merge the 1st and 2nd of the series, I will keep working on the policy one and sort one torrmorow :) | |
| 14:26:39 | pumaranikar | cdent, again checking https://review.openstack.org/#/c/395194/ json formatter for placements. I am wondering if we can use @wsgify.middleware decorator to alter webob response to check error codes. | |
| 14:27:44 | pumaranikar | cdent, not sure about it. Any suggestions on it ? | |
| 14:29:46 | cdent | pumaranikar: I'll have to get back to you about this a bit later, digging in some log files right now. In general however I'm not sure about middleware for this. It might work. Basically you would want to check for an error response and a body which is application/json, then modify the json | |
| 14:31:37 | pumaranikar | cdent, sure thanks. I will try to find more on this. Will catch up later when you are free. | |
| 14:32:03 | cdent | EmilienM: what is making these requests: http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/keystone/keystone.txt.gz#_2017-01-17_17_36_38_614 | |
| 14:33:33 | EmilienM | cdent: probably nova compute | |
| 14:33:45 | EmilienM | cdent: but probably keystone resources haven't been created yet | |
| 14:33:51 | EmilienM | "Could not find user: placement" | |
| 14:35:41 | eglynn | cdent, EmilienM: worth re-running the CI tests with debug logging? (not much detail to go on currently in the logs) | |
| 14:36:25 | cdent | is there anything in the logs anywhere that is listing the service catalog, because from the resource tracker code side, that error is what it says: I can't find the endpoint I'm looking for | |
| 14:36:38 | EmilienM | eglynn: I ran 20 CI jobs yesterday and 10 today | |
| 14:36:44 | EmilienM | eglynn: it's all consistent, it fails all the time | |
| 14:37:01 | EmilienM | cdent: not now but I can do it for debug | |
| 14:37:29 | eglynn | EmilienM: that would potentially be useful | |
| 14:37:32 | EmilienM | cdent: but puppet does it http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_19 | |
| 14:37:38 | EmilienM | 2017-01-13 21:21:19 +0000 /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure (notice): created | |
| 14:37:48 | EmilienM | which means we have a service called "placement" | |
| 14:38:53 | EmilienM | let me add some debug in the CI job | |
| 14:43:41 | cdent | EmilienM, eglynn: based on the logs, the resource tracker code is working as expected: it can't find the placement endpoint so it doesn't do the requested action. Later it tries again and still can't find the placement endpoint. | |
| 14:43:44 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 14:44:31 | EmilienM | cdent: let me add some debug in puppet logs so we can see keystone resources | |