| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 14:03:55 | gmann | alex_xu: johnthetubaguy need to go to bed, ll check logs and patches tomorrow. | |
| 14:04:02 | alex_xu | gmann: johnthetubaguy ok, got it, that works for me | |
| 14:04:11 | alex_xu | gmann: see you tomorrow | |
| 14:04:32 | owalsh | EmilienM: just fishing for something that might differ from running locally | |
| 14:09:06 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell_and_hosts https://review.openstack.org/421420 | |
| 14:09:06 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_instances https://review.openstack.org/421404 | |
| 14:09:07 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 simple_cell_setup https://review.openstack.org/421333 | |
| 14:09:07 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 verify_instance https://review.openstack.org/421436 | |
| 14:09:08 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 delete_cell https://review.openstack.org/421459 | |
| 14:09:08 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell0 https://review.openstack.org/421384 | |
| 14:09:09 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 list_cells https://review.openstack.org/421458 | |
| 14:09:09 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 discover_hosts https://review.openstack.org/421442 | |
| 14:09:10 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 create_cell https://review.openstack.org/421444 | |
| 14:09:30 | eglynn | cdent: are the placement API logs normally found under /logs/nova ? | |
| 14:10:01 | cdent | eglynn: the logs only show up after it has actually had its first request | |
| 14:10:12 | cdent | so if the resource-tracker can't see the service, it will never make a request | |
| 14:10:16 | cdent | and never make a log entry | |
| 14:10:46 | eglynn | a-ha, /me was wondering why none to be found under http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova | |
| 14:11:13 | cdent | it's one of the side effects of being hosted under mod-wsgi | |
| 14:11:52 | jaypipes | eglynn: heyo, how are you Eoghan? :) | |
| 14:11:55 | takashin | alex_xu: Could you review https://review.openstack.org/#/c/408955/ again? | |
| 14:12:07 | eglynn | jaypipes: all good :) ... and you? | |
| 14:12:11 | EmilienM | eglynn, cdent: yes logs are in /var/log/nova | |
| 14:12:12 | cdent | eglynn, EmilienM : the presenting error "The placement API endpoint not found." suggests either that the rt can't reach keystone, or that keystone is not yet up to date | |
| 14:12:15 | EmilienM | they always were at least | |
| 14:12:24 | jaypipes | eglynn: busy as always, which is good :) | |
| 14:12:28 | EmilienM | cdent: it worked 2 days ago :( | |
| 14:12:31 | eglynn | indeed | |
| 14:12:35 | EmilienM | cdent: I can show you logs, a sec | |
| 14:13:33 | EmilienM | cdent: here are logs from 5 days ago: | |
| 14:13:35 | EmilienM | http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/ | |
| 14:13:43 | EmilienM | you can see nova placement api log, and everything works | |
| 14:14:11 | EmilienM | we haven't changed anything on this topic https://github.com/openstack/puppet-nova/commits/master | |
| 14:15:18 | cdent | EmilienM: http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_20_29_245 | |
| 14:16:00 | EmilienM | cdent: credentials are created by Puppet, we never changed that since December | |
| 14:16:12 | EmilienM | let me show the logs | |
| 14:16:14 | cdent | I don't dispute that | |
| 14:16:31 | cdent | all I'm saying is that the logs you pointed to me as being successful are saying they have bad credentials | |
| 14:16:35 | EmilienM | http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_14 | |
| 14:16:44 | cdent | I don't know if that means anything or not | |
| 14:16:54 | EmilienM | they are created at 21:21:14 | |
| 14:17:02 | EmilienM | and your log is at 21:20 | |
| 14:17:25 | EmilienM | if you look later, it worked: http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_21_32_490 | |
| 14:17:33 | EmilienM | so like I said, it worked on January 13th | |
| 14:17:36 | EmilienM | but not now | |
| 14:17:48 | cdent | I get it, I'm just trying to clarify the pieces | |
| 14:18:08 | EmilienM | something changed in Nova between January 13th and at least 17th (I saw the thing yesterday) | |
| 14:19:31 | EmilienM | cdent: maybe https://github.com/openstack/nova/commit/c17772e1f202c9ffb651adc3299a1990c35766f3 ? | |
| 14:19:59 | openstackgerrit | Zhenyu Zheng proposed openstack/nova: Strict pattern match query parameters https://review.openstack.org/420494 | |
| 14:20:29 | cdent | EmilienM no impact on the presence of the service. I think the problem is most likely on the keystone side, give me a chance to read through the logs more closely | |
| 14:20:58 | EmilienM | cdent: sure, and thanks for helping | |
| 14:21:57 | eglynn | the timing of the credentials creation shouldn't be a show-stopper, right, as the report will be retried since https://review.openstack.org/#/c/418590/4/nova/scheduler/client/report.py landed on Jan 11th | |
| 14:22:48 | Kevin_Zheng | johnthetubaguy: https://review.openstack.org/420494 rebased to the fitler whitelist patch, so it will be the 2nd of the series, please have a check when you got a sec, hope we can merge the 1st and 2nd of the series, I will keep working on the policy one and sort one torrmorow :) | |
| 14:26:39 | pumaranikar | cdent, again checking https://review.openstack.org/#/c/395194/ json formatter for placements. I am wondering if we can use @wsgify.middleware decorator to alter webob response to check error codes. | |
| 14:27:44 | pumaranikar | cdent, not sure about it. Any suggestions on it ? | |
| 14:29:46 | cdent | pumaranikar: I'll have to get back to you about this a bit later, digging in some log files right now. In general however I'm not sure about middleware for this. It might work. Basically you would want to check for an error response and a body which is application/json, then modify the json | |
| 14:31:37 | pumaranikar | cdent, sure thanks. I will try to find more on this. Will catch up later when you are free. | |
| 14:32:03 | cdent | EmilienM: what is making these requests: http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/keystone/keystone.txt.gz#_2017-01-17_17_36_38_614 | |
| 14:33:33 | EmilienM | cdent: probably nova compute | |
| 14:33:45 | EmilienM | cdent: but probably keystone resources haven't been created yet | |
| 14:33:51 | EmilienM | "Could not find user: placement" | |
| 14:35:41 | eglynn | cdent, EmilienM: worth re-running the CI tests with debug logging? (not much detail to go on currently in the logs) | |
| 14:36:25 | cdent | is there anything in the logs anywhere that is listing the service catalog, because from the resource tracker code side, that error is what it says: I can't find the endpoint I'm looking for | |
| 14:36:38 | EmilienM | eglynn: I ran 20 CI jobs yesterday and 10 today | |
| 14:36:44 | EmilienM | eglynn: it's all consistent, it fails all the time | |
| 14:37:01 | EmilienM | cdent: not now but I can do it for debug | |
| 14:37:29 | eglynn | EmilienM: that would potentially be useful | |
| 14:37:32 | EmilienM | cdent: but puppet does it http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_19 | |
| 14:37:38 | EmilienM | 2017-01-13 21:21:19 +0000 /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure (notice): created | |
| 14:37:48 | EmilienM | which means we have a service called "placement" | |
| 14:38:53 | EmilienM | let me add some debug in the CI job | |
| 14:43:41 | cdent | EmilienM, eglynn: based on the logs, the resource tracker code is working as expected: it can't find the placement endpoint so it doesn't do the requested action. Later it tries again and still can't find the placement endpoint. | |
| 14:43:44 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 14:44:31 | EmilienM | cdent: let me add some debug in puppet logs so we can see keystone resources | |
| 14:44:47 | cdent | so something, I'm not sure what, is going wrong in the communication with keystone, unfortunately we're not actually logging much of that interaction and the error (that I can find) | |
| 14:45:11 | eglynn | EmilienM: with debug logging, we should see the endpoint creation (o lack thereof) in the keystone logs? | |
| 14:45:17 | eglynn | *or | |
| 14:46:10 | EmilienM | eglynn: yes, let me look | |
| 14:46:22 | mriedem | well, why doesn't the RT try to connect again? | |
| 14:46:27 | mriedem | first failure is here http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_19_30_090 | |
| 14:46:38 | mriedem | because the placement endpoint isn't setup yet http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_19 | |
| 14:46:41 | mriedem | so that's expected | |
| 14:46:50 | mriedem | but later attempts should work | |
| 14:46:58 | cdent | mriedem: and they don't | |
| 14:47:07 | EmilienM | eglynn: well no I don't see it | |
| 14:47:17 | cdent | http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_43_51_918 | |
| 14:47:37 | mriedem | cdent: that's still before the endpoint is added in keystone | |
| 14:47:39 | mriedem | so that's expected | |
| 14:47:45 | cdent | still | |
| 14:47:48 | mriedem | anything after 21:21:19 should be fine | |
| 14:48:10 | cdent | I think I've got log confusion | |
| 14:48:29 | eglynn | agreed, if the retry logic is working as intended | |
| 14:48:30 | cdent | mriedem: the 01-13 logs are not in question | |
| 14:48:40 | cdent | it's the 01-17 | |
| 14:49:21 | EmilienM | cdent: I'm adding more debug https://review.openstack.org/421980 | |
| 14:49:39 | EmilienM | oops typo | |
| 14:49:57 | mriedem | so in the 1/17 logs the placement endpoint is registered here right? http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/puppet-20170117_173440.txt.gz#_2017-01-17_17_33_42 | |
| 14:50:02 | EmilienM | what is weird, is that it worked 5 days ago | |
| 14:50:07 | mriedem | at 17:33 | |
| 14:50:09 | openstackgerrit | Nadya Shakhat proposed openstack/nova: Make all datastamps formats equal https://review.openstack.org/421981 | |