Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-18
14:12:35 EmilienM cdent: I can show you logs, a sec
14:13:33 EmilienM cdent: here are logs from 5 days ago:
14:13:35 EmilienM http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/
14:13:43 EmilienM you can see nova placement api log, and everything works
14:14:11 EmilienM we haven't changed anything on this topic https://github.com/openstack/puppet-nova/commits/master
14:15:18 cdent EmilienM: http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_20_29_245
14:16:00 EmilienM cdent: credentials are created by Puppet, we never changed that since December
14:16:12 EmilienM let me show the logs
14:16:14 cdent I don't dispute that
14:16:31 cdent all I'm saying is that the logs you pointed to me as being successful are saying they have bad credentials
14:16:35 EmilienM http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_14
14:16:44 cdent I don't know if that means anything or not
14:16:54 EmilienM they are created at 21:21:14
14:17:02 EmilienM and your log is at 21:20
14:17:25 EmilienM if you look later, it worked: http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_21_32_490
14:17:33 EmilienM so like I said, it worked on January 13th
14:17:36 EmilienM but not now
14:17:48 cdent I get it, I'm just trying to clarify the pieces
14:18:08 EmilienM something changed in Nova between January 13th and at least 17th (I saw the thing yesterday)
14:19:31 EmilienM cdent: maybe https://github.com/openstack/nova/commit/c17772e1f202c9ffb651adc3299a1990c35766f3 ?
14:19:59 openstackgerrit Zhenyu Zheng proposed openstack/nova: Strict pattern match query parameters https://review.openstack.org/420494
14:20:29 cdent EmilienM no impact on the presence of the service. I think the problem is most likely on the keystone side, give me a chance to read through the logs more closely
14:20:58 EmilienM cdent: sure, and thanks for helping
14:21:57 eglynn the timing of the credentials creation shouldn't be a show-stopper, right, as the report will be retried since https://review.openstack.org/#/c/418590/4/nova/scheduler/client/report.py landed on Jan 11th
14:22:48 Kevin_Zheng johnthetubaguy: https://review.openstack.org/420494 rebased to the fitler whitelist patch, so it will be the 2nd of the series, please have a check when you got a sec, hope we can merge the 1st and 2nd of the series, I will keep working on the policy one and sort one torrmorow :)
14:26:39 pumaranikar cdent, again checking https://review.openstack.org/#/c/395194/ json formatter for placements. I am wondering if we can use @wsgify.middleware decorator to alter webob response to check error codes.
14:27:44 pumaranikar cdent, not sure about it. Any suggestions on it ?
14:29:46 cdent pumaranikar: I'll have to get back to you about this a bit later, digging in some log files right now. In general however I'm not sure about middleware for this. It might work. Basically you would want to check for an error response and a body which is application/json, then modify the json
14:31:37 pumaranikar cdent, sure thanks. I will try to find more on this. Will catch up later when you are free.
14:32:03 cdent EmilienM: what is making these requests: http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/keystone/keystone.txt.gz#_2017-01-17_17_36_38_614
14:33:33 EmilienM cdent: probably nova compute
14:33:45 EmilienM cdent: but probably keystone resources haven't been created yet
14:33:51 EmilienM "Could not find user: placement"
14:35:41 eglynn cdent, EmilienM: worth re-running the CI tests with debug logging? (not much detail to go on currently in the logs)
14:36:25 cdent is there anything in the logs anywhere that is listing the service catalog, because from the resource tracker code side, that error is what it says: I can't find the endpoint I'm looking for
14:36:38 EmilienM eglynn: I ran 20 CI jobs yesterday and 10 today
14:36:44 EmilienM eglynn: it's all consistent, it fails all the time
14:37:01 EmilienM cdent: not now but I can do it for debug
14:37:29 eglynn EmilienM: that would potentially be useful
14:37:32 EmilienM cdent: but puppet does it http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_19
14:37:38 EmilienM 2017-01-13 21:21:19 +0000 /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure (notice): created
14:37:48 EmilienM which means we have a service called "placement"
14:38:53 EmilienM let me add some debug in the CI job
14:43:41 cdent EmilienM, eglynn: based on the logs, the resource tracker code is working as expected: it can't find the placement endpoint so it doesn't do the requested action. Later it tries again and still can't find the placement endpoint.
14:43:44 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209
14:44:31 EmilienM cdent: let me add some debug in puppet logs so we can see keystone resources
14:44:47 cdent so something, I'm not sure what, is going wrong in the communication with keystone, unfortunately we're not actually logging much of that interaction and the error (that I can find)
14:45:11 eglynn EmilienM: with debug logging, we should see the endpoint creation (o lack thereof) in the keystone logs?
14:45:17 eglynn *or
14:46:10 EmilienM eglynn: yes, let me look
14:46:22 mriedem well, why doesn't the RT try to connect again?
14:46:27 mriedem first failure is here http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_19_30_090
14:46:38 mriedem because the placement endpoint isn't setup yet http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_19
14:46:41 mriedem so that's expected
14:46:50 mriedem but later attempts should work
14:46:58 cdent mriedem: and they don't
14:47:07 EmilienM eglynn: well no I don't see it
14:47:17 cdent http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_43_51_918
14:47:37 mriedem cdent: that's still before the endpoint is added in keystone
14:47:39 mriedem so that's expected
14:47:45 cdent still
14:47:48 mriedem anything after 21:21:19 should be fine
14:48:10 cdent I think I've got log confusion
14:48:29 eglynn agreed, if the retry logic is working as intended
14:48:30 cdent mriedem: the 01-13 logs are not in question
14:48:40 cdent it's the 01-17
14:49:21 EmilienM cdent: I'm adding more debug https://review.openstack.org/421980
14:49:39 EmilienM oops typo
14:49:57 mriedem so in the 1/17 logs the placement endpoint is registered here right? http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/puppet-20170117_173440.txt.gz#_2017-01-17_17_33_42
14:50:02 EmilienM what is weird, is that it worked 5 days ago
14:50:07 mriedem at 17:33
14:50:09 openstackgerrit Nadya Shakhat proposed openstack/nova: Make all datastamps formats equal https://review.openstack.org/421981
14:50:39 mriedem and this would fail before that http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862
14:50:44 mriedem at 17:31
14:50:45 openstackgerrit Eric Fried proposed openstack/nova: Introduce base PowerVM driver https://review.openstack.org/391288
14:50:55 EmilienM mriedem: yes exactly
14:51:39 mriedem but it should work here http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_34_40_920
14:52:17 EmilienM yes
14:58:31 gcb johnthetubaguy: need +A for the python 3 unit test https://review.openstack.org/#/c/406435/
15:03:34 mriedem lyarwood: i -1ed https://review.openstack.org/#/c/421261/ just for the sheer confusion
15:12:05 jdandrea Is there a time limit for nova-scheduler to do its job in scheduling a server, and where is that setting (even if it's hardcoded/fixed)?
15:19:32 cdent there's a retry limit, isn't there?
15:20:40 markus_z adreznec: Got a minute? I'd like to setup the RTD job for nova-dpm and I'm curious what you did on the RTD side to enable that.
15:21:06 lyarwood mriedem: thanks and apologies for missing that it hadn't landed on master etc
15:22:28 jdandrea markus_z *nod* that's good from an overall perspective, got it. I'm wondering if there's a timeout specific to the scheduler as well, in the process of trying conf.scheduler.max_attempts (3) times.
15:23:03 markus_z cdent: I thought that's used for the RetryFilter. I assumed the scheduler *always* finds 0..* compute nodes. The question from above sounds like it never ends.
15:24:00 mriedem alaski: when you get a minute, why do we need the cell_info stuff in https://review.openstack.org/#/c/369634/ ? - doesn't that defeat the purpose of having to run discover_hosts, i.e. we want auto-registration of hosts into a cell so that people/tools don't have to run discover_hosts manually. i'm thinking in at least the single-cell case auto-register is pretty simple.
15:25:01 cdent jdandrea: do you want to stop sooner or less soon?
15:25:04 jdandrea markus_z I'm looking at https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L519 ... hmm, no timeout to speak of (that I can see).
15:25:15 adreznec Hey markus_z. It's been a while since I did that, but I'd be happy to help with anything I can remember
15:25:30 jdandrea cdent I don't want to stop sooner or less soon, neither. I am just trying to determine if there's a scheduling-specific timeout at all.
15:25:49 openstackgerrit Claudiu Belu proposed openstack/nova: Hyper-V PCI Passthrough https://review.openstack.org/420614
15:26:30 jdandrea cdent Looking at this I suspect there ... isn't? (a scheduler-specific timeout) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L519
15:27:46 jdandrea cdent That is, there's a 60 second overall timeout to boot a server. But is there anything that puts a time limit on nova-scheduler to come up with an answer to begin with?
15:28:05 edleafe jdandrea: nothing that I'm aware of
15:28:19 jdandrea edleafe excellent, ty!
15:28:25 jdandrea (and that matches up with what I'm finding as well)
15:28:35 jdandrea I just wanted to be sure I wasn't missing something. ;)
15:30:22 markus_z adreznec: thanks in advance. What do you remember? :) I was wondering which account I have to use. My own? A new one?
15:30:44 adreznec markus_z: So on the rtd side I went through the basic registration steps (set up my account, create project, etc) w/ sphinx docs and set the URL to point at the git.o.o for the repo.

Earlier   Later