| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 14:31:37 | pumaranikar | cdent, sure thanks. I will try to find more on this. Will catch up later when you are free. | |
| 14:32:03 | cdent | EmilienM: what is making these requests: http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/keystone/keystone.txt.gz#_2017-01-17_17_36_38_614 | |
| 14:33:33 | EmilienM | cdent: probably nova compute | |
| 14:33:45 | EmilienM | cdent: but probably keystone resources haven't been created yet | |
| 14:33:51 | EmilienM | "Could not find user: placement" | |
| 14:35:41 | eglynn | cdent, EmilienM: worth re-running the CI tests with debug logging? (not much detail to go on currently in the logs) | |
| 14:36:25 | cdent | is there anything in the logs anywhere that is listing the service catalog, because from the resource tracker code side, that error is what it says: I can't find the endpoint I'm looking for | |
| 14:36:38 | EmilienM | eglynn: I ran 20 CI jobs yesterday and 10 today | |
| 14:36:44 | EmilienM | eglynn: it's all consistent, it fails all the time | |
| 14:37:01 | EmilienM | cdent: not now but I can do it for debug | |
| 14:37:29 | eglynn | EmilienM: that would potentially be useful | |
| 14:37:32 | EmilienM | cdent: but puppet does it http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_19 | |
| 14:37:38 | EmilienM | 2017-01-13 21:21:19 +0000 /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure (notice): created | |
| 14:37:48 | EmilienM | which means we have a service called "placement" | |
| 14:38:53 | EmilienM | let me add some debug in the CI job | |
| 14:43:41 | cdent | EmilienM, eglynn: based on the logs, the resource tracker code is working as expected: it can't find the placement endpoint so it doesn't do the requested action. Later it tries again and still can't find the placement endpoint. | |
| 14:43:44 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 14:44:31 | EmilienM | cdent: let me add some debug in puppet logs so we can see keystone resources | |
| 14:44:47 | cdent | so something, I'm not sure what, is going wrong in the communication with keystone, unfortunately we're not actually logging much of that interaction and the error (that I can find) | |
| 14:45:11 | eglynn | EmilienM: with debug logging, we should see the endpoint creation (o lack thereof) in the keystone logs? | |
| 14:45:17 | eglynn | *or | |
| 14:46:10 | EmilienM | eglynn: yes, let me look | |
| 14:46:22 | mriedem | well, why doesn't the RT try to connect again? | |
| 14:46:27 | mriedem | first failure is here http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/nova/nova-compute.txt.gz#_2017-01-13_21_19_30_090 | |
| 14:46:38 | mriedem | because the placement endpoint isn't setup yet http://logs.openstack.org/66/420166/2/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/591bf96/logs/puppet-20170113_212224.txt.gz#_2017-01-13_21_21_19 | |
| 14:46:41 | mriedem | so that's expected | |
| 14:46:50 | mriedem | but later attempts should work | |
| 14:46:58 | cdent | mriedem: and they don't | |
| 14:47:07 | EmilienM | eglynn: well no I don't see it | |
| 14:47:17 | cdent | http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_43_51_918 | |
| 14:47:37 | mriedem | cdent: that's still before the endpoint is added in keystone | |
| 14:47:39 | mriedem | so that's expected | |
| 14:47:45 | cdent | still | |
| 14:47:48 | mriedem | anything after 21:21:19 should be fine | |
| 14:48:10 | cdent | I think I've got log confusion | |
| 14:48:29 | eglynn | agreed, if the retry logic is working as intended | |
| 14:48:30 | cdent | mriedem: the 01-13 logs are not in question | |
| 14:48:40 | cdent | it's the 01-17 | |
| 14:49:21 | EmilienM | cdent: I'm adding more debug https://review.openstack.org/421980 | |
| 14:49:39 | EmilienM | oops typo | |
| 14:49:57 | mriedem | so in the 1/17 logs the placement endpoint is registered here right? http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/puppet-20170117_173440.txt.gz#_2017-01-17_17_33_42 | |
| 14:50:02 | EmilienM | what is weird, is that it worked 5 days ago | |
| 14:50:07 | mriedem | at 17:33 | |
| 14:50:09 | openstackgerrit | Nadya Shakhat proposed openstack/nova: Make all datastamps formats equal https://review.openstack.org/421981 | |
| 14:50:39 | mriedem | and this would fail before that http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862 | |
| 14:50:44 | mriedem | at 17:31 | |
| 14:50:45 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver https://review.openstack.org/391288 | |
| 14:50:55 | EmilienM | mriedem: yes exactly | |
| 14:51:39 | mriedem | but it should work here http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_34_40_920 | |
| 14:52:17 | EmilienM | yes | |
| 14:58:31 | gcb | johnthetubaguy: need +A for the python 3 unit test https://review.openstack.org/#/c/406435/ | |
| 15:03:34 | mriedem | lyarwood: i -1ed https://review.openstack.org/#/c/421261/ just for the sheer confusion | |
| 15:12:05 | jdandrea | Is there a time limit for nova-scheduler to do its job in scheduling a server, and where is that setting (even if it's hardcoded/fixed)? | |
| 15:19:32 | cdent | there's a retry limit, isn't there? | |
| 15:20:40 | markus_z | adreznec: Got a minute? I'd like to setup the RTD job for nova-dpm and I'm curious what you did on the RTD side to enable that. | |
| 15:21:06 | lyarwood | mriedem: thanks and apologies for missing that it hadn't landed on master etc | |
| 15:22:28 | jdandrea | markus_z *nod* that's good from an overall perspective, got it. I'm wondering if there's a timeout specific to the scheduler as well, in the process of trying conf.scheduler.max_attempts (3) times. | |
| 15:23:03 | markus_z | cdent: I thought that's used for the RetryFilter. I assumed the scheduler *always* finds 0..* compute nodes. The question from above sounds like it never ends. | |
| 15:24:00 | mriedem | alaski: when you get a minute, why do we need the cell_info stuff in https://review.openstack.org/#/c/369634/ ? - doesn't that defeat the purpose of having to run discover_hosts, i.e. we want auto-registration of hosts into a cell so that people/tools don't have to run discover_hosts manually. i'm thinking in at least the single-cell case auto-register is pretty simple. | |
| 15:25:01 | cdent | jdandrea: do you want to stop sooner or less soon? | |
| 15:25:04 | jdandrea | markus_z I'm looking at https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L519 ... hmm, no timeout to speak of (that I can see). | |
| 15:25:15 | adreznec | Hey markus_z. It's been a while since I did that, but I'd be happy to help with anything I can remember | |
| 15:25:30 | jdandrea | cdent I don't want to stop sooner or less soon, neither. I am just trying to determine if there's a scheduling-specific timeout at all. | |
| 15:25:49 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V PCI Passthrough https://review.openstack.org/420614 | |
| 15:26:30 | jdandrea | cdent Looking at this I suspect there ... isn't? (a scheduler-specific timeout) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L519 | |
| 15:27:46 | jdandrea | cdent That is, there's a 60 second overall timeout to boot a server. But is there anything that puts a time limit on nova-scheduler to come up with an answer to begin with? | |
| 15:28:05 | edleafe | jdandrea: nothing that I'm aware of | |
| 15:28:19 | jdandrea | edleafe excellent, ty! | |
| 15:28:25 | jdandrea | (and that matches up with what I'm finding as well) | |
| 15:28:35 | jdandrea | I just wanted to be sure I wasn't missing something. ;) | |
| 15:30:22 | markus_z | adreznec: thanks in advance. What do you remember? :) I was wondering which account I have to use. My own? A new one? | |
| 15:30:44 | adreznec | markus_z: So on the rtd side I went through the basic registration steps (set up my account, create project, etc) w/ sphinx docs and set the URL to point at the git.o.o for the repo. | |
| 15:31:39 | cdent | EmilienM: can you ping me please when that debug job comes through with info? | |
| 15:31:50 | adreznec | Then in the advanced settings I had to configure it to use a virtualenv for builds w/ the right requirements file | |
| 15:32:05 | jdandrea | markus_z, cdent ... unless there's an oslo *messaging* timeout? https://github.com/openstack/nova/blob/4e747092bcb015303efc2ab13da98ef5ce575ec8/nova/scheduler/rpcapi.py#L127 | |
| 15:32:42 | adreznec | I think that was about it actually, the infra hook took care of the rest (notifying for rebuild, etc) | |
| 15:32:52 | EmilienM | cdent: yes | |
| 15:32:56 | EmilienM | cdent: I will | |
| 15:34:14 | markus_z | adreznec: Did you use the email address of a "functional Notes ID" for the registration? Or simply your own? | |
| 15:34:29 | openstackgerrit | Vladik Romanovsky proposed openstack/nova: libvirt: set vlan tag for macvtap on SRIOV VFs https://review.openstack.org/421186 | |
| 15:36:45 | adreznec | markus_z: I originally just used my own, then later on added our CI email address as a maintainer to the docs projects | |
| 15:37:54 | markus_z | adreznec: Ah, ok, makes sense. I'll simply start so as well. Thanks! | |
| 15:38:26 | adreznec | markus_z: Np, if you have any issues I can help with let me know | |
| 15:42:00 | cdent | edleafe, bauzas, jaypipes: I'm going to be on PTO tomorrow and friday, so if an rp/placement update is desired if someone could do it that would be great. Otherwise I'll pick it back up next week. | |
| 15:42:39 | edleafe | cdent: roger that | |
| 15:47:18 | bauzas | cdent: ack | |
| 15:48:09 | openstackgerrit | Mihail proposed openstack/nova: proposed fix for bug 1646896 - System hangs when using NFS storage backend with loopback mounts https://review.openstack.org/422041 | |
| 15:48:09 | openstack | bug 1646896 in OpenStack Compute (nova) "System hangs when using NFS storage backend with loopback mounts" [Low,Confirmed] https://launchpad.net/bugs/1646896 | |
| 15:48:15 | EmilienM | cdent: wait wait wait | |
| 15:48:23 | EmilienM | cdent: you're going on PTO? no please | |
| 15:48:25 | openstackgerrit | Vladik Romanovsky proposed openstack/nova: Adding vlans filed to Device tagging metadata https://review.openstack.org/405951 | |
| 15:48:34 | cdent | EmilienM: just for two days | |
| 15:48:51 | cdent | and I'm still here for another few hours today | |
| 15:49:03 | cdent | for you in particular | |
| 15:49:03 | cdent | because you said please | |
| 15:49:13 | bauzas | EmilienM, cdent: I missed today's discussion | |
| 15:49:22 | bauzas | anything I need to look at ? | |
| 15:49:31 | EmilienM | bauzas: https://bugs.launchpad.net/tripleo/+bug/1657467 | |
| 15:49:31 | openstack | Launchpad bug 1657467 in tripleo "placement: unable to refresh compute resource provider record" [High,Triaged] | |
| 15:49:45 | EmilienM | cdent: ♥ | |