| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 16:12:58 | bauzas | EmilienM: I got a shit ton of "placement API endpoint not found" even after keystone is started http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_38_02_282 | |
| 16:13:14 | markus_z | adreznec: I think I have it up and running http://nova-dpm.readthedocs.io/en/latest/ Thanks again for your time! | |
| 16:13:26 | bauzas | EmilienM: correct me if I'm wrong but you told me that keystone is ready by 17:35 ? | |
| 16:13:47 | bauzas | oh, even 17:33 | |
| 16:13:49 | bauzas | "(17:02:45) EmilienM: so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31" | |
| 16:13:56 | bauzas | that's... weird | |
| 16:14:09 | adreznec | markus_z: Awesome, looks good! | |
| 16:14:15 | bauzas | EmilienM: because it seems we litterally fail to find the correct endpoint even after that | |
| 16:14:17 | openstackgerrit | Eric Fried proposed openstack/nova: WIP(2): Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 16:14:48 | bauzas | EmilienM: ie. keystoneclient returns an EndpointNotFound exception | |
| 16:14:54 | EmilienM | bauzas: 17h33 | |
| 16:15:10 | EmilienM | bauzas: let's see how my endpoint is structured | |
| 16:15:13 | EmilienM | maybe it's wrong | |
| 16:16:23 | EmilienM | bauzas: http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/ | |
| 16:16:50 | EmilienM | cdent: ^ | |
| 16:17:10 | bauzas | EmilienM: maybe the placement service failed to start | |
| 16:17:25 | bauzas | because the URL seems correct | |
| 16:17:27 | EmilienM | bauzas: apache would have failed | |
| 16:17:32 | EmilienM | let's see apache logs | |
| 16:17:39 | cdent | the wsgi process shows up in ps.txt | |
| 16:18:05 | EmilienM | yeah | |
| 16:18:14 | bauzas | I mean, something is wrong with that HTTP call | |
| 16:18:20 | cdent | I'll need to review the wsgi config, but I'm surprised to see: http://127.0.0.1:8778/placement instead of either | |
| 16:18:36 | bauzas | so, either it's the service catalog which is incorrect or the placement service wsgi stack not fully up | |
| 16:18:38 | cdent | http://127.0.0.1/placement or http://127.0.0.1:8778/ | |
| 16:18:42 | cdent | however | |
| 16:18:50 | bauzas | cdent: you're correct, that's the reason | |
| 16:19:05 | bauzas | EmilienM: I need to see your apache conf | |
| 16:19:10 | cdent | except that the exception message is that the endpoint is never found | |
| 16:19:12 | bauzas | because I guess the root cause is ^ | |
| 16:19:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:19:24 | bauzas | cdent: yeah, because it could fail | |
| 16:19:24 | cdent | and the placement-api logs never start, which suggests the wsgi app is never contacted | |
| 16:19:30 | bauzas | I mean getting the VirtualHost | |
| 16:19:35 | bauzas | cdent: exactly | |
| 16:19:46 | bauzas | FWIW http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz | |
| 16:19:56 | openstackgerrit | Scott DAngelo proposed openstack/nova: Fix Nova to allow using cinder v3 endpoint https://review.openstack.org/422078 | |
| 16:20:23 | bauzas | I think this configuration is incorrect | |
| 16:20:56 | cdent | bauzas: except that is the same as a previous working config, I _think_ | |
| 16:21:02 | cdent | but that ought to be confirmed | |
| 16:21:05 | EmilienM | so if I look at Puppet CI this morning it seems to work but not in TripleO | |
| 16:21:20 | EmilienM | I don't see placement-api logs http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/nova/ | |
| 16:21:44 | EmilienM | but I know the vhost has been created, see http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/httpd/ | |
| 16:22:20 | cdent | the critical point on the placement-api logs is that they will not exist until at least one request is made to the placement API | |
| 16:22:27 | cdent | whether or not it is up | |
| 16:22:37 | EmilienM | ok | |
| 16:22:59 | EmilienM | eg: when you spawn a VM, I guess | |
| 16:23:06 | bauzas | yup | |
| 16:23:15 | bauzas | https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template is the devstack config | |
| 16:23:29 | bauzas | there is an alias that is out of the virtualhost | |
| 16:23:32 | EmilienM | so in tripleo, we spawn overcloud nodes with nova/ironic and i don't see placement logs: http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/undercloud/var/log/nova/ | |
| 16:23:46 | bauzas | hence why you get either :8778/ or :80/placement | |
| 16:24:08 | bauzas | we share the same wsgi process app, but that's two different directives | |
| 16:24:36 | EmilienM | because we use HAproxy and 80 is used for horizon | |
| 16:24:39 | bauzas | EmilienM: yeah, got the problem | |
| 16:24:52 | bauzas | EmilienM: in devstack, we alias on / for the vhost https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template#L8 | |
| 16:25:12 | bauzas | EmilienM: in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz you alias on /placement | |
| 16:25:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:25:22 | bauzas | but you don't provide the location directive | |
| 16:25:24 | EmilienM | yep | |
| 16:25:33 | cdent | if that's the case why did it work before? | |
| 16:25:48 | EmilienM | yeah and I just noticed it works in some CI jobs this morning | |
| 16:25:59 | EmilienM | let me grab the more recent one | |
| 16:26:23 | EmilienM | http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/ | |
| 16:26:39 | cdent | I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? | |
| 16:28:05 | _gryf | seems similar to https://bugs.launchpad.net/nova/+bug/1656075 | |
| 16:28:05 | openstack | Launchpad bug 1656075 in OpenStack Compute (nova) "DiscoveryFailure when trying to get resource providers from the scheduler report client" [Low,Confirmed] - Assigned to Roman Dobosz (roman-dobosz) | |
| 16:28:13 | bauzas | EmilienM: looks all good for that one http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/nova-compute.txt.gz#_2017-01-18_15_44_36_293 | |
| 16:28:50 | EmilienM | ok so good news: it works in Puppet OpenStack CI this morning. Now trying to figure why it doesn't work in TripleO | |
| 16:29:03 | EmilienM | bauzas: I swear it didn't work yesterday | |
| 16:29:08 | bauzas | :) | |
| 16:29:14 | bauzas | I haven't touched anything ! | |
| 16:29:15 | EmilienM | if you look at https://bugs.launchpad.net/tripleo/+bug/1657467 | |
| 16:29:15 | openstack | Launchpad bug 1657467 in tripleo "placement: unable to refresh compute resource provider record" [High,Triaged] | |
| 16:29:21 | EmilienM | logs are from yesterday and it didn't work | |
| 16:29:38 | bauzas | I'd look at the vhost config | |
| 16:29:41 | mriedem | melwitt: yeah good point, nova.compute.manager calling nova.compute.api to call compute rpcapi to cast the | |
| 16:29:42 | bauzas | and compare both | |
| 16:29:45 | mriedem | 'stop' back to the same compute | |
| 16:29:50 | EmilienM | bauzas: we didn't change it | |
| 16:30:34 | mriedem | melwitt: i was thinking if that already was in place the compute manager could call the compute api to register with the cell mapping (in the single cell case) in the api db, but that's not really going to work | |
| 16:31:01 | mriedem | unless the compute rpcapi did a cast to something else that has access to the api, like conductor | |
| 16:31:03 | cdent | EmilienM, bauzas: I'll say again: [t 43Pk] | |
| 16:31:03 | purplerbot | <cdent> I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? [2017-01-18 16:26:39] [n 43Pk] | |
| 16:31:47 | bauzas | cdent: well, I agree with you on the point that EndpointNotFound is kinda obscure to me | |
| 16:32:01 | bauzas | cdent: I'm supposing that keystone failed finding the endpoint | |
| 16:32:14 | bauzas | but the endpoint can be / | |
| 16:32:21 | sdague | bauzas: it's not that keystone failed | |
| 16:32:21 | cdent | it happens at two different places in the keystoneauth1 stack | |
| 16:32:29 | cdent | once when actually making a request and filtering the results | |
| 16:32:35 | melwitt | mriedem: I think if something in compute queried cell mapping through the object, it would go through conductor and be able to read from the api db | |
| 16:32:35 | cdent | but also when trying to construct a url | |
| 16:32:35 | bauzas | sdague: I'm all ears :) | |
| 16:32:46 | sdague | it's that the resolution of the endpoint from the token ended up with something unknown | |
| 16:33:16 | bauzas | a-ha | |
| 16:34:54 | bauzas | sdague: I don't see any problem with http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/ | |
| 16:35:02 | mriedem | melwitt: hmm, so like, list cell mappings, if one, get that one and register a host with it? | |
| 16:35:12 | mriedem | s/a host/my host/ | |
| 16:35:15 | bauzas | sdague: except the fact that they call http://ip:8778/placement | |
| 16:35:16 | cdent | sdague: by "resolution" do you mean looking at what's at the endpoint, for finding the endpoint in what's in the token? | |