| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 16:18:20 | cdent | I'll need to review the wsgi config, but I'm surprised to see: http://127.0.0.1:8778/placement instead of either | |
| 16:18:36 | bauzas | so, either it's the service catalog which is incorrect or the placement service wsgi stack not fully up | |
| 16:18:38 | cdent | http://127.0.0.1/placement or http://127.0.0.1:8778/ | |
| 16:18:42 | cdent | however | |
| 16:18:50 | bauzas | cdent: you're correct, that's the reason | |
| 16:19:05 | bauzas | EmilienM: I need to see your apache conf | |
| 16:19:10 | cdent | except that the exception message is that the endpoint is never found | |
| 16:19:12 | bauzas | because I guess the root cause is ^ | |
| 16:19:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:19:24 | cdent | and the placement-api logs never start, which suggests the wsgi app is never contacted | |
| 16:19:24 | bauzas | cdent: yeah, because it could fail | |
| 16:19:30 | bauzas | I mean getting the VirtualHost | |
| 16:19:35 | bauzas | cdent: exactly | |
| 16:19:46 | bauzas | FWIW http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz | |
| 16:19:56 | openstackgerrit | Scott DAngelo proposed openstack/nova: Fix Nova to allow using cinder v3 endpoint https://review.openstack.org/422078 | |
| 16:20:23 | bauzas | I think this configuration is incorrect | |
| 16:20:56 | cdent | bauzas: except that is the same as a previous working config, I _think_ | |
| 16:21:02 | cdent | but that ought to be confirmed | |
| 16:21:05 | EmilienM | so if I look at Puppet CI this morning it seems to work but not in TripleO | |
| 16:21:20 | EmilienM | I don't see placement-api logs http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/nova/ | |
| 16:21:44 | EmilienM | but I know the vhost has been created, see http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/httpd/ | |
| 16:22:20 | cdent | the critical point on the placement-api logs is that they will not exist until at least one request is made to the placement API | |
| 16:22:27 | cdent | whether or not it is up | |
| 16:22:37 | EmilienM | ok | |
| 16:22:59 | EmilienM | eg: when you spawn a VM, I guess | |
| 16:23:06 | bauzas | yup | |
| 16:23:15 | bauzas | https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template is the devstack config | |
| 16:23:29 | bauzas | there is an alias that is out of the virtualhost | |
| 16:23:32 | EmilienM | so in tripleo, we spawn overcloud nodes with nova/ironic and i don't see placement logs: http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/undercloud/var/log/nova/ | |
| 16:23:46 | bauzas | hence why you get either :8778/ or :80/placement | |
| 16:24:08 | bauzas | we share the same wsgi process app, but that's two different directives | |
| 16:24:36 | EmilienM | because we use HAproxy and 80 is used for horizon | |
| 16:24:39 | bauzas | EmilienM: yeah, got the problem | |
| 16:24:52 | bauzas | EmilienM: in devstack, we alias on / for the vhost https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template#L8 | |
| 16:25:12 | bauzas | EmilienM: in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz you alias on /placement | |
| 16:25:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:25:22 | bauzas | but you don't provide the location directive | |
| 16:25:24 | EmilienM | yep | |
| 16:25:33 | cdent | if that's the case why did it work before? | |
| 16:25:48 | EmilienM | yeah and I just noticed it works in some CI jobs this morning | |
| 16:25:59 | EmilienM | let me grab the more recent one | |
| 16:26:23 | EmilienM | http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/ | |
| 16:26:39 | cdent | I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? | |
| 16:28:05 | openstack | Launchpad bug 1656075 in OpenStack Compute (nova) "DiscoveryFailure when trying to get resource providers from the scheduler report client" [Low,Confirmed] - Assigned to Roman Dobosz (roman-dobosz) | |
| 16:28:05 | _gryf | seems similar to https://bugs.launchpad.net/nova/+bug/1656075 | |
| 16:28:13 | bauzas | EmilienM: looks all good for that one http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/nova-compute.txt.gz#_2017-01-18_15_44_36_293 | |
| 16:28:50 | EmilienM | ok so good news: it works in Puppet OpenStack CI this morning. Now trying to figure why it doesn't work in TripleO | |
| 16:29:03 | EmilienM | bauzas: I swear it didn't work yesterday | |
| 16:29:08 | bauzas | :) | |
| 16:29:14 | bauzas | I haven't touched anything ! | |
| 16:29:15 | openstack | Launchpad bug 1657467 in tripleo "placement: unable to refresh compute resource provider record" [High,Triaged] | |
| 16:29:15 | EmilienM | if you look at https://bugs.launchpad.net/tripleo/+bug/1657467 | |
| 16:29:21 | EmilienM | logs are from yesterday and it didn't work | |
| 16:29:38 | bauzas | I'd look at the vhost config | |
| 16:29:41 | mriedem | melwitt: yeah good point, nova.compute.manager calling nova.compute.api to call compute rpcapi to cast the | |
| 16:29:42 | bauzas | and compare both | |
| 16:29:45 | mriedem | 'stop' back to the same compute | |
| 16:29:50 | EmilienM | bauzas: we didn't change it | |
| 16:30:34 | mriedem | melwitt: i was thinking if that already was in place the compute manager could call the compute api to register with the cell mapping (in the single cell case) in the api db, but that's not really going to work | |
| 16:31:01 | mriedem | unless the compute rpcapi did a cast to something else that has access to the api, like conductor | |
| 16:31:03 | purplerbot | <cdent> I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? [2017-01-18 16:26:39] [n 43Pk] | |
| 16:31:03 | cdent | EmilienM, bauzas: I'll say again: [t 43Pk] | |
| 16:31:47 | bauzas | cdent: well, I agree with you on the point that EndpointNotFound is kinda obscure to me | |
| 16:32:01 | bauzas | cdent: I'm supposing that keystone failed finding the endpoint | |
| 16:32:14 | bauzas | but the endpoint can be / | |
| 16:32:21 | cdent | it happens at two different places in the keystoneauth1 stack | |
| 16:32:21 | sdague | bauzas: it's not that keystone failed | |
| 16:32:29 | cdent | once when actually making a request and filtering the results | |
| 16:32:35 | bauzas | sdague: I'm all ears :) | |
| 16:32:35 | cdent | but also when trying to construct a url | |
| 16:32:35 | melwitt | mriedem: I think if something in compute queried cell mapping through the object, it would go through conductor and be able to read from the api db | |
| 16:32:46 | sdague | it's that the resolution of the endpoint from the token ended up with something unknown | |
| 16:33:16 | bauzas | a-ha | |
| 16:34:54 | bauzas | sdague: I don't see any problem with http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/ | |
| 16:35:02 | mriedem | melwitt: hmm, so like, list cell mappings, if one, get that one and register a host with it? | |
| 16:35:12 | mriedem | s/a host/my host/ | |
| 16:35:15 | bauzas | sdague: except the fact that they call http://ip:8778/placement | |
| 16:35:16 | cdent | sdague: by "resolution" do you mean looking at what's at the endpoint, for finding the endpoint in what's in the token? | |
| 16:35:32 | cdent | s/for finding/or finding/ | |
| 16:35:34 | bauzas | I thought the latter | |
| 16:36:48 | sdague | cdent: it's looking through the token | |
| 16:37:00 | sdague | the end point information isn't an api call | |
| 16:37:05 | sdague | it's all embedded in the token | |
| 16:37:38 | melwitt | mriedem: it seems like that would work. technically it would be two cell mappings because of cell0 | |
| 16:38:46 | mriedem | melwitt: true, so list cell mappings, exclude cell0 by uuid, and if one left, register with it | |
| 16:39:31 | mriedem | melwitt: i'm not sure why we didn't think of just auto-registering that way, unless we wanted to auto-register in a more generic way for the multi-cell case | |
| 16:39:47 | melwitt | mriedem: I think so | |
| 16:40:09 | mriedem | but it seems we could do the simple thing for now and hit most small/medium deployment scenarios that don't have any cells today | |
| 16:40:20 | mriedem | idk | |
| 16:40:24 | mriedem | just thinking of options | |
| 16:40:47 | mriedem | melwitt: not sure if you saw the discussion here https://review.openstack.org/#/c/420981/ | |
| 16:41:24 | melwitt | mriedem: yeah, I dunno the history on the auto-register stuff. it's not straightforward for multi-cell because of the need for a "staging cell" and then a command to re-map something. for single-cell, I didn't know what the original plan was | |
| 16:41:55 | melwitt | I saw you refer to alaski's patch from some weeks ago, but I hadn't looked at it before | |
| 16:42:03 | cdent | EmilienM, bauzas: so, where are we? | |
| 16:42:18 | bauzas | honestly, I'm puzzled | |
| 16:42:18 | EmilienM | I'm currently in the matrix reading tripleo logs | |
| 16:42:30 | mriedem | melwitt: yeah i was looking at it this morning, n-cpu does a new rest api call to n-api to register it's host with a cell mapping | |
| 16:42:45 | bauzas | I'm trying to compare the req-id in nova and match it in the keystone logs | |
| 16:42:46 | mriedem | but there are some parts in there about storing the parent cell uuid in the main db for some reason | |
| 16:42:56 | sdague | mriedem: is there a reason the conductor can't just do that? | |