| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 16:21:44 | EmilienM | but I know the vhost has been created, see http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/httpd/ | |
| 16:22:20 | cdent | the critical point on the placement-api logs is that they will not exist until at least one request is made to the placement API | |
| 16:22:27 | cdent | whether or not it is up | |
| 16:22:37 | EmilienM | ok | |
| 16:22:59 | EmilienM | eg: when you spawn a VM, I guess | |
| 16:23:06 | bauzas | yup | |
| 16:23:15 | bauzas | https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template is the devstack config | |
| 16:23:29 | bauzas | there is an alias that is out of the virtualhost | |
| 16:23:32 | EmilienM | so in tripleo, we spawn overcloud nodes with nova/ironic and i don't see placement logs: http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/undercloud/var/log/nova/ | |
| 16:23:46 | bauzas | hence why you get either :8778/ or :80/placement | |
| 16:24:08 | bauzas | we share the same wsgi process app, but that's two different directives | |
| 16:24:36 | EmilienM | because we use HAproxy and 80 is used for horizon | |
| 16:24:39 | bauzas | EmilienM: yeah, got the problem | |
| 16:24:52 | bauzas | EmilienM: in devstack, we alias on / for the vhost https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template#L8 | |
| 16:25:12 | bauzas | EmilienM: in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz you alias on /placement | |
| 16:25:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:25:22 | bauzas | but you don't provide the location directive | |
| 16:25:24 | EmilienM | yep | |
| 16:25:33 | cdent | if that's the case why did it work before? | |
| 16:25:48 | EmilienM | yeah and I just noticed it works in some CI jobs this morning | |
| 16:25:59 | EmilienM | let me grab the more recent one | |
| 16:26:23 | EmilienM | http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/ | |
| 16:26:39 | cdent | I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? | |
| 16:28:05 | openstack | Launchpad bug 1656075 in OpenStack Compute (nova) "DiscoveryFailure when trying to get resource providers from the scheduler report client" [Low,Confirmed] - Assigned to Roman Dobosz (roman-dobosz) | |
| 16:28:05 | _gryf | seems similar to https://bugs.launchpad.net/nova/+bug/1656075 | |
| 16:28:13 | bauzas | EmilienM: looks all good for that one http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/nova-compute.txt.gz#_2017-01-18_15_44_36_293 | |
| 16:28:50 | EmilienM | ok so good news: it works in Puppet OpenStack CI this morning. Now trying to figure why it doesn't work in TripleO | |
| 16:29:03 | EmilienM | bauzas: I swear it didn't work yesterday | |
| 16:29:08 | bauzas | :) | |
| 16:29:14 | bauzas | I haven't touched anything ! | |
| 16:29:15 | openstack | Launchpad bug 1657467 in tripleo "placement: unable to refresh compute resource provider record" [High,Triaged] | |
| 16:29:15 | EmilienM | if you look at https://bugs.launchpad.net/tripleo/+bug/1657467 | |
| 16:29:21 | EmilienM | logs are from yesterday and it didn't work | |
| 16:29:38 | bauzas | I'd look at the vhost config | |
| 16:29:41 | mriedem | melwitt: yeah good point, nova.compute.manager calling nova.compute.api to call compute rpcapi to cast the | |
| 16:29:42 | bauzas | and compare both | |
| 16:29:45 | mriedem | 'stop' back to the same compute | |
| 16:29:50 | EmilienM | bauzas: we didn't change it | |
| 16:30:34 | mriedem | melwitt: i was thinking if that already was in place the compute manager could call the compute api to register with the cell mapping (in the single cell case) in the api db, but that's not really going to work | |
| 16:31:01 | mriedem | unless the compute rpcapi did a cast to something else that has access to the api, like conductor | |
| 16:31:03 | purplerbot | <cdent> I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? [2017-01-18 16:26:39] [n 43Pk] | |
| 16:31:03 | cdent | EmilienM, bauzas: I'll say again: [t 43Pk] | |
| 16:31:47 | bauzas | cdent: well, I agree with you on the point that EndpointNotFound is kinda obscure to me | |
| 16:32:01 | bauzas | cdent: I'm supposing that keystone failed finding the endpoint | |
| 16:32:14 | bauzas | but the endpoint can be / | |
| 16:32:21 | cdent | it happens at two different places in the keystoneauth1 stack | |
| 16:32:21 | sdague | bauzas: it's not that keystone failed | |
| 16:32:29 | cdent | once when actually making a request and filtering the results | |
| 16:32:35 | bauzas | sdague: I'm all ears :) | |
| 16:32:35 | cdent | but also when trying to construct a url | |
| 16:32:35 | melwitt | mriedem: I think if something in compute queried cell mapping through the object, it would go through conductor and be able to read from the api db | |
| 16:32:46 | sdague | it's that the resolution of the endpoint from the token ended up with something unknown | |
| 16:33:16 | bauzas | a-ha | |
| 16:34:54 | bauzas | sdague: I don't see any problem with http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/ | |
| 16:35:02 | mriedem | melwitt: hmm, so like, list cell mappings, if one, get that one and register a host with it? | |
| 16:35:12 | mriedem | s/a host/my host/ | |
| 16:35:15 | bauzas | sdague: except the fact that they call http://ip:8778/placement | |
| 16:35:16 | cdent | sdague: by "resolution" do you mean looking at what's at the endpoint, for finding the endpoint in what's in the token? | |
| 16:35:32 | cdent | s/for finding/or finding/ | |
| 16:35:34 | bauzas | I thought the latter | |
| 16:36:48 | sdague | cdent: it's looking through the token | |
| 16:37:00 | sdague | the end point information isn't an api call | |
| 16:37:05 | sdague | it's all embedded in the token | |
| 16:37:38 | melwitt | mriedem: it seems like that would work. technically it would be two cell mappings because of cell0 | |
| 16:38:46 | mriedem | melwitt: true, so list cell mappings, exclude cell0 by uuid, and if one left, register with it | |
| 16:39:31 | mriedem | melwitt: i'm not sure why we didn't think of just auto-registering that way, unless we wanted to auto-register in a more generic way for the multi-cell case | |
| 16:39:47 | melwitt | mriedem: I think so | |
| 16:40:09 | mriedem | but it seems we could do the simple thing for now and hit most small/medium deployment scenarios that don't have any cells today | |
| 16:40:20 | mriedem | idk | |
| 16:40:24 | mriedem | just thinking of options | |
| 16:40:47 | mriedem | melwitt: not sure if you saw the discussion here https://review.openstack.org/#/c/420981/ | |
| 16:41:24 | melwitt | mriedem: yeah, I dunno the history on the auto-register stuff. it's not straightforward for multi-cell because of the need for a "staging cell" and then a command to re-map something. for single-cell, I didn't know what the original plan was | |
| 16:41:55 | melwitt | I saw you refer to alaski's patch from some weeks ago, but I hadn't looked at it before | |
| 16:42:03 | cdent | EmilienM, bauzas: so, where are we? | |
| 16:42:18 | bauzas | honestly, I'm puzzled | |
| 16:42:18 | EmilienM | I'm currently in the matrix reading tripleo logs | |
| 16:42:30 | mriedem | melwitt: yeah i was looking at it this morning, n-cpu does a new rest api call to n-api to register it's host with a cell mapping | |
| 16:42:45 | bauzas | I'm trying to compare the req-id in nova and match it in the keystone logs | |
| 16:42:46 | mriedem | but there are some parts in there about storing the parent cell uuid in the main db for some reason | |
| 16:42:56 | sdague | mriedem: is there a reason the conductor can't just do that? | |
| 16:43:05 | mriedem | sdague: that's what melwitt and i are talking about | |
| 16:43:13 | sdague | mriedem: I'm not sure I get why n-cpu needs to make an n-api call | |
| 16:43:14 | sdague | ok | |
| 16:43:35 | melwitt | yeah, I thought it could just go through conductor via objects | |
| 16:44:04 | sdague | ok, got to run to lunch, biab | |
| 16:45:33 | cdent | mriedem, sdague, bauzas: if this merges, all the functional tests pass in python35: https://review.openstack.org/#/c/421529/ | |
| 16:45:34 | bauzas | cdent: EmilienM: FWIW, I'll need to drop very soon, I have a meetup to go | |
| 16:45:59 | mriedem | so in the short-term i probably need to just look at making devstack-gate call back into the primary node at the end of the setup (before tempest) to call discover_hosts instead of doing this https://review.openstack.org/#/c/420981/ | |
| 16:47:10 | melwitt | ah, I see | |
| 16:47:11 | mriedem | +W | |
| 16:47:33 | cdent | thanks bauzas and mriedem . dims we have arrived in the future. | |
| 16:47:51 | mriedem | melwitt: yeah so sean's main point is that my patch relies on the fact that it sets up the computes after the controller (i think) | |
| 16:48:00 | mriedem | as in i think that's the main point | |
| 16:48:12 | melwitt | right | |
| 16:48:48 | mriedem | regardless of whether or not i can rely on the db connection in nova.conf, because DATABASE_HOST is still on the subnode when it's setup so i could build my own one-off nova.conf for running discover_hosts if i wanted, | |
| 16:48:49 | bauzas | cdent: EmilienM: humpf, nothing looks wrong when I see the keystone logs http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache/keystone_wsgi_main_access_ssl.txt.gz | |
| 16:48:55 | mriedem | but i think that defeats the point he's making | |
| 16:49:06 | bauzas | cdent: EmilienM: but I can't introspect the token | |
| 16:49:15 | bauzas | so, honestly, I'm really stuck | |
| 16:50:37 | melwitt | mriedem: oh, to get to the db regardless of whether the controller is up yet? yeah | |