Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-18
16:24:39 bauzas EmilienM: yeah, got the problem
16:24:52 bauzas EmilienM: in devstack, we alias on / for the vhost https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template#L8
16:25:12 bauzas EmilienM: in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz you alias on /placement
16:25:20 openstackgerrit Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225
16:25:22 bauzas but you don't provide the location directive
16:25:24 EmilienM yep
16:25:33 cdent if that's the case why did it work before?
16:25:48 EmilienM yeah and I just noticed it works in some CI jobs this morning
16:25:59 EmilienM let me grab the more recent one
16:26:23 EmilienM http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/
16:26:39 cdent I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know?
16:28:05 _gryf seems similar to https://bugs.launchpad.net/nova/+bug/1656075
16:28:05 openstack Launchpad bug 1656075 in OpenStack Compute (nova) "DiscoveryFailure when trying to get resource providers from the scheduler report client" [Low,Confirmed] - Assigned to Roman Dobosz (roman-dobosz)
16:28:13 bauzas EmilienM: looks all good for that one http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/nova-compute.txt.gz#_2017-01-18_15_44_36_293
16:28:50 EmilienM ok so good news: it works in Puppet OpenStack CI this morning. Now trying to figure why it doesn't work in TripleO
16:29:03 EmilienM bauzas: I swear it didn't work yesterday
16:29:08 bauzas :)
16:29:14 bauzas I haven't touched anything !
16:29:15 EmilienM if you look at https://bugs.launchpad.net/tripleo/+bug/1657467
16:29:15 openstack Launchpad bug 1657467 in tripleo "placement: unable to refresh compute resource provider record" [High,Triaged]
16:29:21 EmilienM logs are from yesterday and it didn't work
16:29:38 bauzas I'd look at the vhost config
16:29:41 mriedem melwitt: yeah good point, nova.compute.manager calling nova.compute.api to call compute rpcapi to cast the
16:29:42 bauzas and compare both
16:29:45 mriedem 'stop' back to the same compute
16:29:50 EmilienM bauzas: we didn't change it
16:30:34 mriedem melwitt: i was thinking if that already was in place the compute manager could call the compute api to register with the cell mapping (in the single cell case) in the api db, but that's not really going to work
16:31:01 mriedem unless the compute rpcapi did a cast to something else that has access to the api, like conductor
16:31:03 cdent EmilienM, bauzas: I'll say again: [t 43Pk]
16:31:03 purplerbot <cdent> I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? [2017-01-18 16:26:39] [n 43Pk]
16:31:47 bauzas cdent: well, I agree with you on the point that EndpointNotFound is kinda obscure to me
16:32:01 bauzas cdent: I'm supposing that keystone failed finding the endpoint
16:32:14 bauzas but the endpoint can be /
16:32:21 sdague bauzas: it's not that keystone failed
16:32:21 cdent it happens at two different places in the keystoneauth1 stack
16:32:29 cdent once when actually making a request and filtering the results
16:32:35 melwitt mriedem: I think if something in compute queried cell mapping through the object, it would go through conductor and be able to read from the api db
16:32:35 cdent but also when trying to construct a url
16:32:35 bauzas sdague: I'm all ears :)
16:32:46 sdague it's that the resolution of the endpoint from the token ended up with something unknown
16:33:16 bauzas a-ha
16:34:54 bauzas sdague: I don't see any problem with http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/
16:35:02 mriedem melwitt: hmm, so like, list cell mappings, if one, get that one and register a host with it?
16:35:12 mriedem s/a host/my host/
16:35:15 bauzas sdague: except the fact that they call http://ip:8778/placement
16:35:16 cdent sdague: by "resolution" do you mean looking at what's at the endpoint, for finding the endpoint in what's in the token?
16:35:32 cdent s/for finding/or finding/
16:35:34 bauzas I thought the latter
16:36:48 sdague cdent: it's looking through the token
16:37:00 sdague the end point information isn't an api call
16:37:05 sdague it's all embedded in the token
16:37:38 melwitt mriedem: it seems like that would work. technically it would be two cell mappings because of cell0
16:38:46 mriedem melwitt: true, so list cell mappings, exclude cell0 by uuid, and if one left, register with it
16:39:31 mriedem melwitt: i'm not sure why we didn't think of just auto-registering that way, unless we wanted to auto-register in a more generic way for the multi-cell case
16:39:47 melwitt mriedem: I think so
16:40:09 mriedem but it seems we could do the simple thing for now and hit most small/medium deployment scenarios that don't have any cells today
16:40:20 mriedem idk
16:40:24 mriedem just thinking of options
16:40:47 mriedem melwitt: not sure if you saw the discussion here https://review.openstack.org/#/c/420981/
16:41:24 melwitt mriedem: yeah, I dunno the history on the auto-register stuff. it's not straightforward for multi-cell because of the need for a "staging cell" and then a command to re-map something. for single-cell, I didn't know what the original plan was
16:41:55 melwitt I saw you refer to alaski's patch from some weeks ago, but I hadn't looked at it before
16:42:03 cdent EmilienM, bauzas: so, where are we?
16:42:18 EmilienM I'm currently in the matrix reading tripleo logs
16:42:18 bauzas honestly, I'm puzzled
16:42:30 mriedem melwitt: yeah i was looking at it this morning, n-cpu does a new rest api call to n-api to register it's host with a cell mapping
16:42:45 bauzas I'm trying to compare the req-id in nova and match it in the keystone logs
16:42:46 mriedem but there are some parts in there about storing the parent cell uuid in the main db for some reason
16:42:56 sdague mriedem: is there a reason the conductor can't just do that?
16:43:05 mriedem sdague: that's what melwitt and i are talking about
16:43:13 sdague mriedem: I'm not sure I get why n-cpu needs to make an n-api call
16:43:14 sdague ok
16:43:35 melwitt yeah, I thought it could just go through conductor via objects
16:44:04 sdague ok, got to run to lunch, biab
16:45:33 cdent mriedem, sdague, bauzas: if this merges, all the functional tests pass in python35: https://review.openstack.org/#/c/421529/
16:45:34 bauzas cdent: EmilienM: FWIW, I'll need to drop very soon, I have a meetup to go
16:45:59 mriedem so in the short-term i probably need to just look at making devstack-gate call back into the primary node at the end of the setup (before tempest) to call discover_hosts instead of doing this https://review.openstack.org/#/c/420981/
16:47:10 melwitt ah, I see
16:47:11 mriedem +W
16:47:33 cdent thanks bauzas and mriedem . dims we have arrived in the future.
16:47:51 mriedem melwitt: yeah so sean's main point is that my patch relies on the fact that it sets up the computes after the controller (i think)
16:48:00 mriedem as in i think that's the main point
16:48:12 melwitt right
16:48:48 mriedem regardless of whether or not i can rely on the db connection in nova.conf, because DATABASE_HOST is still on the subnode when it's setup so i could build my own one-off nova.conf for running discover_hosts if i wanted,
16:48:49 bauzas cdent: EmilienM: humpf, nothing looks wrong when I see the keystone logs http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache/keystone_wsgi_main_access_ssl.txt.gz
16:48:55 mriedem but i think that defeats the point he's making
16:49:06 bauzas cdent: EmilienM: but I can't introspect the token
16:49:15 bauzas so, honestly, I'm really stuck
16:50:37 melwitt mriedem: oh, to get to the db regardless of whether the controller is up yet? yeah
16:51:02 dims cdent : yay :)
16:51:15 openstackgerrit Anusha Unnam proposed openstack/nova: Add keypairs field to InstancePayload https://review.openstack.org/419730
16:52:40 mriedem melwitt: right. i mean i'm not sure how many deployments actually setup computes first.
16:53:32 melwitt mriedem: yeah. we didn't, computes were last/parallel after controllers
16:54:21 openstackgerrit Sujitha proposed openstack/nova: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185
17:00:11 openstackgerrit Matt Riedemann proposed openstack/nova: DNM: Test interface attach/detach for Ironic https://review.openstack.org/421638
17:03:31 mriedem why god does this fail functional tests on newton but is no different from the change on master https://review.openstack.org/#/c/416760/
17:03:38 mriedem why hast thou forsaken my backport
17:03:46 _gryf bauzas, cdent, EmilienM, does any of you was able to reproduce/isolate that error on devstack? I was spent some time yesterday, but without any luck
17:04:04 bauzas _gryf: which error ?
17:04:14 bauzas the one we discussed above?
17:04:22 alaski mriedem: the reason for the cell_info is that a compute does not know what cell it is in. But it is configured to use a specific db. So it queries the db it knows about for its cell, then calls to the API and says "hey, I'm a compute in cell 'foo', register me"

Earlier   Later