Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-18
16:31:01 mriedem unless the compute rpcapi did a cast to something else that has access to the api, like conductor
16:31:03 purplerbot <cdent> I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? [2017-01-18 16:26:39] [n 43Pk]
16:31:03 cdent EmilienM, bauzas: I'll say again: [t 43Pk]
16:31:47 bauzas cdent: well, I agree with you on the point that EndpointNotFound is kinda obscure to me
16:32:01 bauzas cdent: I'm supposing that keystone failed finding the endpoint
16:32:14 bauzas but the endpoint can be /
16:32:21 cdent it happens at two different places in the keystoneauth1 stack
16:32:21 sdague bauzas: it's not that keystone failed
16:32:29 cdent once when actually making a request and filtering the results
16:32:35 bauzas sdague: I'm all ears :)
16:32:35 cdent but also when trying to construct a url
16:32:35 melwitt mriedem: I think if something in compute queried cell mapping through the object, it would go through conductor and be able to read from the api db
16:32:46 sdague it's that the resolution of the endpoint from the token ended up with something unknown
16:33:16 bauzas a-ha
16:34:54 bauzas sdague: I don't see any problem with http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/
16:35:02 mriedem melwitt: hmm, so like, list cell mappings, if one, get that one and register a host with it?
16:35:12 mriedem s/a host/my host/
16:35:15 bauzas sdague: except the fact that they call http://ip:8778/placement
16:35:16 cdent sdague: by "resolution" do you mean looking at what's at the endpoint, for finding the endpoint in what's in the token?
16:35:32 cdent s/for finding/or finding/
16:35:34 bauzas I thought the latter
16:36:48 sdague cdent: it's looking through the token
16:37:00 sdague the end point information isn't an api call
16:37:05 sdague it's all embedded in the token
16:37:38 melwitt mriedem: it seems like that would work. technically it would be two cell mappings because of cell0
16:38:46 mriedem melwitt: true, so list cell mappings, exclude cell0 by uuid, and if one left, register with it
16:39:31 mriedem melwitt: i'm not sure why we didn't think of just auto-registering that way, unless we wanted to auto-register in a more generic way for the multi-cell case
16:39:47 melwitt mriedem: I think so
16:40:09 mriedem but it seems we could do the simple thing for now and hit most small/medium deployment scenarios that don't have any cells today
16:40:20 mriedem idk
16:40:24 mriedem just thinking of options
16:40:47 mriedem melwitt: not sure if you saw the discussion here https://review.openstack.org/#/c/420981/
16:41:24 melwitt mriedem: yeah, I dunno the history on the auto-register stuff. it's not straightforward for multi-cell because of the need for a "staging cell" and then a command to re-map something. for single-cell, I didn't know what the original plan was
16:41:55 melwitt I saw you refer to alaski's patch from some weeks ago, but I hadn't looked at it before
16:42:03 cdent EmilienM, bauzas: so, where are we?
16:42:18 bauzas honestly, I'm puzzled
16:42:18 EmilienM I'm currently in the matrix reading tripleo logs
16:42:30 mriedem melwitt: yeah i was looking at it this morning, n-cpu does a new rest api call to n-api to register it's host with a cell mapping
16:42:45 bauzas I'm trying to compare the req-id in nova and match it in the keystone logs
16:42:46 mriedem but there are some parts in there about storing the parent cell uuid in the main db for some reason
16:42:56 sdague mriedem: is there a reason the conductor can't just do that?
16:43:05 mriedem sdague: that's what melwitt and i are talking about
16:43:13 sdague mriedem: I'm not sure I get why n-cpu needs to make an n-api call
16:43:14 sdague ok
16:43:35 melwitt yeah, I thought it could just go through conductor via objects
16:44:04 sdague ok, got to run to lunch, biab
16:45:33 cdent mriedem, sdague, bauzas: if this merges, all the functional tests pass in python35: https://review.openstack.org/#/c/421529/
16:45:34 bauzas cdent: EmilienM: FWIW, I'll need to drop very soon, I have a meetup to go
16:45:59 mriedem so in the short-term i probably need to just look at making devstack-gate call back into the primary node at the end of the setup (before tempest) to call discover_hosts instead of doing this https://review.openstack.org/#/c/420981/
16:47:10 melwitt ah, I see
16:47:11 mriedem +W
16:47:33 cdent thanks bauzas and mriedem . dims we have arrived in the future.
16:47:51 mriedem melwitt: yeah so sean's main point is that my patch relies on the fact that it sets up the computes after the controller (i think)
16:48:00 mriedem as in i think that's the main point
16:48:12 melwitt right
16:48:48 mriedem regardless of whether or not i can rely on the db connection in nova.conf, because DATABASE_HOST is still on the subnode when it's setup so i could build my own one-off nova.conf for running discover_hosts if i wanted,
16:48:49 bauzas cdent: EmilienM: humpf, nothing looks wrong when I see the keystone logs http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache/keystone_wsgi_main_access_ssl.txt.gz
16:48:55 mriedem but i think that defeats the point he's making
16:49:06 bauzas cdent: EmilienM: but I can't introspect the token
16:49:15 bauzas so, honestly, I'm really stuck
16:50:37 melwitt mriedem: oh, to get to the db regardless of whether the controller is up yet? yeah
16:51:02 dims cdent : yay :)
16:51:15 openstackgerrit Anusha Unnam proposed openstack/nova: Add keypairs field to InstancePayload https://review.openstack.org/419730
16:52:40 mriedem melwitt: right. i mean i'm not sure how many deployments actually setup computes first.
16:53:32 melwitt mriedem: yeah. we didn't, computes were last/parallel after controllers
16:54:21 openstackgerrit Sujitha proposed openstack/nova: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185
17:00:11 openstackgerrit Matt Riedemann proposed openstack/nova: DNM: Test interface attach/detach for Ironic https://review.openstack.org/421638
17:03:31 mriedem why god does this fail functional tests on newton but is no different from the change on master https://review.openstack.org/#/c/416760/
17:03:38 mriedem why hast thou forsaken my backport
17:03:46 _gryf bauzas, cdent, EmilienM, does any of you was able to reproduce/isolate that error on devstack? I was spent some time yesterday, but without any luck
17:04:04 bauzas _gryf: which error ?
17:04:14 bauzas the one we discussed above?
17:04:22 alaski mriedem: the reason for the cell_info is that a compute does not know what cell it is in. But it is configured to use a specific db. So it queries the db it knows about for its cell, then calls to the API and says "hey, I'm a compute in cell 'foo', register me"
17:04:25 _gryf bauzas, yes
17:04:32 bauzas _gryf: nope
17:04:39 _gryf bummer
17:04:59 alaski mriedem: that avoids the api cell from needing to go through and discover the new host
17:05:17 _gryf I'll try to investigate it further anyway
17:09:31 mriedem alaski: so that's plumbed into discover_hosts here https://review.openstack.org/#/c/369634/1/nova/cmd/manage.py@1530 in the case that the compute_nodes list is empty?
17:09:53 mriedem i.e. if compute_nodes then we create the host mappings for them in discover_hosts, if not we provide the info into the cell db to auto-register itself when it creates the compute node
17:10:19 gabor_antal Hey there, can I have a question? In kilo, there was a config flag name "allow_migrate_to_same_host". But in liberty (and since then) it is disappeared. Why it was removed? How can I get something like that nowadays for testing?
17:10:51 mriedem gabor_antal: allow_resize_to_same_host
17:11:21 alaski mriedem: it wasn't fully fleshed out, but it was written under the assumption that discover_hosts would be run once per cell to set it up and then after that auto-registration would happen.
17:11:42 alaski mriedem: if auto-registration is the path forward finding a way to avoid the initial discover_hosts would be preferable
17:11:53 gabor_antal mriedem: thanks, checking that out.
17:12:13 mriedem alaski: yeah we were talking about a simple auto-register in the single-cell case, which most deploys will be at first in ocata
17:12:33 mriedem alaski: since we know that besides cell0, if there is just a single cell, it's the one we want the host mapping in when creating a compute node record
17:13:15 mriedem alternatively we could mark a specific cell as the 'default' cell or 'staging' cell or something like that
17:13:19 mriedem which new computes show up in
17:14:06 melwitt when you say that, it makes me think we'd need a list_cells_and_hosts that shows what hosts are mapped to what cells
17:14:06 mriedem anyway with <2 weeks to FF auto-registration is probably not going to happen this release anyway so i should probably stop thinking about it
17:14:31 cdent mriedem: I've got a few cycles to poke at https://review.openstack.org/#/c/416760/ if you like?
17:14:34 alaski mriedem: yeah, there's a fair amount to consider to get this done
17:14:50 mriedem melwitt: i think we could just do like a nova-manage cell_v2 list_hosts or something that shows which cell each host is in, and vice-versa
17:14:52 alaski melwitt: that seems generally useful anyways
17:15:03 mriedem cdent: go nuts
17:15:10 cdent jfak;fjdasopvicu9v8
17:15:12 mriedem cdent: i suspect i'm just missing something before that backport
17:15:20 melwitt mriedem: yeah
17:15:31 mriedem cdent: but i can recreate this locally http://logs.openstack.org/60/416760/1/check/gate-nova-tox-db-functional-ubuntu-xenial/f192837/testr_results.html.gz

Earlier   Later