Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-12
23:37:16 mriedem i'll just eff up the db connection then
23:37:19 sean-k-mooney right because you want to allow the other dbs to work
23:37:21 dansmith mriedem: you could revoke privs on the db
23:37:54 dansmith mriedem: or try dropping the tables altogether I guess, but tbh we don't really know what the failure modes will look like I'd guess, and most of the time in reality it would be a connection timeout,
23:38:02 dansmith which will be hard to mimic with one mysql
23:38:26 dansmith one thing you could do, is change the connection string to point to some ip black hole for you
23:38:39 dansmith that would get you the timeout on that connection
23:38:58 dansmith depends on if you're trying to make it happen at runtime, or just start it up unable to reach one of the dbs
23:40:23 sean-k-mooney dansmith: you might be able to do it with ssh forwarding
23:40:35 dansmith do what/
23:41:01 mriedem yeah i just changed cell1 db connection to mysql+pymysql://root:openstack1@127.0.0.2/nova_cell1?charset=utf8
23:41:03 mriedem will see what happens
23:41:16 sean-k-mooney e.g. expose the mysql server port locally on anoter port point nova at it and then kill the ssh tunnel e.g. using ssh -L ...
23:41:21 dansmith mriedem: that might not be a black hole for you :)
23:41:43 dansmith mriedem: I would change it to some 172. or 192. or 10. address you definitely can't reach
23:42:31 mriedem oh yeah the connections are cached,
23:42:31 sean-k-mooney using ssh that way would allow nova to start up normmally and then you can kill the connect when ever you like by killing the ssh tunnel
23:42:33 mriedem restarting n-api
23:43:04 mriedem bingo
23:43:05 mriedem Feb 12 23:42:52 downcell devstack@n-api.service[13288]: WARNING oslo_db.sqlalchemy.engines [None req-8002ab53-453c-488e-ae15-04e2a74e7e6f None None] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'192.0.0.1' ([Errno 101] ENETUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8)
23:43:18 dansmith right on
23:43:20 mriedem oh goody and it's just spewing in the api logs
23:43:34 mriedem well i guess 10 retries
23:43:44 dansmith sean-k-mooney: he could just do that with some outbound nat to whatever black hole address he chooses
23:44:02 mriedem Feb 12 23:43:49 downcell devstack@n-api.service[13288]: WARNING nova.context [None req-f257afc4-18c9-4989-a500-a4a1f017a322 None None] Timed out waiting for response from cell 0641a06e-4efa-4c25-acac-b92d1f9082b7: CellTimeout: Timeout waiting for response from cell
23:44:02 mriedem Feb 12 23:43:49 downcell devstack@n-api.service[13288]: WARNING nova.objects.service [None req-f257afc4-18c9-4989-a500-a4a1f017a322 None None] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 did not respond when getting minimum service version
23:44:07 sean-k-mooney dansmith: ya you can do it with iptables but that is more annoying
23:44:54 mriedem how about i just changed the db connectoin in the db to a bad ip :)
23:44:56 sean-k-mooney with ssh its basically "ssh 127.0.0.1 -L 53306:127.0.0.1:3306"
23:44:57 mriedem much easier
23:45:31 sean-k-mooney mriedem: it is but do you want to be able to kill the connect at an arbitray point or just not have it work
23:46:08 mriedem hmm, nova list with 2.69 hanging...this is probably expected while it's trying to gather results from the down cell db
23:48:29 mriedem i seem to be stuck in an infinite reconnect loop blocking the api response
23:49:39 mriedem and i got a 500
23:50:18 mriedem oh heh,
23:50:19 sean-k-mooney proably due to a timeout of the connect waiting for the responce
23:50:36 mriedem dansmith: i think when nova client is trying to do version negotiation it's getting hung up on that
23:50:48 mriedem nova --debug list isn't even getting to the GET /servers/detail request
23:51:10 dansmith mriedem: hmm, why would we be striping cells at that point?
23:51:46 mriedem this is basically the api log http://paste.openstack.org/show/744983/
23:53:16 dansmith when it fails we should get an exception that tells us where it is right?
23:53:28 dansmith but yeah, ten connect attempts at a slow timeout is going to suck
23:53:54 dansmith and if we swallow, might need to print it out somewhere so you can see
23:54:08 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#database.max_retries
23:54:15 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#database.retry_interval
23:54:22 mriedem 10 attempts at 10 seconds between attempts
23:54:50 mriedem or this.. https://docs.openstack.org/nova/latest/configuration/config.html#database.db_max_retries
23:54:54 sean-k-mooney its connecting to the cell deb to get to the service table to figure out the mimium service version
23:54:57 mriedem there are a lot of similar sounding options
23:55:01 sean-k-mooney do we not have that in the api db?
23:55:13 mriedem sean-k-mooney: we shouldn't need that in GET /
23:55:22 mriedem the API version doesn't care what the compute services are
23:55:25 mriedem or shouldn't
23:55:44 sean-k-mooney did you mess with the nova_cell0 connection
23:55:57 sean-k-mooney isint that wehere the nova api service record would be
23:56:13 mriedem client side error isn't helpful http://paste.openstack.org/show/744984/
23:56:51 mriedem but yeah this is what is in the server API logs
23:56:51 mriedem Feb 12 23:54:50 downcell devstack@n-api.service[13288]: WARNING nova.context [None req-e9c1b49d-437b-47f9-a28e-7d5811908b58 None None] Timed out waiting for response from cell 0641a06e-4efa-4c25-acac-b92d1f9082b7: CellTimeout: Timeout waiting for response from cell
23:56:51 mriedem Feb 12 23:54:50 downcell devstack@n-api.service[13288]: WARNING nova.objects.service [None req-e9c1b49d-437b-47f9-a28e-7d5811908b58 None None] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 did not respond when getting minimum service version
23:57:09 dansmith ah
23:57:10 dansmith service version
23:57:19 dansmith needs that for microversion discovery I guess?
23:57:45 mriedem the api version shouldn't depend on the service version though
23:57:50 mriedem the service version is about rpc yeah?
23:58:03 dansmith I thought we didn't offer some microversions if computes hadn't ugpraded yet?
23:58:12 dansmith or do we just refuse to do the thing?
23:58:16 sean-k-mooney mriedem: when you curl / it has the service version in the responce i think
23:58:22 dansmith sean-k-mooney: no
23:58:42 mriedem curl -g -i -X GET http://199.204.45.32/compute/v2.1 -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}121cea354034cc9d2b715383c002e81f067eeb1207dd273ff76587143e309ae5"
23:58:45 mriedem that's the request
23:59:07 mriedem i don't see what in the version router request handler is trying to do something with service versions
#openstack-nova - 2019-02-13
00:01:15 mriedem besides the API service is registered in cell0 http://paste.openstack.org/show/744985/
00:02:20 sean-k-mooney http://paste.openstack.org/show/744986/
00:02:24 mriedem this should be the route handler https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/versionsV21.py#L23
00:02:31 dansmith yeah I thought we surveyed the service version of the computes somewhere early like that
00:02:39 dansmith have to do some debug printing
00:03:04 mriedem so we have the recentish change where on service startup we try to get the minimum compute service version,
00:03:07 sean-k-mooney so if you http://192.168.100.19/compute/ it has the microverion in the responce. isnt that how it fitures out the version to use
00:03:27 mriedem sean-k-mooney: the services table version is not the same as microversions
00:03:44 sean-k-mooney ok
00:10:54 mriedem hmm, i think it's the compute.API code trying to create an RPC API client
00:10:55 mriedem Feb 13 00:09:57 downcell devstack@n-api.service[14623]: DEBUG nova.compute.rpcapi [None req-53ebccae-d210-4b14-af5c-02775f3d36e8 None None] Not caching compute RPC version_cap, because min service_version is 0. Please ensure a nova-compute service has been started. Defaulting to current version. {{(pid=14625) _determine_version_cap /opt/stack/nova/nova/compute/rpcapi.py:410}}
00:11:56 dansmith ah could be
00:12:24 sean-k-mooney mriedem could this be related https://github.com/openstack/nova/blob/master/nova/cmd/api.py#L47
00:12:27 mriedem n-api service reports as active
00:12:35 mriedem sean-k-mooney: this isn't using that code
00:13:12 mriedem with default devstack n-api is running under uwsgi so https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L76
00:13:27 sean-k-mooney its also done in the standalone api https://github.com/openstack/nova/blob/master/nova/cmd/api_os_compute.py#L42
00:13:29 mriedem which now that i look at that i guess we don't cache the service versions
00:13:40 mriedem sean-k-mooney: NOT USING nova/cmd/*
00:13:52 mriedem dansmith: ^ aside that n-api under uwsgi isn't caching service versions
00:14:07 sean-k-mooney on right because the api is under uwsig in stead of python webserver
00:14:18 dansmith mriedem: ack
00:16:39 mriedem https://bugs.launchpad.net/nova/+bug/1815692
00:16:40 openstack Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged]
00:17:52 mriedem so i think the hang is on
00:17:52 mriedem https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/api.py#L261
00:18:10 mriedem https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L373
00:18:22 mriedem https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L395
00:19:38 mriedem and i suppose i'm seeing that for every controller that is initializing a compute.API object

Earlier   Later