| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-12 | |||
| 23:50:36 | mriedem | dansmith: i think when nova client is trying to do version negotiation it's getting hung up on that | |
| 23:50:48 | mriedem | nova --debug list isn't even getting to the GET /servers/detail request | |
| 23:51:10 | dansmith | mriedem: hmm, why would we be striping cells at that point? | |
| 23:51:46 | mriedem | this is basically the api log http://paste.openstack.org/show/744983/ | |
| 23:53:16 | dansmith | when it fails we should get an exception that tells us where it is right? | |
| 23:53:28 | dansmith | but yeah, ten connect attempts at a slow timeout is going to suck | |
| 23:53:54 | dansmith | and if we swallow, might need to print it out somewhere so you can see | |
| 23:54:08 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#database.max_retries | |
| 23:54:15 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#database.retry_interval | |
| 23:54:22 | mriedem | 10 attempts at 10 seconds between attempts | |
| 23:54:50 | mriedem | or this.. https://docs.openstack.org/nova/latest/configuration/config.html#database.db_max_retries | |
| 23:54:54 | sean-k-mooney | its connecting to the cell deb to get to the service table to figure out the mimium service version | |
| 23:54:57 | mriedem | there are a lot of similar sounding options | |
| 23:55:01 | sean-k-mooney | do we not have that in the api db? | |
| 23:55:13 | mriedem | sean-k-mooney: we shouldn't need that in GET / | |
| 23:55:22 | mriedem | the API version doesn't care what the compute services are | |
| 23:55:25 | mriedem | or shouldn't | |
| 23:55:44 | sean-k-mooney | did you mess with the nova_cell0 connection | |
| 23:55:57 | sean-k-mooney | isint that wehere the nova api service record would be | |
| 23:56:13 | mriedem | client side error isn't helpful http://paste.openstack.org/show/744984/ | |
| 23:56:51 | mriedem | but yeah this is what is in the server API logs | |
| 23:56:51 | mriedem | Feb 12 23:54:50 downcell devstack@n-api.service[13288]: WARNING nova.context [None req-e9c1b49d-437b-47f9-a28e-7d5811908b58 None None] Timed out waiting for response from cell 0641a06e-4efa-4c25-acac-b92d1f9082b7: CellTimeout: Timeout waiting for response from cell | |
| 23:56:51 | mriedem | Feb 12 23:54:50 downcell devstack@n-api.service[13288]: WARNING nova.objects.service [None req-e9c1b49d-437b-47f9-a28e-7d5811908b58 None None] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 did not respond when getting minimum service version | |
| 23:57:09 | dansmith | ah | |
| 23:57:10 | dansmith | service version | |
| 23:57:19 | dansmith | needs that for microversion discovery I guess? | |
| 23:57:45 | mriedem | the api version shouldn't depend on the service version though | |
| 23:57:50 | mriedem | the service version is about rpc yeah? | |
| 23:58:03 | dansmith | I thought we didn't offer some microversions if computes hadn't ugpraded yet? | |
| 23:58:12 | dansmith | or do we just refuse to do the thing? | |
| 23:58:16 | sean-k-mooney | mriedem: when you curl / it has the service version in the responce i think | |
| 23:58:22 | dansmith | sean-k-mooney: no | |
| 23:58:42 | mriedem | curl -g -i -X GET http://199.204.45.32/compute/v2.1 -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}121cea354034cc9d2b715383c002e81f067eeb1207dd273ff76587143e309ae5" | |
| 23:58:45 | mriedem | that's the request | |
| 23:59:07 | mriedem | i don't see what in the version router request handler is trying to do something with service versions | |
| #openstack-nova - 2019-02-13 | |||
| 00:01:15 | mriedem | besides the API service is registered in cell0 http://paste.openstack.org/show/744985/ | |
| 00:02:20 | sean-k-mooney | http://paste.openstack.org/show/744986/ | |
| 00:02:24 | mriedem | this should be the route handler https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/versionsV21.py#L23 | |
| 00:02:31 | dansmith | yeah I thought we surveyed the service version of the computes somewhere early like that | |
| 00:02:39 | dansmith | have to do some debug printing | |
| 00:03:04 | mriedem | so we have the recentish change where on service startup we try to get the minimum compute service version, | |
| 00:03:07 | sean-k-mooney | so if you http://192.168.100.19/compute/ it has the microverion in the responce. isnt that how it fitures out the version to use | |
| 00:03:27 | mriedem | sean-k-mooney: the services table version is not the same as microversions | |
| 00:03:44 | sean-k-mooney | ok | |
| 00:10:54 | mriedem | hmm, i think it's the compute.API code trying to create an RPC API client | |
| 00:10:55 | mriedem | Feb 13 00:09:57 downcell devstack@n-api.service[14623]: DEBUG nova.compute.rpcapi [None req-53ebccae-d210-4b14-af5c-02775f3d36e8 None None] Not caching compute RPC version_cap, because min service_version is 0. Please ensure a nova-compute service has been started. Defaulting to current version. {{(pid=14625) _determine_version_cap /opt/stack/nova/nova/compute/rpcapi.py:410}} | |
| 00:11:56 | dansmith | ah could be | |
| 00:12:24 | sean-k-mooney | mriedem could this be related https://github.com/openstack/nova/blob/master/nova/cmd/api.py#L47 | |
| 00:12:27 | mriedem | n-api service reports as active | |
| 00:12:35 | mriedem | sean-k-mooney: this isn't using that code | |
| 00:13:12 | mriedem | with default devstack n-api is running under uwsgi so https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L76 | |
| 00:13:27 | sean-k-mooney | its also done in the standalone api https://github.com/openstack/nova/blob/master/nova/cmd/api_os_compute.py#L42 | |
| 00:13:29 | mriedem | which now that i look at that i guess we don't cache the service versions | |
| 00:13:40 | mriedem | sean-k-mooney: NOT USING nova/cmd/* | |
| 00:13:52 | mriedem | dansmith: ^ aside that n-api under uwsgi isn't caching service versions | |
| 00:14:07 | sean-k-mooney | on right because the api is under uwsig in stead of python webserver | |
| 00:14:18 | dansmith | mriedem: ack | |
| 00:16:39 | mriedem | https://bugs.launchpad.net/nova/+bug/1815692 | |
| 00:16:40 | openstack | Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged] | |
| 00:17:52 | mriedem | so i think the hang is on | |
| 00:17:52 | mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/api.py#L261 | |
| 00:18:10 | mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L373 | |
| 00:18:22 | mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L395 | |
| 00:19:38 | mriedem | and i suppose i'm seeing that for every controller that is initializing a compute.API object | |
| 00:19:42 | mriedem | which is most of them.... | |
| 00:20:00 | mriedem | but doesn't really explain why GET / hangs | |
| 00:20:53 | sean-k-mooney | look like that was intoduced by dan to fix https://bugs.launchpad.net/nova/+bug/1807044 | |
| 00:20:54 | openstack | Launchpad bug 1807044 in OpenStack Compute (nova) queens "nova-api startup does not scan cells looking for minimum nova-compute service version" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 00:21:13 | mriedem | yes i know, i approved it | |
| 00:21:49 | sean-k-mooney | sorry just loading context for myself that said im going to sleep soon | |
| 00:22:11 | mriedem | what i need to figure out is why that impacts the versions controller route handler which doesn't use the compute API object | |
| 00:34:24 | mriedem | ok stopped the crazy startup noise by setting upgrade_levels.compute=rocky in nova.conf | |
| 00:34:56 | mriedem | dansmith: huh and this time nova list worked | |
| 00:35:01 | mriedem | with 2.69, i got the partial results back | |
| 00:35:14 | dansmith | well, because no =auto right? | |
| 00:35:17 | mriedem | still get tracebacks in the api logs but didn't hang | |
| 00:35:18 | mriedem | yes | |
| 00:35:41 | mriedem | Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.multi_cell_list [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence is being omitted from the results | |
| 00:35:41 | mriedem | Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.api [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence only partial results are available from this cell if any. | |
| 00:35:49 | dansmith | nice | |
| 00:36:00 | mriedem | i also turned those retry/interval db options to 1 | |
| 00:36:04 | openstackgerrit | Michael Still proposed openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151 | |
| 00:36:04 | openstackgerrit | Michael Still proposed openstack/nova master: create_veth_pair is unused, remove it. https://review.openstack.org/624226 | |
| 00:36:05 | openstackgerrit | Michael Still proposed openstack/nova master: Create specialist set_macaddr_and_vlan helper. https://review.openstack.org/624227 | |
| 00:36:05 | openstackgerrit | Michael Still proposed openstack/nova master: Move create_tap_dev into privsep. https://review.openstack.org/624228 | |
| 00:36:06 | openstackgerrit | Michael Still proposed openstack/nova master: Move set_vf_interface_vlan to be with its only caller. https://review.openstack.org/624229 | |
| 00:36:06 | openstackgerrit | Michael Still proposed openstack/nova master: Move DHCP releasing to privsep. https://review.openstack.org/624230 | |
| 00:36:07 | openstackgerrit | Michael Still proposed openstack/nova master: Convert additional IP management calls to privsep. https://review.openstack.org/624231 | |
| 00:36:07 | openstackgerrit | Michael Still proposed openstack/nova master: Move route management to privsep. https://review.openstack.org/624232 | |
| 00:36:08 | openstackgerrit | Michael Still proposed openstack/nova master: Move additional IP address management to privsep. https://review.openstack.org/624591 | |
| 00:36:08 | openstackgerrit | Michael Still proposed openstack/nova master: Move slight bonkers IP management to privsep. https://review.openstack.org/624592 | |
| 00:36:09 | openstackgerrit | Michael Still proposed openstack/nova master: Move arping to privsep. https://review.openstack.org/624593 | |
| 00:36:09 | openstackgerrit | Michael Still proposed openstack/nova master: Move cleaning conntrack to privsep. https://review.openstack.org/624594 | |
| 00:36:10 | openstackgerrit | Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431 | |
| 00:36:10 | openstackgerrit | Michael Still proposed openstack/nova master: Move killing processes to privsep. https://review.openstack.org/635432 | |
| 00:36:11 | openstackgerrit | Michael Still proposed openstack/nova master: Move FP device creation and deletion to privsep. https://review.openstack.org/635433 | |
| 00:36:11 | openstackgerrit | Michael Still proposed openstack/nova master: Cleanup the exec_ebtables code a little. https://review.openstack.org/635434 | |
| 00:36:12 | openstackgerrit | Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435 | |
| 00:36:12 | openstackgerrit | Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436 | |
| 00:37:47 | mriedem | http://paste.openstack.org/show/744987/ | |