Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-13
00:01:15 mriedem besides the API service is registered in cell0 http://paste.openstack.org/show/744985/
00:02:20 sean-k-mooney http://paste.openstack.org/show/744986/
00:02:24 mriedem this should be the route handler https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/versionsV21.py#L23
00:02:31 dansmith yeah I thought we surveyed the service version of the computes somewhere early like that
00:02:39 dansmith have to do some debug printing
00:03:04 mriedem so we have the recentish change where on service startup we try to get the minimum compute service version,
00:03:07 sean-k-mooney so if you http://192.168.100.19/compute/ it has the microverion in the responce. isnt that how it fitures out the version to use
00:03:27 mriedem sean-k-mooney: the services table version is not the same as microversions
00:03:44 sean-k-mooney ok
00:10:54 mriedem hmm, i think it's the compute.API code trying to create an RPC API client
00:10:55 mriedem Feb 13 00:09:57 downcell devstack@n-api.service[14623]: DEBUG nova.compute.rpcapi [None req-53ebccae-d210-4b14-af5c-02775f3d36e8 None None] Not caching compute RPC version_cap, because min service_version is 0. Please ensure a nova-compute service has been started. Defaulting to current version. {{(pid=14625) _determine_version_cap /opt/stack/nova/nova/compute/rpcapi.py:410}}
00:11:56 dansmith ah could be
00:12:24 sean-k-mooney mriedem could this be related https://github.com/openstack/nova/blob/master/nova/cmd/api.py#L47
00:12:27 mriedem n-api service reports as active
00:12:35 mriedem sean-k-mooney: this isn't using that code
00:13:12 mriedem with default devstack n-api is running under uwsgi so https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L76
00:13:27 sean-k-mooney its also done in the standalone api https://github.com/openstack/nova/blob/master/nova/cmd/api_os_compute.py#L42
00:13:29 mriedem which now that i look at that i guess we don't cache the service versions
00:13:40 mriedem sean-k-mooney: NOT USING nova/cmd/*
00:13:52 mriedem dansmith: ^ aside that n-api under uwsgi isn't caching service versions
00:14:07 sean-k-mooney on right because the api is under uwsig in stead of python webserver
00:14:18 dansmith mriedem: ack
00:16:39 mriedem https://bugs.launchpad.net/nova/+bug/1815692
00:16:40 openstack Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged]
00:17:52 mriedem so i think the hang is on
00:17:52 mriedem https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/api.py#L261
00:18:10 mriedem https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L373
00:18:22 mriedem https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L395
00:19:38 mriedem and i suppose i'm seeing that for every controller that is initializing a compute.API object
00:19:42 mriedem which is most of them....
00:20:00 mriedem but doesn't really explain why GET / hangs
00:20:53 sean-k-mooney look like that was intoduced by dan to fix https://bugs.launchpad.net/nova/+bug/1807044
00:20:54 openstack Launchpad bug 1807044 in OpenStack Compute (nova) queens "nova-api startup does not scan cells looking for minimum nova-compute service version" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
00:21:13 mriedem yes i know, i approved it
00:21:49 sean-k-mooney sorry just loading context for myself that said im going to sleep soon
00:22:11 mriedem what i need to figure out is why that impacts the versions controller route handler which doesn't use the compute API object
00:34:24 mriedem ok stopped the crazy startup noise by setting upgrade_levels.compute=rocky in nova.conf
00:34:56 mriedem dansmith: huh and this time nova list worked
00:35:01 mriedem with 2.69, i got the partial results back
00:35:14 dansmith well, because no =auto right?
00:35:17 mriedem still get tracebacks in the api logs but didn't hang
00:35:18 mriedem yes
00:35:41 mriedem Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.multi_cell_list [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence is being omitted from the results
00:35:41 mriedem Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.api [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence only partial results are available from this cell if any.
00:35:49 dansmith nice
00:36:00 mriedem i also turned those retry/interval db options to 1
00:36:04 openstackgerrit Michael Still proposed openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151
00:36:04 openstackgerrit Michael Still proposed openstack/nova master: create_veth_pair is unused, remove it. https://review.openstack.org/624226
00:36:05 openstackgerrit Michael Still proposed openstack/nova master: Create specialist set_macaddr_and_vlan helper. https://review.openstack.org/624227
00:36:05 openstackgerrit Michael Still proposed openstack/nova master: Move create_tap_dev into privsep. https://review.openstack.org/624228
00:36:06 openstackgerrit Michael Still proposed openstack/nova master: Move set_vf_interface_vlan to be with its only caller. https://review.openstack.org/624229
00:36:06 openstackgerrit Michael Still proposed openstack/nova master: Move DHCP releasing to privsep. https://review.openstack.org/624230
00:36:07 openstackgerrit Michael Still proposed openstack/nova master: Convert additional IP management calls to privsep. https://review.openstack.org/624231
00:36:07 openstackgerrit Michael Still proposed openstack/nova master: Move route management to privsep. https://review.openstack.org/624232
00:36:08 openstackgerrit Michael Still proposed openstack/nova master: Move additional IP address management to privsep. https://review.openstack.org/624591
00:36:08 openstackgerrit Michael Still proposed openstack/nova master: Move slight bonkers IP management to privsep. https://review.openstack.org/624592
00:36:09 openstackgerrit Michael Still proposed openstack/nova master: Move arping to privsep. https://review.openstack.org/624593
00:36:09 openstackgerrit Michael Still proposed openstack/nova master: Move cleaning conntrack to privsep. https://review.openstack.org/624594
00:36:10 openstackgerrit Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431
00:36:10 openstackgerrit Michael Still proposed openstack/nova master: Move killing processes to privsep. https://review.openstack.org/635432
00:36:11 openstackgerrit Michael Still proposed openstack/nova master: Move FP device creation and deletion to privsep. https://review.openstack.org/635433
00:36:11 openstackgerrit Michael Still proposed openstack/nova master: Cleanup the exec_ebtables code a little. https://review.openstack.org/635434
00:36:12 openstackgerrit Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435
00:36:12 openstackgerrit Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436
00:37:47 mriedem http://paste.openstack.org/show/744987/
00:38:40 dansmith also, eye mowt
00:41:05 sean-k-mooney mriedem: so that is effectivly returning the instance ids which we no because they are in the instance mapping table in the api db but everything else is the blank/the default value since the cell is down
00:42:22 mriedem yes
00:56:43 mriedem dansmith: ok, tried to capture the startup "hang" https://bugs.launchpad.net/nova/+bug/1815697
00:56:44 openstack Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed]
00:56:54 mriedem by my calculation by default each API worker will take ~52 minutes to start...
06:34:01 openstackgerrit Merged openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151
06:41:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix resetting non-persistent fields when saving obj https://review.openstack.org/636271
09:00:21 kashyap Huzzah, all unit tests pass: https://review.openstack.org/#/c/632507/
09:00:48 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP Restore connection_info after live migration rollback https://review.openstack.org/551349
09:01:11 kashyap gibi: Morning, when you have a minute, would really appreciate a couple of eyes on this: https://review.openstack.org/#/c/632507/ (libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein")
09:01:19 kashyap stephenfin: ^
09:01:55 gibi kashyap: looking
09:02:54 kashyap The commit message explains it (hopefully). And getting this in reduces expensive rebase churn.
09:03:24 kashyap I'll be here to answer any questions.
09:05:59 kashyap The code that is being removed in driver.py is what the original author (markus_z) asked us to do in his "NOTE"
09:06:33 kashyap And FWIW, I double-checked with the libvirt upstream developers to ensure what I'm removing makes sense.
09:06:43 bauzas gibi: FWIW, I had to bail out yesterday early
09:06:55 bauzas gibi: so I'll upload my series now with a big -W
09:07:09 bauzas so we could discuss with other folks later
09:07:43 gibi bauzas: ack, I will look at it
09:08:11 gibi kashyap: is there a connection between the min version bump and the removed virtlogd code?
09:08:36 kashyap gibi: Yes, it is unavoidable (as noted in the commit message) in this case
09:08:46 kashyap Due to the version constant check
09:10:08 gibi kashyap: so the min version now bigger than what the virtlogd needs
09:10:27 openstackgerrit Lee Yarwood proposed openstack/nova master: Avoid redundant initialize_connection on source post live migration https://review.openstack.org/551302
09:10:27 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP Restore connection_info after live migration rollback https://review.openstack.org/551349
09:10:46 gibi kashyap: so we can remove the check
09:10:54 kashyap Yes, indeed.
09:12:00 kashyap gibi: Oh, just noticed, I actually didn't remove two constant files themselves :-):
09:12:03 kashyap # Use the "logd" backend for handling stdout/stderr from QEMU processes.
09:12:08 kashyap MIN_LIBVIRT_VIRTLOGD = (1, 3, 3)
09:12:10 kashyap MIN_QEMU_VIRTLOGD = (2, 7, 0)
09:12:41 gibi kashyap: OK, then what I miss is the reason of the removal of self._create_file_device from _create_consoles_qemu_kvm()
09:12:53 gibi and the reason of the change in _create_consoles_s390x()

Earlier   Later