| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-13 | |||
| 00:13:52 | mriedem | dansmith: ^ aside that n-api under uwsgi isn't caching service versions | |
| 00:14:07 | sean-k-mooney | on right because the api is under uwsig in stead of python webserver | |
| 00:14:18 | dansmith | mriedem: ack | |
| 00:16:39 | mriedem | https://bugs.launchpad.net/nova/+bug/1815692 | |
| 00:16:40 | openstack | Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged] | |
| 00:17:52 | mriedem | so i think the hang is on | |
| 00:17:52 | mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/api.py#L261 | |
| 00:18:10 | mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L373 | |
| 00:18:22 | mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L395 | |
| 00:19:38 | mriedem | and i suppose i'm seeing that for every controller that is initializing a compute.API object | |
| 00:19:42 | mriedem | which is most of them.... | |
| 00:20:00 | mriedem | but doesn't really explain why GET / hangs | |
| 00:20:53 | sean-k-mooney | look like that was intoduced by dan to fix https://bugs.launchpad.net/nova/+bug/1807044 | |
| 00:20:54 | openstack | Launchpad bug 1807044 in OpenStack Compute (nova) queens "nova-api startup does not scan cells looking for minimum nova-compute service version" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 00:21:13 | mriedem | yes i know, i approved it | |
| 00:21:49 | sean-k-mooney | sorry just loading context for myself that said im going to sleep soon | |
| 00:22:11 | mriedem | what i need to figure out is why that impacts the versions controller route handler which doesn't use the compute API object | |
| 00:34:24 | mriedem | ok stopped the crazy startup noise by setting upgrade_levels.compute=rocky in nova.conf | |
| 00:34:56 | mriedem | dansmith: huh and this time nova list worked | |
| 00:35:01 | mriedem | with 2.69, i got the partial results back | |
| 00:35:14 | dansmith | well, because no =auto right? | |
| 00:35:17 | mriedem | still get tracebacks in the api logs but didn't hang | |
| 00:35:18 | mriedem | yes | |
| 00:35:41 | mriedem | Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.multi_cell_list [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence is being omitted from the results | |
| 00:35:41 | mriedem | Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.api [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence only partial results are available from this cell if any. | |
| 00:35:49 | dansmith | nice | |
| 00:36:00 | mriedem | i also turned those retry/interval db options to 1 | |
| 00:36:04 | openstackgerrit | Michael Still proposed openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151 | |
| 00:36:04 | openstackgerrit | Michael Still proposed openstack/nova master: create_veth_pair is unused, remove it. https://review.openstack.org/624226 | |
| 00:36:05 | openstackgerrit | Michael Still proposed openstack/nova master: Create specialist set_macaddr_and_vlan helper. https://review.openstack.org/624227 | |
| 00:36:05 | openstackgerrit | Michael Still proposed openstack/nova master: Move create_tap_dev into privsep. https://review.openstack.org/624228 | |
| 00:36:06 | openstackgerrit | Michael Still proposed openstack/nova master: Move set_vf_interface_vlan to be with its only caller. https://review.openstack.org/624229 | |
| 00:36:06 | openstackgerrit | Michael Still proposed openstack/nova master: Move DHCP releasing to privsep. https://review.openstack.org/624230 | |
| 00:36:07 | openstackgerrit | Michael Still proposed openstack/nova master: Convert additional IP management calls to privsep. https://review.openstack.org/624231 | |
| 00:36:07 | openstackgerrit | Michael Still proposed openstack/nova master: Move route management to privsep. https://review.openstack.org/624232 | |
| 00:36:08 | openstackgerrit | Michael Still proposed openstack/nova master: Move additional IP address management to privsep. https://review.openstack.org/624591 | |
| 00:36:08 | openstackgerrit | Michael Still proposed openstack/nova master: Move slight bonkers IP management to privsep. https://review.openstack.org/624592 | |
| 00:36:09 | openstackgerrit | Michael Still proposed openstack/nova master: Move arping to privsep. https://review.openstack.org/624593 | |
| 00:36:09 | openstackgerrit | Michael Still proposed openstack/nova master: Move cleaning conntrack to privsep. https://review.openstack.org/624594 | |
| 00:36:10 | openstackgerrit | Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431 | |
| 00:36:10 | openstackgerrit | Michael Still proposed openstack/nova master: Move killing processes to privsep. https://review.openstack.org/635432 | |
| 00:36:11 | openstackgerrit | Michael Still proposed openstack/nova master: Move FP device creation and deletion to privsep. https://review.openstack.org/635433 | |
| 00:36:11 | openstackgerrit | Michael Still proposed openstack/nova master: Cleanup the exec_ebtables code a little. https://review.openstack.org/635434 | |
| 00:36:12 | openstackgerrit | Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435 | |
| 00:36:12 | openstackgerrit | Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436 | |
| 00:37:47 | mriedem | http://paste.openstack.org/show/744987/ | |
| 00:38:40 | dansmith | also, eye mowt | |
| 00:41:05 | sean-k-mooney | mriedem: so that is effectivly returning the instance ids which we no because they are in the instance mapping table in the api db but everything else is the blank/the default value since the cell is down | |
| 00:42:22 | mriedem | yes | |
| 00:56:43 | mriedem | dansmith: ok, tried to capture the startup "hang" https://bugs.launchpad.net/nova/+bug/1815697 | |
| 00:56:44 | openstack | Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed] | |
| 00:56:54 | mriedem | by my calculation by default each API worker will take ~52 minutes to start... | |
| 06:34:01 | openstackgerrit | Merged openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151 | |
| 06:41:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix resetting non-persistent fields when saving obj https://review.openstack.org/636271 | |
| 09:00:21 | kashyap | Huzzah, all unit tests pass: https://review.openstack.org/#/c/632507/ | |
| 09:00:48 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Restore connection_info after live migration rollback https://review.openstack.org/551349 | |
| 09:01:11 | kashyap | gibi: Morning, when you have a minute, would really appreciate a couple of eyes on this: https://review.openstack.org/#/c/632507/ (libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein") | |
| 09:01:19 | kashyap | stephenfin: ^ | |
| 09:01:55 | gibi | kashyap: looking | |
| 09:02:54 | kashyap | The commit message explains it (hopefully). And getting this in reduces expensive rebase churn. | |
| 09:03:24 | kashyap | I'll be here to answer any questions. | |
| 09:05:59 | kashyap | The code that is being removed in driver.py is what the original author (markus_z) asked us to do in his "NOTE" | |
| 09:06:33 | kashyap | And FWIW, I double-checked with the libvirt upstream developers to ensure what I'm removing makes sense. | |
| 09:06:43 | bauzas | gibi: FWIW, I had to bail out yesterday early | |
| 09:06:55 | bauzas | gibi: so I'll upload my series now with a big -W | |
| 09:07:09 | bauzas | so we could discuss with other folks later | |
| 09:07:43 | gibi | bauzas: ack, I will look at it | |
| 09:08:11 | gibi | kashyap: is there a connection between the min version bump and the removed virtlogd code? | |
| 09:08:36 | kashyap | gibi: Yes, it is unavoidable (as noted in the commit message) in this case | |
| 09:08:46 | kashyap | Due to the version constant check | |
| 09:10:08 | gibi | kashyap: so the min version now bigger than what the virtlogd needs | |
| 09:10:27 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Avoid redundant initialize_connection on source post live migration https://review.openstack.org/551302 | |
| 09:10:27 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Restore connection_info after live migration rollback https://review.openstack.org/551349 | |
| 09:10:46 | gibi | kashyap: so we can remove the check | |
| 09:10:54 | kashyap | Yes, indeed. | |
| 09:12:00 | kashyap | gibi: Oh, just noticed, I actually didn't remove two constant files themselves :-): | |
| 09:12:03 | kashyap | # Use the "logd" backend for handling stdout/stderr from QEMU processes. | |
| 09:12:08 | kashyap | MIN_LIBVIRT_VIRTLOGD = (1, 3, 3) | |
| 09:12:10 | kashyap | MIN_QEMU_VIRTLOGD = (2, 7, 0) | |
| 09:12:41 | gibi | kashyap: OK, then what I miss is the reason of the removal of self._create_file_device from _create_consoles_qemu_kvm() | |
| 09:12:53 | gibi | and the reason of the change in _create_consoles_s390x() | |
| 09:14:30 | kashyap | gibi: First is that: we're adding two serial devices — one of them bypasses Nova | |
| 09:15:20 | kashyap | And Nova should not allow an extra serial console (so that one can use external tools like `virsh` to monitor the guest serial console) | |
| 09:17:05 | kashyap | I'll the reasoning in the commit message (and for the _create_consoles_s390x() change) | |
| 09:17:30 | gibi | hm as far as I understand _create_file_device only makes sure that there is a console.log file | |
| 09:17:43 | gibi | ahh I see now | |
| 09:17:51 | gibi | that code is actually noop with virtlogd | |
| 09:18:19 | gibi | then I got the reason of the change in _create_consoles_qemu_kvm() | |
| 09:18:26 | kashyap | Exactly | |
| 09:18:44 | kashyap | If 'virtlogd' is available, the _create_file_device() just returns. | |
| 09:18:49 | gibi | kashyap: correct | |
| 09:19:02 | kashyap | So, we just elided the whole function, and the places where it is called | |
| 09:19:08 | kashyap | Including the one in s390x | |
| 09:19:36 | kashyap | (_create_file_device() was called in two places, and both the two occurrences were removed) | |
| 09:20:01 | gibi | but in the s390x you replaced the removed _create_file_device call with a _create_pty_device call | |
| 09:21:13 | gibi | is that _create_pty_device for sclplm needed for our always-use-virtlogd case? | |
| 09:21:44 | kashyap | That I can use some s390x eyes :-( | |
| 09:22:26 | kashyap | Let me check with a libvirt dev on that | |
| 09:22:37 | gibi | kashyap: thanks for taking care of it | |
| 09:23:10 | kashyap | gibi: Thanks for the careful review | |