Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-18
11:58:51 kashyap virNetClientProgramDispatchError:177 : Cannot open log file:
11:58:54 kashyap '/opt/stack/data/nova/instances/6cabfc83-8056-4282-9918-f4e349bb6169/console.log':
11:58:57 kashyap Device or resource busy
11:59:18 kashyap Because:
11:59:30 kashyap [8 lines of paste]
11:59:31 kashyap Feb 15 08:39:43.093482 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: <console type='pty'>
11:59:35 kashyap Feb 15 08:39:43.093639 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: <log file='/opt/stack/data/nova/instances/35329c2e-d278-4629-87ad-08795d0a68c8/console.log' append='off'/>
11:59:39 kashyap Feb 15 08:39:43.093782 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: <target type='sclplm' port='0'/>
11:59:42 kashyap Feb 15 08:39:43.093922 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: </console>
11:59:45 kashyap Feb 15 08:39:43.094057 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: <console type='pty'>
11:59:48 kashyap Feb 15 08:39:43.094190 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: <log file='/opt/stack/data/nova/instances/35329c2e-d278-4629-87ad-08795d0a68c8/console.log' append='off'/>
11:59:52 kashyap Feb 15 08:39:43.094328 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: <target type='sclp' port='1'/>
11:59:56 kashyap Feb 15 08:39:43.094470 ubuntu-xenial-s390x-cloud-zfwcec175-6757 nova-compute[18734]: </console>
12:00:47 kashyap Actually, better seen on the pastebin (observe line-14 & line-18): http://paste.openstack.org/show/745255/
12:00:58 kashyap Anyway, I'll address this, and note it in the commit.
12:43:41 openstackgerrit Chris Dent proposed openstack/nova stable/rocky: Lock detach_volume https://review.openstack.org/637534
12:44:03 openstackgerrit Chris Dent proposed openstack/nova stable/queens: Lock detach_volume https://review.openstack.org/637535
12:44:26 openstackgerrit Chris Dent proposed openstack/nova stable/pike: Lock detach_volume https://review.openstack.org/637536
12:46:59 cdent we'll see how that goes
13:14:11 openstackgerrit Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239
13:14:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240
13:14:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528
13:14:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459
13:14:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529
13:14:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421
13:14:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
13:14:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add pf_interface_name tag to passthrough_whitelist https://review.openstack.org/625311
13:14:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Record requester in the InstancePCIRequest https://review.openstack.org/625310
13:14:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360
13:14:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ensure that bandwidth and VF are from the same PF https://review.openstack.org/623543
13:32:04 openstackgerrit Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239
13:32:05 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240
13:32:05 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528
13:32:06 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459
13:32:06 openstackgerrit Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529
13:32:07 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421
13:32:07 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
13:32:08 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add pf_interface_name tag to passthrough_whitelist https://review.openstack.org/625311
13:32:08 openstackgerrit Balazs Gibizer proposed openstack/nova master: Record requester in the InstancePCIRequest https://review.openstack.org/625310
13:32:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ensure that bandwidth and VF are from the same PF https://review.openstack.org/623543
13:33:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360
13:33:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360
13:36:04 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Rewrite _create_pty_device() to be clearer https://review.openstack.org/637152
13:36:04 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein" https://review.openstack.org/632507
13:36:05 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop MIN_LIBVIRT_PARALLELS_SET_ADMIN_PASSWD https://review.openstack.org/632514
13:38:01 gibi fried_rice, leakypipes: added the extra check for leftover allocations to https://review.openstack.org/#/c/616239
13:39:44 leakypipes gibi: ++ will review after my 9am appt.
13:43:04 gibi jaypipes: thanks!
13:45:22 kashyap gibi: fried_rice: Appreciate one more look on the above, when you get a few minutes.
13:45:39 kashyap gibi: It was already ACKed, I fixed the issue discovered by zKVM CI
13:45:47 gibi kashyap: looking
13:47:18 kashyap gibi: Err, 1 sec, let me quickly respin -- I moved the code comment to the wrong function.
13:50:19 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Rewrite _create_pty_device() to be clearer https://review.openstack.org/637152
13:50:19 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein" https://review.openstack.org/632507
13:50:20 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop MIN_LIBVIRT_PARALLELS_SET_ADMIN_PASSWD https://review.openstack.org/632514
13:51:40 efried n-sch meeting in 9 minutes in #openstack-meeting-alt
13:51:56 kashyap Done.
13:55:39 kashyap gibi: ^ Thanks! (If you're deep in the middle of a different change, probably look at it later. The context-switch can be expensive here.)
13:56:00 gibi kashyap: done
13:56:43 kashyap That's fast.
13:56:52 kashyap gibi: Wauw; that's crazy
13:57:04 openstackgerrit Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.69: Handles Down Cells https://review.openstack.org/579563
13:57:09 kashyap gibi: This will come in handy ;-) -- https://tobytripp.github.io/meeting-ticker/
13:58:48 gibi kashyap: thanks :D
14:03:41 openstackgerrit Matt Riedemann proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147
14:10:50 sean-k-mooney general question is it leagal to delete an instance when it is building?
14:11:22 sean-k-mooney if so we have a bug in our network cleanup logic which i need to go fix
14:12:04 sean-k-mooney ill fix it regardless of they answer to my first question but im wondering if we expect you to be able to delete a currently building instance.
14:13:53 sean-k-mooney looking at https://developer.openstack.org/api-ref/compute/?expanded=delete-server-detail#delete-server it does not look like there is a precondition preventing deltion while it is spawning
14:14:27 efried sean-k-mooney: I know a delete request is certainly accepted during spawn, and seems to delete the instance.
14:15:21 sean-k-mooney efried: ya so https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2000-L2036 we dont clean up host networking if we raise exception.UnexpectedDeletingTaskStateError
14:16:02 efried okay
14:16:22 sean-k-mooney so between the time os-vif plugs the vif and the managner updates the db to set the stat to active if we recive a delete request we dont clean up properly
14:33:32 dansmith mriedem: tssurya: we should probably have some stuff in the docs around best practices for effectively tolerating down cells.. like db and mq timeout settings
14:37:32 mriedem dansmith: e.g. https://bugs.launchpad.net/nova/+bug/1815697 ?
14:37:33 openstack Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed]
14:37:45 dansmith mriedem: yeah and that
14:37:56 dansmith very op-focused docs
14:38:40 mriedem sure, that would go in https://docs.openstack.org/nova/latest/admin/
14:39:04 mriedem https://review.openstack.org/#/c/635147/ is already pretty big, so i'd suggest we split that off
14:39:13 dansmith yep
14:39:46 NewBruce sean-k-mooney wrt your open question above, i think there are a few paths that don’t clean up when an error occurs
14:40:12 NewBruce for instance, the issue i’m debugging which we were discussing last week is one as well -
14:40:30 sean-k-mooney NewBruce: yes in this case i dont think we ever call the drivers destroy or cleanup fucntions
14:41:14 NewBruce yep, and in some cases it tends to leave allocations - which i suspect will eventually play havoc with the scheduler/placement
14:41:37 NewBruce except Exception as ex:
14:41:37 NewBruce return updated_port
14:41:37 NewBruce port_id, {'port': {BINDING_PROFILE: port_profile}})
14:41:37 NewBruce updated_port = admin_client.update_port(
14:41:37 NewBruce try:
14:41:37 NewBruce self, instance, port_id, port_profile, admin_client):
14:41:37 NewBruce def _update_port_with_migration_profile(
14:41:38 NewBruce LOG.error("Unable to update binding profile "
14:41:38 NewBruce with excutils.save_and_reraise_exception():
14:41:39 NewBruce {'port': port_id, 'error': ex},
14:41:39 NewBruce "for port: %(port)s due to failure: %(error)s",
14:41:40 NewBruce instance=instance)
14:41:58 sean-k-mooney in this case the allcoation are being cleaned up but its not cleaning upu the network ports
14:42:04 sean-k-mooney e.g. unplginning the vifs
14:42:18 NewBruce Aha - ok

Earlier   Later