Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-25
23:04:04 mriedem so it can delete it
23:04:17 jaypipes oh... before shutting down the instance?
23:04:46 mriedem in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7839
23:05:18 jaypipes mriedem: trying to figure out why this sequence of events is occurring in the order it is... http://paste.openstack.org/show/610689/
23:05:27 jaypipes mriedem: this is a Liberty installation.
23:05:30 mriedem that's called from cleanup() which is called from destroy() which is called during shutdown_instance in the compute manager
23:06:05 jaypipes mriedem: from that paste, it almost looks like we're deleting the instance before successfully detaching the vNIC.
23:06:13 mriedem "Neutron deleted interface" is via an event from neutron
23:06:16 mriedem for network-vif-deleted
23:06:24 mriedem concurrent
23:06:32 mriedem we deallocate the network, which deletes the port,
23:06:38 mriedem neutron sends an event saying the port was deleted,
23:06:43 mriedem we log that and refresh the cache
23:06:46 mriedem nw info cache that is
23:06:55 mriedem assuming this was a port that nova created when the server was created
23:07:19 mriedem note req-cbfbe2ef-3403-42a9-aef2-9a0e9babe9ed is a separate request
23:07:21 mriedem that's the instance request
23:07:26 mriedem that InstanceNotFound is ignorable
23:07:34 mriedem b/c we already refreshed the nw info cache when we deleted the port
23:07:37 mriedem during deallocate_for_instance
23:08:21 mriedem https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L6500
23:08:24 mriedem ^ is what you hit
23:08:24 jaypipes mriedem: well, it's leaving qvo devices on the host...
23:09:51 openstackgerrit guang-yee proposed openstack/nova master: make sure to rebuild claim on recreate https://review.openstack.org/466143
23:10:04 mriedem https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L2265
23:10:07 mriedem should unplug the vif
23:10:26 mriedem we deallocate the port here https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L2280
23:10:36 mriedem vif unplug should clean things up on the host shoudln't it?
23:11:32 mriedem driver.destroy calls cleanup which calls unplug_vifs
23:12:57 mriedem gyee: i was just updating that :)
23:13:27 jaypipes mriedem: I've asked eugene to get more log information. he says there's lots of stale qvo devices from deleted instances on the customer's hosts...
23:15:36 openstackgerrit Matt Riedemann proposed openstack/nova master: make sure to rebuild claim on recreate https://review.openstack.org/466143
23:18:02 mriedem jaypipes: i think dansmith fixed a bug in the libvirt driver where failed spawns weren't cleaning up properly
23:18:13 mriedem leaving a bunch of garbage around
23:19:23 openstackgerrit Chris Friesen proposed openstack/nova master: Add validation for cpu_realtime_mask https://review.openstack.org/468203
23:20:33 cfriesen sahid: ^
23:21:18 gyee mriedem: all good :-)
23:22:01 mriedem jaypipes: https://review.openstack.org/#/c/409706/ and https://review.openstack.org/#/c/428671/
23:27:59 jaypipes mriedem: thx
23:28:12 gyee jaypipes: nice comment :-)
23:28:19 jaypipes gyee: you disagree?
23:28:29 gyee I'll take the 5th
23:30:06 jaypipes omfg the gate is killing me today.
23:39:11 cfriesen is it possible to run a subset of tox functional tests?
23:39:49 mriedem cfriesen: yes
23:40:01 mriedem tox -e functional -- nova.tests.functional.test_servers
23:40:04 mriedem or whateever
23:40:06 mriedem same as unit tests
23:41:13 cfriesen ah, okay. with unit tests it seems to accept wildcards anywhere in the test path and it doesn't need the "--"
23:42:01 melwitt I thought it worked the same. I don't use "--" but I also might not be wildcarding the way you are
23:43:51 cfriesen hmm...it doesn't even like the format you suggested. but it works if I just run "tox -e functional"
23:43:57 cfriesen ah well
23:44:32 melwitt tox -efunctional test_blah
23:46:00 cfriesen hmm....probably helps if I point to actual tests and not template directories.
23:49:01 openstackgerrit John Griffith proposed openstack/nova master: WIP/POC Implement new attach Cinder flow https://review.openstack.org/330285
#openstack-nova - 2017-05-26
00:26:36 openstackgerrit melanie witt proposed openstack/nova master: Add FixedIP.get_count_by_project() https://review.openstack.org/446246
00:26:36 openstackgerrit melanie witt proposed openstack/nova master: Add FloatingIP.get_count_by_project() https://review.openstack.org/446247
00:26:37 openstackgerrit melanie witt proposed openstack/nova master: Add get_count_by_vm_state() to Instance object https://review.openstack.org/446244
00:26:38 openstackgerrit melanie witt proposed openstack/nova master: Add SecurityGroup.get_counts() https://review.openstack.org/446245
00:26:40 openstackgerrit melanie witt proposed openstack/nova master: Remove 'reserved' count from used limits https://review.openstack.org/446242
00:26:40 openstackgerrit melanie witt proposed openstack/nova master: Remove useless quota_usage_refresh from nova-manage https://review.openstack.org/446243
00:26:42 openstackgerrit melanie witt proposed openstack/nova master: Count server groups to check quota https://review.openstack.org/446240
00:26:43 openstackgerrit melanie witt proposed openstack/nova master: Count networks to check quota https://review.openstack.org/446241
00:26:47 openstackgerrit melanie witt proposed openstack/nova master: Count instances to check quota https://review.openstack.org/416521
00:26:48 mriedem breath
00:26:48 openstackgerrit melanie witt proposed openstack/nova master: Add InstanceGroup.get_counts() https://review.openstack.org/457857
00:26:49 openstackgerrit melanie witt proposed openstack/nova master: Add check_deltas() and limit_check_project_and_user() to Quotas https://review.openstack.org/446239
00:26:51 openstackgerrit melanie witt proposed openstack/nova master: Add InstanceGroup._remove_members_in_db https://review.openstack.org/457858
00:26:54 openstackgerrit melanie witt proposed openstack/nova master: Count server group members to check quota https://review.openstack.org/457859
00:26:57 openstackgerrit melanie witt proposed openstack/nova master: Count security groups to check quota https://review.openstack.org/457860
00:27:03 openstackgerrit melanie witt proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945
00:27:05 openstackgerrit melanie witt proposed openstack/nova master: Count fixed ips to check quota https://review.openstack.org/457861
00:27:07 openstackgerrit melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946
00:27:08 openstackgerrit melanie witt proposed openstack/nova master: Count floating ips to check quota https://review.openstack.org/457862
00:27:18 mriedem it's a series!
00:27:22 mriedem 10lb 6oz
00:27:28 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608
00:27:45 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483
00:28:28 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.46 - List/Show all server migration types https://review.openstack.org/430839
00:29:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Refactor a test method including 7 test cases https://review.openstack.org/395436
00:29:07 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove unused exceptions https://review.openstack.org/464525
00:29:14 openstackgerrit Takashi NATSUME proposed openstack/nova master: Avoid forcing translation on logging calls https://review.openstack.org/413876
00:30:33 openstackgerrit Takashi NATSUME proposed openstack/nova master: Use PCIAddressField in oslo.versionedobjects https://review.openstack.org/466201
00:35:31 prometheanfire mriedem: :D
00:42:11 mriedem gyee: we're going to have a problem backporting that evacuate rebuild_claim fix to newton
00:42:23 mriedem getting the resource tracker in newton requires scheduled_node to be not None
00:42:28 mriedem which we don't have in the forced evacuate case
00:48:20 mriedem gyee: i think i have a workaround
00:49:01 gyee mriedem: yes, I think we can set the host prior to the resource_tracker
00:49:21 mriedem yeah i've got the patch local
00:49:24 mriedem running tests
00:49:34 mriedem this will likely need an additional test
00:53:55 mriedem eh maybe not that bad
00:54:12 gyee I think there are two checks there
00:54:46 gyee first check if recreate and scheduled_node is not none prior to resource_tracker
00:55:19 gyee then if not recreate and not scheduled_node after that
00:56:41 gyee the second check covers the scenario where instance is being rebuilt on the same host
00:56:58 mriedem yup i'm just fixing tests
01:08:19 mriedem this is going to take some fancy word-smithing in the commit message
01:15:20 mriedem gyee: https://review.openstack.org/#/c/468227/

Earlier   Later