Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-25
20:01:34 mriedem and they'd revive somehow and blow up
20:01:52 mriedem i believe they'd be pulled back when listing instances
20:01:55 mriedem because they weren't deleted
20:02:08 mriedem and the from_db_object or whatever assumed the new data model format
20:02:32 mriedem as i said, it was a weird one
20:02:54 mriedem dansmith: +2 on https://review.openstack.org/#/c/460616/ - time to get someone to pay you back
20:08:44 edleafe cd
20:08:47 edleafe ugh
20:11:01 mriedem dansmith: on this cells rpcapi thing https://review.openstack.org/#/c/461532/6/nova/cells/rpcapi.py - i'm not sure what type of exception should be raised here
20:11:17 mriedem just anything?
20:11:28 dansmith mriedem: yeah, we've done it a couple other places
20:11:30 mriedem ServiceTooOld ?
20:11:45 dansmith mriedem: since that would be an upgrade concern, and it's cellsv1, I'd just raise anything to call attention
20:11:49 dansmith sure, sounds fine tome
20:13:24 gyee mriedem, jaypipes, can you guys please take a look? https://review.hpcloud.net/135895
20:13:31 gyee I am trying to get a resolution on this one
20:13:54 mriedem gyee: i don't have access to your internal gerrit :)
20:14:01 mriedem i can request access of course
20:14:53 gyee oh shit, wrong link
20:14:55 gyee my bad
20:15:28 gyee https://review.openstack.org/#/c/466143/
20:16:05 gyee my brain is fried trying to context switch upstream/downstream
20:17:00 openstackgerrit Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987
20:19:14 openstackgerrit Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987
20:22:23 mriedem gyee: looking
20:45:24 mriedem gyee: comments inline
20:46:47 gyee mriedem: thanks!
20:50:05 mriedem nova meeting in #openstack-meeting in 10 minutes
20:53:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532
20:53:19 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Use uuid for id in Services API https://review.openstack.org/464280
21:02:15 openstackgerrit Matt Rabe proposed openstack/nova master: Query deleted instance records during _destroy_evacuated_instances https://review.openstack.org/467774
21:04:34 openstackgerrit Matt Rabe proposed openstack/nova master: Query deleted instance records during _destroy_evacuated_instances https://review.openstack.org/467774
21:20:44 oomichi mriedem: thanks, lgtm on https://review.openstack.org/#/c/456896
21:21:33 openstackgerrit Eric Fried proposed openstack/nova master: PowerVM Driver: SSP emphemeral disk support https://review.openstack.org/443189
21:27:00 mriedem oomichi: thanks. there are 2 more on top of that one with +2s if you can take a look at those also.
21:42:58 mriedem melwitt: jaypipes: dansmith: we talked about spot instances in one of the forum sessions, do you remember which one by chance?
21:43:16 dansmith reservations or something I think
21:43:27 mriedem https://etherpad.openstack.org/p/BOS-forum-advanced-instance-scheduling
21:43:27 mriedem ah yes
21:56:48 openstackgerrit Eric Fried proposed openstack/nova master: Nix redundant dict in set_inventory_for_provider https://review.openstack.org/468123
22:08:25 mriedem is it just me,
22:08:33 mriedem or are the examples in here missing some things https://docs.openstack.org/ocata/config-reference/compute/schedulers.html#aggregateimagepropertiesisolation
22:29:42 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/468089
23:03:24 jaypipes mriedem: "Deleting instance files /var/lib/nova/instances/c929d839-a55d-4834-b044-cd70e9f6d79a_del"
23:03:45 jaypipes mriedem: seeing the above in a customer log file. Does the _del at the end of the file mean the instance was soft-deleted?
23:03:53 mriedem nope
23:03:59 jaypipes ah, ok
23:04:00 mriedem the libvirt driver renames the file i think
23:04:04 mriedem so it can delete it
23:04:17 jaypipes oh... before shutting down the instance?
23:04:46 mriedem in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7839
23:05:18 jaypipes mriedem: trying to figure out why this sequence of events is occurring in the order it is... http://paste.openstack.org/show/610689/
23:05:27 jaypipes mriedem: this is a Liberty installation.
23:05:30 mriedem that's called from cleanup() which is called from destroy() which is called during shutdown_instance in the compute manager
23:06:05 jaypipes mriedem: from that paste, it almost looks like we're deleting the instance before successfully detaching the vNIC.
23:06:13 mriedem "Neutron deleted interface" is via an event from neutron
23:06:16 mriedem for network-vif-deleted
23:06:24 mriedem concurrent
23:06:32 mriedem we deallocate the network, which deletes the port,
23:06:38 mriedem neutron sends an event saying the port was deleted,
23:06:43 mriedem we log that and refresh the cache
23:06:46 mriedem nw info cache that is
23:06:55 mriedem assuming this was a port that nova created when the server was created
23:07:19 mriedem note req-cbfbe2ef-3403-42a9-aef2-9a0e9babe9ed is a separate request
23:07:21 mriedem that's the instance request
23:07:26 mriedem that InstanceNotFound is ignorable
23:07:34 mriedem b/c we already refreshed the nw info cache when we deleted the port
23:07:37 mriedem during deallocate_for_instance
23:08:21 mriedem https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L6500
23:08:24 jaypipes mriedem: well, it's leaving qvo devices on the host...
23:08:24 mriedem ^ is what you hit
23:09:51 openstackgerrit guang-yee proposed openstack/nova master: make sure to rebuild claim on recreate https://review.openstack.org/466143
23:10:04 mriedem https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L2265
23:10:07 mriedem should unplug the vif
23:10:26 mriedem we deallocate the port here https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L2280
23:10:36 mriedem vif unplug should clean things up on the host shoudln't it?
23:11:32 mriedem driver.destroy calls cleanup which calls unplug_vifs
23:12:57 mriedem gyee: i was just updating that :)
23:13:27 jaypipes mriedem: I've asked eugene to get more log information. he says there's lots of stale qvo devices from deleted instances on the customer's hosts...
23:15:36 openstackgerrit Matt Riedemann proposed openstack/nova master: make sure to rebuild claim on recreate https://review.openstack.org/466143
23:18:02 mriedem jaypipes: i think dansmith fixed a bug in the libvirt driver where failed spawns weren't cleaning up properly
23:18:13 mriedem leaving a bunch of garbage around
23:19:23 openstackgerrit Chris Friesen proposed openstack/nova master: Add validation for cpu_realtime_mask https://review.openstack.org/468203
23:20:33 cfriesen sahid: ^
23:21:18 gyee mriedem: all good :-)
23:22:01 mriedem jaypipes: https://review.openstack.org/#/c/409706/ and https://review.openstack.org/#/c/428671/
23:27:59 jaypipes mriedem: thx
23:28:12 gyee jaypipes: nice comment :-)
23:28:19 jaypipes gyee: you disagree?
23:28:29 gyee I'll take the 5th
23:30:06 jaypipes omfg the gate is killing me today.
23:39:11 cfriesen is it possible to run a subset of tox functional tests?
23:39:49 mriedem cfriesen: yes
23:40:01 mriedem tox -e functional -- nova.tests.functional.test_servers
23:40:04 mriedem or whateever
23:40:06 mriedem same as unit tests
23:41:13 cfriesen ah, okay. with unit tests it seems to accept wildcards anywhere in the test path and it doesn't need the "--"
23:42:01 melwitt I thought it worked the same. I don't use "--" but I also might not be wildcarding the way you are
23:43:51 cfriesen hmm...it doesn't even like the format you suggested. but it works if I just run "tox -e functional"
23:43:57 cfriesen ah well

Earlier   Later