| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-25 | |||
| 19:51:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Avoid lazy-load error when getting instance AZ https://review.openstack.org/468147 | |
| 19:51:38 | mriedem | dansmith: easy stat pad ^ | |
| 19:54:10 | mriedem | cfriesen: that one was really odd | |
| 19:54:12 | cfriesen | mriedem: never mind, I see we introduced the change in Newton so no way to backport the migration | |
| 19:54:16 | mriedem | correct | |
| 19:55:21 | cfriesen | mriedem: I'm assuming it'd be okay if we deleted those entries separately prior to the upgrade | |
| 19:55:38 | mriedem | the old busted build requests? | |
| 19:55:38 | cfriesen | the failed BuildRequest I mean | |
| 19:55:39 | mriedem | i think so | |
| 19:57:46 | cfriesen | mriedem: what would have happened to successful old build requests? | |
| 20:00:43 | mriedem | happened where? when? | |
| 20:01:00 | mriedem | successful build requests are cleaned up when the instance is created | |
| 20:01:17 | mriedem | failed ones weren't getting cleaned up at some point, which i think was part of the issue | |
| 20:01:26 | cfriesen | ah, okay. so at steady-state there would only be failed ones left in the DB | |
| 20:01:29 | mriedem | so you'd have these old busted build requests with the old data format, | |
| 20:01:34 | mriedem | and they'd revive somehow and blow up | |
| 20:01:52 | mriedem | i believe they'd be pulled back when listing instances | |
| 20:01:55 | mriedem | because they weren't deleted | |
| 20:02:08 | mriedem | and the from_db_object or whatever assumed the new data model format | |
| 20:02:32 | mriedem | as i said, it was a weird one | |
| 20:02:54 | mriedem | dansmith: +2 on https://review.openstack.org/#/c/460616/ - time to get someone to pay you back | |
| 20:08:44 | edleafe | cd | |
| 20:08:47 | edleafe | ugh | |
| 20:11:01 | mriedem | dansmith: on this cells rpcapi thing https://review.openstack.org/#/c/461532/6/nova/cells/rpcapi.py - i'm not sure what type of exception should be raised here | |
| 20:11:17 | mriedem | just anything? | |
| 20:11:28 | dansmith | mriedem: yeah, we've done it a couple other places | |
| 20:11:30 | mriedem | ServiceTooOld ? | |
| 20:11:45 | dansmith | mriedem: since that would be an upgrade concern, and it's cellsv1, I'd just raise anything to call attention | |
| 20:11:49 | dansmith | sure, sounds fine tome | |
| 20:13:24 | gyee | mriedem, jaypipes, can you guys please take a look? https://review.hpcloud.net/135895 | |
| 20:13:31 | gyee | I am trying to get a resolution on this one | |
| 20:13:54 | mriedem | gyee: i don't have access to your internal gerrit :) | |
| 20:14:01 | mriedem | i can request access of course | |
| 20:14:53 | gyee | oh shit, wrong link | |
| 20:14:55 | gyee | my bad | |
| 20:15:28 | gyee | https://review.openstack.org/#/c/466143/ | |
| 20:16:05 | gyee | my brain is fried trying to context switch upstream/downstream | |
| 20:17:00 | openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 20:19:14 | openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 20:22:23 | mriedem | gyee: looking | |
| 20:45:24 | mriedem | gyee: comments inline | |
| 20:46:47 | gyee | mriedem: thanks! | |
| 20:50:05 | mriedem | nova meeting in #openstack-meeting in 10 minutes | |
| 20:53:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 20:53:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Use uuid for id in Services API https://review.openstack.org/464280 | |
| 21:02:15 | openstackgerrit | Matt Rabe proposed openstack/nova master: Query deleted instance records during _destroy_evacuated_instances https://review.openstack.org/467774 | |
| 21:04:34 | openstackgerrit | Matt Rabe proposed openstack/nova master: Query deleted instance records during _destroy_evacuated_instances https://review.openstack.org/467774 | |
| 21:20:44 | oomichi | mriedem: thanks, lgtm on https://review.openstack.org/#/c/456896 | |
| 21:21:33 | openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: SSP emphemeral disk support https://review.openstack.org/443189 | |
| 21:27:00 | mriedem | oomichi: thanks. there are 2 more on top of that one with +2s if you can take a look at those also. | |
| 21:42:58 | mriedem | melwitt: jaypipes: dansmith: we talked about spot instances in one of the forum sessions, do you remember which one by chance? | |
| 21:43:16 | dansmith | reservations or something I think | |
| 21:43:27 | mriedem | ah yes | |
| 21:43:27 | mriedem | https://etherpad.openstack.org/p/BOS-forum-advanced-instance-scheduling | |
| 21:56:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Nix redundant dict in set_inventory_for_provider https://review.openstack.org/468123 | |
| 22:08:25 | mriedem | is it just me, | |
| 22:08:33 | mriedem | or are the examples in here missing some things https://docs.openstack.org/ocata/config-reference/compute/schedulers.html#aggregateimagepropertiesisolation | |
| 22:29:42 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/468089 | |
| 23:03:24 | jaypipes | mriedem: "Deleting instance files /var/lib/nova/instances/c929d839-a55d-4834-b044-cd70e9f6d79a_del" | |
| 23:03:45 | jaypipes | mriedem: seeing the above in a customer log file. Does the _del at the end of the file mean the instance was soft-deleted? | |
| 23:03:53 | mriedem | nope | |
| 23:03:59 | jaypipes | ah, ok | |
| 23:04:00 | mriedem | the libvirt driver renames the file i think | |
| 23:04:04 | mriedem | so it can delete it | |
| 23:04:17 | jaypipes | oh... before shutting down the instance? | |
| 23:04:46 | mriedem | in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7839 | |
| 23:05:18 | jaypipes | mriedem: trying to figure out why this sequence of events is occurring in the order it is... http://paste.openstack.org/show/610689/ | |
| 23:05:27 | jaypipes | mriedem: this is a Liberty installation. | |
| 23:05:30 | mriedem | that's called from cleanup() which is called from destroy() which is called during shutdown_instance in the compute manager | |
| 23:06:05 | jaypipes | mriedem: from that paste, it almost looks like we're deleting the instance before successfully detaching the vNIC. | |
| 23:06:13 | mriedem | "Neutron deleted interface" is via an event from neutron | |
| 23:06:16 | mriedem | for network-vif-deleted | |
| 23:06:24 | mriedem | concurrent | |
| 23:06:32 | mriedem | we deallocate the network, which deletes the port, | |
| 23:06:38 | mriedem | neutron sends an event saying the port was deleted, | |
| 23:06:43 | mriedem | we log that and refresh the cache | |
| 23:06:46 | mriedem | nw info cache that is | |
| 23:06:55 | mriedem | assuming this was a port that nova created when the server was created | |
| 23:07:19 | mriedem | note req-cbfbe2ef-3403-42a9-aef2-9a0e9babe9ed is a separate request | |
| 23:07:21 | mriedem | that's the instance request | |
| 23:07:26 | mriedem | that InstanceNotFound is ignorable | |
| 23:07:34 | mriedem | b/c we already refreshed the nw info cache when we deleted the port | |
| 23:07:37 | mriedem | during deallocate_for_instance | |
| 23:08:21 | mriedem | https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L6500 | |
| 23:08:24 | mriedem | ^ is what you hit | |
| 23:08:24 | jaypipes | mriedem: well, it's leaving qvo devices on the host... | |
| 23:09:51 | openstackgerrit | guang-yee proposed openstack/nova master: make sure to rebuild claim on recreate https://review.openstack.org/466143 | |
| 23:10:04 | mriedem | https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L2265 | |
| 23:10:07 | mriedem | should unplug the vif | |
| 23:10:26 | mriedem | we deallocate the port here https://github.com/openstack/nova/blob/liberty-eol/nova/compute/manager.py#L2280 | |
| 23:10:36 | mriedem | vif unplug should clean things up on the host shoudln't it? | |
| 23:11:32 | mriedem | driver.destroy calls cleanup which calls unplug_vifs | |
| 23:12:57 | mriedem | gyee: i was just updating that :) | |
| 23:13:27 | jaypipes | mriedem: I've asked eugene to get more log information. he says there's lots of stale qvo devices from deleted instances on the customer's hosts... | |
| 23:15:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: make sure to rebuild claim on recreate https://review.openstack.org/466143 | |
| 23:18:02 | mriedem | jaypipes: i think dansmith fixed a bug in the libvirt driver where failed spawns weren't cleaning up properly | |
| 23:18:13 | mriedem | leaving a bunch of garbage around | |
| 23:19:23 | openstackgerrit | Chris Friesen proposed openstack/nova master: Add validation for cpu_realtime_mask https://review.openstack.org/468203 | |
| 23:20:33 | cfriesen | sahid: ^ | |
| 23:21:18 | gyee | mriedem: all good :-) | |