| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-23 | |||
| 20:39:40 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add a new section: "Upgrade impact" to the template https://review.openstack.org/456756 | |
| 20:40:38 | openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - add alternate hosts https://review.openstack.org/486215 | |
| 20:40:39 | openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - Add allocations to the values returned from the scheduler https://review.openstack.org/495854 | |
| 20:40:39 | openstackgerrit | Ed Leafe proposed openstack/nova master: return alternates along with their allocations https://review.openstack.org/486253 | |
| 20:41:26 | cfriesen_ | with placement/allocations where do we free up resources on deletion of an instance? | |
| 20:43:20 | exarr | I have a pastebin link if anyone is willing to take a look :-( https://pastebin.com/7Qqfv6SU | |
| 20:44:21 | mriedem | cfriesen_: the resource tracker in the compute service | |
| 20:44:31 | openstackgerrit | Merged openstack/nova master: nova-manage: Deprecate 'cell' commands https://review.openstack.org/496815 | |
| 20:46:10 | cfriesen_ | mriedem: which call? I see _update() calling self.scheduler_client.set_inventory_for_provider(), but that looks like it's just setting the inventory. I was expecting to see something removing an allocation. | |
| 20:49:55 | mriedem | _remove_deleted_instances_allocations | |
| 20:50:04 | mriedem | called from _update_usage_from_instances | |
| 20:50:24 | mriedem | called from update_usage | |
| 20:50:35 | mriedem | which is called from ComputeManager._update_resource_tracker | |
| 20:50:45 | cdent | cfriesen_: it begins wiith a great comment: “all this code sucks” | |
| 20:50:48 | mriedem | which is called from _complete_deletion | |
| 20:52:10 | cfriesen_ | mriedem: cdent: update_usage() calls _update_usage_from_instance(), with no "s" on the end | |
| 20:52:30 | mriedem | yeah you're right, which would call update_instance_allocation if we had ocata computes | |
| 20:52:58 | mriedem | _update_usage_from_instances is called via the periodic task | |
| 20:53:06 | openstackgerrit | Ed Leafe proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810 | |
| 20:53:11 | mriedem | update_available_resource in the manager | |
| 20:53:23 | cfriesen_ | mriedem: okay, so currently pike won't free up deleted resources until the audit runs? that sucks. | |
| 20:54:06 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add a spec for minimal cache headers in placement https://review.openstack.org/496853 | |
| 20:54:13 | mriedem | that appears to be the case | |
| 20:55:04 | mriedem | yeah the ServerMovingTests functional tests paper over this by forcing the run of the periodic before checking the allocations are gone in Placement | |
| 20:55:09 | mriedem | cfriesen_: open a bug | |
| 20:55:12 | cfriesen_ | will do | |
| 20:55:31 | cfriesen_ | do we have a unit test for resource update on instance deletion? | |
| 20:57:06 | mriedem | we have functional tests that assert the allocations are removed after the periodic runs, that's the ServerMovingTests functional i mentioned | |
| 20:57:24 | mriedem | but since those force the periodic to run, we glossed over the fact that we're having to wait for the audit | |
| 21:01:01 | cdent | I was under the impression that periodic being required for deletes was effectively a known issue, something we decided was just how it is for now. I agree that we should have a bug for it. | |
| 21:02:37 | cfriesen_ | seems potentially confusing that it'll get freed up immediately in a mixed Ocata/Pike cloud, but once you're fully Pike it's audit-based. | |
| 21:03:59 | mriedem | i just don't know that we thought about the delete case | |
| 21:04:05 | mriedem | the ocata/pike stuff was for a different issue | |
| 21:04:16 | mriedem | where ocata computes would overwrite non-deleted instances being moved to another host | |
| 21:04:24 | mriedem | overwrite allocations i mean | |
| 21:04:38 | mriedem | so not surprisingly while fixing one thing, another issue is introduced | |
| 21:06:09 | cfriesen_ | https://bugs.launchpad.net/nova/+bug/1712684 | |
| 21:06:10 | openstack | Launchpad bug 1712684 in OpenStack Compute (nova) "allocations not immediately removed when instance deleted" [Undecided,New] | |
| 21:16:07 | mriedem | rt.delete_allocation_for_shelve_offloaded_instance(instance) | |
| 21:16:07 | mriedem | yeah so recently (last week), this was added when shelve offloading an instance | |
| 21:16:45 | mriedem | which is basically the exact same thing that happens during the audit when getting the no longer tracked instance results in an InstanceNotFound | |
| 21:17:46 | cdent | there’s for_migrated and for_evacuated as well | |
| 21:17:57 | mriedem | those aren't deleted instances | |
| 21:19:01 | cdent | yeah, I just stumbled on them and realized they are identical | |
| 21:19:30 | cdent | (supporting the theory that the fixing going in concurrently has left some gaps) | |
| 21:20:05 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20clean%20allocation%20of%20a%20shelve%20offloaded%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=10d | |
| 21:20:07 | mriedem | shite ^ | |
| 21:21:34 | mriedem | ffs you know why | |
| 21:21:37 | mriedem | b/c if True | |
| 21:22:10 | mriedem | gdi, ok patching that quick | |
| 21:25:44 | cfriesen_ | delete_allocation_for_migrated_instance() was explicitly copied from the evacuate case | |
| 21:26:00 | mriedem | both of those call a different method | |
| 21:26:06 | mriedem | which returns a boolean | |
| 21:26:11 | mriedem | i just looked at those | |
| 21:28:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: How about not logging errors every time we shelve offload? https://review.openstack.org/496930 | |
| 21:28:35 | mriedem | dansmith: are you going to make me change this commit message title? ^ | |
| 21:29:21 | dansmith | proper capitalization, grammar, and punctuation.. it's better than 100% of sdague's messages, so I don't see the problem | |
| 21:29:33 | mriedem | ha | |
| 21:29:39 | mriedem | i'm going to say it | |
| 21:29:46 | mriedem | I like cookies. I also like pizza. | |
| 21:29:50 | dansmith | lol | |
| 21:29:55 | cfriesen_ | Betteridge's law of headlines says the answer is "no" | |
| 21:30:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Add placeholder migrations for Pike backports https://review.openstack.org/496932 | |
| 21:30:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add uuid to migration object and migrate-on-load https://review.openstack.org/496934 | |
| 21:30:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add uuid to migration table https://review.openstack.org/496933 | |
| 21:31:12 | dansmith | mriedem: need to land that placeholder patch fairly soonish | |
| 21:32:07 | cfriesen_ | it feels wrong somehow to +1 a patch that has no test changes. :) | |
| 21:35:17 | openstackgerrit | Chris Dent proposed openstack/nova master: De-duplicate two delete_allocation_for_* methods https://review.openstack.org/496936 | |
| 21:35:55 | cdent | mriedem: that ^ is not necessary, but would be great to merge once the dust settles, so we can avoid some duplication | |
| 21:36:10 | mriedem | dansmith: good point, i hadn't looked over https://wiki.openstack.org/wiki/Nova/ReleaseChecklist | |
| 21:36:48 | mriedem | cdent: yeah that came up when the 2nd method was added | |
| 21:38:58 | cdent | welp, now it’s ready for whenever | |
| 21:39:08 | cdent | I think that’s the end of my day | |
| 21:39:18 | cdent | ta ra | |
| 21:39:25 | mriedem | o/ | |
| 21:43:01 | mriedem | cfriesen_: dansmith: ^ here is the fix for cfriesen_'s bug | |
| 21:43:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete instance allocations when the instance is deleted https://review.openstack.org/496942 | |
| 21:46:07 | mriedem | ok with that i've got to run to pick up my kid, | |
| 21:46:10 | mriedem | back online later tonight | |
| 21:46:27 | cfriesen_ | mriedem: is there any sensitivity between when we call that and when we call _delete_scheduler_instance_info() ? | |
| 21:46:36 | openstackgerrit | Chris Dent proposed openstack/nova master: De-duplicate two delete_allocation_for_* methods https://review.openstack.org/496936 | |
| 21:46:46 | cfriesen_ | no rush on that, go do kid stuff | |
| 21:47:43 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add a new section: "Upgrade impact" to the template https://review.openstack.org/456756 | |
| 23:08:40 | mwynne | Hi guys. I'm running Ocata and have a bunch of instances that are stuck in a "Deleting" state. | |
| 23:08:59 | mwynne | Resetting the state didn't help. | |
| 23:09:02 | mwynne | Can I get rid of these? | |
| 23:11:02 | mwynne | reset-state just hangs | |
| 23:12:57 | mwynne | force-delete also doesn't work | |
| 23:33:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/488086 | |
| 23:51:55 | mriedem | mwynne: check the nova-api logs for error messages or issues related to the failed requests | |
| 23:59:45 | mriedem | alex_xu: we have a couple more fixes which i think need to get into rc2: https://review.openstack.org/#/c/496930/ and https://review.openstack.org/#/c/496942/ - they are very trivial at least | |
| #openstack-nova - 2017-08-24 | |||
| 00:38:44 | mwynne | mriedem: There's nothing of any use in the logs. | |
| 00:38:52 | mwynne | I had to reboot all my compute nodes. No idea why. | |
| 00:39:54 | mwynne | mriedem: Would you happen to know if I can specify a specific subnet for nota-manage's discover_hosts to search? | |
| 01:14:23 | mriedem | mwynne: nope, doesn't work that way | |
| 01:14:33 | mriedem | it's not discovering hosts based on IPs | |
| 01:19:50 | alex_xu | mriedem: yea, I will check them | |
| 01:21:03 | mriedem | thanks | |
| 01:40:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Centralize allocation deletion in ComputeManager https://review.openstack.org/496976 | |
| 03:14:54 | alex_xu | mriedem: looks like we didn't remove allocations after rescheduling also | |