| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-23 | |||
| 20:55:31 | cfriesen_ | do we have a unit test for resource update on instance deletion? | |
| 20:57:06 | mriedem | we have functional tests that assert the allocations are removed after the periodic runs, that's the ServerMovingTests functional i mentioned | |
| 20:57:24 | mriedem | but since those force the periodic to run, we glossed over the fact that we're having to wait for the audit | |
| 21:01:01 | cdent | I was under the impression that periodic being required for deletes was effectively a known issue, something we decided was just how it is for now. I agree that we should have a bug for it. | |
| 21:02:37 | cfriesen_ | seems potentially confusing that it'll get freed up immediately in a mixed Ocata/Pike cloud, but once you're fully Pike it's audit-based. | |
| 21:03:59 | mriedem | i just don't know that we thought about the delete case | |
| 21:04:05 | mriedem | the ocata/pike stuff was for a different issue | |
| 21:04:16 | mriedem | where ocata computes would overwrite non-deleted instances being moved to another host | |
| 21:04:24 | mriedem | overwrite allocations i mean | |
| 21:04:38 | mriedem | so not surprisingly while fixing one thing, another issue is introduced | |
| 21:06:09 | cfriesen_ | https://bugs.launchpad.net/nova/+bug/1712684 | |
| 21:06:10 | openstack | Launchpad bug 1712684 in OpenStack Compute (nova) "allocations not immediately removed when instance deleted" [Undecided,New] | |
| 21:16:07 | mriedem | yeah so recently (last week), this was added when shelve offloading an instance | |
| 21:16:07 | mriedem | rt.delete_allocation_for_shelve_offloaded_instance(instance) | |
| 21:16:45 | mriedem | which is basically the exact same thing that happens during the audit when getting the no longer tracked instance results in an InstanceNotFound | |
| 21:17:46 | cdent | there’s for_migrated and for_evacuated as well | |
| 21:17:57 | mriedem | those aren't deleted instances | |
| 21:19:01 | cdent | yeah, I just stumbled on them and realized they are identical | |
| 21:19:30 | cdent | (supporting the theory that the fixing going in concurrently has left some gaps) | |
| 21:20:05 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20clean%20allocation%20of%20a%20shelve%20offloaded%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=10d | |
| 21:20:07 | mriedem | shite ^ | |
| 21:21:34 | mriedem | ffs you know why | |
| 21:21:37 | mriedem | b/c if True | |
| 21:22:10 | mriedem | gdi, ok patching that quick | |
| 21:25:44 | cfriesen_ | delete_allocation_for_migrated_instance() was explicitly copied from the evacuate case | |
| 21:26:00 | mriedem | both of those call a different method | |
| 21:26:06 | mriedem | which returns a boolean | |
| 21:26:11 | mriedem | i just looked at those | |
| 21:28:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: How about not logging errors every time we shelve offload? https://review.openstack.org/496930 | |
| 21:28:35 | mriedem | dansmith: are you going to make me change this commit message title? ^ | |
| 21:29:21 | dansmith | proper capitalization, grammar, and punctuation.. it's better than 100% of sdague's messages, so I don't see the problem | |
| 21:29:33 | mriedem | ha | |
| 21:29:39 | mriedem | i'm going to say it | |
| 21:29:46 | mriedem | I like cookies. I also like pizza. | |
| 21:29:50 | dansmith | lol | |
| 21:29:55 | cfriesen_ | Betteridge's law of headlines says the answer is "no" | |
| 21:30:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Add placeholder migrations for Pike backports https://review.openstack.org/496932 | |
| 21:30:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add uuid to migration table https://review.openstack.org/496933 | |
| 21:30:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add uuid to migration object and migrate-on-load https://review.openstack.org/496934 | |
| 21:31:12 | dansmith | mriedem: need to land that placeholder patch fairly soonish | |
| 21:32:07 | cfriesen_ | it feels wrong somehow to +1 a patch that has no test changes. :) | |
| 21:35:17 | openstackgerrit | Chris Dent proposed openstack/nova master: De-duplicate two delete_allocation_for_* methods https://review.openstack.org/496936 | |
| 21:35:55 | cdent | mriedem: that ^ is not necessary, but would be great to merge once the dust settles, so we can avoid some duplication | |
| 21:36:10 | mriedem | dansmith: good point, i hadn't looked over https://wiki.openstack.org/wiki/Nova/ReleaseChecklist | |
| 21:36:48 | mriedem | cdent: yeah that came up when the 2nd method was added | |
| 21:38:58 | cdent | welp, now it’s ready for whenever | |
| 21:39:08 | cdent | I think that’s the end of my day | |
| 21:39:18 | cdent | ta ra | |
| 21:39:25 | mriedem | o/ | |
| 21:43:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete instance allocations when the instance is deleted https://review.openstack.org/496942 | |
| 21:43:01 | mriedem | cfriesen_: dansmith: ^ here is the fix for cfriesen_'s bug | |
| 21:46:07 | mriedem | ok with that i've got to run to pick up my kid, | |
| 21:46:10 | mriedem | back online later tonight | |
| 21:46:27 | cfriesen_ | mriedem: is there any sensitivity between when we call that and when we call _delete_scheduler_instance_info() ? | |
| 21:46:36 | openstackgerrit | Chris Dent proposed openstack/nova master: De-duplicate two delete_allocation_for_* methods https://review.openstack.org/496936 | |
| 21:46:46 | cfriesen_ | no rush on that, go do kid stuff | |
| 21:47:43 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add a new section: "Upgrade impact" to the template https://review.openstack.org/456756 | |
| 23:08:40 | mwynne | Hi guys. I'm running Ocata and have a bunch of instances that are stuck in a "Deleting" state. | |
| 23:08:59 | mwynne | Resetting the state didn't help. | |
| 23:09:02 | mwynne | Can I get rid of these? | |
| 23:11:02 | mwynne | reset-state just hangs | |
| 23:12:57 | mwynne | force-delete also doesn't work | |
| 23:33:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/488086 | |
| 23:51:55 | mriedem | mwynne: check the nova-api logs for error messages or issues related to the failed requests | |
| 23:59:45 | mriedem | alex_xu: we have a couple more fixes which i think need to get into rc2: https://review.openstack.org/#/c/496930/ and https://review.openstack.org/#/c/496942/ - they are very trivial at least | |
| #openstack-nova - 2017-08-24 | |||
| 00:38:44 | mwynne | mriedem: There's nothing of any use in the logs. | |
| 00:38:52 | mwynne | I had to reboot all my compute nodes. No idea why. | |
| 00:39:54 | mwynne | mriedem: Would you happen to know if I can specify a specific subnet for nota-manage's discover_hosts to search? | |
| 01:14:23 | mriedem | mwynne: nope, doesn't work that way | |
| 01:14:33 | mriedem | it's not discovering hosts based on IPs | |
| 01:19:50 | alex_xu | mriedem: yea, I will check them | |
| 01:21:03 | mriedem | thanks | |
| 01:40:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Centralize allocation deletion in ComputeManager https://review.openstack.org/496976 | |
| 03:14:54 | alex_xu | mriedem: looks like we didn't remove allocations after rescheduling also | |
| 03:17:23 | openstackgerrit | Alex Xu proposed openstack/nova master: Add functional test for rescheduling https://review.openstack.org/496995 | |
| 03:20:14 | openstackgerrit | Alex Xu proposed openstack/nova master: Add functional test for rescheduling https://review.openstack.org/496995 | |
| 03:22:50 | mriedem | alex_xu: hmm, yeah probably not, which sounds similar to https://bugs.launchpad.net/nova/+bug/1712411 | |
| 03:22:51 | openstack | Launchpad bug 1712411 in OpenStack Compute (nova) pike "Allocations may not be removed from dest node during failed migrations" [High,Triaged] | |
| 03:23:11 | mriedem | remove allocations during reschedule is now going to be like deallocating ports from a host | |
| 03:23:39 | mriedem | alex_xu: do you want to open a new bug for the reschedule issue? | |
| 03:29:51 | openstackgerrit | Merged openstack/nova master: Delete instance allocations when the instance is deleted https://review.openstack.org/496942 | |
| 04:02:07 | alex_xu | mriedem: yea, I already opened one https://bugs.launchpad.net/nova/+bug/1712718 | |
| 04:02:08 | openstack | Launchpad bug 1712718 in OpenStack Compute (nova) "Allocations records aren't removed after rescheduling" [Undecided,New] | |
| 04:04:23 | alex_xu | mrhillsman: yea, very similar with https://bugs.launchpad.net/nova/+bug/1712411 | |
| 04:04:24 | openstack | Launchpad bug 1712411 in OpenStack Compute (nova) pike "Allocations may not be removed from dest node during failed migrations" [High,Triaged] | |
| 04:04:43 | alex_xu | mriedem: yea, very similar with https://bugs.launchpad.net/nova/+bug/1712411 | |
| 04:05:11 | alex_xu | just for the case instance boot rescheduling | |
| 06:14:16 | openstackgerrit | Alex Xu proposed openstack/nova master: Add functional test for rescheduling https://review.openstack.org/496995 | |
| 06:54:41 | openstackgerrit | Alex Xu proposed openstack/nova master: Remove allocation when booting instance rescheduled or aborted https://review.openstack.org/496995 | |
| 07:14:56 | openstackgerrit | liusheng proposed openstack/nova master: Trivail: remove unused instance_group_members_add method https://review.openstack.org/497115 | |
| 07:17:05 | openstackgerrit | liusheng proposed openstack/nova master: Trivial: remove unused instance_group_member_delete method https://review.openstack.org/497115 | |
| 08:30:33 | openstackgerrit | jichenjc proposed openstack/nova master: Do not attempt volume swap when guest is stopped/suspended https://review.openstack.org/389798 | |
| 08:49:31 | openstackgerrit | jichenjc proposed openstack/nova master: WIP: use conf for keystone session creation https://review.openstack.org/485121 | |
| 09:02:17 | openstackgerrit | jichenjc proposed openstack/nova master: fix race condition of instance host https://review.openstack.org/494458 | |
| 09:10:16 | ratailor | Can anybody tell me how to debug functional test ? | |
| 09:28:33 | openstackgerrit | jichenjc proposed openstack/nova master: propagate OSError to MigrationPreCheckError https://review.openstack.org/469779 | |
| 09:30:07 | openstackgerrit | jichenjc proposed openstack/nova master: WIP: fix warning: enable-extensions https://review.openstack.org/497170 | |
| 09:30:53 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Websockify security proxy framework https://review.openstack.org/496160 | |
| 09:48:51 | ratailor | Dinesh_Bhor, do you know how to debug functional test ? | |
| 09:56:33 | owalsh | moshele: do you have a minute to confirm some details about pci alias config? | |