| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-23 | |||
| 18:52:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add missing tests for _remove_deleted_instances_allocations https://review.openstack.org/496847 | |
| 18:53:43 | mriedem | knikolla: cold migrate it? | |
| 18:53:58 | mriedem | you can cold migrate from stopped state | |
| 18:54:30 | knikolla | mriedem: i stopped it and ran `openstack server migrate <id>` | |
| 18:54:34 | knikolla | after that it goes back to error | |
| 18:54:37 | knikolla | saying it's not running | |
| 18:55:21 | mriedem | what says it's not running? | |
| 18:55:27 | mriedem | the api, compute service, other? | |
| 18:55:53 | knikolla | mriedem: http://paste.openstack.org/show/nfBmtEhE7SScwd0EimVQ/ | |
| 18:59:02 | mriedem | that doesn't tell much - but assuming something failed on the compute service side | |
| 18:59:08 | mriedem | which is why it's recorded as an instance fault | |
| 19:00:49 | knikolla | checking compute logs | |
| 19:01:23 | knikolla | there's a InterfaceDetachFailed: Failed to detach network adapter device from c7703251-96d6-4a2e-b055-7c204ab104e7 | |
| 19:12:20 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add a spec for minimal cache headers in placement https://review.openstack.org/496853 | |
| 19:15:36 | cdent | mriedem: do you recall when sdague comes back on the scene? I’d like to get his input on the microversion-parse adjustments I’ve got in progress | |
| 19:15:55 | mriedem | cdent: next week | |
| 19:16:02 | cdent | ✔ | |
| 19:16:11 | mriedem | his wood paneled station wagon can tour the country only so long | |
| 19:16:45 | mriedem | https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSIjcZl5KWDWlIYAkdWMwyv-yuZYuBUFN8RRc0MzWUwOfn_Uwfs | |
| 19:16:56 | mriedem | ^ is sdague last week | |
| 19:17:32 | cdent | ah the grand ol family truckster | |
| 19:17:35 | cdent | a fine fine auto | |
| 19:18:47 | dansmith | mriedem: he is in a station wagon, but I'm sure it doesn't have tree flesh on it | |
| 19:18:56 | dansmith | mriedem: and it might be powered by love and french fry oil | |
| 19:19:14 | mriedem | ha | |
| 19:19:28 | mriedem | subaru? | |
| 19:19:33 | knikolla | mriedem: the easy fix was to live-migrate another vm which was fine to make enough space for that to start, haha | |
| 19:19:35 | dansmith | yeah | |
| 19:19:50 | mriedem | knikolla: whatever works :) | |
| 19:46:36 | mriedem | now that i'm looking at the cold migration code, i see something odd | |
| 19:46:46 | mriedem | besides the fact it's called resize | |
| 19:47:15 | exarr | Hi all. I have applied the patch as noted here: https://review.openstack.org/#/c/428481/ to allow me to create volumes now, which is great, and seems to work. | |
| 19:47:19 | mriedem | i don't see how this would ever happen https://github.com/openstack/nova/blob/f1f55d48cc753ef3b6ab9b626148dec06e02f8cd/nova/compute/manager.py#L3773 | |
| 19:47:26 | mriedem | added in https://review.openstack.org/#/c/73387/ | |
| 19:47:33 | exarr | However, i'm stuck now with the following error: Exception during message handling: CantStartEngineError: No sql_connection parameter is establ | |
| 19:47:36 | exarr | ished | |
| 19:47:48 | mriedem | that exception is raised from driver.migrate_disk_and_power_off which is called from resize_instance, which is on the source host, after the dest host does and rpc cast from _prep_resize | |
| 19:51:46 | mriedem | there were like 10 people reviewing that change though so i'm not sure how they would all miss that | |
| 19:57:58 | exarr | mriedem: I assume you're not talking to me? :-) | |
| 19:58:03 | mriedem | correct | |
| 20:00:37 | cdent | mriedem: maybe something else (non-local) changed in the interim? | |
| 20:01:11 | mriedem | it has been 3 years, but looking at the code at the time that change was added, it's basically the same code | |
| 20:06:29 | dansmith | mriedem: did you figure out anything with the oslo timeutils thing yesterday? | |
| 20:06:52 | dansmith | I'm getting a weird native/naive comparison failure in unit tests when loading an object | |
| 20:06:55 | dansmith | for no apparent reason | |
| 20:09:01 | mriedem | dansmith: nope | |
| 20:09:03 | mriedem | wasn't looking | |
| 20:09:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add some inline code docs tracing the cold migrate flow https://review.openstack.org/496861 | |
| 20:09:09 | dansmith | okay | |
| 20:09:26 | mriedem | and got totally distracted by https://review.openstack.org/#/c/73387/ | |
| 20:13:21 | mriedem | dansmith: melwitt: are we having a sassy cells meeting this week? we should talk about cells v2 stuff for the ptg at some point, but that could be next week | |
| 20:13:45 | dansmith | mriedem: melwitt is still out today afaik, so whatever you want | |
| 20:14:01 | dansmith | I have a few things I'm planning on starting here pretty soon, but nothing pressing that anyone needs to know about | |
| 20:14:18 | dansmith | so if you want to punt again that's cool, and if you want it to be just us, then that's cool too | |
| 20:17:33 | mriedem | is there anything that's *not* cool? | |
| 20:17:37 | mriedem | fonzy | |
| 20:17:52 | mriedem | let's just punt until next week when mel is back | |
| 20:21:19 | cdent | dansmith: i saw that comparison problem yesterday while messing with timestamps and ovo. somewhere I had utcnow() and using utcnow(with_timezone=True) fixed it. May be totally unrelated but your comment collided with a memory | |
| 20:21:33 | dansmith | yeah totally unrelated | |
| 20:21:46 | dansmith | I was not doing anything with timestamps at all, | |
| 20:21:52 | dansmith | but I figured out why I was triggering the comparison | |
| 20:22:14 | dansmith | don't know why it was doing a naive/aware comparison in the first place, but once I stopped doing my stupid thing it went away | |
| 20:22:20 | dansmith | mriedem: ack on the meeting | |
| 20:22:24 | cdent | no, worries, it cost me little to type that | |
| 20:24:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add missing tests for _remove_deleted_instances_allocations https://review.openstack.org/496847 | |
| 20:35:50 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add a spec for minimal cache headers in placement https://review.openstack.org/496853 | |
| 20:39:40 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add a new section: "Upgrade impact" to the template https://review.openstack.org/456756 | |
| 20:40:38 | openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - add alternate hosts https://review.openstack.org/486215 | |
| 20:40:39 | openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - Add allocations to the values returned from the scheduler https://review.openstack.org/495854 | |
| 20:40:39 | openstackgerrit | Ed Leafe proposed openstack/nova master: return alternates along with their allocations https://review.openstack.org/486253 | |
| 20:41:26 | cfriesen_ | with placement/allocations where do we free up resources on deletion of an instance? | |
| 20:43:20 | exarr | I have a pastebin link if anyone is willing to take a look :-( https://pastebin.com/7Qqfv6SU | |
| 20:44:21 | mriedem | cfriesen_: the resource tracker in the compute service | |
| 20:44:31 | openstackgerrit | Merged openstack/nova master: nova-manage: Deprecate 'cell' commands https://review.openstack.org/496815 | |
| 20:46:10 | cfriesen_ | mriedem: which call? I see _update() calling self.scheduler_client.set_inventory_for_provider(), but that looks like it's just setting the inventory. I was expecting to see something removing an allocation. | |
| 20:49:55 | mriedem | _remove_deleted_instances_allocations | |
| 20:50:04 | mriedem | called from _update_usage_from_instances | |
| 20:50:24 | mriedem | called from update_usage | |
| 20:50:35 | mriedem | which is called from ComputeManager._update_resource_tracker | |
| 20:50:45 | cdent | cfriesen_: it begins wiith a great comment: “all this code sucks” | |
| 20:50:48 | mriedem | which is called from _complete_deletion | |
| 20:52:10 | cfriesen_ | mriedem: cdent: update_usage() calls _update_usage_from_instance(), with no "s" on the end | |
| 20:52:30 | mriedem | yeah you're right, which would call update_instance_allocation if we had ocata computes | |
| 20:52:58 | mriedem | _update_usage_from_instances is called via the periodic task | |
| 20:53:06 | openstackgerrit | Ed Leafe proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810 | |
| 20:53:11 | mriedem | update_available_resource in the manager | |
| 20:53:23 | cfriesen_ | mriedem: okay, so currently pike won't free up deleted resources until the audit runs? that sucks. | |
| 20:54:06 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add a spec for minimal cache headers in placement https://review.openstack.org/496853 | |
| 20:54:13 | mriedem | that appears to be the case | |
| 20:55:04 | mriedem | yeah the ServerMovingTests functional tests paper over this by forcing the run of the periodic before checking the allocations are gone in Placement | |
| 20:55:09 | mriedem | cfriesen_: open a bug | |
| 20:55:12 | cfriesen_ | will do | |
| 20:55:31 | cfriesen_ | do we have a unit test for resource update on instance deletion? | |
| 20:57:06 | mriedem | we have functional tests that assert the allocations are removed after the periodic runs, that's the ServerMovingTests functional i mentioned | |
| 20:57:24 | mriedem | but since those force the periodic to run, we glossed over the fact that we're having to wait for the audit | |
| 21:01:01 | cdent | I was under the impression that periodic being required for deletes was effectively a known issue, something we decided was just how it is for now. I agree that we should have a bug for it. | |
| 21:02:37 | cfriesen_ | seems potentially confusing that it'll get freed up immediately in a mixed Ocata/Pike cloud, but once you're fully Pike it's audit-based. | |
| 21:03:59 | mriedem | i just don't know that we thought about the delete case | |
| 21:04:05 | mriedem | the ocata/pike stuff was for a different issue | |
| 21:04:16 | mriedem | where ocata computes would overwrite non-deleted instances being moved to another host | |