| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-19 | |||
| 23:52:54 | mnaser | i wouldnt be able to test the migrate and conductor changes as those are too critical tbh | |
| 23:53:42 | mnaser | by conductor, the conductor manager change that is | |
| 23:55:05 | mriedem | yeah | |
| 23:55:08 | mnaser | oh fun times | |
| 23:55:15 | mnaser | this will conflict in stable/pike | |
| 23:55:46 | mnaser | _get_request_spec_for_select_destinations doesnt exist in stable/pike .. not in my stable/pike | |
| 23:56:03 | mnaser | tasks | |
| 23:57:52 | mriedem | that code is in _find_destination in pike | |
| 23:58:33 | mnaser | ok i see | |
| 23:58:33 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/conductor/tasks/live_migrate.py#L264 | |
| #openstack-nova - 2017-12-20 | |||
| 00:00:34 | mnaser | well, here goes nothing | |
| 00:01:05 | mnaser | ok, conductor backup with patch, gonna try a live migration | |
| 00:01:28 | mnaser | status=MIGRATING | |
| 00:01:47 | mnaser | let me watch nova-compute to check everything is ok | |
| 00:02:20 | mnaser | live migrated successfully *without* forcing host, let me try forcing host | |
| 00:03:43 | mnaser | mriedem: fixed for both forced host and unspecified host for live migrations | |
| 00:04:02 | mriedem | sweet | |
| 00:04:05 | mnaser | thank you so much, moving forwards i guess we'll have to find a way to clean this up | |
| 00:04:08 | mriedem | about done with tests for the workaround | |
| 00:06:32 | openstackgerrit | Merged openstack/nova master: Remove the objects which related to the old v2 API implementation https://review.openstack.org/519997 | |
| 00:06:40 | openstackgerrit | Merged openstack/nova master: Remove the objects for describing the extension for v2.1 API https://review.openstack.org/519998 | |
| 00:06:58 | mnaser | found some other VMs in error state which failed to delete ... "{"message": "Field `disabled' cannot be None", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 205," | |
| 00:07:03 | mnaser | investigation round #2 | |
| 00:07:04 | openstackgerrit | Merged openstack/nova master: libvirt: throw NotImplementedError if qga is not responsive when setting password https://review.openstack.org/523828 | |
| 00:07:11 | openstackgerrit | Merged openstack/nova master: Update and complete volume attachments during resize https://review.openstack.org/527228 | |
| 00:08:25 | mriedem | mnaser: service record maybe? | |
| 00:08:33 | mriedem | nova.services table | |
| 00:09:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround missing RequestSpec.project_id when moving an instance https://review.openstack.org/529185 | |
| 00:09:30 | mriedem | here is the workaround patch with tests ^ | |
| 00:09:34 | mnaser | http://paste.openstack.org/show/629406/ | |
| 00:09:41 | mnaser | got the full trace back, ill start digging into why | |
| 00:10:07 | mnaser | i guess its a flavor where disabled is none | |
| 00:10:14 | mriedem | yeah... | |
| 00:10:32 | mnaser | instance created at 2015-03-24T09:14:09Z | |
| 00:10:50 | mriedem | so i think in that case, the flavor we're using is pulled out of the embedded flavor stored with the instance | |
| 00:10:59 | mriedem | from the instance_extra table | |
| 00:11:03 | mnaser | most very likely | |
| 00:11:20 | mriedem | not sure why disabled would be null | |
| 00:11:32 | mnaser | let me double check the db | |
| 00:12:01 | mriedem | the disabled column in the db defaults to False | |
| 00:13:29 | mnaser | this is from instance_extra? | |
| 00:14:20 | mriedem | yeah, self.flavor = flavor_payload.FlavorPayload(flavor=instance.flavor) | |
| 00:14:26 | mnaser | http://paste.openstack.org/show/629410/ | |
| 00:14:31 | mriedem | instance.flavor pulls the embedded flavor out of the instance_extra table | |
| 00:15:54 | mnaser | if i recall, the embedded flavor didnt exist before | |
| 00:16:08 | mnaser | could it be possible that when whatever migration ran that did that, it didnt add it? | |
| 00:16:10 | mriedem | we started embedding the flavor in kilo | |
| 00:16:22 | mriedem | flavors were the first online data migration in nova i think | |
| 00:16:30 | mriedem | moving the flavor from the instance_system_metadata table | |
| 00:16:45 | mnaser | i remember something along these lines | |
| 00:16:57 | mriedem | so the serialized form of this flavor says it's version 1.1 of the object which has the disabled field, but maybe it just wasn't set when the flavor was serialized with the instance... | |
| 00:17:29 | mriedem | if you look at a more recent instance, is it set there? | |
| 00:17:34 | mnaser | i think thats what happened, the disabled field seems to be part of when the object was first created | |
| 00:17:39 | mnaser | mriedem: willing to be it is, but let me check to be sure | |
| 00:18:22 | mnaser | mriedem: yup | |
| 00:18:30 | mriedem | ok | |
| 00:23:26 | mnaser | https://github.com/openstack/nova/commit/b4f07f42763489ddf459e4e795477530bdcbd994 | |
| 00:26:15 | mriedem | this is where we set the flavor on the instance during create https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1450 | |
| 00:26:48 | mnaser | im not sure how we ended up in that situation honestly | |
| 00:27:33 | mnaser | actually, instance was created on '2015-03-24T09:14:09Z' and instance_extra record was on '2015-03-24 09:14:09' which means that it wasnt part of the migration (sorry if you knew this but it just clicked for me) | |
| 00:28:29 | mnaser | updated on 2017-10-04 22:30:33 hmm | |
| 00:29:52 | mnaser | ok that was the date the instance was attempted to be deleted | |
| 00:32:31 | mriedem | ok hacking something up | |
| 00:45:50 | mnaser | i feel really bad just finding all these issues all at once lol | |
| 00:45:57 | mnaser | i just found a keyerror bug :x | |
| 00:46:22 | mriedem | i don't have a great solution for the flavor thing in the notification. you could disable versioned notifications for now, not sure if you are relying on those. | |
| 00:46:59 | mnaser | mriedem: we're not atm, but i didnt see a config option to disable it afaik | |
| 00:47:28 | mriedem | CONF.notifications.notification_format | |
| 00:47:42 | mriedem | set that to 'unversioned' | |
| 00:48:18 | mriedem | you'll still get the old unversioned notifications which things like ceilometer consume | |
| 00:50:17 | mnaser | mriedem: https://bugs.launchpad.net/nova/+bug/1739323 not sure how far back this goes | |
| 00:50:18 | openstack | Launchpad bug 1739323 in OpenStack Compute (nova) "KeyError in host_manager for _get_host_states" [Undecided,New] | |
| 00:51:17 | mnaser | looks like its different in stable/newton but ocata+ are probably affected | |
| 00:51:58 | mnaser | i dont understand why its iterating it back.. is it trying to return a set? | |
| 00:52:13 | mriedem | can you put a stacktrace in the bug? | |
| 00:52:42 | mnaser | sure | |
| 00:52:48 | mriedem | this is all really old i think | |
| 00:53:48 | mriedem | https://github.com/openstack/nova/commit/b0582603a401b2e71348e9a862b83acc426097df | |
| 00:53:50 | openstackgerrit | Merged openstack/nova master: Improve error message on invalid BDM fields https://review.openstack.org/527915 | |
| 00:53:54 | mriedem | been around since grizzly | |
| 00:54:17 | mnaser | mriedem: https://github.com/openstack/nova/commit/4660333d0d97d8e00cf290ea1d4ed932f5edc1dc#diff-978b9f8734365934eaf8fbb01f11a7d7L624 this is what broke it though | |
| 00:54:34 | mnaser | before it would return it as is.. which means that the KeyError didnt matter.. now it loops over it for some reason | |
| 00:55:02 | mriedem | oh nice catch, that's ocata | |
| 00:55:35 | mnaser | ok so six.itervalues => Returns an iterator over dictionary‘s values. | |
| 00:55:36 | mriedem | i thought you were saying the KeyError was this | |
| 00:55:37 | mriedem | del self.host_state_map[state_key] | |
| 00:55:44 | mnaser | no sorry | |
| 00:55:48 | mnaser | return (self.host_state_map[host] for host in seen_nodes) | |
| 00:56:26 | mnaser | why not just "return self.host_state_map[host]" i dont know | |
| 00:56:32 | mnaser | err i mean "return self.host_state_map" | |
| 00:56:49 | mriedem | not sure, i don't remember why that change was necessary, would have to ask bauzas | |
| 00:56:52 | mriedem | but it's clearly wrong | |
| 00:57:37 | mnaser | lol | |
| 00:57:37 | mnaser | this only took 41 patchsets to merge | |
| 00:57:43 | mnaser | https://review.openstack.org/#/c/417961/ | |
| 00:58:10 | mriedem | yeah man, and i was in i think cancun at the time on vacation | |
| 01:00:00 | mriedem | https://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py@100 was also wrong | |
| 01:01:52 | mnaser | now let me disable versioned notification and see if that gets things moving | |
| 01:10:43 | mnaser | unversioned notifications did the trick mriedem. that lets me delete the weird vms and live migrate the others. thanks so much for your time, ill track those bugs and hopefully we can get them merged/backported | |
| 01:12:03 | mriedem | mnaser: can you report a bug for the flavor versioned notification thing? | |