| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-20 | |||
| 00:07:04 | openstackgerrit | Merged openstack/nova master: libvirt: throw NotImplementedError if qga is not responsive when setting password https://review.openstack.org/523828 | |
| 00:07:11 | openstackgerrit | Merged openstack/nova master: Update and complete volume attachments during resize https://review.openstack.org/527228 | |
| 00:08:25 | mriedem | mnaser: service record maybe? | |
| 00:08:33 | mriedem | nova.services table | |
| 00:09:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround missing RequestSpec.project_id when moving an instance https://review.openstack.org/529185 | |
| 00:09:30 | mriedem | here is the workaround patch with tests ^ | |
| 00:09:34 | mnaser | http://paste.openstack.org/show/629406/ | |
| 00:09:41 | mnaser | got the full trace back, ill start digging into why | |
| 00:10:07 | mnaser | i guess its a flavor where disabled is none | |
| 00:10:14 | mriedem | yeah... | |
| 00:10:32 | mnaser | instance created at 2015-03-24T09:14:09Z | |
| 00:10:50 | mriedem | so i think in that case, the flavor we're using is pulled out of the embedded flavor stored with the instance | |
| 00:10:59 | mriedem | from the instance_extra table | |
| 00:11:03 | mnaser | most very likely | |
| 00:11:20 | mriedem | not sure why disabled would be null | |
| 00:11:32 | mnaser | let me double check the db | |
| 00:12:01 | mriedem | the disabled column in the db defaults to False | |
| 00:13:29 | mnaser | this is from instance_extra? | |
| 00:14:20 | mriedem | yeah, self.flavor = flavor_payload.FlavorPayload(flavor=instance.flavor) | |
| 00:14:26 | mnaser | http://paste.openstack.org/show/629410/ | |
| 00:14:31 | mriedem | instance.flavor pulls the embedded flavor out of the instance_extra table | |
| 00:15:54 | mnaser | if i recall, the embedded flavor didnt exist before | |
| 00:16:08 | mnaser | could it be possible that when whatever migration ran that did that, it didnt add it? | |
| 00:16:10 | mriedem | we started embedding the flavor in kilo | |
| 00:16:22 | mriedem | flavors were the first online data migration in nova i think | |
| 00:16:30 | mriedem | moving the flavor from the instance_system_metadata table | |
| 00:16:45 | mnaser | i remember something along these lines | |
| 00:16:57 | mriedem | so the serialized form of this flavor says it's version 1.1 of the object which has the disabled field, but maybe it just wasn't set when the flavor was serialized with the instance... | |
| 00:17:29 | mriedem | if you look at a more recent instance, is it set there? | |
| 00:17:34 | mnaser | i think thats what happened, the disabled field seems to be part of when the object was first created | |
| 00:17:39 | mnaser | mriedem: willing to be it is, but let me check to be sure | |
| 00:18:22 | mnaser | mriedem: yup | |
| 00:18:30 | mriedem | ok | |
| 00:23:26 | mnaser | https://github.com/openstack/nova/commit/b4f07f42763489ddf459e4e795477530bdcbd994 | |
| 00:26:15 | mriedem | this is where we set the flavor on the instance during create https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1450 | |
| 00:26:48 | mnaser | im not sure how we ended up in that situation honestly | |
| 00:27:33 | mnaser | actually, instance was created on '2015-03-24T09:14:09Z' and instance_extra record was on '2015-03-24 09:14:09' which means that it wasnt part of the migration (sorry if you knew this but it just clicked for me) | |
| 00:28:29 | mnaser | updated on 2017-10-04 22:30:33 hmm | |
| 00:29:52 | mnaser | ok that was the date the instance was attempted to be deleted | |
| 00:32:31 | mriedem | ok hacking something up | |
| 00:45:50 | mnaser | i feel really bad just finding all these issues all at once lol | |
| 00:45:57 | mnaser | i just found a keyerror bug :x | |
| 00:46:22 | mriedem | i don't have a great solution for the flavor thing in the notification. you could disable versioned notifications for now, not sure if you are relying on those. | |
| 00:46:59 | mnaser | mriedem: we're not atm, but i didnt see a config option to disable it afaik | |
| 00:47:28 | mriedem | CONF.notifications.notification_format | |
| 00:47:42 | mriedem | set that to 'unversioned' | |
| 00:48:18 | mriedem | you'll still get the old unversioned notifications which things like ceilometer consume | |
| 00:50:17 | mnaser | mriedem: https://bugs.launchpad.net/nova/+bug/1739323 not sure how far back this goes | |
| 00:50:18 | openstack | Launchpad bug 1739323 in OpenStack Compute (nova) "KeyError in host_manager for _get_host_states" [Undecided,New] | |
| 00:51:17 | mnaser | looks like its different in stable/newton but ocata+ are probably affected | |
| 00:51:58 | mnaser | i dont understand why its iterating it back.. is it trying to return a set? | |
| 00:52:13 | mriedem | can you put a stacktrace in the bug? | |
| 00:52:42 | mnaser | sure | |
| 00:52:48 | mriedem | this is all really old i think | |
| 00:53:48 | mriedem | https://github.com/openstack/nova/commit/b0582603a401b2e71348e9a862b83acc426097df | |
| 00:53:50 | openstackgerrit | Merged openstack/nova master: Improve error message on invalid BDM fields https://review.openstack.org/527915 | |
| 00:53:54 | mriedem | been around since grizzly | |
| 00:54:17 | mnaser | mriedem: https://github.com/openstack/nova/commit/4660333d0d97d8e00cf290ea1d4ed932f5edc1dc#diff-978b9f8734365934eaf8fbb01f11a7d7L624 this is what broke it though | |
| 00:54:34 | mnaser | before it would return it as is.. which means that the KeyError didnt matter.. now it loops over it for some reason | |
| 00:55:02 | mriedem | oh nice catch, that's ocata | |
| 00:55:35 | mnaser | ok so six.itervalues => Returns an iterator over dictionary‘s values. | |
| 00:55:36 | mriedem | i thought you were saying the KeyError was this | |
| 00:55:37 | mriedem | del self.host_state_map[state_key] | |
| 00:55:44 | mnaser | no sorry | |
| 00:55:48 | mnaser | return (self.host_state_map[host] for host in seen_nodes) | |
| 00:56:26 | mnaser | why not just "return self.host_state_map[host]" i dont know | |
| 00:56:32 | mnaser | err i mean "return self.host_state_map" | |
| 00:56:49 | mriedem | not sure, i don't remember why that change was necessary, would have to ask bauzas | |
| 00:56:52 | mriedem | but it's clearly wrong | |
| 00:57:37 | mnaser | this only took 41 patchsets to merge | |
| 00:57:37 | mnaser | lol | |
| 00:57:43 | mnaser | https://review.openstack.org/#/c/417961/ | |
| 00:58:10 | mriedem | yeah man, and i was in i think cancun at the time on vacation | |
| 01:00:00 | mriedem | https://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py@100 was also wrong | |
| 01:01:52 | mnaser | now let me disable versioned notification and see if that gets things moving | |
| 01:10:43 | mnaser | unversioned notifications did the trick mriedem. that lets me delete the weird vms and live migrate the others. thanks so much for your time, ill track those bugs and hopefully we can get them merged/backported | |
| 01:12:03 | mriedem | mnaser: can you report a bug for the flavor versioned notification thing? | |
| 01:12:09 | mnaser | mriedem: oh sure | |
| 01:15:04 | jaypipes | mriedem, mnaser: just getting back to the house after dinner... have you figured out what was up? | |
| 01:15:15 | mriedem | jaypipes: https://review.openstack.org/#/q/topic:bug/1739318+(status:open+OR+status:merged) | |
| 01:16:35 | mriedem | latent bug from newton bites us in the ass starting in pike | |
| 01:18:18 | mnaser | mriedem: fyi https://bugs.launchpad.net/nova/+bug/1739325 | |
| 01:18:20 | openstack | Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications" [Undecided,New] | |
| 01:18:38 | mnaser | also thanks for asking jaypipes but yeah i think we're good .. for now, going to wait to get that backported so we can upgrade to latest stable/pike | |
| 01:19:03 | mriedem | i can feel my wife getting ready to yell at me for still being in my office | |
| 01:20:01 | mnaser | lol | |
| 01:20:11 | mnaser | i still have some gift shopping to do.. | |
| 01:20:14 | mnaser | but i guess that's not happening tonight | |
| 01:20:35 | mnaser | though not looking forward to how much snow is dumped on my car since 8am today and cleaning all that | |
| 01:21:33 | mriedem | we're virtually snow free in SE minnesota | |
| 01:21:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194 | |
| 01:21:41 | mriedem | mnaser: so this is a start on that versioned notification bug ^ | |
| 01:21:46 | mriedem | super hacky though i think, | |
| 01:21:53 | mriedem | but would need other input, likely from dansmith on that one | |
| 01:22:01 | mriedem | no idea how we could get in that situation to begin with | |
| 01:22:20 | mnaser | yeah i tried looking everywhere and i couldnt identify a source of how it ended up happening unfortunately | |
| 01:22:59 | mnaser | it was def not converted because the creation date matched the vm launch date | |
| 01:25:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194 | |
| 01:27:51 | mriedem | as for https://bugs.launchpad.net/nova/+bug/1739323 i don't think ill be able to crack that tonight | |
| 01:27:52 | openstack | Launchpad bug 1739323 in OpenStack Compute (nova) pike "KeyError in host_manager for _get_host_states" [High,Confirmed] | |