| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-20 | |||
| 00:16:08 | mnaser | could it be possible that when whatever migration ran that did that, it didnt add it? | |
| 00:16:10 | mriedem | we started embedding the flavor in kilo | |
| 00:16:22 | mriedem | flavors were the first online data migration in nova i think | |
| 00:16:30 | mriedem | moving the flavor from the instance_system_metadata table | |
| 00:16:45 | mnaser | i remember something along these lines | |
| 00:16:57 | mriedem | so the serialized form of this flavor says it's version 1.1 of the object which has the disabled field, but maybe it just wasn't set when the flavor was serialized with the instance... | |
| 00:17:29 | mriedem | if you look at a more recent instance, is it set there? | |
| 00:17:34 | mnaser | i think thats what happened, the disabled field seems to be part of when the object was first created | |
| 00:17:39 | mnaser | mriedem: willing to be it is, but let me check to be sure | |
| 00:18:22 | mnaser | mriedem: yup | |
| 00:18:30 | mriedem | ok | |
| 00:23:26 | mnaser | https://github.com/openstack/nova/commit/b4f07f42763489ddf459e4e795477530bdcbd994 | |
| 00:26:15 | mriedem | this is where we set the flavor on the instance during create https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1450 | |
| 00:26:48 | mnaser | im not sure how we ended up in that situation honestly | |
| 00:27:33 | mnaser | actually, instance was created on '2015-03-24T09:14:09Z' and instance_extra record was on '2015-03-24 09:14:09' which means that it wasnt part of the migration (sorry if you knew this but it just clicked for me) | |
| 00:28:29 | mnaser | updated on 2017-10-04 22:30:33 hmm | |
| 00:29:52 | mnaser | ok that was the date the instance was attempted to be deleted | |
| 00:32:31 | mriedem | ok hacking something up | |
| 00:45:50 | mnaser | i feel really bad just finding all these issues all at once lol | |
| 00:45:57 | mnaser | i just found a keyerror bug :x | |
| 00:46:22 | mriedem | i don't have a great solution for the flavor thing in the notification. you could disable versioned notifications for now, not sure if you are relying on those. | |
| 00:46:59 | mnaser | mriedem: we're not atm, but i didnt see a config option to disable it afaik | |
| 00:47:28 | mriedem | CONF.notifications.notification_format | |
| 00:47:42 | mriedem | set that to 'unversioned' | |
| 00:48:18 | mriedem | you'll still get the old unversioned notifications which things like ceilometer consume | |
| 00:50:17 | mnaser | mriedem: https://bugs.launchpad.net/nova/+bug/1739323 not sure how far back this goes | |
| 00:50:18 | openstack | Launchpad bug 1739323 in OpenStack Compute (nova) "KeyError in host_manager for _get_host_states" [Undecided,New] | |
| 00:51:17 | mnaser | looks like its different in stable/newton but ocata+ are probably affected | |
| 00:51:58 | mnaser | i dont understand why its iterating it back.. is it trying to return a set? | |
| 00:52:13 | mriedem | can you put a stacktrace in the bug? | |
| 00:52:42 | mnaser | sure | |
| 00:52:48 | mriedem | this is all really old i think | |
| 00:53:48 | mriedem | https://github.com/openstack/nova/commit/b0582603a401b2e71348e9a862b83acc426097df | |
| 00:53:50 | openstackgerrit | Merged openstack/nova master: Improve error message on invalid BDM fields https://review.openstack.org/527915 | |
| 00:53:54 | mriedem | been around since grizzly | |
| 00:54:17 | mnaser | mriedem: https://github.com/openstack/nova/commit/4660333d0d97d8e00cf290ea1d4ed932f5edc1dc#diff-978b9f8734365934eaf8fbb01f11a7d7L624 this is what broke it though | |
| 00:54:34 | mnaser | before it would return it as is.. which means that the KeyError didnt matter.. now it loops over it for some reason | |
| 00:55:02 | mriedem | oh nice catch, that's ocata | |
| 00:55:35 | mnaser | ok so six.itervalues => Returns an iterator over dictionary‘s values. | |
| 00:55:36 | mriedem | i thought you were saying the KeyError was this | |
| 00:55:37 | mriedem | del self.host_state_map[state_key] | |
| 00:55:44 | mnaser | no sorry | |
| 00:55:48 | mnaser | return (self.host_state_map[host] for host in seen_nodes) | |
| 00:56:26 | mnaser | why not just "return self.host_state_map[host]" i dont know | |
| 00:56:32 | mnaser | err i mean "return self.host_state_map" | |
| 00:56:49 | mriedem | not sure, i don't remember why that change was necessary, would have to ask bauzas | |
| 00:56:52 | mriedem | but it's clearly wrong | |
| 00:57:37 | mnaser | this only took 41 patchsets to merge | |
| 00:57:37 | mnaser | lol | |
| 00:57:43 | mnaser | https://review.openstack.org/#/c/417961/ | |
| 00:58:10 | mriedem | yeah man, and i was in i think cancun at the time on vacation | |
| 01:00:00 | mriedem | https://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py@100 was also wrong | |
| 01:01:52 | mnaser | now let me disable versioned notification and see if that gets things moving | |
| 01:10:43 | mnaser | unversioned notifications did the trick mriedem. that lets me delete the weird vms and live migrate the others. thanks so much for your time, ill track those bugs and hopefully we can get them merged/backported | |
| 01:12:03 | mriedem | mnaser: can you report a bug for the flavor versioned notification thing? | |
| 01:12:09 | mnaser | mriedem: oh sure | |
| 01:15:04 | jaypipes | mriedem, mnaser: just getting back to the house after dinner... have you figured out what was up? | |
| 01:15:15 | mriedem | jaypipes: https://review.openstack.org/#/q/topic:bug/1739318+(status:open+OR+status:merged) | |
| 01:16:35 | mriedem | latent bug from newton bites us in the ass starting in pike | |
| 01:18:18 | mnaser | mriedem: fyi https://bugs.launchpad.net/nova/+bug/1739325 | |
| 01:18:20 | openstack | Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications" [Undecided,New] | |
| 01:18:38 | mnaser | also thanks for asking jaypipes but yeah i think we're good .. for now, going to wait to get that backported so we can upgrade to latest stable/pike | |
| 01:19:03 | mriedem | i can feel my wife getting ready to yell at me for still being in my office | |
| 01:20:01 | mnaser | lol | |
| 01:20:11 | mnaser | i still have some gift shopping to do.. | |
| 01:20:14 | mnaser | but i guess that's not happening tonight | |
| 01:20:35 | mnaser | though not looking forward to how much snow is dumped on my car since 8am today and cleaning all that | |
| 01:21:33 | mriedem | we're virtually snow free in SE minnesota | |
| 01:21:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194 | |
| 01:21:41 | mriedem | mnaser: so this is a start on that versioned notification bug ^ | |
| 01:21:46 | mriedem | super hacky though i think, | |
| 01:21:53 | mriedem | but would need other input, likely from dansmith on that one | |
| 01:22:01 | mriedem | no idea how we could get in that situation to begin with | |
| 01:22:20 | mnaser | yeah i tried looking everywhere and i couldnt identify a source of how it ended up happening unfortunately | |
| 01:22:59 | mnaser | it was def not converted because the creation date matched the vm launch date | |
| 01:25:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194 | |
| 01:27:51 | mriedem | as for https://bugs.launchpad.net/nova/+bug/1739323 i don't think ill be able to crack that tonight | |
| 01:27:52 | openstack | Launchpad bug 1739323 in OpenStack Compute (nova) pike "KeyError in host_manager for _get_host_states" [High,Confirmed] | |
| 01:28:10 | mriedem | should be a simple 'in' check or .get() to handle the key not being in there | |
| 01:29:41 | mnaser | mriedem: i would have done it but it seemed weird why it was changed so i figured maybe there was a... purpose to the decision | |
| 01:30:20 | mriedem | not sure, kind of why i wanted bauzas to look at it | |
| 02:32:39 | openstackgerrit | Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753 | |
| 02:32:39 | openstackgerrit | Jay Pipes proposed openstack/nova master: Remove server group sched filter support caching https://review.openstack.org/529200 | |
| 02:32:40 | openstackgerrit | Jay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters https://review.openstack.org/529201 | |
| 03:01:10 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: libvirt: add Linux distribution guest only description for inject_xxx options https://review.openstack.org/528441 | |
| 03:12:41 | openstackgerrit | Alex Xu proposed openstack/nova master: Remove extensions module https://review.openstack.org/529206 | |
| 03:13:20 | alex_xu | oomichi: ^ remove extensions module entirely | |
| 03:24:10 | openstackgerrit | Alex Xu proposed openstack/nova master: Remove the LoadedExtensionInfo object https://review.openstack.org/529207 | |
| 03:56:07 | openstackgerrit | blkart proposed openstack/nova master: Force delete an instance in any vm_state/task_state https://review.openstack.org/529212 | |
| 04:22:21 | openstackgerrit | Alex Xu proposed openstack/nova master: Merge ResourceV21 obj into Resource obj https://review.openstack.org/529217 | |
| 04:22:22 | openstackgerrit | Alex Xu proposed openstack/nova master: Remove the inherits parameter for the Resource object https://review.openstack.org/529218 | |
| 05:43:32 | jichen | mriedem: hi, canyou please help to take a look at https://review.openstack.org/#/c/523387/ ? | |
| 06:12:58 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: trivial: conf: libvirt: fix a typo https://review.openstack.org/529229 | |
| 06:16:07 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795 | |
| 06:38:15 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add migration db and object pagination support. https://review.openstack.org/514904 | |
| 06:38:54 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add cross cell sort support for get_migrations https://review.openstack.org/517273 | |
| 06:42:25 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406 | |
| 06:47:08 | openstackgerrit | Minho Ban proposed openstack/nova master: prevent removing numa_topology usage info from DB https://review.openstack.org/529236 | |
| 07:09:49 | openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | |
| 07:25:45 | openstackgerrit | TommyLike proposed openstack/nova master: Remove redundant try/except block when authorize https://review.openstack.org/528991 | |