Posted	Nick	Remark
#openstack-nova - 2017-12-20
00:26:48	mnaser	im not sure how we ended up in that situation honestly
00:27:33	mnaser	actually, instance was created on '2015-03-24T09:14:09Z' and instance_extra record was on '2015-03-24 09:14:09' which means that it wasnt part of the migration (sorry if you knew this but it just clicked for me)
00:28:29	mnaser	updated on 2017-10-04 22:30:33 hmm
00:29:52	mnaser	ok that was the date the instance was attempted to be deleted
00:32:31	mriedem	ok hacking something up
00:45:50	mnaser	i feel really bad just finding all these issues all at once lol
00:45:57	mnaser	i just found a keyerror bug :x
00:46:22	mriedem	i don't have a great solution for the flavor thing in the notification. you could disable versioned notifications for now, not sure if you are relying on those.
00:46:59	mnaser	mriedem: we're not atm, but i didnt see a config option to disable it afaik
00:47:28	mriedem	CONF.notifications.notification_format
00:47:42	mriedem	set that to 'unversioned'
00:48:18	mriedem	you'll still get the old unversioned notifications which things like ceilometer consume
00:50:17	mnaser	mriedem: https://bugs.launchpad.net/nova/+bug/1739323 not sure how far back this goes
00:50:18	openstack	Launchpad bug 1739323 in OpenStack Compute (nova) "KeyError in host_manager for _get_host_states" [Undecided,New]
00:51:17	mnaser	looks like its different in stable/newton but ocata+ are probably affected
00:51:58	mnaser	i dont understand why its iterating it back.. is it trying to return a set?
00:52:13	mriedem	can you put a stacktrace in the bug?
00:52:42	mnaser	sure
00:52:48	mriedem	this is all really old i think
00:53:48	mriedem	https://github.com/openstack/nova/commit/b0582603a401b2e71348e9a862b83acc426097df
00:53:50	openstackgerrit	Merged openstack/nova master: Improve error message on invalid BDM fields https://review.openstack.org/527915
00:53:54	mriedem	been around since grizzly
00:54:17	mnaser	mriedem: https://github.com/openstack/nova/commit/4660333d0d97d8e00cf290ea1d4ed932f5edc1dc#diff-978b9f8734365934eaf8fbb01f11a7d7L624 this is what broke it though
00:54:34	mnaser	before it would return it as is.. which means that the KeyError didnt matter.. now it loops over it for some reason
00:55:02	mriedem	oh nice catch, that's ocata
00:55:35	mnaser	ok so six.itervalues => Returns an iterator over dictionary‘s values.
00:55:36	mriedem	i thought you were saying the KeyError was this
00:55:37	mriedem	del self.host_state_map[state_key]
00:55:44	mnaser	no sorry
00:55:48	mnaser	return (self.host_state_map[host] for host in seen_nodes)
00:56:26	mnaser	why not just "return self.host_state_map[host]" i dont know
00:56:32	mnaser	err i mean "return self.host_state_map"
00:56:49	mriedem	not sure, i don't remember why that change was necessary, would have to ask bauzas
00:56:52	mriedem	but it's clearly wrong
00:57:37	mnaser	lol
00:57:37	mnaser	this only took 41 patchsets to merge
00:57:43	mnaser	https://review.openstack.org/#/c/417961/
00:58:10	mriedem	yeah man, and i was in i think cancun at the time on vacation
01:00:00	mriedem	https://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py@100 was also wrong
01:01:52	mnaser	now let me disable versioned notification and see if that gets things moving
01:10:43	mnaser	unversioned notifications did the trick mriedem. that lets me delete the weird vms and live migrate the others. thanks so much for your time, ill track those bugs and hopefully we can get them merged/backported
01:12:03	mriedem	mnaser: can you report a bug for the flavor versioned notification thing?
01:12:09	mnaser	mriedem: oh sure
01:15:04	jaypipes	mriedem, mnaser: just getting back to the house after dinner... have you figured out what was up?
01:15:15	mriedem	jaypipes: https://review.openstack.org/#/q/topic:bug/1739318+(status:open+OR+status:merged)
01:16:35	mriedem	latent bug from newton bites us in the ass starting in pike
01:18:18	mnaser	mriedem: fyi https://bugs.launchpad.net/nova/+bug/1739325
01:18:20	openstack	Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications" [Undecided,New]
01:18:38	mnaser	also thanks for asking jaypipes but yeah i think we're good .. for now, going to wait to get that backported so we can upgrade to latest stable/pike
01:19:03	mriedem	i can feel my wife getting ready to yell at me for still being in my office
01:20:01	mnaser	lol
01:20:11	mnaser	i still have some gift shopping to do..
01:20:14	mnaser	but i guess that's not happening tonight
01:20:35	mnaser	though not looking forward to how much snow is dumped on my car since 8am today and cleaning all that
01:21:33	mriedem	we're virtually snow free in SE minnesota
01:21:37	openstackgerrit	Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194
01:21:41	mriedem	mnaser: so this is a start on that versioned notification bug ^
01:21:46	mriedem	super hacky though i think,
01:21:53	mriedem	but would need other input, likely from dansmith on that one
01:22:01	mriedem	no idea how we could get in that situation to begin with
01:22:20	mnaser	yeah i tried looking everywhere and i couldnt identify a source of how it ended up happening unfortunately
01:22:59	mnaser	it was def not converted because the creation date matched the vm launch date
01:25:20	openstackgerrit	Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194
01:27:51	mriedem	as for https://bugs.launchpad.net/nova/+bug/1739323 i don't think ill be able to crack that tonight
01:27:52	openstack	Launchpad bug 1739323 in OpenStack Compute (nova) pike "KeyError in host_manager for _get_host_states" [High,Confirmed]
01:28:10	mriedem	should be a simple 'in' check or .get() to handle the key not being in there
01:29:41	mnaser	mriedem: i would have done it but it seemed weird why it was changed so i figured maybe there was a... purpose to the decision
01:30:20	mriedem	not sure, kind of why i wanted bauzas to look at it
02:32:39	openstackgerrit	Jay Pipes proposed openstack/nova master: Remove server group sched filter support caching https://review.openstack.org/529200
02:32:39	openstackgerrit	Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753
02:32:40	openstackgerrit	Jay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters https://review.openstack.org/529201
03:01:10	openstackgerrit	Chen Hanxiao proposed openstack/nova master: libvirt: add Linux distribution guest only description for inject_xxx options https://review.openstack.org/528441
03:12:41	openstackgerrit	Alex Xu proposed openstack/nova master: Remove extensions module https://review.openstack.org/529206
03:13:20	alex_xu	oomichi: ^ remove extensions module entirely
03:24:10	openstackgerrit	Alex Xu proposed openstack/nova master: Remove the LoadedExtensionInfo object https://review.openstack.org/529207
03:56:07	openstackgerrit	blkart proposed openstack/nova master: Force delete an instance in any vm_state/task_state https://review.openstack.org/529212
04:22:21	openstackgerrit	Alex Xu proposed openstack/nova master: Merge ResourceV21 obj into Resource obj https://review.openstack.org/529217
04:22:22	openstackgerrit	Alex Xu proposed openstack/nova master: Remove the inherits parameter for the Resource object https://review.openstack.org/529218
05:43:32	jichen	mriedem: hi, canyou please help to take a look at https://review.openstack.org/#/c/523387/ ?
06:12:58	openstackgerrit	Chen Hanxiao proposed openstack/nova master: trivial: conf: libvirt: fix a typo https://review.openstack.org/529229
06:16:07	openstackgerrit	OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795
06:38:15	openstackgerrit	Yikun Jiang (Kero) proposed openstack/nova master: Add migration db and object pagination support. https://review.openstack.org/514904
06:38:54	openstackgerrit	Yikun Jiang (Kero) proposed openstack/nova master: Add cross cell sort support for get_migrations https://review.openstack.org/517273
06:42:25	openstackgerrit	Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406
06:47:08	openstackgerrit	Minho Ban proposed openstack/nova master: prevent removing numa_topology usage info from DB https://review.openstack.org/529236
07:09:49	openstackgerrit	Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900
07:25:45	openstackgerrit	TommyLike proposed openstack/nova master: Remove redundant try/except block when authorize https://review.openstack.org/528991
07:32:14	openstackgerrit	龚肖 proposed openstack/nova master: compute: Catch binding failed exception while init host https://review.openstack.org/529249
07:44:26	openstackgerrit	Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900
07:47:29	openstackgerrit	guanzuoyu proposed openstack/nova-specs master: Fix an spelling error of driver https://review.openstack.org/529251
07:50:54	openstackgerrit	Chen Hanxiao proposed openstack/nova master: InstanceInfo: store more runtime information of instance https://review.openstack.org/526348
07:53:57	openstackgerrit	Marc Koderer proposed openstack/nova master: Enhance error handling for InvalidBDMVolume https://review.openstack.org/529254
07:56:35	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: Fix duplicate allocation candidates https://review.openstack.org/528591
08:16:49	openstackgerrit	Minho Ban proposed openstack/nova master: prevent removing numa_topology usage info from DB https://review.openstack.org/529236
08:17:38	maciejjozefczyk	mriedem: jaypipes thanks for https://review.openstack.org/#/c/520024/
08:22:17	openstackgerrit	龚肖 proposed openstack/nova master: compute: Catch binding failed exception while init host https://review.openstack.org/529249
08:57:15	ameeda	can you please review my code here ? https://review.openstack.org/#/c/528069/ also here https://review.openstack.org/#/c/528385/
09:36:10	openstackgerrit	Lee Yarwood proposed openstack/nova master: libvirt: Refactor encryptor attach and detach calls https://review.openstack.org/460243
09:36:11	openstackgerrit	Lee Yarwood proposed openstack/nova master: WIP libvirt: QEMU native LUKS decryption for volumes https://review.openstack.org/523958
09:36:11	openstackgerrit	Lee Yarwood proposed openstack/nova master: libvirt: Introduce disk encryption config classes https://review.openstack.org/464008

Earlier   Later