Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-18
16:30:52 openstackgerrit Merged openstack/nova master: Fix inconsistency in docs https://review.openstack.org/569348
16:34:09 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300
16:54:42 superdan mriedem: so, I don't think my instance change actually caused that error
16:54:54 superdan mriedem: here's another example from a different unrelated patch: http://logs.openstack.org/38/564838/6/gate/nova-cells-v1/5fc0493/logs/screen-n-cell-child.txt.gz?level=TRACE#_May_18_14_09_53_542120
16:56:08 superdan mriedem: my DNM testing patch above should help uncover what it's actually choking on
17:38:32 superdan mriedem: that cells failure is trying to load keypairs
17:39:06 superdan mriedem: which is probably because the instance that came out of the cell database can't have loaded that from the api db down there or some such
17:40:30 jgwentworth I wonder why the cells v1 job is otherwise on the fritz today
17:40:48 jgwentworth seeing a lot of fails and rechecks in my gerrit notifications
17:40:59 superdan jgwentworth: what I just said :)
17:41:17 jgwentworth oh, that change already landed? I had thought it was stuck on the cells v1 job
17:41:30 jgwentworth if so, nevermind
17:41:35 superdan no
17:41:40 superdan it's unrelated to that change
17:42:16 jgwentworth oh. I thought when you mentioned failure to load keypairs it was in the context of the "load everything" patch
17:42:38 jgwentworth so weird that it's suddenly acting up if not for that
17:42:56 superdan we thought it was caused by that patch but it's not
17:43:06 superdan yeah I dunno why it just started coming up,
17:43:10 superdan but it's clearly racy so ..
17:43:16 jgwentworth yeah ++
17:44:55 superdan I'm going to put it underneath the bottom one though just so it will squash the issue and make it easier to land that set
17:47:01 superdan heh we actually have a test that ensures that refresh does this exactly
17:47:36 superdan but I don't think the cells code can do anything about this, so probably better to just alter the refresh() behavior
17:51:58 jgwentworth guh, I gotta pack
18:12:59 openstackgerrit Dan Smith proposed openstack/nova master: Make instance.refresh() avoid recursion better https://review.openstack.org/569461
18:13:00 openstackgerrit Dan Smith proposed openstack/nova master: Make instance able to lazy-load almost everything https://review.openstack.org/569267
18:13:01 openstackgerrit Dan Smith proposed openstack/nova master: Expose instance_get_all_uuids_by_host() from DB API and use it https://review.openstack.org/569268
18:13:18 superdan first one there ^ should fix the cellsv1 thing
18:18:10 mriedem superdan: ok i was wondering how your change would have caused that
18:18:18 openstackgerrit Merged openstack/nova master: Fix typo in NUMATopologyFilter docs https://review.openstack.org/543505
18:26:19 mriedem jgwentworth: superdan: this is why we're seeing the spike in cells v1 job failures https://review.openstack.org/#/c/563851/
18:26:21 mriedem the test is new
18:26:33 superdan ah
18:26:51 superdan how fail-y is it?
18:27:19 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ServerShowV254Test%5C%22%20AND%20message%3A%5C%22rebuild_instance%5C%22%20AND%20message%3A%5C%22OrphanedObjectError%5C%22%20AND%20tags%3A%5C%22screen-n-cell-child.txt%5C%22&from=7d
18:27:44 mriedem it's a new rebuild test + key_name change
18:27:51 mriedem that doesn't really need to live in tempest...
18:27:53 mriedem gmann: ^
18:28:47 superdan mriedem: that didn't load anything for me
18:29:06 mriedem yeah fixing the query
18:31:13 mriedem we seem to have lost multiline error logging, which is why that doesn't work
18:31:32 superdan is it 100% or no?
18:31:55 mriedem idk, but i'm seeing a lot of stuff failing with that
18:32:04 superdan okay
18:32:25 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ServerShowV254Test%5C%22%20AND%20message%3A%5C%22Error%20processing%20message%20locally%3A%20OrphanedObjectError%3A%20Cannot%20call%20obj_load_attr%20on%20orphaned%20Instance%20object%5C%22%20AND%20tags%3A%5C%22screen-n-cell-child.txt%5C%22&from=7d
18:32:37 mriedem https://bugs.launchpad.net/nova/+bug/1772088
18:32:38 openstack Launchpad bug 1772088 in OpenStack Compute (nova) "ServerShowV254Test.test_rebuild_server fails with OrphanedObjectError in cells v1" [High,In progress] - Assigned to Dan Smith (danms)
18:33:28 jgwentworth mriedem: ack, nice find
18:34:21 superdan 45 since yesterday so probably not 100%
18:34:23 superdan but clearly bad
18:34:24 mriedem fried_rice: happy with this now https://review.openstack.org/#/c/549300/
19:02:27 openstackgerrit Matthew Edmonds proposed openstack/nova master: Update PowerVM hypervisor docs https://review.openstack.org/569497
19:04:06 openstackgerrit Arvind Nadendla proposed openstack/nova master: WIP:Handle rebuild of instances with image traits https://review.openstack.org/569498
19:04:10 mriedem superdan: +2 on https://review.openstack.org/#/c/569461/ if you can find another core
19:05:38 superdan cool
19:07:57 fried_rice mriedem: ack, reviewing the bug fix that's standing in the way first...
19:36:38 openstackgerrit Eric Fried proposed openstack/nova master: Normalize inventory from update_provider_tree https://review.openstack.org/569132
19:36:39 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: get_inventory => update_provider_tree https://review.openstack.org/560444
19:36:40 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: Don't report DISK_GB if sharing https://review.openstack.org/560459
19:52:38 openstackgerrit Matthew Edmonds proposed openstack/nova master: Update PowerVM hypervisor docs https://review.openstack.org/569497
19:57:20 openstackgerrit Arvind Nadendla proposed openstack/nova master: WIP:Handle rebuild of instances with image traits https://review.openstack.org/569498
21:05:41 openstackgerrit Arvind Nadendla proposed openstack/nova master: WIP:Handle rebuild of instances with image traits https://review.openstack.org/569498
21:08:45 mriedem boo live migration job failed
21:12:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Update PowerVM hypervisor docs https://review.openstack.org/569497
21:20:24 fried_rice son of a
21:21:19 fried_rice superdan: mriedem: I noticed that patch seems to be parented pretty far back in history. Perhaps since we need to recheck it anyway, we could rebase it?
21:34:23 superdan fried_rice: eh?
21:34:35 fried_rice The cells recursion fix
21:34:42 superdan fried_rice: it's the base of a series
21:34:47 superdan it can't go any earlier
21:35:02 fried_rice I'm saying the series could be rebased on a more current version of master.
21:35:19 superdan that will only further delay it getting in
21:35:33 fried_rice superdan: how so? It needs to be rechecked anyway.
21:35:44 superdan does it?
21:35:50 fried_rice yeah, live-migration failed
21:35:57 superdan rebasing won't make it go any quicker, what's the point of doing that?
21:36:20 openstackgerrit Dan Smith proposed openstack/nova master: Make instance.refresh() avoid recursion better https://review.openstack.org/569461
21:36:21 openstackgerrit Dan Smith proposed openstack/nova master: Make instance able to lazy-load almost everything https://review.openstack.org/569267
21:36:22 openstackgerrit Dan Smith proposed openstack/nova master: Expose instance_get_all_uuids_by_host() from DB API and use it https://review.openstack.org/569268
21:36:33 fried_rice actually cause I want to start rebasing other failing patches on top of it so they can get through the gate sooner :)
21:36:43 superdan aight, well ^
21:36:48 fried_rice cool
21:38:12 openstackgerrit Eric Fried proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300
21:39:45 fried_rice edmondsw, esberglu: FYI, rebased this on top of the fix for the cellsv1 job so I could +W it now without babysitting.
21:40:16 esberglu fried_rice: sweet tx
21:41:36 openstackgerrit Eric Fried proposed openstack/nova master: ProviderTree.has_inventory_changed for new fields https://review.openstack.org/569228
21:41:37 openstackgerrit Eric Fried proposed openstack/nova master: Normalize inventory from update_provider_tree https://review.openstack.org/569132
21:41:38 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: get_inventory => update_provider_tree https://review.openstack.org/560444
21:41:39 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: Don't report DISK_GB if sharing https://review.openstack.org/560459
21:43:03 openstackgerrit Eric Fried proposed openstack/nova master: Granular requests to get_allocation_candidates https://review.openstack.org/515811
21:44:00 fried_rice superdan: FYI, them's the rebases in question ^^^^^^ (there may be more on the way).
21:44:31 fried_rice Cause I don't want to have to wait for yours to merge and then come back in a couple hours to recheck the world.
21:46:26 openstackgerrit Eric Fried proposed openstack/nova master: Update nova-status and docs for required placement 1.24 https://review.openstack.org/568923
21:47:16 openstackgerrit Eric Fried proposed openstack/nova master: Enforce placement minimum in nova.cmd.status https://review.openstack.org/568965
21:55:01 openstackgerrit Matthew Edmonds proposed openstack/nova master: Update PowerVM hypervisor docs https://review.openstack.org/569497
21:55:10 edmondsw fried_rice ^
22:00:23 openstackgerrit Eric Fried proposed openstack/nova master: Update PowerVM hypervisor docs https://review.openstack.org/569497
23:04:28 openstackgerrit Eric Fried proposed openstack/nova master: Placement: allow to set reserved value equal to total for inventory https://review.openstack.org/564838
23:04:35 openstackgerrit Tsuyoshi Nagata proposed openstack/nova master: nova improvement of maximum attach volumes more than 26 vols https://review.openstack.org/567472
23:05:47 openstackgerrit Eric Fried proposed openstack/nova master: Avoid unnecessary joins in HostManager._get_instances_by_host https://review.openstack.org/569218

Earlier   Later