Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-13
20:53:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505
21:08:28 openstackgerrit Merged openstack/nova master: Deprecate file injection https://review.openstack.org/522027
21:10:41 mriedem ooo eff yeah
21:12:05 mriedem yikes 500 from placement http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-sch.txt.gz#_Dec_13_17_07_40_977103
21:13:48 mriedem wth, tracing req-c9d0d841-a466-4b8a-aea1-aae993781b4f we get a NoValidHost,
21:13:52 mriedem http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-super-cond.txt.gz#_Dec_13_17_07_41_007395
21:14:13 mriedem but looking at that in the scheduler logs, we don't even hit the filters for that request
21:14:14 mriedem http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-sch.txt.gz#_Dec_13_17_07_40_529855
21:14:56 mriedem right around the time that request is starting to run in the scheduler, another one blows up with a 500 response from placement
21:15:00 mriedem but those should be separate threads yeah?
21:15:06 mriedem edleafe: ^ you see anything suspicious in here?
21:16:30 mriedem so uh
21:16:31 mriedem http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-placement-api.txt.gz#_Dec_13_17_07_40_968321
21:16:34 mriedem that doesn't look good
21:16:38 mriedem jay_tampa: ^
21:20:57 mriedem sdague: you wanna just push this through? https://review.openstack.org/#/c/519458/
21:21:04 mriedem it sucks not being able to e-r check placement logs
21:21:51 mriedem oh maybe that's not an issue anymore
21:22:25 sdague it shouldn't impact e-r
21:22:35 sdague oh, actually maybe it does
21:22:42 sdague otherwise you get DEBUG logs, which overload it
21:23:16 mriedem logstash appears to be finding these
21:23:23 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22DBDeadlock%5C%22%20AND%20tags%3A%5C%22screen-placement-api.txt%5C%22%20AND%20voting%3A1&from=7d
21:24:09 mriedem i think the logstash problem with placement logs is multiline
21:24:20 mriedem yeah, i can't do message:"DBDeadlock" AND message:"_trait_sync"
21:24:31 mriedem clarkb: can we get https://review.openstack.org/#/c/519458/ in so i can do multiline queries on placement logs?
21:25:03 mriedem or maybe ^ doesn't fix that either...
21:29:03 clarkb mriedem: the filename does have screen in it right?
21:29:09 clarkb mriedem: so thats a valid fix regardless?
21:29:30 clarkb mriedem: as for multiline queries I would guess that is more a failing of the logstash parsing rules particularly for multiline events
21:30:11 mriedem clarkb: i definitely don't get the nice log level filters on the placement logs today
21:30:26 clarkb mriedem: ya thats going to be the chagne you linked then
21:30:40 mriedem i'm unable to live without log level filters
21:38:14 edleafe mriedem: no idea what's causing that error
21:39:09 mriedem edleafe: the dbdeadlock error in placement is causing the 500 in the scheduler logs,
21:39:15 mriedem i just can't figure out why that also aborts the other request
21:39:19 mriedem which should be on a different thread
21:39:29 mriedem https://bugs.launchpad.net/nova/+bug/1738083
21:39:30 openstack Launchpad bug 1738083 in OpenStack Compute (nova) "DBDeadlock when when syncing traits in Placement during list_allocation_candidates" [High,Triaged]
21:39:30 edleafe mriedem: yeah, I meant the deadlock
21:39:35 mriedem traits sync
21:41:25 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643
21:41:26 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644
21:41:26 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646
21:41:27 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647
21:41:27 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: [WIP] Get resource provider by uuid or name https://review.openstack.org/527791
21:42:05 mriedem this reminds me,
21:42:14 mriedem we should run multiple scheduler workers (2) in our CI
21:42:18 mriedem because we said we can since pike
21:42:41 edleafe mriedem: since we're only expecting db_exc.DBDuplicateEntry, not a DBDeadlock
22:11:47 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova-status check for ironic flavor migration https://review.openstack.org/527541
22:26:25 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/527799
22:26:35 dansmith mriedem: this is what I'm thinking for the functional change ^
22:31:01 dansmith except with less obvious bugs
22:31:26 mriedem one question, but seems ok at first glance
23:06:53 openstackgerrit Dan Smith proposed openstack/nova master: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/527799
23:13:55 jaypipes mriedem: that's odd... looks like multiple threads are all trying to insert the standard traits at the same time.
23:42:12 openstackgerrit melanie witt proposed openstack/nova master: Optionalize instance_uuid in console_auth_token_get_valid() https://review.openstack.org/481700
23:42:13 openstackgerrit melanie witt proposed openstack/nova master: Add console connection object https://review.openstack.org/320063
23:42:13 openstackgerrit melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381
23:42:14 openstackgerrit melanie witt proposed openstack/nova master: Use ConsoleConnection object to generate authorizations https://review.openstack.org/325414
23:42:14 openstackgerrit melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990
23:46:31 melwitt grr looks like I did something wrong to set the tempest conf option
23:48:38 melwitt need dashes instead of underscores ... okay
#openstack-nova - 2017-12-14
00:00:41 openstackgerrit Dan Smith proposed openstack/nova master: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/527799
00:15:40 openstackgerrit melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990
00:29:50 openstackgerrit Merged openstack/nova master: Update Instance action's updated_at when action event updated. https://review.openstack.org/507473
00:59:03 openstackgerrit Merged openstack/nova master: Add instance action db and obj pagination support. https://review.openstack.org/526422
01:16:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Pass mountpoint to volume attachment_update https://review.openstack.org/527468
01:16:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Update and complete volume attachments during resize https://review.openstack.org/527228
01:16:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Pass bdms to versioned notifications during finish_revert_resize https://review.openstack.org/527446
01:16:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Pass mountpoint to volume attachment_create with connector https://review.openstack.org/527479
01:16:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unnecessary connector stash in attachment_update https://review.openstack.org/527484
01:17:39 openstackgerrit Merged openstack/nova master: Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker https://review.openstack.org/527564
01:41:00 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and changes-since for instance-actions https://review.openstack.org/326326
01:47:39 yikun ^ rebase 2.57 --> 2.58
01:59:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix missing marker functions https://review.openstack.org/514579
02:02:45 mriedem yikun: on it
02:03:56 yikun mriedem, :), thanks
02:05:37 mriedem yikun: you dropped something from the rest api history doc
02:07:24 yikun mr
02:07:39 yikun mriedem, oops, my bad, I will address soon
02:09:51 mriedem yikun: i can get it from gerrit
02:10:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Add pagination and changes-since for instance-actions https://review.openstack.org/326326
02:10:26 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and changes-since for instance-actions https://review.openstack.org/326326
02:14:34 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and changes-since for instance-actions https://review.openstack.org/326326
02:19:34 yikun mriedem, another typo fix, so, I update again...now I think it's a final one, ╮(╯▽╰)╭
02:20:13 Kevin_Zheng https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L391 are we ever going to set this to False?
02:24:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Retry _trait_sync on deadlock https://review.openstack.org/527836
02:28:02 mriedem Kevin_Zheng: we can, but it has to be on a microversion
02:28:21 Kevin_Zheng Yeah, thats what I thought
02:29:04 gmann i feel if we do then we can do all together for query things of all APIs
02:30:12 mriedem yikun: got it, thanks
02:30:27 Kevin_Zheng Yeah, so maybe we should add them to all APIs first :)
02:30:34 Kevin_Zheng still alot missing?
02:30:59 mriedem Kevin_Zheng: there is a blacklist of filters that we don't allow
02:31:11 Kevin_Zheng yeah, I know
02:31:35 Kevin_Zheng Just curious :)
02:31:39 mriedem https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L295
02:31:45 mriedem because those will 500

Earlier   Later