Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-21
19:28:52 melwitt I feel like I'm seeing tempest jobs time out fairly often lately too
19:29:36 openstackgerrit Chris Friesen proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208
19:30:50 cfriesen mriedem: dansmith: is this ^ the right way to handle the issue? I need to deal with the testcases still.
19:34:55 sdague melwitt: yeh, I noticed that as well. I haven't gone and looked yet
19:35:03 sdague melwitt: you have a link to some?
19:35:37 melwitt sdague: I think this is one, the most recent I've seen http://logs.openstack.org/75/419975/20/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/17f4349/
19:36:54 sdague melwitt: also, if you are inclined, bringing back request logging under uwsgi - https://review.openstack.org/#/c/485602
19:38:11 melwitt sdague: this one says "Request timed out" http://logs.openstack.org/88/485088/6/gate/gate-tempest-dsvm-cells-ubuntu-xenial/17bb8bf/logs/testr_results.html.gz
19:38:30 melwitt k, looking
19:53:34 cdent figleaf: that is what it is currently doing
19:54:01 figleaf cdent: ah, I thought it was taking shared resources into account, too
19:54:16 cdent it was supposed to be but that was never actually implemented
19:54:43 figleaf cdent: then if it isn't needed, I would prefer to leave it as is
19:54:52 figleaf since we're using allocation_candidates
19:54:59 sdague melwitt: yeh, trying to figure out with infra if this is persistent issue or not
19:55:11 cdent figleaf: yeah, that’s kinda where I landed too
19:55:49 figleaf cdent: maybe later if it is needed, we could add an additional query param to indicate "include shared"
19:58:10 leakypipes mriedem, cdent, melwitt, superdan, figleaf: I swear I need to hire someone to just keep rechecking my patches through the gate :(
19:58:16 leakypipes http://logs.openstack.org/66/483566/8/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/eda58dd/console.html
19:58:21 leakypipes latest random failure...
19:58:28 melwitt you and me both
19:58:33 melwitt recheck bots
19:58:42 figleaf leakypipes: I thought you had a team of lackeys for such mundane tasks
19:59:01 leakypipes honestly, our gate should see 1 test failure like that out of 1319 tempest tests and say to itself "fuck it, close enough. merge."
19:59:13 melwitt heh
20:00:09 openstackgerrit Chris Dent proposed openstack/nova master: retry on authentication failure in api_client https://review.openstack.org/486190
20:00:27 cdent mriedem: you want +w that ^ again, had a pep8 failure
20:02:21 openstackgerrit Merged openstack/nova master: Change default policy to view quota details https://review.openstack.org/386008
20:02:42 cdent leakypipes: that auth retry thing _may_ help the gate
20:03:59 leakypipes cdent: +Wallaby'd
20:04:20 melwitt whoa something merged, amaze
20:09:34 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - add alternate hosts https://review.openstack.org/486215
20:09:47 figleaf leakypipes: ^^ very rough for now. Just wanted to get feedback on direction
20:10:45 figleaf leakypipes: planning on a subsequent patch to return allocation_candidates along with hosts, and then return both the alternates with the candidates
20:11:22 figleaf return both to the compute build_and_run_instances() call
20:16:49 openstackgerrit Merged openstack/nova master: doc: Switch to openstackdocstheme https://review.openstack.org/477751
20:17:58 melwitt sdague: what's the idea here? if you get an exception while logging, log again? https://review.openstack.org/#/c/485602/5/nova/api/openstack/requestlog.py@92
20:19:38 sdague melwitt: because of the way the paste pipelines work, if we get an unexpected exception it's going to end up going through that layer
20:19:49 sdague and handled at the fault layer above it
20:20:16 sdague if the log message was not emitted in the except block, you'd never have a record of the request
20:20:23 sdague you'd just have the fault error
20:20:34 melwitt sdague: ah, cool. thanks
20:21:06 mriedem sdague: in case you didn't know, the neutron linuxbridge job is the one that has been timing out all week
20:21:12 mriedem kevinbenton: ^ know what's going on there?
20:21:22 sdague mriedem: it's also set at 130minutes
20:21:26 sdague which is shorter than the rest
20:21:57 sdague I've definitely seen passing neutron jobs at 1:45 and 2:00
20:22:06 sdague which means everything is just long
20:22:27 sdague so a 130 is going to fail just because of statistics on slow node events
20:22:32 melwitt huh, interesting. I would have assumed they all had the same timeout
20:22:55 sdague I think at some point the linux bridge job was running less stuff
20:23:06 sdague but who knows :)
20:23:43 melwitt heh
20:27:29 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Expect id and disabled_reason in GET /os-services response https://review.openstack.org/485409
20:27:29 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: WIP: Microversion 2.53 - services and hypervisors using UUIDs https://review.openstack.org/485435
20:52:07 cfriesen has anyone considered a "run this command in parallel on all cells and aggregate the responses" helper function? there seem to be quite a few "for cell in cells:" loops in the code.
20:53:40 cfriesen separate issue...nova/compute/api.py has a CELLS variable. How does this get refreshed to pick up new cells created after the variable was initially set?
20:53:58 melwitt cfriesen: yes https://github.com/openstack/nova/blob/master/nova/context.py#L531
20:54:13 openstackgerrit Ildiko Vancsa proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285
20:55:28 melwitt currently you have to restart the services that are caching them. refreshing them is a TODO https://github.com/openstack/nova/blob/master/nova/context.py#L50
20:55:31 cfriesen melwitt: nice work
20:57:16 melwitt gdi and my patch just hit the linuxbridge job timeout
20:57:17 cfriesen if we don't fix that refresh thing for Pike we should document it...I don't remember seeing any mention of that in the discussion about adding new cells
20:57:34 cfriesen make that documentation, not discussion
20:58:02 melwitt yeah, true. I'll put it on the TODOs etherpad
20:58:03 kevinbenton sdague: where can we propose to increase that timeout?
20:58:14 kevinbenton (for linux bridge job)
20:59:28 melwitt cfriesen: the todo etherpad is here if you think of other stuff that would help (documentation or otherwise) https://etherpad.openstack.org/p/nova-pike-cells-v2-todos
21:01:16 cfriesen melwitt: good to know
21:09:56 cfriesen melwitt: was there discusson on how to handle the refresh? for what it's worth, I think a trigger would be more deterministic
21:11:00 melwitt cfriesen: not really yet. I was thinking we'd do it upon a SIGHUP or something, like we do for config file refreshes
21:14:15 openstackgerrit Chris Friesen proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208
21:18:13 openstackgerrit Merged openstack/nova master: Use plain routes list for versions instead of stevedore https://review.openstack.org/485011
21:19:10 openstackgerrit Merged openstack/nova master: Remove the unittest for plugin framework https://review.openstack.org/485060
21:20:39 cfriesen melwitt: what about making it an RPC call and hooking it into the "nova-manage cell_v2 create_cell" command so that all the necessary services would get kicked when we add a cell? Otherwise it's a requirement on the operator to arrange for a SIGHUP to all the services which may be on separate physical servers.
21:26:52 melwitt cfriesen: nova-manage doesn't work that way, it just does direct DB access stuff, no RPC
21:27:16 melwitt but yeah, I agree SIGHUP isn't ideal. probably we can come up with a better idea
22:02:03 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use wsgi_intercept in PlacementFixture https://review.openstack.org/486237
22:34:44 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - add alternate hosts https://review.openstack.org/486215
22:34:45 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - return alternates along with their allocations https://review.openstack.org/486253
22:34:59 figleaf leakypipes: ^^ and with that I'm heading out
23:22:23 openstackgerrit Merged openstack/nova master: placement: proper JOIN order for shared resources https://review.openstack.org/485088
23:23:36 openstackgerrit Merged openstack/nova master: Removed unused 'wrap' property https://review.openstack.org/481465
23:55:50 openstackgerrit Chris Friesen proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208
#openstack-nova - 2017-07-22
00:30:18 openstackgerrit Merged openstack/nova master: [placement] Add api-ref for traits https://review.openstack.org/474186
01:07:39 openstackgerrit Chris Friesen proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208
01:16:55 openstackgerrit Merged openstack/nova master: retry on authentication failure in api_client https://review.openstack.org/486190
02:03:47 openstackgerrit Merged openstack/nova master: Remove 'reserved' count from used limits https://review.openstack.org/446242
02:39:55 openstackgerrit Moshe Levi proposed openstack/nova master: hardware offload support for openvswitch https://review.openstack.org/398265
08:07:38 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955
09:48:42 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use wsgi_intercept in PlacementFixture https://review.openstack.org/486237
09:58:37 openstackgerrit Chris Dent proposed openstack/nova master: [placement] gabbi tests for shared custom resource class https://review.openstack.org/485209
10:02:13 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use wsgi_intercept in PlacementFixture https://review.openstack.org/486237
10:46:06 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964
11:33:18 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host https://review.openstack.org/414926
11:33:28 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host https://review.openstack.org/414926
11:39:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042
11:52:36 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091
12:15:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Dump versioned notifications when test_create_delete_server https://review.openstack.org/486301
12:33:28 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: WIP: Microversion 2.53 - services and hypervisors using UUIDs https://review.openstack.org/485435

Earlier   Later