Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-13
15:50:55 tasker the above is git master as of a few minutes ago.
15:51:24 tasker commit a4fc1bcd
15:52:07 tasker I have Mitaka in production where I noticed this problem and I see that function unchanged between Mitaka (commit 9825c80) and current master.
16:01:34 tasker in my case, the snapshot failed due to OverLimit returned by the cinderclient. The exeception bubbles up, and I'm still not seeing anywhere that unquiesces / thaws the instance after the exception.
16:05:03 mriedem ok i remember someone fixing the handling of the OverLimit from cinder, but would have to see if the unquiesce is ever run on failure
16:05:18 mriedem the overlimit used to cause a 500 response from the API
16:07:32 maciejjozefczyk mriedem: We talked during summit about bug in calculating used resources in resource_tracker. The bug is there: https://bugs.launchpad.net/nova/+bug/1729621
16:07:32 openstack Launchpad bug 1729621 in OpenStack Compute (nova) "Inconsistent value for vcpu_used" [Undecided,New]
16:07:41 tasker the traceback from this morning: http://paste.openstack.org/show/626180/
16:08:02 maciejjozefczyk I mentioned possible solutions (1. option imo seems to be the right one); could you take a look?
16:18:24 mriedem maciejjozefczyk: sure, thanks for pointing out the bu
16:18:25 mriedem *bug
16:20:47 mriedem maciejjozefczyk: nice recreate :)
16:28:18 openstackgerrit Merged openstack/nova master: Test allocation_candidates with only sharing RPs https://review.openstack.org/518382
16:29:44 tasker mriedem: I'm not seeing so. the only time I see `unquiesce` called is from compute/api.py#L2807 after the volumes finish snapping. but there is no exception handling between quiesce and unquiesce. if an exception is raised, it goes right past the unuqiesce on its way out the stack.
16:32:18 mriedem tasker: same here
16:32:26 mriedem tasker: want to report a bug?
16:34:14 openstackgerrit Merged openstack/nova master: Update contributor microversion doc for compute https://review.openstack.org/517626
16:39:38 efried mriedem Do you want me to do anything with https://review.openstack.org/#/c/519034/ ?
16:40:09 mriedem efried: not at this moment no
16:40:17 mriedem efried: i'd like jaypipes to also know about this and look at it
16:40:30 mriedem i'm playing catch up on internal stuff today
16:40:57 efried mriedem Okay. Should I bother to keep rechecking it? I'm convinced that it at least avoids the problem, though of course it's probably just masking the root cause.
16:42:05 openstackgerrit Merged openstack/nova master: placement: Contributor doc microversion checklist https://review.openstack.org/516783
16:42:41 mriedem efried: i'd like others more involved with placement code to look to see if they have any ideas on root cause
16:42:57 mriedem because i'm not sure why we'd be hitting this with a single scheduler CI job
16:43:07 efried rgr
16:44:06 mriedem johnthetubaguy: sdague: want to slam through this bit of spec accounting? https://review.openstack.org/#/c/516033/
16:44:44 mriedem stephenfin: you were +W on this before, just needed a rebase https://review.openstack.org/#/c/514782/
17:02:56 dtantsur folks, who's left the nova_cells meeting running? :)
17:03:40 efried mriedem dansmith Do you have the power to #endmeeting for dtantsur ?
17:04:23 dtantsur I suspect no
17:04:31 dtantsur somebody unrelated has started it, sorry for noise here
17:04:50 openstackgerrit Elod Illes proposed openstack/nova master: Transform instance-live_migration_pre notification https://review.openstack.org/482070
17:31:14 openstackgerrit Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/519165
17:38:58 openstackgerrit Elod Illes proposed openstack/nova master: Functional test: evacuate with no compute https://review.openstack.org/498482
17:42:56 mriedem sdague: i'm +2 on the keypair reset during rebuild change now https://review.openstack.org/#/c/379128/
17:45:38 tasker mriedem: https://bugs.launchpad.net/nova/+bug/1731986
17:45:38 openstack Launchpad bug 1731986 in OpenStack Compute (nova) "nova snapshot_volume_backed failure does not thaw filesystems" [Undecided,New]
17:46:11 tasker I'm in the proccess of writing and testing a patch to get my production clusters moving. I"ll hopefully have a review for the patch this afternoon.
17:46:35 mriedem tasker: cool, thanks
18:03:43 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/519127
18:06:31 sdague mriedem: +A
18:06:46 sdague took me a second to understand the or on the json schema, but I think I get it now
18:07:17 mriedem yeah the oneOf
18:07:28 mriedem we have that in the 'networks' entry in the server create schema too
18:19:44 mriedem hmm, why is it that we don't dump config options when n-api starts up?
18:20:04 mriedem i'm trying to fix the log formatting for placement-api in devstack, and the differences in the logs are a bit weird
18:20:41 mriedem oh i see why
18:21:04 mriedem https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/wsgi.py#L60
18:24:26 openstackgerrit Surya Seetharaman proposed openstack/nova master: instance_mapping/request_spec records exist even after the instances are archived https://review.openstack.org/515034
18:38:14 openstackgerrit Merged openstack/nova master: Fix missing versioned notification sample https://review.openstack.org/516582
19:49:36 xarlos Hi all. I have just configured a number of nova node in separate region configs.
19:50:48 xarlos However, when I run "openstack hypervisor list" I get 2 listed from one region in this region, and none over in another region. I'm a bit confused.
19:51:01 xarlos I think I failed to add the region statement when I registered them using nova-manage.
19:51:13 xarlos How can I clear this out and get to re-add them? :-/
20:07:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Log options at debug when starting API services under wsgi https://review.openstack.org/519462
20:12:24 openstackgerrit Eric M Gonzalez (tasker) proposed openstack/nova master: unquiesce instance on volume snapshot failure https://review.openstack.org/519464
20:12:53 mriedem tasker: we'll want a test for that
20:12:55 tasker hope I got that submitted all right.
20:12:56 mriedem when you get the time
20:13:06 tasker oh, of course you will. . )
20:13:10 tasker let me get some lunch.
20:24:40 mriedem tasker: comments inline when you get back
20:25:35 mriedem smcginnis: any idea how graceful cinder is about deleting a snapshot that is being created? concurrent requests i mean.
20:35:40 tasker mriedem: regarding your summary comment about splitting it up. after reviewing your comments I agree with you. how do you want me to proceed on that? resubmit with just the unqueisce and revisit the snapshot deletion later (after discussion about race) in another blueprint?
20:36:21 mriedem tasker: i don't think we need a blueprint, it's just another patch related to the same bug
20:36:35 mriedem tasker: but yeah split them up so quiesce is first and handled separately, we can backport that
20:36:45 mriedem then we'll see about the automatic cleanup of snapshots
20:37:20 tasker ok. I'll have that and a test (i hope) submitted in a little bit.
20:59:35 openstackgerrit Merged openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128
21:26:17 tasker this url https://docs.openstack.org/nova/latest/reference/unit_tests.html is no longer valid. does anyone know where it ended up?
21:31:45 mriedem tasker: https://docs.openstack.org/nova/latest/contributor/testing.html#unit-tests ?
21:33:26 mriedem that page linking to the hacking repo docs seems like the wrong thing to do...
21:37:55 mriedem it was probably supposed to link to this https://github.com/openstack/nova/blob/master/HACKING.rst#creating-unit-tests
21:40:22 tasker sorry -- got sidetracked because of /.
21:41:06 tasker the latter is where I ended up after some trolling.
21:52:14 mriedem i'll fix the link in our docs to point at nova/HACKING.rst, although that is pretty outdated
21:52:14 mriedem 1. use mock, not stubs or mox; 2. easiest to start with an existing test over the same function, copy it and tweak for what you need to test
21:52:14 tasker thanks for the suggestion. I'm doing 2 now and just saw 1 in one of the other tests.
21:52:15 mriedem there might be an existing test for the OverLimit failure, and your change makes it fail if yo'ure not mocking out the unquiesce_instance call you added
21:52:15 mriedem mtreinish: nova uses os-testr which uses stestr now right?
21:52:15 mriedem so nova saying it's using testr isn't really accurate anymore
21:53:22 mtreinish mriedem: actually nova is using stestr directly in most places now
21:53:45 mriedem ah yeah stestr run '{posargs}'
21:59:17 mtreinish mriedem: if you're updating the nova docs, the user manual for stestr is: http://stestr.readthedocs.io/en/latest/MANUAL.html
21:59:45 mriedem yup, just linking to the main index
22:04:29 efried patchbot seems to be down
22:15:27 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: fix link to creating unit tests in contributor guide https://review.openstack.org/519482
22:15:41 mriedem efried: looks like it's working for me
22:16:28 efried mriedem Okay. Maybe it just hates me. I just rebased this whole series https://review.openstack.org/#/c/516778/ and nothing popped up.
22:20:35 openstackgerrit Michael Still proposed openstack/nova master: Move remaining uses of parted to privsep. https://review.openstack.org/519483
22:20:35 openstackgerrit Michael Still proposed openstack/nova master: Convert users of tune2fs to privsep. https://review.openstack.org/519484
22:52:34 squid hi, i seem to sporadically have nova snapshots failing, with rbd as the backend, it throws the error "ERROR oslo_messaging.rpc.server ImageNotAuthorized: Not authorized for image fc378659-9d6f-41d6-8c7f-a7d08c2bdb54"
22:52:39 squid any idea what might be causing this?
22:54:38 calebb this is using the faster rbd backend snapshots as well
22:58:11 calebb sorry i'll put this in #openstack
22:59:24 tasker most places I see ImageNotAuthorized being thrown are a synonym to HTTPForbidden. check the OpenStack user that's initiating the call. Also check the glance / rbd authentication and make sure that glance has proper access to the requisite pools.
22:59:58 openstackgerrit Eric Fried proposed openstack/nova master: Test alloc_cands with non overlapping sharing RPs https://review.openstack.org/519380
23:00:37 efried gibi Hey, I noticed that one of the other tests related to bug 1730730 started passing when incorporated into Jay's series, so I rebased yours on the same.

Earlier   Later