Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-13
15:22:52 efried gibi There's also one more bug-provin' patch for allocation candidates: https://review.openstack.org/#/c/518633/
15:23:16 efried gibi ...and the one you looked at earlier which I updated per your comments: https://review.openstack.org/#/c/518982/
15:27:00 gibi efried: yeah, I opened https://review.openstack.org/#/c/518633/ before but I think I need a fresh mind to walk throught he complexity there, so maybe tomorrow :)
15:27:15 efried okay
15:27:26 gibi I looking that the test helper patch now
15:32:29 gibi that looks good too
15:34:45 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test alloc_cands with non overlapping sharing RPs https://review.openstack.org/519380
15:35:05 gibi I think I managed to make it even stranger ^^
15:35:22 gibi I managed to get the same allocation candidate from placement twice
15:44:00 tasker when making a snapshot of a volume-backed instance, the nova driver will queisce the instance's volumes prior to the snapshot. it looks like if the instance fails a snapshot, the exception isn't caught and the instance and its volumes are never thawed: master/nova/compute/api.py#L2738
15:45:00 tasker am I missing something? I'm still trying to trace through the code to see if the volumes are thawed after and exception. I can't find anything in launchpad regarding this either (could be that I'm not using the right search terms).
15:48:54 mriedem tasker: which release?
15:50:46 openstackgerrit Elod Illes proposed openstack/nova master: factor out compute service start in ServerMovingTest https://review.openstack.org/503037
15:50:46 openstackgerrit Elod Illes proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase https://review.openstack.org/499539
15:50:47 openstackgerrit Elod Illes proposed openstack/nova master: Change live_migrate tests to use fakedriver https://review.openstack.org/505202
15:50:48 openstackgerrit Elod Illes proposed openstack/nova master: Use FakeLiveMigrateDriver in notification test https://review.openstack.org/519405
15:50:55 tasker the above is git master as of a few minutes ago.
15:51:24 tasker commit a4fc1bcd
15:52:07 tasker I have Mitaka in production where I noticed this problem and I see that function unchanged between Mitaka (commit 9825c80) and current master.
16:01:34 tasker in my case, the snapshot failed due to OverLimit returned by the cinderclient. The exeception bubbles up, and I'm still not seeing anywhere that unquiesces / thaws the instance after the exception.
16:05:03 mriedem ok i remember someone fixing the handling of the OverLimit from cinder, but would have to see if the unquiesce is ever run on failure
16:05:18 mriedem the overlimit used to cause a 500 response from the API
16:07:32 openstack Launchpad bug 1729621 in OpenStack Compute (nova) "Inconsistent value for vcpu_used" [Undecided,New]
16:07:32 maciejjozefczyk mriedem: We talked during summit about bug in calculating used resources in resource_tracker. The bug is there: https://bugs.launchpad.net/nova/+bug/1729621
16:07:41 tasker the traceback from this morning: http://paste.openstack.org/show/626180/
16:08:02 maciejjozefczyk I mentioned possible solutions (1. option imo seems to be the right one); could you take a look?
16:18:24 mriedem maciejjozefczyk: sure, thanks for pointing out the bu
16:18:25 mriedem *bug
16:20:47 mriedem maciejjozefczyk: nice recreate :)
16:28:18 openstackgerrit Merged openstack/nova master: Test allocation_candidates with only sharing RPs https://review.openstack.org/518382
16:29:44 tasker mriedem: I'm not seeing so. the only time I see `unquiesce` called is from compute/api.py#L2807 after the volumes finish snapping. but there is no exception handling between quiesce and unquiesce. if an exception is raised, it goes right past the unuqiesce on its way out the stack.
16:32:18 mriedem tasker: same here
16:32:26 mriedem tasker: want to report a bug?
16:34:14 openstackgerrit Merged openstack/nova master: Update contributor microversion doc for compute https://review.openstack.org/517626
16:39:38 efried mriedem Do you want me to do anything with https://review.openstack.org/#/c/519034/ ?
16:40:09 mriedem efried: not at this moment no
16:40:17 mriedem efried: i'd like jaypipes to also know about this and look at it
16:40:30 mriedem i'm playing catch up on internal stuff today
16:40:57 efried mriedem Okay. Should I bother to keep rechecking it? I'm convinced that it at least avoids the problem, though of course it's probably just masking the root cause.
16:42:05 openstackgerrit Merged openstack/nova master: placement: Contributor doc microversion checklist https://review.openstack.org/516783
16:42:41 mriedem efried: i'd like others more involved with placement code to look to see if they have any ideas on root cause
16:42:57 mriedem because i'm not sure why we'd be hitting this with a single scheduler CI job
16:43:07 efried rgr
16:44:06 mriedem johnthetubaguy: sdague: want to slam through this bit of spec accounting? https://review.openstack.org/#/c/516033/
16:44:44 mriedem stephenfin: you were +W on this before, just needed a rebase https://review.openstack.org/#/c/514782/
17:02:56 dtantsur folks, who's left the nova_cells meeting running? :)
17:03:40 efried mriedem dansmith Do you have the power to #endmeeting for dtantsur ?
17:04:23 dtantsur I suspect no
17:04:31 dtantsur somebody unrelated has started it, sorry for noise here
17:04:50 openstackgerrit Elod Illes proposed openstack/nova master: Transform instance-live_migration_pre notification https://review.openstack.org/482070
17:31:14 openstackgerrit Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/519165
17:38:58 openstackgerrit Elod Illes proposed openstack/nova master: Functional test: evacuate with no compute https://review.openstack.org/498482
17:42:56 mriedem sdague: i'm +2 on the keypair reset during rebuild change now https://review.openstack.org/#/c/379128/
17:45:38 openstack Launchpad bug 1731986 in OpenStack Compute (nova) "nova snapshot_volume_backed failure does not thaw filesystems" [Undecided,New]
17:45:38 tasker mriedem: https://bugs.launchpad.net/nova/+bug/1731986
17:46:11 tasker I'm in the proccess of writing and testing a patch to get my production clusters moving. I"ll hopefully have a review for the patch this afternoon.
17:46:35 mriedem tasker: cool, thanks
18:03:43 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/519127
18:06:31 sdague mriedem: +A
18:06:46 sdague took me a second to understand the or on the json schema, but I think I get it now
18:07:17 mriedem yeah the oneOf
18:07:28 mriedem we have that in the 'networks' entry in the server create schema too
18:19:44 mriedem hmm, why is it that we don't dump config options when n-api starts up?
18:20:04 mriedem i'm trying to fix the log formatting for placement-api in devstack, and the differences in the logs are a bit weird
18:20:41 mriedem oh i see why
18:21:04 mriedem https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/wsgi.py#L60
18:24:26 openstackgerrit Surya Seetharaman proposed openstack/nova master: instance_mapping/request_spec records exist even after the instances are archived https://review.openstack.org/515034
18:38:14 openstackgerrit Merged openstack/nova master: Fix missing versioned notification sample https://review.openstack.org/516582
19:49:36 xarlos Hi all. I have just configured a number of nova node in separate region configs.
19:50:48 xarlos However, when I run "openstack hypervisor list" I get 2 listed from one region in this region, and none over in another region. I'm a bit confused.
19:51:01 xarlos I think I failed to add the region statement when I registered them using nova-manage.
19:51:13 xarlos How can I clear this out and get to re-add them? :-/
20:07:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Log options at debug when starting API services under wsgi https://review.openstack.org/519462
20:12:24 openstackgerrit Eric M Gonzalez (tasker) proposed openstack/nova master: unquiesce instance on volume snapshot failure https://review.openstack.org/519464
20:12:53 mriedem tasker: we'll want a test for that
20:12:55 tasker hope I got that submitted all right.
20:12:56 mriedem when you get the time
20:13:06 tasker oh, of course you will. . )
20:13:10 tasker let me get some lunch.
20:24:40 mriedem tasker: comments inline when you get back
20:25:35 mriedem smcginnis: any idea how graceful cinder is about deleting a snapshot that is being created? concurrent requests i mean.
20:35:40 tasker mriedem: regarding your summary comment about splitting it up. after reviewing your comments I agree with you. how do you want me to proceed on that? resubmit with just the unqueisce and revisit the snapshot deletion later (after discussion about race) in another blueprint?
20:36:21 mriedem tasker: i don't think we need a blueprint, it's just another patch related to the same bug
20:36:35 mriedem tasker: but yeah split them up so quiesce is first and handled separately, we can backport that
20:36:45 mriedem then we'll see about the automatic cleanup of snapshots
20:37:20 tasker ok. I'll have that and a test (i hope) submitted in a little bit.
20:59:35 openstackgerrit Merged openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128
21:26:17 tasker this url https://docs.openstack.org/nova/latest/reference/unit_tests.html is no longer valid. does anyone know where it ended up?
21:31:45 mriedem tasker: https://docs.openstack.org/nova/latest/contributor/testing.html#unit-tests ?
21:33:26 mriedem that page linking to the hacking repo docs seems like the wrong thing to do...
21:37:55 mriedem it was probably supposed to link to this https://github.com/openstack/nova/blob/master/HACKING.rst#creating-unit-tests
21:40:22 tasker sorry -- got sidetracked because of /.
21:41:06 tasker the latter is where I ended up after some trolling.
21:52:14 tasker thanks for the suggestion. I'm doing 2 now and just saw 1 in one of the other tests.
21:52:14 mriedem 1. use mock, not stubs or mox; 2. easiest to start with an existing test over the same function, copy it and tweak for what you need to test
21:52:14 mriedem i'll fix the link in our docs to point at nova/HACKING.rst, although that is pretty outdated
21:52:15 mriedem so nova saying it's using testr isn't really accurate anymore
21:52:15 mriedem mtreinish: nova uses os-testr which uses stestr now right?
21:52:15 mriedem there might be an existing test for the OverLimit failure, and your change makes it fail if yo'ure not mocking out the unquiesce_instance call you added
21:53:22 mtreinish mriedem: actually nova is using stestr directly in most places now

Earlier   Later