Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-01
19:42:29 zigo It's never been that smooth ... :P
19:46:24 mriedem zigo: that's good news, thanks for sharing
19:46:32 mriedem we'll have an RC2, just fyi
19:46:53 zigo mriedem: For the moment, there's no "but x is broken" part, as much as I know... :P
19:47:32 zigo Still waiting for Glance RC1 though, it still not out.
19:47:53 zigo Looks like it Glance from Stein is enough. :)
20:23:22 mnaser hmm
20:23:29 mnaser i remember there was the rabbitmq api stuff we talked about a while ago
20:23:52 mnaser is it possible that at some point if nova heartbeats get messed up, all processes seem to be forever stuck (heartbeat misses accumulate)
20:24:58 mnaser >missed heartbeats from client, timeout: 60s
20:25:07 mnaser with some "client unexpectedly closed TCP connection"
20:25:26 mnaser and a whole bunch of: Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed
20:25:44 mnaser but also "Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer" (assuming this is when rabbitmq kills it cause it hasnt gotten heartbeat)
20:25:57 mnaser and for context this is a very busy api so its not sitting idle
20:26:13 mriedem are you thinking of this? https://review.opendev.org/#/c/662095/5/releasenotes/notes/eventlet-monkey-patch-5f734ef581aa550e.yaml
20:27:11 mnaser hmm, threads=1 in this deployment (osa with uwsgi)
20:27:26 mnaser and again this is a quite busy api endpoint so i dont imagine it sat idle for more than.. 5-10s at a time
20:28:16 mriedem maybe check this out https://review.opendev.org/#/c/663074/
20:28:30 mnaser simple math shows 1000 requests per last 26 minutes so 38 requests per minute.. doubt thats enough to trigger something lik ethis
20:31:13 mnaser hmm
20:31:17 mnaser it'll be hard to run that
20:31:27 mnaser but im also a bit curious why its triggering here at all
20:36:48 mnaser also we run HEAD / every few seconds too
20:43:02 mriedem efried: explain to me why this wouldn't work in a test:
20:43:04 mriedem 1. do something,
20:43:07 mriedem 2. patch x
20:43:11 mriedem 3. do something else
20:43:19 mriedem 4. stop the patch on
20:43:21 mriedem x
20:43:28 mriedem 5. do another thing that hits the real thing that x was patching
20:43:42 mriedem 5 is blowing up b/c it doesn't appear that stop is stopping the patch
20:44:27 mriedem seems like it should work given https://docs.python.org/3.6/library/unittest.mock.html#patch-methods-start-and-stop
20:46:52 mriedem calling mock.patch.stopall() seems to do what 4 isn't
20:48:51 mnaser welp, it restarted happenign again not long after a service restart
20:50:42 mriedem maybe a bug in the mock library
20:52:31 mriedem anyway, i have a workaround, ignore me
20:58:13 efried mriedem: without seeing the code...
20:58:16 efried can I see the code?
20:58:29 efried Cause what you're describing seems like it ought to work
21:00:28 efried mriedem: a classic blunder is calling stop() on the *mock* (rather than the patcher)
21:00:41 mriedem yeah it wasn't that
21:00:50 mriedem p = mock.patch('x')
21:00:52 mriedem p.start()
21:00:55 mriedem ...
21:00:57 mriedem p.stop()
21:01:05 mriedem x() -> kablammo
21:01:22 mriedem err, p = mock.patch('x', side_effect=kablammo)
21:03:04 mriedem threads could be at play somehow, idk
21:03:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional regression test for build part of bug 1781286 https://review.opendev.org/685998
21:03:40 openstack bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem)
21:03:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017
21:03:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Selection.availability_zone field https://review.opendev.org/685807
21:03:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047
21:04:07 mriedem if you're a glutton for punishment, what i was doing was at this point https://review.opendev.org/#/c/686047/1/nova/tests/functional/regressions/test_bug_1781286.py@72
21:04:12 efried In [8]: foo()
21:04:12 efried real
21:04:12 efried In [9]: p.start()
21:04:12 efried Out[9]: <MagicMock name='foo' id='139999791540112'>
21:04:12 efried In [10]: foo()
21:04:12 mriedem i changed that to be something like:
21:04:13 efried patched
21:04:13 efried In [11]: p.stop()
21:04:14 efried In [12]: foo()
21:04:14 efried real
21:04:29 mriedem self.agg_mock = mock.patch(...)
21:04:33 mriedem self.agg_mock.start()
21:04:52 mriedem and then later after fake_notifier.wait_for_versioned_notifications where i have mock.patch.stopall(), i had self.agg_mock.stop()
21:04:55 mriedem and that wasn't working
21:05:52 efried because you subsequently called get_by_host?
21:06:05 efried as opposed to build_and_run_instance?
21:06:22 mriedem the test goes on to call self._wait_for_state_change which hits the API which hit the thing i had previously mockd to fail yeah
21:06:51 efried cause if you had called build_and_run_instance again, it would have reinstated your mock.
21:07:08 efried but stop_all would have reverted your mock of build_and_run_instance
21:07:18 mriedem what probably happened is there were 2 mocks
21:07:25 mriedem and i only stopped one
21:07:35 efried mm, because two instances?
21:07:45 mriedem https://github.com/testing-cabal/mock/blob/master/mock/mock.py#L1533
21:07:52 mriedem 2 calls to build_and_run_instance
21:07:54 mriedem 2 hosts
21:07:56 mriedem it's a reschedule test
21:07:59 efried that'd do it :)
21:08:10 mriedem yup, so stopall to the rescue
21:08:22 efried well, except that's going to stop... *all* of your mocks.
21:08:31 efried o
21:08:38 efried only the ones you started with start()
21:08:40 efried brittle
21:08:42 mriedem that's fine for this test; anyway, thanks for making me think through it - now i can explain why i'm using stopall when artom -1s my change
21:10:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047
21:33:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during migrate reschedule https://review.opendev.org/686050
21:36:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during migrate reschedule https://review.opendev.org/686050
21:46:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Update cells v2 up-call caveats doc https://review.opendev.org/686053
21:55:01 openstackgerrit Ghanshyam Mann proposed openstack/nova-specs master: Re-propose policy-defaults-refresh spec for Ussuri https://review.opendev.org/686058
22:08:19 openstackgerrit melanie witt proposed openstack/nova stable/stein: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686063
22:14:03 efried melwitt: did you miss the train cherry-pick?
22:14:19 efried or did I?
22:15:06 melwitt efried: no, I missed it. sorry, I forgot the branch was cut
22:15:11 melwitt will fix
22:15:40 efried at least it's clean :P
22:16:57 openstackgerrit melanie witt proposed openstack/nova stable/train: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686066
22:17:50 openstackgerrit melanie witt proposed openstack/nova stable/stein: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686063

Earlier   Later