| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-01 | |||
| 19:23:30 | efried | stand by. | |
| 19:24:19 | sean-k-mooney | i was going to say we shoudl navoe nova serial testing in the gate but it actuly just i nthe unit tests | |
| 19:25:01 | sean-k-mooney | getting spice testing should be pretty simple we jus tneed to pick one of our exisitg jobs to swap over ot it | |
| 19:25:12 | sean-k-mooney | im not sure if devstack deploy nova-serial | |
| 19:25:19 | sean-k-mooney | it proably does but i have never used it | |
| 19:25:27 | sean-k-mooney | melwitt: ^ | |
| 19:27:47 | efried | mriedem, melwitt: +A | |
| 19:27:54 | mriedem | hot | |
| 19:28:10 | sean-k-mooney | if we change greande-py3 to nova-serial-proxy and sawped nova-greade-multione to spice and kept the neturon-greade multinoe as the default novnc it should cover all three? | |
| 19:38:55 | efried | above my pay grade, sounds like a melwitt question. | |
| 19:42:13 | zigo | Heya ! Got my first VM running on Train with all the RC1 packaged and uploaded to Debian already ! :) | |
| 19:42:29 | zigo | It's never been that smooth ... :P | |
| 19:46:24 | mriedem | zigo: that's good news, thanks for sharing | |
| 19:46:32 | mriedem | we'll have an RC2, just fyi | |
| 19:46:53 | zigo | mriedem: For the moment, there's no "but x is broken" part, as much as I know... :P | |
| 19:47:32 | zigo | Still waiting for Glance RC1 though, it still not out. | |
| 19:47:53 | zigo | Looks like it Glance from Stein is enough. :) | |
| 20:23:22 | mnaser | hmm | |
| 20:23:29 | mnaser | i remember there was the rabbitmq api stuff we talked about a while ago | |
| 20:23:52 | mnaser | is it possible that at some point if nova heartbeats get messed up, all processes seem to be forever stuck (heartbeat misses accumulate) | |
| 20:24:58 | mnaser | >missed heartbeats from client, timeout: 60s | |
| 20:25:07 | mnaser | with some "client unexpectedly closed TCP connection" | |
| 20:25:26 | mnaser | and a whole bunch of: Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed | |
| 20:25:44 | mnaser | but also "Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer" (assuming this is when rabbitmq kills it cause it hasnt gotten heartbeat) | |
| 20:25:57 | mnaser | and for context this is a very busy api so its not sitting idle | |
| 20:26:13 | mriedem | are you thinking of this? https://review.opendev.org/#/c/662095/5/releasenotes/notes/eventlet-monkey-patch-5f734ef581aa550e.yaml | |
| 20:27:11 | mnaser | hmm, threads=1 in this deployment (osa with uwsgi) | |
| 20:27:26 | mnaser | and again this is a quite busy api endpoint so i dont imagine it sat idle for more than.. 5-10s at a time | |
| 20:28:16 | mriedem | maybe check this out https://review.opendev.org/#/c/663074/ | |
| 20:28:30 | mnaser | simple math shows 1000 requests per last 26 minutes so 38 requests per minute.. doubt thats enough to trigger something lik ethis | |
| 20:31:13 | mnaser | hmm | |
| 20:31:17 | mnaser | it'll be hard to run that | |
| 20:31:27 | mnaser | but im also a bit curious why its triggering here at all | |
| 20:36:48 | mnaser | also we run HEAD / every few seconds too | |
| 20:43:02 | mriedem | efried: explain to me why this wouldn't work in a test: | |
| 20:43:04 | mriedem | 1. do something, | |
| 20:43:07 | mriedem | 2. patch x | |
| 20:43:11 | mriedem | 3. do something else | |
| 20:43:19 | mriedem | 4. stop the patch on | |
| 20:43:21 | mriedem | x | |
| 20:43:28 | mriedem | 5. do another thing that hits the real thing that x was patching | |
| 20:43:42 | mriedem | 5 is blowing up b/c it doesn't appear that stop is stopping the patch | |
| 20:44:27 | mriedem | seems like it should work given https://docs.python.org/3.6/library/unittest.mock.html#patch-methods-start-and-stop | |
| 20:46:52 | mriedem | calling mock.patch.stopall() seems to do what 4 isn't | |
| 20:48:51 | mnaser | welp, it restarted happenign again not long after a service restart | |
| 20:50:42 | mriedem | maybe a bug in the mock library | |
| 20:52:31 | mriedem | anyway, i have a workaround, ignore me | |
| 20:58:13 | efried | mriedem: without seeing the code... | |
| 20:58:16 | efried | can I see the code? | |
| 20:58:29 | efried | Cause what you're describing seems like it ought to work | |
| 21:00:28 | efried | mriedem: a classic blunder is calling stop() on the *mock* (rather than the patcher) | |
| 21:00:41 | mriedem | yeah it wasn't that | |
| 21:00:50 | mriedem | p = mock.patch('x') | |
| 21:00:52 | mriedem | p.start() | |
| 21:00:55 | mriedem | ... | |
| 21:00:57 | mriedem | p.stop() | |
| 21:01:05 | mriedem | x() -> kablammo | |
| 21:01:22 | mriedem | err, p = mock.patch('x', side_effect=kablammo) | |
| 21:03:04 | mriedem | threads could be at play somehow, idk | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for build part of bug 1781286 https://review.opendev.org/685998 | |
| 21:03:40 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem) | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017 | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Selection.availability_zone field https://review.opendev.org/685807 | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047 | |
| 21:04:07 | mriedem | if you're a glutton for punishment, what i was doing was at this point https://review.opendev.org/#/c/686047/1/nova/tests/functional/regressions/test_bug_1781286.py@72 | |
| 21:04:12 | efried | In [8]: foo() | |
| 21:04:12 | efried | real | |
| 21:04:12 | efried | In [9]: p.start() | |
| 21:04:12 | efried | Out[9]: <MagicMock name='foo' id='139999791540112'> | |
| 21:04:12 | efried | In [10]: foo() | |
| 21:04:12 | mriedem | i changed that to be something like: | |
| 21:04:13 | efried | patched | |
| 21:04:13 | efried | In [11]: p.stop() | |
| 21:04:14 | efried | In [12]: foo() | |
| 21:04:14 | efried | real | |
| 21:04:29 | mriedem | self.agg_mock = mock.patch(...) | |
| 21:04:33 | mriedem | self.agg_mock.start() | |
| 21:04:52 | mriedem | and then later after fake_notifier.wait_for_versioned_notifications where i have mock.patch.stopall(), i had self.agg_mock.stop() | |
| 21:04:55 | mriedem | and that wasn't working | |
| 21:05:52 | efried | because you subsequently called get_by_host? | |
| 21:06:05 | efried | as opposed to build_and_run_instance? | |
| 21:06:22 | mriedem | the test goes on to call self._wait_for_state_change which hits the API which hit the thing i had previously mockd to fail yeah | |
| 21:06:51 | efried | cause if you had called build_and_run_instance again, it would have reinstated your mock. | |
| 21:07:08 | efried | but stop_all would have reverted your mock of build_and_run_instance | |
| 21:07:18 | mriedem | what probably happened is there were 2 mocks | |
| 21:07:25 | mriedem | and i only stopped one | |
| 21:07:35 | efried | mm, because two instances? | |
| 21:07:45 | mriedem | https://github.com/testing-cabal/mock/blob/master/mock/mock.py#L1533 | |
| 21:07:52 | mriedem | 2 calls to build_and_run_instance | |
| 21:07:54 | mriedem | 2 hosts | |
| 21:07:56 | mriedem | it's a reschedule test | |
| 21:07:59 | efried | that'd do it :) | |
| 21:08:10 | mriedem | yup, so stopall to the rescue | |
| 21:08:22 | efried | well, except that's going to stop... *all* of your mocks. | |
| 21:08:31 | efried | o | |
| 21:08:38 | efried | only the ones you started with start() | |
| 21:08:40 | efried | brittle | |
| 21:08:42 | mriedem | that's fine for this test; anyway, thanks for making me think through it - now i can explain why i'm using stopall when artom -1s my change | |
| 21:10:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047 | |
| 21:33:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during migrate reschedule https://review.opendev.org/686050 | |