| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-01 | |||
| 18:57:52 | sean-k-mooney | yes hence my comment above | |
| 18:59:00 | sean-k-mooney | when i see patches merge with TODO comments from jaypipes i assume that was not a recent TODO and there for form nova | |
| 19:00:57 | sean-k-mooney | the bit that worres my is fi the share with nova flag is set | |
| 19:02:15 | sean-k-mooney | the pass the full provider tree for the nova compute node RP to there update_provider_tree function | |
| 19:03:46 | sean-k-mooney | they are also modifing trait possible on the root RP | |
| 19:17:42 | mriedem | sean-k-mooney: bw providers are nested under the compute node rp | |
| 19:18:48 | mriedem | the case that sean-k-mooney mentioned about nova deleting the compute node and root resource provider when you delete a compute service, i can definitely see that screwing with zun if they assume the same resource provider with the same uuid is going to exist, i don't know how they link that up, maybe via hostname | |
| 19:19:03 | mriedem | we have our own problems managing that stuff, i.e. the orphaned allocations/providers bug we have right now, | |
| 19:19:11 | mriedem | let alone worrying about how we're going to be breaking external things | |
| 19:19:21 | efried | mriedem: Did you have something to do with https://review.opendev.org/#/c/684326/ not being combined with its (current) successor https://review.opendev.org/#/c/685937/ ? (USE_NEUTRON refactoring in tests) | |
| 19:19:37 | sean-k-mooney | ya so depending on the valude of "share with nova" they either create there own root RP or share nova | |
| 19:20:13 | sean-k-mooney | and they use the hostname to look up the one created by nova if "shareing" is enabled | |
| 19:20:14 | mriedem | efried: you can see my comments on PS1, i don't know why it changed like it did, i haven't looked yet, but it definitely isn't what i was expecting | |
| 19:20:29 | mriedem | efried: he had https://review.opendev.org/#/c/684326/1 | |
| 19:20:44 | mriedem | and then in the next patch in the series reverted a bunch of what he added with https://review.opendev.org/#/c/684327/ | |
| 19:20:45 | efried | mriedem: yeah, the earlier comments have been addressed to squash together a now-abandoned change | |
| 19:20:48 | mriedem | i said just squash those | |
| 19:21:02 | mriedem | sure, he did but then there were other things tacked on that i wasn't expecting | |
| 19:21:05 | efried | I'm just wondering why there's still two patches, one that sets False and the other that removes True | |
| 19:21:09 | mriedem | and haven't taken the time to grok the diff | |
| 19:21:29 | mriedem | iow, i said "just do x" and he did x + 4 | |
| 19:22:43 | mriedem | btw, unrelated to this, but we have another train rc2 fix on master with a +2 and ack from the bug reporter that it fixes the regression https://review.opendev.org/#/c/685194/ | |
| 19:23:28 | efried | o, I was reviewing that and then... didn't finish for some reason. | |
| 19:23:30 | efried | stand by. | |
| 19:24:19 | sean-k-mooney | i was going to say we shoudl navoe nova serial testing in the gate but it actuly just i nthe unit tests | |
| 19:25:01 | sean-k-mooney | getting spice testing should be pretty simple we jus tneed to pick one of our exisitg jobs to swap over ot it | |
| 19:25:12 | sean-k-mooney | im not sure if devstack deploy nova-serial | |
| 19:25:19 | sean-k-mooney | it proably does but i have never used it | |
| 19:25:27 | sean-k-mooney | melwitt: ^ | |
| 19:27:47 | efried | mriedem, melwitt: +A | |
| 19:27:54 | mriedem | hot | |
| 19:28:10 | sean-k-mooney | if we change greande-py3 to nova-serial-proxy and sawped nova-greade-multione to spice and kept the neturon-greade multinoe as the default novnc it should cover all three? | |
| 19:38:55 | efried | above my pay grade, sounds like a melwitt question. | |
| 19:42:13 | zigo | Heya ! Got my first VM running on Train with all the RC1 packaged and uploaded to Debian already ! :) | |
| 19:42:29 | zigo | It's never been that smooth ... :P | |
| 19:46:24 | mriedem | zigo: that's good news, thanks for sharing | |
| 19:46:32 | mriedem | we'll have an RC2, just fyi | |
| 19:46:53 | zigo | mriedem: For the moment, there's no "but x is broken" part, as much as I know... :P | |
| 19:47:32 | zigo | Still waiting for Glance RC1 though, it still not out. | |
| 19:47:53 | zigo | Looks like it Glance from Stein is enough. :) | |
| 20:23:22 | mnaser | hmm | |
| 20:23:29 | mnaser | i remember there was the rabbitmq api stuff we talked about a while ago | |
| 20:23:52 | mnaser | is it possible that at some point if nova heartbeats get messed up, all processes seem to be forever stuck (heartbeat misses accumulate) | |
| 20:24:58 | mnaser | >missed heartbeats from client, timeout: 60s | |
| 20:25:07 | mnaser | with some "client unexpectedly closed TCP connection" | |
| 20:25:26 | mnaser | and a whole bunch of: Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed | |
| 20:25:44 | mnaser | but also "Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer" (assuming this is when rabbitmq kills it cause it hasnt gotten heartbeat) | |
| 20:25:57 | mnaser | and for context this is a very busy api so its not sitting idle | |
| 20:26:13 | mriedem | are you thinking of this? https://review.opendev.org/#/c/662095/5/releasenotes/notes/eventlet-monkey-patch-5f734ef581aa550e.yaml | |
| 20:27:11 | mnaser | hmm, threads=1 in this deployment (osa with uwsgi) | |
| 20:27:26 | mnaser | and again this is a quite busy api endpoint so i dont imagine it sat idle for more than.. 5-10s at a time | |
| 20:28:16 | mriedem | maybe check this out https://review.opendev.org/#/c/663074/ | |
| 20:28:30 | mnaser | simple math shows 1000 requests per last 26 minutes so 38 requests per minute.. doubt thats enough to trigger something lik ethis | |
| 20:31:13 | mnaser | hmm | |
| 20:31:17 | mnaser | it'll be hard to run that | |
| 20:31:27 | mnaser | but im also a bit curious why its triggering here at all | |
| 20:36:48 | mnaser | also we run HEAD / every few seconds too | |
| 20:43:02 | mriedem | efried: explain to me why this wouldn't work in a test: | |
| 20:43:04 | mriedem | 1. do something, | |
| 20:43:07 | mriedem | 2. patch x | |
| 20:43:11 | mriedem | 3. do something else | |
| 20:43:19 | mriedem | 4. stop the patch on | |
| 20:43:21 | mriedem | x | |
| 20:43:28 | mriedem | 5. do another thing that hits the real thing that x was patching | |
| 20:43:42 | mriedem | 5 is blowing up b/c it doesn't appear that stop is stopping the patch | |
| 20:44:27 | mriedem | seems like it should work given https://docs.python.org/3.6/library/unittest.mock.html#patch-methods-start-and-stop | |
| 20:46:52 | mriedem | calling mock.patch.stopall() seems to do what 4 isn't | |
| 20:48:51 | mnaser | welp, it restarted happenign again not long after a service restart | |
| 20:50:42 | mriedem | maybe a bug in the mock library | |
| 20:52:31 | mriedem | anyway, i have a workaround, ignore me | |
| 20:58:13 | efried | mriedem: without seeing the code... | |
| 20:58:16 | efried | can I see the code? | |
| 20:58:29 | efried | Cause what you're describing seems like it ought to work | |
| 21:00:28 | efried | mriedem: a classic blunder is calling stop() on the *mock* (rather than the patcher) | |
| 21:00:41 | mriedem | yeah it wasn't that | |
| 21:00:50 | mriedem | p = mock.patch('x') | |
| 21:00:52 | mriedem | p.start() | |
| 21:00:55 | mriedem | ... | |
| 21:00:57 | mriedem | p.stop() | |
| 21:01:05 | mriedem | x() -> kablammo | |
| 21:01:22 | mriedem | err, p = mock.patch('x', side_effect=kablammo) | |
| 21:03:04 | mriedem | threads could be at play somehow, idk | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for build part of bug 1781286 https://review.opendev.org/685998 | |
| 21:03:40 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem) | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017 | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Selection.availability_zone field https://review.opendev.org/685807 | |
| 21:03:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047 | |
| 21:04:07 | mriedem | if you're a glutton for punishment, what i was doing was at this point https://review.opendev.org/#/c/686047/1/nova/tests/functional/regressions/test_bug_1781286.py@72 | |
| 21:04:12 | efried | In [8]: foo() | |
| 21:04:12 | efried | real | |
| 21:04:12 | efried | In [9]: p.start() | |
| 21:04:12 | efried | Out[9]: <MagicMock name='foo' id='139999791540112'> | |
| 21:04:12 | efried | In [10]: foo() | |
| 21:04:12 | mriedem | i changed that to be something like: | |
| 21:04:13 | efried | patched | |
| 21:04:13 | efried | In [11]: p.stop() | |
| 21:04:14 | efried | In [12]: foo() | |
| 21:04:14 | efried | real | |
| 21:04:29 | mriedem | self.agg_mock = mock.patch(...) | |
| 21:04:33 | mriedem | self.agg_mock.start() | |