| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 15:50:04 | dansmith | imacdonn: okay if it's stopping the interpreter, why does using another real thread change it? does it just pause the main thread's interpreter and release the GIL?\ | |
| 15:50:32 | sean-k-mooney | cdent: i did not know https://github.com/openstack/nova/blob/master/nova/cmd/api.py was implement the wsgi progocaol but it look like it is | |
| 15:50:32 | dansmith | if the wsgi server is pausing the interpreter, I take that to mean it expects the thing to stop doing stuff, and thus, shouldn't really be doing heartbeats? | |
| 15:51:02 | cdent | imacdonn: have you tried melwitt's patch? does it "fix it". Because if not, maybe it doesn't... | |
| 15:51:23 | cdent | and what we said above about reopening the connection is the real fix | |
| 15:52:09 | imacdonn | cdent: depends which patch, I guess .. there was a set of 3 related ones, and the last one removed the monkey patching ... I did some basic testing of that, and it seemed to work OK | |
| 15:52:26 | imacdonn | by "that", I mean "all three" | |
| 15:52:38 | cdent | imacdonn: this one: https://review.opendev.org/#/c/650172/ | |
| 15:53:09 | imacdonn | I think I did (most of) these: https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged) | |
| 15:54:57 | dansmith | so if you disable monkeypatching but still use eventlet, I think everything just becomes synchronous, right? | |
| 15:55:35 | dansmith | in that case, the "it works" is probably just because you're doing everything serially, so unless you apply the other patches to make it actually use the native threading, it's not a relevant test, IMHO | |
| 15:55:40 | sean-k-mooney | dansmith: if you dont call eventlet directly i think so | |
| 15:55:54 | sean-k-mooney | im not sure what happens if you do call eventlet spawn et al | |
| 15:56:26 | dansmith | again, I think we need to repro this in a clean devstack and/or the gate before we get too much further.. or figure out why we can't. | |
| 15:57:57 | sean-k-mooney | you mean the rabbitmq issues | |
| 15:58:22 | sean-k-mooney | is the theroy that they will start happening if the api is not actively used after a few minutes | |
| 15:58:37 | sean-k-mooney | e.g. it shoudl break if we stack go have coffee and come back? | |
| 16:01:09 | bnemec | There's some analysis of what was happening in http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005310.html | |
| 16:02:37 | imacdonn | I missed the last ~5 mins ... *^&&*$% stupid VPN | |
| 16:03:04 | imacdonn | for repro, I think the best way is to enable debug logging for oslo_messaging and observe heartbeats (or lack thereof) | |
| 16:03:36 | cdent | dansmith, mriedem I put a comment with links to this discussion and some other summaries on the ThreadPoolExecutor patch | |
| 16:03:46 | imacdonn | (after making an API call that requires RPC - like getting the VNC console URL for an instance) | |
| 16:04:30 | dansmith | imacdonn: is it specific to vnc console by chance? because many api calls make rpc calls.. pretty much anything other than get/list instances | |
| 16:04:30 | fungi | mriedem et al: i found what was wrong with http://status.openstack.org/reviews/ and got content updating again, though i'm at a loss for explanation as to how it broke (one of the dependencies we pip install for reviewday, cheetah, was somehow not installed on the server and all reasonable explanations have been ruled out) | |
| 16:04:57 | fungi | anyway, let us know if you're relying on it and see it go stale again somehow | |
| 16:05:04 | fungi | maybe i'll be able to track it down that way | |
| 16:05:11 | fungi | should update roughly every 30 minutes | |
| 16:05:18 | imacdonn | dansmith: I don't believe so .. that's just an easy way to make an RPC call happen | |
| 16:07:37 | dansmith | confirming that would be good, because I think we have mucked with console stuff recently | |
| 16:07:49 | dansmith | and would also help narrow down the differences | |
| 16:09:06 | imacdonn | I have observed other things failing, but didn't make note of what specifically .. is there a particular call that you would suggest? | |
| 16:09:27 | dansmith | I suggest reproducing this with a devstack | |
| 16:09:38 | imacdonn | working on that (devstack) now | |
| 16:11:05 | mriedem | fungi: thanks | |
| 16:11:59 | fungi | any time! | |
| 16:40:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478 | |
| 16:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269 | |
| 16:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656 | |
| 16:40:57 | aspiers | kashyap, efried: are you good with https://review.opendev.org/#/c/655268/ now? | |
| 16:41:33 | mriedem | efried: i never +2ed this https://review.opendev.org/#/c/659374/ | |
| 16:41:40 | mriedem | well, | |
| 16:41:43 | mriedem | i did before -1ing it | |
| 16:44:03 | mriedem | but now i am :) | |
| 16:51:07 | imacdonn | dansmith: I've reproduced the lack of heartbeats on devstack (and confirmed that the heartbeats happen if monkey patching is removed) ... but somehow (so-far) devstack is able to recover from the unexpected loss of amqp connection, where my RDO environment is (sometimes) not | |
| 16:51:56 | dansmith | okay so devstack is behaving like we would expect, assuming the pause-the-interpreter wsgi behavior you describe | |
| 16:52:21 | imacdonn | so-far, I suppose so | |
| 16:52:40 | imacdonn | if the pause is considered "expected" | |
| 16:56:48 | dansmith | if the pause isn't, then the problem is in the configuration of the wsgi container or something else | |
| 16:57:29 | imacdonn | but it only happens when python has been monkeyed-with | |
| 16:58:06 | dansmith | mm, I think you're misunderstanding | |
| 16:58:13 | dansmith | I think that wsgi is pausing regardless, | |
| 16:58:22 | dansmith | and in the eventlet case, there is only one thing to pause, so it pauses | |
| 16:58:38 | dansmith | and without eventlet, we spawn a real thread that the wsgi container does not pause (only the main thread) | |
| 16:59:07 | dansmith | which is why I say that if the design of the wsgi container is to pause the whole thing between requests, then us subverting that with a real thread seems wrong | |
| 16:59:17 | imacdonn | I definitely don't understand the mechanisms well enough ... so my observation is fairly high-level ... in one case, the heartbeats happen, so it seems awake .. in the other case, it goes completely quiet | |
| 17:00:10 | imacdonn | so if it's wrong, the whole oslo_messaging model seems inappropriate ..... ??? | |
| 17:00:28 | imacdonn | cos it does heartbeats quite intentionally, AFAIK ... | |
| 17:00:49 | dansmith | well, o.msg _was_ developed with eventlet servers in mind | |
| 17:06:11 | dansmith | so here's a random person asking a similar question: https://github.com/benoitc/gunicorn/issues/1924 | |
| 17:06:29 | dansmith | not much response there, except that it basically says "don't spawn worker threads from your wsgi app" | |
| 17:08:50 | dansmith | check this out: https://docs.newrelic.com/docs/agents/python-agent/web-frameworks-servers/python-agent-uwsgi-web-server | |
| 17:09:05 | dansmith | "By default uWSGI does not enable threading support within the Python interpreter core. This means it is not possible to create background threads from Python code. " | |
| 17:09:35 | dansmith | ...which might mean that disabling eventlet and letting threading be unpatched just gets you synchronous behavior | |
| 17:11:04 | dansmith | I wonder if using --enable-threads but leaving the monkeypatching in place would allow the eventlet background threads to run? | |
| 17:12:02 | imacdonn | "By default the Python plugin does not initialize the GIL. This means your app-generated threads will not run. If you need threads, remember to enable them with enable-threads. Running uWSGI in multithreading mode (with the threads options) will automatically enable threading support. This “strange” default behaviour is for performance reasons, no shame in that." | |
| 17:12:17 | imacdonn | ^ from https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html | |
| 17:12:39 | dansmith | yeah, just saw that.. so... do you have threads enabled? | |
| 17:12:59 | imacdonn | checking ... probably not | |
| 17:14:02 | dansmith | if I were you, I'd really want to find out why devstack recovers properly and yours does not though | |
| 17:14:28 | imacdonn | or... | |
| 17:14:29 | imacdonn | # grep threads /etc/nova/nova-api-uwsgi.ini | |
| 17:14:29 | imacdonn | enable-threads = true | |
| 17:14:29 | imacdonn | # | |
| 17:14:34 | imacdonn | (this is on devstack) | |
| 17:15:41 | imacdonn | thinking about that some more ... I think if enable-threads was not on, the heartbeat thread would not work in the non-patched case (?) | |
| 17:15:47 | imacdonn | guess I could attempt to test that | |
| 17:16:59 | imacdonn | confirmed that I do have enable-threads in my "real" (non-devstack) config too | |
| 17:18:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix some issues with the newton release notes https://review.opendev.org/659623 | |
| 17:18:56 | dansmith | so, | |
| 17:19:00 | dansmith | thinking about this some more | |
| 17:19:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use new hotness for upper-constraints https://review.opendev.org/659624 | |
| 17:19:26 | dansmith | I'm not sure how eventlet could really ever work in the wsgi mode, because background tasks really require the main thread to block on the event loop | |
| 17:19:46 | dansmith | I'm struggling to find where the entry point is for nova-api in wsgi mode... | |
| 17:21:10 | imacdonn | that seems to be the consensus ... eventlet and wsgi containers seem fundamentally incompatible | |
| 17:21:23 | dansmith | well, | |
| 17:21:25 | dansmith | not really, | |
| 17:21:32 | dansmith | just wsgi+eventlet with a background thread | |
| 17:21:59 | imacdonn | ok | |
| 17:23:38 | imacdonn | FWIW, with enable-threads turned off, and monkey patching still removed, it looks the same - no heartbeats after the API request that opens an RPC connection | |
| 17:23:59 | dansmith | yeah, not running the event loop means we'll just never do that thing | |
| 17:25:17 | dansmith | tbh, it kinda seems like what we should be doing is splitting nova-api into two pieces in a wsgi world | |
| 17:25:23 | dansmith | which is, of course, not how it was designed | |
| 17:25:32 | imacdonn | someone else suggested that | |
| 17:27:33 | dansmith | mnaser: you around? | |
| 17:28:05 | dansmith | imacdonn: how instantaneous is the reconnect to rabbit on devstack? | |
| 17:28:48 | imacdonn | no perceptible delay | |
| 17:29:08 | mnaser | dansmith: hiya | |
| 17:29:30 | dansmith | mnaser: you are on stein and running nova-api in wsgi something right? | |
| 17:29:47 | mnaser | Correct. Uwsgi | |
| 17:30:01 | dansmith | mnaser: and you don't see this rabbit heartbeat timeout freezeup thing correct? | |