| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 15:47:24 | imacdonn | cos I only know of the other guys who are seeing the same issue as me | |
| 15:47:36 | cdent | it's always running on wsgi, whether you're using nova-api script or using mod_wsgi or uwsgi. the distinguishing factor is standalone eventlet server versus everything else | |
| 15:48:09 | imacdonn | dansmith: see also https://bugs.launchpad.net/nova/+bug/1825584/comments/7 - I haven't really tried to digest that yet | |
| 15:48:11 | openstack | Launchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,New] | |
| 15:48:31 | sean-k-mooney | cdent: is it running on wsgi when using the console script | |
| 15:48:37 | sean-k-mooney | i did not think that was the case | |
| 15:49:16 | cdent | yes, wsgi is the protocol by which a web request is translated into an eviron dict that is passed to a __callable__ handler with a start_response function | |
| 15:50:01 | cdent | it's the reason why it is possible to run what amounts to the same code with the eventlet server based console script and "everything else" | |
| 15:50:04 | dansmith | imacdonn: okay if it's stopping the interpreter, why does using another real thread change it? does it just pause the main thread's interpreter and release the GIL?\ | |
| 15:50:32 | dansmith | if the wsgi server is pausing the interpreter, I take that to mean it expects the thing to stop doing stuff, and thus, shouldn't really be doing heartbeats? | |
| 15:50:32 | sean-k-mooney | cdent: i did not know https://github.com/openstack/nova/blob/master/nova/cmd/api.py was implement the wsgi progocaol but it look like it is | |
| 15:51:02 | cdent | imacdonn: have you tried melwitt's patch? does it "fix it". Because if not, maybe it doesn't... | |
| 15:51:23 | cdent | and what we said above about reopening the connection is the real fix | |
| 15:52:09 | imacdonn | cdent: depends which patch, I guess .. there was a set of 3 related ones, and the last one removed the monkey patching ... I did some basic testing of that, and it seemed to work OK | |
| 15:52:26 | imacdonn | by "that", I mean "all three" | |
| 15:52:38 | cdent | imacdonn: this one: https://review.opendev.org/#/c/650172/ | |
| 15:53:09 | imacdonn | I think I did (most of) these: https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged) | |
| 15:54:57 | dansmith | so if you disable monkeypatching but still use eventlet, I think everything just becomes synchronous, right? | |
| 15:55:35 | dansmith | in that case, the "it works" is probably just because you're doing everything serially, so unless you apply the other patches to make it actually use the native threading, it's not a relevant test, IMHO | |
| 15:55:40 | sean-k-mooney | dansmith: if you dont call eventlet directly i think so | |
| 15:55:54 | sean-k-mooney | im not sure what happens if you do call eventlet spawn et al | |
| 15:56:26 | dansmith | again, I think we need to repro this in a clean devstack and/or the gate before we get too much further.. or figure out why we can't. | |
| 15:57:57 | sean-k-mooney | you mean the rabbitmq issues | |
| 15:58:22 | sean-k-mooney | is the theroy that they will start happening if the api is not actively used after a few minutes | |
| 15:58:37 | sean-k-mooney | e.g. it shoudl break if we stack go have coffee and come back? | |
| 16:01:09 | bnemec | There's some analysis of what was happening in http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005310.html | |
| 16:02:37 | imacdonn | I missed the last ~5 mins ... *^&&*$% stupid VPN | |
| 16:03:04 | imacdonn | for repro, I think the best way is to enable debug logging for oslo_messaging and observe heartbeats (or lack thereof) | |
| 16:03:36 | cdent | dansmith, mriedem I put a comment with links to this discussion and some other summaries on the ThreadPoolExecutor patch | |
| 16:03:46 | imacdonn | (after making an API call that requires RPC - like getting the VNC console URL for an instance) | |
| 16:04:30 | fungi | mriedem et al: i found what was wrong with http://status.openstack.org/reviews/ and got content updating again, though i'm at a loss for explanation as to how it broke (one of the dependencies we pip install for reviewday, cheetah, was somehow not installed on the server and all reasonable explanations have been ruled out) | |
| 16:04:30 | dansmith | imacdonn: is it specific to vnc console by chance? because many api calls make rpc calls.. pretty much anything other than get/list instances | |
| 16:04:57 | fungi | anyway, let us know if you're relying on it and see it go stale again somehow | |
| 16:05:04 | fungi | maybe i'll be able to track it down that way | |
| 16:05:11 | fungi | should update roughly every 30 minutes | |
| 16:05:18 | imacdonn | dansmith: I don't believe so .. that's just an easy way to make an RPC call happen | |
| 16:07:37 | dansmith | confirming that would be good, because I think we have mucked with console stuff recently | |
| 16:07:49 | dansmith | and would also help narrow down the differences | |
| 16:09:06 | imacdonn | I have observed other things failing, but didn't make note of what specifically .. is there a particular call that you would suggest? | |
| 16:09:27 | dansmith | I suggest reproducing this with a devstack | |
| 16:09:38 | imacdonn | working on that (devstack) now | |
| 16:11:05 | mriedem | fungi: thanks | |
| 16:11:59 | fungi | any time! | |
| 16:40:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478 | |
| 16:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656 | |
| 16:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269 | |
| 16:40:57 | aspiers | kashyap, efried: are you good with https://review.opendev.org/#/c/655268/ now? | |
| 16:41:33 | mriedem | efried: i never +2ed this https://review.opendev.org/#/c/659374/ | |
| 16:41:40 | mriedem | well, | |
| 16:41:43 | mriedem | i did before -1ing it | |
| 16:44:03 | mriedem | but now i am :) | |
| 16:51:07 | imacdonn | dansmith: I've reproduced the lack of heartbeats on devstack (and confirmed that the heartbeats happen if monkey patching is removed) ... but somehow (so-far) devstack is able to recover from the unexpected loss of amqp connection, where my RDO environment is (sometimes) not | |
| 16:51:56 | dansmith | okay so devstack is behaving like we would expect, assuming the pause-the-interpreter wsgi behavior you describe | |
| 16:52:21 | imacdonn | so-far, I suppose so | |
| 16:52:40 | imacdonn | if the pause is considered "expected" | |
| 16:56:48 | dansmith | if the pause isn't, then the problem is in the configuration of the wsgi container or something else | |
| 16:57:29 | imacdonn | but it only happens when python has been monkeyed-with | |
| 16:58:06 | dansmith | mm, I think you're misunderstanding | |
| 16:58:13 | dansmith | I think that wsgi is pausing regardless, | |
| 16:58:22 | dansmith | and in the eventlet case, there is only one thing to pause, so it pauses | |
| 16:58:38 | dansmith | and without eventlet, we spawn a real thread that the wsgi container does not pause (only the main thread) | |
| 16:59:07 | dansmith | which is why I say that if the design of the wsgi container is to pause the whole thing between requests, then us subverting that with a real thread seems wrong | |
| 16:59:17 | imacdonn | I definitely don't understand the mechanisms well enough ... so my observation is fairly high-level ... in one case, the heartbeats happen, so it seems awake .. in the other case, it goes completely quiet | |
| 17:00:10 | imacdonn | so if it's wrong, the whole oslo_messaging model seems inappropriate ..... ??? | |
| 17:00:28 | imacdonn | cos it does heartbeats quite intentionally, AFAIK ... | |
| 17:00:49 | dansmith | well, o.msg _was_ developed with eventlet servers in mind | |
| 17:06:11 | dansmith | so here's a random person asking a similar question: https://github.com/benoitc/gunicorn/issues/1924 | |
| 17:06:29 | dansmith | not much response there, except that it basically says "don't spawn worker threads from your wsgi app" | |
| 17:08:50 | dansmith | check this out: https://docs.newrelic.com/docs/agents/python-agent/web-frameworks-servers/python-agent-uwsgi-web-server | |
| 17:09:05 | dansmith | "By default uWSGI does not enable threading support within the Python interpreter core. This means it is not possible to create background threads from Python code. " | |
| 17:09:35 | dansmith | ...which might mean that disabling eventlet and letting threading be unpatched just gets you synchronous behavior | |
| 17:11:04 | dansmith | I wonder if using --enable-threads but leaving the monkeypatching in place would allow the eventlet background threads to run? | |
| 17:12:02 | imacdonn | "By default the Python plugin does not initialize the GIL. This means your app-generated threads will not run. If you need threads, remember to enable them with enable-threads. Running uWSGI in multithreading mode (with the threads options) will automatically enable threading support. This “strange” default behaviour is for performance reasons, no shame in that." | |
| 17:12:17 | imacdonn | ^ from https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html | |
| 17:12:39 | dansmith | yeah, just saw that.. so... do you have threads enabled? | |
| 17:12:59 | imacdonn | checking ... probably not | |
| 17:14:02 | dansmith | if I were you, I'd really want to find out why devstack recovers properly and yours does not though | |
| 17:14:28 | imacdonn | or... | |
| 17:14:29 | imacdonn | # | |
| 17:14:29 | imacdonn | enable-threads = true | |
| 17:14:29 | imacdonn | # grep threads /etc/nova/nova-api-uwsgi.ini | |
| 17:14:34 | imacdonn | (this is on devstack) | |
| 17:15:41 | imacdonn | thinking about that some more ... I think if enable-threads was not on, the heartbeat thread would not work in the non-patched case (?) | |
| 17:15:47 | imacdonn | guess I could attempt to test that | |
| 17:16:59 | imacdonn | confirmed that I do have enable-threads in my "real" (non-devstack) config too | |
| 17:18:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix some issues with the newton release notes https://review.opendev.org/659623 | |
| 17:18:56 | dansmith | so, | |
| 17:19:00 | dansmith | thinking about this some more | |
| 17:19:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use new hotness for upper-constraints https://review.opendev.org/659624 | |
| 17:19:26 | dansmith | I'm not sure how eventlet could really ever work in the wsgi mode, because background tasks really require the main thread to block on the event loop | |
| 17:19:46 | dansmith | I'm struggling to find where the entry point is for nova-api in wsgi mode... | |
| 17:21:10 | imacdonn | that seems to be the consensus ... eventlet and wsgi containers seem fundamentally incompatible | |
| 17:21:23 | dansmith | well, | |
| 17:21:25 | dansmith | not really, | |
| 17:21:32 | dansmith | just wsgi+eventlet with a background thread | |
| 17:21:59 | imacdonn | ok | |
| 17:23:38 | imacdonn | FWIW, with enable-threads turned off, and monkey patching still removed, it looks the same - no heartbeats after the API request that opens an RPC connection | |
| 17:23:59 | dansmith | yeah, not running the event loop means we'll just never do that thing | |
| 17:25:17 | dansmith | tbh, it kinda seems like what we should be doing is splitting nova-api into two pieces in a wsgi world | |
| 17:25:23 | dansmith | which is, of course, not how it was designed | |