Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-16
15:43:59 imacdonn re. "things where you might be expecting to hear something from the other side" - I wonder if there are cases where that could break - maybe an async RPC call that takes a long time to complete ?
15:44:25 dansmith imacdonn: in that case you're processing a request still
15:44:58 dansmith what I don't understand is, if the wsgi container has spawned a process to do a thing, and has not killed it off during times of inactivity, what is it doing that prevents the normal threading background stuff from running like any other process?
15:45:33 dansmith unless we've tickled something in the python threading library before patching or whatever, but if that's the problem that's the kind of thing we need to know
15:46:03 dansmith I'd be interested in what the differences are between you and everyone else.. same exact (unmodified) nova, oslo, et al code? same version of deps, etc?
15:46:19 dansmith these are the things worth chasing, imho
15:46:41 imacdonn do we know of anyone running stein nova-api on wsgi successfully ?
15:47:24 imacdonn cos I only know of the other guys who are seeing the same issue as me
15:47:36 cdent it's always running on wsgi, whether you're using nova-api script or using mod_wsgi or uwsgi. the distinguishing factor is standalone eventlet server versus everything else
15:48:09 imacdonn dansmith: see also https://bugs.launchpad.net/nova/+bug/1825584/comments/7 - I haven't really tried to digest that yet
15:48:11 openstack Launchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,New]
15:48:31 sean-k-mooney cdent: is it running on wsgi when using the console script
15:48:37 sean-k-mooney i did not think that was the case
15:49:16 cdent yes, wsgi is the protocol by which a web request is translated into an eviron dict that is passed to a __callable__ handler with a start_response function
15:50:01 cdent it's the reason why it is possible to run what amounts to the same code with the eventlet server based console script and "everything else"
15:50:04 dansmith imacdonn: okay if it's stopping the interpreter, why does using another real thread change it? does it just pause the main thread's interpreter and release the GIL?\
15:50:32 sean-k-mooney cdent: i did not know https://github.com/openstack/nova/blob/master/nova/cmd/api.py was implement the wsgi progocaol but it look like it is
15:50:32 dansmith if the wsgi server is pausing the interpreter, I take that to mean it expects the thing to stop doing stuff, and thus, shouldn't really be doing heartbeats?
15:51:02 cdent imacdonn: have you tried melwitt's patch? does it "fix it". Because if not, maybe it doesn't...
15:51:23 cdent and what we said above about reopening the connection is the real fix
15:52:09 imacdonn cdent: depends which patch, I guess .. there was a set of 3 related ones, and the last one removed the monkey patching ... I did some basic testing of that, and it seemed to work OK
15:52:26 imacdonn by "that", I mean "all three"
15:52:38 cdent imacdonn: this one: https://review.opendev.org/#/c/650172/
15:53:09 imacdonn I think I did (most of) these: https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged)
15:54:57 dansmith so if you disable monkeypatching but still use eventlet, I think everything just becomes synchronous, right?
15:55:35 dansmith in that case, the "it works" is probably just because you're doing everything serially, so unless you apply the other patches to make it actually use the native threading, it's not a relevant test, IMHO
15:55:40 sean-k-mooney dansmith: if you dont call eventlet directly i think so
15:55:54 sean-k-mooney im not sure what happens if you do call eventlet spawn et al
15:56:26 dansmith again, I think we need to repro this in a clean devstack and/or the gate before we get too much further.. or figure out why we can't.
15:57:57 sean-k-mooney you mean the rabbitmq issues
15:58:22 sean-k-mooney is the theroy that they will start happening if the api is not actively used after a few minutes
15:58:37 sean-k-mooney e.g. it shoudl break if we stack go have coffee and come back?
16:01:09 bnemec There's some analysis of what was happening in http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005310.html
16:02:37 imacdonn I missed the last ~5 mins ... *^&&*$% stupid VPN
16:03:04 imacdonn for repro, I think the best way is to enable debug logging for oslo_messaging and observe heartbeats (or lack thereof)
16:03:36 cdent dansmith, mriedem I put a comment with links to this discussion and some other summaries on the ThreadPoolExecutor patch
16:03:46 imacdonn (after making an API call that requires RPC - like getting the VNC console URL for an instance)
16:04:30 dansmith imacdonn: is it specific to vnc console by chance? because many api calls make rpc calls.. pretty much anything other than get/list instances
16:04:30 fungi mriedem et al: i found what was wrong with http://status.openstack.org/reviews/ and got content updating again, though i'm at a loss for explanation as to how it broke (one of the dependencies we pip install for reviewday, cheetah, was somehow not installed on the server and all reasonable explanations have been ruled out)
16:04:57 fungi anyway, let us know if you're relying on it and see it go stale again somehow
16:05:04 fungi maybe i'll be able to track it down that way
16:05:11 fungi should update roughly every 30 minutes
16:05:18 imacdonn dansmith: I don't believe so .. that's just an easy way to make an RPC call happen
16:07:37 dansmith confirming that would be good, because I think we have mucked with console stuff recently
16:07:49 dansmith and would also help narrow down the differences
16:09:06 imacdonn I have observed other things failing, but didn't make note of what specifically .. is there a particular call that you would suggest?
16:09:27 dansmith I suggest reproducing this with a devstack
16:09:38 imacdonn working on that (devstack) now
16:11:05 mriedem fungi: thanks
16:11:59 fungi any time!
16:40:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478
16:40:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269
16:40:41 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656
16:40:57 aspiers kashyap, efried: are you good with https://review.opendev.org/#/c/655268/ now?
16:41:33 mriedem efried: i never +2ed this https://review.opendev.org/#/c/659374/
16:41:40 mriedem well,
16:41:43 mriedem i did before -1ing it
16:44:03 mriedem but now i am :)
16:51:07 imacdonn dansmith: I've reproduced the lack of heartbeats on devstack (and confirmed that the heartbeats happen if monkey patching is removed) ... but somehow (so-far) devstack is able to recover from the unexpected loss of amqp connection, where my RDO environment is (sometimes) not
16:51:56 dansmith okay so devstack is behaving like we would expect, assuming the pause-the-interpreter wsgi behavior you describe
16:52:21 imacdonn so-far, I suppose so
16:52:40 imacdonn if the pause is considered "expected"
16:56:48 dansmith if the pause isn't, then the problem is in the configuration of the wsgi container or something else
16:57:29 imacdonn but it only happens when python has been monkeyed-with
16:58:06 dansmith mm, I think you're misunderstanding
16:58:13 dansmith I think that wsgi is pausing regardless,
16:58:22 dansmith and in the eventlet case, there is only one thing to pause, so it pauses
16:58:38 dansmith and without eventlet, we spawn a real thread that the wsgi container does not pause (only the main thread)
16:59:07 dansmith which is why I say that if the design of the wsgi container is to pause the whole thing between requests, then us subverting that with a real thread seems wrong
16:59:17 imacdonn I definitely don't understand the mechanisms well enough ... so my observation is fairly high-level ... in one case, the heartbeats happen, so it seems awake .. in the other case, it goes completely quiet
17:00:10 imacdonn so if it's wrong, the whole oslo_messaging model seems inappropriate ..... ???
17:00:28 imacdonn cos it does heartbeats quite intentionally, AFAIK ...
17:00:49 dansmith well, o.msg _was_ developed with eventlet servers in mind
17:06:11 dansmith so here's a random person asking a similar question: https://github.com/benoitc/gunicorn/issues/1924
17:06:29 dansmith not much response there, except that it basically says "don't spawn worker threads from your wsgi app"
17:08:50 dansmith check this out: https://docs.newrelic.com/docs/agents/python-agent/web-frameworks-servers/python-agent-uwsgi-web-server
17:09:05 dansmith "By default uWSGI does not enable threading support within the Python interpreter core. This means it is not possible to create background threads from Python code. "
17:09:35 dansmith ...which might mean that disabling eventlet and letting threading be unpatched just gets you synchronous behavior
17:11:04 dansmith I wonder if using --enable-threads but leaving the monkeypatching in place would allow the eventlet background threads to run?
17:12:02 imacdonn "By default the Python plugin does not initialize the GIL. This means your app-generated threads will not run. If you need threads, remember to enable them with enable-threads. Running uWSGI in multithreading mode (with the threads options) will automatically enable threading support. This “strange” default behaviour is for performance reasons, no shame in that."
17:12:17 imacdonn ^ from https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
17:12:39 dansmith yeah, just saw that.. so... do you have threads enabled?
17:12:59 imacdonn checking ... probably not
17:14:02 dansmith if I were you, I'd really want to find out why devstack recovers properly and yours does not though
17:14:28 imacdonn or...
17:14:29 imacdonn # grep threads /etc/nova/nova-api-uwsgi.ini
17:14:29 imacdonn enable-threads = true
17:14:29 imacdonn #
17:14:34 imacdonn (this is on devstack)
17:15:41 imacdonn thinking about that some more ... I think if enable-threads was not on, the heartbeat thread would not work in the non-patched case (?)
17:15:47 imacdonn guess I could attempt to test that
17:16:59 imacdonn confirmed that I do have enable-threads in my "real" (non-devstack) config too
17:18:22 openstackgerrit Stephen Finucane proposed openstack/nova master: Fix some issues with the newton release notes https://review.opendev.org/659623
17:18:56 dansmith so,
17:19:00 dansmith thinking about this some more
17:19:16 openstackgerrit Stephen Finucane proposed openstack/nova master: Use new hotness for upper-constraints https://review.opendev.org/659624
17:19:26 dansmith I'm not sure how eventlet could really ever work in the wsgi mode, because background tasks really require the main thread to block on the event loop
17:19:46 dansmith I'm struggling to find where the entry point is for nova-api in wsgi mode...
17:21:10 imacdonn that seems to be the consensus ... eventlet and wsgi containers seem fundamentally incompatible
17:21:23 dansmith well,

Earlier   Later