| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 15:36:57 | dansmith | I thought imacdonn said that it repros the same in uwsgi? | |
| 15:36:58 | cdent | but since the only people we've heard of this problem from are tripleo, it's not clear | |
| 15:37:01 | melwitt | I think imacdonn said it behaves the same uwsgi and mod_wsgi, IIRC | |
| 15:37:05 | imacdonn | I have easily reproduced the exact symptoms (including absense of heartbeats with debug logging) with both mod_wsgi and uWSGI | |
| 15:37:05 | cdent | ah, okay | |
| 15:37:10 | cdent | cool | |
| 15:38:07 | imacdonn | and I'm not tripleo, FWIW | |
| 15:38:26 | dansmith | ...said with beaming pride :) | |
| 15:38:29 | imacdonn | heh | |
| 15:38:33 | cdent | so is the other way around the problem to catch the reset by peer and reopen? because expecting an idle web service to make rpc heartbeats is ... odd | |
| 15:38:44 | cdent | imacdonn: sorry, like I said, it's been hard to keep track of the details on this one | |
| 15:38:45 | dansmith | yeah | |
| 15:38:47 | dansmith | so that's the other thing, | |
| 15:38:57 | dansmith | why is the connection not just immediately re-youknow-connected? | |
| 15:39:03 | dansmith | because that would be a lot more sane | |
| 15:39:10 | imacdonn | the whole point of heartbeats is to keep connections alive when there's no traffic | |
| 15:39:25 | imacdonn | for example, a firewall idle timeout could bite you otherwise | |
| 15:39:46 | cdent | sure, but a web server process may not even exist in some deployment scenarios when there are no requests | |
| 15:40:09 | cdent | who is going to send the heartbeat? | |
| 15:40:16 | dansmith | imacdonn: but only really for things where you might be expecting to hear something from the other side | |
| 15:40:30 | dansmith | imacdonn: the api server receives no messages other than replies, | |
| 15:40:36 | imacdonn | the wsgi app container, I guess? and if that doesn't exist, there's no problem ... a new one would start a new connection | |
| 15:40:46 | dansmith | so there's really no reason for a connection except when it's processing a reuest | |
| 15:41:41 | imacdonn | I guess I might have assumed that setting up a new connection incurs some expense ... so it's better to keep it open once you have it | |
| 15:42:13 | dansmith | open connections that aren't used are pretty expensive in rabbit terms ;) | |
| 15:42:14 | imacdonn | kindof like the REST concept | |
| 15:43:59 | imacdonn | re. "things where you might be expecting to hear something from the other side" - I wonder if there are cases where that could break - maybe an async RPC call that takes a long time to complete ? | |
| 15:44:25 | dansmith | imacdonn: in that case you're processing a request still | |
| 15:44:58 | dansmith | what I don't understand is, if the wsgi container has spawned a process to do a thing, and has not killed it off during times of inactivity, what is it doing that prevents the normal threading background stuff from running like any other process? | |
| 15:45:33 | dansmith | unless we've tickled something in the python threading library before patching or whatever, but if that's the problem that's the kind of thing we need to know | |
| 15:46:03 | dansmith | I'd be interested in what the differences are between you and everyone else.. same exact (unmodified) nova, oslo, et al code? same version of deps, etc? | |
| 15:46:19 | dansmith | these are the things worth chasing, imho | |
| 15:46:41 | imacdonn | do we know of anyone running stein nova-api on wsgi successfully ? | |
| 15:47:24 | imacdonn | cos I only know of the other guys who are seeing the same issue as me | |
| 15:47:36 | cdent | it's always running on wsgi, whether you're using nova-api script or using mod_wsgi or uwsgi. the distinguishing factor is standalone eventlet server versus everything else | |
| 15:48:09 | imacdonn | dansmith: see also https://bugs.launchpad.net/nova/+bug/1825584/comments/7 - I haven't really tried to digest that yet | |
| 15:48:11 | openstack | Launchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,New] | |
| 15:48:31 | sean-k-mooney | cdent: is it running on wsgi when using the console script | |
| 15:48:37 | sean-k-mooney | i did not think that was the case | |
| 15:49:16 | cdent | yes, wsgi is the protocol by which a web request is translated into an eviron dict that is passed to a __callable__ handler with a start_response function | |
| 15:50:01 | cdent | it's the reason why it is possible to run what amounts to the same code with the eventlet server based console script and "everything else" | |
| 15:50:04 | dansmith | imacdonn: okay if it's stopping the interpreter, why does using another real thread change it? does it just pause the main thread's interpreter and release the GIL?\ | |
| 15:50:32 | sean-k-mooney | cdent: i did not know https://github.com/openstack/nova/blob/master/nova/cmd/api.py was implement the wsgi progocaol but it look like it is | |
| 15:50:32 | dansmith | if the wsgi server is pausing the interpreter, I take that to mean it expects the thing to stop doing stuff, and thus, shouldn't really be doing heartbeats? | |
| 15:51:02 | cdent | imacdonn: have you tried melwitt's patch? does it "fix it". Because if not, maybe it doesn't... | |
| 15:51:23 | cdent | and what we said above about reopening the connection is the real fix | |
| 15:52:09 | imacdonn | cdent: depends which patch, I guess .. there was a set of 3 related ones, and the last one removed the monkey patching ... I did some basic testing of that, and it seemed to work OK | |
| 15:52:26 | imacdonn | by "that", I mean "all three" | |
| 15:52:38 | cdent | imacdonn: this one: https://review.opendev.org/#/c/650172/ | |
| 15:53:09 | imacdonn | I think I did (most of) these: https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged) | |
| 15:54:57 | dansmith | so if you disable monkeypatching but still use eventlet, I think everything just becomes synchronous, right? | |
| 15:55:35 | dansmith | in that case, the "it works" is probably just because you're doing everything serially, so unless you apply the other patches to make it actually use the native threading, it's not a relevant test, IMHO | |
| 15:55:40 | sean-k-mooney | dansmith: if you dont call eventlet directly i think so | |
| 15:55:54 | sean-k-mooney | im not sure what happens if you do call eventlet spawn et al | |
| 15:56:26 | dansmith | again, I think we need to repro this in a clean devstack and/or the gate before we get too much further.. or figure out why we can't. | |
| 15:57:57 | sean-k-mooney | you mean the rabbitmq issues | |
| 15:58:22 | sean-k-mooney | is the theroy that they will start happening if the api is not actively used after a few minutes | |
| 15:58:37 | sean-k-mooney | e.g. it shoudl break if we stack go have coffee and come back? | |
| 16:01:09 | bnemec | There's some analysis of what was happening in http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005310.html | |
| 16:02:37 | imacdonn | I missed the last ~5 mins ... *^&&*$% stupid VPN | |
| 16:03:04 | imacdonn | for repro, I think the best way is to enable debug logging for oslo_messaging and observe heartbeats (or lack thereof) | |
| 16:03:36 | cdent | dansmith, mriedem I put a comment with links to this discussion and some other summaries on the ThreadPoolExecutor patch | |
| 16:03:46 | imacdonn | (after making an API call that requires RPC - like getting the VNC console URL for an instance) | |
| 16:04:30 | dansmith | imacdonn: is it specific to vnc console by chance? because many api calls make rpc calls.. pretty much anything other than get/list instances | |
| 16:04:30 | fungi | mriedem et al: i found what was wrong with http://status.openstack.org/reviews/ and got content updating again, though i'm at a loss for explanation as to how it broke (one of the dependencies we pip install for reviewday, cheetah, was somehow not installed on the server and all reasonable explanations have been ruled out) | |
| 16:04:57 | fungi | anyway, let us know if you're relying on it and see it go stale again somehow | |
| 16:05:04 | fungi | maybe i'll be able to track it down that way | |
| 16:05:11 | fungi | should update roughly every 30 minutes | |
| 16:05:18 | imacdonn | dansmith: I don't believe so .. that's just an easy way to make an RPC call happen | |
| 16:07:37 | dansmith | confirming that would be good, because I think we have mucked with console stuff recently | |
| 16:07:49 | dansmith | and would also help narrow down the differences | |
| 16:09:06 | imacdonn | I have observed other things failing, but didn't make note of what specifically .. is there a particular call that you would suggest? | |
| 16:09:27 | dansmith | I suggest reproducing this with a devstack | |
| 16:09:38 | imacdonn | working on that (devstack) now | |
| 16:11:05 | mriedem | fungi: thanks | |
| 16:11:59 | fungi | any time! | |
| 16:40:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478 | |
| 16:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269 | |
| 16:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656 | |
| 16:40:57 | aspiers | kashyap, efried: are you good with https://review.opendev.org/#/c/655268/ now? | |
| 16:41:33 | mriedem | efried: i never +2ed this https://review.opendev.org/#/c/659374/ | |
| 16:41:40 | mriedem | well, | |
| 16:41:43 | mriedem | i did before -1ing it | |
| 16:44:03 | mriedem | but now i am :) | |
| 16:51:07 | imacdonn | dansmith: I've reproduced the lack of heartbeats on devstack (and confirmed that the heartbeats happen if monkey patching is removed) ... but somehow (so-far) devstack is able to recover from the unexpected loss of amqp connection, where my RDO environment is (sometimes) not | |
| 16:51:56 | dansmith | okay so devstack is behaving like we would expect, assuming the pause-the-interpreter wsgi behavior you describe | |
| 16:52:21 | imacdonn | so-far, I suppose so | |
| 16:52:40 | imacdonn | if the pause is considered "expected" | |
| 16:56:48 | dansmith | if the pause isn't, then the problem is in the configuration of the wsgi container or something else | |
| 16:57:29 | imacdonn | but it only happens when python has been monkeyed-with | |
| 16:58:06 | dansmith | mm, I think you're misunderstanding | |
| 16:58:13 | dansmith | I think that wsgi is pausing regardless, | |
| 16:58:22 | dansmith | and in the eventlet case, there is only one thing to pause, so it pauses | |
| 16:58:38 | dansmith | and without eventlet, we spawn a real thread that the wsgi container does not pause (only the main thread) | |
| 16:59:07 | dansmith | which is why I say that if the design of the wsgi container is to pause the whole thing between requests, then us subverting that with a real thread seems wrong | |
| 16:59:17 | imacdonn | I definitely don't understand the mechanisms well enough ... so my observation is fairly high-level ... in one case, the heartbeats happen, so it seems awake .. in the other case, it goes completely quiet | |
| 17:00:10 | imacdonn | so if it's wrong, the whole oslo_messaging model seems inappropriate ..... ??? | |
| 17:00:28 | imacdonn | cos it does heartbeats quite intentionally, AFAIK ... | |
| 17:00:49 | dansmith | well, o.msg _was_ developed with eventlet servers in mind | |
| 17:06:11 | dansmith | so here's a random person asking a similar question: https://github.com/benoitc/gunicorn/issues/1924 | |