Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-16
15:27:59 efried agreed
15:28:03 mriedem i'll push that up
15:28:05 efried ight
15:28:30 efried mriedem: I made a story
15:28:31 mriedem this would also suggest all ironic jobs would be busted since stein and if that were the case i'd think we would have heard about it by now...
15:28:36 efried mriedem: https://storyboard.openstack.org/#!/story/2005725
15:28:42 efried you would think, yah
15:29:33 mriedem cdent: melwitt: dansmith: you don't think it's a simple as adding an https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.sleep between calls in the scatter gather do you?
15:30:30 imacdonn melwitt: I don't know enough about what happens in the gate ... but the problem manifests when nova-api does something that requires an API call, then goes quiet for more than a minute or so, then gets another request that requires RPC
15:30:36 dansmith the problem is that we're not monkeypatched properly such that background threads aren't being run and thus the heartbeat for the rabbit connection isn't working
15:31:20 dansmith which seems like it should manifest in the gate easily if it's fundamental to how we're doing service setup, and which also means other things shouldn't really be working
15:31:42 dansmith so I'm kinda suspicious of the whole thing, and don't really want to "solve" it by changing our whole threading model for one service
15:31:49 dansmith but I don't really have time to dig deep, as I said
15:32:18 cdent dansmith: I thought the "more than a minute or so" was a critical factor as it is resulting in mod_wsgi doing $something to the python process that breaks the monkey patching
15:32:47 cdent but I have to admit that finding the sensible thread through all this is hard so I'm not sure if I'm remembering the right stuff
15:33:05 mriedem going quite for a minute or so is something that wouldn't happen in the gate yeah
15:33:07 imacdonn cdent: rabbitmq, in a typical configuration, will drop a connection if it has not seen heartbeats for over a minute
15:33:13 mriedem tempest is all hammer until it's done and then we teardown
15:33:38 imacdonn cdent: that's what leads to a "connection reset by peer" when nova-api wakes up again and tries to use the connection that it thinks is still healthy
15:33:45 dansmith so you think we're just not running hearbeat background tasks if no api requests are coming in?
15:33:57 cdent it does sound a bit that way
15:34:12 imacdonn I have observed that with debug logging
15:34:47 dansmith seems like that should be easy to reproduce in a devstack no?
15:34:52 mriedem could we maybe tickle this in a post test script after tempest is done, sleep a minute or something, and then try to create or do something to a server, like stop it?
15:35:05 kashyap efried: Thanks for representing on #openstack-meeting (still haven't caught up). Stuck in consecutive calls, afraid.
15:35:20 dansmith mriedem: or just a devstack
15:35:39 mriedem yeah..i will leave devstack running for many more minutes before coming back to it to do something
15:35:42 dansmith changing the whole threading model for this seems like a big hammer
15:36:10 mriedem also note that vexxhost is running stein and haven't reported this being a problem...
15:36:20 cdent mriedem: is your devstack using mod_wsgi, by default it won't be
15:36:30 mriedem cdent: oh no i use the default (uwsgi)
15:36:31 cdent and it's not clear the problem is the same uwsgi as mod_wsgi
15:36:37 cdent it _might_ be
15:36:57 dansmith I thought imacdonn said that it repros the same in uwsgi?
15:36:58 cdent but since the only people we've heard of this problem from are tripleo, it's not clear
15:37:01 melwitt I think imacdonn said it behaves the same uwsgi and mod_wsgi, IIRC
15:37:05 imacdonn I have easily reproduced the exact symptoms (including absense of heartbeats with debug logging) with both mod_wsgi and uWSGI
15:37:05 cdent ah, okay
15:37:10 cdent cool
15:38:07 imacdonn and I'm not tripleo, FWIW
15:38:26 dansmith ...said with beaming pride :)
15:38:29 imacdonn heh
15:38:33 cdent so is the other way around the problem to catch the reset by peer and reopen? because expecting an idle web service to make rpc heartbeats is ... odd
15:38:44 cdent imacdonn: sorry, like I said, it's been hard to keep track of the details on this one
15:38:45 dansmith yeah
15:38:47 dansmith so that's the other thing,
15:38:57 dansmith why is the connection not just immediately re-youknow-connected?
15:39:03 dansmith because that would be a lot more sane
15:39:10 imacdonn the whole point of heartbeats is to keep connections alive when there's no traffic
15:39:25 imacdonn for example, a firewall idle timeout could bite you otherwise
15:39:46 cdent sure, but a web server process may not even exist in some deployment scenarios when there are no requests
15:40:09 cdent who is going to send the heartbeat?
15:40:16 dansmith imacdonn: but only really for things where you might be expecting to hear something from the other side
15:40:30 dansmith imacdonn: the api server receives no messages other than replies,
15:40:36 imacdonn the wsgi app container, I guess? and if that doesn't exist, there's no problem ... a new one would start a new connection
15:40:46 dansmith so there's really no reason for a connection except when it's processing a reuest
15:41:41 imacdonn I guess I might have assumed that setting up a new connection incurs some expense ... so it's better to keep it open once you have it
15:42:13 dansmith open connections that aren't used are pretty expensive in rabbit terms ;)
15:42:14 imacdonn kindof like the REST concept
15:43:59 imacdonn re. "things where you might be expecting to hear something from the other side" - I wonder if there are cases where that could break - maybe an async RPC call that takes a long time to complete ?
15:44:25 dansmith imacdonn: in that case you're processing a request still
15:44:58 dansmith what I don't understand is, if the wsgi container has spawned a process to do a thing, and has not killed it off during times of inactivity, what is it doing that prevents the normal threading background stuff from running like any other process?
15:45:33 dansmith unless we've tickled something in the python threading library before patching or whatever, but if that's the problem that's the kind of thing we need to know
15:46:03 dansmith I'd be interested in what the differences are between you and everyone else.. same exact (unmodified) nova, oslo, et al code? same version of deps, etc?
15:46:19 dansmith these are the things worth chasing, imho
15:46:41 imacdonn do we know of anyone running stein nova-api on wsgi successfully ?
15:47:24 imacdonn cos I only know of the other guys who are seeing the same issue as me
15:47:36 cdent it's always running on wsgi, whether you're using nova-api script or using mod_wsgi or uwsgi. the distinguishing factor is standalone eventlet server versus everything else
15:48:09 imacdonn dansmith: see also https://bugs.launchpad.net/nova/+bug/1825584/comments/7 - I haven't really tried to digest that yet
15:48:11 openstack Launchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,New]
15:48:31 sean-k-mooney cdent: is it running on wsgi when using the console script
15:48:37 sean-k-mooney i did not think that was the case
15:49:16 cdent yes, wsgi is the protocol by which a web request is translated into an eviron dict that is passed to a __callable__ handler with a start_response function
15:50:01 cdent it's the reason why it is possible to run what amounts to the same code with the eventlet server based console script and "everything else"
15:50:04 dansmith imacdonn: okay if it's stopping the interpreter, why does using another real thread change it? does it just pause the main thread's interpreter and release the GIL?\
15:50:32 sean-k-mooney cdent: i did not know https://github.com/openstack/nova/blob/master/nova/cmd/api.py was implement the wsgi progocaol but it look like it is
15:50:32 dansmith if the wsgi server is pausing the interpreter, I take that to mean it expects the thing to stop doing stuff, and thus, shouldn't really be doing heartbeats?
15:51:02 cdent imacdonn: have you tried melwitt's patch? does it "fix it". Because if not, maybe it doesn't...
15:51:23 cdent and what we said above about reopening the connection is the real fix
15:52:09 imacdonn cdent: depends which patch, I guess .. there was a set of 3 related ones, and the last one removed the monkey patching ... I did some basic testing of that, and it seemed to work OK
15:52:26 imacdonn by "that", I mean "all three"
15:52:38 cdent imacdonn: this one: https://review.opendev.org/#/c/650172/
15:53:09 imacdonn I think I did (most of) these: https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged)
15:54:57 dansmith so if you disable monkeypatching but still use eventlet, I think everything just becomes synchronous, right?
15:55:35 dansmith in that case, the "it works" is probably just because you're doing everything serially, so unless you apply the other patches to make it actually use the native threading, it's not a relevant test, IMHO
15:55:40 sean-k-mooney dansmith: if you dont call eventlet directly i think so
15:55:54 sean-k-mooney im not sure what happens if you do call eventlet spawn et al
15:56:26 dansmith again, I think we need to repro this in a clean devstack and/or the gate before we get too much further.. or figure out why we can't.
15:57:57 sean-k-mooney you mean the rabbitmq issues
15:58:22 sean-k-mooney is the theroy that they will start happening if the api is not actively used after a few minutes
15:58:37 sean-k-mooney e.g. it shoudl break if we stack go have coffee and come back?
16:01:09 bnemec There's some analysis of what was happening in http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005310.html
16:02:37 imacdonn I missed the last ~5 mins ... *^&&*$% stupid VPN
16:03:04 imacdonn for repro, I think the best way is to enable debug logging for oslo_messaging and observe heartbeats (or lack thereof)
16:03:36 cdent dansmith, mriedem I put a comment with links to this discussion and some other summaries on the ThreadPoolExecutor patch
16:03:46 imacdonn (after making an API call that requires RPC - like getting the VNC console URL for an instance)
16:04:30 dansmith imacdonn: is it specific to vnc console by chance? because many api calls make rpc calls.. pretty much anything other than get/list instances
16:04:30 fungi mriedem et al: i found what was wrong with http://status.openstack.org/reviews/ and got content updating again, though i'm at a loss for explanation as to how it broke (one of the dependencies we pip install for reviewday, cheetah, was somehow not installed on the server and all reasonable explanations have been ruled out)
16:04:57 fungi anyway, let us know if you're relying on it and see it go stale again somehow
16:05:04 fungi maybe i'll be able to track it down that way

Earlier   Later