| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 15:11:24 | dansmith | ..a thing for which eventlet is ideal | |
| 15:11:24 | sean-k-mooney | eventlet and thread pools both allwo that with different tradeoffs | |
| 15:11:50 | sean-k-mooney | the issue isnt the use of eventlets its self | |
| 15:12:38 | sean-k-mooney | its that mod_wsgi and uwsgi both have exposed diffreent bugs caused by using eventlets with them | |
| 15:15:06 | sean-k-mooney | does wsgi provide a way to run request like this in the background that we could leverag and fall back to eventlest when not runnign under wsgi? | |
| 15:16:03 | sean-k-mooney | google found me https://pypi.org/project/uwsgi-tasks/ but no idea if that would help | |
| 15:18:52 | cdent | that could work for the uwsgi situation, but not mod_wsgi, presumably | |
| 15:19:18 | sean-k-mooney | i dont know i personally would prefer to try melwitt's approch first and maybe ask cern to try it and get feed back | |
| 15:20:38 | cdent | sean-k-mooney: you mean the current code under review? that seems reasonable. I guess the issue is that we don't know if the problem is a problem until it gets hit... | |
| 15:20:56 | sean-k-mooney | cdent: right | |
| 15:21:06 | cdent | mriedem seemed to think it could be caused with some effort in devstack | |
| 15:21:12 | sean-k-mooney | and yes the current code under review | |
| 15:21:57 | efried | mriedem: I'm going to propose that fix unless you've arleady got it in the pipe | |
| 15:23:43 | mriedem | efried: i don't, i'm trying to recreate with bash locally | |
| 15:23:56 | efried | ack | |
| 15:24:17 | mriedem | cdent: i know how to manually test multiple cells in a single node devstack and make one or more down, but that doesn't mean i know how to recreate the issue | |
| 15:24:38 | mriedem | we haven't hit this problem in the gate as far as i know with nova-api under wsgi + 2 cells | |
| 15:24:54 | mriedem | we also now have the nova-multi-cell job which is effectively 3 cells | |
| 15:25:00 | mriedem | cell0/1/2 | |
| 15:26:28 | melwitt | imacdonn: do you happen to know why/how the uwsgi/mod_wsgi problem doesn't seem to show itself in the gate? | |
| 15:26:29 | cdent | as I remmeber (which may be wrong) the problem with eventlet needs some work, a long pause, some attempted work to show up. And the expected problem with threads requires connections to databases (or other I/O) that blocks weirdly and doesn't time out in natural ways | |
| 15:26:30 | mriedem | efried: removing the brackets gives me this: | |
| 15:26:30 | mriedem | jq: error (at <stdin>:1): array (["COMPUTE_N...) and string ("CUSTOM_GOLD") cannot have their containment checked | |
| 15:26:40 | cdent | melwitt: I reckon it the lack of pause | |
| 15:26:49 | melwitt | aye | |
| 15:27:01 | efried | mriedem: ugh, I guess it would help me to know wtf jq is. | |
| 15:27:08 | mriedem | efried: json parser | |
| 15:27:24 | mriedem | efried: i considered just re-writing this with osc-placement and grep | |
| 15:27:42 | efried | seems like that would be better. | |
| 15:27:47 | mriedem | openstack --os-placement-api-version 1.6 resource provider trait list $provider_id | grep $trait | |
| 15:27:55 | mriedem | it would definitely be easier to f'ing read | |
| 15:27:59 | efried | agreed | |
| 15:28:03 | mriedem | i'll push that up | |
| 15:28:05 | efried | ight | |
| 15:28:30 | efried | mriedem: I made a story | |
| 15:28:31 | mriedem | this would also suggest all ironic jobs would be busted since stein and if that were the case i'd think we would have heard about it by now... | |
| 15:28:36 | efried | mriedem: https://storyboard.openstack.org/#!/story/2005725 | |
| 15:28:42 | efried | you would think, yah | |
| 15:29:33 | mriedem | cdent: melwitt: dansmith: you don't think it's a simple as adding an https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.sleep between calls in the scatter gather do you? | |
| 15:30:30 | imacdonn | melwitt: I don't know enough about what happens in the gate ... but the problem manifests when nova-api does something that requires an API call, then goes quiet for more than a minute or so, then gets another request that requires RPC | |
| 15:30:36 | dansmith | the problem is that we're not monkeypatched properly such that background threads aren't being run and thus the heartbeat for the rabbit connection isn't working | |
| 15:31:20 | dansmith | which seems like it should manifest in the gate easily if it's fundamental to how we're doing service setup, and which also means other things shouldn't really be working | |
| 15:31:42 | dansmith | so I'm kinda suspicious of the whole thing, and don't really want to "solve" it by changing our whole threading model for one service | |
| 15:31:49 | dansmith | but I don't really have time to dig deep, as I said | |
| 15:32:18 | cdent | dansmith: I thought the "more than a minute or so" was a critical factor as it is resulting in mod_wsgi doing $something to the python process that breaks the monkey patching | |
| 15:32:47 | cdent | but I have to admit that finding the sensible thread through all this is hard so I'm not sure if I'm remembering the right stuff | |
| 15:33:05 | mriedem | going quite for a minute or so is something that wouldn't happen in the gate yeah | |
| 15:33:07 | imacdonn | cdent: rabbitmq, in a typical configuration, will drop a connection if it has not seen heartbeats for over a minute | |
| 15:33:13 | mriedem | tempest is all hammer until it's done and then we teardown | |
| 15:33:38 | imacdonn | cdent: that's what leads to a "connection reset by peer" when nova-api wakes up again and tries to use the connection that it thinks is still healthy | |
| 15:33:45 | dansmith | so you think we're just not running hearbeat background tasks if no api requests are coming in? | |
| 15:33:57 | cdent | it does sound a bit that way | |
| 15:34:12 | imacdonn | I have observed that with debug logging | |
| 15:34:47 | dansmith | seems like that should be easy to reproduce in a devstack no? | |
| 15:34:52 | mriedem | could we maybe tickle this in a post test script after tempest is done, sleep a minute or something, and then try to create or do something to a server, like stop it? | |
| 15:35:05 | kashyap | efried: Thanks for representing on #openstack-meeting (still haven't caught up). Stuck in consecutive calls, afraid. | |
| 15:35:20 | dansmith | mriedem: or just a devstack | |
| 15:35:39 | mriedem | yeah..i will leave devstack running for many more minutes before coming back to it to do something | |
| 15:35:42 | dansmith | changing the whole threading model for this seems like a big hammer | |
| 15:36:10 | mriedem | also note that vexxhost is running stein and haven't reported this being a problem... | |
| 15:36:20 | cdent | mriedem: is your devstack using mod_wsgi, by default it won't be | |
| 15:36:30 | mriedem | cdent: oh no i use the default (uwsgi) | |
| 15:36:31 | cdent | and it's not clear the problem is the same uwsgi as mod_wsgi | |
| 15:36:37 | cdent | it _might_ be | |
| 15:36:57 | dansmith | I thought imacdonn said that it repros the same in uwsgi? | |
| 15:36:58 | cdent | but since the only people we've heard of this problem from are tripleo, it's not clear | |
| 15:37:01 | melwitt | I think imacdonn said it behaves the same uwsgi and mod_wsgi, IIRC | |
| 15:37:05 | imacdonn | I have easily reproduced the exact symptoms (including absense of heartbeats with debug logging) with both mod_wsgi and uWSGI | |
| 15:37:05 | cdent | ah, okay | |
| 15:37:10 | cdent | cool | |
| 15:38:07 | imacdonn | and I'm not tripleo, FWIW | |
| 15:38:26 | dansmith | ...said with beaming pride :) | |
| 15:38:29 | imacdonn | heh | |
| 15:38:33 | cdent | so is the other way around the problem to catch the reset by peer and reopen? because expecting an idle web service to make rpc heartbeats is ... odd | |
| 15:38:44 | cdent | imacdonn: sorry, like I said, it's been hard to keep track of the details on this one | |
| 15:38:45 | dansmith | yeah | |
| 15:38:47 | dansmith | so that's the other thing, | |
| 15:38:57 | dansmith | why is the connection not just immediately re-youknow-connected? | |
| 15:39:03 | dansmith | because that would be a lot more sane | |
| 15:39:10 | imacdonn | the whole point of heartbeats is to keep connections alive when there's no traffic | |
| 15:39:25 | imacdonn | for example, a firewall idle timeout could bite you otherwise | |
| 15:39:46 | cdent | sure, but a web server process may not even exist in some deployment scenarios when there are no requests | |
| 15:40:09 | cdent | who is going to send the heartbeat? | |
| 15:40:16 | dansmith | imacdonn: but only really for things where you might be expecting to hear something from the other side | |
| 15:40:30 | dansmith | imacdonn: the api server receives no messages other than replies, | |
| 15:40:36 | imacdonn | the wsgi app container, I guess? and if that doesn't exist, there's no problem ... a new one would start a new connection | |
| 15:40:46 | dansmith | so there's really no reason for a connection except when it's processing a reuest | |
| 15:41:41 | imacdonn | I guess I might have assumed that setting up a new connection incurs some expense ... so it's better to keep it open once you have it | |
| 15:42:13 | dansmith | open connections that aren't used are pretty expensive in rabbit terms ;) | |
| 15:42:14 | imacdonn | kindof like the REST concept | |
| 15:43:59 | imacdonn | re. "things where you might be expecting to hear something from the other side" - I wonder if there are cases where that could break - maybe an async RPC call that takes a long time to complete ? | |
| 15:44:25 | dansmith | imacdonn: in that case you're processing a request still | |
| 15:44:58 | dansmith | what I don't understand is, if the wsgi container has spawned a process to do a thing, and has not killed it off during times of inactivity, what is it doing that prevents the normal threading background stuff from running like any other process? | |
| 15:45:33 | dansmith | unless we've tickled something in the python threading library before patching or whatever, but if that's the problem that's the kind of thing we need to know | |
| 15:46:03 | dansmith | I'd be interested in what the differences are between you and everyone else.. same exact (unmodified) nova, oslo, et al code? same version of deps, etc? | |
| 15:46:19 | dansmith | these are the things worth chasing, imho | |
| 15:46:41 | imacdonn | do we know of anyone running stein nova-api on wsgi successfully ? | |
| 15:47:24 | imacdonn | cos I only know of the other guys who are seeing the same issue as me | |
| 15:47:36 | cdent | it's always running on wsgi, whether you're using nova-api script or using mod_wsgi or uwsgi. the distinguishing factor is standalone eventlet server versus everything else | |
| 15:48:09 | imacdonn | dansmith: see also https://bugs.launchpad.net/nova/+bug/1825584/comments/7 - I haven't really tried to digest that yet | |