| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 14:57:45 | cdent | every request or thread or process? | |
| 14:58:43 | dansmith | idk | |
| 14:58:44 | cdent | process | |
| 14:59:40 | sean-k-mooney | so we could revert to createing the thread pool per call and set it to one thread per cell | |
| 15:00:09 | sean-k-mooney | but the issue with that is a its expensive to spawn treads and b we nolonger have a limit on them | |
| 15:00:47 | sean-k-mooney | but we will not have the issue of a potention exausting of the thread pool | |
| 15:01:30 | sean-k-mooney | cdent: the pool i a module global in the current patch so it would associated with the process | |
| 15:03:13 | purplerbot | <cdent> process [2019-05-16 14:58:44.001687] [n 6MVa] | |
| 15:03:13 | cdent | sean-k-mooney: [t 6MVa] | |
| 15:03:55 | sean-k-mooney | ? | |
| 15:04:23 | purplerbot | http://p.anticdent.org/6MVa | |
| 15:04:23 | cdent | [l 6MVa] | |
| 15:04:39 | cdent | in other words, I figured it out by looking at the code a few minutes ago | |
| 15:04:50 | sean-k-mooney | oh right :) | |
| 15:06:13 | mriedem | efried: on that ironic job failure, it looks like that job has probably been broken since stein when we started reporting capability traits for nodes http://logs.openstack.org/32/634832/29/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/fba9197/controller/logs/devstacklog.txt.gz#_2019-05-16_03_28_21_590 | |
| 15:06:21 | jangutter | cdent: I really thought you were mistyping 6 million volt-ampere. | |
| 15:06:36 | mriedem | if i'm reading that correctly it's expecting only the CUSTOM_GOLD trait | |
| 15:06:40 | sean-k-mooney | dansmith: what was the behavior that motivated the scater gather code in the first place to you rememebr? was it to deal with down cells or was it a performce imporvoment? | |
| 15:07:12 | dansmith | sean-k-mooney: it's to avoid a sequential walk of all the cell databases, serializing their latencies | |
| 15:07:28 | sean-k-mooney | ah ok so it was to addres perfromace ok | |
| 15:07:51 | cdent | Are there numbers on those latencies? | |
| 15:07:57 | dansmith | sean-k-mooney: cern has 70 cells, some of which are remote | |
| 15:08:06 | cdent | ah. the remote part | |
| 15:08:08 | dansmith | as in, in another country | |
| 15:08:18 | efried | mriedem: looks like the fix is just to take out the square brackets, nah? | |
| 15:08:40 | sean-k-mooney | ya even if they wer all in the same data center it could be a proble as the number of cell grow | |
| 15:09:18 | dansmith | sean-k-mooney: especially if one in the middle is timing out, and we wait for that to complete before moving to the next one and get its response | |
| 15:09:52 | dansmith | and everyone has two of these at minimum so moving to sequential means 2N latency, which is twice as bad, regardless of what N is | |
| 15:10:00 | sean-k-mooney | i was just wondering if the initall approch to solving the problem was patinting us in to a corner that could be avoid by revisiting the originial usecase cool | |
| 15:11:04 | sean-k-mooney | effectivly waht we want to do is queue up a bunch of asyc request and wait for them to complete | |
| 15:11:24 | sean-k-mooney | eventlet and thread pools both allwo that with different tradeoffs | |
| 15:11:24 | dansmith | ..a thing for which eventlet is ideal | |
| 15:11:50 | sean-k-mooney | the issue isnt the use of eventlets its self | |
| 15:12:38 | sean-k-mooney | its that mod_wsgi and uwsgi both have exposed diffreent bugs caused by using eventlets with them | |
| 15:15:06 | sean-k-mooney | does wsgi provide a way to run request like this in the background that we could leverag and fall back to eventlest when not runnign under wsgi? | |
| 15:16:03 | sean-k-mooney | google found me https://pypi.org/project/uwsgi-tasks/ but no idea if that would help | |
| 15:18:52 | cdent | that could work for the uwsgi situation, but not mod_wsgi, presumably | |
| 15:19:18 | sean-k-mooney | i dont know i personally would prefer to try melwitt's approch first and maybe ask cern to try it and get feed back | |
| 15:20:38 | cdent | sean-k-mooney: you mean the current code under review? that seems reasonable. I guess the issue is that we don't know if the problem is a problem until it gets hit... | |
| 15:20:56 | sean-k-mooney | cdent: right | |
| 15:21:06 | cdent | mriedem seemed to think it could be caused with some effort in devstack | |
| 15:21:12 | sean-k-mooney | and yes the current code under review | |
| 15:21:57 | efried | mriedem: I'm going to propose that fix unless you've arleady got it in the pipe | |
| 15:23:43 | mriedem | efried: i don't, i'm trying to recreate with bash locally | |
| 15:23:56 | efried | ack | |
| 15:24:17 | mriedem | cdent: i know how to manually test multiple cells in a single node devstack and make one or more down, but that doesn't mean i know how to recreate the issue | |
| 15:24:38 | mriedem | we haven't hit this problem in the gate as far as i know with nova-api under wsgi + 2 cells | |
| 15:24:54 | mriedem | we also now have the nova-multi-cell job which is effectively 3 cells | |
| 15:25:00 | mriedem | cell0/1/2 | |
| 15:26:28 | melwitt | imacdonn: do you happen to know why/how the uwsgi/mod_wsgi problem doesn't seem to show itself in the gate? | |
| 15:26:29 | cdent | as I remmeber (which may be wrong) the problem with eventlet needs some work, a long pause, some attempted work to show up. And the expected problem with threads requires connections to databases (or other I/O) that blocks weirdly and doesn't time out in natural ways | |
| 15:26:30 | mriedem | jq: error (at <stdin>:1): array (["COMPUTE_N...) and string ("CUSTOM_GOLD") cannot have their containment checked | |
| 15:26:30 | mriedem | efried: removing the brackets gives me this: | |
| 15:26:40 | cdent | melwitt: I reckon it the lack of pause | |
| 15:26:49 | melwitt | aye | |
| 15:27:01 | efried | mriedem: ugh, I guess it would help me to know wtf jq is. | |
| 15:27:08 | mriedem | efried: json parser | |
| 15:27:24 | mriedem | efried: i considered just re-writing this with osc-placement and grep | |
| 15:27:42 | efried | seems like that would be better. | |
| 15:27:47 | mriedem | openstack --os-placement-api-version 1.6 resource provider trait list $provider_id | grep $trait | |
| 15:27:55 | mriedem | it would definitely be easier to f'ing read | |
| 15:27:59 | efried | agreed | |
| 15:28:03 | mriedem | i'll push that up | |
| 15:28:05 | efried | ight | |
| 15:28:30 | efried | mriedem: I made a story | |
| 15:28:31 | mriedem | this would also suggest all ironic jobs would be busted since stein and if that were the case i'd think we would have heard about it by now... | |
| 15:28:36 | efried | mriedem: https://storyboard.openstack.org/#!/story/2005725 | |
| 15:28:42 | efried | you would think, yah | |
| 15:29:33 | mriedem | cdent: melwitt: dansmith: you don't think it's a simple as adding an https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.sleep between calls in the scatter gather do you? | |
| 15:30:30 | imacdonn | melwitt: I don't know enough about what happens in the gate ... but the problem manifests when nova-api does something that requires an API call, then goes quiet for more than a minute or so, then gets another request that requires RPC | |
| 15:30:36 | dansmith | the problem is that we're not monkeypatched properly such that background threads aren't being run and thus the heartbeat for the rabbit connection isn't working | |
| 15:31:20 | dansmith | which seems like it should manifest in the gate easily if it's fundamental to how we're doing service setup, and which also means other things shouldn't really be working | |
| 15:31:42 | dansmith | so I'm kinda suspicious of the whole thing, and don't really want to "solve" it by changing our whole threading model for one service | |
| 15:31:49 | dansmith | but I don't really have time to dig deep, as I said | |
| 15:32:18 | cdent | dansmith: I thought the "more than a minute or so" was a critical factor as it is resulting in mod_wsgi doing $something to the python process that breaks the monkey patching | |
| 15:32:47 | cdent | but I have to admit that finding the sensible thread through all this is hard so I'm not sure if I'm remembering the right stuff | |
| 15:33:05 | mriedem | going quite for a minute or so is something that wouldn't happen in the gate yeah | |
| 15:33:07 | imacdonn | cdent: rabbitmq, in a typical configuration, will drop a connection if it has not seen heartbeats for over a minute | |
| 15:33:13 | mriedem | tempest is all hammer until it's done and then we teardown | |
| 15:33:38 | imacdonn | cdent: that's what leads to a "connection reset by peer" when nova-api wakes up again and tries to use the connection that it thinks is still healthy | |
| 15:33:45 | dansmith | so you think we're just not running hearbeat background tasks if no api requests are coming in? | |
| 15:33:57 | cdent | it does sound a bit that way | |
| 15:34:12 | imacdonn | I have observed that with debug logging | |
| 15:34:47 | dansmith | seems like that should be easy to reproduce in a devstack no? | |
| 15:34:52 | mriedem | could we maybe tickle this in a post test script after tempest is done, sleep a minute or something, and then try to create or do something to a server, like stop it? | |
| 15:35:05 | kashyap | efried: Thanks for representing on #openstack-meeting (still haven't caught up). Stuck in consecutive calls, afraid. | |
| 15:35:20 | dansmith | mriedem: or just a devstack | |
| 15:35:39 | mriedem | yeah..i will leave devstack running for many more minutes before coming back to it to do something | |
| 15:35:42 | dansmith | changing the whole threading model for this seems like a big hammer | |
| 15:36:10 | mriedem | also note that vexxhost is running stein and haven't reported this being a problem... | |
| 15:36:20 | cdent | mriedem: is your devstack using mod_wsgi, by default it won't be | |
| 15:36:30 | mriedem | cdent: oh no i use the default (uwsgi) | |
| 15:36:31 | cdent | and it's not clear the problem is the same uwsgi as mod_wsgi | |
| 15:36:37 | cdent | it _might_ be | |
| 15:36:57 | dansmith | I thought imacdonn said that it repros the same in uwsgi? | |
| 15:36:58 | cdent | but since the only people we've heard of this problem from are tripleo, it's not clear | |
| 15:37:01 | melwitt | I think imacdonn said it behaves the same uwsgi and mod_wsgi, IIRC | |
| 15:37:05 | cdent | ah, okay | |
| 15:37:05 | imacdonn | I have easily reproduced the exact symptoms (including absense of heartbeats with debug logging) with both mod_wsgi and uWSGI | |
| 15:37:10 | cdent | cool | |