| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 14:21:01 | sean-k-mooney | which is your fix for bug #1809095 | |
| 14:21:02 | openstack | bug 1809095 in OpenStack Compute (nova) "Wrong representor port was unplugged from OVS during cold migration" [Medium,In progress] https://launchpad.net/bugs/1809095 - Assigned to Adrian Chiris (adrian.chiris) | |
| 14:32:32 | mriedem | btw, i remember mtreinish trying to get glance-api moved to wsgi in pike and it got hung up on threading stuff with eventlet, hence https://review.opendev.org/#/c/549743/ | |
| 14:32:57 | dansmith | yup | |
| 14:34:52 | mriedem | oh and https://review.opendev.org/#/c/531498/ | |
| 14:35:04 | mriedem | ^ taskflow evenlet async stuff + wsgi + glance-api | |
| 14:36:21 | mtreinish | yep, but I couldn't get anyone to ever really review that or explain how to use the async glance APIs. And there is no dsvm coverage for any of it | |
| 14:37:55 | mriedem | mtreinish: for context we're going over similar in nova-api https://review.opendev.org/#/c/650172/ | |
| 14:41:14 | cdent | I reckon the root of the problems in both glance and nova is that we are trying to async workings in an environment (an HTTP API) where it doesn't make sense. The traditional python solution to this problem is to dump into an offboard async task queue (celery and the like). The scatter/gather thing is not quite in the same boat as the glance stuff (which is more amenable to the celery thing perhaps) | |
| 14:46:13 | mtreinish | cdent: yeah, that was the best way to fix it, to just do the async bits in another service | |
| 14:46:37 | mtreinish | but I assumed just launching another process or thread to handle it should work as a quick workaround | |
| 14:46:56 | cdent | it ought to | |
| 14:47:24 | cdent | but I would guess that zombie blocked processes would present the same concern as blocked threads | |
| 14:47:36 | cdent | which might suggest: that's the way unix goes, deal... | |
| 14:53:11 | sean-k-mooney | i mean we could just not set a limit on the worker trhead in the tread pool or make it a config option | |
| 14:53:42 | sean-k-mooney | python will grow and srink the thread pool automatically if you dont specify the number of workers | |
| 14:53:53 | dansmith | that's the problem, these are a threadpool, which means they're re-used and capped at some max number, so enough api requests come in during a cell down event, and you immediately exhaust the pool of workers until things time out | |
| 14:54:15 | dansmith | sean-k-mooney: only up to N*CPUs I think I read | |
| 14:54:27 | dansmith | where N is like 3 or something | |
| 14:54:53 | sean-k-mooney | dansmith: maybe not sure | |
| 14:55:09 | sean-k-mooney | but we could leave it as a config option and defult to -1 mean let python figure it out | |
| 14:55:26 | sean-k-mooney | or set it to X if you care and have mesured for your system | |
| 14:55:34 | mriedem | https://docs.python.org/3.6/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor | |
| 14:55:40 | mriedem | "If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor." | |
| 14:56:25 | dansmith | right, so with 5 cells, that becomes $CPUs | |
| 14:56:26 | dansmith | and everyone has at least two | |
| 14:57:15 | cdent | when calculating the number of threads that will be around, another thing to keep in mind is mod_wsgi will often run with multiple process and multiple threads in those processes. what is the pool associated with? | |
| 14:57:45 | cdent | every request or thread or process? | |
| 14:58:43 | dansmith | idk | |
| 14:58:44 | cdent | process | |
| 14:59:40 | sean-k-mooney | so we could revert to createing the thread pool per call and set it to one thread per cell | |
| 15:00:09 | sean-k-mooney | but the issue with that is a its expensive to spawn treads and b we nolonger have a limit on them | |
| 15:00:47 | sean-k-mooney | but we will not have the issue of a potention exausting of the thread pool | |
| 15:01:30 | sean-k-mooney | cdent: the pool i a module global in the current patch so it would associated with the process | |
| 15:03:13 | cdent | sean-k-mooney: [t 6MVa] | |
| 15:03:13 | purplerbot | <cdent> process [2019-05-16 14:58:44.001687] [n 6MVa] | |
| 15:03:55 | sean-k-mooney | ? | |
| 15:04:23 | cdent | [l 6MVa] | |
| 15:04:23 | purplerbot | http://p.anticdent.org/6MVa | |
| 15:04:39 | cdent | in other words, I figured it out by looking at the code a few minutes ago | |
| 15:04:50 | sean-k-mooney | oh right :) | |
| 15:06:13 | mriedem | efried: on that ironic job failure, it looks like that job has probably been broken since stein when we started reporting capability traits for nodes http://logs.openstack.org/32/634832/29/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/fba9197/controller/logs/devstacklog.txt.gz#_2019-05-16_03_28_21_590 | |
| 15:06:21 | jangutter | cdent: I really thought you were mistyping 6 million volt-ampere. | |
| 15:06:36 | mriedem | if i'm reading that correctly it's expecting only the CUSTOM_GOLD trait | |
| 15:06:40 | sean-k-mooney | dansmith: what was the behavior that motivated the scater gather code in the first place to you rememebr? was it to deal with down cells or was it a performce imporvoment? | |
| 15:07:12 | dansmith | sean-k-mooney: it's to avoid a sequential walk of all the cell databases, serializing their latencies | |
| 15:07:28 | sean-k-mooney | ah ok so it was to addres perfromace ok | |
| 15:07:51 | cdent | Are there numbers on those latencies? | |
| 15:07:57 | dansmith | sean-k-mooney: cern has 70 cells, some of which are remote | |
| 15:08:06 | cdent | ah. the remote part | |
| 15:08:08 | dansmith | as in, in another country | |
| 15:08:18 | efried | mriedem: looks like the fix is just to take out the square brackets, nah? | |
| 15:08:40 | sean-k-mooney | ya even if they wer all in the same data center it could be a proble as the number of cell grow | |
| 15:09:18 | dansmith | sean-k-mooney: especially if one in the middle is timing out, and we wait for that to complete before moving to the next one and get its response | |
| 15:09:52 | dansmith | and everyone has two of these at minimum so moving to sequential means 2N latency, which is twice as bad, regardless of what N is | |
| 15:10:00 | sean-k-mooney | i was just wondering if the initall approch to solving the problem was patinting us in to a corner that could be avoid by revisiting the originial usecase cool | |
| 15:11:04 | sean-k-mooney | effectivly waht we want to do is queue up a bunch of asyc request and wait for them to complete | |
| 15:11:24 | dansmith | ..a thing for which eventlet is ideal | |
| 15:11:24 | sean-k-mooney | eventlet and thread pools both allwo that with different tradeoffs | |
| 15:11:50 | sean-k-mooney | the issue isnt the use of eventlets its self | |
| 15:12:38 | sean-k-mooney | its that mod_wsgi and uwsgi both have exposed diffreent bugs caused by using eventlets with them | |
| 15:15:06 | sean-k-mooney | does wsgi provide a way to run request like this in the background that we could leverag and fall back to eventlest when not runnign under wsgi? | |
| 15:16:03 | sean-k-mooney | google found me https://pypi.org/project/uwsgi-tasks/ but no idea if that would help | |
| 15:18:52 | cdent | that could work for the uwsgi situation, but not mod_wsgi, presumably | |
| 15:19:18 | sean-k-mooney | i dont know i personally would prefer to try melwitt's approch first and maybe ask cern to try it and get feed back | |
| 15:20:38 | cdent | sean-k-mooney: you mean the current code under review? that seems reasonable. I guess the issue is that we don't know if the problem is a problem until it gets hit... | |
| 15:20:56 | sean-k-mooney | cdent: right | |
| 15:21:06 | cdent | mriedem seemed to think it could be caused with some effort in devstack | |
| 15:21:12 | sean-k-mooney | and yes the current code under review | |
| 15:21:57 | efried | mriedem: I'm going to propose that fix unless you've arleady got it in the pipe | |
| 15:23:43 | mriedem | efried: i don't, i'm trying to recreate with bash locally | |
| 15:23:56 | efried | ack | |
| 15:24:17 | mriedem | cdent: i know how to manually test multiple cells in a single node devstack and make one or more down, but that doesn't mean i know how to recreate the issue | |
| 15:24:38 | mriedem | we haven't hit this problem in the gate as far as i know with nova-api under wsgi + 2 cells | |
| 15:24:54 | mriedem | we also now have the nova-multi-cell job which is effectively 3 cells | |
| 15:25:00 | mriedem | cell0/1/2 | |
| 15:26:28 | melwitt | imacdonn: do you happen to know why/how the uwsgi/mod_wsgi problem doesn't seem to show itself in the gate? | |
| 15:26:29 | cdent | as I remmeber (which may be wrong) the problem with eventlet needs some work, a long pause, some attempted work to show up. And the expected problem with threads requires connections to databases (or other I/O) that blocks weirdly and doesn't time out in natural ways | |
| 15:26:30 | mriedem | efried: removing the brackets gives me this: | |
| 15:26:30 | mriedem | jq: error (at <stdin>:1): array (["COMPUTE_N...) and string ("CUSTOM_GOLD") cannot have their containment checked | |
| 15:26:40 | cdent | melwitt: I reckon it the lack of pause | |
| 15:26:49 | melwitt | aye | |
| 15:27:01 | efried | mriedem: ugh, I guess it would help me to know wtf jq is. | |
| 15:27:08 | mriedem | efried: json parser | |
| 15:27:24 | mriedem | efried: i considered just re-writing this with osc-placement and grep | |
| 15:27:42 | efried | seems like that would be better. | |
| 15:27:47 | mriedem | openstack --os-placement-api-version 1.6 resource provider trait list $provider_id | grep $trait | |
| 15:27:55 | mriedem | it would definitely be easier to f'ing read | |
| 15:27:59 | efried | agreed | |
| 15:28:03 | mriedem | i'll push that up | |
| 15:28:05 | efried | ight | |
| 15:28:30 | efried | mriedem: I made a story | |
| 15:28:31 | mriedem | this would also suggest all ironic jobs would be busted since stein and if that were the case i'd think we would have heard about it by now... | |
| 15:28:36 | efried | mriedem: https://storyboard.openstack.org/#!/story/2005725 | |
| 15:28:42 | efried | you would think, yah | |
| 15:29:33 | mriedem | cdent: melwitt: dansmith: you don't think it's a simple as adding an https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.sleep between calls in the scatter gather do you? | |
| 15:30:30 | imacdonn | melwitt: I don't know enough about what happens in the gate ... but the problem manifests when nova-api does something that requires an API call, then goes quiet for more than a minute or so, then gets another request that requires RPC | |
| 15:30:36 | dansmith | the problem is that we're not monkeypatched properly such that background threads aren't being run and thus the heartbeat for the rabbit connection isn't working | |
| 15:31:20 | dansmith | which seems like it should manifest in the gate easily if it's fundamental to how we're doing service setup, and which also means other things shouldn't really be working | |
| 15:31:42 | dansmith | so I'm kinda suspicious of the whole thing, and don't really want to "solve" it by changing our whole threading model for one service | |