| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 17:48:04 | mnaser | dunno how relevant it is but | |
| 17:48:06 | mnaser | a good read none the less | |
| 17:48:50 | dansmith | imacdonn: yeah, I just don't think I agree :) | |
| 17:48:58 | imacdonn | if nova-api isn't going to use persistent AMQP connections, it should gracefully close the connection before it goes to sleep, IMO | |
| 17:49:23 | mnaser | my question is: is there a problem? are your requests forever 500-ing after? | |
| 17:49:24 | dansmith | nova-api doesn't go to sleep | |
| 17:49:26 | melwitt | what's the behavior when it doesn't properly reconnect? nova-api responds with a failure? | |
| 17:49:32 | mnaser | ^^ | |
| 17:49:51 | imacdonn | yeah, I get some sort of 50x error | |
| 17:50:26 | dansmith | presumably because something is getting stuck and not letting the connections close off, which is why I want to know why imacdonn is different from the rest of the world | |
| 17:50:28 | mnaser | if you get a 50x that means you have aa traceback, right? | |
| 17:51:02 | imacdonn | dansmith: hey now .. there was at least one other person experiencing the failures | |
| 17:51:29 | mnaser | should you have a traceback if you're getting a 500? | |
| 17:51:33 | dansmith | imacdonn: I meant your deployment, not you as a person :) | |
| 17:51:59 | imacdonn | well, yeah, but the other person was using a different deployment, and mod_wsgi | |
| 17:52:16 | dansmith | imacdonn: ^ | |
| 17:53:14 | efried | mriedem: Were you going to put up a null nova change depending on https://review.opendev.org/659611 to make sure the ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa job passes? | |
| 17:53:19 | melwitt | I had thought someone else faced this behavior too, but looking at the bug report, they say it reconnects https://bugs.launchpad.net/tripleo/+bug/1827744 | |
| 17:53:20 | openstack | Launchpad bug 1827744 in tripleo "Running eventlet.monkey_patch in nova_api breaks the AMQP heartbeat thread" [High,In progress] - Assigned to Damien Ciabrini (dciabrin) | |
| 17:53:23 | efried | or does ironic run that job too? | |
| 17:53:26 | imacdonn | I'll have to rebuild my "real" environment with uWSGI and reproduce again, if we need more context around the failure to reconnect ... I'd already started down the path of reverting to running nova-api standalone, since there didn't seem to be a path forward | |
| 17:54:02 | openstack | Launchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,New] | |
| 17:54:02 | imacdonn | the "other person" is here: https://bugs.launchpad.net/nova/+bug/1825584/comments/7 | |
| 17:54:07 | dansmith | imacdonn: can you also try disabling rabbit heartbeats on a wsgi api node? just to see if that eliminates the log noise | |
| 17:54:11 | dansmith | imacdonn: even if on the devstack | |
| 17:54:40 | melwitt | imacdonn: that's the bug I linked, I think, and the proposed patch is the stop monkey patching nova-api | |
| 17:54:55 | imacdonn | what do you mean by "log noise"? or are you just asking me to test if it's possible to disable heartbeats at all ? | |
| 17:55:09 | melwitt | but in that bug report, he says that rabbit reconnects | |
| 17:55:25 | imacdonn | that's my bug report, but he commented on it | |
| 17:55:42 | melwitt | no, the bug I linked you | |
| 17:55:45 | dansmith | imacdonn: I mean the log noise where nova-api complains that the connection was reset before reconnecting (in mnaser's case) | |
| 17:55:52 | melwitt | imacdonn: https://bugs.launchpad.net/tripleo/+bug/1827744 | |
| 17:55:54 | dansmith | i.e. this: [10:46:02] <mnaser>2019-05-16 17:38:53.644 3897 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer | |
| 17:55:54 | openstack | Launchpad bug 1827744 in tripleo "Running eventlet.monkey_patch in nova_api breaks the AMQP heartbeat thread" [High,In progress] - Assigned to Damien Ciabrini (dciabrin) | |
| 17:56:35 | sean-k-mooney | actully in the api we dont really care about hearbeat to rabbitmq right | |
| 17:57:08 | dansmith | sean-k-mooney: I've been shouting that for hours, yes:) | |
| 17:57:22 | dansmith | there's really no reason to maintain a persistent connection to rabbit from an idle api worker | |
| 17:57:27 | sean-k-mooney | actullly i was going to say we never actully listen for anyting from the api but i guess we do if we make a blocking call | |
| 17:57:31 | imacdonn | melwitt: ah, I see .. OK ... yeah, maybe it was non-fatal in his case | |
| 17:57:37 | mnaser | I think the connection reset by peer is the real issue, I think the warning is log noise | |
| 17:57:41 | dansmith | sean-k-mooney: only in the reply case, which isn't a problem here | |
| 17:57:47 | sean-k-mooney | ya | |
| 17:58:01 | dansmith | mnaser: the connection reset by peer is because we're not running the heartbeat thread, and thus is expected | |
| 17:58:16 | sean-k-mooney | so even if the connect drops due to in activity in the normal case we just fire and forget | |
| 17:58:19 | dansmith | mnaser: it's the "this is dead let me reconnect immediately" signal, which seems to be working fine for you | |
| 17:58:28 | dansmith | sean-k-mooney: right | |
| 17:58:50 | mnaser | I'm in china and its 2am but for thee last time I'll say it | |
| 17:59:03 | mnaser | if you're getting 500s, it probably threw a traceback, and that'll probably contain something useful | |
| 17:59:14 | dansmith | mnaser: imacdonn said he needs to rebuild his env to get that traceback | |
| 17:59:14 | mnaser | (which will be the real issue) | |
| 17:59:25 | mnaser | aaaha | |
| 17:59:25 | imacdonn | ^ that | |
| 17:59:36 | mnaser | okay, well in that case, good luck | |
| 17:59:41 | mnaser | fwiw let me share the OSA config | |
| 17:59:50 | efried | mriedem: I guess it does run it. It failed. And looking closer, the 'contains' match was working properly. The problem is further up: the inventory is never getting set. | |
| 18:00:36 | dansmith | the other thing I'm interested in is, if you just don't patch, and everything seems to work, is that just because eventlet is being synchronous, or because eventlet primitives are spawning real threads? | |
| 18:01:06 | mnaser | https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/templates/nova-uwsgi.ini.j2 | |
| 18:01:13 | mnaser | this is our uwsgi config | |
| 18:01:57 | melwitt | I'm also wondering about that | |
| 18:02:07 | dansmith | I think that if disabling the heartbeat thread removes the log noise and we reconnect as we would expect for all cases except the unexplained ones, that'd eb my preference as a path forward | |
| 18:02:39 | dansmith | because changing our threading model for some of nova, and maintaining a thread off each wsgi worker just to hold a useless connection to rabbit does not seem like an improvement to me | |
| 18:03:18 | mriedem | efried: ack ok just getting back, haven't looked | |
| 18:04:03 | efried | mriedem: Left comment on https://review.opendev.org/#/c/659611/ | |
| 18:04:10 | imacdonn | dansmith: not sure I like the idea of turning off heartbeat .. that could have side-effects ... like leaving stale connections on the rabbitmq-server side for ever (??) | |
| 18:04:48 | dansmith | imacdonn: show us that it causes negative effects | |
| 18:05:27 | imacdonn | not sure that I can come up with every possible scenario off the top of my head | |
| 18:05:42 | dansmith | the other things we could do are: | |
| 18:06:05 | dansmith | 1. establish and drop connections before/after request | |
| 18:06:36 | efried | mriedem: n-cpu thinks the inventory is being set: http://logs.openstack.org/11/659611/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/9c644a2/controller/logs/screen-n-cpu.txt.gz#_May_16_17_17_55_788562 | |
| 18:06:36 | dansmith | 2. leave heartbeats in place, as they are today so that rabbit will close the stale connections, which seems to be working for most people, | |
| 18:06:43 | sean-k-mooney | if i do | |
| 18:06:46 | sean-k-mooney | >>> def func(): | |
| 18:06:47 | dansmith | 3. change our threading model for part of nova to run a heartbeat thread | |
| 18:06:48 | sean-k-mooney | ... print("test") | |
| 18:06:50 | sean-k-mooney | ... | |
| 18:06:52 | sean-k-mooney | >>> eventlet.greenthread.spawn(func) | |
| 18:06:54 | sean-k-mooney | it never prints anything | |
| 18:06:57 | dansmith | 4. split nova-api into the wsgi request processing side and the persistent bit | |
| 18:07:20 | sean-k-mooney | so if you dont monkeypatch it looks liek spawn does nothing but queue the fucntion to run | |
| 18:07:22 | sean-k-mooney | and never run it | |
| 18:07:25 | dansmith | sean-k-mooney: that makes me wonder how disabling monkeypatching does anything | |
| 18:07:49 | sean-k-mooney | dansmith: as long a s you dont call the scater gater code it should apper to work | |
| 18:08:10 | dansmith | sean-k-mooney: right, but we do that for instance list, and they say that disabling monkeypatching just makes everything work normally, right? | |
| 18:08:13 | sean-k-mooney | that is the only part of the api that actully uses eventlet | |
| 18:08:36 | efried | mriedem: oh, reserved is 1 and never gets set to 0 | |
| 18:08:49 | mriedem | efried: hmm, i wonder if it things the node isn't available | |
| 18:08:52 | mriedem | *thinks | |
| 18:08:54 | sean-k-mooney | dansmith: maybe its different on python 2 | |
| 18:08:58 | mriedem | b/c i think that's when the driver sets reserved=1 | |
| 18:09:04 | sean-k-mooney | i jsut checked with py36 | |
| 18:09:05 | efried | yup | |
| 18:09:17 | mriedem | Node 7c5e9503-1e20-42bf-9ae7-d14cef0f38a9 is not ready for a deployment, reporting resources as reserved for it. Node's provision state is cleaning, power state is power off and maintenance is False. | |
| 18:09:58 | efried | just so | |
| 18:10:50 | mriedem | http://logs.openstack.org/11/659611/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/9c644a2/controller/logs/screen-ir-cond.txt.gz#_May_16_17_28_09_311564 | |
| 18:10:50 | mriedem | yeah so the node is dead | |
| 18:10:55 | imacdonn | sean-k-mooney dansmith: side-note ... my "real" deployment is py2, devstack is py3 ... may be a factor in the failure to reconnect | |
| 18:13:42 | mriedem | yeah i just abandoned the change | |
| 18:15:45 | sean-k-mooney | dansmith: if you save teh greanthread object and then call wait() it runs all the queued greenthreads | |
| 18:16:07 | dansmith | sean-k-mooney: right that's the main loop part we will never do in wsgi mode | |
| 18:16:08 | sean-k-mooney | so if i do gt = eventlet.greenthread.spawn(func) | |