libevent http fixes

laanwj commented at 1:59 pm on September 18, 2015: member

Fix a few rough edges discovered in use of the new libevent http server:;

init: Ignore SIGPIPE

Ignore SIGPIPE on all non-win32 OSes, otherwise an unexpectedly disconnecting RPC client will terminate the application. This problem was introduced with the libhttp-based RPC server.

Fixes #6660.

http: Disable libevent debug logging, if not explicitly enabled

Add a option “-debug=libevent” to enable libevent debugging for troubleshooting. Libevent logging is redirected to our own log.

Addresses stdout spew in #6655.

rpc: Split option -rpctimeout into -rpcservertimeout and -rpcclienttimeout

The two timeouts for the server and client, are essentially different:

In the case of the server it should be a lower value to avoid clients clogging up connection slots
In the case of the client it should be a high value to accomedate slow responses from the server, for example for slow queries or when the lock is contended

Split the options into -rpcservertimeout and -rpcclienttimeout with respective defaults of 30 and 900.

Should fix, or make it possible to fix timeout issues in #6655.

init: Ignore SIGPIPE

Ignore SIGPIPE on all non-win32 OSes, otherwise an unexpectedly disconnecting
RPC client will terminate the application. This problem was introduced
with the libhttp-based RPC server.

Fixes #6660.

5ce43da03d

laanwj added the label RPC on Sep 18, 2015

jonasschnelli commented at 3:02 pm on September 18, 2015: contributor

utACK (plans to test this soon). Not sure about the insides of getblocktemplate, but hows does the rpctimeout affect the long poll?

laanwj commented at 3:13 pm on September 18, 2015: member

Not sure about the insides of getblocktemplate, but hows does the rpctimeout affect the long poll?

On the server side it doesn’t affect that - it doesn’t spend time waiting for input from the client. On the client side, the request will time out if the reply takes longer than rpcclienttimeout.

morcos commented at 4:37 pm on September 18, 2015: member

Unfortunately I’m 0/2 on testing these. I couldn’t recreate the SIGPIPE issue, in master or here, so thats’s a start. But I still get what looks like a timeout from getblocktemplate_longpoll.py and I still get errors spewed to stdout when I don’t have an IPv6 address.

EDIT: as per IRC:

SIGPIPE appears to be an OS X issue, which is why I couldn’t recreate.
older versions of libevent don’t support turning off the function call to turn off debugging

laanwj commented at 5:04 pm on September 18, 2015: member

From IRC:

0<morcos> regarding the rpc timeouts..  before 6695, you could fix the problem by starting the bitcoind's in the python tests with -rpctimeout=1500 (or whatever)
1<morcos> doesn't that imply that somehow it was the timeout on the server end that was the problem?

Looks like the issue is not in bitcoin-cli but in the python http functionality that cannot cope with disconnects between requests (as said by @sdaftuar ). Which would be strange, but I’ll take a look at it.

laanwj commented at 5:07 pm on September 18, 2015: member

The rpcbind.py test hang in #6655 is caused by bitcoin-cli being unable to connect to IPv6 IPs. Not sure why. Also not sure whether this affects all libevent versions. But I can reproduce it with libevent 0x02001500 on Ubuntu 14.04 (which is relatively ancient).

jonasschnelli commented at 6:38 pm on September 18, 2015: contributor

SIGPIPE appears to be an OS X issue, which is why I couldn’t recreate.

I could reproduce it on debian as well. Use a bitcoin-cli command like getblocktemplate or getrawmempool and enter ctrl-C short after sending-off the command. Very hard to reproduce (requires a few tries). With a script it would probably easy.

The SIGPIPE gets emitted when bitcoind starts to write to a closed socket (ctrl-c will terminate bitcoin-cli which ends up in writing to a closed socket on the server side).

morcos commented at 10:12 pm on September 18, 2015: member

@laanwj I know I said I didn’t care, but I tracked down the silly warning messages. We need to call event_set_log_callback. I tried it and it works.

dcousens commented at 0:53 am on September 19, 2015: contributor

concept ACK

laanwj commented at 3:15 am on September 19, 2015: member

@morcos Ah, that’d allow redirecting the messages to our own log. That makes sense in addition to what is done here.

Edit: done, the option is -debug=libevent now

laanwj force-pushed on Sep 19, 2015

laanwj commented at 12:15 pm on September 19, 2015: member

The Python BadStatusLine on disconnect issue sounds like https://bugs.python.org/issue3566 “httplib persistent connections violate MUST in RFC2616 sec 8.1.4.”

This was fixed in Python 3.5.

I’ve pushed a commit that should work around it.

laanwj force-pushed on Sep 19, 2015

morcos commented at 3:55 pm on September 20, 2015: member

ACK (modulo missing newlines)

http: Disable libevent debug logging, if not explicitly enabled

Add a option "-debug=libevent" to enable libevent debugging for troubleshooting.
Libevent logging is redirected to our own log.

8b2d6edaa9

laanwj force-pushed on Sep 21, 2015

laanwj commented at 11:40 am on September 21, 2015: member

Fixed the newline issue (thanks @morcos), removed a spurious change in util.py

rpc: Split option -rpctimeout into -rpcservertimeout and -rpcclienttimeout

The two timeouts for the server and client, are essentially different:

- In the case of the server it should be a lower value to avoid clients
clogging up connection slots

- In the case of the client it should be a high value to accomedate slow
  responses from the server, for example for slow queries or when the
  lock is contended

Split the options into `-rpcservertimeout` and `-rpcclienttimeout` with
respective defaults of 30 and 900.

2190ea6c4e

Make RPC tests cope with server-side timeout between requests

Python's httplib does not graciously handle disconnections from the http server, resulting in BadStatusLine errors.
See https://bugs.python.org/issue3566 "httplib persistent connections violate MUST in RFC2616 sec 8.1.4."

This was fixed in Python 3.5.

Work around it for now.

ddf98d1d84

laanwj force-pushed on Sep 21, 2015

laanwj merged this on Sep 21, 2015

laanwj closed this on Sep 21, 2015

laanwj referenced this in commit 5e21388c48 on Sep 21, 2015

zkbot referenced this in commit b0afc4ba1d on Mar 22, 2017

zkbot referenced this in commit f9f48667be on Mar 25, 2017

MarcoFalke locked this on Sep 8, 2021

libevent http fixes #6695

init: Ignore SIGPIPE

http: Disable libevent debug logging, if not explicitly enabled

rpc: Split option -rpctimeout into -rpcservertimeout and -rpcclienttimeout