RPC-related hangs on shutdown #2739

I have been debugging some RPC-related shutdown hangs in bitcoin-qt on windows. The behavior is worse in 0.8.2, but there were problems before 0.8.2 as well.

1. In ServiceConnection, there is a loop that processes multiple requests from the same connection as long as "Connection: close" is not specified in the request. During application shutdown, nothing stops that processing, so a client can continue to make requests indefinitely using an already established connection. This is new in 0.8.2 -- in 0.8.1 there was a check for fShutdown before processing each request.
2. If an RPC connection is blocked, for example in ReadHTTPRequestLine, shutdown can be delayed indefinitely while waiting for input from the client. That can happen due to a misbehaving client, but it can also happen with a well-behaved client that makes multiple requests on the same connection. After the first request and reply, we will reach ReadHTTPRequestLine and wait until the client has another request to make. This is not new in 0.8.2.

For now I have worked around the problems by forcing ServiceConnection to close the connection after the first request in all cases. There is still a possibility of hangs if the client fails to send the first request promptly.

The thread code plays nicely with boost::asio.

I would guess any solution that keeps the calls to std::getline() will be problematic, because they don't get interrupted by boost::thread::interrupt.

As of 0.10, command line argument -rpckeepalive=0 can be used to disable the keep-alive behavior. The underlying issue will only be solved by switching to another HTTP server. e.g. #5677.

HTTP server has been switched to libevent, which properly times out connections, this issue should be solved now.