We had an instance of a single RPC request blocking for a while. I suspect it was because the RPC client socket got closed ungracefully and the RPC request's reply was just blocking there until the server eventually dropped the socket. (probably a network related problem)
The problem is that while the request was blocking, other requests got queued in the RPC queue for processing. But those requests failed to be processed before the other clients timeout, because of the first RPC request blocking all processing until the socket timeout.
Some of those requests were not idempotent (like sendtoaddress). The caller of this request, not seeing any response from in a certain timeout, assumed the request failed, however this request got eventually processed once the RPC queue started getting processed again. The caller, assuming sendtoaddress failed, made another request, essentially sending twice.
rpcservertimeout is only referring to the timeout it takes to read the HTTP request, but HTTP requests get read immediately and queued in the RPC queue. Once queued, bitcoin core makes no check after dequeuing the request that it got timed out.
I was thinking, the best way to solve this is to have a timeout at RPC protocol level, that clients can specify in the RPC request. When bitcoin core unqueue the request from the rpc queue, it can check if the request expired and not running it if that's the case.
What do you think about such solution? I agree this is to solve a corner case issue, but nevertheless this issue could have resulted in loss of funds, and I need to find a way so it does not happen again.
