Disconnected clients fill rpcworkqueue, block json-rpc #10436

Describe the issue

Requests are kept waiting even after clients disconnect. The queue can go full, blocking further json-rpc calls. Usually not a problem, except when long polling. Long polling getblocktemplate will hold a slot until another block has been received.

Can you reliably reproduce the issue?

If so, please list the steps to reproduce below:

1. Long poll getblocktemplate
2. Disconnect
3. Repeat 1+2 until requests fail

Expected behaviour

Working json-rpc.

Suggestion: throw away requests from disconnected clients. Maybe reserve some slots for quick (non long polling) requests so long polling requests can't as easily lock up json-rpc.

Actual behaviour

After the long polling client reconnects a few times, there is the following error:

WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

What version of bitcoin-core are you using?

0.14.1

are you sure you're actually disconnecting? some http libraries are broken and leave the connections up.

Yes, I'm sure I'm disconnecting.

I have a simple regtest network set up with 2 nodes. I have bit1 and bit2 shell commands to run bitcoin-cli for the two separate nodes.

First I run "bit1 getblocktemplate" to get a longpollid. Then I construct a simple longpoll request ala:

POST / HTTP/1.0
Host: localhost
Authorization: Basic BASE64-AUTH-HERE
Content-Length: 131

{"method":"getblocktemplate","params":[{"longpollid":"3bb959dad7927110f61d66b60cbf185513adf3b7e0a1879d672e82d2f41660f252"}],"id":1}

Now in one window I telnet to the rpc port of the bit1 node and paste in this request. In another window I do "killall telnet" to disconnect. I repeat this a few times to fill all the slots.

At this point all the telnet processes are dead. There is no active connection to the RPC port.

Now I do "bit1 getinfo" in one window. It hangs. No RPC works with node 1 at this point. Then in another window I do "bit2 generate 1". Now the "bit1 getinfo" completes. As soon as node 1 receives a block from node 2 the longpoll is triggered and all slots are freed up. RPC works again.

I've created more than 100 longpolls, but couldn't reproduce.

This was the diff I used:

diff --git a/test/functional/mining_getblocktemplate_longpoll.py b/test/functional/mining_getblocktemplate_longpoll.py
index 6d0b241e57..eacf71cf22 100755
--- a/test/functional/mining_getblocktemplate_longpoll.py
+++ b/test/functional/mining_getblocktemplate_longpoll.py
@@ -9,20 +9,19 @@ from decimal import Decimal
 from test_framework.test_framework import BitcoinTestFramework
 from test_framework.util import get_rpc_proxy, random_transaction
 
+from multiprocessing import Process
+
 import threading
 
-class LongpollThread(threading.Thread):
-    def __init__(self, node):
-        threading.Thread.__init__(self)
+def LongpollThread(node):
         # query current longpollid
         template = node.getblocktemplate({'rules': ['segwit']})
-        self.longpollid = template['longpollid']
+        longpollid = template['longpollid']
         # create a new connection to the node, we can't use the same
         # connection from two threads
-        self.node = get_rpc_proxy(node.url, 1, timeout=600, coveragedir=node.coverage_dir)
+        node = get_rpc_proxy(node.url, 1, timeout=600, coveragedir=node.coverage_dir)
 
-    def run(self):
-        self.node.getblocktemplate({'longpollid': self.longpollid, 'rules': ['segwit']})
+        node.getblocktemplate({'longpollid': longpollid, 'rules': ['segwit']})
 
 class GetBlockTemplateLPTest(BitcoinTestFramework):
     def set_test_params(self):
@@ -42,8 +41,13 @@ class GetBlockTemplateLPTest(BitcoinTestFramework):
         assert template2['longpollid'] == longpollid
 
         # Test 1: test that the longpolling wait if we do nothing
-        thr = LongpollThread(self.nodes[0])
-        thr.start()
+        for i in range(1000):
+         print(i)
+         p = Process(target=LongpollThread,args=(self.nodes[0],))
+         p.start()
+         import time
+         time.sleep(1)
+         p.terminate()
         # check that thread still lives
         thr.join(5)  # wait 5 seconds or until thread exits
         assert thr.is_alive()

Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce?