bitcoind crashes if RPC is spammed #11322

bitcoind v0.15.0rc3 (signed binaries), Arch Linux rolling amd64 with 16GB RAM

Upon attempting to connect to the JSON-RPC mechanism using the Python requests library, bitcoind dies unexpectedly;

terminate called without an active exception
Aborted (core dumped)

debug.log

2017-09-13 16:50:28 libevent: evhttp_read_cb: illegal connection state 7

(no other relevant entries).

I cannot reproduce this at the moment, if I manage it I will update this issue.

edit: Reproduction steps as follows:

while True:
    r = requests.get("http://localhost:8332", data=json.dumps({}))

Any payload passed seems to trigger it, passing no payload does not trigger the crash.

Obviously, spamming it in a while loop is an extreme case, but I'm curious as to why this happens. It is pretty much immediate.

Okay, narrowing this down a bit more, running with -debug=http;

GET requests with a valid endpoint (e.g. /rest/chaininfo.json) do not appear to cause issues. I've tried 20,000 requests as quickly as python can throw them out from two request threads.

GET requests on / end up in a number of Received a GET request for / from [::1]:port followed by the libevent illegal connection state. Sometimes it's as few as 5.

Occasionally a segfault is caused, other times the 'terminate called without an active exception' error.

0.14.2 has the same issue.

Further testing: Hitting any endpoint that is registered with a handler causes the problem. "/", "/rest/tx/" if enabled, "/wallet/", etc, but not "/wallet" or "/arbitrarystring".

The response "JSONRPC server handles only POST requests" is getting through to the client until the crash.

I've dug around a bit and been unable to find the problem. The work queue is not filling up; adding new items is fine; possibly a libevent issue?

possibly a libevent issue

Don't know but I doubt. I'll try to reproduce.

Bit more digging:

$ nc -l -p 18334
GET / HTTP/1.1                           
Host: localhost:18334                    
User-Agent: python-requests/2.18.4       
Accept-Encoding: gzip, deflate           
Accept: */*                              
Connection: keep-alive                   
Content-Length: 2                        

{}

Tweaking cURL to give the same output:

for x in {1..2000}; do curl -s -X GET -H "User-Agent: python-requests/2.18.4" -H "Accept-Encoding: gzip, deflate" -H "Accept: */*" -H "Connection: keep-alive" -H "Content-Type:" --data "{}" http://localhost:18332/ > /dev/null; done

(the output of nc is identical given this modification, checked with diff).

Can't reproduce using cURL. It does hit the server much more slowly, though.

I've been up to my eyeballs in libevent lately, so I'll take this one.

Edit: Not to say anyone should stop investigating, of course!

I've tracked down the root cause of this and am working on a proper fix.

Cannot reproduce with fa8a0639f7b0ce04030b72b4d5be4f0aa36fc5cb

python 3

import requests
import json

while True:
        print("get")
        r = requests.get("http://127.0.0.1:9332", data=json.dumps({}))

@theuni Missing lock? :-)

@theuni was this actually an issue?

Hi @theuni. Did you find any solution to this? Thanks.

@theuni ping. Can you share your findings?

Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce?