bitcoind crashes if RPC is spammed #11322

1. 
   esotericnonsense commented at 5:01 pm on September 13, 2017: contributor

   bitcoind v0.15.0rc3 (signed binaries), Arch Linux rolling amd64 with 16GB RAM

   Upon attempting to connect to the JSON-RPC mechanism using the Python requests library, bitcoind dies unexpectedly;

   0terminate called without an active exception
   1Aborted (core dumped)
   

   debug.log

   02017-09-13 16:50:28 libevent: evhttp_read_cb: illegal connection state 7
   

   (no other relevant entries).

   I cannot reproduce this at the moment, if I manage it I will update this issue.

   edit: Reproduction steps as follows:

   0while True:
   1    r = requests.get("http://localhost:8332", data=json.dumps({}))
   

   Any payload passed seems to trigger it, passing no payload does not trigger the crash.

   Obviously, spamming it in a while loop is an extreme case, but I’m curious as to why this happens. It is pretty much immediate.

2. esotericnonsense renamed this:
   Unreproducable crash on RPC connection
   bitcoind crashes if RPC is spammed
   on Sep 13, 2017
3. MarcoFalke added the label RPC/REST/ZMQ on Sep 13, 2017
4. 
   esotericnonsense commented at 8:35 pm on September 13, 2017: contributor

   Okay, narrowing this down a bit more, running with -debug=http;

   GET requests with a valid endpoint (e.g. /rest/chaininfo.json) do not appear to cause issues. I’ve tried 20,000 requests as quickly as python can throw them out from two request threads.

   GET requests on / end up in a number of Received a GET request for / from [::1]:port followed by the libevent illegal connection state. Sometimes it’s as few as 5.

   Occasionally a segfault is caused, other times the ’terminate called without an active exception’ error.

   0.14.2 has the same issue.

   Further testing: Hitting any endpoint that is registered with a handler causes the problem. “/”, “/rest/tx/” if enabled, “/wallet/”, etc, but not “/wallet” or “/arbitrarystring”.

   The response “JSONRPC server handles only POST requests” is getting through to the client until the crash.

5. 
   esotericnonsense commented at 10:38 pm on September 13, 2017: contributor

   I’ve dug around a bit and been unable to find the problem. The work queue is not filling up; adding new items is fine; possibly a libevent issue?
6. 
   promag commented at 10:59 pm on September 13, 2017: member

   possibly a libevent issue

   Don’t know but I doubt. I’ll try to reproduce.

7. 
   esotericnonsense commented at 0:04 am on September 14, 2017: contributor

   Bit more digging:

   0$ nc -l -p 18334
   1GET / HTTP/1.1                           
   2Host: localhost:18334                    
   3User-Agent: python-requests/2.18.4       
   4Accept-Encoding: gzip, deflate           
   5Accept: */*                              
   6Connection: keep-alive                   
   7Content-Length: 2                        
   8
   9{}
   

   Tweaking cURL to give the same output:

   0for x in {1..2000}; do curl -s -X GET -H "User-Agent: python-requests/2.18.4" -H "Accept-Encoding: gzip, deflate" -H "Accept: */*" -H "Connection: keep-alive" -H "Content-Type:" --data "{}" http://localhost:18332/ > /dev/null; done
   

   (the output of nc is identical given this modification, checked with diff).

   Can’t reproduce using cURL. It does hit the server much more slowly, though.

8. 
   theuni commented at 0:14 am on September 14, 2017: member

   I’ve been up to my eyeballs in libevent lately, so I’ll take this one.

   Edit: Not to say anyone should stop investigating, of course!

9. theuni assigned theuni on Sep 14, 2017
10. 
    theuni commented at 1:41 pm on September 18, 2017: member

    I’ve tracked down the root cause of this and am working on a proper fix.
11. 
    dcousens commented at 5:11 am on September 19, 2017: contributor

    Cannot reproduce with fa8a0639f7b0ce04030b72b4d5be4f0aa36fc5cb

    python 3

    0import requests
    1import json
    2
    3while True:
    4        print("get")
    5        r = requests.get("http://127.0.0.1:9332", data=json.dumps({}))
    

12. 
    practicalswift commented at 1:42 pm on September 19, 2017: contributor

    @theuni Missing lock? :-)
13. 
    dcousens commented at 9:07 am on November 1, 2017: contributor

    @theuni was this actually an issue?
14. 
    ITsolution-git commented at 12:51 pm on March 6, 2018: none

    Hi @theuni. Did you find any solution to this? Thanks.
15. 
    promag commented at 1:11 am on October 18, 2018: member

    @theuni ping. Can you share your findings?
16. MarcoFalke closed this on Apr 27, 2020

    ---

17. 
    MarcoFalke commented at 0:35 am on April 27, 2020: member

    Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce?
18. MarcoFalke locked this on Feb 15, 2022

Contributors
esotericnonsense promag theuni dcousens practicalswift ITsolution-git MarcoFalke

Labels
RPC/REST/ZMQ