Rapidly getting blocks via RPC causes bitcoind to temporarily stop responding to RPC commands #2889

issue ryanxcharles opened this issue on August 9, 2013
  1. ryanxcharles commented at 10:57 PM on August 9, 2013: contributor

    When retrieving blocks from the blockchain using bitcoind RPC commands, bitcoind will temporarily stop sending responses to RPC commands after many thousands of commands if the commands are sent in quick succession. The issue has something to do with timing, since adding a delay to retrieving blocks will make the problem go away.

    This node.js code will display the issue (obviously, replace "password" with your bitcoind password):

    var rpc=require('json-rpc2');
    var bitcoind=rpc.Client.create(8332, "localhost",
      "bitcoinrpc", "password");
    var chain=function(blockhash){
      bitcoind.call('getblock',[blockhash],function(err,block){
        if (err){
          console.log(String(err));
          if (String(err)=="Error: Have no response object"){
            console.log("Thought so.");
          }
          return;
        }
        console.log(block.hash, block.height);
        if (block.nextblockhash){
          chain(block.nextblockhash);
        };
      });
    };
    chain("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f");
    

    Note that my development machine has an SSD, and so blocks are retrieved quickly by bitcoind - it may be necessary to use an SSD to reproduce the problem.

    What this code does is to retrieve the genesis block, then get the next block, and the next block, and so on, until it has got all the blocks. Except at around 25,000 or so blocks in (different every time), it will quit when it encounters a "no response object" error. This error has nothing to do with node.js or json-rpc2 - it is a bitcoind error. This can be seen by running bitcoin commands from the command prompt, which result in a "couldn't connect to server" error, e.g.:

    > bitcoind getblock 00000000722ecede1cc4edd52a7d416bdafd6528f630926b0021b41a6b57ba5d
    error: couldn't connect to server
    

    After a few seconds, bitcoind will begin responding to commads again. It doesn't crash. It just stops responding to commands for a short while.

    By delaying repeated RPC commands, the above code can be fixed to run without freezing bitcoind (notice the 10ms setTimeout):

    var rpc=require('json-rpc2');
    var bitcoind=rpc.Client.create(8332, "localhost",
      "bitcoinrpc", "password");
    var chain=function(blockhash){
      bitcoind.call('getblock',[blockhash],function(err,block){
        if (err){
          console.log(String(err));
          if (String(err)=="Error: Have no response object"){
            console.log("Thought so.");
          }
          return;
        }
        console.log(block.hash, block.height);
        if (block.nextblockhash){
          setTimeout(function(){
            chain(block.nextblockhash);
          },10);
        };
      });
    };
    chain("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f");
    
  2. ryanxcharles commented at 11:12 PM on August 9, 2013: contributor

    After further tests, it seems to have nothing to do with blocks at all, but even a simple command like "getdifficulty" will cause the RPC server to stop sending responses after about 28,000 requests:

    var rpc=require('json-rpc2');
    var bitcoind=rpc.Client.create(8332, "localhost",
      "bitcoinrpc", "password");
    var chain=function(i){
      bitcoind.call('getdifficulty',[],function(err,result){
        if (err){
          console.log(String(err));
          if (String(err)=="Error: Have no response object"){
            console.log("Thought so.");
          }
          return;
        }
        console.log(result, i);
        chain(i+1);
      });
    };
    chain(0);
    
  3. Diapolo commented at 12:22 PM on August 10, 2013: none

    Perhaps there is a memory leak in the RPC server we didn't yet catch? Does bitcoind consume a weird amount of memory after 28000 requests?

  4. sipa commented at 12:27 PM on August 10, 2013: member

    How long is "temporarily"?

    EDIT: a few seconds, apparently. That can just be a new block that arrives and is being processed, I think. RPC commands (and pretty much everything) stops working then. A few seconds sounds long though.

  5. ryanxcharles commented at 11:36 PM on August 10, 2013: contributor

    Perhaps there is a memory leak in the RPC server we didn't yet catch? Does bitcoind consume a weird amount of memory after 28000 requests?

    No, the memory usage is low. It does use a lot of CPU though - about 50% CPU usage according to top.

    How long is "temporarily"?

    It depends. Sometimes it last a few seconds, sometimes it lasts tens of seconds.

    EDIT: a few seconds, apparently. That can just be a new block that arrives and is being processed, I think. RPC commands (and pretty much everything) stops working then. A few seconds sounds long though.

    It's not due to a new block. I just tested it and it froze several times before a new block was found.

  6. npiguet commented at 3:31 PM on February 25, 2014: none

    I can actually confirm that myself. I can add this the stopping doesn't happen abruptly. For me, responses to RPC requests that are fired in quick succession become gradually slower and slower. After some time though, bitcoind seems to recover and the sequence of requests can be attempted again, but the pattern repeats (high number of requests fullfilled at first, then gradually becomes slower and slower)

    Interestingly, bitcoind's memory usage doesn't grow, and remains more or less constant during the whole process. Its CPU usage also doesn't grow, but instead seems to be proportional to the rate of the responses it gives. Looking at the CPU usage, you can see high usage at first (when the rate of requests is high), and then the CPU usage gradually declines.

    This looks to me like bitcoind is really sleeping (or waiting for something) for some time before, during or after completing the request.

  7. espringe commented at 6:55 PM on March 14, 2014: none

    I'm have the same problem -- albeit with bitcoind stop responding to RPC for minutes. During this time, running even bitcoind help locally will just hang.

    Perhaps coincidentally, but I'm also using node to make rpc calls. I tried a very basic, local bash loop executing commands -- and did not have the same pause problem. I am wondering if it could be something related to the library used making these rpc calls. Perhaps it's not immediately ending connections?

  8. espringe commented at 7:01 PM on March 14, 2014: none

    Bash loop that appears to not cause problems:

    COUNTER = 1
    while true; do
    echo $COUNTER
    bitcoind getdifficulty
    let COUNTER+=1
    done
    
  9. npiguet commented at 7:05 PM on March 14, 2014: none

    I've been using Java's HttpUrlConnection mechanism to do those calls. I know that it makes use of the HTTP keepalive feature to avoid having to setup and teardown a new TCP connection for each request. NodeJS probably does the same.

    However, the bash loop obviously doesn't do that. Could this be what causes the difference?

  10. npiguet commented at 7:07 PM on March 14, 2014: none

    I'll try doing some tests that add the header "Connection: close" to the request so that keepalive is disabled, and see if that makes a difference.

  11. leofidus commented at 10:47 PM on March 14, 2014: none

    Since a 10ms timeout is enough to make the problem go away in the first example, I suspect the bash loop is simply too slow to reproduce the problem.

  12. laanwj added the label RPC on May 13, 2014
  13. laanwj added the label Bug on May 13, 2014
  14. shivaenigma commented at 12:48 PM on June 12, 2015: none

    This is hapenning on my server after every few weeks . We have a getblock call to bitcoind for each blocks. After few block numbers bitcoind RPC hangs ( I have tried waiting for 30 min , but doesn't help, have to kill bitcoind and restart). Running bitcoind 0.10.2 .

    Is this fixed ? I can attempt to write a script to reproduce if this is not fixed yet... in the latest source

  15. laanwj commented at 12:54 PM on June 12, 2015: member

    No, it's not fixed, that's why this issue is still open. A script to reproduce it would be very useful.

  16. shivaenigma commented at 6:47 AM on July 10, 2015: none

    I am not sure if this is bcoz of some throtlling feature on bitcoin , but this is how to reproduce

    from bitcoinrpc.authproxy import AuthServiceProxy
    import sys
    import threading
    import traceback
    import time
    
    number = 5
    conn = [None] * number
    while(True):
      for i in range(0, number):
        try:
          conn[i] = AuthServiceProxy(sys.argv[1])
          count = conn[i].getblockcount()
          raw_block=conn[i].getblock(conn[i].getblockhash(count))
          print "Got block %d"%(count)  
        except:
          traceback.print_exc()
          #Ignore and continue
    

    Run the above script and then u can see the bitcoin-cli getinfo will stop responding.

  17. shivaenigma commented at 6:50 AM on July 10, 2015: none

    Basically on my server sometimes lots of these connections are going into CLOSE_WAIT state

    python   6388 ubuntu   11u  IPv4 156823      0t0  TCP localhost:37917->localhost:8332 (ESTABLISHED)
    python   6388 ubuntu   13u  IPv4 125106      0t0  TCP localhost:36759->localhost:8332 (CLOSE_WAIT)
    python   6898 ubuntu   11u  IPv4 128729      0t0  TCP localhost:36958->localhost:8332 (ESTABLISHED)
    

    When the number of these CLOSE_WAIT connections is becoming 5 ... rpc calls wont respond anymore and I have to restart my client

  18. mcelrath commented at 3:06 PM on October 29, 2015: none

    This is likely the same as #6454 and fixed by #5677

  19. laanwj closed this on Apr 28, 2016

  20. winteraz commented at 2:37 AM on January 22, 2018: none

    This is not fixed. I'm running a bitcoin node (with large amount of ram (i.e. 64GB) and after several requests to get the unspent transactions the rpc service get stuck and stops to responds to new requests.

  21. MarcoFalke commented at 1:49 PM on January 22, 2018: member

    @winteraz This issue is about five years old and was closed two years ago. If you are certain that you are running into the same issue, please open a new one and reference (link to) this issue.

    Also, make sure to include steps to reproduce and the version you are using.

  22. IntegralTeam referenced this in commit 8ffdcbf999 on Jun 4, 2019
  23. DrahtBot locked this on Sep 8, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-13 15:16 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me