v0.14.0 node shutdown failure #10001

Describe the issue

bitcoind-v0.14.0 can't shutdown.

Can you reliably reproduce the issue?

Yes. Always happen.

If so, please list the steps to reproduce below:

1. build bitcoind with docker: https://github.com/btccom/btcpool/tree/master/docker/bitcoind/v0.14.0
2. get into docker, than shutdown node with rpc call bitcoin-cli stop
3. bitcoind process will hang up.

Expected behaviour

Actual behaviour

debug.log:

2017-03-15 11:49:28 CreateNewBlock(): total size: 998181 block weight: 3992724 txs: 1516 fees: 94050467 sigops 14524
2017-03-15 11:49:38 CreateNewBlock(): total size: 998191 block weight: 3992764 txs: 1533 fees: 96320496 sigops 14772
2017-03-15 11:49:44 tor: Thread interrupt
2017-03-15 11:49:44 torcontrol thread exit
2017-03-15 11:49:44 opencon thread exit
2017-03-15 11:49:44 scheduler thread interrupt
2017-03-15 11:49:44 Shutdown: In progress...
2017-03-15 11:49:44 net thread exit
2017-03-15 11:49:44 msghand thread exit

$ strace -p `pgrep bitcoind`
Process 31913 attached
futex(0x7fd03d5899d0, FUTEX_WAIT, 61, NULL^CProcess 31913 detached

What version of bitcoin-core are you using?

tag v0.14.0

Machine specs:

• OS: ubuntu 14.04 LTS, 64Bits
  • Docker: Ubuntu 16.04 LTS, 64Bits
• CPU: Intel
• RAM: 8G+
• Disk size: 200G+
• Disk Type (HD/SDD): SSD

debug log from another server:

2017-03-15 12:12:30 connect() to [2001:0:34af:cdd1:414:187b:b2dd:20ab]:8333 failed: Network is unreachable (101)
2017-03-15 12:13:36 connect() to [2001:0:9d38:78cf:3c1f:3759:a844:c08]:8333 failed: Network is unreachable (101)
2017-03-15 12:13:48 socket recv error Connection reset by peer (104)
2017-03-15 12:14:06 tor: Thread interrupt
2017-03-15 12:14:06 torcontrol thread exit
2017-03-15 12:14:06 opencon thread exit
2017-03-15 12:14:06 scheduler thread interrupt
2017-03-15 12:14:06 Shutdown: In progress...
2017-03-15 12:14:06 net thread exit
2017-03-15 12:14:06 msghand thread exit

------------------------------------

$ strace -p 21360
Process 21360 attached
futex(0x7f6a9ddea9d0, FUTEX_WAIT, 19190, NULL

#9950 was another issue where 0.14 hung on shutdown. That was caused by having >=8 addnodes in the bitcoin.conf file. Do you have that? Are you able to paste your bitcoin.conf here?

@jnewbery yeah, I put about 20 nodes in the bitcoin.conf file.

rpcallowip=172.17.0.0/16
rpcallowip=192.168.0.0/16
rpcallowip=10.0.0.0/8

 # we use ZMQ to get notification when new block is coming
zmqpubrawblock=tcp://0.0.0.0:8331
zmqpubrawtx=tcp://0.0.0.0:8331
zmqpubhashtx=tcp://0.0.0.0:8331
zmqpubhashblock=tcp://0.0.0.0:8331

 # use 2G memory for utxo, depends on your machine's memory
 dbcache=1000

 # use 1MB block when call GBT
 blockmaxsize=1000000

 ####################################

 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx
 addnode=xx.xx.xx.xx

@jnewbery temporary fix: https://github.com/btccom/btcpool/commit/656dd3dbd09ccc5e679b4ce13fe5e79adf470073

using patch: #9953

 RUN cd ~/source \
   && tar zxf v0.14.0.tar.gz && cd bitcoin-0.14.0 \
+  && wget https://patch-diff.githubusercontent.com/raw/bitcoin/bitcoin/pull/9953.patch -O 9953.patch \
+  && patch -p1 < 9953.patch \
   && ./autogen.sh \
   && ./configure --disable-wallet --disable-tests \

Right now, restart node works fine. I will put it on pool.btc.com production env.

Do note that, as per the 0.14 release notes, there is now an 8-addnode limit - after the first 8 the rest are ignored (this was previously true as well, but indirectly because they were simply used as nodes to consider for the 8 general outbound connections).

Matt

On March 15, 2017 5:31:56 AM PDT, kevin notifications@github.com wrote:

Describe the issue

bitcoind-v0.14.0 can't shutdown.

Can you reliably reproduce the issue?

Yes. Always happen.

If so, please list the steps to reproduce below:

1. build bitcoind with docker: https://github.com/btccom/btcpool/tree/master/docker/bitcoind/v0.14.0
2. get into docker, than shutdown node with rpc call bitcoin-cli stop
3. bitcoind process will hang up.

Expected behaviour

Actual behaviour

debug.log:

2017-03-15 11:49:28 CreateNewBlock(): total size: 998181 block weight:
3992724 txs: 1516 fees: 94050467 sigops 14524
2017-03-15 11:49:38 CreateNewBlock(): total size: 998191 block weight:
3992764 txs: 1533 fees: 96320496 sigops 14772
2017-03-15 11:49:44 tor: Thread interrupt
2017-03-15 11:49:44 torcontrol thread exit
2017-03-15 11:49:44 opencon thread exit
2017-03-15 11:49:44 scheduler thread interrupt
2017-03-15 11:49:44 Shutdown: In progress...
2017-03-15 11:49:44 net thread exit
2017-03-15 11:49:44 msghand thread exit

$ strace -p `pgrep bitcoind`
Process 31913 attached
futex(0x7fd03d5899d0, FUTEX_WAIT, 61, NULL^CProcess 31913 detached

What version of bitcoin-core are you using?

tag v0.14.0

Machine specs:

• OS: ubuntu 14.04 LTS, 64Bits
• CPU: Intel
• RAM: 8G+
• Disk size: 200G+
• Disk Type (HD/SDD): SSD

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/10001

@TheBlueMatt why there is an 8-addnode limit? Eight is far from enough for a pool's node. At least I need to connect with mine own nodes, it's surely more that 8.

Heh, I figured I should mention it :)

There has always been an 8 addnode limit, because addnodes used to be simply extra nodes to consider when making the 8 outbound convections, now they're split out and there is a separate limit of 8. This limit was kept because there has historically been a lot of folks who just add addnodes to their bitcoin.conf without realizing the implication of potentially DoSing the network :/.

If you want to override it, you can either patch the MAX_ADDNODE_CONNECTIONS constant higher or use the "addnode $NODE onetry" RPC command, which bypasses the limit for the given connection, but you'd have to run that in a loop.

On March 15, 2017 10:31:17 PM PDT, kevin notifications@github.com wrote:

@TheBlueMatt why there is an 8-addnode limit? Eight is far from enough for a pool's node. At least I need to connect with mine own nodes, it's surely more that 8.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/10001#issuecomment-286962084

fixed in 0.14.1