crabel99 commented at 5:59 am on January 24, 2018: none

Setting up a new linux box with Ubuntu 16.0.4 LTS on a 2009 MacPro. during initial network sync downloading hangs at around block 477890. This has been observed with ppa version 0.15.1 and with 0.15.99.0-b5e4b9b. The command bicoin-cli reconsiderblock [“block hash”] will occasionally advance, but usually 1-2 blocks at most.

Expected bitcoind to sync with no problems. Even with a stuck block, I expected reconsiderblock to be able to unstick and allow the network syncing to proceed.

snipet of debug.log when problem occured 2018-01-23 22:48:30 UpdateTip: new best=000000000000000000a87c7f93a33b243350a891c275a7f64febb7377ae4bb65 height=477892 version=0x20000002 log2_work=86.83599 tx=242351572 date=‘2017-07-28 0$ 2018-01-23 22:48:30 UpdateTip: new best=00000000000000000131d93cad98313b8c64ad61220b704c62dd9ef454972403 height=477893 version=0x20000012 log2_work=86.836028 tx=242353398 date=‘2017-07-28 $ 2018-01-23 22:48:31 ERROR: ConnectBlock: CheckQueue failed 2018-01-23 22:48:31 Misbehaving: 199.188.204.106:8333 peer=27 (0 -> 100) BAN THRESHOLD EXCEEDED 2018-01-23 22:48:31 InvalidChainFound: invalid block=000000000000000000cba89109e4baf645122d0211d93e808456e0c9cfb53a35 height=477894 log2_work=86.836067 date=2017-07-28 02:03:09 2018-01-23 22:48:31 InvalidChainFound: current best=00000000000000000131d93cad98313b8c64ad61220b704c62dd9ef454972403 height=477893 log2_work=86.836028 date=2017-07-28 01:43:21 2018-01-23 22:48:31 ERROR: ConnectTip(): ConnectBlock 000000000000000000cba89109e4baf645122d0211d93e808456e0c9cfb53a35 failed 2018-01-23 22:48:31 InvalidChainFound: invalid block=000000000000000000cba89109e4baf645122d0211d93e808456e0c9cfb53a35 height=477894 log2_work=86.836067 date=2017-07-28 02:03:09 2018-01-23 22:48:31 InvalidChainFound: current best=00000000000000000131d93cad98313b8c64ad61220b704c62dd9ef454972403 height=477893 log2_work=86.836028 date=2017-07-28 01:43:21 2018-01-23 22:48:31 Pre-allocating up to position 0x6000000 in blk00951.dat 2018-01-23 22:48:31 Pre-allocating up to position 0x7000000 in blk00951.dat 2018-01-23 22:48:32 New outbound peer connected: version: 70015, blocks=505769, peer=1274 2018-01-23 22:48:34 Pre-allocating up to position 0x8000000 in blk00951.dat 2018-01-23 22:48:36 Leaving block file 951: CBlockFileInfo(blocks=156, size=134139846, heights=477850…478413, time=2017-07-27…2017-07-31) 2018-01-23 22:48:37 Pre-allocating up to position 0x1000000 in blk00952.dat 2018-01-23 22:48:40 Pre-allocating up to position 0x2000000 in blk00952.dat 2018-01-23 22:49:09 Pre-allocating up to position 0x3000000 in blk00952.dat 2018-01-23 22:49:43 ERROR: AcceptBlockHeader: prev block invalid 2018-01-23 22:49:43 Misbehaving: 110.10.81.224:8333 peer=338 (0 -> 100) BAN THRESHOLD EXCEEDED 2018-01-23 22:49:43 ERROR: invalid header received 2018-01-23 22:49:43 ERROR: AcceptBlockHeader: prev block invalid

snipet of a reconsiderblock from debug.log 2018-01-24 00:39:14 ERROR: ConnectBlock: CheckQueue failed 2018-01-24 00:39:14 InvalidChainFound: invalid block=0000000000000000012f0e4974243ef26af7317366ffccdaac013105582d34c5 height=478058 log2_work=86.842382 date=2017-07-29 03:40:27 2018-01-24 00:39:14 InvalidChainFound: current best=00000000000000000094c2fdb70a5b2ff7e0ddbf664eb63d08d0215151c8fa5b height=478057 log2_work=86.842344 date=2017-07-29 03:39:12 2018-01-24 00:39:14 ERROR: ConnectTip(): ConnectBlock 0000000000000000012f0e4974243ef26af7317366ffccdaac013105582d34c5 failed 2018-01-24 00:39:14 InvalidChainFound: invalid block=0000000000000000012f0e4974243ef26af7317366ffccdaac013105582d34c5 height=478058 log2_work=86.842382 date=2017-07-29 03:40:27 2018-01-24 00:39:14 InvalidChainFound: current best=00000000000000000094c2fdb70a5b2ff7e0ddbf664eb63d08d0215151c8fa5b height=478057 log2_work=86.842344 date=2017-07-29 03:39:12 2018-01-24 00:39:16 UpdateTip: new best=0000000000000000012f0e4974243ef26af7317366ffccdaac013105582d34c5 height=478058 version=0x20000002 log2_work=86.842382 tx=242624309 date=‘2017-07-29 $ 2018-01-24 00:39:16 UpdateTip: new best=00000000000000000139f00fb5619f60df2ea7382939bd03b95a3d671248eb4b height=478059 version=0x20000002 log2_work=86.842421 tx=242625377 date=‘2017-07-29 $ 2018-01-24 00:39:17 Pre-allocating up to position 0xf00000 in rev00949.dat 2018-01-24 00:39:17 UpdateTip: new best=0000000000000000002efc240da85b16fdcaa0ed5de19481797291fd153949db height=478060 version=0x20000012 log2_work=86.842459 tx=242626018 date=‘2017-07-29 $ 2018-01-24 00:39:17 UpdateTip: new best=00000000000000000077ec2249bd5635244183fbde7088f33c34e4e63934f51b height=478061 version=0x20000002 log2_work=86.842497 tx=242627176 date=‘2017-07-29 $ 2018-01-24 00:39:17 ERROR: ConnectBlock: CheckQueue failed 2018-01-24 00:39:17 InvalidChainFound: invalid block=0000000000000000008987b70cb2bc48e9c670ff4b7d8cd17b64f68e21c14c63 height=478062 log2_work=86.842536 date=2017-07-29 04:05:25 2018-01-24 00:39:17 InvalidChainFound: current best=00000000000000000077ec2249bd5635244183fbde7088f33c34e4e63934f51b height=478061 log2_work=86.842497 date=2017-07-29 03:55:33 2018-01-24 00:39:17 ERROR: ConnectTip(): ConnectBlock 0000000000000000008987b70cb2bc48e9c670ff4b7d8cd17b64f68e21c14c63 failed 2018-01-24 00:39:17 InvalidChainFound: invalid block=0000000000000000008987b70cb2bc48e9c670ff4b7d8cd17b64f68e21c14c63 height=478062 log2_work=86.842536 date=2017-07-29 04:05:25 2018-01-24 00:39:17 InvalidChainFound: current best=00000000000000000077ec2249bd5635244183fbde7088f33c34e4e63934f51b height=478061 log2_work=86.842497 date=2017-07-29 03:55:33 2018-01-24 00:40:12 socket recv error Connection reset by peer (104) 2018-01-24 00:41:16 socket recv error Connection reset by peer (104) 2018-01-24 00:42:05 version handshake timeout from 357

I have been able to reproduce this 4 times on fresh OS installs using the ppa 0.15.1. I then compiled the source for 0.15.99 to see if that would cause the problem. The machine is clean with just enough dependencies to compile and test the code without error. I have not tried downloading a 0.14.2 version as that was the released version for the blocks in question. I will do this tomorrow. There was similar problem with bitcoinx reported here at the same block height. https://www.reddit.com/r/BitcoinAirdrops/comments/7r4jmc/anyone_else_troubles_syncing_btx_bitcoinx_at/

Various: ppa 0.15.1 Self compiled 0.15.99.0-b5e4b9b

Ubuntu 16.0.4 LTS (fully up to date fresh install) 2009 MacPro w/ (x8) 2.2GHz Intel/ 64GB RAM and 3TB (apple hardware) J5 RAID with (4) Seagate 2TB hard drives (the swap space is 69GB but isn’t used)

See above

fanquake added the label Linux/Unix on Jan 24, 2018

Willtech commented at 8:10 am on January 26, 2018: contributor

I currently have:

019:07:21  getblockhash 477893 
119:07:47  00000000000000000131d93cad98313b8c64ad61220b704c62dd9ef454972403

But, am running that node pruned so don’t know if I have that block or if the hash was fetched.

crabel99 commented at 12:25 pm on January 26, 2018: none

I am now having a similar problem with 0.14.2 occurring earlier in the blockchain. I have a node running on another machine that is synced to the network. on my unix box that I’m trying to get up I edited the bitcoin.conf file to connect directly to the full node I had running. Next I wrote a script to keep the sync going: `#!/bin/bash

bin directory for bitcoind and bitcoin-cli

launchdir=$HOME/Downloads/bitcoin-0.14.2/bin

launch the bitcoin daemon and count the number of errors in the debug log

sudo $launchdir/bitcoind -datadir=/var/lib/bitcoind -conf=/etc/bitcoin/bitcoin.conf & counterold=$(grep -o ‘ERROR’ /var/log/bitcoind/debug.log | wc -l)

wait for a minute to allow the daemon to sync with the network

sleep 60

initialize the block height

BLOCK=$($launchdir/bitcoin-cli getblockcount)

set the period to run until approximately synched with the network

while [ $BLOCK -lt 506095 ] do # count the number of errors in the log file and determine the block height counter=$(grep -o ‘ERROR’ /var/log/bitcoind/debug.log | wc -l) BLOCK=$($launchdir/bitcoin-cli getblockcount) # skip the block reconsideration if there is no change in error numbers in the log file while [ $counterold -lt $counter ] do # report the location of the hung block and reconsider that block echo Block synchronization is stuck on block height $BLOCK HASH=$($launchdir/bitcoin-cli getblockhash $BLOCK) $launchdir/bitcoin-cli reconsiderblock $HASH counterold=$counter done # wait 2 seconds sleep 2 done

congrats you are done

echo Successfully synced to the network!`

My current block height is 468188 and is slowly progressing. The error counter in the debug.log file is at 6327 and rising.

Willtech commented at 12:18 pm on January 27, 2018: contributor

Could it be that this is a hardware issue with that machine?

crabel99 commented at 4:04 pm on January 27, 2018: none

I don’t think so. I ran bitcoind on this computer under Mac OS from 2012-2017. It chugged along the entire time without issue. I used to even mine with it back in the day… The blocks sync a few at a time when I force it to reconsider the block in the above script. There has got to be a configuration that is off that is causing the syncing difficulty. I have the stock bitcoin.conf file with a few updates for RPC and connecting to trusted peers, but other than that it runs on defaults. On my other MacOS and Windows 10 boxes, the defaults are adequate to sync with the network.

crabel99 commented at 4:01 pm on January 28, 2018: none

Using the script above I was able to force the client to sync to the network. Once I was synced to the network, I shifted bitcoin.conf to the default (moving my known node from connect to addnode). I shifted the path from 0.14.2 to 0.15.99 and the daemon converted the downloaded chain to the newer version and with a few more bitcoin-cli reconsiderblock commands it finally started relaying transactions and keeping the chain up to date.

I am not sure why this was so difficult. Some things to observe. Because my hardware was able to sync to the network and is now properly relaying transactions, it is not a hardware issue. There is some other problem that I have no idea what it is. There are still a considerable amount of connection errors.

2018-01-28 15:55:29 New outbound peer connected: version: 70015, blocks=506512, peer=96 2018-01-28 15:55:33 Misbehaving: 35.185.46.13:8333 peer=96 (0 -> 100) BAN THRESHOLD EXCEEDED 2018-01-28 15:55:37 New outbound peer connected: version: 70015, blocks=506412, peer=94 2018-01-28 15:55:42 New outbound peer connected: version: 70015, blocks=506512, peer=97 2018-01-28 15:55:54 connect() to 91.205.47.233:8333 failed after select(): No route to host (113) 2018-01-28 15:55:56 New outbound peer connected: version: 70013, blocks=224331, peer=98 2018-01-28 15:56:00 version handshake timeout from 93 2018-01-28 15:56:07 Misbehaving: 171.8.83.19:8333 peer=79 (60 -> 80) 2018-01-28 15:56:07 ERROR: non-continuous headers sequence 2018-01-28 15:56:19 New outbound peer connected: version: 70015, blocks=506512, peer=99 2018-01-28 15:56:35 Misbehaving: 5.9.101.190:8333 peer=97 (0 -> 100) BAN THRESHOLD EXCEEDED 2018-01-28 15:56:36 Misbehaving: 185.145.130.92:8333 peer=99 (0 -> 100) BAN THRESHOLD EXCEEDED 2018-01-28 15:56:44 connect() to 66.172.112.181:8333 failed after select(): Connection refused (111) 2018-01-28 15:56:46 Misbehaving: 10.1.10.139:8333 peer=0 (21700 -> 21800) 2018-01-28 15:56:46 Misbehaving: 10.1.10.139:8333 peer=0 (21800 -> 21900) 2018-01-28 15:56:46 Misbehaving: 10.1.10.139:8333 peer=0 (21900 -> 22000) 2018-01-28 15:56:46 Misbehaving: 10.1.10.139:8333 peer=0 (22000 -> 22100) 2018-01-28 15:56:46 Misbehaving: 10.1.10.139:8333 peer=0 (22100 -> 22200) 2018-01-28 15:56:46 Misbehaving: 10.1.10.139:8333 peer=0 (22200 -> 22300) 2018-01-28 15:56:47 New outbound peer connected: version: 70015, blocks=506512, peer=101 2018-01-28 15:56:50 connect() to [2604:9a00:2010:a08d:10::23]:8333 failed after select(): Connection refused (111) 2018-01-28 15:56:51 connect() to 91.214.82.68:8333 failed after select(): Connection refused (111) 2018-01-28 15:56:52 Misbehaving: 120.79.83.179:8333 peer=101 (0 -> 100) BAN THRESHOLD EXCEEDED 2018-01-28 15:56:52 Misbehaving: 10.1.10.139:8333 peer=0 (22300 -> 22400) 2018-01-28 15:56:58 connect() to 51.15.7.224:8333 failed after select(): Connection refused (111)

Willtech commented at 8:32 am on January 30, 2018: contributor

You have some serious errors noted in your log there, you even have: 2018-01-28 15:56:07 ERROR: non-continuous headers sequence and 2018-01-28 15:56:46 Misbehaving: 10.1.10.139:8333 peer=0 (21700 -> 21800) - I presume this is your local peer that is ban exempt because it was specified?

I do not know what the issue is but I will leave further attempts to diagnose to those who have coded the software. There is certainly some issue between your peers.