ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large

eN0Rm commented at 6:12 pm on January 15, 2015: none

Client info

2015-01-15 18:15:21 Bitcoin version v0.10.99.0-4f73a8f-dirty (2015-01-09 20:21:19 -0800) 2015-01-15 18:15:21 Using OpenSSL version OpenSSL 1.0.1f 6 Jan 2014 2015-01-15 18:15:21 Using BerkeleyDB version Berkeley DB 5.3.28: (September 9, 2013) 2015-01-15 18:15:21 Default data directory /home/fredrik/.bitcoin 2015-01-15 18:15:21 Using data directory /home/fredrik/.bitcoin 2015-01-15 18:15:21 Using config file /home/fredrik/.bitcoin/bitcoin.conf 2015-01-15 18:15:21 Using at most 200 connections (1024 file descriptors available) 2015-01-15 18:15:21 Using 0 threads for script verification 2015-01-15 18:15:21 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0) 2015-01-15 18:15:21 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0) 2015-01-15 18:15:21 Using wallet wallet.dat 2015-01-15 18:15:21 init message: Verifying wallet… 2015-01-15 18:15:21 CDBEnv::Open : LogDir=/home/fredrik/.bitcoin/database ErrorFile=/home/fredrik/.bitcoin/db.log 2015-01-15 18:15:21 Bound to [::]:8333 2015-01-15 18:15:21 Bound to 0.0.0.0:8333 2015-01-15 18:15:21 init message: Loading block index… 2015-01-15 18:15:21 Opening LevelDB in /home/fredrik/.bitcoin/blocks/index 2015-01-15 18:15:22 Opened LevelDB successfully 2015-01-15 18:15:22 Opening LevelDB in /home/fredrik/.bitcoin/chainstate 2015-01-15 18:15:28 Opened LevelDB successfully 2015-01-15 18:15:32 LoadBlockIndexDB: last block file = 218 2015-01-15 18:15:32 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=274, size=115501816, heights=338812…339085, time=2015-01-13…2015-01-15) 2015-01-15 18:15:32 Checking all blk files are present… 2015-01-15 18:15:32 LoadBlockIndexDB(): transaction index disabled 2015-01-15 18:15:32 LoadBlockIndexDB(): hashBestChain=0000000000000000188de542fd76b1676c4be6c380b39ddea119358c290cebd7 height=339085 date=2015-01-15 18:07:14 progress=0.999987 2015-01-15 18:15:32 init message: Verifying blocks… 2015-01-15 18:15:32 Verifying last 288 blocks at level 3

log before crash

2015-01-13 22:15:27 UpdateTip: new best=0000000000000000159e7d66c954312bccb5f12de808f2991b3859205665c442 height=338819 log2_work=81.997891 tx=56658614 date=2015-01-13 22:14:41 progress=0.999999 cache=6969 2015-01-13 22:15:58 ResendWalletTransactions() 2015-01-13 22:17:06 receive version message: /Satoshi:0.9.2/opennodes.org:0.1/: version 70002, blocks=338819, us=84.215.171.162:8333, peer=2855 2015-01-13 22:17:25 receive version message: /libbitcoin:2.0/: version 60000, blocks=0, us=10.0.0.1:8333, peer=2856 2015-01-13 22:17:25 Added time data, samples 200, offset -5 (+0 minutes) 2015-01-13 22:18:06 socket recv error Connection reset by peer (104) 2015-01-13 22:18:13 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2857 2015-01-13 22:18:31 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=338818, us=84.215.171.162:8333, peer=2858 2015-01-13 22:19:16 receive version message: /Satoshi:0.9.3/: version 70002, blocks=338817, us=84.215.171.162:8333, peer=2859 2015-01-13 22:19:16 Added time data, samples 200, offset -2 (+0 minutes) 2015-01-13 22:19:32 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=2860 2015-01-13 22:19:47 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2861 2015-01-13 22:22:36 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=338819, us=84.215.171.162:8333, peer=2862 2015-01-13 22:22:44 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:22:45 receive version message: /BTCXchange.ro Mapper:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2863 2015-01-13 22:22:47 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=338819, us=127.0.0.1:8333, peer=2864 2015-01-13 22:22:47 Added time data, samples 200, offset +10 (+0 minutes) 2015-01-13 22:23:00 receive version message: /Satoshi:0.8.4/: version 70001, blocks=338819, us=84.215.171.162:8333, peer=2865 2015-01-13 22:23:00 Added time data, samples 200, offset -7 (+0 minutes) 2015-01-13 22:23:53 receive version message: /Satoshi:0.8.2.2/: version 70001, blocks=336635, us=84.215.171.162:8333, peer=2866 2015-01-13 22:23:53 Added time data, samples 200, offset +14 (+0 minutes) 2015-01-13 22:24:13 receive version message: /Satoshi:0.9.3/: version 70002, blocks=338735, us=84.215.171.162:8333, peer=2867 2015-01-13 22:24:13 Added time data, samples 200, offset -5 (+0 minutes) 2015-01-13 22:25:06 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=2868 2015-01-13 22:25:49 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2869 2015-01-13 22:26:33 UpdateTip: new best=00000000000000001829e20898eaff72aa667d3715a94535afae5ae7ada07bc0 height=338820 log2_work=81.997947 tx=56659414 date=2015-01-13 22:25:41 progress=0.999999 cache=9596 2015-01-13 22:26:35 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:26:43 ERROR: AcceptToMemoryPool : nonstandard transaction: dust 2015-01-13 22:26:48 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:26:58 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:27:01 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2870 2015-01-13 22:27:20 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:27:22 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:27:46 UpdateTip: new best=000000000000000008b332686e0043a4b95ca7e46d1b0785536981b543dfc755 height=338821 log2_work=81.998004 tx=56659415 date=2015-01-13 22:36:09 progress=1.000013 cache=9706 2015-01-13 22:28:29 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:28:34 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:28:48 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:28:52 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=338819, us=84.215.171.162:8333, peer=2871 2015-01-13 22:29:22 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:30:29 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large 2015-01-15 18:02:07

Diapolo commented at 7:40 am on January 16, 2015: none

ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large means there was a read from a block file on disk, which is larger than 32 MiB. I dunno why that can happen to be honest…

eN0Rm commented at 8:49 am on January 16, 2015: none

I’ve used the version that comes with Ubuntu 14.04, I can do an upgrade if that is the cause of the problem.

-f-

On Fri, Jan 16, 2015 at 8:41 AM, P. Kaufmann notifications@github.com wrote:

Just a hint before I further check debug.log, your OpenSSL version is rather out of date (OpenSSL 1.0.1f and 1.0.1k is current)!

— Reply to this email directly or view it on GitHub #5668 (comment).

Mvh -fredrik-normann-

Sent from my Gmail Account

laanwj commented at 8:51 am on January 16, 2015: member

@diapolo NOOOOOO don’t go around telling people to upgrade their OpenSSL. Remember it’s 1.0.1k that breaks things.

laanwj commented at 8:52 am on January 16, 2015: member

BTW: this looks like disk corruption in the block files (.bitcoin/blocks/blk*.dat).

Diapolo commented at 8:56 am on January 16, 2015: none

@laanwj Sorry, removed the comment! You are right!

Diapolo commented at 9:01 am on January 16, 2015: none

@laanwj Back to the actual error, IMHO (after looking through the code) I dived down until reaching void Unserialize(Stream& is, std::set<K, Pred, A>& m, int nType, int nVersion) in serialize.h. There ReadCompactSize() is called which contains nSizeRet > (uint64_t)MAX_SIZE check. Am I right, that this means we tried to read a chunk larger that 32MiB, which fails this check?

Addition: You also mind checking where MAX_SIZE is used in our code, do we need some better/safer limits there???

laanwj commented at 9:13 am on January 16, 2015: member

That’s a belt-and-suspenders sanity check. Blocks are never larger then 1 MB at the moment, so if there is data on disk that pretends to be a >32MiB chunk, something is corrupted and it’s correct to throw an error. (better than, say, trying to read 4GB of data and running out of memory before throwing an error, or worse, overwriting other parts of memory) Unless you spot critical issues I’d argue against making changes to MAX_SIZE handling, it’s deeply entwined with block handling and thus consensus.

Diapolo commented at 10:45 am on January 16, 2015: none

@laanwj IMHO it seems we use MAX_SIZE (remember thats 32MiB) also in CMessageHeader::IsValid(), ReadHTTPMessage() and CNetMessage::readHeader(). Isn’t that a bit large for these things?

laanwj commented at 11:24 am on January 16, 2015: member

@diapolo IMO it’s exactly the right size. It allows some breathing room. E.g. if block sizes would increase, or if other protocol messages would encode for larger objects. As said - let’s not change this unless necessary by a concrete issue.

It is a bit weird that the same constant is used for both P2P/serialization message checking and HTTP limitations.

kef commented at 10:53 am on January 17, 2015: none

So then why not have separate constants for these that coincidentally have the same 32MB value?

eN0Rm commented at 1:07 pm on January 17, 2015: none

this could be or usb controller bugs. The node runs in a vm on a disk that is connected with usb 3.0 I’ve had problems with this before, but it’s hard to tell from the error message.

On Fri, Jan 16, 2015 at 9:53 AM, Wladimir J. van der Laan < notifications@github.com> wrote:

BTW: this looks like disk corruption in the block files.

— Reply to this email directly or view it on GitHub #5668 (comment).

Mvh -fredrik-normann-

Sent from my Gmail Account

eN0Rm commented at 8:01 pm on January 19, 2015: none

Still getting this. Disk is stable now.

 02015-01-19 19:40:15 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=924
 12015-01-19 19:40:35 Pre-allocating up to position 0xa00000 in rev00220.dat
 22015-01-19 19:40:35 UpdateTip: new best=00000000000000000301f5d8111767a37a888add1e259cbd0394abe396859aa4  height=339685  log2_work=82.045948  tx=57249563  date=2015-01-19 19:40:35 progress=1.000000  cache=11199
 32015-01-19 19:40:37 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=339684, us=84.215.171.162:8333, peer=925
 42015-01-19 19:40:52 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=926
 52015-01-19 19:41:41 ERROR: AcceptToMemoryPool : inputs already spent
 62015-01-19 19:41:46 ERROR: AcceptToMemoryPool : inputs already spent
 72015-01-19 19:42:07 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=339684, us=84.215.171.162:8333, peer=927
 82015-01-19 19:42:34 ResendWalletTransactions()
 92015-01-19 19:42:38 UpdateTip: new best=000000000000000008a41249e3270e550e07c77bc939d7920d638eb7fb8d9a4e  height=339686  log2_work=82.046002  tx=57249914  date=2015-01-19 19:39:26 progress=0.999995  cache=11899
102015-01-19 19:42:46 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:4.16/: version 70001, blocks=339471, us=127.0.0.1:8333, peer=928
112015-01-19 19:42:46 Added time data, samples 185, offset -5 (+0 minutes)
122015-01-19 19:42:46 nTimeOffset = -4  (+0 minutes)
132015-01-19 19:42:50 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:4.16/: version 70001, blocks=339686, us=127.0.0.1:8333, peer=929
142015-01-19 19:42:50 Added time data, samples 186, offset -5 (+0 minutes)
152015-01-19 19:43:41 ERROR: AcceptToMemoryPool : inputs already spent
162015-01-19 19:45:15 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=930
172015-01-19 19:45:41 ERROR: AcceptToMemoryPool : inputs already spent
182015-01-19 19:46:45 UpdateTip: new best=000000000000000005737ecae89bdadbe6eb9a3d1710b101b83ce0fd28410f23  height=339687  log2_work=82.046057  tx=57250283  date=2015-01-19 19:46:37 progress=1.000000  cache=12835
192015-01-19 19:46:48 ERROR: AcceptToMemoryPool : inputs already spent
202015-01-19 19:46:49 ERROR: AcceptToMemoryPool : inputs already spent
212015-01-19 19:47:31 ERROR: AcceptToMemoryPool : inputs already spent
222015-01-19 19:47:33 ERROR: AcceptToMemoryPool : inputs already spent
232015-01-19 19:47:41 ERROR: AcceptToMemoryPool : inputs already spent
242015-01-19 19:48:29 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=339686, us=84.215.171.162:8333, peer=931
252015-01-19 19:48:49 ERROR: AcceptToMemoryPool : inputs already spent
262015-01-19 19:49:04 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
272015-01-19 19:49:04 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
282015-01-19 19:49:12 UpdateTip: new best=000000000000000009e1cc70b4e19e7f031b2e3aa0de38cf72e4a87802864887  height=339688  log2_work=82.046111  tx=57250407  date=2015-01-19 19:49:40 progress=1.000001  cache=13443
292015-01-19 19:49:33 ERROR: AcceptToMemoryPool : inputs already spent
302015-01-19 19:49:53 ERROR: AcceptToMemoryPool : inputs already spent
312015-01-19 19:50:15 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=932
322015-01-19 19:50:50 ERROR: AcceptToMemoryPool : inputs already spent
332015-01-19 19:51:33 ERROR: AcceptToMemoryPool : inputs already spent
342015-01-19 19:52:49 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=934
352015-01-19 19:52:49 ERROR: AcceptToMemoryPool : inputs already spent
362015-01-19 19:54:43 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=339688, us=84.215.171.162:8333, peer=935
372015-01-19 19:54:49 ERROR: AcceptToMemoryPool : inputs already spent
382015-01-19 19:55:15 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=936
392015-01-19 19:57:28 UpdateTip: new best=00000000000000000fbce81a5c1457f5cf720541454c58d62eb4e777c14f2af4  height=339689  log2_work=82.046166  tx=57251146  date=2015-01-19 19:56:37 progress=0.999999  cache=14697
402015-01-19 19:57:49 ERROR: AcceptToMemoryPool : inputs already spent
412015-01-19 19:57:50 ERROR: AcceptToMemoryPool : inputs already spent

laanwj commented at 8:07 pm on January 19, 2015: member

It is accepting blocks succesfully. AcceptToMemoryPool aren’t an issue, they are errors in a transaction that a peer is sending.

eN0Rm commented at 8:09 pm on January 19, 2015: none

Thanks, then I’ll close it.

eN0Rm closed this on Jan 19, 2015

carnesen commented at 7:07 pm on February 3, 2015: contributor

I’ve hit this a couple times now running Bitcoin Core Daemon version v0.10.0rc3 on Debian 7.

carnesen commented at 10:49 pm on February 7, 2015: contributor

My bitcoind crashed a third time this week now on “ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large”. I’m running a pretty vanilla setup: v0.10.0rc3 on an EBS-backed AWS instance whose only job in life is to run bitcoind, which does nothing but act as a relay node with the default params. I’ll take the opportunity of my bitcoind being down to upgrade to rc4, which I see was released yesterday, and see if that helps. Please let me know if there’s anything I can do to provide information about this recurring failure mode in the release candidate.

eN0Rm commented at 12:14 pm on February 15, 2015: none

Assert

0bitcoind: main.cpp:3377: void ProcessGetData(CNode*): Assertion `!"cannot load block from disk"' failed.
1Aborted (core dumped)

debug.log

 02015-02-15 04:29:04 UpdateTip: new best=0000000000000000136de7125e7ffa9197aecec1a690c1d064e2d862138f5bf4  height=343527  log2_work=82.236249  tx=59677940  date=2015-02-15 04:27:54 progress=0.999998  cache=11784
 12015-02-15 04:31:05 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=850
 22015-02-15 04:31:52 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=851
 32015-02-15 04:31:58 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:4.19/: version 70001, blocks=343071, us=127.0.0.1:8333, peer=852
 42015-02-15 04:31:58 Added time data, samples 162, offset -5 (+0 minutes)
 52015-02-15 04:33:32 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=343526, us=84.215.171.162:8333, peer=853
 62015-02-15 04:35:55 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=343527, us=84.215.171.162:8333, peer=854
 72015-02-15 04:35:56 receive version message: /Satoshi:0.8.6/: version 70001, blocks=343524, us=84.215.171.162:8333, peer=855
 82015-02-15 04:36:57 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=856
 92015-02-15 04:37:04 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=857
102015-02-15 04:38:25 receive version message: /BTCXchange.ro Mapper:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=858
112015-02-15 04:38:49 receive version message: /Satoshi:0.8.6/: version 70001, blocks=343524, us=84.215.171.162:8333, peer=859
122015-02-15 04:38:51 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=860
132015-02-15 04:39:37 receive version message: /Satoshi:0.8.6/: version 70001, blocks=343524, us=84.215.171.162:8333, peer=861
142015-02-15 04:40:33 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=343527, us=84.215.171.162:8333, peer=862
152015-02-15 04:40:38 receive version message: /Shibetoshi:0.1/: version 70002, blocks=0, us=[::]:0, peer=863
162015-02-15 04:40:40 ResendWalletTransactions()
172015-02-15 04:42:33 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=864
182015-02-15 04:45:25 Pre-allocating up to position 0x1000000 in rev00230.dat
192015-02-15 04:45:25 UpdateTip: new best=000000000000000011642065c017236e7e4833c05eb3fd3790ace2c43e363449  height=343528  log2_work=82.236297  tx=59678662  date=2015-02-15 04:54:19 progress=1.000012  cache=2600
202015-02-15 04:45:54 ping timeout: 1200.015820s
212015-02-15 04:46:20 UpdateTip: new best=00000000000000001455720d22a9091679ee7997d7e81f90f6b986def4e28ee2  height=343529  log2_work=82.236346  tx=59678722  date=2015-02-15 04:45:32 progress=0.999999  cache=3227
222015-02-15 04:46:26 socket recv error Connection reset by peer (104)
232015-02-15 04:46:59 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=865
242015-02-15 04:47:22 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=343527, us=84.215.171.162:8333, peer=866
252015-02-15 04:47:25 ERROR: CheckTransaction(): vin empty
262015-02-15 04:47:25 ERROR: AcceptToMemoryPool: CheckTransaction failed
272015-02-15 04:47:25 Misbehaving: 83.222.55.161:63380 (30 -> 40)
282015-02-15 04:47:25 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
292015-02-15 04:47:25 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
302015-02-15 04:47:26 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
312015-02-15 04:47:26 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
322015-02-15 04:47:26 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
332015-02-15 04:47:26 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
342015-02-15 04:47:26 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
352015-02-15 04:47:26 ERROR: AcceptToMemoryPool: nonstandard transaction: dust
362015-02-15 04:47:44 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=867
372015-02-15 04:47:51 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large

Info from init

 02015-02-15 12:14:00 Bitcoin version v0.10.99.0-d8ed3bd (2015-02-13 22:11:30 -0800)
 12015-02-15 12:14:00 Using OpenSSL version OpenSSL 1.0.1f 6 Jan 2014
 22015-02-15 12:14:00 Using BerkeleyDB version Berkeley DB 5.3.28: (September  9, 2013)
 32015-02-15 12:14:00 Default data directory /home/fredrik/.bitcoin
 42015-02-15 12:14:00 Using data directory /home/fredrik/.bitcoin
 52015-02-15 12:14:00 Using config file /home/fredrik/.bitcoin/bitcoin.conf
 62015-02-15 12:14:00 Using at most 200 connections (1024 file descriptors available)
 72015-02-15 12:14:00 Using 0 threads for script verification
 82015-02-15 12:14:00 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0)
 92015-02-15 12:14:00 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0)
102015-02-15 12:14:00 Using wallet wallet.dat
112015-02-15 12:14:00 init message: Verifying wallet...
122015-02-15 12:14:00 CDBEnv::Open: LogDir=/home/fredrik/.bitcoin/database ErrorFile=/home/fredrik/.bitcoin/db.log
132015-02-15 12:14:01 Bound to [::]:8333
142015-02-15 12:14:01 Bound to 0.0.0.0:8333
152015-02-15 12:14:01 init message: Loading block index...
162015-02-15 12:14:01 Opening LevelDB in /home/fredrik/.bitcoin/blocks/index
172015-02-15 12:14:01 Opened LevelDB successfully
182015-02-15 12:14:01 Opening LevelDB in /home/fredrik/.bitcoin/chainstate
192015-02-15 12:14:02 Opened LevelDB successfully
202015-02-15 12:14:07 LoadBlockIndexDB: last block file = 230
212015-02-15 12:14:07 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=271, size=125224105, heights=343260...343527, time=2015-02-13...2015-02-15)
222015-02-15 12:14:07 Checking all blk files are present...
232015-02-15 12:14:07 LoadBlockIndexDB(): transaction index disabled
242015-02-15 12:14:07 LoadBlockIndexDB(): hashBestChain=0000000000000000136de7125e7ffa9197aecec1a690c1d064e2d862138f5bf4 height=343527 date=2015-02-15 04:27:54 progress=0.999362
252015-02-15 12:14:07 init message: Verifying blocks...
262015-02-15 12:14:07 Verifying last 288 blocks at level 3

I’ve run fsck on host drive. Bitcoin node is running in kvm guest. No bad blocks.

eN0Rm reopened this on Feb 15, 2015

MineForeman commented at 0:57 am on February 17, 2015: none

I have had this error 3 times now over the past few weeks. I am using the BitcoinXT fork though (https://github.com/bitcoinxt/bitcoinxt).

eN0Rm commented at 5:49 pm on February 18, 2015: none

 02015-02-17 18:40:05 ResendWalletTransactions()
 12015-02-17 18:40:51 ERROR: CheckTransaction(): vin empty
 22015-02-17 18:40:51 ERROR: AcceptToMemoryPool: CheckTransaction failed
 32015-02-17 18:40:51 Misbehaving: 96.46.28.172:33442 (0 -> 10)
 42015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
 52015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
 62015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
 72015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
 82015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
 92015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
102015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
112015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
122015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
132015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
142015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
152015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
162015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
172015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
182015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
192015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
202015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
212015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
222015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
232015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
242015-02-17 18:40:51 ERROR: AcceptToMemoryPool: inputs already spent
252015-02-17 18:43:19 receive version message: /Satoshi:0.8.6/: version 70001, blocks=343923, us=84.215.171.162:8333, peer=1550
262015-02-17 18:43:31 UpdateTip: new best=0000000000000000001a0590e06217704b4630ed2d6856749444a6abd596c955  height=343926  log2_work=82.255417  tx=59933233  date=2015-02-17 18:42:21 progress=0.999998  cache=16219
272015-02-17 18:43:53 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=1551
282015-02-17 18:44:42 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=1552
292015-02-17 18:45:02 receive version message: /Satoshi:0.8.6/: version 70001, blocks=343923, us=84.215.171.162:8333, peer=1553
302015-02-17 18:45:34 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=343925, us=84.215.171.162:8333, peer=1554
312015-02-17 18:46:01 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large

Exit status from bitcoind

0fredrik@bitNORm:~> bitcoind 
1bitcoind: main.cpp:3377: void ProcessGetData(CNode*): Assertion `!"cannot load block from disk"' failed.
2Aborted (core dumped)

jlopp commented at 5:43 pm on February 20, 2015: contributor

Reporting in: just experienced the same error, also on an EBS-backed AWS instance. The node in question was built from the 0.10.0 release and was completely synced.

“ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large”

No errors reported by the AWS instance that indicate any issues with EBS.

paveljanik commented at 9:17 pm on February 20, 2015: contributor

We really need to see the blocks directory when this happens to be able to analyse this…

eN0Rm commented at 9:24 pm on February 20, 2015: none

What do you mean? What do you need access to? ssh?

On Fri, Feb 20, 2015 at 10:17 PM, paveljanik notifications@github.com wrote:

We really need to see the blocks directory when this happen to be able to analyse this…

— Reply to this email directly or view it on GitHub #5668 (comment).

Mvh -fredrik-normann-

Sent from my Gmail Account

paveljanik commented at 9:39 pm on February 20, 2015: contributor

No! blocks directory only. But I can’t help here because of my connectivity ;-)

eN0Rm commented at 9:42 pm on February 20, 2015: none

Not a big deal with giving you ssh access. It’s just a vm and I can set a random password for that user. But maybe I can help with checking out the blocks dir.

eN0Rm commented at 9:44 pm on February 20, 2015: none

http://paste.ubuntu.com/10330823/ <– This is what it looks like

jlopp commented at 8:17 pm on February 23, 2015: contributor

My AWS node crashed on this error again and is currently in its post-crash state. Is there any information I can provide that would help to debug the issue? Alternatively, should I set debug logging on “db” and start the node up again?

jlopp commented at 9:04 pm on February 23, 2015: contributor

Digging more into the debug.log for the recent crash:

2015-02-21 05:08:50 UpdateTip: new best=00000000000000000e0d175f7f6c657d6dce5e0d909cafeb643b82d776110f25 height=344475 log2_work=82.281381 tx=60272692 date=2015-02-21 05:08:53 progress=1.000000 cache=5508 [SNIP peers connecting] 2015-02-21 05:18:21 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large

Without actually having a stack trace to reference, my best guess based upon the timing of the crash was that bitcoind was in the ConnectTip function of main.cpp at line 2061. This is because the crash occurred right when Block #344476 was announced: https://blockchain.info/block/0000000000000000080315867d97c97a55f23fea6998c9067e73c16abc73b791

However, it also makes sense that it’s occurring at main.cpp:3377: void ProcessGetData(CNode*) if this is some sort of race condition. Perhaps bitcoind is handling a GetData request for this block before it has been completely stored to disk? The timing seems too coincidental.

Aside from the fact that there seems to be a bug in the logic that streams the blocks from disk, is it really the appropriate behavior for this error to result in a shutdown of bitcoind? It seems like we could handle the error more gracefully.

ajweiss commented at 9:39 pm on February 23, 2015: contributor

Just out of curiosity, what happens if you fsck your EBS volume?

MineForeman commented at 9:55 pm on February 23, 2015: none

@ajweiss I get;-

cloudimg-rootfs: clean, 94748/3932160 files, 11933728/15725626 blocks

I have even gone as far as provisioning a new ebs disk, re-syncing to the new disk and I will still occasionally get the crash. Nothing has shown up in a fsck yet.

jlopp commented at 9:56 pm on February 23, 2015: contributor

fsck reports that the file system is clean

ajweiss commented at 10:07 pm on February 23, 2015: contributor

This is very interesting. It hasn’t shown up on any of our nodes, but none of ours are on AWS/EBS. Are all of you that are seeing this on AWS/EBS? Which AMI?

jlopp commented at 10:16 pm on February 23, 2015: contributor

This has only occurred to me on nodes running AWS where bitcoind’s data directory is on an EBS volume. This instance is using the ubuntu-trusty-14.04-amd64-server-20150123 AMI

MineForeman commented at 10:25 pm on February 23, 2015: none

ubuntu-trusty-14.04-amd64-server-20140927 (ami-3d50120d) on my AWS install.

eN0Rm commented at 9:15 am on February 24, 2015: none

libvirt-bin and kvm virtulized on a Intel NUC @ home. fsck both on host and guest reports that filesystem is clean.

On Mon, Feb 23, 2015 at 11:25 PM, Neil Fincham notifications@github.com wrote:

ubuntu-trusty-14.04-amd64-server-20140927 (ami-3d50120d) on my AWS install.

— Reply to this email directly or view it on GitHub #5668 (comment).

Mvh -fredrik-normann-

Sent from my Gmail Account

jlopp commented at 9:40 pm on March 4, 2015: contributor

Update:

We created a new EBS volume on the machine that was crashing from this error and rsynced the block chain data over to it. We then ran bitcoind while pointing to that data. It experienced the same error and crashed every 12 - 24 hours.

We then took the original EBS volume and rsynced block chain data from a different AWS machine (also running 0.10) that has not been crashing. We started bitcoind pointing to this data and it has been running solidly for nearly a week without crashing.

Conclusions: while this issue may be somehow related to EBS, it doesn’t appear to be due to a specific EBS volume being “bad.” The problem appears to be in the block chain data itself. We have a copy of this data that we can make available if any core developers want to look at it.

bloxton commented at 7:39 am on March 5, 2015: none

I’m having the same issue. I’m running bitcoind v0.10 (no GUI, no wallet compiled in) on a Raspberry Pi 2 running the latest Raspbian. I know the issue appears disk-related but I don’t believe my root filesystem - or reads/writes to it - are the problem. Root has been moved off the micro sdcard and onto a Sandisk Extreme Pro USB drive. When the problem occurs (every 12-24 hours typically) I simply restart bitcoind, it catches up on the blockchain and it continues processing. fsck has not found any corruptions. Latest instance of the issue:

2015-03-04 10:26:40 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=118.208.160.171:8333, peer=7925 2015-03-04 10:27:11 ERROR: AcceptToMemoryPool : inputs already spent 2015-03-04 10:27:18 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=118.208.160.171:8333, peer=7926 2015-03-04 10:27:23 receive version message: /Satoshi:0.8.6/: version 70001, blocks=346126, us=118.208.160.171:8333, peer=7927 2015-03-04 10:27:38 ERROR: AcceptToMemoryPool : inputs already spent 2015-03-04 10:27:53 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=118.208.160.171:8333, peer=7928 2015-03-04 10:28:05 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=118.208.160.171:8333, peer=7929 2015-03-04 10:30:18 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large bitcoind: main.cpp:3335: void ProcessGetData(CNode*): Assertion `!“cannot load block from disk”’ failed.

Happy to collect further data if it will help to diagnose.

jlopp commented at 9:36 pm on March 5, 2015: contributor

You can grab our block data here: http://bg.bitcoind.debug.s3-website-us-west-2.amazonaws.com/bitcoind.tar

eN0Rm commented at 8:49 pm on March 7, 2015: none

 02015-03-07 15:00:19 UpdateTip: new
 1best=000000000000013071bb9b1c874a6600187d1656a7371f5ab2c38f6cac6664d8
 2 height=238620  log2_work=70.168506  tx=18645945  date=2013-05-30 03:44:23
 3progress=0.137042  cache=69964
 42015-03-07 15:00:19 UpdateTip: new
 5best=00000000000000882460df59ac4558dd8cc110972e0c71473131aa972c7d528e
 6 height=238621  log2_work=70.168563  tx=18646442  date=2013-05-30 03:55:22
 7progress=0.137046  cache=70127
 82015-03-07 15:00:19 Pre-allocating up to position 0x700000 in rev00063.dat
 92015-03-07 15:00:19 UpdateTip: new
10best=00000000000000b7e87a19a995e882bd33c48e6e2f0ec20761dd29291f091d3f
11 height=238622  log2_work=70.168619  tx=18647017  date=2013-05-30 04:08:50
12progress=0.137050  cache=70400
132015-03-07 15:00:19 UpdateTip: new
14best=0000000000000022c1a2e7f6cdbe8b9b018196bd838a38cfcbe673023bb04dc6
15 height=238623  log2_work=70.168676  tx=18647376  date=2013-05-30 04:14:13
16progress=0.137053  cache=70590
172015-03-07 15:00:19 UpdateTip: new
18best=00000000000000f103611c15c2e28fc784d0761e293ebc7c15d013dd119d6e83
19 height=238624  log2_work=70.168733  tx=18647705  date=2013-05-30 04:29:25
20progress=0.137055  cache=71009
212015-03-07 15:00:19 UpdateTip: new
22best=000000000000002c7f7234f4290e5012b26a98ce4b045fa656e002fdbc57aacb
23 height=238625  log2_work=70.16879  tx=18648399  date=2013-05-30 04:39:28
24progress=0.137060  cache=71407
252015-03-07 15:00:20 UpdateTip: new
26best=0000000000000030f5f983e7718c6af574fcce1230cb5e8f126755a4fe51f604
27 height=238626  log2_work=70.168846  tx=18648606  date=2013-05-30 04:41:21
28progress=0.137062  cache=71412
292015-03-07 15:00:20 UpdateTip: new
30best=00000000000000485937686cfc567e9cec3e4126f3c408d99811b5edf545e3f8
31 height=238627  log2_work=70.168903  tx=18648734  date=2013-05-30 04:42:59
32progress=0.137063  cache=71500
332015-03-07 15:00:20 UpdateTip: new
34best=00000000000000712d0309b7c1bf22ad47c91ec518d84334bd29a2b761443387
35 height=238628  log2_work=70.16896  tx=18648862  date=2013-05-30 04:46:04
36progress=0.137064  cache=71985
372015-03-07 15:00:20 UpdateTip: new
38best=0000000000000119a25e432b0ef56796aae611c4cc778b088b73c67ed1b83265
39 height=238629  log2_work=70.169017  tx=18649257  date=2013-05-30 04:52:24
40progress=0.137067  cache=72155
412015-03-07 15:00:21 ERROR: ReadBlockFromDisk : OpenBlockFile failed
42a

Will it be usefull to have a look at this file? rev00063.dat

On Thu, Mar 5, 2015 at 10:37 PM, Jameson Lopp notifications@github.com wrote:

If you install the aws cli (brew install awscli on OS X) on your machine you should be able to access our block data here:

aws s3 ls s3://bg.bitcoind.debug/

— Reply to this email directly or view it on GitHub #5668 (comment).

Mvh -fredrik-normann-

Sent from my Gmail Account

theuni commented at 6:44 pm on March 9, 2015: member

For those who have run into this: have you previously re-indexed? If so, it could be related to #5864

jlopp commented at 7:08 pm on March 9, 2015: contributor

Our nodes do index all transactions, though to my knowledge no re-index was performed on the node that was experiencing this issue; it was a brand new node that began crashing within a few days of us bringing it online.

rromanchuk commented at 7:43 pm on March 9, 2015: none

Just got this on aws/ebs setup as well. @theuni i can confirm we use indexed as well, but have never re-indexed.

Amazon Linux AMI release 2014.09

MineForeman commented at 8:01 pm on March 9, 2015: none

No re-index here either BUT i have added a new EBS volume, mounted it at /home and have done a full sync of the blockchain.

For the past 5 days it has been stable whereas the original has gone down 3 times (I run daemontools so it has not been a huge issue). I am about to switch to using it for my apps and will report back if there are any issues.

theuni commented at 8:07 pm on March 9, 2015: member

@jlopp / @rromanchuk It’d be very useful if you could run @laanwj’s script from here: #5850 on the block files in order to narrow down the problem. I’d really like to look at the block files, but it’ll take me forever to download the whole thing.

Steps to run: wget https://download.visucore.com/tmp/2015_03_arm_error.tar.xz untar and cd into the dir ./list-blocks.py /path/to/blkfile.dat

You’ll need to do that for each .dat file, so it’d probably be quicker to write up a quick script. In the output, you’re looking for “Overlap with last block” or “Block doesn’t start at expected position”. If you find one of those, we could start by analyzing that file rather than the entire set.

rromanchuk commented at 8:42 pm on March 9, 2015: none

@MineForeman you might be on to something, i totally forgot I ALSO detached and remounted a volume (resized)

jlopp commented at 8:43 pm on March 9, 2015: contributor

We’ve uploaded the bad block files to http://bg.bitcoind.debug.s3-website-us-west-2.amazonaws.com/badblocks.tar

 0$ for f in blocks/blk*.dat; do python list-blocks.py $f | grep -i block; done
 1Input file blocks/blk00023.dat
 205741ff8-535fbbaf 05742000 f892b2437a142747e6b593176c767f0e9dad30d4ba52791aacdc1bebd23dff58     -1 Overlap with last block (05741ff8,535fbbaf)
 305742001-b1fcbbb8 05742009 09e6829d8dffae38ad5c10343f7347b421db2294d6360eab8256f932fc8cce69     -1 Overlap with last block (05742001,b1fcbbb8)
 40574b23d-53604df4 0574b245 05fd918c3e7456a26d3f6e9d72777558eb4b7c3a3da5dde0556993592fb8add3     -1 Overlap with last block (0574b23d,53604df4)
 50574b246-53604dfd 0574b24e 39cc348070b9a7a8e3bbb3caf973aa7b80651e1f012dd86fe84656a8c3506c4e     -1 Overlap with last block (0574b246,53604dfd)
 60574b262-df297163 0574b26a 46fbe70e2fe21609042d1c2f7d7de78be7d6a4739163fc8dffe71c02313604e2     -1 Overlap with last block (0574b262,df297163)
 70574b266-df297167 0574b26e f521065b25eb4f3a0431132084f19bba0f2dc3ae39108b0fed414e5ce0129320     -1 Overlap with last block (0574b266,df297167)
 80574b26a-df29716b 0574b272 5579819e0c88b3759b28dc8f7526af5e2b457f7c3a55ca89cd8ffd3e812b6e7b     -1 Overlap with last block (0574b26a,df29716b)
 90574b26e-06213b6f 0574b276 de627541c042f591b352a9ccdcb844cd51820c7b73c1a2cc402fafd65ab06f8c     -1 Overlap with last block (0574b26e,06213b6f)
100574fe88-05760c1b 0574fe90 00000000000005d9f163bc54c2030dbf59c0076ae16af3bbb0dd3ce385a79f5d 200333 Overlap with last block (0574fe88,05760c1b)
11
12Input file blocks/blk00047.dat
130546b882-d5cf1c8f 0546b88a cd0f5125199d2e872ca2f894f292b42eeb6646992755ec85d7dfdabdd5fd8ac5     -1 Overlap with last block (0546b882,d5cf1c8f)
14054fe5eb-054fe6d2 054fe5f3 0000000000000048464248019cf5979a30bcb788ce575f547456496597e164f4 225723 Overlap with last block (054fe5eb,054fe6d2)
15
16Input file blocks/blk00075.dat
1702d5100b-02d8dcee 02d51013 00000000000000126cd5ccc13d8b7fd24b4ec1c5de73c0299d5ace2be3c475ed 252261 Overlap with last block (02d5100b,02d8dcee)
18
19Input file blocks/blk00103.dat
200146ba57-2537f817 0146ba5f d549a393beda9d833022cdf966e63b543a6fe424897a19310280a0dac59cf51d     -1 Overlap with last block (0146ba57,2537f817)
2101477757-0147e654 0147775f 0000000000000003241adf2d3cd721501778a5b830377ff7025d0d84d71173be 276657 Overlap with last block (01477757,0147e654)
22
23Input file blocks/blk00126.dat
2406ad4850-8baf6e58 06ad4858 a8528116edd7914f7c132d5bbbb4ee147a34a53a0e1f4e60fdd1d445ddb987ab     -1 Overlap with last block (06ad4850,8baf6e58)
2506b05cd6-06b666fe 06b05cde 0000000000000000aec2cc916b840c1f9345153745c9380831825497d976cdf7 292616 Overlap with last block (06b05cd6,06b666fe)
2606c33711-8bc55d19 06c33719 413000f2aa16953e9f1eb86e72074e798b5a3cb4aafc5fe2df0e06199553855b     -1 Overlap with last block (06c33711,8bc55d19)
2706c82949-06cd36c2 06c82951 0000000000000000c68462ab6298c8d44b8197fe44ef6ea729941f3c6d8f9237 292635 Overlap with last block (06c82949,06cd36c2)
28
29Input file blocks/blk00127.dat
3000162cf4-851852fc 00162cfc 2359b6fe12feae9c761c00ffe9885c7c5c28200b92fcb7081c0b3f6227ccbf54     -1 Overlap with last block (00162cf4,851852fc)
31001b5ad2-001cfd27 001b5ada 00000000000000009e578f4333ea2f3970d1698decb0b6db39b7e65c4bb2edd3 292709 Overlap with last block (001b5ad2,001cfd27)
32021ba0f2-871dc6fa 021ba0fa 93d796919bd2906217839eba4257b0e4e949a5304cc100f6f11fc2acd3a462eb     -1 Overlap with last block (021ba0f2,871dc6fa)
330224e47a-02265f3f 0224e482 000000000000000034d932ad79ca48c1a4dab27c975213edf9e3cea7ecdcde5c 292873 Overlap with last block (0224e47a,02265f3f)
34
35Input file blocks/blk00135.dat
3603686a64-886a906c 03686a6c b0906ed7a180d0fe92f92697acdbc8e79ef6b06c53b88122b7def6d3d80ca0d9     -1 Overlap with last block (03686a64,886a906c)
37036b2d8d-036b6e48 036b2d95 000000000000000083725fb70cdc7c7d6182f521f11b8c27a0342e53c1e90c53 297916 Overlap with last block (036b2d8d,036b6e48)
3803e2c36a-88e4e972 03e2c372 b480a3912cb6d1c463e87e1a8d0e97ee91dc35405167a911bcba58ec0b7b4361     -1 Overlap with last block (03e2c36a,88e4e972)
3903e4035a-03e5dfc5 03e40362 0000000000000000935dfee26f6412e15e4714cab7b10c558123ece412455626 297999 Overlap with last block (03e4035a,03e5dfc5)
4004fb2946-89fd4f4e 04fb294e 93f86dd0ebb350799b9163cdcf7125e5b24311fab57b74a94da3939bd611360b     -1 Overlap with last block (04fb2946,89fd4f4e)
4104fd1732-04fdf146 04fd173a 000000000000000041e279661f1de9f5c408fa295ebd10bbdfd29d2f24c710e4 298080 Overlap with last block (04fd1732,04fdf146)
42
43Input file blocks/blk00168.dat
4401e2210e-cecfe62c 01e22116 6dfcd17e90d976033c1eb409ef5c2319dafe9221b7db435abcbf7690b6803e8d     -1 Overlap with last block (01e2210e,cecfe62c)
4501e37831-01e81bf4 01e37839 000000000000000020d471f2f6d2012fe95cef6862a0b57e04d9cc64df75b18d 317074 Overlap with last block (01e37831,01e81bf4)

rromanchuk commented at 9:11 pm on March 9, 2015: none

@theuni

 0[ec2-user]$  for f in /home/ec2-user/.bitcoin/blocks/blk*.dat; do python list-blocks.py $f | grep -i block; done
 1Input file /home/ec2-user/.bitcoin/blocks/blk00000.dat
 2Input file /home/ec2-user/.bitcoin/blocks/blk00001.dat
 3Input file /home/ec2-user/.bitcoin/blocks/blk00002.dat
 4Input file /home/ec2-user/.bitcoin/blocks/blk00003.dat
 5Input file /home/ec2-user/.bitcoin/blocks/blk00004.dat
 6Input file /home/ec2-user/.bitcoin/blocks/blk00005.dat
 7Input file /home/ec2-user/.bitcoin/blocks/blk00006.dat
 8Input file /home/ec2-user/.bitcoin/blocks/blk00007.dat
 9Input file /home/ec2-user/.bitcoin/blocks/blk00008.dat
10Input file /home/ec2-user/.bitcoin/blocks/blk00009.dat
11Input file /home/ec2-user/.bitcoin/blocks/blk00010.dat
12Input file /home/ec2-user/.bitcoin/blocks/blk00011.dat
13Input file /home/ec2-user/.bitcoin/blocks/blk00012.dat
14Input file /home/ec2-user/.bitcoin/blocks/blk00013.dat
15Input file /home/ec2-user/.bitcoin/blocks/blk00014.dat
16Input file /home/ec2-user/.bitcoin/blocks/blk00015.dat
17Input file /home/ec2-user/.bitcoin/blocks/blk00016.dat
18Input file /home/ec2-user/.bitcoin/blocks/blk00017.dat
19Input file /home/ec2-user/.bitcoin/blocks/blk00018.dat
20Input file /home/ec2-user/.bitcoin/blocks/blk00019.dat
21Input file /home/ec2-user/.bitcoin/blocks/blk00020.dat
22Input file /home/ec2-user/.bitcoin/blocks/blk00021.dat
23Input file /home/ec2-user/.bitcoin/blocks/blk00022.dat
24Input file /home/ec2-user/.bitcoin/blocks/blk00023.dat
25054e71db-533a0d92 054e71e3 f892b2437a142747e6b593176c767f0e9dad30d4ba52791aacdc1bebd23dff58     -1 Overlap with last block (054e71db,533a0d92)
26054e71e4-b1d70d9b 054e71ec 09e6829d8dffae38ad5c10343f7347b421db2294d6360eab8256f932fc8cce69     -1 Overlap with last block (054e71e4,b1d70d9b)
27054f0420-533a9fd7 054f0428 05fd918c3e7456a26d3f6e9d72777558eb4b7c3a3da5dde0556993592fb8add3     -1 Overlap with last block (054f0420,533a9fd7)
28054f0429-533a9fe0 054f0431 39cc348070b9a7a8e3bbb3caf973aa7b80651e1f012dd86fe84656a8c3506c4e     -1 Overlap with last block (054f0429,533a9fe0)
29054f0445-df03c346 054f044d 46fbe70e2fe21609042d1c2f7d7de78be7d6a4739163fc8dffe71c02313604e2     -1 Overlap with last block (054f0445,df03c346)
30054f0449-df03c34a 054f0451 f521065b25eb4f3a0431132084f19bba0f2dc3ae39108b0fed414e5ce0129320     -1 Overlap with last block (054f0449,df03c34a)
31054f044d-df03c34e 054f0455 5579819e0c88b3759b28dc8f7526af5e2b457f7c3a55ca89cd8ffd3e812b6e7b     -1 Overlap with last block (054f044d,df03c34e)
32054f0451-05fb8d52 054f0459 de627541c042f591b352a9ccdcb844cd51820c7b73c1a2cc402fafd65ab06f8c     -1 Overlap with last block (054f0451,05fb8d52)
33054f506b-054fc7dd 054f5073 0000000000000315cfdf7f2d83f8632f5042a6c346b62476ab4c7cc48e9c8695 199768 Overlap with last block (054f506b,054fc7dd)
34Input file /home/ec2-user/.bitcoin/blocks/blk00024.dat
35Input file /home/ec2-user/.bitcoin/blocks/blk00025.dat
36Input file /home/ec2-user/.bitcoin/blocks/blk00026.dat
37Input file /home/ec2-user/.bitcoin/blocks/blk00027.dat
3804ef0373-04ef1172 04ef037b 0000000000000451285856e4260cc8c7f89f2a1c9f966f38a1bf3a65e618707c 205993 Overlap with last block (04ef0373,04ef1172)
39Input file /home/ec2-user/.bitcoin/blocks/blk00028.dat
40Input file /home/ec2-user/.bitcoin/blocks/blk00029.dat
41Input file /home/ec2-user/.bitcoin/blocks/blk00030.dat

theuni commented at 9:38 pm on March 9, 2015: member

@rromanchuk interesting, blk00023.dat looks to be pretty well clobbered (notice the hashes). Can you make that file available somewhere?

rromanchuk commented at 9:53 pm on March 9, 2015: none

@theuni https://dl.dropboxusercontent.com/u/13326666/blk00023.dat.zip

ajweiss commented at 10:00 pm on March 9, 2015: contributor

Interestingly, the datadir that @jlopp posted also shows bogus hash values… Even more interestingly, they match!

052625 05741ff8-535fbbaf 05742000 f892b2437a142747e6b593176c767f0e9dad30d4ba52791aacdc1bebd23dff58     -1 WARN Overlap with last block (05741ff8,535fbbaf)
19 05742001-b1fcbbb8 05742009 09e6829d8dffae38ad5c10343f7347b421db2294d6360eab8256f932fc8cce69     -1 WARN Overlap with last block (05742001,b1fcbbb8)
237436 0574b23d-53604df4 0574b245 05fd918c3e7456a26d3f6e9d72777558eb4b7c3a3da5dde0556993592fb8add3     -1 WARN Overlap with last block (0574b23d,53604df4)
39 0574b246-53604dfd 0574b24e 39cc348070b9a7a8e3bbb3caf973aa7b80651e1f012dd86fe84656a8c3506c4e     -1 WARN Overlap with last block (0574b246,53604dfd)
428 0574b262-df297163 0574b26a 46fbe70e2fe21609042d1c2f7d7de78be7d6a4739163fc8dffe71c02313604e2     -1 WARN

theuni commented at 10:13 pm on March 9, 2015: member

@ajweiss whoa.. really nice catch. Ok, @rromanchuk and @jlopp, where did you get your initial blocks? Did you bootstrap from somewhere?

jlopp commented at 10:17 pm on March 9, 2015: contributor

Synced normally over the network using the 0.10 release.

rromanchuk commented at 10:23 pm on March 9, 2015: none

Also did not bootstrap

theuni commented at 10:31 pm on March 9, 2015: member

Ok, false alrm. A quick look shows the magic bytes repeated several times very close together. See tx 4ce5a9b7141543cd8bb0a18caa94087f8d87356559e00c823f4493ef0826ef77 .

The script thinks that’s the start of a block, but those should just be ignored.

MineForeman commented at 10:33 pm on March 9, 2015: none

No bootstrap here either BUT I am using non-standard fork (BIP 64 UTXO’s) so take everything I say with a grain of salt.

theuni commented at 10:44 pm on March 9, 2015: member

I just compared the values from @rromanchuk and @jlopp against my own, and there are several instances of magic bytes that we need to ignore. The ones still unaccounted for are: @rromanchuk: blk00027.dat @jlopp: blk00075.dat

It would be great if we could get those hosted somewhere.

jlopp commented at 10:47 pm on March 9, 2015: contributor

@theuni You can grab my blk00075.dat from http://bg.bitcoind.debug.s3-website-us-west-2.amazonaws.com/badblocks.tar

theuni commented at 0:16 am on March 10, 2015: member

Something interesting in blk00075.dat: The following blocks are repeated:

The hashes appear to be the same and correct correct for each duplication. The first repeat (252262) comes directly after the overlapping one.

Edit: if anyone’s curious, here’s more detail

 002a33b62-02a4fe0c 02a33b6a 000000000000000ef3824393dd8ec05f1efbd128b492d0c798b08b23b9ed07ba 252262 
 102a4fe0c-02a5b3ab 02a4fe14 0000000000000026900535948804f23466b4f367a60da6fdbec559b96221e9a0 252240 
 202a5b3ab-02a83411 02a5b3b3 00000000000000186acb75140c13a3d53fe66c21cb3f9848c4f9072cfe4ebb48 252226 
 302a83411-02a90bfa 02a83419 000000000000001f802536a22331535bb6c46c74ebb05277dacc5048c76d77c1 252270 
 402a90bfa-02a99f4f 02a90c02 0000000000000047ff2009cd7168bfb3e529e29841c35de048294bc0e8346372 252266 
 502a99f4f-02aab303 02a99f57 000000000000003c6bbd6df1f346c95be8741743f44973efc19a21106929bdb7 252244 
 602aab303-02abeca2 02aab30b 000000000000002c88ad190715e287ff1e0b539ebf5a3db5895b65894e753228 252236 
 702abeca2-02ae69de 02abecaa 0000000000000036abb445a7266e2d7babd9d4e1754b5d0054b60ffd5d242f47 252231 
 802ae69de-02b0152d 02ae69e6 0000000000000037234bbe5fda97175b8aab8d4155597d8c57298f18e4fff735 252258 
 902b0152d-02b40183 02b01535 000000000000001d976e500e4e880c9d16a8e7a30eff2b4638f1ce8e8a3d29c4 252276 
1002b40183-02b7c5c3 02b4018b 000000000000002c4853f34e72ed741abe5c65833a23d694dbcc387fb3617812 252176 
1102b7c5c3-02b823d5 02b7c5cb 0000000000000048ba25c4d2336337ca63bd78b1dde0ac1b47d3bfb26eae43e4 252273 
1202b823d5-02c0c269 02b823dd 000000000000001514684f6f04348420660fa46ff02b36104623f9fcdfc75052 252241 
1302c0c269-02c39db0 02c0c271 000000000000001eefcf81515912cd4c5fadfcbeb9846165a850b924cbc354d1 252242 
1402c39db0-02c61c04 02c39db8 0000000000000021d2ca3bb70769caf1fd13957fe967a573581291a37f2fb278 252281 
1502c61c04-02c9e940 02c61c0c 000000000000004a85ed53ee82ebd955f58e73e89a2b8b840ef73c545e0f8f39 252278 
1602c9e940-02cb0b6d 02c9e948 000000000000002c8e5c5d9a46da003a0ff8ec9d3dc42b0216129e8bd70fef08 252249 
1702cb0b6d-02cc6e3a 02cb0b75 00000000000000225822fd4d41f2ccf833f15b771637f3f23041829bb82e8828 252245 
1802cc6e3a-02cfa307 02cc6e42 0000000000000036900ab56a565ceb05397ba81d7522ea06f2632a0cb75937cf 252263 
1902cfa307-02d308da 02cfa30f 00000000000000050b87fd847547a036a6bc37c70eeb98799f2efff80936a77c 252295 
2002d308da-02d404fc 02d308e2 000000000000002c8d5d970623566f6a2118f8dd87d5bb0e3d64115007f80cbf 252252 
2102d404fc-02d5068c 02d40504 000000000000004fa499adca211f04bea1d7410425aa02292e1711173cbaa721 252183 
2202d5068c-02d5db55 02d50694 0000000000000032ab0b1568d94f3986877e925141789ba1f8d6bb45b3b9d88b 252283 
2302d5100b-02d8dcee 02d51013 00000000000000126cd5ccc13d8b7fd24b4ec1c5de73c0299d5ace2be3c475ed 252261 Overlap with last block (02d5100b,02d8dcee)
2402d8dcee-02da9f98 02d8dcf6 000000000000000ef3824393dd8ec05f1efbd128b492d0c798b08b23b9ed07ba 252262 
2502da9f98-02ddd465 02da9fa0 0000000000000036900ab56a565ceb05397ba81d7522ea06f2632a0cb75937cf 252263 
2602ddd465-02de947c 02ddd46d 00000000000000520fc022e3303786b162ec91fe393200880229afb94ba9846d 252264 
2702de947c-02e13742 02de9484 000000000000004cd36483d79dfd32cbb0eda8c83e147969467e2d680777677f 252265 
2802e13742-02e1ca97 02e1374a 0000000000000047ff2009cd7168bfb3e529e29841c35de048294bc0e8346372 252266 
2902e1ca97-02e4705f 02e1ca9f 00000000000000299380b28be0926835bd512d264e0f13ace67e14a01f165797 252267 
3002e4705f-02e53376 02e47067 00000000000000455de655a4c6b37963389ad73bf8046fffc1c6d40e4cf48559 252268 
3102e53376-02e7b857 02e5337e 00000000000000174c80916cc99420946d3a6767f2eaf298d9f62e2e8b23e363 252269 
3202e7b857-02e89040 02e7b85f 000000000000001f802536a22331535bb6c46c74ebb05277dacc5048c76d77c1 252270 
3302e89040-02ea88ec 02e89048 0000000000000045eea1e1f9fb28e599e1d99d0792933968a2c39ff99bd5ed80 252271 
3402ea88ec-02ee5152 02ea88f4 0000000000000038de210c039a07d6cbdfe1c14ff250245d787d6989e6ef6a9f 252272 
3502ee5152-02eeaf64 02ee515a 0000000000000048ba25c4d2336337ca63bd78b1dde0ac1b47d3bfb26eae43e4 252273

rromanchuk commented at 0:39 am on March 10, 2015: none

https://dl.dropboxusercontent.com/u/13326666/blk00027.dat.zip

you can ping me on #bitcoin if you need something faster from me

theuni commented at 0:42 am on March 10, 2015: member

Same thing with @rromanchuk’s: repeated blocks:

ajweiss commented at 1:04 am on March 10, 2015: contributor

Here’s a quick and dirty script that will dump all the blockpos data from the leveldb. I had trouble getting Core to read jlopp’s data so I threw this together. Hope it’s helpful.

https://gist.github.com/ajweiss/68a832d9cf26e89b9e0a

theuni commented at 1:08 am on March 10, 2015: member

@ajweiss could you paste the result of the dump? I’m unable to grab his whole .tar.

ajweiss commented at 1:16 am on March 10, 2015: contributor

https://www.dropbox.com/s/fpv19nyqlpl3lnt/jlopp-blockindex.txt.gz?dl=0

Format is “offset hash” followed by a line that prints out the details of the CDiskBlockIndex (in case you want to see the status).

theuni commented at 3:11 am on March 10, 2015: member

Based on @jlopp’s db and blocks (using @ajweiss’s dump, thanks!), here’s the analysis: blk00075.dat has the following added (duplicated) blocks:

002a33b6a 000000000000000ef3824393dd8ec05f1efbd128b492d0c798b08b23b9ed07ba
102a83419 000000000000001f802536a22331535bb6c46c74ebb05277dacc5048c76d77c1
202a90c02 0000000000000047ff2009cd7168bfb3e529e29841c35de048294bc0e8346372
302b01535 000000000000001d976e500e4e880c9d16a8e7a30eff2b4638f1ce8e8a3d29c4
402b7c5cb 0000000000000048ba25c4d2336337ca63bd78b1dde0ac1b47d3bfb26eae43e4
502c39db8 0000000000000021d2ca3bb70769caf1fd13957fe967a573581291a37f2fb278
602c61c0c 000000000000004a85ed53ee82ebd955f58e73e89a2b8b840ef73c545e0f8f39
702cc6e42 0000000000000036900ab56a565ceb05397ba81d7522ea06f2632a0cb75937cf
802cfa30f 00000000000000050b87fd847547a036a6bc37c70eeb98799f2efff80936a77c
902d50694 0000000000000032ab0b1568d94f3986877e925141789ba1f8d6bb45b3b9d88b

and the db expects to find the following blocks, but they’re missing:

002d5db5d 000000000000004b919486ebc14e79b29234b9d9db6572656043c3b1d282ef22
102d9a886 0000000000000024e67d083a058e1d9f64430b509fa5b622c2d430e625c8525c
202e35f8f 000000000000000ea356291d79ac3ba94a9256a3815a37247a8c4a7a2c739e52
302e5fdf3 000000000000001dda82a38ecb4a47f67435e18207f79a053e836bf8e96c9dac
402edcf95 000000000000000f244140e70d84b1ac575b7d81b251a9b5538e872ffc89d774
502f303d2 0000000000000007d9725c5257bd1ca70d4894b518b69b9e88edb9426fd0acab
6034985db 000000000000002453986ca03cf97fa5dd9c06c577baf150ef9f6fdee425adfd
7035d267b 000000000000002e266fee36a0bd8dd10fb1daec13b59b88b5fc5c90deafc259
803681a06 00000000000000004ba71bb94e7b93ade15d6ef3b1542e16220f0cc024467ca3

It looks as though in each of these cases of corruption (including @laanwj’s), this pattern of duplicated blocks/missing blocks is present.

I’m still not able to explain why the wrong blocks/offsets are being inserted.

theuni commented at 6:47 pm on March 10, 2015: member

@jlopp @rromanchuk do you have ebs snapshots enabled? if so, has this data been restored from a shapshot?

rromanchuk commented at 6:55 pm on March 10, 2015: none

@theuni yeah, almost positive this is the one common thread. I accidently launched an instance with the default ebs size so i had to generate snapshot/relaunch

Here is roughly the steps

ran ./bitcoind -d -txindex -daemon
Ran for a couple hours 3 ./bitcoin-cli stop
Stopped instance
snapshot ebs and relaunched with 150G EBS standard ssd
./bitcoind -daemon

jlopp commented at 6:55 pm on March 10, 2015: contributor

@theuni No, we don’t use EBS snapshots. It seems to me that the block files became corrupted during our initial sync from peers on the network and once other peers began requesting blocks contained in the corrupted files, the node began crashing.

theuni commented at 7:02 pm on March 10, 2015: member

@rromanchuk Thanks. I’m not sure how much of that is necessary though, if @jlopp managed to hit the issue without a resize/snapshot. @jlopp I agree with that. I’m trying to determine how the blocks can become corrupt under normal circumstances.

theuni commented at 7:51 pm on March 10, 2015: member

@jlopp How many cores/cpus? Which AWS tier? I’ve set one up and I’m trying to reproduce with testnet.

jlopp commented at 7:59 pm on March 10, 2015: contributor

@theuni Zone us-west-2a; we used a c4.8xlarge for the initial syncing and then dropped to a t2.med once it was at the current tip.

rromanchuk commented at 9:18 pm on March 10, 2015: none

@jlopp did you ever restart the machine when the daemon was running?

jlopp commented at 9:35 pm on March 10, 2015: contributor

Nope. On Mar 10, 2015 5:19 PM, “Ryan Romanchuk” notifications@github.com wrote:

@jlopp https://github.com/jlopp did you ever restart the machine when the daemon was running?

— Reply to this email directly or view it on GitHub #5668 (comment).

laanwj added the label UTXO Db and Indexes on Mar 11, 2015

theuni commented at 3:57 am on March 12, 2015: member

I spent quite a while trying to reproduce this on a c4.8xlarge setup to match @jlopp’s with no luck. 32cores is definitely out of the norm and could be a contributing factor, though.

I synced 100% normally, sync’d with some clean shutdowns, sync’d with some kill -9’s… nothing. @rromanchuk was your original sync on a high-core machine as well?

rubensayshi commented at 10:09 am on March 12, 2015: contributor

0.10 is running fine for us on ec2, but we just upgraded instead of a full resync.

maybe the problem happened after detaching and attaching the disk to the new instance

ajweiss commented at 3:50 pm on March 12, 2015: contributor

@jlopp, @rromanchuk were these nodes running with listen=1 and the appropriate hole punched in the amazon firewall or did they only have outbound connections?

jlopp commented at 3:57 pm on March 12, 2015: contributor

Our nodes have port 8333 open and accept inbound connections. I’d be surprised if that makes a difference though, because I thought a node doesn’t advertise its address while it is still performing the historical sync.

ajweiss commented at 4:00 pm on March 12, 2015: contributor

They’re not supposed to, but the block files aren’t supposed to be corrupted, either. : )

rromanchuk commented at 5:44 pm on March 12, 2015: none

@theuni surprisingly i started syncing on a micro to just start configuring the ami and then i detached volumes, and relaunched into the medium class. The machine was also opened *:8333 and when occasionally checking netstat didn’t see any active connections until after the sync @jlopp were you running magnetic or ssd ebs?

jlopp commented at 5:48 pm on March 12, 2015: contributor

@rromanchuk Our EBS volumes are SSD

theuni commented at 9:41 pm on March 12, 2015: member

@rromanchuk This is starting to make some sense. I just did a reindex on a micro and it took ~40min to get to blk00027.dat. Does that sound like roughly the amount of time you left your sync running before detaching and moving to the medium class?

Edit: I see above that you mentioned “Ran for a couple hours” before creating the snapshot and stopping the instance. Any chance it was ~40min? If not, did anything interesting happen at about that point?

rromanchuk commented at 10:27 pm on March 12, 2015: none

@theuni It was something like that, i wish i would have had detailed cloudwatch events being logged so I could have played back exactly what I did, it would help reveal what really happened….just incase i did something loose and fast with ebs snapshotting/mounting/etc. @theuni if i were to do a “fresh install” is the best way to just shut down and wipe out my blocks/chainstate directory?

This really shouldn’t be a factor but i’m going to say it anyways, after i synced, i ran insight’s sync, which is 100% not compatible with 0.10.0 unless you manually send in a forced RPC flag. Of course they don’t document this except for a closed issue. They load their db by reading the indexed blocks, i can’t imagine how/why they would do anything but read, but programmers are programmers. The fact that anything is accessing data from that folder besides bitcoind makes me a bit suspicious.

ajweiss commented at 10:44 pm on March 12, 2015: contributor

Another note on @jlopp ’s datadir:

The undo data seems to lag the block data by about 44 minutes, and this seems to suggest to me that a reindex took place.

0Feb 18 16:52 blk00000.dat
1Feb 18 17:36 rev00000.dat

The time delta disappears on the problematic block file, implying that this is where it resumed downloading blocks:

0Feb 18 17:34 blk00074.dat
1Feb 18 18:14 rev00074.dat
2
3Feb 18 18:15 blk00075.dat
4Feb 18 18:15 rev00075.dat

This really smells like the issue @theuni fixed in the write position accounting, but I still don’t have a satisfying explanation. It seems it would have to ignore the perfectly valid looking repeated blocks while scanning the block file, but then pick up the missing blocks after. I was thinking maybe something that looks like a magic could be written out in some corruption followed by a skip length followed by the missing blocks, but then there should be evidence of the corruption still in the block file.

Also, why is it reindexing when people think it hasn’t? What do the timestamps look like on the blk/rev files in others’ datadirs?

ajweiss commented at 10:51 pm on March 12, 2015: contributor

I should also note that I tried reproducing this by syncing on a c4.8xlarge and then remounting on a t2.med. I also played with taking snapshots during busy points in sync and remounting, no dice, I was able to validate at level 0 in all cases.

rromanchuk commented at 10:58 pm on March 12, 2015: none

@ajweiss i just looked at my aws console and it looks like i threw another wrench into the situation by taking a magnetic EBS volume -> snapshot -> general purpose SSD. This process alone sounds scary

theuni commented at 11:01 pm on March 12, 2015: member

@rromanchuk did your first EBS disk get 100% full before you noticed and switched?

rromanchuk commented at 11:06 pm on March 12, 2015: none

@theuni good catch! I can hardly remember that fateful night, but maybe that’s when i made the switch. It would make sense if a new node on t1.micro would fill the default 8G around the time my blocks got trashed.

theuni commented at 11:15 pm on March 12, 2015: member

@rromanchuk ok. I’m doing a fresh sync now on an 8gb disk, and I’m up to blk00024.dat and 70% full. Assuming you installed some other stuff and dumped some more data on the drive before sync, that does indeed seem to line up with your point of corruption.

It would also make sense since (I believe) everyone that’s seen this has moved data from one drive to another before continuing the sync. Maybe the common element is filling up the drive, moving, then continuing.

I’ll see if I can get it to blow up when this gets full. @ajweiss the timestamps there are also very interesting, nice catch there. It does seem to strongly imply that @jlopp’s db was re-indexed.

theuni commented at 6:05 pm on March 13, 2015: member

No luck, no matter how I try to corrupt the data. @jlopp @rromanchuk what filesystems are you using? @jlopp are you absolutely certain that you did a fresh sync with no existing data, starting with the final 0.10 release? There are many avenues for issues that are introduced when migrating blocks, so if we can truly rule those out then we have a more narrow scope. Also, as @ajweiss points out, it really looks as though a reindex was performed. Are you 100% certain that’s not the case?

It would be great if someone could reproduce this so that we have more data to go on. It’s pretty clear by now that the initial sync is the problem, and not the crash that could occur days/weeks later. So I’m not sure that we’ll get anywhere with this until someone manages to hit it on purpose.

jlopp commented at 6:18 pm on March 13, 2015: contributor

I wish I could be more help. We definitely didn’t perform a reindex and I don’t know how to reproduce this issue - we built two 0.10 nodes from scratch at the same time with the exact same configurations. One of those nodes ended up having this corruption issue while the other node has been running solidly for a month.

rromanchuk commented at 8:29 pm on March 13, 2015: none

Just got another shutdown yesterday, this is the same node so don’t panic :P

 02015-03-12 23:24:54 UpdateTip: new best=0000000000000000129c03b59501e32a816644e1e4d1e564f68b5dfb534ccdc8  height=347360  log2_work=82.4167  tx=62216231  date=2015-03-12 23:22:01 progress=0.999996  cache=7387
 12015-03-12 23:25:42 ERROR: AcceptToMemoryPool : inputs already spent
 22015-03-12 23:26:40 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=347358, us=54.173.214.35:8333, peer=2885
 32015-03-12 23:26:45 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=54.173.214.35:8333, peer=2886
 42015-03-12 23:27:39 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=54.173.214.35:8333, peer=2887
 52015-03-12 23:27:52 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=54.173.214.35:8333, peer=2888
 62015-03-12 23:29:54 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:4.21/: version 70001, blocks=347340, us=127.0.0.1:8333, peer=2889
 72015-03-12 23:29:54 Added time data, samples 200, offset +0 (+0 minutes)
 82015-03-12 23:30:51 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=54.173.214.35:8333, peer=2890
 92015-03-12 23:31:46 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=54.173.214.35:8333, peer=2891
102015-03-12 23:32:35 socket recv error Connection timed out (110)
112015-03-12 23:33:54 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=347360, us=54.173.214.35:8333, peer=2892
122015-03-12 23:33:59 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=54.173.214.35:8333, peer=2893
132015-03-12 23:34:07 ResendWalletTransactions()
142015-03-12 23:35:55 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=54.173.214.35:8333, peer=2894
152015-03-12 23:36:46 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=54.173.214.35:8333, peer=2895
162015-03-12 23:36:46 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=347360, us=54.173.214.35:8333, peer=2896
172015-03-12 23:37:21 receive version message: /Shibetoshi:0.1/: version 70002, blocks=0, us=[::]:0, peer=2897
182015-03-12 23:38:29 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=54.173.214.35:8333, peer=2898
192015-03-12 23:38:52 socket send error Broken pipe (32)
202015-03-12 23:39:14 receive version message: /Satoshi:0.10.0/: version 70002, blocks=138014, us=54.173.214.35:8333, peer=2899
212015-03-12 23:39:30 socket send error Broken pipe (32)
222015-03-12 23:39:49 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large

keystrike commented at 7:03 am on March 17, 2015: contributor

Just received the same error. Fresh install on a Debian box running 0.10.

~/bitcoin-0.10.0/bin$ bitcoind: main.cpp:3335: void ProcessGetData(CNode*): Assertion `!“cannot load block from disk”’ failed.

uname -a Linux delysid 3.2.0-4-686-pae #1 SMP Debian 3.2.63-2+deb7u2 i686 GNU/Linux

Last lines of debug.log:

2015-03-17 06:55:02 UpdateTip: new best=000000000001012cfcc8516330999654efb4c90109173d66d0a5f91832e632a7 height=92176 log2_work=56.847009 tx=154876 date=2010-11-16 14:23:00$ 2015-03-17 06:55:02 UpdateTip: new best=00000000000e316f5176eb751e67da48e93d9bce0b0696885081273cc2650d2c height=92177 log2_work=56.847225 tx=154877 date=2010-11-16 14:25:05$ 2015-03-17 06:55:02 UpdateTip: new best=000000000003fd604e30a67ae86e9850d4aed2d4ac40f85b9405ab129ebfa2c3 height=92178 log2_work=56.847442 tx=154887 date=2010-11-16 14:28:58$ 2015-03-17 06:55:02 UpdateTip: new best=000000000002ffba92d0769e4b7da59fc8c4128f6066ca3103a30341250dd174 height=92179 log2_work=56.847659 tx=155101 date=2010-11-16 14:45:31$ 2015-03-17 06:55:03 ERROR: ReadBlockFromDisk : OpenBlockFile failed 2015-03-17 06:59:12

This is an old web server, it has been running for years. Standalone hardware in a London datacenter, not Amazon.

Edit: Looking closer now, this isn’t actually the same error as I do not receive “Deserialize or I/O error - ReadCompactSize() : size too large”.

EhVedadoOAnonimato commented at 4:11 pm on March 20, 2015: none

Hello,

Just to add that I’m also seeing the same error, which prevents me to boot my bitcon-qt.

My block database is stored on a Truecrypt encrypted external disk. My system is a Kubuntu 14.10. I’m running bitcoin-qt v0.10.0.0-g047a898 (64-bit)

I suppose rebuilding the database will do me no good, right?

jlopp commented at 9:49 pm on March 21, 2015: contributor

One of our other nodes that has been running fine for over a month just crashed from the same error.

2015-03-21 15:11:06 ERROR: ReadBlockFromDisk : OpenBlockFile failed

ajweiss commented at 10:06 pm on March 21, 2015: contributor

can you make the datadir available please?

On Sat, Mar 21, 2015, 5:49 PM Jameson Lopp notifications@github.com wrote:

One of our other nodes that has been running fine for over a month just crashed from the same error.

2015-03-21 15:11:06 ERROR: ReadBlockFromDisk : OpenBlockFile failed

— Reply to this email directly or view it on GitHub #5668 (comment).

jlopp commented at 5:29 pm on March 22, 2015: contributor

I’ll post the bad block files soon. The output from the check blocks tools is quite similar:

 0Input file /blocks/blk00023.dat
 1054bead8-5337868f 054beae0 f892b2437a142747e6b593176c767f0e9dad30d4ba52791aacdc1bebd23dff58     -1 Overlap with last block (054bead8,5337868f)
 2054beae1-b1d48698 054beae9 09e6829d8dffae38ad5c10343f7347b421db2294d6360eab8256f932fc8cce69     -1 Overlap with last block (054beae1,b1d48698)
 3054c7d1d-533818d4 054c7d25 05fd918c3e7456a26d3f6e9d72777558eb4b7c3a3da5dde0556993592fb8add3     -1 Overlap with last block (054c7d1d,533818d4)
 4054c7d26-533818dd 054c7d2e 39cc348070b9a7a8e3bbb3caf973aa7b80651e1f012dd86fe84656a8c3506c4e     -1 Overlap with last block (054c7d26,533818dd)
 5054c7d42-df013c43 054c7d4a 46fbe70e2fe21609042d1c2f7d7de78be7d6a4739163fc8dffe71c02313604e2     -1 Overlap with last block (054c7d42,df013c43)
 6054c7d46-df013c47 054c7d4e f521065b25eb4f3a0431132084f19bba0f2dc3ae39108b0fed414e5ce0129320     -1 Overlap with last block (054c7d46,df013c47)
 7054c7d4a-df013c4b 054c7d52 5579819e0c88b3759b28dc8f7526af5e2b457f7c3a55ca89cd8ffd3e812b6e7b     -1 Overlap with last block (054c7d4a,df013c4b)
 8054c7d4e-05f9064f 054c7d56 de627541c042f591b352a9ccdcb844cd51820c7b73c1a2cc402fafd65ab06f8c     -1 Overlap with last block (054c7d4e,05f9064f)
 9054cc968-054e3198 054cc970 000000000000006c3ec2668d792c990799d91e72d8e93d80052dcafb55aa8542 200399 Overlap with last block (054cc968,054e3198)
10Input file /blocks/blk00047.dat
110326a258-d3af0665 0326a260 cd0f5125199d2e872ca2f894f292b42eeb6646992755ec85d7dfdabdd5fd8ac5     -1 Overlap with last block (0326a258,d3af0665)
12032fcfc1-0336bbda 032fcfc9 000000000000033584d783a674f0a406748f81d779786b8cdf8fbfe8af83d5b6 225539 Overlap with last block (032fcfc1,0336bbda)
13Input file /blocks/blk00103.dat
14013cc2e7-252e00a7 013cc2ef d549a393beda9d833022cdf966e63b543a6fe424897a19310280a0dac59cf51d     -1 Overlap with last block (013cc2e7,252e00a7)
15013d7fe7-0140592e 013d7fef 0000000000000003683570b2d0412ebaf73440ba1205dc556eb3954ecd39cbff 276712 Overlap with last block (013d7fe7,0140592e)
16Input file /blocks/blk00126.dat
1705f066a0-8af28ca8 05f066a8 413000f2aa16953e9f1eb86e72074e798b5a3cb4aafc5fe2df0e06199553855b     -1 Overlap with last block (05f066a0,8af28ca8)
1805f558d8-05faac0f 05f558e0 0000000000000000b03b44d66683c1c844e4639224dc0f79d65709d6408b1b0c 292650 Overlap with last block (05f558d8,05faac0f)
1907796dc6-8c7b93ce 07796dce a8528116edd7914f7c132d5bbbb4ee147a34a53a0e1f4e60fdd1d445ddb987ab     -1 Overlap with last block (07796dc6,8c7b93ce)
20077c824c-07852ef9 077c8254 00000000000000004cb4f430b5a66cafcd20b656d7d66242dadf91d603228b7a 292754 Overlap with last block (077c824c,07852ef9)
21Input file /blocks/blk00127.dat
2200df4681-85e16c89 00df4689 2359b6fe12feae9c761c00ffe9885c7c5c28200b92fcb7081c0b3f6227ccbf54     -1 Overlap with last block (00df4681,85e16c89)
2300e4745f-00e8ab8d 00e47467 00000000000000001048b08274249b592148c0af5eab644afeb2e36f7a6a4a06 292820 Overlap with last block (00e4745f,00e8ab8d)
2401bc6f33-86be953b 01bc6f3b 93d796919bd2906217839eba4257b0e4e949a5304cc100f6f11fc2acd3a462eb     -1 Overlap with last block (01bc6f33,86be953b)
2501c5b2bb-01c5c91d 01c5b2c3 000000000000000017aee58faf59d2ecf3a7cb0821938328d49e7b13ccc1d3fa 292561 Overlap with last block (01c5b2bb,01c5c91d)
26Input file /blocks/blk00135.dat
2703800eb3-888234bb 03800ebb 93f86dd0ebb350799b9163cdcf7125e5b24311fab57b74a94da3939bd611360b     -1 Overlap with last block (03800eb3,888234bb)
280381fc9f-03822d3e 0381fca7 000000000000000047d2f2eb7278e3f4aded9acaf502f5ec27bab5018b5871f2 298000 Overlap with last block (0381fc9f,03822d3e)
2903c56d6a-88c79372 03c56d72 b480a3912cb6d1c463e87e1a8d0e97ee91dc35405167a911bcba58ec0b7b4361     -1 Overlap with last block (03c56d6a,88c79372)
3003c6ad5a-03ca3256 03c6ad62 00000000000000006608c7d7e72ea8b6cd59af8126f432108fda5394a44fc97a 297957 Overlap with last block (03c6ad5a,03ca3256)
3104295595-892b7b9d 0429559d b0906ed7a180d0fe92f92697acdbc8e79ef6b06c53b88122b7def6d3d80ca0d9     -1 Overlap with last block (04295595,892b7b9d)
32042c18be-04313a10 042c18c6 00000000000000007ad73454bb619f73e8634efcc7ab46be53969341b1d245bb 298056 Overlap with last block (042c18be,04313a10)
33Input file /blocks/blk00168.dat
340378c0b9-d06685d7 0378c0c1 6dfcd17e90d976033c1eb409ef5c2319dafe9221b7db435abcbf7690b6803e8d     -1 Overlap with last block (0378c0b9,d06685d7)
35037a17dc-037f2fb6 037a17e4 000000000000000023f49c794399a27ab70af711b6be789529b37d0bf8d6f20f 317064 Overlap with last block (037a17dc,037f2fb6)

EhVedadoOAnonimato commented at 1:16 pm on March 23, 2015: none

One question to the others who are having this problem: you talk about your nodes crashing. But are you even able to start it? My node fails on startup. I simply cannot start it with the current blockchain copy I have on my encrypted drive. This is the error:

2015-03-23 13:13:18 Verifying last 288 blocks at level 3 2015-03-23 13:13:18 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large 2015-03-23 13:13:18 ERROR: VerifyDB() : *** ReadBlockFromDisk failed at 331486, hash=0000000000000000101c32bf9fd355ac2754fa56ef44db1f98c89e3783e9a2df 2015-03-23 13:13:20 Aborted block database rebuild. Exiting.

Is there anything I could do in order to start my node? Does rebuilding the database help?

Thank you!

sipa commented at 1:21 pm on March 23, 2015: member

@EhVedadoOAnonimato It seems your corruption is in the last 288 blocks (which get checked at startup), so it fails immediately. Otherwise there doesn’t really seem to be a difference with the other reports.

Yes, rebuilding the database will fix it, but that will wipe information from your data dir at this stage would be welcome for debugging.

EhVedadoOAnonimato commented at 1:30 pm on March 23, 2015: none

Thank you for the reply, Pieter.

Actually it seems the corruption is on the very last block I have stored on this volume. I used the -checkblocks and look:

2015-03-23 13:27:00 Verifying last 1 blocks at level 3 2015-03-23 13:27:00 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large 2015-03-23 13:27:00 ERROR: VerifyDB() : *** ReadBlockFromDisk failed at 331486, hash=0000000000000000101c32bf9fd355ac2754fa56ef44db1f98c89e3783e9a2df

Is there no way to bypass this check?

I think I might end rebuilding it eventually. But if there’s anything I should keep or send you to so that this could be better analyzed please let me know.

On 3/23/15, Pieter Wuille notifications@github.com wrote:

@EhVedadoOAnonimato It seems your corruption is in the last 288 blocks (which get checked at startup), so it fails immediately. Otherwise there doesn’t really seem to be a difference with the other reports.

Yes, rebuilding the database will fix it, but that will wipe information from your data dir at this stage would be welcome for debugging.

Reply to this email directly or view it on GitHub: #5668 (comment)

sipa commented at 1:34 pm on March 23, 2015: member

@EhVedadoOAnonimato You could bypass the check, but you’ll likely fail shortly afterwards.

It would be useful if you could run @ajweiss’s script on your datadir: https://gist.github.com/ajweiss/68a832d9cf26e89b9e0a

EhVedadoOAnonimato commented at 1:43 pm on March 23, 2015: none

Hum, I tried but as I’m not a bitcoin developer, my environment seems to lack the necessary libraries. See:

Traceback (most recent call last): File “./gistfile1.py”, line 3, in import leveldb ImportError: No module named leveldb

I don’t know python. Is there a simple set of commands I can execute to install all necessary libraries?

On 3/23/15, Pieter Wuille notifications@github.com wrote:

@EhVedadoOAnonimato You could bypass the check, but you’ll likely fail shortly afterwards.

It would be useful if you could run @ajweiss’s script on your datadir: https://gist.github.com/ajweiss/68a832d9cf26e89b9e0a

Reply to this email directly or view it on GitHub: #5668 (comment)

ajweiss commented at 7:49 pm on March 23, 2015: contributor

can you post a tarball of the whole datadir?

On Mon, Mar 23, 2015 at 12:25 PM, Jameson Lopp notifications@github.com wrote:

Our bad blocks from our second crashed node are available at http://bg.bitcoind.debug.s3-website-us-west-2.amazonaws.com/badblocks-2-23.tar

— Reply to this email directly or view it on GitHub #5668 (comment).

jlopp commented at 8:14 pm on March 23, 2015: contributor

I can, though this morning I performed reindexes on our nodes that have been crashing. I just ran your block checking script after the reindex and it’s finding the same corruption - shouldn’t the reindex clean that up?

Edit: I guess I want to use -checkblocks=0 instead?

ajweiss commented at 8:37 pm on March 23, 2015: contributor

No, in fact there’s evidence that reindexing may cause the problem.

A couple of notes here:

When you see this issue, if at all possible, please take a snapshot of the full datadir before restarting. It helps a great deal in not just looking at the corruption, but also trying to figure out why it occurred in the first place.
Right now, it’s looking like something is going wrong either during initial sync or during a reindex. Running a reindex with released 0.10 is not advised for resolving this issue as @cfields found a bug in the reindexing code that can be triggered when there is block file corruption.
Run bitcoind -checkblocks=0 -checklevel=0 to verify block locations in the block index. I’d run this now on your reindexed node to see if it succeeds. If it doesn’t, the reindex failed and you will need to wipe out the datadir and start over. If you do start over, run this at the end to ensure it succeeded. (Please do take a snapshot before you wipe, if possible)

Does this make sense?

On Mon, Mar 23, 2015 at 4:14 PM, Jameson Lopp notifications@github.com wrote:

I can, though this morning I performed reindexes on our nodes that have been crashing. I just ran your block checking scrip after the reindex and it’s finding the same corruption - shouldn’t the reindex clean that up?

— Reply to this email directly or view it on GitHub #5668 (comment).

ajweiss commented at 8:49 pm on March 23, 2015: contributor

The issues encountered here should be caught by -checklevel=0 as it reads each block and verifies the data hashes to what it should. Setting a higher -checklevel will slow things down significantly.

jlopp commented at 10:14 pm on March 23, 2015: contributor

The rest of our data files are available here: http://bg.bitcoind.debug.s3-website-us-west-2.amazonaws.com/

ajweiss commented at 10:34 pm on March 23, 2015: contributor

Hi Jameson,

Can you please also include the block index? Or to make things simpler, wipe out the debug.log, wallet.dat and anything sensitive in the config file and post a tarball?

Thanks!

–adam

On Mon, Mar 23, 2015 at 6:15 PM, Jameson Lopp notifications@github.com wrote:

The rest of our data files are available here: http://bg.bitcoind.debug.s3-website-us-west-2.amazonaws.com/

— Reply to this email directly or view it on GitHub #5668 (comment).

laanwj commented at 10:27 am on March 24, 2015: member

@jlopp I wouldn’t pay too much attention to those overlapping blocks. Look at the unrealistic ranges, those are misidentified

0Overlap with last block (0378c0b9,d06685d7)

I’ve created a repository here https://github.com/laanwj/blockdb-troubleshoot for these troubleshooting tools. Let’s try to maintain them there.

It contains a newer version of list-blocks.py without this problem, a version of printblockdb.py as well as everything needed to build the leveldb.so module for python for that (as the default distro py-leveldb will corrupt your database due to compression, see my this comment).

jlopp commented at 1:14 pm on March 24, 2015: contributor

Thanks @laanwj, I’ll check out those tools. I’ve posted the full contents of our recently crashed node’s data here: http://bg.bitcoind.debug.s3-website-us-west-2.amazonaws.com/bitcoind.tar

ajweiss commented at 3:56 pm on March 24, 2015: contributor

@jlopp Thanks for posting this! At first glance at the block index and debug.log, it looks like either the snapshot was taken in the middle of the reindex, or the reindex crashed. Do you know which is true?

jlopp commented at 4:01 pm on March 24, 2015: contributor

mid-index - I started a reindex on the node yesterday morning and it’s ongoing. Once it completes I’ll run checkblocks.

Update: ran checkblocks and then ran the block checker tool and it says the files are all fixed. Will report back if we experience any more crashes.

lontivero commented at 3:27 am on April 21, 2015: contributor

My node fails with the same error. I was running perfectly well for around a week.

 02015-04-20 01:53:44 Bitcoin version v0.10.0 (2015-02-13 09:55:11 +0100)
 12015-04-20 01:53:44 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
 22015-04-20 01:53:44 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
 32015-04-20 01:53:44 Default data directory C:\Users\lontivero\AppData\Roaming\Bitcoin
 42015-04-20 01:53:44 Using data directory C:\Users\lontivero\AppData\Roaming\Bitcoin
 52015-04-20 01:53:44 Using config file C:\Users\lontivero\AppData\Roaming\Bitcoin\bitcoin.conf
 62015-04-20 01:53:44 Using at most 125 connections (2048 file descriptors available)
 72015-04-20 01:53:44 Using 4 threads for script verification
 82015-04-20 01:53:44 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0)
 92015-04-20 01:53:44 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0)
102015-04-20 01:53:44 Using wallet wallet.dat
112015-04-20 01:53:44 init message: Verificando monedero...
122015-04-20 01:53:44 CDBEnv::Open : LogDir=C:\Users\lontivero\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\lontivero\AppData\Roaming\Bitcoin\db.log
132015-04-20 01:53:44 Bound to [::]:8333
142015-04-20 01:53:44 Bound to 0.0.0.0:8333
152015-04-20 01:53:44 init message: Cargando el Ãndice de bloques...
162015-04-20 01:53:44 Opening LevelDB in C:\Users\lontivero\AppData\Roaming\Bitcoin\blocks\index
172015-04-20 01:53:44 Opened LevelDB successfully
182015-04-20 01:53:44 Opening LevelDB in C:\Users\lontivero\AppData\Roaming\Bitcoin\chainstate
192015-04-20 01:53:44 Opened LevelDB successfully
202015-04-20 01:53:56 LoadBlockIndexDB: last block file = 252
212015-04-20 01:53:56 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=244, size=95301518, heights=350714...351103, time=2015-04-04...2015-04-07)
222015-04-20 01:53:56 Checking all blk files are present...
232015-04-20 01:54:09 LoadBlockIndexDB(): transaction index enabled
242015-04-20 01:54:10 LoadBlockIndexDB(): hashBestChain=000000000000000001bd8a044a084bbee53c8a6b7bb118145f71c31edab15764 height=350919 date=2015-04-06 02:53:23 progress=0.976874
252015-04-20 01:54:10 init message: Verificando bloques...
262015-04-20 01:54:10 Verifying last 288 blocks at level 3
272015-04-20 02:09:31 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large
282015-04-20 02:09:31 ERROR: VerifyDB() : *** ReadBlockFromDisk failed at 350690, hash=000000000000000013290aecbe160e27e4d92b71dd2e397e9a8abeaee85ac7ac

lontivero commented at 3:28 am on April 21, 2015: contributor

My node fails with the same error. I was running perfectly well for around a week.

 02015-04-20 01:53:44 Bitcoin version v0.10.0 (2015-02-13 09:55:11 +0100)
 12015-04-20 01:53:44 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
 22015-04-20 01:53:44 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
 32015-04-20 01:53:44 Default data directory C:\Users\lontivero\AppData\Roaming\Bitcoin
 42015-04-20 01:53:44 Using data directory C:\Users\lontivero\AppData\Roaming\Bitcoin
 52015-04-20 01:53:44 Using config file C:\Users\lontivero\AppData\Roaming\Bitcoin\bitcoin.conf
 62015-04-20 01:53:44 Using at most 125 connections (2048 file descriptors available)
 72015-04-20 01:53:44 Using 4 threads for script verification
 82015-04-20 01:53:44 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0)
 92015-04-20 01:53:44 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0)
102015-04-20 01:53:44 Using wallet wallet.dat
112015-04-20 01:53:44 init message: Verificando monedero...
122015-04-20 01:53:44 CDBEnv::Open : LogDir=C:\Users\lontivero\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\lontivero\AppData\Roaming\Bitcoin\db.log
132015-04-20 01:53:44 Bound to [::]:8333
142015-04-20 01:53:44 Bound to 0.0.0.0:8333
152015-04-20 01:53:44 init message: Cargando el Ãndice de bloques...
162015-04-20 01:53:44 Opening LevelDB in C:\Users\lontivero\AppData\Roaming\Bitcoin\blocks\index
172015-04-20 01:53:44 Opened LevelDB successfully
182015-04-20 01:53:44 Opening LevelDB in C:\Users\lontivero\AppData\Roaming\Bitcoin\chainstate
192015-04-20 01:53:44 Opened LevelDB successfully
202015-04-20 01:53:56 LoadBlockIndexDB: last block file = 252
212015-04-20 01:53:56 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=244, size=95301518, heights=350714...351103, time=2015-04-04...2015-04-07)
222015-04-20 01:53:56 Checking all blk files are present...
232015-04-20 01:54:09 LoadBlockIndexDB(): transaction index enabled
242015-04-20 01:54:10 LoadBlockIndexDB(): hashBestChain=000000000000000001bd8a044a084bbee53c8a6b7bb118145f71c31edab15764 height=350919 date=2015-04-06 02:53:23 progress=0.976874
252015-04-20 01:54:10 init message: Verificando bloques...
262015-04-20 01:54:10 Verifying last 288 blocks at level 3
272015-04-20 02:09:31 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large
282015-04-20 02:09:31 ERROR: VerifyDB() : *** ReadBlockFromDisk failed at 350690, hash=000000000000000013290aecbe160e27e4d92b71dd2e397e9a8abeaee85ac7ac

jlopp commented at 1:35 pm on May 14, 2015: contributor

Anyone still digging into this? I’m no longer experiencing the ReadCompactSize() error on the nodes I repaired, but I have a completely different node on a VPS (not AWS) that has started to crash several times per week with a similar error:

2015-05-14 07:53:39 ERROR: ReadBlockFromDisk: OpenBlockFile failed

lontivero commented at 2:25 pm on May 14, 2015: contributor

I downloaded the blockchain again and didn’t see this problem again. Anyway, I think it is just a matter of time before it crash again because the node worked perfectly well for a month before the crashing.

funbucks commented at 1:51 am on June 21, 2015: none

I had this issue and I also moved my blockchain from disk to disk and have my OS on a seperate disk than the blockchain.

yamamushi commented at 9:37 pm on July 16, 2015: none

I’m running on a Raspberry Pi, and had moved my blockchain between disks as well.

I’ve been seeing this error over the past few days, and have had to restart bitcoind to restore functionality.

jlopp commented at 8:31 pm on October 15, 2015: contributor

I haven’t had any issues on my AWS instances in months, but I recently had my Bitnodes node crash with:

2015-10-15 20:04:28 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large at CBlockDiskPos(nFile=356, nPos=43062353)

rromanchuk commented at 9:32 pm on October 15, 2015: none

@jlopp most likely purely coincidence, but I had a USA east node die yesterday for the first time since this post…but haven’t investigated why yet. When I looked at the tail briefly, I didn’t see anything in the tail besides a get version.

gmaxwell commented at 9:39 pm on October 15, 2015: contributor

@jlopp what software are they running, precisely?

dcousens commented at 9:46 pm on October 15, 2015: contributor

@rromanchuk exact same here. Last message was a bitcoinj get version.

edit: To clarify, I did not receive the error above, I didn’t receive any error at all. My situation mirrored that of @rromanchuk’s last comment though, so I figured I’d weigh in.

jlopp commented at 10:21 pm on October 15, 2015: contributor

@gmaxwell I’m actually suspecting it’s a hardware error; I’m seeing other I/O errors now (the storage in question is a microSD card.) Disregard for now; I’ll report back if I rule out hardware.

laanwj added the label DB corruption on Feb 9, 2016

TimCinel commented at 1:07 pm on June 1, 2016: none

I’m also seeing this a few times a week on a dedicated server:

02016-06-01 09:29:56 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large: iostream error at CBlockDiskPos(nFile=4, nPos=31649587)
1Assertion failed: !"cannot load block from disk" (main.cpp: ProcessGetData: 4153)

Restarting bitcoind fixes the problem until the next occurrence, usually several days and several hundred blocks later.

TimCinel commented at 2:58 pm on June 11, 2016: none

While I am running bitcoind on a dedicated server with a physical hard drive, I failed to mention that bitcoind runs from within a docker container and that the data is on a docker volume.

laanwj commented at 10:45 am on June 13, 2016: member

@TimCinel Apparently (some of) your block files are corrupt. Restarting bitcoind only appears to solve the problem until it tries to access the same block again. You could reindex, although as a block in early file 4 is failing it’s better to a) redownload entirely or b) copy the bitcoin folder (excluding wallet) from a known good node.

TimCinel commented at 11:14 am on August 25, 2016: none

@laanwj You were spot on. I bootstrapped this node by copying its blocks from another node, something must have gone wrong. Since redownloaded everything, it’s been running without an issue.

Thanks.

jlopp commented at 8:11 pm on September 9, 2016: contributor

I haven’t experienced this problem on any of my nodes in a long time, but just had a crash on a Raspberry Pi node:

2016-09-09 14:28:14 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large at CBlockDiskPos(nFile=641, nPos=79365832)

joshafest commented at 8:19 am on November 2, 2016: none

Is this issue still open. ?

laanwj commented at 11:41 am on November 2, 2016: member

This is simply disk corruption (block file was truncated)

laanwj closed this on Nov 2, 2016

CnaBhr commented at 7:07 am on July 20, 2017: none

would you plz tell me what’s the problem ? a fatal error has occurred…

2017-07-20 06:42:55 ERROR: ReadBlockFromDisk: Deserialize or I/O error - CAutoFile::read: fread failed at CBlockDiskPos(nFile=78, nPos=8734624) 2017-07-20 06:42:55 *** Failed to read block 2017-07-20 06:43:02 Loading addresses from DNS seeds (could take a while) 2017-07-20 06:43:04 Failed to connect best blockImported mempool transactions from disk: 0 successes, 0 failed, 0 expired 2017-07-20 06:43:04 tor: Thread interrupt 2017-07-20 06:43:04 opencon thread exit 2017-07-20 06:43:04 addcon thread exit 2017-07-20 06:43:04 scheduler thread interrupt 2017-07-20 06:43:04 Shutdown: In progress… 2017-07-20 06:43:04 torcontrol thread exit 2017-07-20 06:43:04 msghand thread exit 2017-07-20 06:43:05 net thread exit 2017-07-20 06:43:06 118 addresses found from DNS seeds 2017-07-20 06:43:06 dnsseed thread exit 2017-07-20 06:43:07 Shutdown: done

TheBlueMatt commented at 3:40 pm on July 24, 2017: member

@CnaBhr almost certainly disk corruption, possibly due to unclean shutdown on consumer-grade drives or OSes which lie to applications about having fully stored data.

vincenzopalazzo commented at 7:10 pm on April 20, 2019: none

Hi guys, I’m writing a parser for bitcoin core and now at the file blk 976 I have an error, into type verInt. I’m using a library bitcoin core and I using serialize.h for decode a data, and I’using a readCompactSize for reading a var int but into when I go read the blk00976.dat I have this error

C ++ exception with description" ReadCompactSize (): size too large: iostream error "thrown in the test body.

But The node bitcoin core run ok, when the ReadCompactSize thow this exception? Is there something I’m missing?

lomonosofff commented at 5:42 am on December 14, 2021: none

Ребята! Кто как решил??? У меня вот аккурат этим числом так раскорячило кошельки, что я до сих пор от туда не могу ничего отправить. Постоянно висит не в мемпуле, конфликтует со всеми и я уже отчаялся честно говоря забрать свой кусочек биткоина…. У меня виндус, кошелек на биткоинкоре. Что делать то? Как бы на новогод не пришлось последний творог без соли доедать… Диск новый, сверсховременный турбопневмагипергидроскоростной.

MarcoFalke locked this on Dec 14, 2021

ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large #5668