Bitcoin disk corruption

mbrix commented at 2:47 am on December 30, 2014: none

Hi guys,

I’m running Bitcoin-0.10.0rc1 on debian linux x64, did a clean sync, been operating without issue until Block 336550 when it stopped syncing with the network.

Shutdown normally, on restart I get: ./bitcoin-qt bitcoin-qt: coins.cpp:78: virtual CCoinsViewCache::~CCoinsViewCache(): Assertion `!hasModifier’ failed.

Gets through about 10% of block validation before that happens.

With debug on I get:

Opened LevelDB successfully Opening LevelDB in /home/matt/.bitcoin/chainstate Opened LevelDB successfully LoadBlockIndexDB: last block file = 212 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=204, size=57773388, heights=336235…336551, time=2014-12-28…2014-12-29) Checking all blk files are present… LoadBlockIndexDB(): transaction index disabled LoadBlockIndexDB(): hashBestChain=000000000000000013186b8af4492f00d5ed950a5633fe2f43410f6f839c26cf height=336544 date=2014-12-29 22:18:28 progress=0.999577 init message: Verifying blocks… Verifying last 288 blocks at level 3 LevelDB read failure: Corruption: block checksum mismatch Corruption: block checksum mismatch

I’m not seeing any other disk corruption issues, no kernel messages that might indicate some hardware failure condition etc. What other information can I provide?

mbrix commented at 3:01 pm on January 2, 2015: none

Just want to follow up. This is a repeatable issue for me, just got corrupted again after a few more days.

Bitcoin version v0.10.0rc1 (2014-12-23 20:14:37 +0100) Using OpenSSL version OpenSSL 1.0.1j 15 Oct 2014 Using BerkeleyDB version Berkeley DB 4.8.30: (April 9, 2010) Default data directory /home/matt/.bitcoin Using data directory /home/matt/.bitcoin Using config file /home/matt/.bitcoin/bitcoin.conf Using at most 125 connections (1024 file descriptors available) Using 4 threads for script verification Using wallet wallet.dat init message: Verifying wallet… CDBEnv::Open : LogDir=/home/matt/.bitcoin/database ErrorFile=/home/matt/.bitcoin/db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 init message: Loading block index… Opening LevelDB in /home/matt/.bitcoin/blocks/index Opened LevelDB successfully Opening LevelDB in /home/matt/.bitcoin/chainstate Opened LevelDB successfully LoadBlockIndexDB: last block file = 213 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=246, size=50429569, heights=336813…337057, time=2014-12-31…2015-01-02) Checking all blk files are present… LoadBlockIndexDB(): transaction index disabled LoadBlockIndexDB(): hashBestChain=000000000000000004cbbc8d7e0dbe4bc1736ebc6eaab0a6216137febf6e283b height=337029 date=2015-01-02 00:34:59 progress=0.998639 init message: Verifying blocks… Verifying last 288 blocks at level 3 No coin database inconsistencies in last 289 blocks (124629 transactions) block index 23277ms init message: Loading wallet… nFileVersion = 100000 Keys: 445 plaintext, 0 encrypted, 117 w/ metadata, 445 total wallet 355ms UpdateTip: new best=000000000000000004e89b8b28dd44a7c73b5b0dd1273ee56e365a40d0e398f8 height=337030 log2_work=81.900832 tx=55541042 date=2015-01-02 00:28:37 progress=0.998628 cache=45 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000000a1bf44402cdb950b8378502c84cd3275612d26ccbda01a height=337031 log2_work=81.900888 tx=55541501 date=2015-01-02 00:45:54 progress=0.998656 cache=1910 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000011ca2e3e4607bf793b191de7a2810cd07cdc3a53006dc6b2 height=337032 log2_work=81.900944 tx=55541611 date=2015-01-02 00:54:40 progress=0.998670 cache=2575 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=00000000000000000743226d47b6492ea774efb1695d5c42ba5babe76d0296b6 height=337033 log2_work=81.900999 tx=55542017 date=2015-01-02 00:57:54 progress=0.998675 cache=3776 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=00000000000000000f9bd7a95527b71e45b8e6dae5063868ef230aaca1ac7e09 height=337034 log2_work=81.901055 tx=55542284 date=2015-01-02 01:03:35 progress=0.998684 cache=4425 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000015d4526f5272a481cd2bb8a96e9f99eccaca704a8b9f7e50 height=337035 log2_work=81.901111 tx=55542603 date=2015-01-02 01:10:57 progress=0.998695 cache=5142 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=0000000000000000115f7f4062ce472da782431cc884a048dc3262d945332ac7 height=337036 log2_work=81.901167 tx=55542812 date=2015-01-02 01:16:40 progress=0.998704 cache=5543 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=00000000000000000ee3f83dae49a4092f33ac018156c875634beeae52674a71 height=337037 log2_work=81.901222 tx=55542981 date=2015-01-02 01:20:00 progress=0.998710 cache=5865 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000012b69cbeffd012744b8c05a35c94f37cde00cdf583c47f1c height=337038 log2_work=81.901278 tx=55543011 date=2015-01-02 01:20:29 progress=0.998710 cache=5998 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=0000000000000000001fffc761f0fe48dfa94dfb42e3c8713c5e14bcef11d930 height=337039 log2_work=81.901334 tx=55543501 date=2015-01-02 01:31:13 progress=0.998727 cache=6863 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=0000000000000000096016907dfb56633d7ef132f2275712788326283a8e5c73 height=337040 log2_work=81.90139 tx=55543532 date=2015-01-02 01:32:03 progress=0.998729 cache=6945 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000000dd5681f0bc2b67a87877deabdda2fd05d03692ff7de82d height=337041 log2_work=81.901445 tx=55544255 date=2015-01-02 01:47:35 progress=0.998753 cache=8589 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000004d63eba91a3f501acc8d866f0c0b8fce5a76234522a08cc height=337042 log2_work=81.901501 tx=55545195 date=2015-01-02 02:05:46 progress=0.998782 cache=10322 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=00000000000000000e623f4667978ba2511edac18e1dd1075b419847d7a6ceda height=337043 log2_work=81.901557 tx=55545589 date=2015-01-02 02:17:27 progress=0.998800 cache=10993 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000004246d1a2666987c44fcf89123d86e9cb115cba00bcf832c height=337044 log2_work=81.901613 tx=55546169 date=2015-01-02 02:27:39 progress=0.998816 cache=11948 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=0000000000000000167ed8a5ba5d8cb5ea27aaac7b4705d0245630daeac192cc height=337045 log2_work=81.901668 tx=55546679 date=2015-01-02 02:36:37 progress=0.998831 cache=12805 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000000124b2813b69fc5356b5f5d91e7969ff6081d66e521b6f8 height=337046 log2_work=81.901724 tx=55546832 date=2015-01-02 02:42:50 progress=0.998840 cache=13039 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=00000000000000000f0de5622037d513181a9ffe911867826eb5a1aac40a477a height=337047 log2_work=81.90178 tx=55547196 date=2015-01-02 02:47:49 progress=0.998848 cache=13540 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000014f63c53e011e13561a423201e1239f30b7474acf63c51df height=337048 log2_work=81.901836 tx=55547444 date=2015-01-02 02:52:00 progress=0.998855 cache=13908 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000018f7fa666a399acf35cd882fedc37c8bfddb050d36c2fe70 height=337049 log2_work=81.901891 tx=55547811 date=2015-01-02 03:00:12 progress=0.998868 cache=14539 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=00000000000000000e39d8d2eb9a64748983cd1986b26acafcde27632e68e774 height=337050 log2_work=81.901947 tx=55548107 date=2015-01-02 03:08:26 progress=0.998881 cache=14909 CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. UpdateTip: new best=000000000000000000eef16b6dc52c28448c7eb2996c3b5a75d9b15d5858bd90 height=337051 log2_work=81.902003 tx=55548435 date=2015-01-02 03:13:42 progress=0.998889 cache=15417 UpdateTip: new best=00000000000000001a059f718edc433af3f15000ed8452958e21e3416614396c height=337052 log2_work=81.902059 tx=55549309 date=2015-01-02 03:35:26 progress=0.998924 cache=16602 UpdateTip: new best=000000000000000002ae38a842176d4b4e0092b5a30a90ddf64e9ad8dcf5323a height=337053 log2_work=81.902114 tx=55549422 date=2015-01-02 03:37:44 progress=0.998927 cache=16764 UpdateTip: new best=00000000000000000038f27237a14acb358d7066efbb5ae3849de267a6d8a818 height=337054 log2_work=81.90217 tx=55549595 date=2015-01-02 03:42:40 progress=0.998935 cache=17150 UpdateTip: new best=000000000000000018df67f45d76bf9f33902351bc648986163dced741ec9cbf height=337055 log2_work=81.902226 tx=55549963 date=2015-01-02 03:50:59 progress=0.998948 cache=17614 UpdateTip: new best=00000000000000000daf09931982f6e1382c5e31c9e35ee805a8f88754022670 height=337056 log2_work=81.902282 tx=55550700 date=2015-01-02 04:08:59 progress=0.998977 cache=18892 Corruption: block checksum mismatch

laanwj commented at 3:26 pm on January 2, 2015: member

These issues are almost always some kind of hw issue. Either corrupt RAM, or an overheating CPU, bitcoin has a knack for exposing them. It starts happening later in the sync process as that is where the CPU starts to be heavily used through e.g., lots of transactions thus secp256k1 verifications.

You could try running with the par=1 option (and -reindex) to restart with only one verification thread. That will make the sync slower, of course.

The issue here is that it shouldn’t continue after a corruption is suspected, but stop the node.

mbrix commented at 2:22 am on January 5, 2015: none

I think I agree with this. Bitcoin is the canary in the coal mine, and even though there isn’t any other indication some kind of disk failure is probably imminent.

So in this instance even though corruption occurs it just stops syncing, and only on shutdown does it issue a warning to that affect. It should probably immediately halt on known corruption.

laanwj added the label UTXO Db and Indexes on Jan 5, 2015

laanwj commented at 8:44 am on January 5, 2015: member

Yes, that is due to an inherent conflict: in network code it’s desirable to continue, what ever happens (otherwise DDoS risk). In the consensus code the program needs to quit on any corruption detected (otherwise fork risk). The network code calls the consensus code to validate…

Closing in favor of #5462, which is that issue.

laanwj closed this on Jan 5, 2015

laanwj added the label Data corruption on Feb 9, 2016

MarcoFalke locked this on Sep 8, 2021

Bitcoin disk corruption #5568