Corruption: block checksum mismatch (System error while flushing: Database corrupted)

poiuty commented at 2:16 pm on June 8, 2016: contributor

 02016-06-08 13:07:20 Relaying wtx 7adf5eac6eddb2d18ebd2eb555bc82fec2e329aa1dfb3cea002f80d74f383836
 12016-06-08 13:07:20 ResendWalletTransactions: rebroadcast 1 unconfirmed transactions
 22016-06-08 13:11:41 UpdateTip: new best=000000000000000003688d5bcf1351e7e9fe39c385f6335c5ab4bb7dd3587662  height=415352  log2_work=84.796227  tx=134396077  date=2016-06-08 13:11:15 progress=1.000000  cache=36.5MiB(23669tx)
 32016-06-08 13:12:43 UpdateTip: new best=0000000000000000043a9975933cee8e0d783c2550ffaa7c16a3a556fa18cb0e  height=415353  log2_work=84.796263  tx=134396943  date=2016-06-08 13:11:52 progress=1.000000  cache=0.5MiB(0tx)
 42016-06-08 13:12:43 AddToWallet 7adf5eac6eddb2d18ebd2eb555bc82fec2e329aa1dfb3cea002f80d74f383836  update
 52016-06-08 13:15:42 UpdateTip: new best=0000000000000000012d454dcdf8485b673f49cdefcd1b5f24dcfc37326c1a26  height=415354  log2_work=84.796299  tx=134397562  date=2016-06-08 13:15:16 progress=1.000000  cache=4.9MiB(2784tx)
 62016-06-08 13:33:58 UpdateTip: new best=0000000000000000058a59f94ef788ec6958c758ee483bb1ba85e911d3f9cf27  height=415355  log2_work=84.796335  tx=134400200  date=2016-06-08 13:33:09 progress=1.000000  cache=25.7MiB(12042tx)
 72016-06-08 13:35:00 UpdateTip: new best=0000000000000000037276bb3fccf25942eac416d5c537eeca39e4c7c7897f31  height=415356  log2_work=84.796371  tx=134401162  date=2016-06-08 13:34:06 progress=1.000000  cache=26.0MiB(13379tx)
 82016-06-08 13:38:16 Pre-allocating up to position 0x3000000 in blk00539.dat
 92016-06-08 13:38:17 Pre-allocating up to position 0x500000 in rev00539.dat
102016-06-08 13:38:17 UpdateTip: new best=00000000000000000409150a8c24b3699e7eeaf93933be6d3d3b0eca1ada552b  height=415357  log2_work=84.796408  tx=134402028  date=2016-06-08 13:37:54 progress=1.000000  cache=48.9MiB(16972tx)
112016-06-08 13:44:22 ERROR: AcceptToMemoryPool: rejecting replacement bca20ebfd6bfa5f3d7e6c3bd4d4bb5b406de5045c27576ab5a52d493094874fc; new feerate 0.00052236 BTC/kB <= old feerate 0.00156710 BTC/kB
122016-06-08 13:53:19 UpdateTip: new best=00000000000000000476e4e2a6dee84669f9737b638d4e550c950b07828463e7  height=415358  log2_work=84.796444  tx=134404706  date=2016-06-08 13:52:07 progress=1.000000  cache=54.9MiB(9044tx)
132016-06-08 13:57:02 UpdateTip: new best=0000000000000000021f679b24f2c67466937a47210e55574046b47a7b943b8f  height=415359  log2_work=84.79648  tx=134405788  date=2016-06-08 13:55:55 progress=1.000000  cache=58.5MiB(13565tx)
142016-06-08 13:57:52 Corruption: block checksum mismatch
152016-06-08 13:57:52 *** System error while flushing: Database corrupted
162016-06-08 13:57:59 Corruption: block checksum mismatch
172016-06-08 13:57:59 *** System error while flushing: Database corrupted
182016-06-08 13:57:59 Corruption: block checksum mismatch
192016-06-08 13:57:59 *** System error while flushing: Database corrupted
202016-06-08 13:57:59 tor: Thread interrupt
212016-06-08 13:57:59 torcontrol thread exit
222016-06-08 13:57:59 opencon thread interrupt
232016-06-08 13:57:59 addcon thread interrupt
242016-06-08 13:57:59 scheduler thread interrupt
252016-06-08 13:57:59 net thread interrupt
262016-06-08 13:58:01 msghand thread interrupt
272016-06-08 13:58:01 Shutdown: In progress...
282016-06-08 13:58:01 StopNode()
292016-06-08 13:58:01 Corruption: block checksum mismatch
302016-06-08 13:58:01 *** System error while flushing: Database corrupted
312016-06-08 13:58:01 Shutdown: done

jonasschnelli added the label Block storage on Jun 9, 2016

jonasschnelli commented at 6:44 am on June 9, 2016: contributor

Your Issue title does not reflect your problem. ERROR: AcceptToMemoryPool happens during normal operations (although I think we should change it from ERROR to INFO).

2016-06-08 13:57:59 Corruption: block checksum mismatch is more important.

Your node has shutdown because of a corrupted block file. Either you have force terminated your node in the wrong moment or your Disk/Storage-Device is malfunctioning.

laanwj commented at 7:54 am on June 9, 2016: member

Your Issue title does not reflect your problem. ERROR: AcceptToMemoryPool happens during normal operations (although I think we should change it from ERROR to INFO).

This is solved in master. See also #7592

laanwj added the label Data corruption on Jun 9, 2016

MarcoFalke renamed this:
~~ERROR: AcceptToMemoryPool: rejecting replacement~~
Corruption: block checksum mismatch (System error while flushing: Database corrupted)
on Jun 9, 2016

vadipp commented at 8:35 am on July 12, 2016: none

I’m having this issue on FreeBSD with ZFS storage (so no on-disk data corruption is possible). I’ve already performed reindexing, but after several days of working in background, bitcoind crashed again.

Bitcoin Core Daemon version v0.12.1.0-g9779e1e (built from ports).

 02016-07-12 06:33:54 UpdateTip: new best=000000000000000004d423b7951f1d2db204ed70eadf0a17125f9d315f734372  height=420315  log2_work=84.972631  tx=141779952  date=2016-07-11 18:39:09 progress=0.999735  cache=57.6MiB(36867tx)
 12016-07-12 06:33:54 UpdateTip: 5 of last 100 blocks have unexpected version
 22016-07-12 06:33:58 UpdateTip: new best=0000000000000000026d11266fe61e9cc64b458c84152673d50d1c8e0d0abb88  height=420316  log2_work=84.972666  tx=141781031  date=2016-07-11 18:44:01 progress=0.999737  cache=59.5MiB(39496tx)
 32016-07-12 06:33:58 UpdateTip: 5 of last 100 blocks have unexpected version
 42016-07-12 06:33:59 Corruption: block checksum mismatch
 52016-07-12 06:33:59 *** System error while flushing: Database corrupted
 62016-07-12 06:33:59 Error: Error: A fatal internal error occurred, see debug.log for details
 72016-07-12 06:33:59 ERROR: ProcessNewBlock: ActivateBestChain failed
 82016-07-12 06:33:59 tor: Thread interrupt
 92016-07-12 06:33:59 scheduler thread interrupt
102016-07-12 06:33:59 torcontrol thread exit
112016-07-12 06:33:59 addcon thread interrupt
122016-07-12 06:33:59 msghand thread interrupt
132016-07-12 06:33:59 net thread interrupt
142016-07-12 06:34:03 opencon thread interrupt
152016-07-12 06:34:03 Shutdown: In progress...
162016-07-12 06:34:03 StopNode()
172016-07-12 06:34:04 Corruption: block checksum mismatch
182016-07-12 06:34:04 *** System error while flushing: Database corrupted
192016-07-12 06:34:04 Error: Error: A fatal internal error occurred, see debug.log for details
202016-07-12 06:34:04 Shutdown: done

andrejpodzimek commented at 10:01 pm on July 19, 2016: none

I’m getting various types of database corruption errors (mostly just like this one) when trying to synchronize bitcoin-qt after quite a long time (~one year) without running it. One of the failures even broke my wallet.dat (no idea how that could happen) to the point where I had to restore it from backups.

The machine runs Btrfs, so silent data corruption is highly unlikely. (Btrfs is a checksummed filesystem, just like ZFS.) The machine doesn’t have any software issues other than the bitcoin problem and passes days of memtesting without failure.

BTW, --reindex doesn’t solve the problem; it simply fails again sooner or later in the reindexing process. The only thing that does help is bitcoin-qt -reindex -rpcthreads=1. This leads me to believe that there’s an ugly race condition lurking somewhere in bitcoin and/or bitcoin-qt.

However, I don’t know why the tweak above helped and whether it was just a coincidence, because IIUC, the -rpcthreads option would only set the number of threads serving requests from the frontend(s) and should not have any direct impact on reindexing. And indeed, the number of threads that run during the reindexing proces is still the same. Yet it just doesn’t fail with -rpcthreads=1.

My system: ArchLinux, kernel 4.6.4 (vanilla kernel with a homebrew configuration, not the distro kernel), bitcoin-qt 0.12.1-1, gcc-multilib 6.1.1-3.

liori commented at 11:52 pm on July 31, 2016: none

Getting exactly the same problem. Manually compiled bitcoin from git tag 0.12.1, with db 4.8.30 straight from Oracle. Debian Jessie. Starting each time from scratch: no preexisting wallet, no blockchain. Tried two different drives (firstly on a HDD inside an external USB enclosure with btrfs, then twice on an internal HDD with ext4, then once again the external drive). btrfs checksums clean. In all cases, the client was downloading the blockchain for 8÷24 hours, then failed with Corruption: block checksum mismatch. In the second of these cases the client started downloading more blocks after a simple restart (with no additional switches), only to fail few hours later. In the third and fourth attempt the client had to be killed with kill -9 to stop. -reindex doesn’t help. -rpcthreads=1 doesn’t help either. Is there anything I can do to debug this problem?

gmaxwell commented at 7:22 pm on January 1, 2017: contributor

Please close this issue, this looks like a standard memory / disk corruption report and the original reporter hasn’t come back.

MarcoFalke closed this on Jan 1, 2017

liori commented at 8:05 pm on January 1, 2017: none

Indeed, my issue turned out to be faulty RAM (sorry for not replying earlier, forgot about this ticket). Both memtest86 and adding memtest to Linux’s parameters were finding the problem. Running with memtest permanently seems to be good enough in my case to have the bitcoin client run correctly.

MarcoFalke locked this on Sep 8, 2021

Corruption: block checksum mismatch (System error while flushing: Database corrupted) #8174