Corrupted database? #2305

Lohoris commented at 9:53 PM on February 13, 2013: none

Launching bitcoind I get the following error:

************************
EXCEPTION: St13runtime_error       
CLevelDB(): error opening database environment Corruption: missing start of fragmented record(2)       
bitcoin in AppInit()

Using version v0.8.0rc1

gavinandresen commented at 12:03 AM on February 14, 2013: contributor

What OS?

Clean shutdown the run before, or did your machine or bitcoin crash?

This might be a case of "your disk drive was flaky for some unknowable reason."

Lohoris commented at 10:00 AM on February 14, 2013: none

OSX 10.6.8, HFS+ journaled disk. Unfortunately I don't know if bitcoind was running or not while the machine crashed (we can assume it was). On SE Nick pointed out the exactly line I'm hitting, so if you want we could add there some more debugging infos to track it down.

i.e. it is not very good if you have to re-download the whole blockchain for a single corrupted element, especially since not it is made of many small files, if would make more sense if only that file was re-downloaded...

sipa commented at 10:08 AM on February 14, 2013: member

@Lohoris The problem is not a corrupted downloaded file - that's easy. In your case, the database is corrupted, despite several systems which try to prevent ever getting an inconsistent state on disk. You also don't need to redownload anything. Starting with -reindex will rebuild the database using the block chain data you already have.

Lohoris commented at 10:17 AM on February 14, 2013: none

Oh! I'll try, thanks. Of course if this works it could be considered fixed... (it's unlikely I'll be able to try before tomorrow, sorry)

Lohoris commented at 10:00 AM on February 15, 2013: none

reindex at block ~100000 started re-downloading, so I can't go on testing it right now (I've got a limited bandwidth). Maybe I'll go on testing it on monday, sorry for the delay.

qubez commented at 11:07 AM on February 15, 2013: none

I'll spam this bug, for interesting reading:

http://www.codeofhonor.com/blog/whose-bug-is-this-anyway

On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate!

A lot of bug hunting on something as processor and ram intensive and requiring 100% math accuracy like Bitcoin, is likely to be unrepeatable - it's going to be Bitcoin revealing to users their hardware problems. A similar hardware fitness test like the article describes, running before initial-download or during idle cpu, to certify a system's error-free crypto and 8GB of it's storage before attempting to assemble and verify a bit-accurate database, may prevent tears later.

Lohoris commented at 6:02 PM on March 7, 2013: none

Didn't have time to test it properly so far, sorry. I'll close it for now, and I guess if someone else encounters this same problem upgrading, he'll be able to re-open it...

Lohoris closed this on Mar 7, 2013

Lohoris commented at 1:43 PM on March 16, 2013: none

Ok this happened again right now.

I'll reindex and see what happens.

Lohoris reopened this on Mar 16, 2013

Lohoris commented at 10:06 AM on March 20, 2013: none

Reindex worked, as expected. Next time this bug happens, what do I need to do before reindexing to debug it?

zainmw91 commented at 5:12 PM on April 18, 2013: none

I am having this error i dunno what to do i have bitcoin wallet on Windows Vista when i open it i get Bitcoin-QT: System Error: Database Corrupted what shall i do to fix it?

shamoons commented at 5:17 PM on April 18, 2013: none

Try running it from command line: bitcoind -reindex

zainmw91 commented at 5:25 PM on April 18, 2013: none

do you have Skype we can chat on sir? I backed up the enitre bitcoin folder in app data and all removed it re-install re-inserting data to see if it works please add me on skype zainmw91

Date: Thu, 18 Apr 2013 10:18:02 -0700 From: notifications@github.com To: bitcoin@noreply.github.com CC: zainmw91@hotmail.com Subject: Re: [bitcoin] Corrupted database? (#2305)

Try running it from command line: bitcoind -reindex

— Reply to this email directly or view it on GitHub.

h0jeZvgoxFepBQ2C commented at 10:31 AM on May 4, 2013: none

I think this functionality should be added to the GUI too.. Not everybody is a computer pro, and knows how to handle the command shell.. if the index is broken, a window should pop up "it crashed - should we reindex blabla... yes/no?"

gavinandresen commented at 8:16 PM on May 7, 2013: contributor

I'm going to close this as "corruption probably due to the run-out-of-file-descriptors" bug that was fixed.

gavinandresen closed this on May 7, 2013

Lohoris commented at 7:50 AM on May 8, 2013: none

Good, I'll update and let you know if anything happens.

thebalaa commented at 6:27 PM on July 15, 2013: none

This keeps happening to me with version v0.8.3-beta on OS X 10.8.3 Bitcoin-qt spends a few hours "Reindexing blocks on disk.." and then all is well for a day or so, until I gracefully shutdown bitcoin-qt and upon opening I am greeted with the Corrupt database error dialog. Any updates on this?

woeisme commented at 4:16 PM on September 13, 2013: none

I'm getting this on ubuntu 12.04. it re-indexed once when upgrading to 0.8.4 and a few days later its doing it again. I'm running on a SSD for another possible data point that might be relevant. Will running -reindex on the cli compete faster?

nikolajsheller commented at 5:27 PM on September 13, 2013: none

I've just stated seeing this issue on ubuntu 13.04 on a laptop with an SSD. I saw the error earlier today, re-indexed successfully and got the error again now. Edit: I'm running v0.8.3.0-g40809ae-beta.

gmaxwell commented at 7:38 PM on September 13, 2013: contributor

See https://bitcointalk.org/index.php?topic=290922.0 or just update to 0.8.5

nikolajsheller commented at 8:54 PM on September 13, 2013: none

Thank you for the information. There does not seem to be a PPA version of 0.8.5 for Ubuntu yet. I assume the correct place to clone and build from is: https://github.com/bitcoin/bitcoin/tree/0.8.5

gmaxwell commented at 10:15 PM on September 13, 2013: contributor

@nikolajsheller Yes, though the official Linux binaries in tar form are for ubuntu.

chevdor commented at 5:23 PM on October 11, 2013: none

I confirm the issue on MacOS. I got my database corrupted a few times already. I always recover using bitcoin-qt -reindex. That works fine but takes hours. The last time I paid attention to close the application in a clean way. I did and re-opened the application several times without any issue. Nonetheless, after rebooting, the database is corrupted again. I am using 0.8.5-beta, the executable has been built on Friday, September 13, 2013 2:18 AM.

Is there a version that does not have this issue?

raamdev commented at 6:50 PM on November 10, 2013: none

On Mac OS X 10.9 (Mavericks) with Bitcoin-Qt 0.8.5, I was getting the "System Error: Database Corrupted" error every time I started the Bitcoin-Qt client. I tried deleting everything in my Bitcoin directory except peers.dat and wallet.dat, but the error persisted.

What fixed it for me was deleting everything including peers.dat.

For the record, here's what debug.log was showing:

Bitcoin version v0.8.5-beta (2013-09-12 13:35:18 +1000)
Using OpenSSL version OpenSSL 1.0.1c 10 May 2012
Startup time: 2013-11-10 14:58:06
Default data directory /Users/raam/Library/Application Support/Bitcoin
Using data directory /Users/raam/Library/Application Support/Bitcoin
Using at most 125 connections (2560 file descriptors available)
Using 4 threads for script verification
init message: Verifying wallet...
dbenv.open LogDir=/Users/raam/Library/Application Support/Bitcoin/database ErrorFile=/Users/raam/Library/Application Support/Bitcoin/db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
init message: Loading block index...
Opening LevelDB in /Users/raam/Library/Application Support/Bitcoin/blocks/index
Opened LevelDB successfully
Opening LevelDB in /Users/raam/Library/Application Support/Bitcoin/chainstate
Opened LevelDB successfully
LoadBlockIndexDB(): last block file = 3
LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=1563, size=37123020, heights=136929...138491, time=2011-07-18...2011-07-28)
LoadBlockIndexDB(): transaction index disabled
LoadBlockIndexDB(): hashBestChain=00000000000008ef809c70a6ca5da21a03e3e69c105ed29cf285d816ac8a1504  height=138490 date=2011-07-28 14:18:58
init message: Verifying blocks...
Verifying last 288 blocks at level 3
No coin database inconsistencies in last 289 blocks (12767 transactions)
 block index            3730ms
init message: Loading wallet...
nFileVersion = 80500
 wallet                   38ms
LevelDB read failure: Corruption: block checksum mismatch
*** System error: Database corrupted
init message: Loading addresses...
Loaded 18043 addresses from peers.dat  66ms
Flush(false)
wallet.dat refcount=0
wallet.dat checkpoint
wallet.dat detach
wallet.dat closed
DBFlush(false) ended              29ms
StopNode()
Flushed 18043 addresses to peers.dat  34ms
Committing 12178 changed transactions to coin database...
Flush(true)
wallet.dat refcount=0
wallet.dat checkpoint
wallet.dat detach
wallet.dat closed
DBFlush(true) ended              40ms

So, it appears to have been an issue with peers.dat. If anyone needs the peers.dat file to debug this issue, let me know. I saved a copy of the bad version.

sipa commented at 7:00 PM on November 10, 2013: member

peers.dat is not a LevelDB file. Only the blocks/index/ and chainstate/ directories contain LevelDB databases, so the problem must have been with either of those.

raamdev commented at 5:43 PM on November 11, 2013: none

@sipa The only two files that I kept were peers.dat and wallet.dat. I deleted everything else and every time I started the Bitcoin-Qt client I received the "System Error: Database Corrupted" message. It wasn't until I deleted everything including the peers.dat file that the problem resolved itself.

tahoeboarder909 commented at 9:14 AM on November 24, 2013: none

Macbook just crashed trying to sync the blockchain for the first time from Bitcoin QT. Now I'm getting the "System error: Database corrupted" I didn't have any coins in the wallet. Does anyone know how I can reset and trying to sync again? The data directory library file -> ~/Library/Application Support/Bitcoin/ isn't found in terminal when I enter it. Thanks.

chevdor commented at 10:44 AM on November 24, 2013: none

I am getting those db corruptions again those days. I tried restoring my db and blocks folders from backups but it does not do the trick. Only thing that works:reindex That takes very long so this is really painful. On Nov 24, 2013 10:14 AM, "tahoeboarder909" notifications@github.com wrote:

Macbook just crashed trying to sync the blockchain for the first time from Bitcoin QT. Now I'm getting the "System error: Database corrupted. I didn't have any coins in the wallet. Does anyone know how I can reset and trying to sync again? The data directory library file -> ~/Library/Application Support/Bitcoin/ isn't found in terminal when I enter it. Thanks.

— Reply to this email directly or view it on GitHubhttps://github.com/bitcoin/bitcoin/issues/2305#issuecomment-29151681 .

tahoeboarder909 commented at 12:26 AM on November 27, 2013: none

How does one reindex the blockchain??

chevdor commented at 12:28 AM on November 27, 2013: none

Bitcoin-qt -reindex On Nov 27, 2013 1:27 AM, "Kyle Hess" notifications@github.com wrote:

How does one reindex the blockchain??

— Reply to this email directly or view it on GitHubhttps://github.com/bitcoin/bitcoin/issues/2305#issuecomment-29350410 .

fanquake commented at 12:56 AM on November 27, 2013: member

@tahoeboarder909 Using Terminal

open path/to/Bitcoin-Qt.app --args -reindex

mapalmalemus commented at 6:18 PM on August 25, 2017: none

2017-08-25 19:26:16 Bitcoin version v0.14.2 2017-08-25 19:26:16 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1 2017-08-25 19:26:17 Assuming ancestors of block 00000000000000000013176bf8d7dfeab4e1db31dc93bc311b436e82ab226b90 have valid signatures. 2017-08-25 19:26:17 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core no se ha cerrado de forma segura todavÃa..." 2017-08-25 19:26:17 Default data directory C:\Users\Miguel\AppData\Roaming\Bitcoin 2017-08-25 19:26:17 Using data directory C:\Users\Miguel\AppData\Roaming\Bitcoin 2017-08-25 19:26:17 Using config file C:\Users\Miguel\AppData\Roaming\Bitcoin\bitcoin.conf 2017-08-25 19:26:17 Using at most 125 automatic connections (2048 file descriptors available) 2017-08-25 19:26:17 Using 32 MiB out of 32 requested for signature cache, able to store 1048576 elements 2017-08-25 19:26:17 Using 0 threads for script verification 2017-08-25 19:26:17 Using BerkeleyDB version Berkeley DB 4.8.30: (April 9, 2010) 2017-08-25 19:26:17 Using wallet wallet.dat 2017-08-25 19:26:17 scheduler thread start 2017-08-25 19:26:17 init message: Verificando cartera... 2017-08-25 19:26:17 CDBEnv::Open: LogDir=C:\Users\Miguel\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\Miguel\AppData\Roaming\Bitcoin\db.log 2017-08-25 19:26:17 Bound to [::]:8333 2017-08-25 19:26:17 Bound to 0.0.0.0:8333 2017-08-25 19:26:17 Cache configuration: 2017-08-25 19:26:17 * Using 2.0MiB for block index database 2017-08-25 19:26:17 * Using 8.0MiB for chain state database 2017-08-25 19:26:17 * Using 440.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space) 2017-08-25 19:26:17 init message: Cargando indice de bloques... 2017-08-25 19:26:17 Opening LevelDB in C:\Users\Miguel\AppData\Roaming\Bitcoin\blocks\index 2017-08-25 19:26:17 Opened LevelDB successfully 2017-08-25 19:26:17 Using obfuscation key for C:\Users\Miguel\AppData\Roaming\Bitcoin\blocks\index: 0000000000000000 2017-08-25 19:26:17 Opening LevelDB in C:\Users\Miguel\AppData\Roaming\Bitcoin\chainstate 2017-08-25 19:26:18 Corruption: 924 missing files; e.g.: C:\Users\Miguel\AppData\Roaming\Bitcoin\chainstate/499631.ldb 2017-08-25 19:26:20 init message: Cargando indice de bloques... 2017-08-25 19:26:20 Wiping LevelDB in C:\Users\Miguel\AppData\Roaming\Bitcoin\blocks\index 2017-08-25 19:26:20 Opening LevelDB in C:\Users\Miguel\AppData\Roaming\Bitcoin\blocks\index 2017-08-25 19:26:20 Opened LevelDB successfully 2017-08-25 19:26:20 Using obfuscation key for C:\Users\Miguel\AppData\Roaming\Bitcoin\blocks\index: 0000000000000000 2017-08-25 19:26:20 Wiping LevelDB in C:\Users\Miguel\AppData\Roaming\Bitcoin\chainstate 2017-08-25 19:26:20 Opening LevelDB in C:\Users\Miguel\AppData\Roaming\Bitcoin\chainstate 2017-08-25 19:26:20 Corruption: 924 missing files; e.g.: C:\Users\Miguel\AppData\Roaming\Bitcoin\chainstate/499631.ldb 2017-08-25 19:26:21 scheduler thread interrupt 2017-08-25 19:26:21 Shutdown: In progress... 2017-08-25 19:26:21 Shutdown: done 2017-08-25 19:28:47

bitcoin-qt.exe -reindex -chainstate, it doesn´´t work.

I have removed chainstate folder, because it was corrupted and I get to access again and recovery (C:\Users\Miguel\AppData\Roaming\Bitcoin).

How can I repair original chainstate folder?

TheBlueMatt commented at 11:50 PM on August 25, 2017: member

@mapalmalemus hmm, I cant reproduce, but if you simply delete the files in C:\Users\Miguel\AppData\Roaming\Bitcoin\chainstate and then run with bitcoin-qt.exe -reinded-chainstate (note its not separate), that should do it.

mapalmalemus commented at 2:04 AM on August 26, 2017: none

Yes, I have done, thanks!

MarcoFalke locked this on Sep 8, 2021