ChainState DB uncorrupted itself #2785

Running 0.8.2-beta on OSX 10.8.4

Opened Qt to a corrupted chainstate, and was told to rebuild. Decided to do it later, so chose abort from the popup.

Opened Qt the next day to no corruption message, and it's just syncing per normal.

This debug snippet below is from the shutdown before corruption, opening the corrupted wallet and choosing to not rebuild and finally opening the wallet and having it load successfully.

Added 1 addresses from 173.197.107.3: 683 tried, 13640 new received getdata for: tx a535a410388497f78b4bb77b9c929242d6174b0d26f396cc10212d9dcfafb95e CTxMemPool::accept() : accepted e3406befa1868f30485c44a460586ec374186b3ce07e1c287b6afdbb44a8909f (poolsz 1451) received getdata for: tx e3406befa1868f30485c44a460586ec374186b3ce07e1c287b6afdbb44a8909f received getdata for: tx e3406befa1868f30485c44a460586ec374186b3ce07e1c287b6afdbb44a8909f Flushed 14323 addresses to peers.dat 24ms Added 1 addresses from 173.197.107.3: 683 tried, 13640 new received getdata for: tx 8a7b68238126e3a167f8d4369bc1b5c832147e6b3aa5a291b2af462f2a4b51ff dumpaddr thread stop msghand thread interrupt opencon thread interrupt net thread interrupt addcon thread interrupt Flush(false) DBFlush(false) ended 0ms StopNode() UPNP_DeletePortMapping() returned : 0 upnp thread interrupt Flushed 14322 addresses to peers.dat 25ms Committing 775 changed transactions to coin database... Flush(true) wallet.dat refcount=0 wallet.dat checkpoint wallet.dat detach wallet.dat closed DBFlush(true) ended 67ms

Bitcoin version v0.8.2-beta (2013-05-25 08:48:25 -0700) Using OpenSSL version OpenSSL 1.0.1c 10 May 2012 Startup time: 2013-06-22 15:18:30 Default data directory /Users/xxxx/Library/Application Support/Bitcoin Using data directory /Users/xxxx/Library/Application Support/Bitcoin Using at most 125 connections (2560 file descriptors available) Using 8 threads for script verification init message: Verifying wallet... dbenv.open LogDir=/Users/xxxx/Library/Application Support/Bitcoin/database ErrorFile=/Users/xxxx/Library/Application Support/Bitcoin/db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 init message: Loading block index... Opening LevelDB in /Users/xxxx/Library/Application Support/Bitcoin/blocks/index Opened LevelDB successfully Opening LevelDB in /Users/xxxx/Library/Application Support/Bitcoin/chainstate Opened LevelDB successfully LoadBlockIndexDB(): last block file = 67 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=175, size=28716039, heights=242380...242554, time=2013-06-20...2013-06-21) LoadBlockIndexDB(): transaction index disabled LoadBlockIndexDB(): hashBestChain=00000000000000241476e0d024a206a2e8555a97f0fc0b5ab522d492b5dab34f height=242554 date=2013-06-21 06:54:39 init message: Verifying blocks... Verifying last 288 blocks at level 3 LevelDB read failure: Corruption: block checksum mismatch Flush(false) DBFlush(false) ended 0ms StopNode() Flushed 0 addresses to peers.dat 1ms Committing 8 changed transactions to coin database... Flush(true) DBFlush(true) ended 0ms

Bitcoin version v0.8.2-beta (2013-05-25 08:48:25 -0700) Using OpenSSL version OpenSSL 1.0.1c 10 May 2012 Startup time: 2013-06-23 04:32:54 Default data directory /Users/xxxx/Library/Application Support/Bitcoin Using data directory /Users/xxxx/Library/Application Support/Bitcoin Using at most 125 connections (2560 file descriptors available) Using 8 threads for script verification init message: Verifying wallet... dbenv.open LogDir=/Users/xxxx/Library/Application Support/Bitcoin/database ErrorFile=/Users/xxxx/Library/Application Support/Bitcoin/db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 init message: Loading block index... Opening LevelDB in /Users/xxxx/Library/Application Support/Bitcoin/blocks/index Opened LevelDB successfully Opening LevelDB in /Users/xxxx/Library/Application Support/Bitcoin/chainstate Opened LevelDB successfully LoadBlockIndexDB(): last block file = 67 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=175, size=28716039, heights=242380...242554, time=2013-06-20...2013-06-21) LoadBlockIndexDB(): transaction index disabled LoadBlockIndexDB(): hashBestChain=00000000000000241476e0d024a206a2e8555a97f0fc0b5ab522d492b5dab34f height=242554 date=2013-06-21 06:54:39 init message: Verifying blocks... Verifying last 288 blocks at level 3 No coin database inconsistencies in last 150 blocks (53329 transactions) block index 17768ms init message: Loading wallet... nFileVersion = 80202 wallet 1171ms init message: Loading addresses... Loaded 0 addresses from peers.dat 1ms mapBlockIndex.size() = 242556 nBestHeight = 242554 setKeyPool.size() = 100 mapWallet.size() = 19 mapAddressBook.size() = 24 dnsseed thread start upnp thread start Loading addresses from DNS seeds (could take a while) net thread start addcon thread start opencon thread start msghand thread start dumpaddr thread start Flushed 0 addresses to peers.dat 1ms init message: Done loading

Same issue again. This time I've opened Qt to see the corruption message, and chosen to not rebuild. I then opened Qt again to be certain that there was corruption, and again chosen to not rebuild.

I've then shutdown my computer, and when I opened Qt the next day with the intent of starting the rebuild, I've seen no corruption message and Qt opens and functions normally. Debug excerpt below.

CTxMemPool::accept() : accepted 0ff7e52e6db85029f84e1deb0555fd7a55ef89517106ff6cc7b20ebb826f8e01 (poolsz 1365) received getdata for: tx 0ff7e52e6db85029f84e1deb0555fd7a55ef89517106ff6cc7b20ebb826f8e01 Added 1 addresses from 108.28.174.70: 144 tried, 14212 new CTxMemPool::accept() : accepted a5d26c75f362f242eb8c69dc1de1a23dce06391705f8a32dde6c9a9ded20f02c (poolsz 1366) CTxMemPool::accept() : accepted 7637ee26947110612af59fb3eacc7197fcb3354d6176f94e6f7bf824905da36f (poolsz 1367) msghand thread interrupt opencon thread interrupt dumpaddr thread stop addcon thread interrupt net thread interrupt Flush(false) DBFlush(false) ended 0ms StopNode() UPNP_DeletePortMapping() returned : 0 upnp thread interrupt Flushed 14356 addresses to peers.dat 25ms Committing 487 changed transactions to coin database... Flush(true) wallet.dat refcount=0 wallet.dat checkpoint wallet.dat detach wallet.dat closed DBFlush(true) ended 87ms

Bitcoin version v0.8.3-beta (2013-06-25 10:27:24 -0400) Using OpenSSL version OpenSSL 1.0.1c 10 May 2012 Startup time: 2013-07-15 09:10:22 Default data directory /Users/xxxxx/Library/Application Support/Bitcoin Using data directory /Users/xxxxx/Library/Application Support/Bitcoin Using at most 125 connections (2560 file descriptors available) Using 8 threads for script verification init message: Verifying wallet... dbenv.open LogDir=/Users/xxxxx/Library/Application Support/Bitcoin/database ErrorFile=/Users/xxxxx/Library/Application Support/Bitcoin/db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 init message: Loading block index... Opening LevelDB in /Users/xxxxx/Library/Application Support/Bitcoin/blocks/index Opened LevelDB successfully Opening LevelDB in /Users/xxxxx/Library/Application Support/Bitcoin/chainstate Opened LevelDB successfully LoadBlockIndexDB(): last block file = 70 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=494, size=61594836, heights=246066...246559, time=2013-07-11...2013-07-14) LoadBlockIndexDB(): transaction index disabled LoadBlockIndexDB(): hashBestChain=00000000000000341ec038963b45848695b667222c4beccaf6b6a930bb2c2335 height=246559 date=2013-07-14 17:43:56 init message: Verifying blocks... Verifying last 288 blocks at level 3 LevelDB read failure: Corruption: block checksum mismatch Flush(false) DBFlush(false) ended 0ms StopNode() Flushed 0 addresses to peers.dat 1ms Committing 5518 changed transactions to coin database... Flush(true) DBFlush(true) ended 0ms

Bitcoin version v0.8.3-beta (2013-06-25 10:27:24 -0400) Using OpenSSL version OpenSSL 1.0.1c 10 May 2012 Startup time: 2013-07-15 09:27:36 Default data directory /Users/xxxxx/Library/Application Support/Bitcoin Using data directory /Users/xxxxx/Library/Application Support/Bitcoin Using at most 125 connections (2560 file descriptors available) Using 8 threads for script verification init message: Verifying wallet... dbenv.open LogDir=/Users/xxxxx/Library/Application Support/Bitcoin/database ErrorFile=/Users/xxxxx/Library/Application Support/Bitcoin/db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 init message: Loading block index... Opening LevelDB in /Users/xxxxx/Library/Application Support/Bitcoin/blocks/index Opened LevelDB successfully Opening LevelDB in /Users/xxxxx/Library/Application Support/Bitcoin/chainstate Opened LevelDB successfully LoadBlockIndexDB(): last block file = 70 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=494, size=61594836, heights=246066...246559, time=2013-07-11...2013-07-14) LoadBlockIndexDB(): transaction index disabled LoadBlockIndexDB(): hashBestChain=00000000000000341ec038963b45848695b667222c4beccaf6b6a930bb2c2335 height=246559 date=2013-07-14 17:43:56 init message: Verifying blocks... Verifying last 288 blocks at level 3 LevelDB read failure: Corruption: block checksum mismatch Flush(false) DBFlush(false) ended 0ms StopNode() Flushed 0 addresses to peers.dat 0ms Committing 5518 changed transactions to coin database... Flush(true) DBFlush(true) ended 0ms

Bitcoin version v0.8.3-beta (2013-06-25 10:27:24 -0400) Using OpenSSL version OpenSSL 1.0.1c 10 May 2012 Startup time: 2013-07-16 07:36:40 Default data directory /Users/xxxxx/Library/Application Support/Bitcoin Using data directory /Users/xxxxx/Library/Application Support/Bitcoin Using at most 125 connections (2560 file descriptors available) Using 8 threads for script verification init message: Verifying wallet... dbenv.open LogDir=/Users/xxxxx/Library/Application Support/Bitcoin/database ErrorFile=/Users/xxxxx/Library/Application Support/Bitcoin/db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 init message: Loading block index... Opening LevelDB in /Users/xxxxx/Library/Application Support/Bitcoin/blocks/index Opened LevelDB successfully Opening LevelDB in /Users/xxxxx/Library/Application Support/Bitcoin/chainstate Opened LevelDB successfully LoadBlockIndexDB(): last block file = 70 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=494, size=61594836, heights=246066...246559, time=2013-07-11...2013-07-14) LoadBlockIndexDB(): transaction index disabled LoadBlockIndexDB(): hashBestChain=00000000000000341ec038963b45848695b667222c4beccaf6b6a930bb2c2335 height=246559 date=2013-07-14 17:43:56 init message: Verifying blocks... Verifying last 288 blocks at level 3 No coin database inconsistencies in last 255 blocks (53474 transactions) block index 16640ms init message: Loading wallet... nFileVersion = 80300 wallet 1251ms init message: Loading addresses... Loaded 0 addresses from peers.dat 1ms mapBlockIndex.size() = 246561 nBestHeight = 246559 setKeyPool.size() = 100 mapWallet.size() = 25 mapAddressBook.size() = 29 dnsseed thread start upnp thread start net thread start addcon thread start Loading addresses from DNS seeds (could take a while) opencon thread start msghand thread start dumpaddr thread start init message: Done loading refreshWallet Added 28 addresses from 178.18.90.41: 0 tried, 28 new

Very strange, this is an error coming from inside leveldb. Could this be some kind of transient disk/memory corruption issue?

LevelDB read failure: Corruption: block checksum mismatch

I suppose it's related to the other MacOSX leveldb corruption issues

Related discussion http://bitcoinstats.com/irc/bitcoin-dev/logs/2013/07/25#l1374759085

Can you guys with the issue please run this on a terminal and paste the output here?

diskutil list

Please describe which disk and partition that the db / bitcoin-qt supporting files is on as well - trying to track down a bigger picture issue.

/dev/disk0 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *120.0 GB disk0 1: EFI 209.7 MB disk0s1 2: Apple_HFS Lion SSD 119.7 GB disk0s2 /dev/disk1 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *1.0 TB disk1 1: EFI 209.7 MB disk1s1 2: Apple_HFS SpinTB 999.9 GB disk1s2

I've had corruption happen running on both of these (the SSD and the non-SSD).

http://www.skypaint.com/leveldb_corruption/ has a couple of corrupt chainstate dbs. (or will soon, second upload is still happening).

Thanks checking it out now! Can you confirm that both corruptions occurred on disks using HFS??? Any if possible can you confirm if these are HFS with case sensitive settings enabled?

I don't have the skills to fix this but I can recreate the problem.

1. Go to system preferences and click on Date/Time. Set the date to 2012. Close System preferences and hit save. Open Bitcoin-qt and get the corrupted database message. Click abort.
2. Set system preferences and click set date and time automatically. Open Bitcoin-qt and the database is fine. Can you use an atomic clock API and rely on that instead of the system clock to prevent leveldb from becoming corrupted?

@jordan314 that's a different issue ("CheckBlock() : block timestamp too far in the future") not related to corruption, see #2007

I believe I have a fix. I've submitted to the LevelDB devs for review: https://groups.google.com/d/msg/leveldb/GXhx8YvFiig/FhMHoWz7MNAJ

Closing, this is the well-known MacOSX corruption issue and should be fixed in 0.8.6 and master.

I'm having a sort of error like this, my node on a VPS server crashes pretty often because of this error. I've made a script that checks if the node is running every minute, but it's kinda annoying. I'm running Bitcoin core v0.10.1 on Ubuntu 12.04_x64 Debug.log file:

3c26bfff664f03a939468cb0 height=298522 log2_work=78.316982 tx=37876897 date=2014-05-01 01:13:17 progress=0.260683 cache=13132 2015-06-13 12:00:20 opencon thread interrupt 2015-06-13 12:00:20 dumpaddr thread stop 2015-06-13 12:00:20 addcon thread interrupt 2015-06-13 12:00:20 net thread interrupt 2015-06-13 12:00:26 UpdateTip: new best=0000000000000000088c21d8a6f9bae5f5009ca14525c21884a83679b9c12b18 height=298523 log2_work=78.317113 tx=37877528 date=2014-05-01 01:29:17 progress=0.260703 cache=14282 2015-06-13 12:00:26 msghand thread interrupt 2015-06-13 12:00:26 Shutdown: In progress... 2015-06-13 12:00:26 RPCAcceptHandler: Error: Operation canceled 2015-06-13 12:00:26 StopNode() 2015-06-13 12:00:28 Shutdown: done 2015-06-13 12:01:01