LevelDB "Corruption: bad entry in block" on 0.9.1-beta #4263

$ bitcoind --help
Bitcoin Core Daemon version v0.9.1-beta
$ uname -a
Linux raspberry 3.12.19+ [#681](/bitcoin-bitcoin/681/) PREEMPT Fri May 9 16:36:10 BST 2014 armv6l GNU/Linux

It's a small system with limited resources (512MB) which may play a role in these problems which happen about once a week. I compiled bitcoind from Github source (./autogen.sh ; ./configure). After ingesting blocks for 2-3 days bitcoind crashed due to a corruption in LevelDB.

2014-05-31 00:07:45 UpdateTip: new best=000000000000000034c0a9f377fdd6331703e255b0bb08fa947fc072f20c0754  height=268080  log2_work=73.566934  tx=26511008  date=2013-11-05 13:25:36 progress=0.359374
2014-05-31 00:07:45 ProcessBlock: ACCEPTED
2014-05-31 00:09:01 Corruption: bad entry in block
2014-05-31 00:09:02 *** System error: Database corrupted
2014-05-31 00:09:02 Error: System error: Database corrupted
2014-05-31 00:09:02 ERROR: ProcessBlock() : AcceptBlock FAILED
2014-05-31 00:09:02 Loaded 904 blocks from external file in 15259139ms
2014-05-31 00:09:02 Reindexing block file blk00091.dat...
2014-05-31 00:09:02 addcon thread interrupt
2014-05-31 00:09:02 net thread interrupt
2014-05-31 00:09:02 opencon thread interrupt
2014-05-31 00:09:02 msghand thread interrupt
2014-05-31 00:09:02 dumpaddr thread stop
2014-05-31 00:09:02 Shutdown : In progress...
2014-05-31 00:09:03 StopNode()
2014-05-31 00:09:43 Corruption: bad entry in block

Upon restart LevelDB rescanned blocks and recovered:

2014-05-31 06:39:48 Default data directory /home/rippler/.bitcoin
2014-05-31 06:39:48 Using data directory /home/rippler/.bitcoin
2014-05-31 06:39:48 Using at most 125 connections (1024 file descriptors available)
2014-05-31 06:39:49 Error: Couldn't open socket for incoming connections (socket returned error 97)
2014-05-31 06:39:49 Bound to 0.0.0.0:8333
2014-05-31 06:39:49 init message: Loading block index...
2014-05-31 06:39:49 Opening LevelDB in /home/rippler/.bitcoin/blocks/index
2014-05-31 06:39:51 Opened LevelDB successfully
2014-05-31 06:39:51 Opening LevelDB in /home/rippler/.bitcoin/chainstate
2014-05-31 06:40:05 Opened LevelDB successfully
2014-05-31 06:45:06 LoadBlockIndexDB(): last block file = 90
2014-05-31 06:45:06 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=905, size=101159236, heights=267177...268081, time=2013-10-31...2013-11-05)
2014-05-31 06:45:06 LoadBlockIndexDB(): transaction index enabled
2014-05-31 06:45:07 LoadBlockIndexDB(): hashBestChain=0000000000000003b30c326310fb143552d5a1dfa12fc5fdd5c4d5e925946be7 height=268019 date=2013-11-05 04:55:36 progress=0.358755
2014-05-31 06:45:07 init message: Verifying blocks...
2014-05-31 06:45:07 Verifying last 288 blocks at level 3

There are known corruption problems on ARM.

I've run quite a few nodes on ARM boxes and have never encountered corruption problems.

The only thing that you shouldn't do is copy the leveldb database files between ARM and x86, they are incompatible in subtle but messy ways (see #2293).

Thanks laanwj, I tried deleting all blocks and chainstate files and it still didn't work for me, it'd just fail in different ways errors were all related to the DB.
I'm building 0.9.2 now, will try with that.

Rippler how big is your db ?

At which level ? *

*http://dailyjs.com/2013/04/19/leveldb-and-node-1/#%21

Sorry it's been 60+ days I think I waited for a week and then tried various things and messed the system up. I'll post back if I resurrect that system. I had several bugs at various stages so I can't remember which bug happened at which level, but I think at the very end when this happened I think blockchain was downloaded up to block 200K or thereabout (hopefully you can guess which level that is).