Running 0.12rc01 with pruning. "Corrupted block database" now.

Xekyo commented at 9:18 am on January 20, 2016: member

Started running 0.12rc01 a couple of days ago. As the host only has limited storage, I was running it in pruning mode. Checking on my node today, it is sporting a “Corrupted block database” now.

debug.log:

2016-01-20 09:15:39 ERROR: ApplyTxInUndo: undo data overwriting existing transaction 2016-01-20 09:15:39 ERROR: ApplyTxInUndo: undo data overwriting existing output 2016-01-20 09:15:40 ERROR: ApplyTxInUndo: undo data adding output to missing 2016-01-20 09:16:31 ERROR: VerifyDB(): *** coin database inconsistencies found (last 1 blocks, 0 good transactions before that) 2016-01-20 09:16:32 : Corrupted block database detected.

Unfortunately, debug log is missing everything older than an hour. Is there anything else I could provide?

jonasschnelli added the label Block storage on Jan 20, 2016

pstratem commented at 9:30 am on January 20, 2016: contributor

@Xekyo Did you receive this error when running 0.12rc1 or 0.11.x ?

Xekyo commented at 9:36 am on January 20, 2016: member

@pstratem 0.12rc1

pstratem commented at 9:37 am on January 20, 2016: contributor

@Xekyo what are the lines before this? (So I can take a guess at what was going on)

Xekyo commented at 9:44 am on January 20, 2016: member

Unfortunately, only hundreds of lines repeating the above ERROR messages.

When I tried to restart the process I got this:

2016-01-20 09:35:02 Bitcoin version v0.12.0rc1 (2016-01-13 18:08:05 +0100) 2016-01-20 09:35:02 Prune configured to target 20480MiB on disk for block and undo files. 2016-01-20 09:35:02 Using OpenSSL version OpenSSL 1.0.1e 11 Feb 2013 2016-01-20 09:35:02 Default data directory /home/pi/.bitcoin 2016-01-20 09:35:02 Using data directory /home/pi/.bitcoin 2016-01-20 09:35:02 Using config file /home/pi/.bitcoin/bitcoin.conf 2016-01-20 09:35:02 Using at most 125 connections (1024 file descriptors available) 2016-01-20 09:35:02 Using 4 threads for script verification 2016-01-20 09:35:02 scheduler thread start 2016-01-20 09:35:02 libevent: getaddrinfo: address family for nodename not supported 2016-01-20 09:35:02 Binding RPC on address ::1 port 8332 failed. 2016-01-20 09:35:02 HTTP: creating work queue of depth 16 2016-01-20 09:35:02 Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to us e cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.HTTP: starting 4 worker threads 2016-01-20 09:35:02 Error: Couldn’t open socket for incoming connections (socket returned error Address family not supported by pr otocol (97)) 2016-01-20 09:35:02 Bound to 0.0.0.0:8333 2016-01-20 09:35:02 Cache configuration: 2016-01-20 09:35:02 * Using 2.0MiB for block index database 2016-01-20 09:35:02 * Using 20.0MiB for chain state database 2016-01-20 09:35:02 * Using 28.0MiB for in-memory UTXO set 2016-01-20 09:35:02 init message: Loading block index… 2016-01-20 09:35:02 Opening LevelDB in /home/pi/.bitcoin/blocks/index 2016-01-20 09:35:02 Opened LevelDB successfully 2016-01-20 09:35:02 Using obfuscation key for /home/pi/.bitcoin/blocks/index: 0000000000000000 2016-01-20 09:35:02 Opening LevelDB in /home/pi/.bitcoin/chainstate 2016-01-20 09:35:02 Opened LevelDB successfully 2016-01-20 09:35:02 Using obfuscation key for /home/pi/.bitcoin/chainstate: 0000000000000000 2016-01-20 09:35:29 LoadBlockIndexDB: last block file = 421 2016-01-20 09:35:29 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=87, size=70324481, heights=393971…394057, time =2016-01-19…2016-01-19) 2016-01-20 09:35:29 Checking all blk files are present… 2016-01-20 09:35:29 LoadBlockIndexDB(): Block files have previously been pruned 2016-01-20 09:35:29 LoadBlockIndexDB: transaction index disabled 2016-01-20 09:35:30 LoadBlockIndexDB: hashBestChain=0000000000000000071f795753f5812d12305dbfda26303378a36b89ff33b15e height=394057 date=2016-01-19 20:40:59 progress=0.999572 2016-01-20 09:35:30 init message: Verifying blocks… 2016-01-20 09:35:30 Verifying last 288 blocks at level 3 2016-01-20 09:35:30 ERROR: DisconnectBlock(): added transaction mismatch? database corrupted 2016-01-20 09:35:30 ERROR: ApplyTxInUndo: undo data overwriting existing output

Xekyo commented at 9:54 am on January 20, 2016: member

@pstratem Is there anything else I could provide or run to get more information?

MarcoFalke commented at 10:10 am on January 20, 2016: member

What is the memory usage of RAM and Storage?

Xekyo commented at 10:25 am on January 20, 2016: member

@MarcoFalke: Since it won’t run right now, I don’t know how I would provide accurate numbers. I’m running this on a RaspberryPi 2B with 1 GiB RAM. I’m pruning to 20GiB, on a budget of 64GiB.

1.2G ./.bitcoin/chainstate 60M ./.bitcoin/blocks/index 21G ./.bitcoin/blocks 22G ./.bitcoin

laanwj commented at 10:51 am on January 20, 2016: member

So to be clear: you were already using pruning in 0.11.x, without problems, but after running 0.12.0rc1 on the same data directory this problem occured?

Do you have a backup of your chainstate before the test with rc1?

Xekyo commented at 11:06 am on January 20, 2016: member

laanwj: I was running 0.11.2 without problems with the same settings (pruning to 20GiB), I replaced it with 0.12.0rc1 on Monday evening (less than 48 hours ago), on the same directory.

I don’t have a backup of the chainstate before the test unfortunately.

pstratem commented at 9:37 pm on January 20, 2016: contributor

@laanwj I’ve encountered the same issue running git master

MarcoFalke commented at 10:09 pm on January 20, 2016: member

@pstratem Is your debug.log truncated as well? Did you save it?

mrbandrews commented at 3:57 pm on January 28, 2016: contributor

I tried to reproduce this error by running 0.11.2 on an old datadir (with pruning on) until Jan 18; then switched to 0.12.0rc1 and continued syncing. No problems here. @pstratem When you say you have this issue running master, does that mean running master consistently (as opposed to, say, running 0.11 and then switching to master)?

laanwj added the label Data corruption on Feb 16, 2016

fanquake closed this on May 18, 2017

MarcoFalke locked this on Sep 8, 2021

Running 0.12rc01 with pruning. “Corrupted block database” now. #7382