Coin database inconsistencies found

bg002h commented at 1:55 am on August 1, 2021: none

Describe the issue Core can not check past blocks correctly

What behavior did you expect? Expected setting checkblocks=0, checklevel=6, and assumevalid=0 to result in verification that the chain is valid. And that it would take nigh upon forever. It doesn’t. It poops out in about 12 hours.

What was the actual behavior Debug.log highlights: (Note: Block height at time of message was 693086) init message: Verifying blocks… Verifying last 693086 blocks at level 4 [0%]… [10%]… [20%]… [30%]… [40%]…ERROR: VerifyDB(): *** coin database inconsistencies found (last 601365 blocks, 142571 good transactions before that)

Corrupted block database detected.

How reliably can you reproduce the issue, what are the steps to do so? 100% reproducible. To reproduce, just try rechecking the chain. I don’t know the fastest way to do this, in fact, I’m not even entirely sure how many checklevels there even are but I read 6 on stack exchange…anyhow, just add this to bitcoin.conf and restart node:

assumevalid=0 checkblocks=0 checklevel=6

What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? 0.21.1.0; self compiled. I got the code two different t ways: over Internet using git at the command line and from the friendly lizards working for blockstream in space via satellite using blocksat-cli.

What type of machine are you observing the error on (OS/CPU and disk type)? I’ve checked two intel architecture nodes I have running. Both are little intel NUCs. Ubuntu 20.04 Lts/2.6 ghz intel i7 underclocked at 2.4 GHz max just in case/16 and 32 GB RAM at conservative BIOS settings/NVME SSD (tried it on one node with a single Samsung 970 evo plus formatted ext4 filesystem and another with 2 Samsung 980 NVME in raid 1 configuration…also ext4)

What is your operating system and its version? If Linux, what is your desktop environment and graphical shell? Whatever is default for Ubuntu

Any extra information that might be useful in the debugging process. Someone far smarter than me can reproduce this too (dflate)

Somehow relates to this…see explorers disagreeing (via dflate)

https://blockstream.info/tx/d5d27987d2a3dfc724e359870c6644b40e497bdc0589a033220fe15429d88599?expand

https://explorer.bitcoin.com/btc/tx/d5d27987d2a3dfc724e359870c6644b40e497bdc0589a033220fe15429d88599

Otherwise, not yet. I keep my node off the internet, because, you know, satellite and stuff…but I’ve typed out by hand what seems relevant above.

snpoold commented at 2:51 am on August 1, 2021: none

Blocks 91812 and 91722 coinbase spends cause that issue. I can confirm the observation. When the full scan reach deep down into the data at block 91721 and try to invalidate older blocks in DisconnectBlock … some memories came up, when digging way back into November 2010 :woman_shrugging:

bg002h commented at 3:45 am on August 1, 2021: none

Blocks 91812 and 91722 coinbase spends cause that issue. I can confirm the observation. When the full scan reach deep down into the data at block 91721 and try to invalidate older blocks in DisconnectBlock … some memories came up, when digging way back into November 2010 🤷‍♀️

Maybe related to the overflow thing? https://bitcointalk.org/index.php?topic=822.0

sipa commented at 5:32 am on August 1, 2021: member

These are the two BIP30 exceptions, where coinbases were overwritten.

snpoold commented at 10:23 am on August 1, 2021: none

These are the two BIP30 exceptions, where coinbases were overwritten.

Although it would be nice if we could skip the detection by setting option checkblocks=0 more gracefully and not error break and maybe just a hint to BIP30, what we hardcode anyway in IBD?

 0diff --git a/src/validation.cpp b/src/validation.cpp
 1index d1dd4f472..49158b588 100644
 2--- a/src/validation.cpp
 3+++ b/src/validation.cpp
 4@@ -3980,8 +3980,13 @@
 5         if (ShutdownRequested()) return true;
 6     }
 7-    if (pindexFailure)
 8-        return error("VerifyDB(): *** coin database inconsistencies found (last %i blocks, %i good transactions before that)\n", chainstate.m_chain.Height() - pindexFailure->nHeight + 1, nGoodTransactions);
 9+    if (pindexFailure) {
10+        if (pindexFailure->nHeight == 91722) {
11+            LogPrintf("VerifyDB(): skiping BIP30 exeption at height %d\n", pindexFailure->nHeight);
12+               } else {
13+            return error("VerifyDB(): *** coin database inconsistencies found (last %i blocks, %i accumulated good transactions vtx size before that)\n", chainstate.m_chain.Height() - pindexFailure->nHeight + 1, nGoodTransactions);
14+        }
15+    }

sipa commented at 2:27 pm on August 1, 2021: member

Sure, this is a bug, and should be fixed. I’m just pointing out what it is related to.

bg002h commented at 3:49 pm on August 1, 2021: none

These are the two BIP30 exceptions, where coinbases were overwritten.

I see. I’m rather glad to hear this is expected behavior! Is the code different when validating from Genesis to tip different than tip to Genesis (eg hard coded ~~deception~~ exception in IBD) or is it the over writing of a txid that makes it impossible to validate “backwards” beyond a certain point?

I can’t see how this has any practical impact on the network (other than someone wanting to write a click baity FUD hysteria “news” article). Bitcoin is quirky like that. Things aren’t always as simple as they seem.

jnewbery commented at 2:22 pm on August 2, 2021: contributor

Is the code different when validating from Genesis to tip different than tip to Genesis (eg hard coded deception in IBD) or is it the over writing of a txid that makes it impossible to validate “backwards” beyond a certain point?

Yes, there is special handling of the duplicate coinbase transactions in ConnectBlock():

https://github.com/bitcoin/bitcoin/blob/b620b2d58a55a88ad21da70cb2000863ef17b651/src/validation.cpp#L1758-L1759

https://github.com/bitcoin/bitcoin/blob/b620b2d58a55a88ad21da70cb2000863ef17b651/src/validation.cpp#L1825-L1834

There is no such special handling in the analogous code DisconnectBlock() which checks that the ouputs of the transactions exist in the UTXO set:

https://github.com/bitcoin/bitcoin/blob/b620b2d58a55a88ad21da70cb2000863ef17b651/src/validation.cpp#L1535-L1544

Which means that when VerifyDB() tries to rewind the effects of the blocks, it finds the inconsistency.

bg002h commented at 3:21 pm on August 2, 2021: none

This seems like a bug worth preserving for someone looking to cut their teeth on Core. I’m just a physician with rather modest coding skills and this has even my interest piqued…maybe someone more qualified (and/or less busy) can pick up this ball and run with it.

All that said, performance for checking all the blocks seems really rather good. I thought it would take a day or more, but it looks like it was gonna be about half that. There’s some deep magic going on to make this happen so quickly.

achow101 closed this on Oct 13, 2022

bitcoin locked this on Oct 13, 2023

Coin database inconsistencies found #22596