LevelDB error: Corruption: block checksum mismatch didn't trigger reindex.

Geremia commented at 4:34 am on May 18, 2024: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

For the past 29 hours, it’s been stuck trying to sync:

Number of blocks left Unknown. Syncing Headers (843771, 100.0%)…

and

Connecting to peers…

I’m also pruning (prune=550).

Expected behaviour

It should sync the full chain and reindex if needed, not try to sync what it can’t (due to possibly corrupt chainstate DB).

Steps to reproduce

Open up bitcoin-qt. It doesn’t say I need to reindex, but everything else works. It connects to many peers, but very little date (<25 KB) is transferred between each of them.

Relevant log output

 02024-05-16T22:38:54Z Writing 0 unbroadcast transactions to disk.
 12024-05-16T22:38:54Z Dumped mempool: 0.055s to copy, 2.286s to dump
 22024-05-16T22:38:54Z Flushed fee estimates to fee_estimates.dat.
 32024-05-16T22:38:54Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/geremia/.bitcoin/chainstate/5753777.ldb
 42024-05-16T22:38:54Z You can use -debug=leveldb to get more complete diagnostic messages
 52024-05-16T22:38:54Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/geremia/.bitcoin/chainstate/5753777.ldb
 62024-05-16T22:38:54Z Error: A fatal internal error occurred, see debug.log for details
 72024-05-16T22:38:54Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/geremia/.bitcoin/chainstate/5753777.ldb)
 82024-05-16T22:38:54Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/geremia/.bitcoin/chainstate/5753777.ldb
 92024-05-16T22:38:54Z You can use -debug=leveldb to get more complete diagnostic messages
102024-05-16T22:38:54Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/geremia/.bitcoin/chainstate/5753777.ldb
112024-05-16T22:38:54Z Error: A fatal internal error occurred, see debug.log for details
122024-05-16T22:38:54Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/geremia/.bitcoin/chainstate/5753777.ldb)
132024-05-16T22:38:54Z [default wallet] Releasing wallet
142024-05-16T22:38:54Z Shutdown: done

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v27.0

Operating system and version

slackware64-current

Machine specifications

No response

maflcko commented at 8:22 am on May 18, 2024: member

Can you clarify this? The debug log says “Shutdown: done”, but you claim that “It connects to many peers”, indicating that the program is still running?

If this is the case, it would be a bug, because the program should abort and shutdown when file corruption is detected in the leveldb database.

maflcko added the label Bug on May 18, 2024

maflcko added the label Data corruption on May 18, 2024

maflcko added the label Questions and Help on May 18, 2024

Geremia commented at 6:56 pm on May 18, 2024: none

@maflcko

The debug log says “Shutdown: done”, but you claim that “It connects to many peers”, indicating that the program is still running?

When I restart bitcoin-qt, it should have asked to reindex, but instead it connects to peers and tries to sync (but stalls):

screenshot

The debug log doesn’t show any more corrupt DB errors than the one I pasted above.

Geremia commented at 7:46 pm on May 18, 2024: none

Later, I tried Knots, but the issue occurs with that fork, too. I also noticed that from 2024-05-18T04:48:21Z to 2024-05-18T04:48:35Z (14 second duration) I got 30 messages in debug.log that say:

 02024-05-18T04:48:21Z UpdateTip: new best=00000000000000000003483012d42cbcd1ebe486364e46a56b8800fadfd43b74 height=843741 version=0x20f12000 log2_work=94.927416 tx=1005741855 date='2024-05-16T18:51:28Z' progress=0.999244 cache=1.4MiB(11882txo) warning='Miner violated version bit protocol'
 12024-05-18T04:48:22Z UpdateTip: new best=0000000000000000000014c2b943de6983dc19f4f0ea6b63d57448bfc8de5038 height=843742 version=0x2035a000 log2_work=94.927430 tx=1005747486 date='2024-05-16T18:53:35Z' progress=0.999245 cache=2.0MiB(16673txo) warning='Miner violated version bit protocol'
 22024-05-18T04:48:22Z UpdateTip: new best=00000000000000000002eff303c89bd3e5b47a9c7272f9ee900526e0b228b8c0 height=843743 version=0x34ecc000 log2_work=94.927444 tx=1005754059 date='2024-05-16T18:58:02Z' progress=0.999247 cache=2.7MiB(21142txo) warning='Miner violated version bit protocol'
 32024-05-18T04:48:23Z UpdateTip: new best=0000000000000000000353a62ed7e20746378acc3bc6dc0b6a4a7a45e1e17bc8 height=843744 version=0x201a6000 log2_work=94.927457 tx=1005756478 date='2024-05-16T19:10:14Z' progress=0.999251 cache=4.0MiB(33339txo) warning='Miner violated version bit protocol'
 42024-05-18T04:48:23Z UpdateTip: new best=000000000000000000033dfde7f3bb2f2685697bd90e19b32a1a77646ffd52e8 height=843745 version=0x20600000 log2_work=94.927471 tx=1005764229 date='2024-05-16T19:10:49Z' progress=0.999251 cache=4.0MiB(34655txo) warning='Miner violated version bit protocol'
 52024-05-18T04:48:23Z UpdateTip: new best=00000000000000000001b12bfcf09ce2b914ef27f3c006356964f28ace662f1f height=843746 version=0x215ce000 log2_work=94.927485 tx=1005771799 date='2024-05-16T19:11:19Z' progress=0.999251 cache=4.3MiB(36062txo) warning='Miner violated version bit protocol'
 62024-05-18T04:48:24Z UpdateTip: new best=00000000000000000002c363be6f60c920b06716cee9e59ce4e2d92422401d80 height=843747 version=0x25dd4000 log2_work=94.927499 tx=1005774696 date='2024-05-16T19:28:35Z' progress=0.999258 cache=5.9MiB(50090txo) warning='Miner violated version bit protocol'
 72024-05-18T04:48:25Z UpdateTip: new best=000000000000000000016e0c9f15c0cbc29d08d27e65a89e44ebf8fb474792d6 height=843748 version=0x20560000 log2_work=94.927512 tx=1005777130 date='2024-05-16T19:35:55Z' progress=0.999261 cache=7.0MiB(60180txo) warning='Miner violated version bit protocol'
 82024-05-18T04:48:26Z UpdateTip: new best=000000000000000000011274fd7cb748fa1fc056f29ec7e9b9e2d6f6e3166681 height=843749 version=0x2f71c000 log2_work=94.927526 tx=1005781435 date='2024-05-16T19:43:35Z' progress=0.999263 cache=8.1MiB(69522txo) warning='Miner violated version bit protocol'
 92024-05-18T04:48:26Z UpdateTip: new best=0000000000000000000230bf5ecccfd4a279fd960dd73215916adb6b58d606d7 height=843750 version=0x2000c000 log2_work=94.927540 tx=1005786143 date='2024-05-16T19:45:25Z' progress=0.999264 cache=8.6MiB(73485txo) warning='Miner violated version bit protocol'
102024-05-18T04:48:26Z UpdateTip: new best=0000000000000000000157f74546dba8178edf0183204270f780c8a881a2b171 height=843751 version=0x228cc004 log2_work=94.927553 tx=1005790255 date='2024-05-16T19:53:07Z' progress=0.999267 cache=9.4MiB(81218txo) warning='Miner violated version bit protocol'
112024-05-18T04:48:27Z UpdateTip: new best=0000000000000000000267aeea2f1f1c584a4cd7f4d8236f964ef38bf01f6326 height=843752 version=0x2fa6e000 log2_work=94.927567 tx=1005793244 date='2024-05-16T20:16:20Z' progress=0.999276 cache=11.3MiB(95581txo) warning='Miner violated version bit protocol'
122024-05-18T04:48:28Z UpdateTip: new best=000000000000000000028daedf7e7c2b711c606dd939f056cdc2d4e65c4e2c3f height=843753 version=0x3e9a8000 log2_work=94.927581 tx=1005795518 date='2024-05-16T20:29:29Z' progress=0.999280 cache=12.9MiB(108600txo) warning='Miner violated version bit protocol'
132024-05-18T04:48:28Z UpdateTip: new best=00000000000000000001955fef1cbb90e9be2f3032ffc0771b048687fbed50e7 height=843754 version=0x22834000 log2_work=94.927594 tx=1005797687 date='2024-05-16T20:44:31Z' progress=0.999286 cache=13.7MiB(117722txo) warning='Miner violated version bit protocol'
142024-05-18T04:48:29Z UpdateTip: new best=00000000000000000000201c6fe53184f0317751669f14f6358ab3987a91ecdf height=843755 version=0x2000e000 log2_work=94.927608 tx=1005801582 date='2024-05-16T20:45:04Z' progress=0.999286 cache=14.5MiB(125347txo) warning='Miner violated version bit protocol'
152024-05-18T04:48:29Z UpdateTip: new best=0000000000000000000049055669f7d22f90eb2961f89879f4469c0e4db1cc6d height=843756 version=0x221da000 log2_work=94.927622 tx=1005805028 date='2024-05-16T21:01:52Z' progress=0.999292 cache=15.5MiB(134238txo) warning='Miner violated version bit protocol'
162024-05-18T04:48:30Z UpdateTip: new best=000000000000000000006ac9f2ab61d6826a7c856d712774f8450e6399380dee height=843757 version=0x20090000 log2_work=94.927635 tx=1005810055 date='2024-05-16T21:08:09Z' progress=0.999295 cache=16.0MiB(139911txo) warning='Miner violated version bit protocol'
172024-05-18T04:48:30Z UpdateTip: new best=00000000000000000002af779a9d511ca85d5b332ee287a00c3c2c1aab084755 height=843758 version=0x2000c000 log2_work=94.927649 tx=1005816645 date='2024-05-16T21:08:58Z' progress=0.999295 cache=16.3MiB(142353txo) warning='Miner violated version bit protocol'
182024-05-18T04:48:31Z UpdateTip: new best=00000000000000000000421e0d2c5c2439789e0b23002207318da65851b5fb2c height=843759 version=0x24938000 log2_work=94.927663 tx=1005817052 date='2024-05-16T21:09:12Z' progress=0.999295 cache=17.1MiB(149587txo) warning='Miner violated version bit protocol'
192024-05-18T04:48:31Z UpdateTip: new best=00000000000000000001bf6d7c4c88a24d7a3f9459f00610a4fbcfc8e05af00f height=843760 version=0x27058000 log2_work=94.927676 tx=1005821260 date='2024-05-16T21:18:54Z' progress=0.999299 cache=17.8MiB(156907txo) warning='Miner violated version bit protocol'
202024-05-18T04:48:32Z UpdateTip: new best=00000000000000000001a013f69917a80afa6d8c4639ac3996db8e4052b720ca height=843762 version=0x2205e000 log2_work=94.927704 tx=1005831678 date='2024-05-16T21:36:27Z' progress=0.999305 cache=19.1MiB(169799txo) warning='Miner violated version bit protocol'
212024-05-18T04:48:32Z UpdateTip: new best=0000000000000000000221c6b3e69a3e998e0e991e86ee8bd91c0eda57cde9cf height=843763 version=0x22790000 log2_work=94.927717 tx=1005835262 date='2024-05-16T21:58:59Z' progress=0.999314 cache=21.5MiB(180764txo) warning='Miner violated version bit protocol'
222024-05-18T04:48:32Z UpdateTip: new best=000000000000000000015ff1104042ddd32519423132ec46839d44f7e6e89634 height=843764 version=0x29536000 log2_work=94.927731 tx=1005835267 date='2024-05-16T22:12:33Z' progress=0.999319 cache=21.5MiB(180770txo) warning='Miner violated version bit protocol'
232024-05-18T04:48:33Z UpdateTip: new best=000000000000000000019b717b4fdff8a0a116deef0990245de104e7563baf85 height=843765 version=0x20198000 log2_work=94.927745 tx=1005841149 date='2024-05-16T22:24:17Z' progress=0.999323 cache=22.4MiB(187212txo) warning='Miner violated version bit protocol'
242024-05-18T04:48:33Z UpdateTip: new best=000000000000000000024ac4c0678d90b5c7ca958388a485631df6e9ddc66734 height=843766 version=0x27baa000 log2_work=94.927758 tx=1005846943 date='2024-05-16T22:25:44Z' progress=0.999324 cache=22.9MiB(191721txo) warning='Miner violated version bit protocol'
252024-05-18T04:48:34Z UpdateTip: new best=00000000000000000000d9e6e86f1ba6a0d17087eca395f706cb4159a67db084 height=843767 version=0x27148000 log2_work=94.927772 tx=1005850674 date='2024-05-16T22:29:33Z' progress=0.999325 cache=23.7MiB(201280txo) warning='Miner violated version bit protocol'
262024-05-18T04:48:34Z UpdateTip: new best=00000000000000000000579be079f1c9b9a46f240c35f01d316e9fc157cfdc5e height=843768 version=0x2a000000 log2_work=94.927786 tx=1005856488 date='2024-05-16T22:34:40Z' progress=0.999327 cache=24.3MiB(206192txo) warning='Miner violated version bit protocol'
272024-05-18T04:48:34Z UpdateTip: new best=00000000000000000003282d343fcaa8352f7bcf69db83e34202bac40e2b0348 height=843769 version=0x3fff0000 log2_work=94.927799 tx=1005862416 date='2024-05-16T22:46:33Z' progress=0.999331 cache=24.9MiB(211489txo) warning='Miner violated version bit protocol'
282024-05-18T04:48:34Z UpdateTip: new best=000000000000000000010d70b30e8fd47a4c52f02e2d88a383512841c1d18d24 height=843770 version=0x2c000000 log2_work=94.927813 tx=1005864462 date='2024-05-16T22:51:59Z' progress=0.999333 cache=25.2MiB(213302txo) warning='Miner violated version bit protocol'
292024-05-18T04:48:35Z UpdateTip: new best=00000000000000000000b02efe5ada29a5aa034fe48423fb621f89a2f9fc335a height=843771 version=0x2d3f8004 log2_work=94.927827 tx=1005865111 date='2024-05-16T22:52:35Z' progress=0.999334 cache=26.2MiB(223592txo) warning='Miner violated version bit protocol'

Geremia commented at 7:37 pm on May 21, 2024: none

If I delete the blocks, it triggers a reindex when I restart the app. Perhaps some of the (pruned) blocks were corrupted?

Also, I have a backup of the chainstate from May 11. I wish I could re-download only the blocks since then, not have to re-download the entire chain again.

luke-jr commented at 7:58 pm on May 21, 2024: member

I guess we should save a “reindex needed” flag somewhere when LevelDB corruption causes the node to shutdown?

I wish I could re-download only the blocks since then, not have to re-download the entire chain again.

LevelDB corruption means you have a bad chainstate. It’s not really possible to rebuild it without a full reindex from genesis, short of us doing something like utreexo commitments (even without a softfork, we could set a flag in the block index db to indicate when the commitment is valid)

maflcko removed the label Questions and Help on Jun 27, 2024

jstefanop commented at 0:04 am on December 20, 2024: none

I guess we should save a “reindex needed” flag somewhere when LevelDB corruption causes the node to shutdown?

I wish I could re-download only the blocks since then, not have to re-download the entire chain again.

LevelDB corruption means you have a bad chainstate. It’s not really possible to rebuild it without a full reindex from genesis, short of us doing something like utreexo commitments (even without a softfork, we could set a flag in the block index db to indicate when the commitment is valid)

This should really be looked into, seeing this a lot more with our hardware now that blocks are full and re-indexing entire blockchain for a corrupt chainstate a few hundred blocks from tip is horrible UX. Should only require reindex from last good block db.

This should also be handled internally than require user intervention to recover.

LevelDB error: Corruption: block checksum mismatch didn’t trigger reindex. #30138