Running out of disk space can leave bitcoin in a desynced state #26112

I noticed my node was not syncing, and through some debugging on #bitcoin-core-dev it seems like it was caused by running out of disk space which left the node in a desynced state, with the valid chain marked as invalid.

After freeing up some space and doing reconsiderblock it fixed it.

Comments from @sipa:

Ugh. That is bad. Out of disk space should not result in database corruption.

Database errors propagating up and being interpreted as (permanent) block invalidity was one of the contributing factors to the BDB/LevelDB fork in the 0.7/0.8 transition.

logs:

2022-09-05T00:28:55Z UpdateTip: new best=000000000000000000087e337539b369b19570a88c493725020266387d22dac0 height=752167 version=0x2ce8e004 log2_work=93.70770
7 tx=761139817 date='2022-09-01T15:00:06Z' progress=0.998887 cache=146.8MiB(1101860txo)
2022-09-05T00:28:55Z Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:55Z You can use -debug=leveldb to get more complete diagnostic messages
2022-09-05T00:28:55Z *** System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:55Z Error: A fatal internal error occurred, see debug.log for details
2022-09-05T00:28:56Z ERROR: ProcessNewBlock: ActivateBestChain failed (System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chains
tate/2963623.ldb: No space left on device)
2022-09-05T00:28:56Z ERROR: ConnectBlock: Consensus::CheckTxInputs: 221fa678c5c9953d6cd17e584f05c12ab10ba0f2fc8e8131e266f3f0e9819848, bad-txns-inputs-missing
orspent, CheckTxInputs: inputs missing/spent
2022-09-05T00:28:56Z InvalidChainFound: invalid block=000000000000000000079ba062298aa7cef1888f870c23e303a361c19b097ff8  height=752168  log2_work=93.707719  date=2022-09-01T15:13:42Z
2022-09-05T00:28:56Z InvalidChainFound:  current best=000000000000000000087e337539b369b19570a88c493725020266387d22dac0  height=752167  log2_work=93.707707  date=2022-09-01T15:00:06Z
2022-09-05T00:28:56Z ERROR: ConnectTip: ConnectBlock 000000000000000000079ba062298aa7cef1888f870c23e303a361c19b097ff8 failed, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
2022-09-05T00:28:56Z InvalidChainFound: invalid block=000000000000000000079ba062298aa7cef1888f870c23e303a361c19b097ff8  height=752168  log2_work=93.707719  date=2022-09-01T15:13:42Z
2022-09-05T00:28:56Z InvalidChainFound:  current best=000000000000000000087e337539b369b19570a88c493725020266387d22dac0  height=752167  log2_work=93.707707  date=2022-09-01T15:00:06Z

So it appears that this was triggered by LevelDB failing to write to disk.

Some questions:

• Why didn't our own disk space check detect this long before it happened? @jb55 was anything else quickly filling your disk at the same time, which could cause our own check to not being frequent enough?
• Did Bitcoin Core shut down after this happened?

Based on your comments on IRC, it seems that normal restarting didn't fix the problem. So that suggests that while there was some LevelDB error... Bitcoin Core still managed to (incorrectly) write to disk that the block was invalid. It shouldn't conclude that in the first place, but it's somewhat strange that even after a system error it still managed to actually commit that to disk.

Why didn't our own disk space check detect this long before it happened? @jb55 was anything else quickly filling your disk at the same time, which could cause our own check to not being frequent enough?

yes this is very possible, I run nixos and frequently use nix-shell, etc which downloads things and fills up my disk pretty quickly.

Did Bitcoin Core shut down after this happened?

yes, here's the full log:

2022-09-05T00:28:55Z UpdateTip: new best=000000000000000000087e337539b369b19570a88c493725020266387d22dac0 height=752167 version=0x2ce8e004 log2_work=93.70770
7 tx=761139817 date='2022-09-01T15:00:06Z' progress=0.998887 cache=146.8MiB(1101860txo)
2022-09-05T00:28:55Z Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:55Z You can use -debug=leveldb to get more complete diagnostic messages
2022-09-05T00:28:55Z *** System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:55Z Error: A fatal internal error occurred, see debug.log for details
2022-09-05T00:28:56Z ERROR: ProcessNewBlock: ActivateBestChain failed (System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chains
tate/2963623.ldb: No space left on device)
2022-09-05T00:28:56Z ERROR: ConnectBlock: Consensus::CheckTxInputs: 221fa678c5c9953d6cd17e584f05c12ab10ba0f2fc8e8131e266f3f0e9819848, bad-txns-inputs-missing
orspent, CheckTxInputs: inputs missing/spent
2022-09-05T00:28:56Z InvalidChainFound: invalid block=000000000000000000079ba062298aa7cef1888f870c23e303a361c19b097ff8  height=752168  log2_work=93.707719  date=2022-09-01T15:13:42Z
2022-09-05T00:28:56Z InvalidChainFound:  current best=000000000000000000087e337539b369b19570a88c493725020266387d22dac0  height=752167  log2_work=93.707707  date=2022-09-01T15:00:06Z
2022-09-05T00:28:56Z ERROR: ConnectTip: ConnectBlock 000000000000000000079ba062298aa7cef1888f870c23e303a361c19b097ff8 failed, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
2022-09-05T00:28:56Z InvalidChainFound: invalid block=000000000000000000079ba062298aa7cef1888f870c23e303a361c19b097ff8  height=752168  log2_work=93.707719  date=2022-09-01T15:13:42Z
orspent, CheckTxInputs: inputs missing/spent
2022-09-05T00:28:57Z ERROR: ProcessNewBlock: ActivateBestChain failed (System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device)
2022-09-05T00:28:57Z msghand thread exit
2022-09-05T00:28:57Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat started
 CheckTxInputs: inputs missing/spent
2022-09-05T00:28:56Z InvalidChainFound: invalid block=000000000000000000079ba062298aa7cef1888f870c23e303a361c19b097ff8  height=752168  log2_work=93.707719  date=2022-09-01T15:13:42Z
2022-09-05T00:28:56Z InvalidChainFound:  current best=000000000000000000087e337539b369b19570a88c493725020266387d22dac0  height=752167  log2_work=93.707707  date=2022-09-01T15:00:06Z
2022-09-05T00:28:57Z tor: Thread interrupt
2022-09-05T00:28:57Z torcontrol thread exit
2022-09-05T00:28:57Z opencon thread exit
2022-09-05T00:28:57Z addcon thread exit
2022-09-05T00:28:57Z Shutdown: In progress...
2022-09-05T00:28:57Z net thread exit
2022-09-05T00:28:57Z Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:57Z You can use -debug=leveldb to get more complete diagnostic messages
2022-09-05T00:28:57Z *** System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:57Z Error: A fatal internal error occurred, see debug.log for details
2022-09-05T00:28:57Z ERROR: ProcessNewBlock: ActivateBestChain failed (System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chains
tate/2963623.ldb: No space left on device)
2022-09-05T00:28:57Z msghand thread exit
2022-09-05T00:28:57Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat started
2022-09-05T00:28:57Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2022-09-05T00:28:57Z scheduler thread exit
2022-09-05T00:28:57Z Writing 0 unbroadcast transactions to disk.
2022-09-05T00:28:57Z Dumped mempool: 0.001682s to copy, 0.015307s to dump
2022-09-05T00:28:57Z Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:57Z You can use -debug=leveldb to get more complete diagnostic messages
2022-09-05T00:28:57Z *** System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:57Z Error: A fatal internal error occurred, see debug.log for details
2022-09-05T00:28:57Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device)
2022-09-05T00:28:57Z Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:57Z You can use -debug=leveldb to get more complete diagnostic messages
2022-09-05T00:28:57Z *** System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device
2022-09-05T00:28:57Z Error: A fatal internal error occurred, see debug.log for details
2022-09-05T00:28:57Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: IO error: /home/jb55/.bitcoin/chainstate/2963623.ldb: No space left on device)
2022-09-05T00:28:57Z [personal] Releasing wallet
2022-09-05T00:28:57Z [old-wallet] Releasing wallet
2022-09-05T00:28:58Z Shutdown: done

A guess about what might be happening:

CCoinsViewErrorCatcher, the wrapper class used around CCoinsViewDB that's supposed to detect these problems and forcefully exit the application, has an override for GetCoins. But in CheckTxInputs, HaveInputs is first invoked, which on its turn calls HaveCoin. HaveCoin is implemented in CCoinsViewDB, but not in CCoinsViewErrorCatcher, and thus the disk read exception escapes.

A solution may be to just add an override for HaveCoin in CCoinsViewErrorCatcher.

Removed from the milestone, as this is not a regression, nor a fix is available right now.

awesome, thanks @aureleoules !

61a6c3b0e9a8dab5c5f845af4becde817539133c