Stuck chainstate when utxo_snapshot.sh fails

hazeycode commented at 0:04 am on June 9, 2023: contributor

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

IBD/chainstate gets stuck if utxo_snapshot.sh fails. No new blocks are downloaded. Restarting bitcoind does not help. Taking another snapshot at an earlier height or reindex is required to repair.

utxo_snapshot.sh output:

0Rewinding chain back to height 788440 (by invalidating 000000000000000000052ac0ce9b8f1a2e7691771e6386f770432c9443dc2af7); this may take a while
1Generating UTXO snapshot...
2error code: -8
3error message:
4/home/chris/Desktop/assumeutxo/test/utxo-788440.dat already exists. If you are sure this is what you want, move it out of the way first

Note that moving the file or choosing another filename and rerunning does not fix. But taking another snapshot at an earlier height will.

Expected behaviour

If utxo_snapshot.sh fails, bitcoind should cancel rewind and recover.

Also, if a filepath that already exists is passed to utxosnapshot.sh, it should early exit before rewinding. This could perhaps be considered a separate issue.

Steps to reproduce

Create a dummy file for the purpose of the test e.g. echo "some bytes so file is not empty" > ./utxo-788440.dat
Run bitcoind and wait for it to sync
Run utxosnapshot.sh with some blockheight and a filepath that already exists e.g. ./contrib/devtools/utxo_snapshot.sh 788440 ./utxo-788440.dat ./src/bitcoin-cli
Wait for chain rewind to happen, snapshot to be generated and write to disk fail
Observe bitcoind is now stuck with an invalid block at the rewind height

Relevant log output

02023-06-08T23:51:50Z [httpworker.3] UpdateTip: new best=000000000000000000014a9d36f6a19aca593f0c228ae1b1b6fe33361fa67be1 height=788440 version=0x2001e000 log2_work=94.162204 tx=832727241 date='2023-05-06T02:54:13Z' progress=0.987258 cache=790.3MiB(1245160txo)
12023-06-08T23:51:50Z [httpworker.3] InvalidChainFound: invalid block=000000000000000000052ac0ce9b8f1a2e7691771e6386f770432c9443dc2af7  height=788441  log2_work=94.162217  date=2023-05-06T02:55:43Z
22023-06-08T23:51:50Z [httpworker.3] InvalidChainFound:  current best=000000000000000000014a9d36f6a19aca593f0c228ae1b1b6fe33361fa67be1  height=788440  log2_work=94.162204  date=2023-05-06T02:54:13Z

0Rewinding chain back to height 788440 (by invalidating 000000000000000000052ac0ce9b8f1a2e7691771e6386f770432c9443dc2af7); this may take a while
1Generating UTXO snapshot...
2error code: -8
3error message:
4/home/chris/Desktop/assumeutxo/test/utxo-788440.dat already exists. If you are sure this is what you want, move it out of the way first

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v25.99.0-3dc6fd620e28

Operating system and version

Linux 6.3.1-arch2-1

Machine specifications

No response

jamesob commented at 2:16 pm on June 9, 2023: member

Thanks for the report. I think to solve this without a reindex, you just need to run reconsiderblock 000000000000000000052ac0ce9b8f1a2e7691771e6386f770432c9443dc2af7. We should probably amend the script to catch failures or interrupts and do this automatically.

hazeycode commented at 3:54 pm on June 9, 2023: contributor

Thanks @jamesob. I can confirm that reconsiderblock works as you expected. I’ll have a go at updating the script with error handing.

hazeycode referenced this in commit 63a510f40f on Jun 9, 2023

hazeycode referenced this in commit 3a1e3d6093 on Jun 9, 2023

hazeycode referenced this in commit 1026068b26 on Jun 9, 2023

hazeycode referenced this in commit ce10d50583 on Jun 9, 2023

hazeycode referenced this in commit 5db4c8affa on Jun 9, 2023

hazeycode referenced this in commit d7969fd81b on Jun 9, 2023

hazeycode referenced this in commit 10d9bef004 on Jul 3, 2023

ryanofsky closed this on Dec 4, 2023

Stuck chainstate when utxo_snapshot.sh fails #27841