Crash-safe pruning after chainstate persistence
This change makes prune selection non-mutating, then delays block-index pruning and file deletion until after the UTXO cache write. That closes the demonstrated process-crash window where ReplayBlocks() needs files that were already deleted. The new order still allows durable prune metadata and deletion to outrun a non-synchronous coins write on machine failure, and it no longer reclaims selected files before the chainstate disk-space preflight. Watch for: confirmed machine-crash recovery failure and confirmed low-disk shutdown regression on the default shared filesystem.
What this branch solves
This branch addresses one cause of #29422. The issue describes a pruned node restarting with an unusable chainstate after the process was killed during IBD. The reproduced failure happens when pruning deletes block files before the coins database finishes writing. On restart, the older coins database needs those deleted blocks and cannot recover without a full reindex.
The branch changes that order so the block files remain available until the coins write returns. The new test proves that this fixes the process-crash/OOM-kill path: unpatched master deletes blk00000.dat and fails during restart, while this branch keeps the file and successfully catches up from disk.
It does not repair a node that is already broken; #36680 adds that startup detection and recovery message. It also does not yet guarantee safety after a whole-machine crash or power loss, because returning from the coins write does not guarantee that every write is durable on disk.
High-level view
For process crashes, selected files remain available through the coins transition. The first block-index write protects newly indexed blocks referenced by the coins tip; the second atomically records pruned metadata before unlink, and startup removes files left after that commit.
Confirmed: the cross-database durability order is incomplete. Coins batches use LevelDB's non-synchronous mode, while the subsequent block-index prune batch is synchronous, so a machine crash can preserve pruning while losing enough of the coins transition to require the deleted files.
Confirmed: the delayed unlink changes minimum free-space behavior. CheckDiskSpace(datadir, 192 * dirty_count) now runs while selected block files still occupy the default shared filesystem, so pruning can issue a fatal low-disk error where the previous order first created enough room for the forced coins write.
Manual and automatic pruning use the same new order. Startup retries automatic pruning, while interrupted manual pruning requires another RPC. During assumeutxo background validation, snapshot and historical pruning operate on disjoint height regions backed by separate blockfile cursors.
Confirmed coverage gap: the added test reproduces one manual-prune process crash during a partial coins batch. It does not exercise automatic pruning, low disk, machine-level durability, the second block-index commit window, or two live chainstates.
<details> <summary>Issues (3)</summary>
- Unsynced coins transition (confirmed) — The synchronous prune metadata commit and unlink can survive a machine crash while the preceding non-synchronous coins batches are lost. Pass a prune-specific durable mode through every coins batch, then synchronously commit prune metadata and unlink.
- Space reclaimed too late (confirmed) — On a shared filesystem, the UTXO disk-space check now runs before selected files are removed and can fatally stop a prune that previously made enough room. Pre-prune only files below every live chainstate's replay floor, or reserve the forced coins-write space before block allocation.
- Crash-test scope (confirmed) — The test covers one manual partial-batch process exit and checks only height. Add focused low-disk and automatic-retry cases, compare the recovered UTXO hash, and do not treat this hook as machine-crash durability coverage.
</details>
<details> <summary>Details</summary>
Durable prune metadata can outrun the coins database
Priority 1, confirmed. The new order treats return from CoinsTip().Flush() or CoinsTip().Sync() as the durability boundary (src/validation.cpp:2798-2803), then synchronously commits the pruned block-index state and deletes the files (src/validation.cpp:2813-2829). Both partial and final UTXO batches call m_db->WriteBatch(batch) without the sync argument (src/txdb.cpp:178-180, src/txdb.cpp:192-197), whose default is fSync = false (src/dbwrapper.h:324). LevelDB states that these writes may be lost when the machine crashes, although they survive a process-only crash (src/leveldb/include/leveldb/options.h:173-182).
BlockTreeDB::WriteBatchSync() gives the later block-index write a stronger guarantee by calling WriteBatch(batch, true) (src/node/blockstorage.cpp:92-102). If power is lost after that batch and unlink, the block database can durably record the replay blocks as pruned while the coins database returns at an old tip or with DB_HEAD_BLOCKS requiring ReplayBlocks(). The replay positions and files are unavailable, recreating the reindex-required failure this change targets.
The smallest viable correction is to pass a durable-prune mode through the coins batch write and use synchronous writes for every partial and final batch when fFlushForPrune is true. Commit the prune flag and pruned file/index records in the following synchronous block-index transition, then unlink. The tradeoff is fsync latency during prune-forced flushes rather than every periodic UTXO flush.
Delayed unlink can prevent pruning on a nearly full shared filesystem
Priority 2, confirmed. The added block unlinks at src/validation.cpp:2829, after the UTXO preflight at src/validation.cpp:2798. That preflight requires 50 MiB + 192 * CoinsTip().GetDirtyCount() of available space (src/util/fs_helpers.cpp:94-98). Block-file allocation only checks that its own allocation leaves the generic 50 MiB reserve (src/flatfile.cpp:59-70), which does not reserve the larger forced UTXO write.
On the default shared filesystem, choose free space F such that 50 MiB <= F < 50 MiB + 192 * dirty_count, with selected files large enough to raise F above the upper threshold. The old order passed after unlinking; the new order calls FatalError before deleting a file. Automatic pruning reaches this order through FindFilesToPrune() (src/validation.cpp:2747-2749), and manual pruning reaches it through FindFilesToPruneManual() (src/validation.cpp:2740-2744). Separate block and chainstate filesystems do not benefit from either order.
The smallest crash-safe correction is to split selected files into early-safe and replay-needed sets. Mark files below the replay floor of every live durable chainstate in the first synchronous block-index batch and unlink them before the UTXO space check. Keep files intersecting any old-to-new replay path until the corresponding durable coins flush, then use the second batch and unlink. If no selected file lies outside every replay path, the node must reserve chainstate write space earlier or fail without pruning.
The second block-index write closes process-crash windows
select files
-> flush block data
-> sync current block index
-> write coins transition
-> mark selected files pruned in memory
-> sync pruned file/index records
-> unlink files
A process exit before or during the coins transition leaves block-index positions and files available to ReplayBlocks(). An exit after the coins transition but before the second block-index batch leaves an advanced coins database plus a conservative block index and intact files. The second batch atomically writes zeroed CBlockFileInfo records and cleared BLOCK_HAVE_DATA/BLOCK_HAVE_UNDO entries (src/node/blockstorage.cpp:270-297, src/node/blockstorage.cpp:527-544). If exit occurs after that batch but before or during unlink, startup retries deletion from zero-sized file records (src/node/blockstorage.cpp:608-623). A thrown second database write skips unlink, leaving the previous durable index matched with intact files.
The second write is necessary and sufficient for process-crash ordering. It cannot act as a durability barrier for the separate coins database addressed in Priority 1.
Manual, automatic, and assumeutxo pruning remain separated
Both helpers now only select file numbers (src/node/blockstorage.cpp:315-325, src/node/blockstorage.cpp:357-401) before converging on the post-coins pruning block (src/validation.cpp:2740-2753, src/validation.cpp:2813-2829). Automatic selection clears the request flag, but startup calls PruneAndFlush() for every chainstate (src/init.cpp:2043-2051), which retries target-based pruning after a process crash. The manual target is a sentinel, so an interrupted manual prune waits for another pruneblockchain call.
For an unvalidated snapshot, GetPruneRange() starts snapshot-chain pruning above the snapshot base (src/validation.cpp:6381-6386); the historical chain prunes below its own lagging tip. BlockfileTypeForHeight() assigns those regions to separate normal and assumed cursors (src/node/blockstorage.cpp:787-793). Each selected range is therefore flushed by the chainstate whose replay can depend on it.
The crash test covers only the first interrupted coins batch
Confirmed coverage gap. The test fixes the mode to manual pruning (test/functional/feature_prune_crash.py:22-23) and uses -dbcrashratio=1 to exit after the first partial coins batch (test/functional/feature_prune_crash.py:31-39; hook at src/txdb.cpp:178-187). It checks that blk00000.dat remains, restart reaches the prior in-memory tip, and a repeated manual prune deletes the file (test/functional/feature_prune_crash.py:43-54).
Because LevelDB's non-synchronous writes survive process exit, this hook cannot test machine-crash durability. There is also no hook after the final coins batch or around the second block-index write, no automatic-prune crash case, and no two-chainstate crash case. Add focused coverage for the low-disk preflight and automatic-prune restart path, and compare recovered UTXO state with a reference hash rather than checking only height. Enforce durable ordering in production code because the functional harness cannot reliably emulate lost kernel writeback state.
</details>
<details> <summary>File map</summary>
src/node/blockstorage.cpp: makes manual and automatic prune discovery selection-only.src/node/blockstorage.h: documents selection, marking, and unlink responsibilities.src/validation.cpp: moves prune marking, persistence, and unlink after the coins write and adds the second block-index batch.test/functional/feature_prune_crash.py: reproduces a manual-prune process crash during a partial coins batch.test/functional/test_runner.py: registers the new functional test.
Full diff: git diff upstream/master...HEAD
</details>
