Factoring out a stateless, side-effect free validation library #35904

Problem

Today, consensus validation is tightly coupled to the node implementation. As a result, unit tests require substantial test setup, and validation cannot be reused independently of the node's state-management implementation.

This issue proposes refactoring the validation code into a stateless layer with explicit inputs while preserving most of the existing code, behavior, and architecture.

Testing

The resulting validation interface will provide a dedicated unit-testing surface. Consensus validation can be tested by supplying only the required consensus inputs through lightweight implementations of ChainView and CoinIndex, without constructing ChainstateManager, CBlockIndex, or CCoinsViewCache.

Refactoring plan

The following actions have been identified to be necessary towards the goal of providing a standalone validation library. For a large part, those actions are independent of each other and can be done independently, in any order.

1. Expose the validation interface

   Make the existing CheckBlockHeader, ContextualCheckBlockHeader, and ContextualCheckBlock functions part of the public validation interface by declaring them in validation.h.

2. Interface Segregation: Remove dependency on ChainstateManager

   ContextualCheck* functions need a ChainstateManager only to access the consensus parameters. Pass Consensus::Params instead and access them at the call site.

3. Make environmental dependencies explicit

   Pass the validation time explicitly to ContextualCheckBlockHeader instead of reading the system clock internally.

4. Replace CBlockIndex with ChainView

   Introduce a ChainView abstraction representing the chain history required by contextual validation. Wrap the existing CBlockIndex-based implementation to provide this interface.

5. Introduce CoinIndex

   Introduce a read-only CoinIndex abstraction for the UTXO lookups performed during validation. Wrap CCoinsViewCache to provide this interface.

6. Extract validation from ConnectBlock

   Extract the consensus-validation logic from ConnectBlock into a stateless operation depending only on the block and its explicit consensus inputs (Consensus::Params, ChainView, CoinIndex). Further abstractions may become necessary (async execution, caching).

   ConnectBlock remains responsible for orchestrating validation and applying the resulting state transition.

More Interface Segregation

Many consensus relevant types, like CBlock, CTransaction, or uint256 provide functionality that is not required for validation, like mutability, stringification, or construction from a string representation. Decoupling such functionality from the type and instead providing it as utilities for testing or the wallet will further narrow the surface of a validation API.

1. Rewrite deserialization code in a way that it does not mutate individual data members

   Current serdes code is hard to reason about because it heavily relies on overload resolution. It is also one of the very few places that impose a mutability requirement on many types. Here is a proof of concept that an alternative is possible: https://github.com/purpleKarrot/std-bitcoin/blob/master/src/serdes_decode.cpp

2. Rewrite test code that mutates individual data members

   Many tests span multiple test cases sharing the same testing data, where each test case performs an assertion after applying a modification to the shared test data. This is problematic because it is hard to diagnose test failures. This is probably the second place out of two where mutability is required. Rewriting those test cases to be independent also removes the mutability requirement.

3. Make primitive types like CBlock, CTransaction, COutPoint, etc immutable

   Types that no longer have the requirement that their data members need to be individually mutable, should no longer provide ways to modify them. Their data members should be made private, with public observers. CMutableTransaction should no longer be necessary by now.

4. A block is not a block header, and its transactions are not optional

   CBlock should have rather than be a CBlockHeader (inheritance establishes an is-a relationship). CBlock should further own CTransaction objects, rather than pointers which could be nullptr. CBlock and CTransaction may be implemented in terms of std::shared_ptr<const impl> internally and provide a never-empty guarantee as part of their invariant.

5. Replace ToString functions with std::formatter specializations

6. Factor out string parsing helpers

   The use case for constructing a type from a string representation is very narrow. Performing the latter in a constant expression is exclusively required for testing. Such functionality should live in utilities rather than the constructor. User-defined literals are also good candidates for such utilities.

Relation to the Bitcoin Kernel library

The validation library provides stateless consensus validation. The Bitcoin Kernel library can build on it by providing concrete implementations of ChainView and CoinIndex while retaining responsibility for state management.

The problem you seem to be identifying here is, to my understanding:

• validating unit tests require substantial setup
• validation code cannot be used independently of chainstate

Those seem to be somewhat plausible but fairly minor advantages to me: the impact is that tests are slower to run (and that we therefore might have fewer tests run less often), and that third-party code is more complicated.

To solve that problem, you then spend far a lot of time describing a large refactoring of consensus code to solve that problem. That is a very high risk approach, and when it gains only fairly small advantages, I think very much the wrong approach.

I also don't think it's helpful to file an issue as "look, a nail: here's why you should buy my hammer". It's possible to agree on the problem, without agreeing on the solution; in this case, eg, I think the identified problems can be effectively solved without all the refactoring, in particular via the approach in #35187.