Fees: add Fee rate Forecaster Manager #31664

1. 
   ismaelsadeeq commented at 9:42 pm on January 15, 2025: member

   • This PR implements the core component of #30392, introducing a new fee rate forecasting module. The primary addition is a ForecasterManager that coordinates multiple forecasting strategies to be able to provide better transaction fee rate predictions.

   Key Changes

   1. Addition of Fee Rate Forecasting Utility Structures

      • Forecaster abstract class, serving as the base class for all fee rate forecasters.
      • ForecastResult (the response from a Forecaster) with all metadata.
      • ConfirmationTarget, which represents the input fee rate forecasters receive. Currently, forecasters provide block-based fee rate forecasts, but this structure allows for potential time-based inputs, offering more flexibility.
      • ForecastType enum, identify and distinguish forecasters.

   2. Refactoring of CBlockPolicyEstimator

      • Renames fees to block_policy_estimator for clarity.
      • Renamesfees_args to block_policy_estimator_args for clarity
      • Renames policy_fee_tests to feerounder_tests also for clarity.
      • Modifies CBlockPolicyEstimator to inherit from the Forecaster class and implement the EstimateFee method.

   3. Introduce FeeRateForecasterManager class

      • Adds the FeeRateForecasterManager class, responsible for managing all fee rate forecasters, including CBlockPolicyEstimator, which is now a forecaster.
      • Updates the node context to hold a unique pointer to FeeRateForecasterManager instead of CBlockPolicyEstimator.
      • Changes CBlockPolicyEstimator instance from unique_ptr to a shared_pointer, allowing it to be registered in the validation interface without requiring explicit unregistration during shutdown (current master behaviour). Registering for CBlockPolicyEstimator flushes also get a shared_pointer.
      • Exposes a raw pointer of CBlockPolicyEstimator through FeeRateForecasterManager for compatibility with estimateSmartFee, friends, and node interfaces.

   4. Introduce MempoolForecaster class

      • Adds the MempoolForecaster, which forecast the fee rate required for a transaction to confirm as soon as possible. It generates a block template and returns the 75th and 50th percentile fee rates as low-priority and high-priority fee rate forecasts.
      • Implements caching for the last fee rate forecast, using a 30-second rate limit to avoid frequent block generation via the block assembler.

   The FeeRateForecasterManager now includes a method GetFeeRateForecast. This method polls forecasts from all registered forecasters and returns the lowest value. This behavior aligns with discussions and research outlined in this post. But the primary aim of the FeeRateForecasterManager is to be able to perform sanity checks and potentially apply heuristics to determine the most appropriate fee rate forecast from polled forecasters, for now it just return the lowest.

   This PR also add unit tests for the classes.

2. 
   DrahtBot commented at 9:42 pm on January 15, 2025: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Code Coverage & Benchmarks

   For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31664.

   Reviews

   See the guideline for information on the review process. A summary of reviews will appear here.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #31382 (kernel: Flush in ChainstateManager destructor by TheCharlatan)
   • #31282 (refactor: Make node_id a const& in RemoveBlockRequest by maflcko)
   • #31260 (scripted-diff: Type-safe settings retrieval by ryanofsky)
   • #30157 (Fee Estimation via Fee rate Forecasters by ismaelsadeeq)
   • #30079 (Fee Estimation: Ignore all transactions that are CPFP’d by ismaelsadeeq)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

3. DrahtBot added the label Needs rebase on Jan 22, 2025
4. fees: add `ForecastResult` class

   - This class represents the response returned by
     a fee rate forecaster.

   1e6ce06bf3
5. fees: add `ConfirmationTarget` struct

   - Defines the types of confirmation targets for fee rate forecasters.
   
   - This enable having a generic input for fee rate forecasters.

   df7ffc9f68
6. fees: add Forecaster abstract class

   - This commit implements `Forecaster` abstract class
     as the base class of fee rate forecasters.
   
   - Derived classes must provide concrete implementation
     of the virtual methods.
   
   Co-authored-by: willcl-ark <will@256k1.dev>

   a2e33262b9
7. fees: add `ForecastType` enum

   - ForecastType will be used to identify forecasters.
   
   - Each time a new forecaster is added, a corresponding
     enum value should be added to ForecastType.
   
   - This allows users to identify which forecasting strategy
     was used to make a fee rate estimate.

   0745dd789b
8. fees: add `ForecasterMan` class

   - Its a module for managing and utilising multiple
     fee rate forecasters to provide fee rate forecast.
   
   - The ForecasterManager class allows for the registration of
     multiple fee rate forecasters.
   
   Co-authored-by: willcl-ark <will@256k1.dev>

   df16b70a71
9. fees: refactor: rename policy_fee_tests.cpp to feerounder_tests.cpp

   - Also remame the test suite name to match the new name.

   3d9a393ef8
10. fees: refactor: rename fees to block_policy_estimator

    - Also move it to policy/fees and update the includes
    - Note: the block_policy_estimator.h include in block_policy_estimator.cpp was done manually.

    85dce07e3b
11. fees: rename fees_args to block_policy_estimator_args

    - Also move them to policy/fees/ and update includes
    - Note: the block_policy_estimator_args.h include in block_policy_estimator_args.cpp was done manually.

    ec92584d55
12. fees: return current block height in `estimateSmartFee` 143a301d3a
13. fees: make block_policy_estimator a forecaster 9355da6de4
14. fees: add block policy estimator to forecaster manager

    - This changes `CBlockPolicyEstimator` to a shared pointer
      this gives us three advantages.
       - Registering to validation interface using shared pointer
       - Scheduling block policy estimator flushes using shared pointer
       - Registering block policy estimator to forecaster_man

    e8f5eb5143
15. fees: add `forecastTypeToString` method

    - This method converts a ForecastType enum to its
      string representation.

    ba44923aeb
16. fees: add `CalculatePercentiles` function

    - The CalculatePercentiles function, given
      a vector of feerates in the order they were added
      to the block, will return the 25th, 50th, 75th,
      and 95th percentile feerates.
    
    - Also add a unit test for this function.

    61e28422f3
17. fees: add `MemPoolForecaster` class

    - The mempool based fee rate forecaster generate a predicted fee rate estimate
      for a given confirmation target using the mempool unconfirmed transactions.
    
    Co-authored-by: willcl-ark <will@256k1.dev>

    c7cdeafd70
18. test: add mempool forecaster unit test eb72afc3ec
19. fees: cache `MemPoolPolicyEstimator` forecasts

    - Provide new estimates only when the time delta from previous
      forecast is older than 30 seconds.
    
    - This caching helps avoid the high cost of frequently generating block templates,
      preventing users from inadvertently calling `estimateFee` repeatedly.
    
    Co-authored-by: willcl-ark <will@256k1.dev>

    5bd2220422
20. fees: add `GetFeeRateForecast` method

    - Fallback to Block policy estimator estimates whenever mempool forecaster
      estimates are higher than block policy estimator.

    c6b94403bf
21. ismaelsadeeq force-pushed on Jan 22, 2025
22. DrahtBot removed the label Needs rebase on Jan 22, 2025
23. glozow added the label TX fees and policy on Feb 12, 2025

Contributors
ismaelsadeeq DrahtBot

Labels
TX fees and policy