Fees: add Fee rate Forecaster Manager #31664

1. 
   ismaelsadeeq commented at 9:42 pm on January 15, 2025: member

   • This PR implements the core component of #30392, introducing a new fee rate forecasting module. The primary addition is a ForecasterManager that coordinates multiple forecasting strategies to be able to provide better transaction fee rate predictions.

   Key Changes

   1. Addition of Fee Rate Forecasting Utility Structures

      • Forecaster abstract class, serving as the base class for all fee rate forecasters.
      • ForecastResult (the response from a Forecaster) with all metadata.
      • ConfirmationTarget, which represents the input fee rate forecasters receive. Currently, forecasters provide block-based fee rate forecasts, but this structure allows for potential time-based inputs, offering more flexibility.
      • ForecastType enum, identify and distinguish forecasters.

   2. Refactoring of CBlockPolicyEstimator

      • Renames fees to block_policy_estimator for clarity.
      • Renamesfees_args to block_policy_estimator_args for clarity
      • Renames policy_fee_tests to feerounder_tests also for clarity.
      • Modifies CBlockPolicyEstimator to inherit from the Forecaster class and implement the EstimateFee method.

   3. Introduce FeeRateForecasterManager class

      • Adds the FeeRateForecasterManager class, responsible for managing all fee rate forecasters, including CBlockPolicyEstimator, which is now a forecaster.
      • Updates the node context to hold a unique pointer to FeeRateForecasterManager instead of CBlockPolicyEstimator.
      • Changes CBlockPolicyEstimator instance from unique_ptr to a shared_pointer, allowing it to be registered in the validation interface without requiring explicit unregistration during shutdown (current master behaviour). Registering for CBlockPolicyEstimator flushes also get a shared_pointer.
      • Exposes a raw pointer of CBlockPolicyEstimator through FeeRateForecasterManager for compatibility with estimateSmartFee, friends, and node interfaces.

   4. Introduce MempoolForecaster class

      • Adds the MempoolForecaster, which forecast the fee rate required for a transaction to confirm as soon as possible. It generates a block template and returns the 75th and 50th percentile fee rates as low-priority and high-priority fee rate forecasts.
      • Implements caching for the last fee rate forecast, using a 30-second rate limit to avoid frequent block generation via the block assembler.

   The FeeRateForecasterManager now includes a method GetFeeRateForecast. This method polls forecasts from all registered forecasters and returns the lowest value. This behavior aligns with discussions and research outlined in this post. But the primary aim of the FeeRateForecasterManager is to be able to perform sanity checks and potentially apply heuristics to determine the most appropriate fee rate forecast from polled forecasters, for now it just return the lowest.

   This PR also add unit tests for the classes.

2. fees: add `ForecastResult` class

   - This class represents the response returned by
     a fee rate forecaster.

   9556ec714b
3. fees: add `ConfirmationTarget` struct

   - Defines the types of confirmation targets for fee rate forecasters.
   
   - This enable having a generic input for fee rate forecasters.

   9401d3a3ad
4. fees: add Forecaster abstract class

   - This commit implements `Forecaster` abstract class
     as the base class of fee rate forecasters.
   
   - Derived classes must provide concrete implementation
     of the virtual methods.
   
   Co-authored-by: willcl-ark <will@256k1.dev>

   36f0c17f7f
5. fees: add `ForecastType` enum

   - ForecastType will be used to identify forecasters.
   
   - Each time a new forecaster is added, a corresponding
     enum value should be added to ForecastType.
   
   - This allows users to identify which forecasting strategy
     was used to make a fee rate estimate.

   9ebf70bb79
6. fees: add `ForecasterMan` class

   - Its a module for managing and utilising multiple
     fee rate forecasters to provide fee rate forecast.
   
   - The ForecasterManager class allows for the registration of
     multiple fee rate forecasters.
   
   Co-authored-by: willcl-ark <will@256k1.dev>

   377d2165a9
7. fees: refactor: rename policy_fee_tests.cpp to feerounder_tests.cpp

   - Also remame the test suite name to match the new name.

   4ddcc08c9b
8. fees: refactor: rename fees to block_policy_estimator

   - Also move it to policy/fees and update the includes
   - Note: the block_policy_estimator.h include in block_policy_estimator.cpp was done manually.

   c31edbbc79
9. fees: rename fees_args to block_policy_estimator_args

   - Also move them to policy/fees/ and update includes
   - Note: the block_policy_estimator_args.h include in block_policy_estimator_args.cpp was done manually.

   5f455b6f8e
10. fees: return current block height in `estimateSmartFee` 50bdad93c8
11. fees: make block_policy_estimator a forecaster 50f5c04884
12. fees: add block policy estimator to forecaster manager

    - This changes `CBlockPolicyEstimator` to a shared pointer
      this gives us three advantages.
       - Registering to validation interface using shared pointer
       - Scheduling block policy estimator flushes using shared pointer
       - Registering block policy estimator to forecaster_man

    f994d355e6
13. fees: add `forecastTypeToString` method

    - This method converts a ForecastType enum to its
      string representation.

    ca2894a7bb
14. fees: add `CalculatePercentiles` function

    - The CalculatePercentiles function, given
      a vector of feerates in the order they were added
      to the block, will return the 25th, 50th, 75th,
      and 95th percentile feerates.
    
    - Also add a unit test for this function.

    2e9096e24b
15. fees: add `MemPoolForecaster` class

    - The mempool based fee rate forecaster generate a predicted fee rate estimate
      for a given confirmation target using the mempool unconfirmed transactions.
    
    Co-authored-by: willcl-ark <will@256k1.dev>

    f9b339bed3
16. test: add mempool forecaster unit test b75ebd30a8
17. fees: cache `MemPoolPolicyEstimator` forecasts

    - Provide new estimates only when the time delta from previous
      forecast is older than 30 seconds.
    
    - This caching helps avoid the high cost of frequently generating block templates,
      preventing users from inadvertently calling `estimateFee` repeatedly.
    
    Co-authored-by: willcl-ark <will@256k1.dev>

    02bfda8325
18. fees: add `GetFeeRateForecast` method

    - Fallback to Block policy estimator estimates whenever mempool forecaster
      estimates are higher than block policy estimator.

    ba0bede407
19. 
    DrahtBot commented at 9:42 pm on January 15, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31664.

    Reviews

    See the guideline for information on the review process. A summary of reviews will appear here.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31583 (rpc: add gettarget , target getmininginfo field and show next block info by Sjors)
    • #31382 (kernel: Flush in ChainstateManager destructor by TheCharlatan)
    • #31282 (refactor: Make node_id a const& in RemoveBlockRequest by maflcko)
    • #30157 (Fee Estimation via Fee rate Forecasters by ismaelsadeeq)
    • #30079 (Fee Estimation: Ignore all transactions that are CPFP’d by ismaelsadeeq)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Contributors
ismaelsadeeq DrahtBot