This PR aims at improving Bitcoin Core fee estimation
The objectives of this improvement are to:
- Reduce overestimation done by the current
CBlockPolicyEstimator - Make the fee estimator aware of the state of the mempool, allowing it to respond to changing conditions immediately.
- Empower node users to be self-sovereign and use their node’s estimates, as is now, the majority rely on third parties for fee estimation.
- Simplify the process of adding new fee estimation strategies in the future.
The PR creates a new FeeEstimator class which is responsible for getting estimates from one or more subscribed Forcaster’s.
A Forcaster will register with the FeeEstimator as being able to provide estimates, and when an estimate is requested, should return two fee rate estimates (high and low) and a confidence level.
The estimator will then take (currently undecided):
- the lowest estimated rate, or
- the weighted average of all the estimated rates
…and return the two ‘high’ and ’low’ estimations to the user.
The FeeEstimator also owns access to the legacy estimator, accessible via a pointer.
This PR deliberately does not try to modify existing fee estimation code, but instead uses new modules with cleaner seperation. This will make new code and interfaces much easier to introduce, and also make it easier to, if desired in the future, drop legacy estimation code.
Currently, this PR implements two forecasters:
-
Mempool-based forecaster (
MemPoolForecaster): It estimates fees simply by generating a block template from the node’s mempool and returns the 25th and 50th percentile mining scores as low-priority and high-priority fee estimates, respectively. -
Last 6 Blocks forecaster (
BlockForecaster): Uses the transactions from the previous six blocks which were also seen in the node’s mempool, to make an estimate.It listens to the validation interface notifications for removed mempool transactions, linearizes the transactions using a mini miner, and saves the 25th and 50th percentile mining scores. When estimating fees, it averages the mining scores of the percentiles from the last six mined blocks.
NB. As this excludes transactions that never entered the node’s mempool, it’s likely to undershoot slightly
These two forecasters both estimate fees for (1, 2) block confirmation targets, referred to as “as soon as possible (ASAP)” fee rate estimates.
For both forcasters use of RBF is strongly recommended in order that transactions can be fee-bumped if desired.
Next steps:
-
Give some statistics of estimates “missing the block” (~percentage of estimates below the 5th percentile mining score of the confirmed block) and “over paying” (~percentage of estimates above the 25th 50th and 75th percentiles mining score)
-
Extend the feature to the wallet; currently it’s only exposed in an RPC
estimatefee. -
Update forecasters to return confidence levels in their estimates.
Confidence levels
Confidence levels are determined by heuristics.
For the mempool-based forecaster:
- Suspected high mining score transactions should be confirmed but aren’t.
- Most transactions in previously mined blocks were in our node’s mempool.
- High mining score mempool transactions are confirmed.
For the last 6 blocks forecaster:
- Presence of empty blocks in the last 6 blocks.
- Most of the last 6 blocks’ transactions were seen in the node’s mempool or not.
The fee estimator will select the fee rate from the forecasters based on confidence level instead of just taking the lowest fee rate estimate.
How does this fit in with CBlockPolicyEstimator?
This fee estimator owns the legacy CBlockPolicyEstimator and does not change any of its interfaces. The wallet still uses estimatesmartfee, and RPCs also access it through the Fee Estimator.
However, this will change when we switch to using estimates from GetFeeEstimateFromForecasters, which works much better than the estimatesmartfee in (1, 2) confirmation target scenarios.
Moreover, when we have another forecaster that supports confirmation targets beyond (1, 2) and works better than estimatesmartfee, we will eventually phase out CBlockPolicyEstimator as the new Fee Estimator module makes it easy to do so.
This PR addresses two open issues and has been requested by users in the past.
FAQ
-
Will miners broadcast high fee rate transactions, and then evict them with a conflicting transaction in the next block, trick users of this fee estimator into making transactions with high fees? https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/6?u=ismaelsadeeq
This PR addresses this concern by ensuring that the fee estimate does not surge immediately whenever this happens the mempool based forecaster estimate will be higher than last 6 block forecaster estimate. As such the last six blocks forecaster estimate will be returned, preventing users to be victims of this attack.
-
In times of high confirmation times or high block space demand, fee estimates tend to rise. If the mempool-based forecaster estimate rises, the fee estimator will fall back to the last six blocks fee estimate, if it’s low, will that make transactions constructed with the estimate get stuck in the mempool? The ASAP fee estimate provides an estimate for transactions to confirm in the next one or two blocks. During periods of high confirmation times, fee rates rise due to congestion. Once a block is mined, the mempool clears high mining score transactions, making room for lower fee rate transactions. If congestion recurs and the transaction is time-sensitive, users can safely fee bump assuming it’s an RBF-enabled transaction. In times of immediate high block demand, if the mempool-based fee estimate surges while the last six blocks fee estimate is low, users can fee bump their transactions if they take longer to confirm. As more blocks are mined, the last six blocks forecaster will adjust and reflect the higher fee rates. When demand decreases, the mempool forecaster will immediately respond to the new conditions by providing a lower fee estimate and the fee estimator will return it instead.
-
Currently, getting a block template is expensive. How do you tackle that in the mempool-based forecaster? The forecaster cache the recently generated fee rate estimates and only generate a new block template when the time delta between the recent estimate and the current time exceeds 30 seconds.
This is a collaborative work with @willcl-ark and a result of collecting insight from other contributors.
