Fee Estimation via Fee rate Forecasters tracking issue

ismaelsadeeq commented at 11:33 am on July 4, 2024: member

This is a tracking issue for an effort to add Fee Estimation with Forecasters (Mempool-Based Fee rate Forecaster) to Bitcoin Core.

For the conceptual motivation behind this work, see issue #27995 and the previous discussion on Delving Bitcoin: Mempool-Based Fee Estimation on Bitcoin Core.

What to Review

#31664

Roadmap

#30391
#31664
Integrate the new feature into estimatesmartfee RPC and add Functional Test
Add Feature to the Wallet

Complete Feature

Design document https://gist.github.com/ismaelsadeeq/bcfa6b1216eef10076665551d9885722

Implementation #30157

Next Steps

Use this feature in the Bitcoin Core wallet
Use heuristics from the mempool to target a higher/lower percentile in the mempool forecaster

Improving the current `BlockPolicyEstimator` in the Short Term

Improving the current `BlockPolicyEstimator` after the Cluster Mempool Project

Make BlockPolicyEstimator package-aware: Delving Bitcoin Post

Previous Refactors

glozow added the label TX fees and policy on Jul 18, 2024

ismaelsadeeq renamed this:
~~Improving Fee estimation tracking issue~~
Fee Estimation via Fee rate Forecasters tracking issue
on Jul 18, 2024

ismaelsadeeq commented at 1:15 pm on October 10, 2024: member

This issue was opened to track the effort to improve the Bitcoin core fee estimator. This comment will provide information on the status of this project and some key ideas on how to achieve this.

Completed Tasks so far:

Enabled asynchronous block policy fee estimator updates: By making it a client of the validation interface.
Empirical data analysis: To determine which forecasting strategy provides the most accurate estimates while withstanding common attack vectors.
Developed a rough model of the fee estimator architecture: This model will enable adding more fee estimation strategies in the future.
Ideas on how policy estimator will improve accuracy post-cluster mempool: By tracking chunks instead of individual transactions.
Brainstorm sanity checks for the fee estimator algorithms: These checks ensure more robust estimates and help avoid common pitfalls when analyzing mempool data.

Current Status:

We have a rough implementation of the fee estimator that attempts to integrate these improvements. However, two things have not yet been implemented:

Sanity checks: Pending because the cluster mempool is still a work in progress. I have a WIP branch rebased on the cluster mempool to address this.
Package-aware block policy fee estimator: Also on hold for the same reason, awaiting the completion of the cluster mempool.

The fee estimator is currently a member of the node context exposed through the chain interfaces for the wallet and through node rpc’s.

The improvement above will require it to access both the mempool and the chainstate for building block templates.

I came across a suggestion that proposes:

https://github.com/bitcoin/bitcoin/blob/9f45062b9b0625cf4c714e2edbc7b6477164aaa9/src/interfaces/chain.h#L121-L123

This implies the need to create a fee estimation interface that would subscribe to chain interface notification events and expose its methods through the new fee interfaces.

IIUC the comment above hints towards having a separate fee estimation process in the future, allowing it to communicate with other interfaces through IPC?

This can be achieved by:

Decouple the notification handler from the chain interfaces so that the fee estimator does not have to interact directly with the chain interfaces. It should only grab validation signals from the node interfaces and subscribe to the relevant events.
Creating an interface for fee estimation, which will instantiate the new fee estimator class and expose some methods for clients.
Update createNewBlock method in mining interface to return the vector of packages and fees in the the created block, and allow skipping some transactions when constructing the block template.
Bitcoin node will instantiate the fee interface by passing node and mining interfaces to it.
The wallet interface should also access the fee interface when it is being instantiated.

Once we have a multi-process architecture, the fee estimator can run in its process and communicate with other components through IPC.

cc @willcl-ark @josibake @glozow

Because I believe you are interesting in this project

cc cc @ryanofsky @TheCharlatan

Because I believe you are working on multiprocess and kernel project

Feedback from other contributors is also welcomed and will be appreciated.

jonatack commented at 1:30 pm on October 10, 2024: member

Create estimatefee RPC

Four years ago I was beginning to implement the roadmap as described in #20484 (comment), beginning with a setfeerate RPC in #20391 and a planned estimatefeerate RPC, both using sat/vB units. Was considering picking these up again. Presumably orthogonal, apart from coordinating on the RPC naming.

ismaelsadeeq commented at 1:57 pm on October 10, 2024: member

Both using sat/vB units.

This is a good idea.

Beginning with a setfeerate RPC in #20391.

I learned about this issue after @murchandamus’s review comment #29278 (review). I think we should just unify settxfee and your setfeerate into one RPC?

Was considering picking these up again. Presumably orthogonal, apart from coordinating on the RPC naming.

Happy to review when you do. I think its related—one thing we have in mind now is to make the estimatefeerate/estimatefee RPC input be time in hours instead of blocks.

jonatack commented at 1:23 pm on October 11, 2024: member

Thanks!

I think we should just unify settxfee and your setfeerate into one RPC?

I’m not sure. Point 3 and the rationales in #20484 (comment) suggest a possibly safer way (perhaps you have a workaround in mind I’m overlooking).

remyers commented at 10:12 am on October 14, 2024: contributor

Concept ACK for improving fee estimation and this approach looks good. While reviwing data from txstats in the context of optimizing how Eclair manages its utxo set I noticed the large variation between estimatesmartfee and, for example, mempool.space fee estimation. Eclair uses estimatesmartfee and recently changed from the (previous) default to economical. Any improvements to fee estimation in bitcoind will directly help to reduce costs for Eclair node operators.

ismaelsadeeq commented at 9:04 pm on October 14, 2024: member

I’m not sure. Point 3 and the rationales in #20484 (comment) suggest a possibly safer way (perhaps you have a workaround in mind I’m overlooking).

You are right! I opened an issue for this #31088

ismaelsadeeq commented at 9:15 pm on October 14, 2024: member

While reviwing data from txstats in the context of optimizing how Eclair manages its utxo set I noticed the large variation between estimatesmartfee and, for example, mempool.space fee estimation.

Also see #30157 (comment)

Eclair uses estimatesmartfee and recently changed from the (previous) default to economical

Bitcoin Core 28.0 also updated its default to economical, so eclair users just need to update the node, and it should reflect the change.

glozow referenced this in commit 66aa6a47bd on Jan 8, 2025

jsarenik commented at 11:45 am on February 6, 2025: none

Just a few related notes, a compilation from what I have seen so far. HTH, may be biased and YMMV:

Yes, getting a transaction into a block (confirmed) is a future event so on all my nodes I do ln -nsf /dev/null ~/.bitcoin/fee_estimates.dat and never call the RPCs guessing “smart fee estimation” from historical values.

Setting a rational initial fee allows for simple (if not m-of-n multisig where n>=m>1) RBF replacements to increment the fee just by subtracting the vsize sats from one (or more, combined) output(s). See for example the RBF dance on mempool.space/signet.

In case of sat/vB units I would suggest at least three decimal places. Works in Blockstream Green Wallet. See a current example on mempool.space, whereas the fee-rate used was 20.001. Further I would suggest to use the transaction weight rather than vsize. The computation would look like ((weight*fee_rate)+3)/4 which is the same way how to get the vsize but once multiplying by the fee_rate (be it decimal or in sat/kvB) the results may differ.

Computaion of the top-left located “rational fee-rate” at Anyone’s Display is currently done by dividing the total_fee by number of blocks in the mempool. Try this shell-script to get the current mempool-sensitive value:

 0#!/bin/sh
 1
 2eval $(bitcoin-cli getmempoolinfo \
 3  |  tr -d ', ".' | tr : = \
 4  | sed 's/=0\+/=/;1,2d;$d')
 5blocks=$((($bytes+999999)/1000000))
 6rfr=$(( ($total_fee+999)/1000/$blocks ))
 7
 8# Exceptions:
 9#  1. if the computed value is below 1001
10#    set it to 1001
11test $rfr -lt 1001 && rfr=1001
12# 2. if there is only one block in the mempool
13#    set the value to 1000
14test $blocks -eq 1 && rfr=1000
15
16echo $rfr

Copying and pasting the above into you favorite Terminal works and to keep things clean I suggest running env -i PATH=$PATH HOME=$HOME sh first.

ismaelsadeeq commented at 4:07 pm on February 6, 2025: member

Hmm, I am not sure whether this is a rational fee rate.

It’s also unclear what confirmation target this “rational fee rate” is aiming for.
I see there is a case where you just 1000 blindly this is not rational at all, it may be that your mempool is sparse because you just started your node, and you only have 1 block worth of transactions.
The script you provided does not also check against the minimum relay fee rate. Hence you might provide a fee rate below that and your transaction might not even propagate to your peers.

But nice website, and thanks for looking into this!

ismaelsadeeq commented at 11:04 am on March 12, 2025: member

Following some positive feedback from the recent in-person Core Dev meeting, a notable one is from @ajtowns who suggest exposing the data used for the mempool health check

I’ve made several improvements:

Updating the design document to reflect recent improvements and is now shared on github gist for more review and feedback https://gist.github.com/ismaelsadeeq/bcfa6b1216eef10076665551d9885722)

Updated the Complete Feature Implementation

Mempool Health Check:
The node’s mempool is validated against miner mempools using a better approach I thought of recently.

This can easily be achieved through listening to validation interface notifications:

MempoolTransactionsRemovedForBlock: Use notification data to computes the total weight of transactions removed after a new block.
BlockConnected: Use notification data to computes the actual block weight.
Tracking this over six blocks provides a reliable indicator of mempool sanity.

This method is lightweight and aligns with the approach used by the current block policy estimator. (https://github.com/bitcoin/bitcoin/pull/28368

See Implementation https://github.com/bitcoin/bitcoin/pull/30157/commits/3e4f293ba3bc89f7a42b7c15179d466f83743730

Filtering Transactions Skewing Fee Estimates:

Transactions that consistently fail to confirm despite being in the top mempool block template are now filtered out. This prevents misleading fee rate estimates due to miner policy deviations. While rare, such cases can occurr (e.g., Ocean Mining).
See Implementation: https://github.com/bitcoin/bitcoin/pull/30157/commits/a81d3bda9c74936bb020d655443b31d692b17eb2

This is an example output (estimatefee 1 true) on mainet running #30157

 0{
 1  "minrelayfee": 0.00001000,
 2  "low": 0.00002006,
 3  "high": 0.00002118,
 4  "forecaster": "Mempool Forecast",
 5  "stats": [
 6    "Tracked 6 most-recent blocks.",
 7   "Block height: 887420, 3.99909e+06WU txs were seen in the mempool, Block weight: 3.99575e+06 WU (100.00% from mempool).",
 8    "Block height: 887421, 3.92792e+06WU txs were seen in the mempool, Block weight: 3.99165e+06 WU (98.40% from mempool).",
 9    "Block height: 887422, 4.00331e+06WU txs were seen in the mempool, Block weight: 3.99194e+06 WU (100.00% from mempool).",
10    "Block height: 887423, 3.995e+06WU txs were seen in the mempool, Block weight: 3.99189e+06 WU (100.00% from mempool).",
11    "Block height: 887424, 3.99632e+06WU txs were seen in the mempool, Block weight: 3.9919e+06 WU (100.00% from mempool).",
12    "Block height: 887425, 4.00402e+06WU txs were seen in the mempool, Block weight: 3.99197e+06 WU (100.00% from mempool).",
13    "We suspect 3 txs filtered by miners",
14    "02ebbd8ca9e05de5fa056031ef....; count 4"
15  ]
16}

cc @willcl-ark, @glozow @instagibbs @davidgumberg @ryanofsky

naiyoma commented at 5:46 pm on April 9, 2025: contributor

Currently, we have two RPCs for fee estimation. This new method will also introduce another RPC. Why not reuse the existing ones, since that’s what users are already familiar with—assuming they expect them to always return the most accurate fee between FeeRateForecasterManager and CBlockPolicyEstimator?

ismaelsadeeq commented at 2:33 pm on April 13, 2025: member

Good suggestion, I’ve done that. thanks @naiyoma see recent push in #30157

Fee Estimation via Fee rate Forecasters tracking issue #30392