Fee Estimation via Fee rate Forecasters tracking issue

ismaelsadeeq commented at 11:33 am on July 4, 2024: member

This is a tracking issue for an effort to add Fee Estimation with Forecasters (Mempool-Based Fee rate Forecaster) to Bitcoin Core.

For the conceptual motivation behind this work, see issue #27995 and the previous discussion on Delving Bitcoin: Mempool-Based Fee Estimation on Bitcoin Core.

What to Review

#30391

Roadmap

#30391
Add Fee Estimator Module / Mempool Forecaster and Unit Tests
Create estimatefee RPC and Functional Test
Add Fee Estimation Traces
Add Feature to the Wallet

Complete Feature

#30157

Future Plans for Fee Estimation with Forecasters after the Cluster Mempool Project

Add sanity checks to ensure it is safe to use mempool-based fee estimation without falling back to BlockPolicyEstimator, using ideas from #27995
Use heuristics from the mempool to target a higher/lower percentile in the mempool forecaster

Improving the current `BlockPolicyEstimator` in the Short Term

Improving the current `BlockPolicyEstimator` after the Cluster Mempool Project

Make BlockPolicyEstimator package-aware: Delving Bitcoin Post

Previous Refactors

glozow added the label TX fees and policy on Jul 18, 2024

ismaelsadeeq renamed this:
~~Improving Fee estimation tracking issue~~
Fee Estimation via Fee rate Forecasters tracking issue
on Jul 18, 2024

ismaelsadeeq commented at 1:15 pm on October 10, 2024: member

This issue was opened to track the effort to improve the Bitcoin core fee estimator. This comment will provide information on the status of this project and some key ideas on how to achieve this.

Completed Tasks so far:

Enabled asynchronous block policy fee estimator updates: By making it a client of the validation interface.
Empirical data analysis: To determine which forecasting strategy provides the most accurate estimates while withstanding common attack vectors.
Developed a rough model of the fee estimator architecture: This model will enable adding more fee estimation strategies in the future.
Ideas on how policy estimator will improve accuracy post-cluster mempool: By tracking chunks instead of individual transactions.
Brainstorm sanity checks for the fee estimator algorithms: These checks ensure more robust estimates and help avoid common pitfalls when analyzing mempool data.

Current Status:

We have a rough implementation of the fee estimator that attempts to integrate these improvements. However, two things have not yet been implemented:

Sanity checks: Pending because the cluster mempool is still a work in progress. I have a WIP branch rebased on the cluster mempool to address this.
Package-aware block policy fee estimator: Also on hold for the same reason, awaiting the completion of the cluster mempool.

The fee estimator is currently a member of the node context exposed through the chain interfaces for the wallet and through node rpc’s.

The improvement above will require it to access both the mempool and the chainstate for building block templates.

I came across a suggestion that proposes:

https://github.com/bitcoin/bitcoin/blob/9f45062b9b0625cf4c714e2edbc7b6477164aaa9/src/interfaces/chain.h#L121-L123

This implies the need to create a fee estimation interface that would subscribe to chain interface notification events and expose its methods through the new fee interfaces.

IIUC the comment above hints towards having a separate fee estimation process in the future, allowing it to communicate with other interfaces through IPC?

This can be achieved by:

Decouple the notification handler from the chain interfaces so that the fee estimator does not have to interact directly with the chain interfaces. It should only grab validation signals from the node interfaces and subscribe to the relevant events.
Creating an interface for fee estimation, which will instantiate the new fee estimator class and expose some methods for clients.
Update createNewBlock method in mining interface to return the vector of packages and fees in the the created block, and allow skipping some transactions when constructing the block template.
Bitcoin node will instantiate the fee interface by passing node and mining interfaces to it.
The wallet interface should also access the fee interface when it is being instantiated.

Once we have a multi-process architecture, the fee estimator can run in its process and communicate with other components through IPC.

cc @willcl-ark @josibake @glozow

Because I believe you are interesting in this project

cc cc @ryanofsky @TheCharlatan

Because I believe you are working on multiprocess and kernel project

Feedback from other contributors is also welcomed and will be appreciated.

jonatack commented at 1:30 pm on October 10, 2024: member

Create estimatefee RPC

Four years ago I was beginning to implement the roadmap as described in #20484 (comment), beginning with a setfeerate RPC in #20391 and a planned estimatefeerate RPC, both using sat/vB units. Was considering picking these up again. Presumably orthogonal, apart from coordinating on the RPC naming.

ismaelsadeeq commented at 1:57 pm on October 10, 2024: member

Both using sat/vB units.

This is a good idea.

Beginning with a setfeerate RPC in #20391.

I learned about this issue after @murchandamus’s review comment #29278 (review). I think we should just unify settxfee and your setfeerate into one RPC?

Was considering picking these up again. Presumably orthogonal, apart from coordinating on the RPC naming.

Happy to review when you do. I think its related—one thing we have in mind now is to make the estimatefeerate/estimatefee RPC input be time in hours instead of blocks.

jonatack commented at 1:23 pm on October 11, 2024: member

Thanks!

I think we should just unify settxfee and your setfeerate into one RPC?

I’m not sure. Point 3 and the rationales in #20484 (comment) suggest a possibly safer way (perhaps you have a workaround in mind I’m overlooking).

remyers commented at 10:12 am on October 14, 2024: contributor

Concept ACK for improving fee estimation and this approach looks good. While reviwing data from txstats in the context of optimizing how Eclair manages its utxo set I noticed the large variation between estimatesmartfee and, for example, mempool.space fee estimation. Eclair uses estimatesmartfee and recently changed from the (previous) default to economical. Any improvements to fee estimation in bitcoind will directly help to reduce costs for Eclair node operators.

ismaelsadeeq commented at 9:04 pm on October 14, 2024: member

I’m not sure. Point 3 and the rationales in #20484 (comment) suggest a possibly safer way (perhaps you have a workaround in mind I’m overlooking).

You are right! I opened an issue for this #31088

ismaelsadeeq commented at 9:15 pm on October 14, 2024: member

While reviwing data from txstats in the context of optimizing how Eclair manages its utxo set I noticed the large variation between estimatesmartfee and, for example, mempool.space fee estimation.

Also see #30157 (comment)

Eclair uses estimatesmartfee and recently changed from the (previous) default to economical

Bitcoin Core 28.0 also updated its default to economical, so eclair users just need to update the node, and it should reflect the change.

Fee Estimation via Fee rate Forecasters tracking issue #30392