node: add block template manager and track waitNext fee inflow #35581

pull ismaelsadeeq wants to merge 19 commits into bitcoin:master from ismaelsadeeq:07-2026-block-template-man changing 47 files +2303 −548

ismaelsadeeq commented at 10:50 AM on June 22, 2026: member
Motivation

The mining interface and rpc helpers (WaitAndCreateNewBlock and friends) are free functions scattered across src/node/miner.cpp, and every caller has to pass ChainstateManager and KernelNotifications into them, which is verbose. SubmitBlock spins up a short-lived CValidationInterface subscriber on every call just to read back the block validation state. This PR adds a BlockTemplateManager that holds that state and exposes those helpers as methods, so callers don't have to thread chainman/notifications (or a throwaway subscriber) around.

It also removes a redundant pattern in waitNext. Currently, to decide whether mempool fee inflow rose enough to be worth a new template, waitNext locks cs_main and assembles a full block template every tick just to sum its fees and compare. After cluster mempool see https://delvingbitcoin.org/t/determining-blocktemplate-fee-increase-using-fee-rate-diagram/2052, we shouldn't have to build a whole block template to answer that question. This PR uses the #34803 mempool notifications to record the previous template state and track the inflow above the lowest included chunk, and only assembles and returns a new template once that inflow is above the threshold.

I added a fuzz harness and unit tests for the manager. The fuzz test already caught a bug in the miner where it skips legitimate transactions due to an incorrect block chunk size limit check #35580.

Changes
- Populate the tx refs in the fee rate diagrams
- Cache fee rate diagrams as chunks in ChangeSet and add SnapshotDiagrams
- Add a MempoolUpdated notification that returns the before and after diagrams
- Track the selected chunk information after each mempool update
- Add fee-inflow staleness detection per template
- Introduce a BlockTemplateManager wrapper around BlockAssembler that holds state and creates templates, and move the free functions (mining args, SubmitBlock, GetTip, WaitTipChanged, CooldownIfHeadersAhead, WaitAndCreateNewBlock) onto it as methods
- Use the tracked inflow in waitNext instead of rebuilding a template each tick
- Route in-process RPC and test template creation through BlockTemplateManager instead of the IPC Mining interface, which also drops the per-access block copies the IPC accessors (getBlock) forced
- Add a fuzz test and unit tests
There are now two entry points for template creation. Mining clients go through the IPC Mining interface and get tracked templates, while in-process RPC and tests call BlockTemplateManager directly and get untracked ones. This is deliberate: one-shot RPC templates don't need staleness tracking, and it also avoids process-static RPC caches owning BlockTemplateImpl objects whose destructor reaches back into NodeContext at shutdown.

50% of the code addition here are tests ~1090 lines
DrahtBot commented at 10:50 AM on June 22, 2026: contributor


The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35581.



Reviews

See the guideline for information on the review process.

Type Reviewers

Concept ACK w0xlt

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.



Conflicts

Reviewers, this pull request conflicts with the following ones:
- #35587 (Remove boost as a unit test runner by rustaceanrob)
- #35580 (bugfix: compare non-adjusted chunk weight against block weight limit by ismaelsadeeq)
- #35511 (RFC: consensus: Make CAmount a class by hodlinator)
- #35474 (node: move index ownership to NodeContext by w0xlt)
- #34803 (mempool: asynchronous mempool fee rate diagram updates via validation interface by ismaelsadeeq)
- #34672 (mining: add reason/debug to submitSolution and unify with submitBlock by w0xlt)
- #34565 (refactor: extract BlockDownloadManager from PeerManagerImpl by w0xlt)
- #34075 (fees: Introduce Mempool Based Fee Estimation to reduce overestimation by ismaelsadeeq)
- #34020 (mining: add getTransactions(ByWitnessID) IPC methods by Sjors)
- #33922 (mining: add getMemoryLoad() and track template non-mempool memory footprint by Sjors)
- #32468 (rpc: generateblock to allow multiple outputs by polespinasa)
- #31260 (scripted-diff: Type-safe settings retrieval by ryanofsky)
- #31117 (miner: Reorg Testnet4 minimum difficulty blocks by fjahr)
- #30342 (kernel, logging: Pass Logger instances to kernel objects by ryanofsky)
- #29700 (kernel, refactor: return error status on all fatal errors by ryanofsky)
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
DrahtBot added the label CI failed on Jun 22, 2026
DrahtBot commented at 12:06 PM on June 22, 2026: contributor


🚧 At least one of the CI tasks failed. <sub>Task test ancestor commits: https://github.com/bitcoin/bitcoin/actions/runs/27947406418/job/82695559978</sub> <sub>LLM reason (✨ experimental): CI failed because the fuzz target (src/test/fuzz/blocktemplatemanager.cpp) would not compile: error: unknown type name 'SteadyClockContext'.</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
- Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
- A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
- An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.

</details>
ismaelsadeeq force-pushed on Jun 22, 2026
w0xlt commented at 8:50 PM on June 22, 2026: contributor

Concept ACK
ismaelsadeeq force-pushed on Jun 23, 2026
DrahtBot removed the label CI failed on Jun 23, 2026
DrahtBot added the label Needs rebase on Jun 23, 2026

Type	Reviewers
Concept ACK	w0xlt

txgraph: populate chunk refs in `GetMainStagingDiagrams` result

GetMainStagingDiagrams currently returns a pair of vector<FeeFrac>
containing only chunk feerates. Some callers need to know which
transactions belong to each chunk in order to compute a chunk hash.

Introduce TxGraph::Chunk, which pairs a chunk feerate with the refs of
all transactions in that chunk. Update GetMainStagingDiagrams to return a
pair of vector<TxGraph::Chunk>, and rename AppendChunkFeerates to
AppendChunks to reflect the richer return type.

Add SanityCheck coverage for the new behaviour.

9d73c1e2ea

refactor: move-only: split GetPackageHash 6cf2f99b63

mempool: cache fee rate diagrams as chunks in ChangeSet

CalculateChunksForRBF extracts only the FeeFrac from each TxGraph::Chunk
and discards the rest. Cache the fee rate diagram chunks in ChangeSet for
use in subsequent commits.

f3bb70bc62

mempool: add SnapshotDiagrams to ChangeSet

Extract SnapshotDiagrams from CalculateChunksForRBF so the diagram
snapshot can be taken on removal paths that do not go through the RBF
codepath.

The const fix to RemoveStaged is required because GetRemovals()
returns a const reference.

d5c9505709

refactor: move-only: extract dependency addition into a separate method 3221a002c9

mempool: add and fire MempoolUpdated signal on each mempool update path

Add MemPoolChunk and MemPoolChunksUpdate structs to
kernel/mempool_entry.h and MakeMemPoolChunks helper to txmempool.cpp
for converting TxGraph::Chunk vectors into MemPoolChunk vectors.

Each removal path now captures the before/after fee rate diagram chunks
and fires MempoolUpdated so subscribers can observe every mempool change.

This signal allows subscribers to observe mempool fee rate diagram
changes in an asynchronous way without holding the mempool lock.

This changes the order of operations in removeForBlock: all block
transactions in the mempool are now removed before conflict
removals.

In blockencodings_tests, SyncWithValidationInterfaceQueue() drains the
async MempoolUpdated signal queue before LOCK2(cs_main, pool.cs).
The LOCK2 moves after the sync to avoid deadlock.

160be0adf6

test: add unit test for mempool update validation events 96a2dc4993

node: introduce block template manager

Add BlockTemplateManager as an encapsulation
that wraps BlockAssembler::CreateNewBlock().

Add a simple unit test that verifies that block
template can be created via the block template manager.

08181220b6

test: add block template manager fuzz test

Add a fuzz target that creates block templates through
BlockTemplateManager under fuzzed mining options and asserts a template
is always produced.

8a4363ce6e

interfaces: create block template via block template manager

Route the Mining interface's createNewBlock() through
BlockTemplateManager::CreateNewTemplate() instead of constructing a
BlockAssembler directly, so all block template creation goes through the
manager.

595acb3762

node: move mining_args to block template manager

Pass the parsed mining args to BlockTemplateManager at construction and
expose them via GetInitBlockCreateOptions(), so the manager owns the
init-time block create options instead of NodeContext.

cbc15a02e2

miner: track selected chunks in block templates

Replace `m_package_feerates` with `m_template_chunks`, which records
each selected chunk's `FeePerWeight` feerate, unadjusted weight, sigops
cost, and the witness IDs of the transactions selected in that chunk.

This gives block templates enough information to describe both the
selection order and the transaction group selected at each step.

a1ca0a5213

node: add block template snapshot tracking

Index template chunks by hash and feerate to update
snapshots without sorting them after each mempool addition.

47f499999a

node: update template snapshots from mempool callbacks

Register BlockTemplateManager for validation callbacks
and keep tracked snapshots synchronized with mempool and
chain-tip changes.

eb04f25dfc

node: add block template staleness detection

Track template fee inflow and report staleness once the fee improvement
reaches the requested threshold. The fee improvement is the lower-bound
difference between the current tracked chunk fees and the original
template fees.

8b1c724aae

miner: move SubmitBlock into BlockTemplateManager

Move SubmitBlockStateCatcher and SubmitBlock from miner.cpp into
BlockTemplateManager as a member function. This groups block submission
with block creation in the same class. The function uses m_chainman
directly instead of taking it as a parameter.

f7f496d8dc

node: move tip and wait helpers into BlockTemplateManager

Move mining tip lookup and block-template waiting helpers into
BlockTemplateManager so the manager owns the template waiting flow,
including the notification wait helpers used by waitNext.

9099f6d875

rpc, test: build block templates via BlockTemplateManager

Use BlockTemplateManager directly for in-process RPC and test block
template creation instead of going through the IPC Mining interface.

This keeps RPC-created templates untracked and prevents process-static RPC
caches from owning BlockTemplateImpl objects whose destructor reaches back
into NodeContext during shutdown.

dcbd4dbeb6

miner: use tracked fee inflow for waitNext

Use BlockTemplateManager snapshot tracking to
decide when waitNext should return a replacement
template due to fee inflow.

This avoids rebuilding full templates on every wait
tick just to compare fees. Tracked snapshots are cleaned
up by BlockTemplateImpl destruction, so an original template
can remain alive and call waitNext again after a replacement is returned.

8c73017dab

ismaelsadeeq force-pushed on Jun 24, 2026
DrahtBot removed the label Needs rebase on Jun 24, 2026
Sjors commented at 2:59 PM on June 25, 2026: member
It would be nice to have a PR focussed on the BlockTemplateManager:
- 08181220b6 node: introduce block template manager
- 8a4363ce6e test: add block template manager fuzz test
- 595acb3762 interfaces: create block template via block template manager
- cbc15a02e2 node: move mining_args to block template manager
- f7f496d8dc miner: move SubmitBlock into BlockTemplateManager
- 9099f6d875 node: move tip and wait helpers into BlockTemplateManager
- dcbd4dbeb6 rpc, test: build block templates via BlockTemplateManager
I should be able to build #33922 / https://github.com/Sjors/bitcoin/pull/120 on top of that.

The waitNext() cluster mempool enhancement would be a natural followup.

Contributors

Linked (view graph)

#34803 mempool: asynchronous mempool fee rate diagram updates via validation interface #35580 bugfix: compare non-adjusted chunk weight against block weight limit

node: add block template manager and track waitNext fee inflow #35581

Motivation

Changes

Code Coverage & Benchmarks

Reviews

Conflicts