node: add block template manager and track waitNext fee inflow #35581

ismaelsadeeq commented at 10:50 AM on June 22, 2026: member

Motivation

The mining interface and rpc helpers (WaitAndCreateNewBlock and friends) are free functions scattered across src/node/miner.cpp, and every caller has to pass ChainstateManager and KernelNotifications into them, which is verbose. SubmitBlock spins up a short-lived CValidationInterface subscriber on every call just to read back the block validation state. This PR adds a BlockTemplateManager that holds that state and exposes those helpers as methods, so callers don't have to thread chainman/notifications (or a throwaway subscriber) around.

It also removes a redundant pattern in waitNext. Currently, to decide whether mempool fee inflow rose enough to be worth a new template, waitNext locks cs_main and assembles a full block template every tick just to sum its fees and compare. After cluster mempool see https://delvingbitcoin.org/t/determining-blocktemplate-fee-increase-using-fee-rate-diagram/2052, we shouldn't have to build a whole block template to answer that question. This PR uses the #34803 mempool notifications to record the previous template state and track the inflow above the lowest included chunk, and only assembles and returns a new template once that inflow is above the threshold.

I added a fuzz harness and unit tests for the manager. The fuzz test already caught a bug in the miner where it skips legitimate transactions due to an incorrect block chunk size limit check #35580.

Changes

Populate the tx refs in the fee rate diagrams
Cache fee rate diagrams as chunks in ChangeSet and add SnapshotDiagrams
Add a MempoolUpdated notification that returns the before and after diagrams
Track the selected chunk information after each mempool update
Add fee-inflow staleness detection per template
Introduce a BlockTemplateManager wrapper around BlockAssembler that holds state and creates templates, and move the free functions (mining args, SubmitBlock, GetTip, WaitTipChanged, CooldownIfHeadersAhead, WaitAndCreateNewBlock) onto it as methods
Use the tracked inflow in waitNext instead of rebuilding a template each tick
Route in-process RPC and test template creation through BlockTemplateManager instead of the IPC Mining interface, which also drops the per-access block copies the IPC accessors (getBlock) forced
Add a fuzz test and unit tests

There are now two entry points for template creation. Mining clients go through the IPC Mining interface and get tracked templates, while in-process RPC and tests call BlockTemplateManager directly and get untracked ones. This is deliberate: one-shot RPC templates don't need staleness tracking, and it also avoids process-static RPC caches owning BlockTemplateImpl objects whose destructor reaches back into NodeContext at shutdown.

50% of the code addition here are tests ~1090 lines

DrahtBot commented at 10:50 AM on June 22, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35581.

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
Concept ACK	w0xlt, pablomartin4btc

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#36244 (validation, net: Process blocks asynchronously and reduce cs_main contention by w0xlt)
#36188 (crypto: plug hardware optimized SHA256 into libsecp256k1's context by furszy)
#36182 (fees: return block_policy fee rate estimate when mempool_policy is not ready by ismaelsadeeq)
#36156 (mining: include chunks that exactly fill block limits by l0rinc)
#36109 (test: harden arbitrary-parent block creation by l0rinc)
#36097 (mining: replace interrupt methods with cancellation arguments by xyzconstant)
#36091 (test: Add debug output to common tested types by rustaceanrob)
#36047 (rpc: handle createNewBlock() returning nullptr during shutdown by chriszeng1010)
#36015 (txorphanage: bound orphan memory by storing transactions serialized by brunoerg)
#36000 (validation: prefetch blocks while connecting by l0rinc)
#35671 (mining: add TxCollection to bandwidth-efficiently validate external block templates by Sjors)
#35646 (RFC: Separate out runtime errors from BlockValidationState using util::Expected by yuvicc)
#35569 (Encapsulation for CTransaction by purpleKarrot)
#35511 (RFC: consensus: Make CAmount a class by hodlinator)
#33922 (mining: add getMemoryLoad() and track template non-mempool memory footprint by Sjors)
#32468 (rpc: generateblock to allow multiple outputs by polespinasa)
#29700 (kernel, refactor: return error status on all fatal errors by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label CI failed on Jun 22, 2026

DrahtBot commented at 12:06 PM on June 22, 2026: contributor

🚧 At least one of the CI tasks failed. Task test ancestor commits: https://github.com/bitcoin/bitcoin/actions/runs/27947406418/job/82695559978 LLM reason (✨ experimental): CI failed because the fuzz target (src/test/fuzz/blocktemplatemanager.cpp) would not compile: error: unknown type name 'SteadyClockContext'.

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

ismaelsadeeq force-pushed on Jun 22, 2026

w0xlt commented at 8:50 PM on June 22, 2026: contributor

Concept ACK

ismaelsadeeq force-pushed on Jun 23, 2026

DrahtBot removed the label CI failed on Jun 23, 2026

DrahtBot added the label Needs rebase on Jun 23, 2026

ismaelsadeeq force-pushed on Jun 24, 2026

DrahtBot removed the label Needs rebase on Jun 24, 2026

Sjors commented at 2:59 PM on June 25, 2026: member

It would be nice to have a PR focussed on the BlockTemplateManager:

08181220b6 node: introduce block template manager
8a4363ce6e test: add block template manager fuzz test
595acb3762 interfaces: create block template via block template manager
cbc15a02e2 node: move mining_args to block template manager
f7f496d8dc miner: move SubmitBlock into BlockTemplateManager
9099f6d875 node: move tip and wait helpers into BlockTemplateManager
dcbd4dbeb6 rpc, test: build block templates via BlockTemplateManager

I should be able to build #33922 / https://github.com/Sjors/bitcoin/pull/120 on top of that.

The waitNext() cluster mempool enhancement would be a natural followup.

in src/node/block_template_manager.cpp:234 in 8c73017dab

 229 | +                                   CAmount fee_threshold) const
 230 | +{
 231 | +    LOCK(m_mutex);
 232 | +    const auto snapshot = m_template_snapshots.find(template_id);
 233 | +    if (snapshot == m_template_snapshots.end()) return false;
 234 | +    if (snapshot->second.options != FlattenMiningOptions(options)) return false;

pablomartin4btc commented at 12:52 PM on July 1, 2026:

IsStale is a public method but has an implicit precondition: options must match the ones used when template_id was created, otherwise it silently returns false. Could a comment document this (if appropriate)?

in src/node/block_template_manager.cpp:341 in 8c73017dab outdated

 336 | +    bool& interrupt_wait,
 337 | +    uint64_t& new_template_id)
 338 | +{
 339 | +    auto now{NodeClock::now()};
 340 | +    const auto deadline = now + options.timeout;
 341 | +    const MillisecondsDouble tick{1000};

pablomartin4btc commented at 1:11 PM on July 1, 2026:

There was a comment there in the old code that was removed as I think part of it was no longer accurate. Perhaps it can be add:

// Alternate waiting for a new tip and checking if fees have risen.
// The staleness check is cheap but template creation is not, so we only rebuild once per second.

ismaelsadeeq commented at 11:07 AM on July 7, 2026:

Fixed in #35675

pablomartin4btc commented at 1:37 PM on July 1, 2026: member

Concept ACK - first time reviewing this area.

I can see this PR is based on top of #34803 (first 7 commits). The approach of moving free functions onto a class that holds state plus incremental snapshot tracking to avoid full template rebuilds seems clean. The fuzz test catching an actual bug (#35580) is a strong signal.

Left a few comments.

ismaelsadeeq commented at 3:46 PM on July 1, 2026: member

Thanks for taking a look at this @pablomartin4btc, I will address these comments when I open up a PR that adds the block template manager alone, as @Sjors suggested #35581 (comment).

ismaelsadeeq marked this as a draft on Jul 7, 2026

ismaelsadeeq force-pushed on Jul 7, 2026

DrahtBot added the label CI failed on Jul 7, 2026

DrahtBot added the label Needs rebase on Jul 7, 2026

ismaelsadeeq force-pushed on Jul 8, 2026

DrahtBot removed the label Needs rebase on Jul 8, 2026

DrahtBot removed the label CI failed on Jul 8, 2026

willcl-ark added the label Mining on Jul 9, 2026

willcl-ark added the label interfaces on Jul 9, 2026

DrahtBot added the label Needs rebase on Jul 23, 2026

node: introduce BlockTemplateManager

Add BlockTemplateManager, a wrapper around
BlockAssembler::CreateNewBlock(), and store it in NodeContext.

Wire it into node init and test setup so it is reset before its
mempool/chainman dependencies, and update tests and fuzz setups that
rebuild chainman or mempool. Add a unit test that verifies a block
template can be created through the manager.

482a5d2712

node: move mining_args to BlockTemplateManager

Pass the parsed mining args to BlockTemplateManager at construction
and expose them via BlockCreateArgs(), so the manager owns the
init-time block create options instead of NodeContext.

bcdadf2b71

miner: move SubmitBlock into BlockTemplateManager

Move SubmitBlockStateCatcher and SubmitBlock from miner.cpp into
BlockTemplateManager as a member function. This groups block submission
with block creation in the same class. The function uses m_chainman
directly instead of taking it as a parameter.

07964c525d

node: move tip and wait helpers into BlockTemplateManager

Move the mining tip lookup and block-template waiting helpers (GetTip,
WaitTipChanged, WaitAndCreateNewBlock, InterruptWait,
CooldownIfHeadersAhead) into BlockTemplateManager so the manager owns
the template waiting flow. The manager now takes KernelNotifications
at construction.

037d3519f2

interfaces: create block template via BlockTemplateManager

Route the Mining interface's createNewBlock() through
BlockTemplateManager::CreateNewTemplate() instead of constructing a
BlockAssembler directly. Merging the init-time defaults into unset
options now happens inside CreateNewTemplate(), so every caller gets
them applied.

400ac328c9

rpc: route getblocktemplate internals through node

Add EnsureBlockTemplateManager() and use it in getblocktemplate for tip
lookup and longpoll waiting, while using ChainstateManager directly for
test-chain and IBD checks.

011cbfbcd2

rpc: build getblocktemplate via BlockTemplateManager 0998c0bbdf

rpc: do not copy template data in getblocktemplate

The transactions, fees and sigops costs, and the coinbase outputs
are only read, so reference them from the template instead of copying
them out of it.

951fac807d

rpc: only copy the header in getblocktemplate

The block copy exists solely to apply header adjustments (time, nonce,
version bits) that must not mutate the cached template. Now that the
transactions are read directly from the template, copy only the header
instead of the entire block.

38556f817e

rpc: build generation templates via BlockTemplateManager

generateblock is no longer safe for fuzzing: it previously threw at
EnsureMining (node.mining is never set in the fuzz setup), but with
EnsureBlockTemplateManager it now executes, mining and submitting a
real block. Submission mutates the chain state shared across fuzz
iterations and writes the block to disk.

8372721f20

rpc: wait for tips via BlockTemplateManager

The waitforblock, waitforblockheight and waitfornewblock RPCs are no
longer safe for fuzzing: they previously threw at EnsureMining
(node.mining is never set in the fuzz setup), but with
EnsureBlockTemplateManager they now really wait. The tip never changes
during fuzzing and nothing interrupts the wait, so a call without a
timeout blocks forever.

ec4f24ef18

test: create templates via BlockTemplateManager 6c71166bcf

node: remove NodeContext::mining and EnsureMining ffe3a631f3

ci: enforce iwyu for block template manager 3d7d190b69

test: fuzz BlockTemplateManager e8b283c35a

txgraph: populate chunk refs in `GetMainStagingDiagrams` result

GetMainStagingDiagrams currently returns a pair of vector<FeeFrac>
containing only chunk feerates. Some callers need to know which
transactions belong to each chunk in order to compute a chunk hash.

Introduce TxGraph::Chunk, which pairs a chunk feerate with the refs of
all transactions in that chunk. Update GetMainStagingDiagrams to return a
pair of vector<TxGraph::Chunk>, and rename AppendChunkFeerates to
AppendChunks to reflect the richer return type.

Add SanityCheck coverage for the new behaviour.

bfe71afd43

refactor: move-only: split GetPackageHash cc79146ba9

mempool: cache fee rate diagrams as chunks in ChangeSet

CalculateChunksForRBF extracts only the FeeFrac from each TxGraph::Chunk
and discards the rest. Cache the fee rate diagram chunks in ChangeSet for
use in subsequent commits.

b8c8feebcb

mempool: add SnapshotDiagrams to ChangeSet

Extract SnapshotDiagrams from CalculateChunksForRBF so the diagram
snapshot can be taken on removal paths that do not go through the RBF
codepath.

The const fix to RemoveStaged is required because GetRemovals()
returns a const reference.

e2dc04ed16

refactor: move-only: extract dependency addition into a separate method d7c2a0fa4e

ismaelsadeeq force-pushed on Sep 13, 2026

DrahtBot removed the label Needs rebase on Sep 13, 2026

DrahtBot added the label CI failed on Sep 13, 2026

DrahtBot commented at 3:05 PM on September 13, 2026: contributor

🚧 At least one of the CI tasks failed. Task FreeBSD Cross: https://github.com/bitcoin/bitcoin/actions/runs/34759582461/job/103729898040 LLM reason (✨ experimental): CI failed because clang’s thread-safety analysis (Wthread-safety-analysis) in src/test/miner_tests.cpp reported cs_main locking violations, and -Werror turned them into build-stopping errors.

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

ismaelsadeeq force-pushed on Sep 13, 2026

mempool: add and fire MempoolUpdated signal on each mempool update path

Add MemPoolChunk and MemPoolChunksUpdate structs to
kernel/mempool_entry.h and MakeMemPoolChunks helper to txmempool.cpp
for converting TxGraph::Chunk vectors into MemPoolChunk vectors.

Each removal path now captures the before/after fee rate diagram chunks
and fires MempoolUpdated so subscribers can observe every mempool change.

This signal allows subscribers to observe mempool fee rate diagram
changes in an asynchronous way without holding the mempool lock.

This changes the order of operations in removeForBlock: all block
transactions in the mempool are now removed before conflict
removals.

In blockencodings_tests, SyncWithValidationInterfaceQueue() drains the
async MempoolUpdated signal queue before LOCK2(cs_main, pool.cs).
The LOCK2 moves after the sync to avoid deadlock.

4428f963e0

test: add unit test for mempool update validation events aa4d25509a

miner: track selected chunks in block templates

Replace `m_package_feerates` with `m_template_chunks`, which records
each selected chunk's `FeePerWeight` feerate, unadjusted weight, sigops
cost, and the witness IDs of the transactions selected in that chunk.

This gives block templates enough information to describe both the
selection order and the transaction group selected at each step.

d05e6996f9

node: add block template snapshot tracking

Index template chunks by hash and feerate to update
snapshots without sorting them after each mempool addition.

038feb8759

node: update template snapshots from mempool callbacks

Register BlockTemplateManager for validation callbacks
and keep tracked snapshots synchronized with mempool and
chain-tip changes.

ae40e1770d

node: add block template staleness detection

Track template fee inflow and report staleness once the fee improvement
reaches the requested threshold. The fee improvement is the lower-bound
difference between the current tracked chunk fees and the original
template fees.

93c7112104

miner: use tracked fee inflow for waitNext

Use BlockTemplateManager snapshot tracking to
decide when waitNext should return a replacement
template due to fee inflow.

This avoids rebuilding full templates on every wait
tick just to compare fees. Tracked snapshots are cleaned
up by BlockTemplateImpl destruction, so an original template
can remain alive and call waitNext again after a replacement is returned.

24a87b7869

ismaelsadeeq force-pushed on Sep 15, 2026

DrahtBot removed the label CI failed on Sep 15, 2026