RFC: Bitcoin Core Node `BlockTemplateManager`

ismaelsadeeq commented at 9:55 am on September 15, 2025: member

Bitcoin Core Node BlockTemplateManager main use should be handling block template creation for other components of the node. It should use the low-level block assembler to do that. This BlockTemplateManager should be initialized with a pointer to the mempool and a ChainstateManager reference. It should have access to the default block creation options and be instantiated during node startup and destroyed when the node is shutting down. This BlockTemplateManager should be available via the node interfaces.

During instantiation of BlockTemplateManager, it should create an instance of the BlockAssembler and subscribe to the validation interface BlockConnected notification.

Other components like the Mining interface (waitNext, createNewBlock), Peer Manager #33191, and Fee Estimation with the mempool #30157 will request a template with their respective block creation options and the number of seconds of how fresh they want the template to be.

The BlockTemplateManager will maintain a vector of previously built block templates as caches and check if there is a template match within the interval specified by the client. If there is, it will return the template; else it will build a new one and add it to the cache.

Whenever a new block is connected, the cache is cleared.

Future Improvement

Detect the fee increase in the mempool and create a new template based on that information, not just using a time interval. See discussion here: https://github.com/bitcoin/bitcoin/pull/31283#discussion_r1937451223 After this, it should be possible to add a push-based technique to the BlockTemplateManager where you can make a subscription to get a notification when a better template is available in the mempool or when the chain tip changes. This can effectively be used in waitNext and make it better than the current approach, which constantly calls createBlockTemplate every second and checks if it is a better template.
Low-priority, nice-to-have: prioritize clients; the mining interface should be served first before other components. (This is low-priority because of cache and template sharing between components. If a component calls BlockTemplateManager with the same config before the mining interface, the only overhead should be the shared pointer copy, which is negligible, I think. E.g., mempool-based fee estimation has the same config as the default configs for the mining interface. However, Peer Manager seems to have a diverging config; hence the priority will result in serving it faster and will have measurable improvement.)
Reuse Block Template with different Config as mentioned by @ajtowns

One thing that might be interesting to think about is whether a cached template can be reused despite different block creation options. I think in practice all the options can be boiled down to “maximum total weight” and “minimum effective feerate”, in which case if you had a 995 kvB template (ie 1MvB with 5kvB reserved), and had a request for a 300 kvB template, you could conceivably quickly generate the latter from the former, without needing to touch the mempool. That could probably be done particularly quickly if you kept a simplified dependency graph of the txs in the template, so that once you had to start dropping some txs, you could easily figure out which ones were missing parents and no longer sensible to consider. (Dealing with a different minimum effective feerate probably requires caching each txs’ effective feerate, and with cpfp considerations is probably difficult to do accurately unless you keep the ancestor information or the cluster mempool chunking information).

FAQ

Is this useful? The current approach of components each creating its instance of block assembler and effectively its own caching mechanism, as shown in #31664 and #33191, results in duplicate code and does not allow for better resource usage between the components.

Is there a POC? Yes, I’ve attempted this and have a branch here: https://github.com/ismaelsadeeq/bitcoin/tree/09-2025-minerman which has the mining interface createNewBlock using the BlockTemplateManager. #30157 will fit in nicely and reduce a large portion of mempool forecaster code, which was the caching code: https://github.com/ismaelsadeeq/bitcoin/commit/ba9b5876ae453315b57af09654d5b12b664e897a #33191 will also fit in nicely; see: https://github.com/ismaelsadeeq/bitcoin/commit/10bd8920fd259aa47d4b96037a2302d7ae6ea41f

ryanofsky commented at 6:08 pm on September 15, 2025: contributor

Another link to the POC branch: https://github.com/bitcoin/bitcoin/compare/master...ismaelsadeeq:bitcoin:09-2025-minerman

This seems like a good idea. The actual change here doesn’t seem very big and it seems like it could avoid some wasted memory and wasted work. It does seem like BlockTemplateManager is currently duplicating some logic in WaitAndCreateNewBlock, but I think the idea would be to replace it?

This idea also seems like it might relate to #31109 (if that issue is not already resolved by having having BlockTemplate::waitNext()).

Another minor piece of feedback is I think shared_ptr<CBlockTemplate> should be replaced by shared_ptr<const CBlockTemplate> in the branch to prevent callers from making changes to shared templates.

ajtowns commented at 5:37 am on September 16, 2025: contributor

[net: Provide block templates to peers on request #33191](https://github.com/bitcoin/bitcoin/pull/33191) will also fit in nicely; see: ismaelsadeeq@10bd892

I don’t understand this – peer template sharing seems a really bad fit for a template manager for three reasons:

the shared templates are probably oversized (2MvB instead of 1MvB) so aren’t directly useful to reuse in other contexts (and would be harmful if directly reused, as they would obviously be consensus invalid)
we probably want to retain knowledge about precisely when the template was generated to avoid leaking more precise information about when we added a tx to the mempool [ref]
the shared templates for peers are cached even after they’re no longer current in order to still be able to reply to requests and potentially to aid compact block reconstruction; so a different caching strategy compared to BlockTemplateManager’s is needed anyway

So as far as I can see these two features don’t really get a lot of benefit from each other.

In an ideal world, I think some of this is better done via cluster mempool – ie if we’re tracking the approximate feerate of the top 1MvB of mempool txs directly via the mempool (as per #31283 (review)), then perhaps the fee estimation stuff won’t need to invoke createblock at all.

Overall, this seems like a fine idea; at the very least the static variables inside getblocktemplate always seemed a bit tacky to me, and unifying gbt and the mining interface seems sensible. This seems worth doing even if it’s only useful for gbt and the mining interface to me.

will request a template with their respective block creation options and the number of seconds of how fresh they want the template to be.

One thing that might be interesting to think about is whether a cached template can be reused despite different block creation options. I think in practice all the options can be boiled down to “maximum total weight” and “minimum effective feerate”, in which case if you had a 995 kvB template (ie 1MvB with 5kvB reserved), and had a request for a 300 kvB template, you could conceivably quickly generate the latter from the former, without needing to touch the mempool. That could probably be done particularly quickly if you kept a simplified dependency graph of the txs in the template, so that once you had to start dropping some txs, you could easily figure out which ones were missing parents and no longer sensible to consider. (Dealing with a different minimum effective feerate probably requires caching each txs’ effective feerate, and with cpfp considerations is probably difficult to do accurately unless you keep the ancestor information or the cluster mempool chunking information).

I’d be tempted to call this BlockTemplateCache rather than ..Manager personally, but ymmv.

ismaelsadeeq commented at 9:34 am on September 16, 2025: member

Thanks for taking a look. @ryanofsky wrote

It does seem like BlockTemplateManager is currently duplicating some logic in WaitAndCreateNewBlock, but I think the idea would be to replace it?

For now I think it won’t exactly replace it; instead,

0            auto new_tmpl{BlockAssembler{
1                chainman.ActiveChainstate(),
2                mempool,
3                assemble_options}
4                              .CreateNewBlock()};

should be replaced with

0auto interval = std::chrono::seconds{0s};
1auto new_tmpl = minerman.GetBlockTemplate(assemble_options, interval);

such that other components apart from peer manager, as @ajtowns mentioned, can reuse this template since it is very consistent and fresh.

The logic should be replaced when F.1 1 has been implemented.

This idea also seems like it might relate to #31109 (if that issue is not already resolved by having having BlockTemplate::waitNext()).

Yes, if I understand correctly, F.I 1 solves part of the issue, which is deciding that the mempool fees have increased to the point where we should create a new template (it will just change internally without the CBT calls each second, to users of waitNext() nothing changes I think).

Another minor piece of feedback is I think shared_ptr should be replaced by shared_ptr in the branch to prevent callers from making changes to shared templates.

Thanks, I will update. @ajtowns wrote

the shared templates are probably oversized (2MvB instead of 1MvB) so aren’t directly useful to reuse in other contexts (and would be harmful if directly reused, as they would obviously be consensus invalid)

Yes, this is correct. But it will benefit from it in the F.I is added (there should be no point in creating a new template when nothing change in the previous one). As of now, it is not benefiting from the cache, and other components won’t benefit from it either (CAVEAT: when your idea is added other components will benefit).

One thing that might be interesting to think about is whether a cached template can be reused despite different block creation options.

This is the CAVEAT i mentioned above. I will add that to the list of F.I’s.

we probably want to retain knowledge about precisely when the template was generated to avoid leaking more precise information about when we added a tx to the mempool [ref]

This is trivial to return, because the time of creation is the main trigger for new block template generation.

the shared templates for peers are cached even after they’re no longer current in order to still be able to reply to requests and potentially to aid compact block reconstruction; so a different caching strategy compared to BlockTemplateManager’s is needed anyway

I don’t think this means an overlap. BlockTemplateManager also doesn’t clear when it is not current; it clears when the tip changes. You should be able to ask for all block templates with a particular config and get a list with the time of creation.

In an ideal world, I think some of this is better done via cluster mempool – ie if we’re tracking the approximate feerate of the top 1MvB of mempool txs directly via the mempool (as per #31283 (comment)), then perhaps the fee estimation stuff won’t need to invoke createblock at all.

Hmm, yes, that’s correct, and that will be better because the fee estimation only needs the package fee rate data anyway. Although I am not sure about the concrete approach to do this yet, I will think about it a bit more and also would like to know what @sipa thinks about this.

That could probably be done particularly quickly if you kept a simplified dependency graph of the txs in the template, so that once you had to start dropping some txs, you could easily figure out which ones were missing parents and no longer sensible to consider. (Dealing with a different minimum effective feerate probably requires caching each txs’ effective feerate, and with cpfp considerations is probably difficult to do accurately unless you keep the ancestor information or the cluster mempool chunking information).

This is a nice idea 👍🏾 , I will add it to the list of future improvements.

ryanofsky commented at 2:42 pm on September 16, 2025: contributor

Interesting discussion. Especially interesting to know about possibility of leaking transaction information if creation times aren’t tracked and used correctly. And to know how this could take more advantage of cluster mempool. I also like idea of calling this BlockTemplateCache instead of BlockTemplateManager to give it more focus.

Overall, notion of a shared cache seems useful. In terms of the implementation approach, I think it’d be nice if new code were integrated with existing code sooner rather than later (for example if waitNext could call this in the same PR it was introduced, or if this could be used to replace static BlockAssembler variables) to avoid having duplicate functionality, and make sure the new code is well tested and the interface is practically useful.

F.I 1

Note: in case anybody else was confused by this, F.I refers the “Future Improvements” section of the OP.

ismaelsadeeq commented at 10:27 am on September 17, 2025: member

Overall, notion of a shared cache seems useful. In terms of the implementation approach, I think it’d be nice if new code were integrated with existing code sooner rather than later (for example if waitNext could call this in the same PR it was introduced, or if this could be used to replace static BlockAssembler variables) to avoid having duplicate functionality, and make sure the new code is well tested and the interface is practically useful

I will be working on polishing the branch and opening it as a PR

ismaelsadeeq commented at 5:26 pm on September 23, 2025: member

I opened #33421 last week and made an update to it today, I think it is ready for for review now.

The fuzz test added in the PR also caught an issue on master that should be backported to v30 see #33421 (comment)

willcl-ark added the label Brainstorming on Sep 25, 2025

willcl-ark added the label Mining on Sep 25, 2025

RFC: Bitcoin Core Node BlockTemplateManager #33389

RFC: Bitcoin Core Node `BlockTemplateManager` #33389