Template Sharing Tracking Issue

ajtowns commented at 4:11 pm on October 23, 2025: contributor

Plan:

Protocol specification BIN25-2 and BIP153 PR#1937
Implementation:
- (See comments below)
Live tests
- (See comments below)
- inquisition.bitcoin-signet.net
- mainnet: cornflower.erisian.com.au provides templates currently
- warnet simulation?
Discussion:
- https://delvingbitcoin.org/t/sharing-block-templates/1906
Potential benefits: (tick with data confirming benefit)
- Quick population of mempool on initial start
- Quicker relay of top-of-mempool transactions
- Improved compact block propagation with diverse mempool policies
- Transaction rebroadcast without participation of sender/receiver
Future ideas:
- Include 2MvB of transactions, not just the next block
- Send templates shortly after construction
- Coordinate template construction with INV announcements better
- Send diffs since prior templates to reduce bandwidth
- Use set reconciliation to improve compact block reconstruction (see delving discussion)
- Use reduced siphash to make compact block calculations faster?

ajtowns added the label Tracking Issue on Oct 23, 2025

bitcoin deleted a comment on Oct 25, 2025

ajtowns commented at 2:53 pm on December 4, 2025: contributor

It looks like reusing compact block messages doesn’t make sense – including two blocks’ worth of data seems pretty useful, but that could result in an ~8MB blocktxns data if your mempool was empty and the top of the mempool was full of inscriptions, which would in turn exceed the 4MB maximum message size we have. We could expand the maximum message size, but large messages can be a DoS vector, so splitting the template response up probably makes more sense. That also provides the opportunity to describe the block template more directly, rather than pretending it’s a block with a header committing to the txids and perhaps a coinbase committing to the wtxids. So current work is on redesigning the p2p messages and making template reconstruction an iterative thing (“here’s the first batch of missing transactions”, …, “here’s the next batch of missing transactions”) rather than a one shot (“here’s all the missing transactions, you now have a template!”)

ajtowns renamed this:
~~SENDTEMPLATE Tracking Issue~~
Template Sharing Tracking Issue
on Mar 2, 2026

ajtowns commented at 6:33 am on March 2, 2026: contributor

Have an updated approach ~90% specced and implemented at:

The main change is it no longer attempts to reuse the compact block messages, but has its own scheme:

GETTMPLT [oldtemplatehash]
TMPLT [newtemplatehash] [tiphash] [what txns to pull from old template] [shortids for new txns]
GETTMPLTTXNS [templatehash] [delta encoded idxs to missing txs]
TMPLTTXNS [templatehash] [~100kB of requested txns]
TMPLTTXNS [templatehash] [~100kB of requested txns]
TMPLTTXNS [templatehash] [~100kB of requested txns]

Splitting the actual txns from the template into chunk avoids messages potentially going over 4MB and also allows the sender to interleave other messages in between (eg their own tx requests, or a block announcement if a real block comes in).

The implementation has a bunch of privacy code (particularly providing different templates on different networks, building the template at the same time as we would be announcing INVs, extremely limited template relay over block-relay-only connections), but the final commits are still a bit vibe-codey, though I think the idea is roughly sound. The spec is still missing proper rationale descriptions for a bunch of concerns, including things the implementation takes into account.

I did a flamegraph as a quick performance check. It ended up dominated by leveldb compaction, but it looks like the ratio between CreateNewBlock() + GenerateTemplate() + GetNextTemplateTx() and the existing TxDownloadManager::GetRequestsToSend() is about 67 to 119, so at idle (when it’s just regularly generating templates for the “outbounds” “network”) cpu usage seems fine at least.

Anyway, if anyone wants to take a look, mess around, or otherwise provide feedback before I do another round of dev to get it more polished, links are above.

ajtowns commented at 1:11 pm on March 9, 2026: contributor

Bandwidth usage on mainnet is a bit higher than I’d like – seems to be comparable to INV traffic. Currently exploring using minisketch to reduce how much data is needed to describe a template update.

ajtowns commented at 10:22 am on April 2, 2026: contributor

Minisketch-based implementation prototype at https://github.com/ajtowns/bitcoin/commits/202603-gettmplt/ ; seems to work on mainnet, but it’s a ways away from being production-ready, and isn’t specced/documented. Hmm, probably has a memory leak too? Anyway, rough idea of the approach:

gettmplt message requests a new template, possibly referencing a previous template from this peer as a basis
8M weight template is generated by the peer at ~30s intervals; txs are represented by sorted 46-bit shortids
tmplt response is given, indicating which entries from the previous template are still present, and include 4x 64-capacity minisketches of the shortids
can repeat gettmplt / tmplt three times, each of which divides the sketches in two, for an eventual total of 32x 64-capacity minisketches
if that doesn’t resolve, a final gettmplt / tmplt round provides literal shortids (minus ~2048 entries, which are resolved via minisketch)
with shortids resolved, attempts to reconstruct txs from templates, mempool and extra pool
gettmplttxn message to request any missing txs
tmplttxn messages to supply the missing txs, each limited to ~100kB

Transactions from templates are added to the mempool if possible, with 1p1c package acceptance where possible, and kept around for compact block reconstruction (and template reconstruction). gettemplateinfo and a field in getpeerinfo gives some insight, gettmplt log category does likewise. Delta to basis and request for missing txs are GR-encoded.

Seems like the sketches have improved the bandwidth usage a bit as hoped, and so far seems like CPU usage is pretty reasonable.

Template Sharing Tracking Issue #33691