Header-only support for waitNext() #33756

issue Sjors opened this issue on October 31, 2025
  1. Sjors commented at 4:38 PM on October 31, 2025: member

    I'd like to expand the waitNext() Mining IPC function to optionally (via BlockWaitOptions) return a (empty) new template as soon as we have a header with enough proof-of-work, pending block download and/or validation.

    https://github.com/bitcoin/bitcoin/blob/3cd4263bf66408760ec3b4ce6ccedcc87e0d53de/src/interfaces/mining.h#L63-L74

    I suspect there's little benefit if compact block reconstruction succeeds, and IIUC we hold cs_main during this process anyway so there's no way to generate a template.

    But if we need a round trip to our peer because compact block construction didn't finish, it might be useful to create an empty block template and push it.

    This can hopefully be implemented without touching validation code, as long as the kernel can provide a notification. Currently it only notifies us of a better tip, not a better header.


    Closely related to this is the Stratum v2 concept of sending an empty (or optimistically filled) "future" template downstream for mining devices to load, followed by a prevhash messages when a new header is found. Whether that's actually more performant than the above probably depends on the details of ASIC firmware.

    Implementation wise, we'd need the same kernel notification for it. At the interface level it might look like a getFuture() method on BlockTemplate which could return a FutureTemplate interface with a wait() method on it. That would return a prevhash once we validated a header. The client then sends that downstream and immediately calls waitNext() on the original template in order to get a fresh full block template as soon as we processed the full block.


    In both cases, we'd need a way to notify the TP if it the block turned out to be invalid. That way it can revert to the previous template.

  2. Fi3 commented at 4:47 PM on October 31, 2025: none

    it make sense for the TP to optimistically create templates as soon as there is a possible new block since this will reduce the time that a miner waste on stale jobs. Is kinda safe since the TP can stop following the optimistic path when 2 or more consecutive invalid blocks are received. If we leave it like it is now the miner will keep mine on the old job until the block template do not provide a new job.

    So worst case we waste a bunch of hundreds milliseconds mining invalid jobs, that we are still going to waste mining on old jobs.

  3. ismaelsadeeq commented at 11:41 AM on November 7, 2025: member

    I'd like to expand the waitNext() Mining IPC function to optionally (via BlockWaitOptions) return an (empty) new template as soon as we have a header with sufficient proof-of-work, pending block download and/or validation.

    Why? you should expand on the motivations behind this and what exactly you aim to achieve with this feature.

    I suspect there’s little benefit if compact block reconstruction succeeds. And, if I understand correctly, we hold cs_main during this process anyway, so there’s no way to generate a template.

    Even if there is way to generate a template you haven’t validated the block, activate a new tip and evicted conflicts from the mempool yet, so generating a block template with the block assembler is likely to produce the same result as the one you’re currently working on.

    To construct an empty block that builds on the header you validated, you have two options:

    1. Update the block assembler code to accept a custom chain tip. This would require some refactoring instead I think it would be better to
    2. Isolate the header creation section of the block assembler into a function that takes the current block index and consensus parameters, then loads the next block header into the proposed block template. This subroutine could then be used by the mining interface/block template manager to generate an empty block template.

    Re: #33389 (comment)

    Yes, it is compatible. However, the approach I am aiming for in #33758 relies on non-blocking validation interface notifications, so we should simply add a new notification that is executed once a new valid header is known. This notification will wake up the block template manager, which will then use the subroutine isolated in step 2 above to construct a header with an empty template and push it to subscribed clients.

    As I mentioned in #33758, I would like to prune WaitAndCreateNewBlock and create a new GetNext() function on a block template manager that is decoupled from kernel notifications internals, the cs_main lock, and the kernel notification’s condition variable. The block template manager should have its own internal locks, maintain a copy of the chain tip, and wake up its own condition variables to execute what’s needed on the scheduler thread after receiving validation interface notifications.

    So subscriptions to next block template, next block header, can easily be added and tested in isolation.

  4. Sjors commented at 5:12 PM on November 7, 2025: member

    I'd like to expand the waitNext() Mining IPC function to optionally (via BlockWaitOptions) return an (empty) new template as soon as we have a header with sufficient proof-of-work, pending block download and/or validation.

    Why? you should expand on the motivations behind this and what exactly you aim to achieve with this feature.

    To allow miners to start hashing on an empty block template as soon as we see a new header, rather than waiting for the block to be fully downloaded and validated. This provides miners with additional revenue because they are less likely to mine a stale block [0]. The risk however is that the new block was invalid, in which case they're (briefly) mining on top of an invalid block.

    [0] if they happen to find a solution during this time, afaik we won't even broadcast it because it's not better than our current tip. But even if we did broadcast it, other nodes won't prefer it because of the first-seen rule.

  5. ismaelsadeeq commented at 6:10 PM on November 7, 2025: member

    Thanks for the context.

    This provides miners with additional revenue because they are less likely to mine a stale block [0]

    Before doing this, I think we should benchmark in practice the usual latency between knowing about a valid new block header, downloading, validating, and switching the chain tip. As you mentioned, for a node with a good connection and reasonable policy rules, compact block reconstruction should prevent any measurable latency. Also, we are likely going to have SENDTEMPLATE, which will help as well so might not be worth it if the time delta is not going to have impact with relation to the time it will take to generate the empty block, send to tp, and sending the new work to the miners in the pool. If the latency is not much, we just increase the chances of getting empty blocks and make miners lose on fees.

    We could also do this smartly, and only send the empty block when the number of transaction not in mempool and extra pool is high after we get the compact block.

    <details> <summary>Possible (costly) theoretical attack that this could open up</summary>

    Malicious miners could collude to reduce the chances of smaller miners who use this mechanism from mining blocks and increasing their hashrate when they do this by default.

    Whenever the chain tip changes on the valid chain, the malicious miners could collaborate and start working on an invalid block, first broadcasting a header with valid PoW but an invalid block (by consensus). As a result, smaller miners who rely on this mechanism might unknowingly keep working on a block that is valid in isolation but built on an invalid chain, which would later be rejected by the network.

    They can do this repeatedly, and since they have more hashrate, their chances of succeeding increase. The chance for smaller miners is further limited to the time between the tip changes and the larger miners mining an invalid block. But I think this is not realistic and the time they spend finding the invalid block they should rather just be honest and use it to mine.

    </details>

  6. Sjors commented at 6:17 PM on November 7, 2025: member

    we should benchmark in practice the usual latency between knowing about a valid new block header, downloading, validating, and switching the chain tip

    Agreed, ideally I'd like to see data from actual miners.

    compact block reconstruction should prevent any measurable latency

    Yes, but with the recent trend of people using incompatible policies, we should prepare for a scenario where compact block relay (often) doesn't work.

    I don't know if SENDTEMPLATE can fully compensate for broken compact relay or is more of a bandaid.

    theoretical attack that this could open up

    I agree that mining on just headers is not without risk. It should be something the miner can turn on or off. Unless of course we decide it's really too unsafe, then we shouldn't support it.

  7. willcl-ark added the label Feature on Nov 18, 2025
  8. willcl-ark added the label Mining on Nov 18, 2025
  9. enochaz commented at 9:29 AM on November 19, 2026: contributor

    Following @ismaelsadeeq's point about benchmarking the latency first, I collected some data on this and also sketched how a first version could look.

    Miners already start on empty templates after a new block

    Over the last year (blocks 923849 to 976408) there were 97 empty blocks. Comparing the header timestamp gap to their parent against all blocks (blocks with a timestamp earlier than their parent counted as neither):

    gap to parent empty blocks all blocks
    0-10s 20% (19/97) 1.1%
    0-30s 58% (56/97) 4.1%
    0-60s 76% (74/97) 8.1%

    Empty blocks are roughly 9-17x more likely than other blocks to come right after their parent. This can't tell apart pools mining on a header before validating the parent (spy mining) from pools sending an empty job while their full template is still being built, and header timestamps are noisy since miners set them. But either way it shows miners already prefer starting right away on an empty template over waiting for a full one. Doing it from the header in Core, as an opt-in, would give that earlier and without relying on their own setups.

    Header to tip latency on my node

    I logged header -> tip timings (logtimemicros, debug=cmpctblock) over 129 blocks (976408 to 976536) and split each block into stages:

    stage median p90 max
    fetch after header (network) 151 ms 556 ms 1.5 s
    getblocktxn round trip (network, 85 blocks) 276 ms 1.8 s 8.6 s
    fill from mempool (local) 8 ms 10 ms 13 ms
    connect / validate (local) 63 ms 107 ms 395 ms
    header -> tip total 512 ms 1.8 s 10.3 s
    • 75% of blocks (85/113) needed a getblocktxn round trip, so reconstruction without a round trip was the minority case here.
    • Local work was around 70 ms, the rest is network.
    • Left out of the table: 12 blocks that hit a link stall (2.8 to 50 s header -> tip) and 4 blocks right after a restart (cold cache).

    To be upfront, this node is a worst case: it's on 4G behind CGNAT, outbound only, ~200-450 ms ping. So I'm not using it to claim how large the gap is for everyone. Modelling the same blocks with local time + round trips x RTT, a node with 50 ms RTT would see ~160 ms median and ~500 ms max. But I think it does represent the miners this is for: home miners, small pools and sv2 job declaring miners, who are more likely to be on links like this. At 1-3 s per block that is roughly 0.2-0.5% of their hashrate spent on a stale tip.

    compact block reconstruction should prevent any measurable latency

    On good links it mostly does, but @0xB10C's measurements on Delving show reconstruction without a round trip hovering between 1-5% from mid July 2026, until the unsolicited tx chains being pushed to nodes stopped on 2026-10-18. This is the "incompatible policies" scenario @Sjors mentioned. #35558 (prefill) and SENDTEMPLATE both target that, and I see this as complementary rather than competing. In my data over half of the round trips (46/85) were for 1-5 missing txs, which prefill should fix. What's left are the big misses (e.g. 976475 needed 6556 txs / 607 kB fetched after the header) and link stalls, which prefill can't cover.

    Possible approach

    A small, low-risk first version could look like this:

    • Opt-in via a new BlockWaitOptions field, off by default, as Sjors suggested.
    • Only an empty template, and only for a header that directly extends our validated tip (1 deep max). Not during IBD.
    • Wake the template manager on a new header. KernelNotifications::headerTip already fires for this but only forwards to the UI, so either hook into that or add the validation interface notification ismaelsadeeq suggested. Either way no changes to validation logic or net_processing.
    • For building it, ismaelsadeeq's option 2 seems the better fit: factor the header creation out of BlockAssembler into a function that takes the prev block index and consensus params. An empty template doesn't need the mempool, so this avoids teaching BlockAssembler about a custom tip. This would sit in the block template manager wait loop.
    • Resolution through the next waitNext() call: a full template once the block connects, or a template on the old tip if the block turns out invalid or doesn't arrive before a timeout.
    • Only hand out the provisional template if the block isn't connected within a short delay after the header. This is close to ismaelsadeeq's idea of only doing it when many txs are missing, and it means good links (where the block connects in ~100-200 ms) would rarely produce one, so little to no increase in empty blocks for them.

    On fee loss: an empty template only matters if the miner finds a block during the gap, and during the gap the alternative is mining on a tip that is already stale. A full block on the new tip can't be built until we have the parent's txs anyway.

    On the attack in ismaelsadeeq's comment: an invalid header still costs a full block's PoW, the flag is opt-in, and depth is capped at 1. @Fi3's suggestion of backing off after consecutive invalid headers seems like a cheap extra guard worth including. A related case is a header announced with the block withheld, which CooldownIfHeadersAhead already guards against for createNewBlock() by not waiting on a header for too long. The timeout here could follow the same reasoning.

    The getFuture() / FutureTemplate part of the issue could come as a follow up, since it needs the same notification but a new interface.

    Open questions:

    • How should the client know a template is provisional? A field on BlockTemplate, or is it enough that it's empty and on a not yet validated prev?
    • What should the default delay before handing out a provisional template be?
    • Does this overlap too much with #35558 / SENDTEMPLATE, or does it make sense to do alongside them?

    The scripts and raw data are in this gist should one want to check them. Curious whether this direction makes sense.

    Scripts were written with help from an AI assistant; I reviewed the code and the results.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-11-20 15:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me