headersync: do parameter search at runtime #35642

pull sipa wants to merge 12 commits into bitcoin:master from sipa:202607_headerssync_cpp changing 18 files +563 −414
  1. sipa commented at 3:06 PM on July 2, 2026: member

    This migrates the headers presync parameters (commitment_period and redownload_buffer_size) from being precomputed chainparams to being computed at runtime during startup. This has as advantages:

    • Less work for maintainers that need to occasionally run the current contrib/devtools/headerssync-params.py script.
    • Less room for mistakes in that process.
    • Automatically adapts based on time (as opposed to needing to predict the lifetime of the software).

    To achieve this, the parameter search algorithm is ported to C++, and significant optimizations are applied to it step-by-step. The combination brings the runtime from minutes to ~3 ms. As an additional advantage, the result is now actually always optimal (within what can be determined using machine precision); the old code relied on (very reasonable) heuristics to guide the search.

    To make the changes reviewable without understanding the details of the algorithmic and mathematical optimizations, the PR starts with a few refactors to bring the existing Python code in a state where it can be tested. A set of test vectors is then introduced (in Python), and those vectors are ported to, and remain valid, in all the further C++ versions too.

    Disclaimer: this was written with significant assistance from Claude Opus 4.8 and Fable 5, which came up with the C++ port, the algorithmic and mathematical optimizations, and their implementation and testing. The code comments and commit messages are almost entirely written by me for clarity, but also to convince myself I understood all the changes.

  2. DrahtBot commented at 3:07 PM on July 2, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35642.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    Concept ACK sedited, darosior
    Approach ACK ajtowns

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35351 (net: Disallow invalid HeadersSyncState due to lagging clock by hodlinator)
    • #35301 (Silent Payments: Implement bip352 (take 2) by Eunovo)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. sipa force-pushed on Jul 2, 2026
  4. DrahtBot added the label CI failed on Jul 2, 2026
  5. sipa force-pushed on Jul 2, 2026
  6. in src/net_processing.cpp:2040 in cafcdc385a outdated
    2031 | @@ -2029,12 +2032,25 @@ std::unique_ptr<PeerManager> PeerManager::make(CConnman& connman, AddrMan& addrm
    2032 |      return std::make_unique<PeerManagerImpl>(connman, addrman, banman, chainman, pool, warnings, opts);
    2033 |  }
    2034 |  
    2035 | +//! Compute the headers sync DoS-protection parameters for the given chain, using its age (genesis
    2036 | +//! to now) and the height at which its minimum chain work was taken, and log the result.
    2037 | +static HeadersSyncParams MakeHeadersSyncParams(const CChainParams& chainparams)
    2038 | +{
    2039 | +    const auto genesis_time{NodeSeconds{std::chrono::seconds{chainparams.GenesisBlock().nTime}}};
    2040 | +    const auto timespan{std::chrono::duration_cast<std::chrono::seconds>(NodeClock::now() - genesis_time)};
    


    ajtowns commented at 3:51 PM on July 2, 2026:

    Is there any anonymity breaking via NodeClock::now() influencing these parameters? (I think maybe not, but might be worth quantizing or similar to guarantee it?)


    sipa commented at 4:18 PM on July 2, 2026:

    Good point, I had not considered that.

    The choice of the commitment period should not be observable (besides a very minor time leak perhaps). However, the redownload buffer is observable in that an attacker can probably disconnect mid-redownload, and see how much you accepted afterwards.

    I'll run some numbers to see how quickly these parameters change and suggest some quantization or noise.


    sipa commented at 5:00 PM on July 2, 2026:

    It looks like the output parameters only change roughly every 22 days. I don't think quantizing further gains us much, especially as the attack only applies during IBD, so peers generally already know you just started.

  7. sipa force-pushed on Jul 2, 2026
  8. sipa force-pushed on Jul 2, 2026
  9. DrahtBot commented at 2:33 PM on July 9, 2026: contributor

    Could turn into draft while CI is red?

  10. sipa force-pushed on Jul 9, 2026
  11. sipa force-pushed on Jul 9, 2026
  12. sipa force-pushed on Jul 9, 2026
  13. sipa force-pushed on Jul 9, 2026
  14. sipa force-pushed on Jul 9, 2026
  15. DrahtBot removed the label CI failed on Jul 9, 2026
  16. sipa force-pushed on Jul 10, 2026
  17. sipa force-pushed on Jul 10, 2026
  18. DrahtBot added the label CI failed on Jul 10, 2026
  19. DrahtBot removed the label CI failed on Jul 10, 2026
  20. sedited commented at 10:58 AM on July 24, 2026: contributor

    Concept ACK

  21. DrahtBot added the label Needs rebase on Jul 25, 2026
  22. ajtowns commented at 2:08 PM on August 6, 2026: contributor

    Approach ACK ; needs rebase if wanted for 32.0 I guess?

  23. darosior commented at 4:23 PM on August 6, 2026: member

    Concept ACK

  24. sipa force-pushed on Aug 6, 2026
  25. sipa commented at 5:46 PM on August 6, 2026: member

    Rebased.

  26. DrahtBot removed the label Needs rebase on Aug 6, 2026
  27. DrahtBot added the label CI failed on Aug 6, 2026
  28. DrahtBot removed the label CI failed on Aug 7, 2026
  29. shuv-amp commented at 4:11 PM on August 12, 2026: none

    next_height here is int, while m_current_height is int64_t and ValidateAndStoreRedownloadedHeader uses int64_t for the same step. For the long timewarp chains you mention, the presync height can exceed INT_MAX and overflow. Not a practical concern, but it should probably be int64_t to match the redownload path?

  30. contrib: reduce headerssync-params.py output to just the parameters
    In future commits, the logic in the headerssync-params.py tool will be
    made significantly faster, and integrated into the Bitcoin Core node
    software itself and run at runtime.
    
    To prepare for that, remove all output from the tool beyond the actual
    computation result:
    * Drop the printing of the search parameters being explored, because
      the search will become fast enough to not need progress output, and
      because it interferes with tests.
    * Drop the final memory configuration to maximally encapsulate the
      computation logic.
    * Drop memory_usage's now-unused second and third return values (the
      per-scenario breakdowns that only the removed output consumed); it
      now returns just the peak memory usage.
    0d4f51b6fe
  31. contrib: drop the RANDOMIZE_OFFSET option from headerssync-params.py
    RANDOMIZE_OFFSET was a configurable option, introduced in the early
    stages of the headers-presync work to investigate whether randomizing
    the commitment offset was worth it. As that turned out to be the case,
    the actual implementation ended up using it all the time.
    
    Thus, remove the configurable and treat it as always True, making the
    logic slightly simpler.
    683621fc13
  32. contrib: parameterize headerssync-params.py computation
    This moves the computationally expensive part of the optimization
    search into a function that only depends on 3 quantities:
    - The maximum number of headers a chain can have.
    - The number of known valid headers in the main chain.
    - The (fractional) number of headers an attacker is allowed to insert
      into a victim's database per attack.
    
    The goal is to make it possible to introduce tests in later commits
    that cover the hard part, without hardcoding config parameters that
    would invalidate the tests if changed.
    1791c6e3a5
  33. contrib: break headerssync-params ties deterministically
    It is possible to have multiple adjacent period values for which the
    peak memory usage is identical. The current randomized search will find
    a random one in this case, which is not great for testing.
    
    Add a postprocessing step that finds the minimum period value for the
    optimal found peak memory usage value, so the result is deterministic.
    
    Also make the attack_rate early-exit more robust: rather than comparing
    a single batch's contribution against the accumulated rate, compare a
    bound on the sum of all remaining batches. This gives a stronger
    guarantee that the reported solution is actually the optimal one, and
    it also turns out to be needed for the results to stay identical across
    the optimizations in later commits.
    b5694329be
  34. contrib: drop the ASSUME_CONVEX option from headerssync-params.py
    ASSUME_CONVEX assumes the period-to-memory-usage mapping (at the optimal buffer
    size for each period) is convex, which lets the search discard whole ranges of
    periods at once. It defaulted to True and was only ever run that way; the False
    path existed solely as a slower fallback with a stronger guarantee.
    
    Drop the option and always assume convexity, simplifying the search. A later
    commit replaces this random search entirely with a solver that finds the true
    optimum directly, without any convexity assumption, so the stronger-guarantee
    fallback is not needed in the meantime.
    6cd3d584a3
  35. contrib: add test vectors to headerssync-params.py
    This adds a collection of 50 randomly generated test vectors to the
    tool. This is in preparation for converting the code to C++ and
    applying significant optimization, while remaining confident that the
    behavior does not change.
    
    The test vectors are chosen to be inputs that are close to ones where
    the optimal output changes, to test the accuracy of the calculation.
    
    Run with pypy3, which evaluates the search far faster than CPython:
    
        pypy3 contrib/devtools/headerssync-params.py --selftest
    14ea0df5c5
  36. headerssync: port the sync-parameter optimizer to C++
    This adds a C++ implementation of the headerssync-params.py tool's
    parameter search logic to headerssync.cpp directly, together with its
    unit test vectors. The tool's commentary explaining the goals of the
    search is copied along with it.
    
    This is rather inefficient for now, but will be optimized in further
    commits. Because of that, the unit test only checks one randomly chosen
    vector per run for now; a later commit makes the computation fast
    enough to check all of them on every run.
    2069c8e258
  37. headerssync: compute attack_rate in closed form
    The attack rate averages the not-yet-detected probability over all
    {period} randomized commitment offsets. Doing so with an explicit loop
    over the alignments costs O(period) per processed batch. This can be
    computed much more efficiently however, by counting how many choices
    for the alignment have floor(forged_headers / period) commitments, how
    many have ceil(forged_headers / period), and weighting both
    accordingly. This results in just an O(1) cost per batch.
    6986a14703
  38. headerssync: drop find_bufsize optimization (preparation)
    The find_bufsize algorithm takes two arguments that help its search: a
    minimum bufsize to consider, and an upper bound on the amount of
    memory.
    
    After the changes in the next commit, these will not be needed anymore.
    Get rid of them beforehand here.
    5bf8096c4e
  39. headerssync: replace randomized search with better algorithm
    Replace the randomized search over all candidate periods with a faster
    deterministic algorithm:
    
    Step 1: Solve the continuous relaxation of the problem, where period and
            bufsize can be arbitrary real numbers, and round the resulting
            period to an integer. This can be done efficiently using
            Newton-Raphson iterations.
    Step 2: Compute the bufsize and peak memory usage corresponding to that
            rounded period.
    Step 3: Scan the remaining integer periods whose peak memory usage could
            still be at or below the peak memory usage reached so far.
    
    The continuous optimum in Step 1 is only used as a starting point: a bad
    guess makes Step 3 scan more candidate periods, but cannot affect the
    result.
    
    This also makes the computation fast enough that the unit test can check
    all test vectors on every run again, instead of one randomly chosen one.
    
    The new algorithm's continuous relaxation relies on max_headers being
    larger than 2 * minchainwork_headers, which any consistent clock
    satisfies by a wide margin. Assume() this in the optimizer, and clamp
    in ComputeHeadersSyncParams, so that mocked or badly wrong clocks
    cannot produce degenerate inputs (which could otherwise fail to
    converge).
    3431e8800d
  40. headerssync: make next_height type consistent 85c82e5f27
  41. net_processing: compute headers sync parameters at runtime
    So far, the headers sync parameters are committed to the repository,
    as part of the chain parameters. This requires maintainers to
    occasionally recompute them, based on a prediction of how long the
    release is likely to be used.
    
    With the code for the optimization algorithm ported to C++ and
    integrated in headerssync.cpp, it becomes possible to instead do this
    computation at runtime. This is less work, with fewer ways for
    maintainers and reviewers to get it wrong, and automatically adjusts
    based on clock time.
    
    Tests that construct many PeerManager instances can avoid recomputing
    the parameters each time by passing a precomputed value through
    PeerManager::Options. The fuzz targets that construct one per iteration
    pass a fixed constant, which both avoids the recomputation and makes
    their behavior independent of the (mocked) clock at construction time.
    635edca83b
  42. sipa force-pushed on Aug 14, 2026
  43. sipa commented at 5:36 PM on August 14, 2026: member

    @shuv-amp That's mostly independent of this PR, but added a commit to address that. I have also made a small simplification in the search algorithm.

    Ready for more review.

  44. shuv-amp commented at 7:30 PM on August 14, 2026: none

    ComputeHeadersSyncParams() doesn't terminate for a small minchainwork_headers combined with a short timespan:

    ComputeHeadersSyncParams(std::chrono::seconds{0}, 30228);  // hangs
    ComputeHeadersSyncParams(std::chrono::seconds{0}, 30229);  // period=2 bufsize=40
    

    30229 is the boundary; below it the region extends out to timespans around 1e4 s.

    In attack_rate(), the left side of future_limit * prob < 1.0e-16 * rate can't be negative, and the right side is exactly 0 once rate is below ~2.5e-308, so that break stops being reachable. The limit check is the only other exit and Step 1's Newton call doesn't pass one, so there it has none at all.

    The Python this replaces has the same shape (HEADER_BATCH_COUNT * prob < 1.0e-16 * rate * len(align_choices)) and doesn't terminate at those arguments either, so it isn't new, it just matters more now that it runs at startup.

    Non-strict is enough, since prob decreases to exactly 0 and takes the left side with it:

    -    if (future_limit * prob < 1.0e-16 * rate) break;
    +    if (future_limit * prob <= 1.0e-16 * rate) break;
    

    The 50 test vectors still reproduce with that, and 196/196 other inputs I compared are unchanged.

    Nothing in chainparams is close: testnet4 has the lowest minchainwork_height at 123613, and reaches an exponent of 531 against the 1075 where prob underflows. It is the region a new chain starts in though, and 0 hangs a different way (attack_headers is 0, so period is NaN and accept_forged_headers > 0.0 never holds), so a guard on the input might be worth having too.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-08-19 11:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me