headersync: do parameter search at runtime #35642

pull sipa wants to merge 11 commits into bitcoin:master from sipa:202607_headerssync_cpp changing 18 files +561 −423
  1. sipa commented at 3:06 PM on July 2, 2026: member

    This migrates the headers presync parameters (commitment_period and redownload_buffer_size) from being precomputed chainparams to being computed at runtime during startup. This has as advantages:

    • Less work for maintainers that need to occasionally run the current contrib/devtools/headerssync-params.py script.
    • Less room for mistakes in that process.
    • Automatically adapts based on time (as opposed to needing to predict the lifetime of the software).

    To achieve this, the parameter search algorithm is ported to C++, and significant optimizations are applied to it step-by-step. The combination brings the runtime from minutes to ~3 ms. As an additional advantage, the result is now actually always optimal (within what can be determined using machine precision); the old code relied on (very reasonable) heuristics to guide the search.

    To make the changes reviewable without understanding the details of the algorithmic and mathematical optimizations, the PR starts with a few refactors to bring the existing Python code in a state where it can be tested. A set of test vectors is then introduced (in Python), and those vectors are ported to, and remain valid, in all the further C++ versions too.

    Disclaimer: this was written with significant assistance from Claude Opus 4.8 and Fable 5, which came up with the C++ port, the algorithmic and mathematical optimizations, and their implementation and testing. The code comments and commit messages are almost entirely written by me for clarity, but also to convince myself I understood all the changes.

  2. DrahtBot commented at 3:07 PM on July 2, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35642.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35351 (net: Disallow invalid HeadersSyncState due to lagging clock by hodlinator)
    • #35302 (Silent Payments: Sending (take 2) by Eunovo)
    • #35301 (Silent Payments: Implement bip352 (take 2) by Eunovo)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. sipa force-pushed on Jul 2, 2026
  4. DrahtBot added the label CI failed on Jul 2, 2026
  5. sipa force-pushed on Jul 2, 2026
  6. in src/net_processing.cpp:2040 in cafcdc385a outdated
    2031 | @@ -2029,12 +2032,25 @@ std::unique_ptr<PeerManager> PeerManager::make(CConnman& connman, AddrMan& addrm
    2032 |      return std::make_unique<PeerManagerImpl>(connman, addrman, banman, chainman, pool, warnings, opts);
    2033 |  }
    2034 |  
    2035 | +//! Compute the headers sync DoS-protection parameters for the given chain, using its age (genesis
    2036 | +//! to now) and the height at which its minimum chain work was taken, and log the result.
    2037 | +static HeadersSyncParams MakeHeadersSyncParams(const CChainParams& chainparams)
    2038 | +{
    2039 | +    const auto genesis_time{NodeSeconds{std::chrono::seconds{chainparams.GenesisBlock().nTime}}};
    2040 | +    const auto timespan{std::chrono::duration_cast<std::chrono::seconds>(NodeClock::now() - genesis_time)};
    


    ajtowns commented at 3:51 PM on July 2, 2026:

    Is there any anonymity breaking via NodeClock::now() influencing these parameters? (I think maybe not, but might be worth quantizing or similar to guarantee it?)


    sipa commented at 4:18 PM on July 2, 2026:

    Good point, I had not considered that.

    The choice of the commitment period should not be observable (besides a very minor time leak perhaps). However, the redownload buffer is observable in that an attacker can probably disconnect mid-redownload, and see how much you accepted afterwards.

    I'll run some numbers to see how quickly these parameters change and suggest some quantization or noise.


    sipa commented at 5:00 PM on July 2, 2026:

    It looks like the output parameters only change roughly every 22 days. I don't think quantizing further gains us much, especially as the attack only applies during IBD, so peers generally already know you just started.

  7. sipa force-pushed on Jul 2, 2026
  8. sipa force-pushed on Jul 2, 2026
  9. DrahtBot commented at 2:33 PM on July 9, 2026: contributor

    Could turn into draft while CI is red?

  10. contrib: reduce headerssync-params.py output to just the parameters
    In future commits, the logic in the headerssync-params.py tool will be
    made significantly faster, and integrated into the Bitcoin Core node
    software itself and run at runtime.
    
    To prepare for that, remove all output from the tool beyond the actual
    computation result:
    * Drop the printing of the search parameters being explored, because
      the search will become fast enough to not need progress output, and
      because it interferes with tests.
    * Drop the final memory configuration to maximally encapsulate the
      computation logic.
    * Drop memory_usage's now-unused second and third return values (the
      per-scenario breakdowns that only the removed output consumed); it
      now returns just the peak memory usage.
    cc958602f7
  11. contrib: drop the RANDOMIZE_OFFSET option from headerssync-params.py
    RANDOMIZE_OFFSET was a configurable option, introduced in the early
    stages of the headers-presync work to investigate whether randomizing
    the commitment offset was worth it. As that turned out to be the case,
    the actual implementation ended up using it all the time.
    
    Thus, remove the configurable and treat it as always True, making the
    logic slightly simpler.
    357e03be97
  12. contrib: parameterize headerssync-params.py computation
    This moves the computationally expensive part of the optimization
    search into a function that only depends on 3 quantities:
    - The maximum number of headers a chain can have.
    - The number of known valid headers in the main chain.
    - The (fractional) number of headers an attacker is allowed to insert
      into a victim's database per attack.
    
    The goal is to make it possible to introduce tests in later commits
    that cover the hard part, without hardcoding config parameters that
    would invalidate the tests if changed.
    bac347a2b7
  13. contrib: break headerssync-params ties deterministically
    It is possible to have multiple adjacent period values for which the
    peak memory usage is identical. The current randomized search will find
    a random one in this case, which is not great for testing.
    
    Add a postprocessing step that finds the minimum period value for the
    optimal found peak memory usage value, so the result is deterministic.
    
    Also make the attack_rate early-exit more robust: rather than comparing
    a single batch's contribution against the accumulated rate, compare a
    bound on the sum of all remaining batches. This gives a stronger
    guarantee that the reported solution is actually the optimal one, and
    it also turns out to be needed for the results to stay identical across
    the optimizations in later commits.
    89610da63f
  14. contrib: drop the ASSUME_CONVEX option from headerssync-params.py
    ASSUME_CONVEX assumes the period-to-memory-usage mapping (at the optimal buffer
    size for each period) is convex, which lets the search discard whole ranges of
    periods at once. It defaulted to True and was only ever run that way; the False
    path existed solely as a slower fallback with a stronger guarantee.
    
    Drop the option and always assume convexity, simplifying the search. A later
    commit replaces this random search entirely with a solver that finds the true
    optimum directly, without any convexity assumption, so the stronger-guarantee
    fallback is not needed in the meantime.
    6b7cc893b4
  15. contrib: add test vectors to headerssync-params.py
    This adds a collection of 50 randomly generated test vectors to the
    tool. This is in preparation for converting the code to C++ and
    applying significant optimization, while remaining confident that the
    behavior does not change.
    
    The test vectors are chosen to be inputs that are close to ones where
    the optimal output changes, to test the accuracy of the calculation.
    
    Run with pypy3, which evaluates the search far faster than CPython:
    
        pypy3 contrib/devtools/headerssync-params.py --selftest
    f2c000e46a
  16. headerssync: port the sync-parameter optimizer to C++
    This adds a C++ implementation of the headerssync-params.py tool's
    parameter search logic to headerssync.cpp directly, together with its
    unit test vectors. The tool's commentary explaining the goals of the
    search is copied along with it.
    
    This is rather inefficient for now, but will be optimized in further
    commits. Because of that, the unit test only checks one randomly chosen
    vector per run for now; a later commit makes the computation fast
    enough to check all of them on every run.
    f75ad4fa1b
  17. headerssync: compute attack_rate in closed form
    The attack rate averages the not-yet-detected probability over all
    {period} randomized commitment offsets. Doing so with an explicit loop
    over the alignments costs O(period) per processed batch. This can be
    computed much more efficiently however, by counting how many choices
    for the alignment have floor(forged_headers / period) commitments, how
    many have ceil(forged_headers / period), and weighting both
    accordingly. This results in just an O(1) cost per batch.
    41f593a613
  18. headerssync: drop find_bufsize optimization (preparation)
    The find_bufsize algorithm takes two arguments that help its search: a
    minimum bufsize to consider, and an upper bound on the amount of
    memory.
    
    After the changes in the next commit, these will not be needed anymore.
    Get rid of them beforehand here.
    7cfd62cf3f
  19. sipa force-pushed on Jul 9, 2026
  20. headerssync: replace randomized search with better algorithm
    Replace the randomized search over all candidate periods with a faster
    deterministic algorithm:
    
    Step 1: Solve the continuous relaxation of the problem, where period and
            bufsize can be arbitrary real numbers, and round the resulting
            period to an integer. This can be done efficiently using
            Newton-Raphson iterations.
    Step 2: Compute the bufsize and peak memory usage corresponding to that
            rounded period.
    Step 3: Scan the remaining integer periods whose peak memory usage could
            still be at or below the peak memory usage reached so far.
    
    The continuous optimum in Step 1 is only used as a starting point: a bad
    guess makes Step 3 scan more candidate periods, but cannot affect the
    result.
    
    This also makes the computation fast enough that the unit test can check
    all test vectors on every run again, instead of one randomly chosen one.
    
    The new algorithm's continuous relaxation relies on max_headers being
    larger than 2 * minchainwork_headers, which any consistent clock
    satisfies by a wide margin. Assume() this in the optimizer, and clamp
    in ComputeHeadersSyncParams, so that mocked or badly wrong clocks
    cannot produce degenerate inputs (which could otherwise fail to
    converge).
    d326057492
  21. net_processing: compute headers sync parameters at runtime
    So far, the headers sync parameters are committed to the repository,
    as part of the chain parameters. This requires maintainers to
    occasionally recompute them, based on a prediction of how long the
    release is likely to be used.
    
    With the code for the optimization algorithm ported to C++ and
    integrated in headerssync.cpp, it becomes possible to instead do this
    computation at runtime. This is less work, with fewer ways for
    maintainers and reviewers to get it wrong, and automatically adjusts
    based on clock time.
    
    Tests that construct many PeerManager instances can avoid recomputing
    the parameters each time by passing a precomputed value through
    PeerManager::Options. The fuzz targets that construct one per iteration
    pass a fixed constant, which both avoids the recomputation and makes
    their behavior independent of the (mocked) clock at construction time.
    bae2fd5ab0
  22. sipa force-pushed on Jul 9, 2026
Labels

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-07-09 19:50 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me