headerssync: Preempt unrealistic unit test behavior #32579

pull hodlinator wants to merge 9 commits into bitcoin:master from hodlinator:2025/05/headerssync_params changing 11 files +285 −137
  1. hodlinator commented at 1:48 pm on May 21, 2025: contributor

    Background

    As part of the release process we often run contrib/devtools/headerssync-params.py and increase the values of the constants HEADER_COMMITMENT_PERIOD and REDOWNLOAD_BUFFER_SIZE in src/headerssync.cpp as per doc/release-process.md (example: 11a2d3a63e90cdc1920ede3c67d52a9c72860e6b). This helps fine tune the memory consumption per HeadersSyncState-instance in the face of malicious peers.

    (The REDOWNLOAD_BUFFER_SIZE/HEADER_COMMITMENT_PERIOD ratio determines how many Headers Sync commitment bits must match between PRESYNC & REDOWNLOAD phases before we start permanently storing headers from a peer. For more details see comments src/headerssync.h and contrib/devtools/headerssync-params.py).

    Problem: Not feeding back headers until completing sync

    In the next (v30) or following release, it is very likely that REDOWNLOAD_BUFFER_SIZE (14827 as of v29) will exceed the target_blocks constant used to control the length of chains generated for testing Headers Sync (15000, headers_sync_chainwork_tests.cpp).

    Date at time of calculation: 2025-05-21 Block height: 897'662 Avg block time: ~9.8min (https://mempool.space). Added 6 months to TIME constant in headerssync-params.py: datetime(2028, 4, 6) Increased block height in headerssync-params.py (MINCHAINWORK_HEADERS) to: 920'000 Block height difference: 22'338 Days: 22'338 * 9.8min / (60min * 24h) = ~152 Estimated date for block height 920'000: 2025-10-20 REDOWNLOAD_BUFFER_SIZE value calculated by headerssync-params.py: 15002

    The HeadersSyncState::m_redownloaded_headers-buffer would not reach the REDOWNLOAD_BUFFER_SIZE-threshold during those unit tests. As a consequence HeadersSyncState::PopHeadersReadyForAcceptance() will not start feeding back headers until the PoW threshold has been met. While this will not cause the unit test to start failing on master, it means it would go from testing behavior that resembles mainnet (way more than 15'000 blocks to reach the PoW limit), to behavior that is not possible/expected there.

    Solution

    Preempt the tests from only testing this unrealistic condition of completing Headers Sync before reaching REDOWNLOAD_BUFFER_SIZE by making tests able to define their own values through the new HeadersSyncParams instead of having them hard-coded for all chains & tests.

    Commits

    • First commit adds checks for the behavior around the REDOWNLOAD_BUFFER_SIZE threshold to illustrate the need for the rest of the PR (see commit message).
    • Following 6 commits refactor and improve the unit tests.
    • The main change: we extract the section from headerssync.cpp containing the constants to kernel/chainparams.cpp, making HeadersSyncState less hard-coded.
    • Test improvements due to prior commit making more precise checks possible.
  2. DrahtBot commented at 1:48 pm on May 21, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32579.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK marcofleon, danielabrozzoni, davidgumberg
    Concept ACK TheCharlatan, jonatack
    Stale ACK l0rinc

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #32740 (refactor: Header sync optimisations & simplifications by danielabrozzoni)
    • #31974 (Drop testnet3 by Sjors)
    • #28122 (Silent Payments: Implement BIP352 by josibake)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. in src/headerssync-params.h:16 in 0c1978b70d outdated
    11+//! Store one header commitment per HEADER_COMMITMENT_PERIOD blocks.
    12+constexpr size_t HEADER_COMMITMENT_PERIOD{624};
    13+
    14+//! Only feed headers to validation once this many headers on top have been
    15+//! received and validated against commitments.
    16+constexpr size_t REDOWNLOAD_BUFFER_SIZE{14827}; // 14827/624 = ~23.8 commitments
    


    l0rinc commented at 2:13 pm on May 21, 2025:
    I know this is just a move, but instead of just bare magic number + magic comment (which also has to be manually updated every time), could we have the something like : Generated by headerssync-params.py (on 2025-03-04)

    hodlinator commented at 12:35 pm on May 30, 2025:

    I like the idea of including the generation date, and was including changes to the Python generator file in an unpublished version of this PR, but decided to keep that separate. Added to that separate WIP branch.

    The magic numbers are not updated manually, they are included in the calculated output: https://github.com/bitcoin/bitcoin/blob/11a2d3a63e90cdc1920ede3c67d52a9c72860e6b/contrib/devtools/headerssync-params.py#L347-L348


    hodlinator commented at 1:43 pm on June 4, 2025:
    Added generation date in latest push as it was making modifications to the Python script anyway.
  4. hodlinator force-pushed on May 21, 2025
  5. DrahtBot added the label CI failed on May 21, 2025
  6. DrahtBot commented at 2:15 pm on May 21, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/runs/42636718296 LLM reason (✨ experimental): The CI failure is due to a missing include guard in the src/headerssync-params.h file, as reported by the lint checks.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  7. DrahtBot removed the label CI failed on May 21, 2025
  8. in src/test/headers_sync_chainwork_tests.cpp:66 in 34e1513efe outdated
    62@@ -63,15 +63,19 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
    63 //    updates to the REDOWNLOAD phase successfully.
    64 // 2. Then we deliver the second set of headers and verify that they fail
    65 //    processing (presumably due to commitments not matching).
    66-// 3. Finally, we verify that repeating with the first set of headers in both
    67-//    phases is successful.
    68+static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
    


    l0rinc commented at 10:02 am on May 27, 2025:

    34e1513efeb193a46d37f184b9ab41bcda974afa: I have inlined the methods and verified that the remaining changes are only comments, reused-and-cleared pointers and vectors 👍

      0diff --git a/src/test/headers_sync_chainwork_tests.cpp b/src/test/headers_sync_chainwork_tests.cpp
      1index 054853bbbf..89bb6682d1 100644
      2--- a/src/test/headers_sync_chainwork_tests.cpp
      3+++ b/src/test/headers_sync_chainwork_tests.cpp
      4@@ -13,29 +13,32 @@
      5 
      6 #include <boost/test/unit_test.hpp>
      7 
      8-struct HeadersGeneratorSetup : public RegTestingSetup {
      9+struct HeadersGeneratorSetup : RegTestingSetup
     10+{
     11     /** Search for a nonce to meet (regtest) proof of work */
     12-    void FindProofOfWork(CBlockHeader& starting_header);
     13+    static void FindProofOfWork(CBlockHeader& starting_header);
     14     /**
     15      * Generate headers in a chain that build off a given starting hash, using
     16      * the given nVersion, advancing time by 1 second from the starting
     17      * prev_time, and with a fixed merkle root hash.
     18      */
     19-    void GenerateHeaders(std::vector<CBlockHeader>& headers, size_t count,
     20-            const uint256& starting_hash, const int nVersion, int prev_time,
     21-            const uint256& merkle_root, const uint32_t nBits);
     22+    static void GenerateHeaders(
     23+        std::vector<CBlockHeader>& headers, size_t count,
     24+        const uint256& starting_hash, int nVersion, int prev_time,
     25+        const uint256& merkle_root, uint32_t nBits);
     26 };
     27 
     28 void HeadersGeneratorSetup::FindProofOfWork(CBlockHeader& starting_header)
     29 {
     30     while (!CheckProofOfWork(starting_header.GetHash(), starting_header.nBits, Params().GetConsensus())) {
     31-        ++(starting_header.nNonce);
     32+        ++starting_header.nNonce;
     33     }
     34 }
     35 
     36-void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     37-        size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     38-        const uint256& merkle_root, const uint32_t nBits)
     39+void HeadersGeneratorSetup::GenerateHeaders(
     40+    std::vector<CBlockHeader>& headers,
     41+    const size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     42+    const uint256& merkle_root, const uint32_t nBits)
     43 {
     44     uint256 prev_hash = starting_hash;
     45 
     46@@ -45,14 +48,13 @@ void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     47         next_header.nVersion = nVersion;
     48         next_header.hashPrevBlock = prev_hash;
     49         next_header.hashMerkleRoot = merkle_root;
     50-        next_header.nTime = prev_time+1;
     51+        next_header.nTime = prev_time + 1;
     52         next_header.nBits = nBits;
     53 
     54         FindProofOfWork(next_header);
     55         prev_hash = next_header.GetHash();
     56         prev_time = next_header.nTime;
     57     }
     58-    return;
     59 }
     60 
     61 BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
     62@@ -63,31 +65,41 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
     63 //    updates to the REDOWNLOAD phase successfully.
     64 // 2. Then we deliver the second set of headers and verify that they fail
     65 //    processing (presumably due to commitments not matching).
     66-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
     67+static void SneakyRedownload(
     68+    const std::vector<CBlockHeader>& first_chain,
     69+    const std::vector<CBlockHeader>& second_chain,
     70+    const CBlockIndex* chain_start,
     71+    const arith_uint256& chain_work);
     72 // 3. Verify that repeating with the first set of headers in both phases is
     73 //    successful.
     74-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
     75+static void HappyPath(
     76+    const std::vector<CBlockHeader>& first_chain,
     77+    const CBlockIndex* chain_start,
     78+    const arith_uint256& chain_work);
     79 // 4. Finally, repeat the second set of headers in both phases to demonstrate
     80 //    behavior when the chain a peer provides has too little work.
     81-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
     82+static void TooLittleWork(
     83+    const std::vector<CBlockHeader>& second_chain,
     84+    const CBlockIndex* chain_start,
     85+    const arith_uint256& chain_work);
     86 
     87 BOOST_AUTO_TEST_CASE(headers_sync_state)
     88 {
     89     std::vector<CBlockHeader> first_chain;
     90     std::vector<CBlockHeader> second_chain;
     91 
     92-    const int target_blocks = 15000;
     93-    arith_uint256 chain_work = target_blocks*2;
     94+    constexpr int target_blocks = 15000;
     95+    const arith_uint256 chain_work = target_blocks * 2;
     96 
     97     // Generate headers for two different chains (using differing merkle roots
     98     // to ensure the headers are different).
     99-    GenerateHeaders(first_chain, target_blocks-1, Params().GenesisBlock().GetHash(),
    100-            Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    101-            ArithToUint256(0), Params().GenesisBlock().nBits);
    102+    GenerateHeaders(first_chain, target_blocks - 1, Params().GenesisBlock().GetHash(),
    103+                    Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    104+                    ArithToUint256(0), Params().GenesisBlock().nBits);
    105 
    106-    GenerateHeaders(second_chain, target_blocks-2, Params().GenesisBlock().GetHash(),
    107-            Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    108-            ArithToUint256(1), Params().GenesisBlock().nBits);
    109+    GenerateHeaders(second_chain, target_blocks - 2, Params().GenesisBlock().GetHash(),
    110+                    Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    111+                    ArithToUint256(1), Params().GenesisBlock().nBits);
    112 
    113     const CBlockIndex* chain_start = WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(Params().GenesisBlock().GetHash()));
    114 
    115@@ -96,7 +108,10 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    116     TooLittleWork(second_chain, chain_start, chain_work);
    117 }
    118 
    119-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work)
    120+static void SneakyRedownload(
    121+    const std::vector<CBlockHeader>& first_chain,
    122+    const std::vector<CBlockHeader>& second_chain,
    123+    const CBlockIndex* chain_start, const arith_uint256& chain_work)
    124 {
    125     std::vector<CBlockHeader> headers_batch;
    126 
    127@@ -109,56 +124,59 @@ static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const
    128     // Pretend the first header is still "full", so we don't abort.
    129     auto result = hss.ProcessNextHeaders(headers_batch, true);
    130 
    131-    // This chain should look valid, and we should have met the proof-of-work
    132-    // requirement.
    133+    // This chain should look valid, and we should have met the proof-of-work requirement.
    134     BOOST_CHECK(result.success);
    135     BOOST_CHECK(result.request_more);
    136-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    137+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    138 
    139     // Try to sneakily feed back the second chain.
    140     result = hss.ProcessNextHeaders(second_chain, true);
    141     BOOST_CHECK(!result.success); // foiled!
    142-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    143+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    144 }
    145 
    146-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work)
    147+static void HappyPath(
    148+    const std::vector<CBlockHeader>& first_chain,
    149+    const CBlockIndex* chain_start,
    150+    const arith_uint256& chain_work)
    151 {
    152     // This time we feed the first chain twice.
    153     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, chain_work};
    154     (void)hss.ProcessNextHeaders(first_chain, true);
    155-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    156+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    157 
    158-    auto result = hss.ProcessNextHeaders(first_chain, true);
    159-    BOOST_CHECK(result.success);
    160-    BOOST_CHECK(!result.request_more);
    161+    const auto [pow_validated_headers, success, request_more] = hss.ProcessNextHeaders(first_chain, true);
    162+    BOOST_CHECK(success);
    163+    BOOST_CHECK(!request_more);
    164     // All headers should be ready for acceptance:
    165-    BOOST_CHECK(result.pow_validated_headers.size() == first_chain.size());
    166+    BOOST_CHECK_EQUAL(pow_validated_headers.size(), first_chain.size());
    167     // Nothing left for the sync logic to do:
    168-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    169+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    170 }
    171 
    172-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work)
    173+static void TooLittleWork(
    174+    const std::vector<CBlockHeader>& second_chain,
    175+    const CBlockIndex* chain_start,
    176+    const arith_uint256& chain_work)
    177 {
    178-    // Verify that just trying to process the second chain would not succeed
    179-    // (too little work).
    180+    // Verify that just trying to process the second chain would not succeed (too little work).
    181     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, chain_work};
    182     BOOST_CHECK(hss.GetState() == HeadersSyncState::State::PRESYNC);
    183-     // Pretend just the first message is "full", so we don't abort.
    184+    // Pretend just the first message is "full", so we don't abort.
    185     (void)hss.ProcessNextHeaders({second_chain.front()}, true);
    186     BOOST_CHECK(hss.GetState() == HeadersSyncState::State::PRESYNC);
    187 
    188     std::vector<CBlockHeader> headers_batch;
    189     headers_batch.insert(headers_batch.end(), std::next(second_chain.begin(), 1), second_chain.end());
    190     // Tell the sync logic that the headers message was not full, implying no
    191-    // more headers can be requested. For a low-work-chain, this should causes
    192+    // more headers can be requested. For a low-work-chain, this should cause
    193     // the sync to end with no headers for acceptance.
    194-    auto result = hss.ProcessNextHeaders(headers_batch, false);
    195+    const auto [pow_validated_headers, success, request_more] = hss.ProcessNextHeaders(headers_batch, false);
    196     BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    197-    BOOST_CHECK(result.pow_validated_headers.empty());
    198-    BOOST_CHECK(!result.request_more);
    199-    // Nevertheless, no validation errors should have been detected with the
    200-    // chain:
    201-    BOOST_CHECK(result.success);
    202+    BOOST_CHECK(pow_validated_headers.empty());
    203+    BOOST_CHECK(!request_more);
    204+    // Nevertheless, no validation errors should have been detected with the chain:
    205+    BOOST_CHECK(success);
    206 }
    207 
    208 BOOST_AUTO_TEST_SUITE_END()
    

    I see that some of these were done in next commits, but I’d prefer doing either moves or refactors or adding new checks in the commits to simplify review


    hodlinator commented at 11:13 am on June 2, 2025:
    (Incorporated most of this, breaking out commits, expanding function prototypes into multiple lines, even removing the HeadersGeneratorSetup type in favor of free functions).
  9. in src/test/headers_sync_chainwork_tests.cpp:124 in 13aa0b860e outdated
    147+    // Try to sneakily feed back the second chain during REDOWNLOAD.
    148     result = hss.ProcessNextHeaders(second_chain, true);
    149+    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    150     BOOST_CHECK(!result.success); // foiled!
    151-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    152+    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    


    l0rinc commented at 10:20 am on May 27, 2025:

    we’re reusing the previous result (which was brace initialized), what if we used braces to delimit the two independent code parts (+ a few other nits you might want to consider):

     0// Verify that just trying to process the second chain would not succeed (too little work).
     1HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
     2BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
     3// Pretend just the first message is "full", so we don't abort.
     4{
     5    const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin(), 1}, true)};
     6    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
     7    BOOST_CHECK(success);
     8    BOOST_CHECK(request_more);
     9    BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    10}
    11
    12// Tell the sync logic that the headers message was not full, implying no
    13// more headers can be requested. For a low-work-chain, this should cause
    14// the sync to end with no headers for acceptance.
    15{
    16    const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin() + 1, second_chain.end()}, false)};
    17    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    18    BOOST_CHECK(!request_more);
    19    BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    20    // Nevertheless, no validation errors should have been detected with the chain:
    21    BOOST_CHECK(success);
    22}
    

    hodlinator commented at 8:22 pm on May 30, 2025:
    Took the scoping idea, still not sure about the structured bindings.

    l0rinc commented at 8:39 am on June 2, 2025:

    not sure about the structured bindings

    That’s what helped me in noticing that we weren’t checking request_more


    hodlinator commented at 11:17 am on June 2, 2025:

    Is there anything to catch wrong order?

    0const auto [pow_validated_headers, success, request_more] = ...
    

    vs

    0const auto [pow_validated_headers, request_more, success] = ...
    

    (There’s clang-tidy logic to catch /*paramname=*/true mismatches for function calls).


    l0rinc commented at 2:13 pm on June 2, 2025:
    I don’t think there could/should be, you could name them as const auto [x, y, z]{hss.ProcessNextHeaders(std::span{second_chain.begin() + 1, second_chain.end()}, false)}; as well

    hodlinator commented at 7:27 pm on June 2, 2025:
    Yes, but when breaking apart a struct and two of the new variable names match struct field names with identical types, it would be nice to have some kind of guard rail.
  10. in src/test/headers_sync_chainwork_tests.cpp:123 in 13aa0b860e outdated
    145 
    146-    // Try to sneakily feed back the second chain.
    147+    // Try to sneakily feed back the second chain during REDOWNLOAD.
    148     result = hss.ProcessNextHeaders(second_chain, true);
    149+    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    150     BOOST_CHECK(!result.success); // foiled!
    


    l0rinc commented at 10:25 am on May 27, 2025:

    This seems to be the only place where request_more isn’t validated.

    nit: what’s the meaning of “foiled” here?


    hodlinator commented at 12:40 pm on May 30, 2025:

    If we didn’t succeed, then request_more shouldn’t matter, but agree it’s best to have it be false (just like pow_validated_headers should be empty). It would be better if ProcessNextHeaders returned something like std::optional<ProcessingResult> instead of having a bool field to indicate success, but this PR already feels large enough.

    “foiled” means the attack attempt was detected and therefore failed. Elaborated the comment a bit in next push.


    hodlinator commented at 8:45 pm on May 30, 2025:

    Hm.. my std::optional<ProcessingResult> suggestion would not work not clarify things here:

    https://github.com/bitcoin/bitcoin/blob/a7b9cc4a0af0dbe4031ec5848d403f61dc9c3356/src/test/headers_sync_chainwork_tests.cpp#L127-L136

    Success seems to mean that PRESYNC- and REDOWNLOAD-headers matched, even if the chain is too short.

  11. in src/arith_uint256.h:34 in 13aa0b860e outdated
    32     uint32_t pn[WIDTH];
    33-public:
    34 
    35-    base_uint()
    36+public:
    37+    constexpr base_uint()
    


    l0rinc commented at 10:32 am on May 27, 2025:
    should we make any of these explicit?

    hodlinator commented at 12:23 pm on May 30, 2025:
    One could argue at least the uint64_t-taking ctor here and in arith_uint256 should be explicit, but I’d rather not have to change src/pow.cpp in this PR.
  12. in src/test/headers_sync_chainwork_tests.cpp:16 in 13aa0b860e outdated
    12@@ -13,6 +13,9 @@
    13 
    14 #include <boost/test/unit_test.hpp>
    15 
    16+constexpr int TARGET_BLOCKS{15000};
    


    l0rinc commented at 10:32 am on May 27, 2025:

    size_t might me more fitting here since GenerateHeaders accepts that instead of an int:

    0constexpr size_t TARGET_BLOCKS{15'000};
    
  13. in src/test/headers_sync_chainwork_tests.cpp:68 in 13aa0b860e outdated
    64@@ -63,99 +65,107 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
    65 //    updates to the REDOWNLOAD phase successfully.
    66 // 2. Then we deliver the second set of headers and verify that they fail
    67 //    processing (presumably due to commitments not matching).
    68-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
    69+static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start);
    


    l0rinc commented at 10:39 am on May 27, 2025:
    Could we use const std::span<const CBlockHeader> in these helpers as well? And maybe split these long lines.

    hodlinator commented at 12:42 pm on May 30, 2025:
    Agree on the split but what would be the point of changing the type?

    l0rinc commented at 8:39 am on June 2, 2025:
    • they’re simpler/less specific
    • they’re shorter (matters for these long lines)
    • you’re already changing them in other places

    i.e. what’s the point of keeping them as vectors when we don’t actually need so much power?


    hodlinator commented at 11:21 am on June 2, 2025:
    Using const vector& communicates that we are sending in the entire chain, not some subset. I like the specificity. const vectors feel more neutered than spans IMO.

    l0rinc commented at 2:17 pm on June 2, 2025:
    When we’re only iterating inside, it makes sense to have a fine-grained, segregated interface and only define the narrowest functionality which supports the operations we actually need inside. It announces to the reader that we shouldn’t expect any extra complexity, only the ones that the interface implements. But if you think vector does it better, I’m also fine with that. For me span is not about a subset of teh data but a subset of the functionality - just a narrowed view to rule out certain operations that we definitely won’t need.
  14. in src/test/headers_sync_chainwork_tests.cpp:136 in 13aa0b860e outdated
    163+    auto result{hss.ProcessNextHeaders(first_chain, true)};
    164+    // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    165+    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    166+    BOOST_CHECK(result.success);
    167+    BOOST_CHECK(result.request_more);
    168+    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    


    l0rinc commented at 10:40 am on May 27, 2025:
    👍 for comparing size instead of empty - the failure message would likely be more meaningful
  15. in src/test/headers_sync_chainwork_tests.cpp:73 in 13aa0b860e outdated
    43@@ -41,7 +44,7 @@ void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
    44 
    45     while (headers.size() < count) {
    46         headers.emplace_back();
    47-        CBlockHeader& next_header = headers.back();;
    48+        CBlockHeader& next_header = headers.back();
    


    l0rinc commented at 10:43 am on May 27, 2025:
    I think we could do these in the previous refactor commit instead - to separate low risk changes from ones where we have to may more attention, while keeping the first one about move-only changes + minor ones. Or do the first commit as strictly move-only and add a second one with these tiny refactors (I’d like the latter more).
  16. in src/arith_uint256.h:1 in 13aa0b860e outdated


    l0rinc commented at 10:44 am on May 27, 2025:

    full patch for 13aa0b860e9692715d96b13acacb3f83a537b905

      0diff --git a/src/arith_uint256.h b/src/arith_uint256.h
      1index e021947381..b07e8e9f41 100644
      2--- a/src/arith_uint256.h
      3+++ b/src/arith_uint256.h
      4@@ -31,7 +31,7 @@ protected:
      5     uint32_t pn[WIDTH];
      6 
      7 public:
      8-    constexpr base_uint()
      9+    explicit constexpr base_uint()
     10     {
     11         for (int i = 0; i < WIDTH; i++)
     12             pn[i] = 0;
     13@@ -243,8 +243,8 @@ class arith_uint256 : public base_uint<256>
     14 {
     15 public:
     16     constexpr arith_uint256() = default;
     17-    constexpr arith_uint256(const base_uint<256>& b) : base_uint<256>(b) {}
     18-    constexpr arith_uint256(uint64_t b) : base_uint<256>(b) {}
     19+    constexpr arith_uint256(const base_uint& b) : base_uint(b) {}
     20+    constexpr arith_uint256(const uint64_t b) : base_uint(b) {}
     21 
     22     /**
     23      * The "compact" format is a representation of a whole
     24diff --git a/src/test/headers_sync_chainwork_tests.cpp b/src/test/headers_sync_chainwork_tests.cpp
     25index b696285d0e..264959f8c3 100644
     26--- a/src/test/headers_sync_chainwork_tests.cpp
     27+++ b/src/test/headers_sync_chainwork_tests.cpp
     28@@ -13,32 +13,35 @@
     29 
     30 #include <boost/test/unit_test.hpp>
     31 
     32-constexpr int TARGET_BLOCKS{15000};
     33+constexpr size_t TARGET_BLOCKS{15'000};
     34 constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
     35 
     36-struct HeadersGeneratorSetup : public RegTestingSetup {
     37+struct HeadersGeneratorSetup : RegTestingSetup
     38+{
     39     /** Search for a nonce to meet (regtest) proof of work */
     40-    void FindProofOfWork(CBlockHeader& starting_header);
     41+    static void FindProofOfWork(CBlockHeader& starting_header);
     42     /**
     43      * Generate headers in a chain that build off a given starting hash, using
     44      * the given nVersion, advancing time by 1 second from the starting
     45      * prev_time, and with a fixed merkle root hash.
     46      */
     47-    void GenerateHeaders(std::vector<CBlockHeader>& headers, size_t count,
     48-            const uint256& starting_hash, const int nVersion, int prev_time,
     49-            const uint256& merkle_root, const uint32_t nBits);
     50+    static void GenerateHeaders(
     51+        std::vector<CBlockHeader>& headers, size_t count,
     52+        const uint256& starting_hash, int nVersion, int prev_time,
     53+        const uint256& merkle_root, uint32_t nBits);
     54 };
     55 
     56 void HeadersGeneratorSetup::FindProofOfWork(CBlockHeader& starting_header)
     57 {
     58     while (!CheckProofOfWork(starting_header.GetHash(), starting_header.nBits, Params().GetConsensus())) {
     59-        ++(starting_header.nNonce);
     60+        ++starting_header.nNonce;
     61     }
     62 }
     63 
     64-void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     65-        size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     66-        const uint256& merkle_root, const uint32_t nBits)
     67+void HeadersGeneratorSetup::GenerateHeaders(
     68+    std::vector<CBlockHeader>& headers,
     69+    const size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     70+    const uint256& merkle_root, const uint32_t nBits)
     71 {
     72     uint256 prev_hash = starting_hash;
     73 
     74@@ -48,7 +51,7 @@ void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     75         next_header.nVersion = nVersion;
     76         next_header.hashPrevBlock = prev_hash;
     77         next_header.hashMerkleRoot = merkle_root;
     78-        next_header.nTime = prev_time+1;
     79+        next_header.nTime = prev_time + 1;
     80         next_header.nBits = nBits;
     81 
     82         FindProofOfWork(next_header);
     83@@ -65,13 +68,20 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
     84 //    updates to the REDOWNLOAD phase successfully.
     85 // 2. Then we deliver the second set of headers and verify that they fail
     86 //    processing (presumably due to commitments not matching).
     87-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start);
     88+static void SneakyRedownload(
     89+    const std::span<const CBlockHeader> first_chain,
     90+    const std::span<const CBlockHeader> second_chain,
     91+    const CBlockIndex* chain_start);
     92 // 3. Verify that repeating with the first set of headers in both phases is
     93 //    successful.
     94-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start);
     95+static void HappyPath(
     96+    const std::span<const CBlockHeader> first_chain,
     97+    const CBlockIndex* chain_start);
     98 // 4. Finally, repeat the second set of headers in both phases to demonstrate
     99 //    behavior when the chain a peer provides has too little work.
    100-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start);
    101+static void TooLittleWork(
    102+    const std::span<const CBlockHeader> second_chain,
    103+    const CBlockIndex* chain_start);
    104 
    105 BOOST_AUTO_TEST_CASE(headers_sync_state)
    106 {
    107@@ -94,81 +104,101 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    108     TooLittleWork(second_chain, chain_start);
    109 }
    110 
    111-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start)
    112+static void SneakyRedownload(
    113+    const std::span<const CBlockHeader> first_chain,
    114+    const std::span<const CBlockHeader> second_chain,
    115+    const CBlockIndex* chain_start)
    116 {
    117     // Feed the first chain to HeadersSyncState, by delivering 1 header
    118     // initially and then the rest.
    119     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
    120-    auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    121-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    122-    BOOST_CHECK(result.success);
    123-    BOOST_CHECK(result.request_more);
    124-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    125-    BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain.front().GetHash());
    126+    {
    127+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({first_chain.begin(), 1}, true)};
    128+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    129+        BOOST_CHECK(success);
    130+        BOOST_CHECK(request_more);
    131+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    132+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain.front().GetHash());
    133+    }
    134 
    135     // Pretend the first header is still "full", so we don't abort.
    136-    result = hss.ProcessNextHeaders(std::span{first_chain.begin() + 1, first_chain.end()}, true);
    137-    // This chain should look valid, and we should have met the proof-of-work
    138-    // requirement during PRESYNC and transitioned to REDOWNLOAD.
    139-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    140-    BOOST_CHECK(result.success);
    141-    BOOST_CHECK(result.request_more);
    142-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    143-    // The locator should reset to genesis.
    144-    BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    145+    {
    146+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({first_chain.begin() + 1, first_chain.end()}, true)};
    147+        // This chain should look valid, and we should have met the proof-of-work
    148+        // requirement during PRESYNC and transitioned to REDOWNLOAD.
    149+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    150+        BOOST_CHECK(success);
    151+        BOOST_CHECK(request_more);
    152+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    153+        // The locator should reset to genesis.
    154+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    155+    }
    156 
    157     // Try to sneakily feed back the second chain during REDOWNLOAD.
    158-    result = hss.ProcessNextHeaders(second_chain, true);
    159-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    160-    BOOST_CHECK(!result.success); // foiled!
    161-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    162+    {
    163+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders(second_chain, true)};
    164+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    165+        BOOST_CHECK(!success); // foiled!
    166+        BOOST_CHECK(!request_more);
    167+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    168+    }
    169 }
    170 
    171-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start)
    172+static void HappyPath(
    173+    const std::span<const CBlockHeader> first_chain,
    174+    const CBlockIndex* chain_start)
    175 {
    176     // This time we feed the first chain twice.
    177     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
    178-    auto result{hss.ProcessNextHeaders(first_chain, true)};
    179-    // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    180-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    181-    BOOST_CHECK(result.success);
    182-    BOOST_CHECK(result.request_more);
    183-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    184-    // The locator should reset to genesis.
    185-    BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    186-
    187-    result = hss.ProcessNextHeaders(first_chain, true);
    188-    // Nothing left for the sync logic to do:
    189-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    190-    BOOST_CHECK(result.success);
    191-    BOOST_CHECK(!result.request_more);
    192-    // All headers should be ready for acceptance:
    193-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), first_chain.size());
    194+    {
    195+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders(first_chain, true)};
    196+        // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    197+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    198+        BOOST_CHECK(success);
    199+        BOOST_CHECK(request_more);
    200+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    201+        // The locator should reset to genesis.
    202+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    203+    }
    204+
    205+    {
    206+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders(first_chain, true)};
    207+        // Nothing left for the sync logic to do:
    208+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    209+        BOOST_CHECK(success);
    210+        BOOST_CHECK(!request_more);
    211+        // All headers should be ready for acceptance:
    212+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), first_chain.size());
    213+    }
    214 }
    215 
    216-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start)
    217+static void TooLittleWork(
    218+    const std::span<const CBlockHeader> second_chain,
    219+    const CBlockIndex* chain_start)
    220 {
    221-    // Verify that just trying to process the second chain would not succeed
    222-    // (too little work).
    223+    // Verify that just trying to process the second chain would not succeed (too little work).
    224     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
    225     BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    226     // Pretend just the first message is "full", so we don't abort.
    227-    auto result{hss.ProcessNextHeaders(std::span{second_chain.begin(), 1}, true)};
    228-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    229-    BOOST_CHECK(result.success);
    230-    BOOST_CHECK(result.request_more);
    231-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    232+    {
    233+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin(), 1}, true)};
    234+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    235+        BOOST_CHECK(success);
    236+        BOOST_CHECK(request_more);
    237+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    238+    }
    239 
    240     // Tell the sync logic that the headers message was not full, implying no
    241     // more headers can be requested. For a low-work-chain, this should cause
    242     // the sync to end with no headers for acceptance.
    243-    result = hss.ProcessNextHeaders(std::span{second_chain.begin() + 1, second_chain.end()}, false);
    244-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    245-    BOOST_CHECK(!result.request_more);
    246-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    247-    // Nevertheless, no validation errors should have been detected with the
    248-    // chain:
    249-    BOOST_CHECK(result.success);
    250+    {
    251+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin() + 1, second_chain.end()}, false)};
    252+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    253+        BOOST_CHECK(!request_more);
    254+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    255+        // Nevertheless, no validation errors should have been detected with the chain:
    256+        BOOST_CHECK(success);
    257+    }
    258 }
    259 
    260 BOOST_AUTO_TEST_SUITE_END()
    

    l0rinc commented at 10:49 am on May 27, 2025:
    For other headers we’ve used an underscore instead of a hyphen in the file name: src/headerssync_params.h

    hodlinator commented at 12:06 pm on May 30, 2025:
    What’s the point of making a constructor that takes no arguments explicit?

    hodlinator commented at 8:18 pm on May 30, 2025:
    Started renaming but then realized that we already have headerssync-params.py. It’s not obvious to me that underscores would be more correct for C++ files. We do have other C++ files with hyphens.

    l0rinc commented at 8:40 am on June 2, 2025:
    Based on the Sonarcloud recommendations I eagerly checked which other constructors can be explicit, but exploring your question in more detail it seems there’s barely any advantage in explicit default constructors.

    l0rinc commented at 8:40 am on June 2, 2025:

    It seems to be a mix indeed:

    0% find . -name "*.h" | grep '_' | wc -l
    1185
    

    vs

    0% find . -name "*.h" | grep '-' | wc -l
    139
    

    and most of those are from the recent ipc changes:

    0% find . -name "*.h" | grep '-' | grep -v ipc | wc -l            
    111
    

    it’s just a nit from my part, but standard lib also uses underscores mostly (e.g. string_view) and in many regexes the \w, i.e. a word character includes _ but usually not - (if I remember it correctly some bash implementations can also trip on -, but that’s likely not a problem here).


    hodlinator commented at 11:08 am on June 2, 2025:

    Ah, the Sonarcloud recommendation was actually for the uint64_t-ctor, similar to my response here: #32579 (review).

    Tested out making it explicit, modifying 13 additional files. It did uncover some implicit casts that went from int64_t Params::nPowTargetTimespan to arith_uint256… But the rest felt a bit forced. I think it is intentional and makes sense for u64s to implicitly cast to u256.


    hodlinator commented at 11:29 am on June 2, 2025:
    Yeah, seems src/ipc/ really went all in on -. Holding off on changing for now to see what others think.

    l0rinc commented at 2:18 pm on June 2, 2025:
    Thanks for checking!

    l0rinc commented at 9:23 am on June 20, 2025:
    typo in commit message of 42e746096b93b8026569817d8925750089d6daf4

    hodlinator commented at 8:22 am on June 23, 2025:
    Fixed.
  17. in doc/release-process.md:56 in e9f4f6a5d9 outdated
    52@@ -53,7 +53,7 @@ Release Process
    53     - Set `MINCHAINWORK_HEADERS` to the height used for the `nMinimumChainWork` calculation above.
    54     - Check that the other variables still look reasonable.
    55   - Run the script. It works fine in CPython, but PyPy is much faster (seconds instead of minutes): `pypy3 contrib/devtools/headerssync-params.py`.
    56-  - Paste the output defining `HEADER_COMMITMENT_PERIOD` and `REDOWNLOAD_BUFFER_SIZE` into the top of [`src/headerssync.cpp`](/src/headerssync.cpp).
    57+  - Paste the output defining `HEADER_COMMITMENT_PERIOD` and `REDOWNLOAD_BUFFER_SIZE` into [`src/headerssync-params.h`](/src/headerssync-params.h).
    


    l0rinc commented at 10:50 am on May 27, 2025:

    hodlinator commented at 12:59 pm on May 30, 2025:
    Thanks, missed that!
  18. in src/headerssync.cpp:14 in e9f4f6a5d9 outdated
    23-constexpr size_t REDOWNLOAD_BUFFER_SIZE{14827}; // 14827/624 = ~23.8 commitments
    24-
    25-// Our memory analysis assumes 48 bytes for a CompressedHeader (so we should
    26-// re-calculate parameters if we compress further)
    27+// Our memory analysis in headerssync-params.py assumes 48 bytes for a
    28+// CompressedHeader (we should re-calculate parameters if we compress further).
    


    l0rinc commented at 10:54 am on May 27, 2025:

    I find the new comment slightly more confusing - the point is that the cpp and py assumptions should align, right?

    0static_assert(sizeof(CompressedHeader) == 48); // align with headerssync-params.py
    

    hodlinator commented at 12:55 pm on May 30, 2025:

    The new comment is replacing the first (mention of python file) and last comments here: https://github.com/bitcoin/bitcoin/blob/11a2d3a63e90cdc1920ede3c67d52a9c72860e6b/src/headerssync.cpp#L12-L25

    Your comment would probably be enough but I’d rather keep some more context.

  19. in src/headerssync-params.h:11 in e9f4f6a5d9 outdated
     6+#define BITCOIN_HEADERSSYNC_PARAMS_H
     7+
     8+// The two constants below are computed using the simulation script in
     9+// contrib/devtools/headerssync-params.py.
    10+
    11+//! Store one header commitment per HEADER_COMMITMENT_PERIOD blocks.
    


    l0rinc commented at 10:56 am on May 27, 2025:

    nit: I find it a bit circular that the comment explaining a constant references the upcoming constant itself

    0//! Commitment period, in blocks
    

    hodlinator commented at 12:49 pm on May 30, 2025:
    Added to separate WIP branch modifying generator script.

    hodlinator commented at 1:44 pm on June 4, 2025:

    Latest push has:

    0struct HeadersSyncParams {
    1    //! Distance in blocks between header commitments.
    2    size_t commitment_period;
    3...
    

    in src/kernel/chainparams.h.

  20. l0rinc changes_requested
  21. l0rinc commented at 11:35 am on May 27, 2025: contributor

    Concept ACK – this PR replaces the imminently outdated hard-coded test target of 15'000 headers with a value derived from the live REDOWNLOAD_BUFFER_SIZE constant. Moving both header-sync tuning constants into a shared header, and making the unit test reference them, keeps the test exercising the ‘buffer-full’ path as the parameters grow each release.

    Recommendations:

    • split the series into more commits, separating low-risk refactors from ones that change behaviour: strict move-only; refactor / cleanup with no behavioural change; extra assertions; wiring the test to the new parameter header;
    • in the test file, wrap each independent ProcessNextHeaders call inside {} blocks and use structured bindings so temporary variables aren’t reused accidentally;
    • rename the new file to headerssync_params.h (underscore instead of -) for consistency with other similar headers;
    • update contrib/devtools/README.md to reference the header instead of the .cpp;
    • Tweak a few comments whose wording is unclear or slightly misleading.
  22. TheCharlatan commented at 10:09 am on May 30, 2025: contributor
    Concept ACK
  23. hodlinator force-pushed on May 30, 2025
  24. hodlinator commented at 8:37 pm on May 30, 2025: contributor
    Thanks for your feedback so far @l0rinc! Took most of your suggestions (https://github.com/bitcoin/bitcoin/pull/32579#pullrequestreview-2857912770).
  25. hodlinator force-pushed on May 30, 2025
  26. jonatack commented at 5:55 pm on June 2, 2025: member
    Concept ACK
  27. in src/headerssync.cpp:17 in bbe5609b84 outdated
    26-// re-calculate parameters if we compress further)
    27+// Our memory analysis in headerssync-params.py assumes 48 bytes for a
    28+// CompressedHeader (we should re-calculate parameters if we compress further).
    29 static_assert(sizeof(CompressedHeader) == 48);
    30 
    31 HeadersSyncState::HeadersSyncState(NodeId id, const Consensus::Params& consensus_params,
    


    hodlinator commented at 9:04 pm on June 2, 2025:

    Just realized a different possible approach: passing HEADER_COMMITMENT_PERIOD and REDOWNLOAD_BUFFER_SIZE values into the HeadersSyncState-ctor. Right now that aspect of the class is hardcoded for mainnet, despite the ctor accepting Consensus::Params. Having these values be sent in would enable tests to send in different (smaller) variables for regtest, while still testing behavior around the threshold.

    In fact, it would fit nicely with other values updated during the release process to have them written into the corresponding section (mainnet/regtest/etc) of src/kernel/chainparams.cpp.

    Started implementing that approach to see how it pans out.


    hodlinator commented at 1:48 pm on June 4, 2025:
    Implemented in latest push.
  28. hodlinator force-pushed on Jun 4, 2025
  29. hodlinator commented at 2:39 pm on June 4, 2025: contributor

    Latest push (42e746096b93b8026569817d8925750089d6daf4, compare):

    • Relaxes the hardcoding of the header commitment period and the redownload buffer size in HeadersSyncState.
      • This makes things more consistent with the constructor already taking Consensus::Params (implying different chains are supported).
      • Moves specification of the above constants from the top of headerssync.cpp into kernel/chainparams.cpp (similar to many other constants).
      • Returns the chainwork test suite back to only using an integer literal for TARGET_BLOCKS (instead of depending on an #included constant). Also now uses test-local values for commitment period and redownload buffer size.
    • Adds a warning in HeadersSyncState::PopHeadersReadyForAcceptance() for the case when we reach minimum PoW before exceeding the redownload buffer size together with tests.

    (Updated PR description).

  30. hodlinator renamed this:
    headerssync: Keep tests ahead of increasing params
    headerssync: Preempt unrealistic unit test behavior
    on Jun 4, 2025
  31. in src/test/headers_sync_chainwork_tests.cpp:53 in fab6f59e2e outdated
    30-
    31-void HeadersGeneratorSetup::FindProofOfWork(CBlockHeader& starting_header)
    32+constexpr size_t TARGET_BLOCKS{15'000};
    33+constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
    34+
    35+/** Search for a nonce to meet (regtest) proof of work */
    


    l0rinc commented at 1:00 pm on June 17, 2025:
    I don’t think it’s regtest specific - is it?

    hodlinator commented at 7:08 am on June 23, 2025:

    The test is running on RegTest:

    https://github.com/bitcoin/bitcoin/blob/42e746096b93b8026569817d8925750089d6daf4/src/test/headers_sync_chainwork_tests.cpp#L72

    I’m guessing including the comment is meant to assuage fears that this would be an intensive operation.


    l0rinc commented at 2:14 pm on June 24, 2025:
    ah, so it’s jut a reassurance that it doesn’t need a fancy asic, it can mine a block in no time - makes sense
  32. in src/test/headers_sync_chainwork_tests.cpp:86 in 7d45872a05 outdated
    85     // initially and then the rest.
    86-    headers_batch.insert(headers_batch.end(), std::next(first_chain.begin()), first_chain.end());
    87-
    88     hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    89-    (void)hss->ProcessNextHeaders({first_chain.front()}, true);
    90+    (void)hss->ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    


    l0rinc commented at 1:07 pm on June 17, 2025:

    Doc states: by delivering 1 header initially and then the rest. Could we clarify this by code instead? We don’t really have head/tail like in many functional programming languages, but maybe we could do:

    0    const std::span headers_batch{second_chain};
    1    (void)hss->ProcessNextHeaders(headers_batch.first(1), true);
    2...
    3    result = hss->ProcessNextHeaders(headers_batch.last(headers_batch.size() - 1), false);
    

    hodlinator commented at 7:32 am on June 23, 2025:

    Would rather not go back to having an intermediate headers_batch variable. Changed to {{second_chain.front()}} + std::span{second_chain}.last(second_chain.size() - 1).

    Kept the almost the existing style in HappyPath since I couldn’t see an improvement to {first_chain.begin() + REDOWNLOAD_BUFFER_SIZE, 1}.

  33. in src/test/headers_sync_chainwork_tests.cpp:132 in 7d45872a05 outdated
    115@@ -119,15 +116,13 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    116     hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    117     BOOST_CHECK(hss->GetState() == HeadersSyncState::State::PRESYNC);
    118      // Pretend just the first message is "full", so we don't abort.
    119-    (void)hss->ProcessNextHeaders({second_chain.front()}, true);
    120+    (void)hss->ProcessNextHeaders(std::span{second_chain.begin(), 1}, true);
    121     BOOST_CHECK(hss->GetState() == HeadersSyncState::State::PRESYNC);
    122 
    123-    headers_batch.clear();
    


    l0rinc commented at 1:24 pm on June 17, 2025:
    👍 for getting rid of this
  34. in src/test/headers_sync_chainwork_tests.cpp:63 in a0cd196d2f outdated
    59@@ -60,15 +60,23 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, RegTestingSetup)
    60 //    updates to the REDOWNLOAD phase successfully.
    61 // 2. Then we deliver the second set of headers and verify that they fail
    62 //    processing (presumably due to commitments not matching).
    63-// 3. Finally, we verify that repeating with the first set of headers in both
    64-//    phases is successful.
    65+static void SneakyRedownload(const CBlockIndex* chain_start,
    


    l0rinc commented at 1:32 pm on June 17, 2025:
    👍 for splitting these up. Is it important to keep 1) and 2) as a single method, given that we thought it’s best described in two separate list items? Since you grouped them into a “redownload”, it’s probably the comments that should be adjusted.

    hodlinator commented at 7:38 am on June 23, 2025:
    Agreed that 1 & 2 should ideally be combined if this were written from scratch, but I prefer keeping the current approach to make the diff less jarring compared to master. Open to re-evaluate if other reviewers weigh in.

    l0rinc commented at 2:55 pm on June 24, 2025:
    Can we add 2 helpers in that case? I really dislike that the comments don’t follow the code’s structure.

    hodlinator commented at 8:33 pm on June 25, 2025:
    Renumbered in the “Improve comments”-commit.
  35. in src/test/headers_sync_chainwork_tests.cpp:110 in 79be6f7744 outdated
    109-
    110     // Feed the first chain to HeadersSyncState, by delivering 1 header
    111     // initially and then the rest.
    112-    hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    113-    (void)hss->ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    114+    HeadersSyncState hss{CreateState(chain_start, Params())};
    


    l0rinc commented at 1:39 pm on June 17, 2025:
    nit: Maybe mention in the previous commit message that “don’t worry, I’m not gonna’ leave the inits like that” :D
  36. in src/test/headers_sync_chainwork_tests.cpp:149 in 79be6f7744 outdated
    163-    HeadersSyncState::ProcessingResult result;
    164     // Verify that just trying to process the second chain would not succeed
    165     // (too little work).
    166-    hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    167-    BOOST_CHECK(hss->GetState() == HeadersSyncState::State::PRESYNC);
    168+    HeadersSyncState hss{CreateState(chain_start, Params())};
    


    l0rinc commented at 1:43 pm on June 17, 2025:
    nit: I don’t particularly like the abbreviated hss name - would const auto sync_state{CreateState(chain_start, Params())}; maybe be more descriptive?

    hodlinator commented at 7:48 am on June 23, 2025:
    net_processing.cpp uses m_headers_sync, but I’m totally fine with hss-abbreviation, especially if we keep spelling out HeadersSyncState. Makes for a slightly less stark diff with master.
  37. in src/test/headers_sync_chainwork_tests.cpp:113 in 667fd31e74 outdated
    108@@ -109,24 +109,35 @@ static void SneakyRedownload(const CBlockIndex* chain_start,
    109     // initially and then the rest.
    110     HeadersSyncState hss{CreateState(chain_start, Params())};
    111     {
    112-        (void)hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    113+        // Just feed one header and check state.
    114+        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    


    l0rinc commented at 7:24 am on June 20, 2025:

    nit:

    0        const auto result{hss.ProcessNextHeaders({first_chain.begin(), 1}, /*full_headers_message=*/true)};
    

    hodlinator commented at 8:05 am on June 23, 2025:
    Replaced with CHECK_RESULT() macro from your suggestion in #32579 (review). Sprinkled some less obvious /*full_headers_message=*/ around.
  38. in src/test/headers_sync_chainwork_tests.cpp:114 in 667fd31e74 outdated
    108@@ -109,24 +109,35 @@ static void SneakyRedownload(const CBlockIndex* chain_start,
    109     // initially and then the rest.
    110     HeadersSyncState hss{CreateState(chain_start, Params())};
    111     {
    112-        (void)hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    113+        // Just feed one header and check state.
    114+        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    115+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    


    l0rinc commented at 7:39 am on June 20, 2025:
    HeadersSyncState::State::PRESYNC is very verbose in a file called headers_sync_chainwork_tests - can we using State = HeadersSyncState::State; here instead to reduce the noise somewhat?

    hodlinator commented at 7:52 am on June 23, 2025:
    Done in the same commit as we switch to BOOST_REQUIRE_EQUAL.
  39. in src/test/headers_sync_chainwork_tests.cpp:68 in a0cd196d2f outdated
    65+static void SneakyRedownload(const CBlockIndex* chain_start,
    66+        const std::vector<CBlockHeader>& first_chain,
    67+        const std::vector<CBlockHeader>& second_chain);
    68+// 3. Verify that repeating with the first set of headers in both phases is
    69+//    successful.
    70+static void HappyPath(const CBlockIndex* chain_start,
    


    l0rinc commented at 7:54 am on June 20, 2025:
    do we really need these forward declarations here? Wouldn’t it be simpler to just put the test at the end of the file like we’re doing in other places e.g. https://github.com/bitcoin/bitcoin/blob/master/src/test/addrman_tests.cpp#L23-L61 ? If we do need the prototypes can we maybe remove the arg names to reduce the noise?

    hodlinator commented at 8:19 am on June 23, 2025:

    do we really need these forward declarations here? Wouldn’t it be simpler to just put the test at the end of the file like we’re doing in other places e.g. https://github.com/bitcoin/bitcoin/blob/master/src/test/addrman_tests.cpp#L23-L61 ?

    Did it in order to weave the breakup into the existing comments, while avoiding to re-order chunks of code to minimize diffs.

    If we do need the prototypes can we maybe remove the arg names to reduce the noise?

    Done - not used to this style, but warming up to it. :)


    l0rinc commented at 2:58 pm on June 24, 2025:

    chunks of code to minimize diffs

    We’ve rewritten the tests at this stage, diffs and the function prototypes don’t help in my opinion. But if you insist, I don’t mind, please resolve the comment

  40. in src/test/headers_sync_chainwork_tests.cpp:212 in 42e746096b outdated
    286+        BOOST_CHECK_EQUAL(result.pow_validated_headers.front().hashPrevBlock, first_after_genesis.GetHash());
    287+    }
    288+}
    289+
    290+static void TooLittleWork(const CBlockIndex* chain_start,
    291+        const std::vector<CBlockHeader>& second_chain)
    


    l0rinc commented at 9:12 am on June 20, 2025:
    first_chain/second_chain is a bit verbose given how often we’re using them

    hodlinator commented at 8:21 am on June 23, 2025:
    Would rather keep the names to minimize diffs. sufficient_chain & insufficient_chain would have been more descriptive but longer.
  41. in src/test/headers_sync_chainwork_tests.cpp:92 in 42e746096b outdated
    88@@ -87,6 +89,9 @@ static void HappyPath(const CBlockIndex* chain_start,
    89 static void TooLittleWork(const CBlockIndex* chain_start,
    90         const std::vector<CBlockHeader>& second_chain);
    91 
    92+static void TooBigBuffer(const CBlockIndex* chain_start,
    


    l0rinc commented at 9:24 am on June 20, 2025:
    this scenario could also use a comment like the rest

    hodlinator commented at 9:06 am on June 23, 2025:
    Done.
  42. in src/test/headers_sync_chainwork_tests.cpp:170 in 42e746096b outdated
    165+        if (str == nullptr) {
    166+            return false; // Disable exception for not finding a match.
    167+        } else {
    168+            throw std::runtime_error{strprintf("'%s' should not exist in debug log\n", str)};
    169+        }
    170+    }};
    


    l0rinc commented at 9:35 am on June 20, 2025:

    This is a bit hacky - can we add a negated helper to logging.h?

     0// Test fails if the pattern *DOES NOT* show up.
     1#define ASSERT_DEBUG_LOG(message) DebugLogHelper UNIQUE_NAME(debugloghelper)(message)
     2// Test fails if the pattern *DOES* show up.
     3#define ASSERT_NO_DEBUG_LOG(message)                                              \
     4    DebugLogHelper UNIQUE_NAME(nologhelper){                                      \
     5        message,                                                                  \
     6        [](const std::string* line) {                                             \
     7            if (line) throw std::runtime_error("unexpected log line: " + *line);  \
     8            return false; /* suppress default 'not found' failure */              \
     9        }                                                                         \
    10    }
    

    And use it here as:

    0    // During normal operation we shouldn't get the redownload buffer size warning.
    1    ASSERT_NO_DEBUG_LOG(BUFFER_SIZE_WARNING);
    

    hodlinator commented at 9:20 am on June 23, 2025:
    Originally resisted the impulse since there is only one case, but done now. It mirrors what we do in functional Python tests after all.
  43. in src/test/headers_sync_chainwork_tests.cpp:264 in 42e746096b outdated
    259+                         HeadersSyncParams{
    260+                             .commitment_period = COMMITMENT_PERIOD,
    261+                             .redownload_buffer_size = first_chain.size(),
    262+                         },
    263+                         chain_start,
    264+                         /*minimum_required_work=*/CHAIN_WORK};
    


    l0rinc commented at 9:36 am on June 20, 2025:

    we should be able to reuse CreateState here - if we make HeadersSyncParams configurable:

    0static HeadersSyncState CreateState(const CChainParams& chain_params, const HeadersSyncParams& sync_params, const CBlockIndex* chain_start)
    1{
    2    return {/*id=*/0,
    3            chain_params.GetConsensus(),
    4            sync_params,
    5            chain_start,
    6            /*minimum_required_work=*/CHAIN_WORK};
    7}
    

    and

    0// Scenario that intentionally uses an oversized buffer to trigger the warning.
    1static void TooBigBuffer(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1)
    2{
    3    // Intentionally too big redownload buffer in order to trigger warning.
    4    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, chain1.size()}, chain_start)};
    5    ...
    

    hodlinator commented at 9:31 am on June 23, 2025:
    Taken in broad strokes.
  44. in src/test/headers_sync_chainwork_tests.cpp:265 in 42e746096b outdated
    260+                             .commitment_period = COMMITMENT_PERIOD,
    261+                             .redownload_buffer_size = first_chain.size(),
    262+                         },
    263+                         chain_start,
    264+                         /*minimum_required_work=*/CHAIN_WORK};
    265+    (void)hss.ProcessNextHeaders(first_chain, true);
    


    l0rinc commented at 9:37 am on June 20, 2025:

    can we validate this as well instead of just voiding it?

    0CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    1             state, State::REDOWNLOAD,
    2             /*exp_success=*/true,
    3             /*exp_request_more=*/true,
    4             /*exp_headers_size=*/0);
    
  45. in src/test/headers_sync_chainwork_tests.cpp:272 in 42e746096b outdated
    334+
    335+    {
    336+        ASSERT_DEBUG_LOG(BUFFER_SIZE_WARNING);
    337+        auto result{hss.ProcessNextHeaders(first_chain, true)};
    338+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    339+        BOOST_CHECK(result.success);
    


    l0rinc commented at 9:39 am on June 20, 2025:

    We could validate request_more and pow_validated_headers here as well:

    0CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    1             state, State::FINAL,
    2             /*exp_success=*/true,
    3             /*exp_request_more=*/false,
    4             /*exp_headers_size=*/chain1.size());
    
  46. in src/test/headers_sync_chainwork_tests.cpp:127 in 875e89c82f outdated
    136+        BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    137+    }
    138+
    139+    {
    140+        // Try to sneakily feed back the second chain.
    141+        auto result{hss.ProcessNextHeaders(second_chain, true)};
    


    l0rinc commented at 9:42 am on June 20, 2025:

    👍 for reducing the scope of each separate check.

    Was hoping we could compare the result object directly as:

    0BOOST_CHECK_EQUAL(result, (HeadersSyncState::ProcessingResult{
    1    .pow_validated_headers = {},
    2    .success = true,
    3    .request_more = true,
    4}));
    

    But unfortunately we’d need to add equality operators to ProcessingResult and CBlockHeader which would defeat the purpose (+ the operator<<(std::ostream& for printing)…

    But if we extract a validation helper function instead, we’d have keep the scope reduction and make the differences between the checks less noisy:

    0// Standard set of checks common to all scenarios; macro keeps failure lines at the call-site.
    1#define CHECK_RESULT(res, state, exp_state, exp_success, exp_request_more, exp_headers_size) \
    2    do {                                                                                     \
    3        const auto& [res_headers, res_success, res_request_more]{res};                       \
    4        BOOST_REQUIRE_EQUAL(state.GetState(), exp_state);                                    \
    5        BOOST_CHECK_EQUAL(res_headers.size(), exp_headers_size);                             \
    6        BOOST_CHECK_EQUAL(res_success, exp_success);                                         \
    7        BOOST_CHECK_EQUAL(res_request_more, exp_request_more);                               \
    8    } while (false)
    

    and using it as e.g.

     0// Repeat the second set of headers in both phases to demonstrate
     1// behavior when the chain a peer provides has too little work.
     2static void TooLittleWork(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain2)
     3{
     4    // Verify that just trying to process the second chain would not succeed (too little work).
     5    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
     6
     7    // Pretend just the first message is "full", so we don't abort.
     8    CHECK_RESULT(state.ProcessNextHeaders({chain2.begin(), 1}, /*full_headers_message=*/true),
     9                 state, State::PRESYNC,
    10                 /*exp_success=*/true,
    11                 /*exp_request_more=*/true,
    12                 /*exp_headers_size=*/0);
    13
    14    // Tell the sync logic that the headers message was not full, implying no
    15    // more headers can be requested. For a low-work-chain, this should cause
    16    // the sync to end with no headers for acceptance.
    17    CHECK_RESULT(state.ProcessNextHeaders({chain2.begin() + 1, chain2.end()}, /*full_headers_message=*/false),
    18                 state, State::FINAL,
    19                 /*exp_success=*/true,
    20                 /*exp_request_more=*/false,
    21                 /*exp_headers_size=*/0);
    22}
    

    hodlinator commented at 12:45 pm on June 23, 2025:
    Taken, added exp_pow_validated_prev and exp_locator_hash-parameters. Feels good to verify all parameters, also always in the same order.
  47. in src/kernel/chainparams.cpp:193 in 6a5914ea5f outdated
    188@@ -189,6 +189,12 @@ class CMainParams : public CChainParams {
    189             .tx_count = 1161875261,
    190             .dTxRate  = 4.620728156243148,
    191         };
    192+
    193+        // Generated by headerssync-params.py on 2025-03-04.
    


    l0rinc commented at 9:48 am on June 20, 2025:
    We’ll update this during the v30 release?

    hodlinator commented at 8:32 am on June 23, 2025:

    As release-process.md documents, it’s not critical to update on every release. It’s mostly about fine-tuning memory usage under adversarial conditions, and the values don’t tend to change that much. But I would not be surprised if it were updated since it was updated during the previous 2 releases:

    • v29: 11a2d3a63e90cdc1920ede3c67d52a9c72860e6b
    • v28: 221809b81cfcecb04050915eebacffda2599da42

    Also, we’re slated to remove checkpoints (#31649) in v30 which might further encourage the release manager to do this.

  48. in contrib/devtools/headerssync-params.py:343 in 6a5914ea5f outdated
    345+    print(f"memory usage on mainchain for release until {TIME:%Y-%m-%d} is:")
    346     print()
    347-    print("//! Only feed headers to validation once this many headers on top have been")
    348-    print("//! received and validated against commitments.")
    349-    print(f"constexpr size_t REDOWNLOAD_BUFFER_SIZE{{{bufsize}}};"
    350+    print(f"        // Generated by headerssync-params.py on {datetime.today():%Y-%m-%d}.")
    


    l0rinc commented at 9:48 am on June 20, 2025:
    👍
  49. in contrib/devtools/headerssync-params.py:347 in 6a5914ea5f outdated
    349-    print(f"constexpr size_t REDOWNLOAD_BUFFER_SIZE{{{bufsize}}};"
    350+    print(f"        // Generated by headerssync-params.py on {datetime.today():%Y-%m-%d}.")
    351+    print( "        m_headers_sync_params = HeadersSyncParams{")
    352+    print(f"            .commitment_period = {period},")
    353+    print(f"            .redownload_buffer_size = {bufsize},"
    354           f" // {bufsize}/{period} = ~{bufsize/period:.1f} commitments")
    


    l0rinc commented at 9:49 am on June 20, 2025:

    nit: I’d join this with the previous line, it’s just confusing this way. And can we round this (given we already have a leading ~) instead of adding a weird decimal (what is 0.8 commitment mean?)

    And why are we repeating the division in the comment, why not just:

    0print(f"            .redownload_buffer_size = {bufsize}}; // ~{round(bufsize / period)} commitments")
    

    resulting in something like:

    0        // Generated by headerssync-params.py on 2025-03-04.
    1        m_headers_sync_params = HeadersSyncParams{
    2            .commitment_period = 624,
    3            .redownload_buffer_size = 14827}; // ~24 commitments
    

    hodlinator commented at 8:54 am on June 23, 2025:

    I’d join this with the previous line, it’s just confusing this way.

    Prefer keeping this line unchanged compared to master (would rather have lines closer to 80 than 120 chars here).

    can we round this (given we already have a leading ~) instead of adding a weird decimal (what is 0.8 commitment mean?)

    Including one decimal makes it more likely that the number changes between releases, indicating some kind of usefulness + precision of simulation. Not sure if that was what the original authors intended.

    And why are we repeating the division in the comment

    I think keeping the division makes it slightly clearer what the number is to readers of the C++ source.


    l0rinc commented at 2:38 pm on June 24, 2025:

    makes it more likely that the number changes between releases

    That sounds like a counter-argument to me, but don’t have strong feelings about it, if you disagree, I don’t mind, please resolve it.

  50. in src/headerssync.cpp:13 in 6a5914ea5f outdated
    20-//! received and validated against commitments.
    21-constexpr size_t REDOWNLOAD_BUFFER_SIZE{14827}; // 14827/624 = ~23.8 commitments
    22-
    23-// Our memory analysis assumes 48 bytes for a CompressedHeader (so we should
    24-// re-calculate parameters if we compress further)
    25+// Our memory analysis in headerssync-params.py assumes 48 bytes for a
    


    l0rinc commented at 9:51 am on June 20, 2025:
    • 48 is already stated in the code, no need to repeat it in the comment
    • we should re-calculate parameters if we compress further - doesn’t the static assert already obviate that?

    hodlinator commented at 12:27 pm on June 23, 2025:

    48 is already stated in the code, no need to repeat it in the comment

    Replaced with “this many”.

    we should re-calculate parameters if we compress further - doesn’t the static assert already obviate that?

    Don’t think so? This is about prompting whoever changes the struct to also alter and re-run the Python script.

  51. in src/test/headers_sync_chainwork_tests.cpp:165 in 04a0b6609f outdated
    161@@ -161,17 +162,45 @@ static void HappyPath(const CBlockIndex* chain_start,
    162         BOOST_CHECK(result.request_more);
    163         BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    164         // The locator should reset to genesis.
    165-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    166+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    


    l0rinc commented at 9:59 am on June 20, 2025:
    nice, we don’t need the comment anymore after this change

    hodlinator commented at 12:22 pm on June 23, 2025:
    Still think it is good to document why something is expected.

    l0rinc commented at 3:01 pm on June 24, 2025:
    Not sure I understand, especially in the new code which states something like locator == genesis - which is basically the same as the comment
  52. in src/headerssync.cpp:283 in 42e746096b outdated
    277@@ -278,6 +278,14 @@ std::vector<CBlockHeader> HeadersSyncState::PopHeadersReadyForAcceptance()
    278     Assume(m_download_state == State::REDOWNLOAD);
    279     if (m_download_state != State::REDOWNLOAD) return ret;
    280 
    281+    if (m_process_all_remaining_headers && m_chain_start->nHeight == 0 &&
    282+        m_redownload_buffer_first_prev_hash == m_chain_start->GetBlockHash() &&
    283+        m_redownloaded_headers.size() <= m_params.redownload_buffer_size) {
    


    l0rinc commented at 10:01 am on June 20, 2025:
    Q: I can’t immediately tell if this should be triggered for equality as well - please just resolve if it’s a stupid question.

    hodlinator commented at 12:34 pm on June 23, 2025:

    Appreciate feedback on this!

    Are you asking whether m_redownloaded_headers.size() < m_params.redownload_buffer_size might be more correct?

    The while-condition below includes m_redownloaded_headers.size() > m_params.redownload_buffer_size as one of the conditions to start feeding back PoW-validated headers. This means using m_redownloaded_headers.size() < m_params.redownload_buffer_size for the condition above would give us no headers and no warning when the buffer size equals the parameter.


    l0rinc commented at 2:46 pm on June 24, 2025:
    Hmmm, if it has to be the opposite of the while below, would it make sense to extract the common parts to explain it with code (your explanation makes sense, would prefer if the code already made it simpler so that I don’t have to check the code that’s after the current one to understand it).

    hodlinator commented at 8:34 pm on June 25, 2025:
    Broke out bool buffer_exceeded, let me know if you think it’s an improvement.

    l0rinc commented at 2:29 pm on June 27, 2025:
    I don’t like the new repetition - will have to debug it locally to understand why we run the condition outside of the loop in the first place 🤔

    hodlinator commented at 10:34 pm on August 9, 2025:
    Latest push improves this part of the code, please check it out!
  53. l0rinc changes_requested
  54. l0rinc commented at 10:17 am on June 20, 2025: contributor

    I might have gone a bit overboard with the test review, but I think we can make it a lot less noisy - here’s the patch of many of my suggestions for convenience:

      0diff --git a/src/kernel/chainparams.cpp b/src/kernel/chainparams.cpp
      1index d64dd43aba..658ea7f2c3 100644
      2--- a/src/kernel/chainparams.cpp
      3+++ b/src/kernel/chainparams.cpp
      4@@ -193,8 +193,7 @@ public:
      5         // Generated by headerssync-params.py on 2025-03-04.
      6         m_headers_sync_params = HeadersSyncParams{
      7             .commitment_period = 624,
      8-            .redownload_buffer_size = 14827, // 14827/624 = ~23.8 commitments
      9-        };
     10+            .redownload_buffer_size = 14827}; // ~24 commitments
     11     }
     12 };
     13 
     14diff --git a/src/test/headers_sync_chainwork_tests.cpp b/src/test/headers_sync_chainwork_tests.cpp
     15index 1f85c69838..ccbbc1be50 100644
     16--- a/src/test/headers_sync_chainwork_tests.cpp
     17+++ b/src/test/headers_sync_chainwork_tests.cpp
     18@@ -13,10 +13,13 @@
     19 #include <validation.h>
     20 
     21 #include <cstddef>
     22+#include <optional>
     23 #include <vector>
     24 
     25 #include <boost/test/unit_test.hpp>
     26 
     27+using State = HeadersSyncState::State;
     28+
     29 constexpr size_t TARGET_BLOCKS{15'000};
     30 constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
     31 constexpr size_t REDOWNLOAD_BUFFER_SIZE{TARGET_BLOCKS - (MAX_HEADERS_RESULTS + 123)};
     32@@ -37,18 +40,18 @@ static void FindProofOfWork(CBlockHeader& starting_header)
     33  * prev_time, and with a fixed merkle root hash.
     34  */
     35 static void GenerateHeaders(std::vector<CBlockHeader>& headers,
     36-        size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     37-        const uint256& merkle_root, const uint32_t nBits)
     38+                            size_t count, const uint256& starting_hash, int nVersion, int prev_time,
     39+                            const uint256& merkle_root, uint32_t nBits)
     40 {
     41-    uint256 prev_hash = starting_hash;
     42+    uint256 prev_hash{starting_hash};
     43 
     44     while (headers.size() < count) {
     45         headers.emplace_back();
     46-        CBlockHeader& next_header = headers.back();
     47+        CBlockHeader& next_header{headers.back()};
     48         next_header.nVersion = nVersion;
     49         next_header.hashPrevBlock = prev_hash;
     50         next_header.hashMerkleRoot = merkle_root;
     51-        next_header.nTime = prev_time+1;
     52+        next_header.nTime = prev_time + 1;
     53         next_header.nBits = nBits;
     54 
     55         FindProofOfWork(next_header);
     56@@ -57,220 +60,167 @@ static void GenerateHeaders(std::vector<CBlockHeader>& headers,
     57     }
     58 }
     59 
     60-static HeadersSyncState CreateState(const CBlockIndex* chain_start, const CChainParams& chain_params)
     61+static HeadersSyncState CreateState(const CChainParams& chain_params, const HeadersSyncParams& sync_params, const CBlockIndex* chain_start)
     62 {
     63     return {/*id=*/0,
     64             chain_params.GetConsensus(),
     65-            HeadersSyncParams{
     66-                .commitment_period = COMMITMENT_PERIOD,
     67-                .redownload_buffer_size = REDOWNLOAD_BUFFER_SIZE,
     68-            },
     69+            sync_params,
     70             chain_start,
     71-            /*minimum_required_work=*/CHAIN_WORK};
     72+           /*minimum_required_work=*/CHAIN_WORK};
     73 }
     74 
     75-BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, RegTestingSetup)
     76-
     77-// In this test, we construct two sets of headers from genesis, one with
     78-// sufficient proof of work and one without.
     79-// 1. We deliver the first set of headers and verify that the headers sync state
     80-//    updates to the REDOWNLOAD phase successfully.
     81-// 2. Then we deliver the second set of headers and verify that they fail
     82-//    processing (presumably due to commitments not matching).
     83-static void SneakyRedownload(const CBlockIndex* chain_start,
     84-        const std::vector<CBlockHeader>& first_chain,
     85-        const std::vector<CBlockHeader>& second_chain);
     86-// 3. Verify that repeating with the first set of headers in both phases is
     87-//    successful.
     88-static void HappyPath(const CBlockIndex* chain_start,
     89-        const std::vector<CBlockHeader>& first_chain);
     90-// 4. Finally, repeat the second set of headers in both phases to demonstrate
     91-//    behavior when the chain a peer provides has too little work.
     92-static void TooLittleWork(const CBlockIndex* chain_start,
     93-        const std::vector<CBlockHeader>& second_chain);
     94-
     95-static void TooBigBuffer(const CBlockIndex* chain_start,
     96-        const std::vector<CBlockHeader>& first_chain);
     97-
     98-BOOST_AUTO_TEST_CASE(headers_sync_state)
     99+// Standard set of checks common to all scenarios; macro keeps failure lines at the call-site.
    100+#define CHECK_RESULT(res, state, exp_state, exp_success, exp_request_more, exp_headers_size) \
    101+    do {                                                                                     \
    102+        const auto& [res_headers, res_success, res_request_more]{res};                       \
    103+        BOOST_REQUIRE_EQUAL(state.GetState(), exp_state);                                    \
    104+        BOOST_CHECK_EQUAL(res_headers.size(), exp_headers_size);                             \
    105+        BOOST_CHECK_EQUAL(res_success, exp_success);                                         \
    106+        BOOST_CHECK_EQUAL(res_request_more, exp_request_more);                               \
    107+    } while (false)
    108+
    109+// Deliver the first set of headers from genesis to reach REDOWNLOAD,
    110+// then deliver the second set and verify that commitments don’t match.
    111+static void SneakyRedownload(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1, const std::vector<CBlockHeader>& chain2)
    112 {
    113-    std::vector<CBlockHeader> first_chain;
    114-    std::vector<CBlockHeader> second_chain;
    115-
    116-    const auto genesis{Params().GenesisBlock()};
    117-
    118-    // Generate headers for two different chains (using differing merkle roots
    119-    // to ensure the headers are different).
    120-    GenerateHeaders(first_chain, TARGET_BLOCKS - 1, genesis.GetHash(), genesis.nVersion,
    121-                    genesis.nTime, /*merkle_root=*/uint256::ZERO, genesis.nBits);
    122-    GenerateHeaders(second_chain, TARGET_BLOCKS - 2, genesis.GetHash(), genesis.nVersion,
    123-                    genesis.nTime, /*merkle_root=*/uint256::ONE, genesis.nBits);
    124-
    125-    const CBlockIndex* chain_start = WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(genesis.GetHash()));
    126+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
    127+    const auto genesis_hash{Params().GenesisBlock().GetHash()};
    128 
    129-    SneakyRedownload(chain_start, first_chain, second_chain);
    130-    HappyPath(chain_start, first_chain);
    131-    TooLittleWork(chain_start, second_chain);
    132-    TooBigBuffer(chain_start, first_chain);
    133+    const std::span headers{chain1};
    134+    CHECK_RESULT(state.ProcessNextHeaders(headers.first(1), /*full_headers_message=*/true),
    135+                 state, State::PRESYNC,
    136+                 /*exp_success=*/true,
    137+                 /*exp_request_more=*/true,
    138+                 /*exp_headers_size=*/0);
    139+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), chain1.front().GetHash());
    140+
    141+    // Pretend the message is still "full", so we don't abort.
    142+    // This chain should look valid, and we should have met the proof-of-work
    143+    // requirement during PRESYNC and transitioned to REDOWNLOAD.
    144+    CHECK_RESULT(state.ProcessNextHeaders(headers.last(headers.size() - 1), /*full_headers_message=*/true),
    145+                 state, State::REDOWNLOAD,
    146+                 /*exp_success=*/true,
    147+                 /*exp_request_more=*/true,
    148+                 /*exp_headers_size=*/0);
    149+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    150+
    151+    // Try to sneakily feed back the second chain during REDOWNLOAD.
    152+    CHECK_RESULT(state.ProcessNextHeaders(chain2, /*full_headers_message=*/true),
    153+                 state, State::FINAL,
    154+                 /*exp_success=*/false,
    155+                 /*exp_request_more=*/false,
    156+                 /*exp_headers_size=*/0);
    157 }
    158 
    159-static void SneakyRedownload(const CBlockIndex* chain_start,
    160-        const std::vector<CBlockHeader>& first_chain,
    161-        const std::vector<CBlockHeader>& second_chain)
    162+// Verify that repeating with the first set of headers in both phases is successful.
    163+static void HappyPath(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1)
    164 {
    165-    // Feed the first chain to HeadersSyncState, by delivering 1 header
    166-    // initially and then the rest.
    167-    HeadersSyncState hss{CreateState(chain_start, Params())};
    168-    {
    169-        // Just feed one header and check state.
    170-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    171-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    172-        BOOST_CHECK(result.success);
    173-        BOOST_CHECK(result.request_more);
    174-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    175-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain.front().GetHash());
    176-    }
    177-
    178-    {
    179-        // Pretend the message is still "full", so we don't abort.
    180-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin() + 1, first_chain.end()}, true)};
    181-        // This chain should look valid, and we should have met the proof-of-work
    182-        // requirement during PRESYNC and transitioned to REDOWNLOAD.
    183-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    184-        BOOST_CHECK(result.success);
    185-        BOOST_CHECK(result.request_more);
    186-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    187-        // The locator should reset to genesis.
    188-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    189-    }
    190+    // Feed the first chain twice.
    191+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
    192+    const auto genesis_hash{Params().GenesisBlock().GetHash()};
    193+    const std::span headers{chain1};
    194 
    195-    {
    196-        // Try to sneakily feed back the second chain during REDOWNLOAD.
    197-        auto result{hss.ProcessNextHeaders(second_chain, true)};
    198-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    199-        BOOST_CHECK(!result.success); // Foiled! We detected mismatching headers.
    200-        BOOST_CHECK(!result.request_more);
    201-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    202-    }
    203+    // During normal operation we shouldn't get the redownload buffer size warning.
    204+    ASSERT_NO_DEBUG_LOG(BUFFER_SIZE_WARNING);
    205+
    206+    // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    207+    CHECK_RESULT(state.ProcessNextHeaders(headers, /*full_headers_message=*/true),
    208+                 state, State::REDOWNLOAD,
    209+                 /*exp_success=*/true,
    210+                 /*exp_request_more=*/true,
    211+                 /*exp_headers_size=*/0);
    212+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    213+
    214+    // Process only so that the internal threshold isn't met, meaning validated headers shouldn't be returned yet:
    215+    CHECK_RESULT(state.ProcessNextHeaders(headers.first(REDOWNLOAD_BUFFER_SIZE), /*full_headers_message=*/true),
    216+                 state, State::REDOWNLOAD,
    217+                 /*exp_success=*/true,
    218+                 /*exp_request_more=*/true,
    219+                 /*exp_headers_size=*/0);
    220+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), chain1[REDOWNLOAD_BUFFER_SIZE - 1].GetHash());
    221+
    222+    // Next header should make us exceed the threshold, but still not be done:
    223+    const auto res_threshold{state.ProcessNextHeaders(headers.subspan(REDOWNLOAD_BUFFER_SIZE, 1), /*full_headers_message=*/true)};
    224+    CHECK_RESULT(res_threshold,
    225+                 state, State::REDOWNLOAD,
    226+                 /*exp_success=*/true,
    227+                 /*exp_request_more=*/true,
    228+                 /*exp_headers_size=*/1);
    229+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), chain1[REDOWNLOAD_BUFFER_SIZE].GetHash());
    230+
    231+    // Feed in remaining headers, meeting the work threshold again and completing the REDOWNLOAD phase:
    232+    CHECK_RESULT(state.ProcessNextHeaders(headers.subspan(REDOWNLOAD_BUFFER_SIZE + 1), /*full_headers_message=*/true),
    233+                 state, State::FINAL,
    234+                 /*exp_success=*/true,
    235+                 /*exp_request_more=*/false,
    236+                 /*exp_headers_size=*/chain1.size() - 1);
    237+    BOOST_CHECK_EQUAL(res_threshold.pow_validated_headers.front().hashPrevBlock, genesis_hash);
    238 }
    239 
    240-static void HappyPath(const CBlockIndex* chain_start,
    241-        const std::vector<CBlockHeader>& first_chain)
    242+// Repeat the second set of headers in both phases to demonstrate
    243+// behavior when the chain a peer provides has too little work.
    244+static void TooLittleWork(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain2)
    245 {
    246-    // This time we feed the first chain twice.
    247-    HeadersSyncState hss{CreateState(chain_start, Params())};
    248-
    249-    // During normal operation we shouldn't get the redownload buffer size warning.
    250-    DebugLogHelper forbidden{BUFFER_SIZE_WARNING, [](const std::string* str) {
    251-        if (str == nullptr) {
    252-            return false; // Disable exception for not finding a match.
    253-        } else {
    254-            throw std::runtime_error{strprintf("'%s' should not exist in debug log\n", str)};
    255-        }
    256-    }};
    257-
    258-    const auto genesis_hash{Params().GenesisBlock().GetHash()};
    259-    {
    260-        auto result{hss.ProcessNextHeaders(first_chain, true)};
    261-        // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    262-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    263-        BOOST_CHECK(result.success);
    264-        BOOST_CHECK(result.request_more);
    265-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    266-        // The locator should reset to genesis.
    267-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    268-    }
    269-
    270-    {
    271-        // Process only so that the internal threshold isn't met, meaning validated
    272-        // headers shouldn't be returned yet:
    273-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), REDOWNLOAD_BUFFER_SIZE}, true)};
    274-        // Not done:
    275-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    276-        BOOST_CHECK(result.success);
    277-        BOOST_CHECK(result.request_more);
    278-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    279-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain[REDOWNLOAD_BUFFER_SIZE - 1].GetHash());
    280-    }
    281-
    282-    CBlockHeader first_after_genesis;
    283-    {
    284-        // Next header should make us exceed the threshold, but still not be done:
    285-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin() + REDOWNLOAD_BUFFER_SIZE, 1}, true)};
    286-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    287-        BOOST_CHECK(result.success);
    288-        BOOST_CHECK(result.request_more);
    289-        BOOST_REQUIRE_EQUAL(result.pow_validated_headers.size(), 1);
    290-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain[REDOWNLOAD_BUFFER_SIZE].GetHash());
    291-        first_after_genesis = result.pow_validated_headers.front();
    292-        BOOST_CHECK_EQUAL(first_after_genesis.hashPrevBlock, genesis_hash);
    293-    }
    294-
    295-    {
    296-        // Feed in remaining headers, meeting the work threshold again and
    297-        // completing the REDOWNLOAD phase.
    298-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin() + REDOWNLOAD_BUFFER_SIZE + 1, first_chain.end()}, true)};
    299-        // Nothing left for the sync logic to do:
    300-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    301-        BOOST_CHECK(result.success);
    302-        BOOST_CHECK(!result.request_more);
    303-        // All headers except the one already returned above:
    304-        BOOST_REQUIRE_EQUAL(result.pow_validated_headers.size(), first_chain.size() - 1);
    305-        BOOST_CHECK_EQUAL(result.pow_validated_headers.front().hashPrevBlock, first_after_genesis.GetHash());
    306-    }
    307+    // Verify that just trying to process the second chain would not succeed (too little work).
    308+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
    309+    const std::span headers{chain2};
    310+
    311+    // Pretend just the first message is "full", so we don't abort.
    312+    CHECK_RESULT(state.ProcessNextHeaders(headers.first(1), /*full_headers_message=*/true),
    313+                 state, State::PRESYNC,
    314+                 /*exp_success=*/true,
    315+                 /*exp_request_more=*/true,
    316+                 /*exp_headers_size=*/0);
    317+
    318+    // Non-full message causes sync to end with no headers accepted.
    319+    CHECK_RESULT(state.ProcessNextHeaders(headers.subspan(1), /*full_headers_message=*/false),
    320+                 state, State::FINAL,
    321+                 /*exp_success=*/true,
    322+                 /*exp_request_more=*/false,
    323+                 /*exp_headers_size=*/0);
    324 }
    325 
    326-static void TooLittleWork(const CBlockIndex* chain_start,
    327-        const std::vector<CBlockHeader>& second_chain)
    328+// Scenario that intentionally uses an oversized buffer to trigger the warning.
    329+static void TooBigBuffer(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1)
    330 {
    331-    // Verify that just trying to process the second chain would not succeed
    332-    // (too little work).
    333-    HeadersSyncState hss{CreateState(chain_start, Params())};
    334-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    335-    {
    336-        // Pretend just the first message is "full", so we don't abort.
    337-        auto result{hss.ProcessNextHeaders(std::span{second_chain.begin(), 1}, true)};
    338-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    339-        BOOST_CHECK(result.success);
    340-        BOOST_CHECK(result.request_more);
    341-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    342-    }
    343-
    344-    {
    345-        // Tell the sync logic that the headers message was not full, implying no
    346-        // more headers can be requested. For a low-work-chain, this should cause
    347-        // the sync to end with no headers for acceptance.
    348-        auto result{hss.ProcessNextHeaders(std::span{second_chain.begin() + 1, second_chain.end()}, false)};
    349-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    350-        BOOST_CHECK(!result.request_more);
    351-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    352-        // Nevertheless, no validation errors should have been detected with the
    353-        // chain:
    354-        BOOST_CHECK(result.success);
    355-    }
    356+    // Intentionally too big redownload buffer in order to trigger warning.
    357+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, chain1.size()}, chain_start)};
    358+
    359+    CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    360+                 state, State::REDOWNLOAD,
    361+                 /*exp_success=*/true,
    362+                 /*exp_request_more=*/true,
    363+                 /*exp_headers_size=*/0);
    364+
    365+    ASSERT_DEBUG_LOG(BUFFER_SIZE_WARNING);
    366+    CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    367+                 state, State::FINAL,
    368+                 /*exp_success=*/true,
    369+                 /*exp_request_more=*/false,
    370+                 /*exp_headers_size=*/chain1.size());
    371 }
    372 
    373-static void TooBigBuffer(const CBlockIndex* chain_start,
    374-        const std::vector<CBlockHeader>& first_chain)
    375+BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, RegTestingSetup)
    376+
    377+BOOST_AUTO_TEST_CASE(headers_sync_state)
    378 {
    379-    // Intentionally too big redownload buffer in order to trigger warning.
    380-    HeadersSyncState hss{/*id=*/0,
    381-                         Params().GetConsensus(),
    382-                         HeadersSyncParams{
    383-                             .commitment_period = COMMITMENT_PERIOD,
    384-                             .redownload_buffer_size = first_chain.size(),
    385-                         },
    386-                         chain_start,
    387-                         /*minimum_required_work=*/CHAIN_WORK};
    388-    (void)hss.ProcessNextHeaders(first_chain, true);
    389-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    390-
    391-    {
    392-        ASSERT_DEBUG_LOG(BUFFER_SIZE_WARNING);
    393-        auto result{hss.ProcessNextHeaders(first_chain, true)};
    394-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    395-        BOOST_CHECK(result.success);
    396-    }
    397+    const auto& genesis{Params().GenesisBlock()};
    398+    const uint256 genesis_hash{genesis.GetHash()};
    399+
    400+    // Generate headers for two different chains (using differing merkle roots
    401+    // to ensure the headers are different).
    402+    std::vector<CBlockHeader> chain1{};
    403+    GenerateHeaders(chain1, TARGET_BLOCKS - 1, genesis_hash, genesis.nVersion, genesis.nTime,
    404+                    /*merkle_root=*/uint256::ZERO, genesis.nBits);
    405+    std::vector<CBlockHeader> chain2{};
    406+    GenerateHeaders(chain2, TARGET_BLOCKS - 2, genesis_hash, genesis.nVersion, genesis.nTime,
    407+                    /*merkle_root=*/uint256::ONE, genesis.nBits);
    408+
    409+    const CBlockIndex* chain_start{WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(genesis_hash))};
    410+    SneakyRedownload(chain_start, chain1, chain2);
    411+    HappyPath(chain_start, chain1);
    412+    TooLittleWork(chain_start, chain2);
    413+    TooBigBuffer(chain_start, chain1);
    414 }
    415 
    416 BOOST_AUTO_TEST_SUITE_END()
    417diff --git a/src/test/util/logging.h b/src/test/util/logging.h
    418index 73ac23825f..622ff58938 100644
    419--- a/src/test/util/logging.h
    420+++ b/src/test/util/logging.h
    421@@ -36,6 +36,16 @@ public:
    422     ~DebugLogHelper() { check_found(); }
    423 };
    424 
    425+// Test fails if the pattern *DOES NOT* show up.
    426 #define ASSERT_DEBUG_LOG(message) DebugLogHelper UNIQUE_NAME(debugloghelper)(message)
    427+// Test fails if the pattern *DOES* show up.
    428+#define ASSERT_NO_DEBUG_LOG(message)                                              \
    429+    DebugLogHelper UNIQUE_NAME(nologhelper){                                      \
    430+        message,                                                                  \
    431+        [](const std::string* line) {                                             \
    432+            if (line) throw std::runtime_error("unexpected log line: " + *line);  \
    433+            return false; /* suppress default 'not found' failure */              \
    434+        }                                                                         \
    435+    }
    436 
    437 #endif // BITCOIN_TEST_UTIL_LOGGING_H
    
  55. hodlinator force-pushed on Jun 23, 2025
  56. hodlinator force-pushed on Jun 23, 2025
  57. DrahtBot added the label CI failed on Jun 23, 2025
  58. hodlinator commented at 5:49 pm on June 23, 2025: contributor

    Thanks for the extensive review @l0rinc! Implemented many of the suggestions and answered the other points.

    Removed the commit breaking out scopes in favor of the suggested CHECK_RESULT macro (https://github.com/bitcoin/bitcoin/pull/32579#discussion_r2158508969). Broke out the comment improvements into their own commit ahead of that since it gives a “skeleton” for the next commit’s diff to hang off of.

  59. hodlinator force-pushed on Jun 23, 2025
  60. DrahtBot removed the label CI failed on Jun 23, 2025
  61. in src/test/headers_sync_chainwork_tests.cpp:36 in 8c96be3306 outdated
    31+        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), exp_headers_size);                          \
    32+        std::optional<uint256> locator_hash_opt{exp_locator_hash};                                         \
    33+        if (locator_hash_opt == std::nullopt) {                                                            \
    34+            BOOST_CHECK_EQUAL(exp_state, State::FINAL);                                                    \
    35+        } else {                                                                                           \
    36+            BOOST_REQUIRE_GT(hss.NextHeadersRequestLocator().vHave.size(), 0);                             \
    


    l0rinc commented at 2:33 pm on June 24, 2025:
    would’t an .at(0) suffice?

    hodlinator commented at 7:56 pm on June 25, 2025:
    Slightly more confusing error, but like that it’s more terse. Taken.

    l0rinc commented at 2:28 pm on June 27, 2025:
    I think it applies to BOOST_REQUIRE_GT(result.pow_validated_headers.size(), 0); as well

    hodlinator commented at 7:36 am on June 28, 2025:
    True, sole change in latest push.
  62. in src/test/headers_sync_chainwork_tests.cpp:171 in 8c96be3306 outdated
    176-    // Nothing left for the sync logic to do:
    177-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    178+    CHECK_RESULT(hss.ProcessNextHeaders(first_chain, /*full_headers_message=*/true),
    179+        hss, /*exp_state=*/State::REDOWNLOAD,
    180+        /*exp_success*/true, /*exp_request_more=*/true,
    181+        // The locator should reset to genesis.
    


    l0rinc commented at 2:35 pm on June 24, 2025:
    this comment is obvious from the code now: /*exp_locator_hash=*/Params().GenesisBlock().GetHash() or /*exp_locator_hash=*/genesis_hash later in 65a96b507c11b3f34efa996919892fa1a2fcf49c
  63. in src/kernel/chainparams.h:71 in 1032f52791 outdated
    66+ */
    67+struct HeadersSyncParams {
    68+    //! Distance in blocks between header commitments.
    69+    size_t commitment_period;
    70+    //! Only feed headers to validation once this many headers on top have been
    71+    //! received and validated against commitments.
    


    l0rinc commented at 2:39 pm on June 24, 2025:
    I’m not sure what “this many headers on top have been received”

    hodlinator commented at 8:11 pm on June 25, 2025:

    Agree. New attempt:

    0//! Only start outputting headers once this many headers have been received
    1//! and validated against commitments.
    
  64. in src/test/headers_sync_chainwork_tests.cpp:189 in 65a96b507c outdated
    187+    // headers shouldn't be returned yet:
    188+    CHECK_RESULT(hss.ProcessNextHeaders({first_chain.begin(), REDOWNLOAD_BUFFER_SIZE}, /*full_headers_message=*/true),
    189+        // Not done:
    190+        hss, /*exp_state=*/State::REDOWNLOAD,
    191+        /*exp_success*/true, /*exp_request_more=*/true,
    192+        // The locator should reset to genesis.
    


    l0rinc commented at 2:42 pm on June 24, 2025:
    is this comment accurate?

    hodlinator commented at 8:01 pm on June 25, 2025:
    No, rebase mishap. Thanks for catching!
  65. in src/test/headers_sync_chainwork_tests.cpp:115 in 14e933ab6a outdated
    111@@ -109,6 +112,9 @@ static void HappyPath(const CBlockIndex*, const std::vector<CBlockHeader>&);
    112 // 4. Repeat the second set of headers in both phases to demonstrate behavior
    113 //    when the chain a peer provides has too little work.
    114 static void TooLittleWork(const CBlockIndex*, const std::vector<CBlockHeader>&);
    115+// 5. Sets the redownload buffer size to be large enough large enough that we
    


    l0rinc commented at 2:47 pm on June 24, 2025:
    0// 5. Sets the redownload buffer size to be large enough that we
    
  66. in src/test/headers_sync_chainwork_tests.cpp:259 in 14e933ab6a outdated
    243@@ -234,4 +244,28 @@ static void TooLittleWork(const CBlockIndex* chain_start,
    244         /*exp_locator_hash=*/std::nullopt);
    245 }
    246 
    247+static void TooBigBuffer(const CBlockIndex* chain_start,
    248+        const std::vector<CBlockHeader>& first_chain)
    249+{
    250+    // Intentionally too big redownload buffer in order to trigger warning.
    


    l0rinc commented at 2:49 pm on June 24, 2025:
    nice, this is exactly how comments should be used to augment what the code cannot easily tell <3
  67. in src/test/headers_sync_chainwork_tests.cpp:261 in 14e933ab6a outdated
    256+
    257+    CHECK_RESULT(hss.ProcessNextHeaders(first_chain, true),
    258+        hss, /*exp_state=*/State::REDOWNLOAD,
    259+        /*exp_success*/true, /*exp_request_more=*/true,
    260+        /*exp_headers_size=*/0, /*exp_pow_validated_prev=*/std::nullopt,
    261+        /*exp_locator_hash=*/Params().GenesisBlock().GetHash());
    


    l0rinc commented at 2:51 pm on June 24, 2025:
    might make sense to extract genesis hash here as well like we did in HappyPath
  68. l0rinc approved
  69. l0rinc commented at 2:53 pm on June 24, 2025: contributor
    I find the current layout of tiny, focused commits a lot easier to follow, thanks! Left a few comments, after that I will test it again locally and I can ACK.
  70. hodlinator force-pushed on Jun 25, 2025
  71. in src/arith_uint256.h:247 in fab6f59e2e outdated
    246-    arith_uint256() = default;
    247-    arith_uint256(const base_uint<256>& b) : base_uint<256>(b) {}
    248-    arith_uint256(uint64_t b) : base_uint<256>(b) {}
    249+    constexpr arith_uint256() = default;
    250+    constexpr arith_uint256(const base_uint& b) : base_uint(b) {}
    251+    constexpr arith_uint256(uint64_t b) : base_uint(b) {}
    


    danielabrozzoni commented at 3:09 pm on June 27, 2025:
    Just to confirm that I understand this correctly, are you removing the <256> because it’s implicit?

    l0rinc commented at 4:35 pm on July 2, 2025:
    Yes, they’re redundant, can be deduced by the compiler
  72. hodlinator force-pushed on Jun 28, 2025
  73. in src/test/headers_sync_chainwork_tests.cpp:162 in 79bdc6a785 outdated
    145     // This chain should look valid, and we should have met the proof-of-work
    146     // requirement during PRESYNC and transitioned to REDOWNLOAD.
    147-    BOOST_CHECK(result.success);
    148-    BOOST_CHECK(result.request_more);
    149-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    150+    CHECK_RESULT(hss.ProcessNextHeaders(std::span{first_chain}.last(first_chain.size() - 1), true),
    


    danielabrozzoni commented at 2:37 pm on July 2, 2025:
    nit in 79bdc6a785bca96dababc698336c04b8625b7d1c: in some CHECK_RESULT you are putting /*full_headers_message=*/, in some others you’re not, like this one, the one below, and the ones in TooLittleWork

    hodlinator commented at 7:29 am on July 7, 2025:

    In this case we have the comment right above describing the same thing: https://github.com/bitcoin/bitcoin/blob/79bdc6a785bca96dababc698336c04b8625b7d1c/src/test/headers_sync_chainwork_tests.cpp#L140

    Went through the code and realized HappyPath was repeating /*full_headers_message=*/ in one piece of newer code, so removed that.

    This boolean parameter to ProcessNextHeaders() is somewhat of a code smell. Maybe could be split into 2 public functions (ProcessNextHeaders() + ProcessLastHeaders()… but that would make the code as a whole more convoluted I think).

  74. in src/kernel/chainparams.h:70 in ced2ddb37c outdated
    65+ * Used to configure Headers Sync memory usage.
    66+ */
    67+struct HeadersSyncParams {
    68+    //! Distance in blocks between header commitments.
    69+    size_t commitment_period;
    70+    //! Only start outputting headers once this many headers have been received
    


    danielabrozzoni commented at 3:29 pm on July 2, 2025:

    nit in ced2ddb37c957e53d4a8aa40d3512ea3786511b4: the previous comment (in headerssync.cpp) said: “Only feed headers to validation once this many headers..”

    I think that phrasing was a bit clearer than “outputting”


    hodlinator commented at 7:20 am on July 7, 2025:

    Changed to the following, let me know what you have further suggestions:

    Start feeding back headers into the permanent block index once more than this number of them have been received and validated against commitments.

  75. in src/test/headers_sync_chainwork_tests.cpp:38 in 79bdc6a785 outdated
    33+        if (locator_hash_opt == std::nullopt) {                                                          \
    34+            BOOST_CHECK_EQUAL(exp_state, State::FINAL);                                                  \
    35+        } else {                                                                                         \
    36+            BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.at(0), locator_hash_opt);            \
    37+        }                                                                                                \
    38+        std::optional<uint256> pow_validated_prev_opt{exp_pow_validated_prev};                           \
    


    danielabrozzoni commented at 4:06 pm on July 2, 2025:
    micro nit in 79bdc6a785bca96dababc698336c04b8625b7d1c: you could check exp_pow_validated_prev first and exp_locator_hash second, to maintain the same order as the macro parameters

    hodlinator commented at 7:18 am on July 7, 2025:
    Agree that’s cleaner, fixed! Also changed from == std::nullopt to .has_value().

    l0rinc commented at 9:36 pm on August 12, 2025:

    nit: we already have opt in the name, consider:

    0if (pow_validated_prev_opt) {                                                                    \
    

    nit2: result is const, consider unifying:

    0        const std::optional<uint256> pow_validated_prev_opt{exp_pow_validated_prev};                     \
    
  76. danielabrozzoni commented at 4:28 pm on July 2, 2025: contributor

    Concept ACK, I’ve left a few questions/nits

    I’ve only done a partial review so far, I still need to go through the last two commits and part of ced2ddb37c957e53d4a8aa40d3512ea3786511b4

  77. in src/test/headers_sync_chainwork_tests.cpp:133 in c3bc47cfb6 outdated
    133     // Generate headers for two different chains (using differing merkle roots
    134     // to ensure the headers are different).
    135-    GenerateHeaders(first_chain, target_blocks-1, Params().GenesisBlock().GetHash(),
    136-            Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    137-            ArithToUint256(0), Params().GenesisBlock().nBits);
    138+    GenerateHeaders(first_chain, TARGET_BLOCKS - 1, genesis.GetHash(), genesis.nVersion,
    


    danielabrozzoni commented at 1:26 pm on July 3, 2025:

    It took me a while to figure this out - I was initially wondering why the chains have TARGET_BLOCKS - 1 and TARGET_BLOCKS - 2 blocks, until I realized that I wasn’t counting the genesis block 😅

    I’m not sure if it might be worth it to add a comment like // The first chain will contain TARGET_BLOCKS, the second one TARGET_BLOCKS - 1

    It might be obvious already, feel free to drop the suggestion


    hodlinator commented at 9:50 pm on July 14, 2025:

    It’s not obvious to me, thanks for sending me down a rabbit hole. :) Please correct me if I’m wrong but this is my refreshed understanding:

    • CBlockHeader::nBits is the PoW target for the current difficulty period (in compact int format). https://learnmeabitcoin.com/technical/block/bits/
    • GetBlockProof() is somewhat terse in comments (since inception in df9eb5e14fa8072bc8a82b59e712c2ba36f13f4c), but as I understand it: Given a target nBits, it returns an estimate for how many SHA256 hash-operations on average are required to mine a given block.
    • Total PoW of a chain, CBlockIndex::nChainWork, is actually the sum of the targets, not the actual block hashes. This somewhat narrows my perception of claims I’ve heard previously that Bitcoin follows the highest PoW chain rather than the longest chain (so it’s true as far as difficulty periods go, but not on a per-block level). This has the gambler side of me a bit disappointed, but okay. https://github.com/bitcoin/bitcoin/blob/78503a41546f6432045d3f9fd39860ead11d5c95/src/node/blockstorage.cpp#L227

    Based on the above, I’ve added comments to the “doc(test): Improve comments”-commit.


    danielabrozzoni commented at 10:47 am on July 24, 2025:
    Thanks a lot for the explanation, and the updated comments! It’s much more understandable now.

    l0rinc commented at 7:42 pm on August 12, 2025:

    It also confused me for a second but for a different reason, I though they’re heights, maybe adding a count hint

    0GenerateHeaders(first_chain, /*count=*/TARGET_BLOCKS - 1, genesis.GetHash(), genesis.nVersion,
    

    would help.

  78. in src/test/headers_sync_chainwork_tests.cpp:25 in ced2ddb37c outdated
    19@@ -19,6 +20,8 @@ using State = HeadersSyncState::State;
    20 
    21 constexpr size_t TARGET_BLOCKS{15'000};
    22 constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
    23+constexpr size_t REDOWNLOAD_BUFFER_SIZE{TARGET_BLOCKS - (MAX_HEADERS_RESULTS + 123)};
    24+constexpr size_t COMMITMENT_PERIOD{600};
    


    danielabrozzoni commented at 1:52 pm on July 3, 2025:

    A couple of questions regarding these two:

    • REDOWNLOAD_BUFFER_SIZE: can we add a comment explaining why subtracting MAX_HEADERS_RESULT + 123? Copy-pasting what’s in the commit description is ok
    • COMMITMENT_PERIOD = 600: I suppose this is just mimicking the mainnet value?

    hodlinator commented at 9:14 pm on July 14, 2025:
    Good call - added comments and removed the redundant part from the commit message.
  79. hodlinator force-pushed on Jul 7, 2025
  80. hodlinator commented at 7:35 am on July 7, 2025: contributor
    Thanks for checking this out @danielabrozzoni! Pushed some minor changes based on your feedback.
  81. danielabrozzoni commented at 2:58 pm on July 9, 2025: contributor

    Review 78503a41546f6432045d3f9fd39860ead11d5c95

    Code looks good to me, tests are passing locally. Thanks for picking up my comments! I left a couple more.

    I did some additional testing by cherry-picking the latest commit of this PR and applying it to master, setting REDOWNLOAD_BUFFER_SIZE to >15000, and verifying that the warning was logged as expected.

     0diff --git a/src/headerssync.cpp b/src/headerssync.cpp
     1index 9e8b190516..0d0a26d5db 100644
     2--- a/src/headerssync.cpp
     3+++ b/src/headerssync.cpp
     4@@ -17,7 +17,7 @@ constexpr size_t HEADER_COMMITMENT_PERIOD{624};
     5 
     6 //! Only feed headers to validation once this many headers on top have been
     7 //! received and validated against commitments.
     8-constexpr size_t REDOWNLOAD_BUFFER_SIZE{14827}; // 14827/624 = ~23.8 commitments
     9+constexpr size_t REDOWNLOAD_BUFFER_SIZE{15002}; // 14827/624 = ~23.8 commitments
    10 
    11 // Our memory analysis assumes 48 bytes for a CompressedHeader (so we should
    12 // re-calculate parameters if we compress further)
    13@@ -285,11 +285,20 @@ std::vector<CBlockHeader> HeadersSyncState::PopHeadersReadyForAcceptance()
    14     Assume(m_download_state == State::REDOWNLOAD);
    15     if (m_download_state != State::REDOWNLOAD) return ret;
    16 
    17-    while (m_redownloaded_headers.size() > REDOWNLOAD_BUFFER_SIZE ||
    18+    bool buffer_exceeded{m_redownloaded_headers.size() > REDOWNLOAD_BUFFER_SIZE};
    19+    if (!buffer_exceeded && m_process_all_remaining_headers && m_chain_start->nHeight == 0 &&
    20+            m_redownload_buffer_first_prev_hash == m_chain_start->GetBlockHash()) {
    21+        LogWarning("Unexpectedly reached minimum required work before filling the headers sync redownload buffer, even though we started all the way back from genesis "
    22+                   "(m_redownload_buffer_last_height: %u, redownload_buffer_size: %u).",
    23+                   m_redownload_buffer_last_height, REDOWNLOAD_BUFFER_SIZE);
    24+    }
    25+
    26+    while (buffer_exceeded ||
    27             (m_redownloaded_headers.size() > 0 && m_process_all_remaining_headers)) {
    28         ret.emplace_back(m_redownloaded_headers.front().GetFullHeader(m_redownload_buffer_first_prev_hash));
    29         m_redownloaded_headers.pop_front();
    30         m_redownload_buffer_first_prev_hash = ret.back().GetHash();
    31+        buffer_exceeded = m_redownloaded_headers.size() > REDOWNLOAD_BUFFER_SIZE;
    32     }
    33     return ret;
    34 }
    35diff --git a/src/test/headers_sync_chainwork_tests.cpp b/src/test/headers_sync_chainwork_tests.cpp
    36index b710ad1801..a14aa0ee6d 100644
    37--- a/src/test/headers_sync_chainwork_tests.cpp
    38+++ b/src/test/headers_sync_chainwork_tests.cpp
    39@@ -7,6 +7,7 @@
    40 #include <consensus/params.h>
    41 #include <headerssync.h>
    42 #include <pow.h>
    43+#include <test/util/logging.h>
    44 #include <test/util/setup_common.h>
    45 #include <validation.h>
    46 #include <vector>
    47@@ -65,8 +66,12 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
    48 //    processing (presumably due to commitments not matching).
    49 // 3. Finally, we verify that repeating with the first set of headers in both
    50 //    phases is successful.
    51+
    52+constexpr char BUFFER_SIZE_WARNING[]{"Unexpectedly reached minimum required work before filling the headers sync redownload buffer, even though we started all the way back from genesis"};
    53+
    54 BOOST_AUTO_TEST_CASE(headers_sync_state)
    55 {
    56+    ASSERT_NO_DEBUG_LOG(BUFFER_SIZE_WARNING);
    57     std::vector<CBlockHeader> first_chain;
    58     std::vector<CBlockHeader> second_chain;
    59 
    60diff --git a/src/test/util/logging.h b/src/test/util/logging.h
    61index 73ac23825f..af7110b279 100644
    62--- a/src/test/util/logging.h
    63+++ b/src/test/util/logging.h
    64@@ -36,6 +36,17 @@ public:
    65     ~DebugLogHelper() { check_found(); }
    66 };
    67 
    68+// Test fails if the pattern *DOES NOT* show up.
    69 #define ASSERT_DEBUG_LOG(message) DebugLogHelper UNIQUE_NAME(debugloghelper)(message)
    70 
    71+// Test fails if the pattern *DOES* show up.
    72+#define ASSERT_NO_DEBUG_LOG(message)                                                        \
    73+    DebugLogHelper UNIQUE_NAME(nologhelper){                                                \
    74+        message,                                                                            \
    75+        [](const std::string* line) {                                                       \
    76+            if (line) throw std::runtime_error{"Encountered forbidden log line: " + *line}; \
    77+            return false; /* Suppress default 'not found' failure. */                       \
    78+        }                                                                                   \
    79+    }
    80+
    81 #endif // BITCOIN_TEST_UTIL_LOGGING_H	
    

    Test output:

    0./build/bin/test_bitcoin --run_test=headers_sync_chainwork_tests
    1Running 1 test case...
    2unknown location(0): fatal error: in "headers_sync_chainwork_tests/headers_sync_state": std::runtime_error: Encountered forbidden log line: 2025-07-09T14:20:58.691984Z [test] [headerssync.cpp:291] [PopHeadersReadyForAcceptance] [warning] Unexpectedly reached minimum required work before filling the headers sync redownload buffer, even though we started all the way back from genesis (m_redownload_buffer_last_height: 14999, redownload_buffer_size: 15002).
    3
    4./test/headers_sync_chainwork_tests.cpp(119): last checkpoint
    5
    6*** 1 failure is detected in the test module "Bitcoin Core Test Suite"
    7error: Recipe `unit-test` failed on line 13 with exit code 201
    
  82. hodlinator force-pushed on Jul 14, 2025
  83. hodlinator commented at 10:05 pm on July 14, 2025: contributor
    Thanks @danielabrozzoni for your latest feedback and testing against master with your diff! Added some comments in latest push.
  84. achow101 added this to the milestone 30.0 on Aug 7, 2025
  85. hodlinator force-pushed on Aug 9, 2025
  86. hodlinator commented at 10:46 pm on August 9, 2025: contributor

    Changes with latest push (compare):

    • Clarified PR description, hopefully better motivation reason for v30 milestone inclusion.
    • Moved checks for behavior around pow_validated_headers and REDOWNLOAD_BUFFER_SIZE to first commit to motivate the rest of the PR instead of being one of the last commits.
    • Simplified condition emitting warning for misconfigured REDOWNLOAD_BUFFER_SIZE.
  87. achow101 commented at 9:19 pm on August 11, 2025: member

    It seems like this PR has several refactors that are really necessary for the goal of getting a configurable redownload window into HeadersSyncState.

    I find the last commit to not be compelling. Users are never going to see that warning, and if they did, there’s nothing actionalbe for them to do. It’s only a warning that is relevant to developers, and even then, it is not done in a way that is extremely obvious to developers as it’s only a warning and not a fatal error. When we add checks for developers, they should really be something that causes a crash for developers, e.g. Assume. And in that case, there also doesn’t need to be a unit test case for that either.

    Lastly, changes are made to the chain params, but with no instructions of how or when to update the parameters for the other chains. And really, if the expectation is that these parameters do need to be updated for the other chains, then the headerssync-params.py script needs to be made easier to run for those other chains.

  88. hodlinator force-pushed on Aug 12, 2025
  89. hodlinator commented at 8:15 am on August 12, 2025: contributor

    Thanks for your feedback @achow101!

    I find the last commit to not be compelling.

    Dropped 538b9a4f4ee84749bd1b2fe32e61658c272ee7f9 as I agree the warning was a bit contrived, and the change feels less necessary, given how the PR has evolved.

    Considered changing the first commit to add the following in HeadersSyncState::PopHeadersReadyForAcceptance():

    0    // We haven't started feeding back headers from genesis but already reached
    1    // the PoW threshold?! This is an unrealistic situation, maybe a fuzz test?
    2    Assume(!(m_process_all_remaining_headers && m_chain_start->nHeight == 0 &&
    3             m_redownload_buffer_first_prev_hash == m_chain_start->GetBlockHash()));
    

    But it would probably cause the fuzz test to fail.

    … no instructions of how or when to update the parameters for the other chains. And really, if the expectation is that these parameters do need to be updated for the other chains, then the headerssync-params.py script needs to be made easier to run for those other chains.

    Added documentation to the commit message about how the HeadersSyncParams constants were calculated and argued for them not needing to be frequently updated. I could add comments with UTC dates corresponding to genesis block Unix epoch times for all CreateGenesisBlock() calls in kernel/chainparams.cpp to make it easier to re-run the script if deemed necessary.

  90. in src/test/headers_sync_chainwork_tests.cpp:116 in 57a351912b outdated
    112@@ -113,11 +113,28 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    113     (void)hss->ProcessNextHeaders(first_chain, true);
    114     BOOST_CHECK(hss->GetState() == HeadersSyncState::State::REDOWNLOAD);
    115 
    116-    result = hss->ProcessNextHeaders(first_chain, true);
    117+    // Process so that the internal threshold isn't met, meaning validated
    


    l0rinc commented at 6:54 pm on August 12, 2025:
    57a351912b057d1a2234d4a3b3dd0c745690e3d0: this new addition temporarily invalidates the introductory comment describing the workings of headers_sync_state

    hodlinator commented at 8:09 pm on August 12, 2025:

    Are you referring to this? https://github.com/bitcoin/bitcoin/blob/57a351912b057d1a2234d4a3b3dd0c745690e3d0/src/test/headers_sync_chainwork_tests.cpp#L66-L67

    We are feeding in the same chain twice, it’s just that we cut it up in smaller pieces.


    l0rinc commented at 9:43 pm on August 12, 2025:
    you’re updating the comments in a later commit, adding the “happy path”, but the code that corresponds to it was added here

    hodlinator commented at 8:59 pm on August 13, 2025:
    Hoping the change to the happy path is clearer now with the extracted final commit.
  91. in src/test/headers_sync_chainwork_tests.cpp:131 in 57a351912b outdated
    127+    BOOST_CHECK_EQUAL(hss->GetState(), HeadersSyncState::State::REDOWNLOAD);
    128+    BOOST_CHECK_GT(result.pow_validated_headers.size(), 0);
    129+    const size_t validated_headers_count = result.pow_validated_headers.size();
    130+
    131+    // Feed final header, meeting the work threshold again and
    132+    // completing the REDOWNLOAD phase.
    


    l0rinc commented at 6:55 pm on August 12, 2025:
    nit: joining the lines would reduce the noise, we have longer lines than that - which would still be below 100 chars anyway nit2: in other cases we ended with a :, not a .
  92. in src/test/headers_sync_chainwork_tests.cpp:137 in 57a351912b outdated
    134+    result = hss->ProcessNextHeaders(headers_batch, true);
    135     BOOST_CHECK(result.success);
    136     BOOST_CHECK(!result.request_more);
    137     // All headers should be ready for acceptance:
    138-    BOOST_CHECK(result.pow_validated_headers.size() == first_chain.size());
    139+    BOOST_CHECK_EQUAL(validated_headers_count + result.pow_validated_headers.size(), first_chain.size());
    


    l0rinc commented at 6:58 pm on August 12, 2025:
    👍, test passes with REDOWNLOAD_BUFFER_SIZE{15827} without this commit, fails after the addition, as expected

    hodlinator commented at 9:11 pm on August 12, 2025:

    (This is the check that fails for me specifically): https://github.com/bitcoin/bitcoin/blob/57a351912b057d1a2234d4a3b3dd0c745690e3d0/src/test/headers_sync_chainwork_tests.cpp#L127

    0../src/test/headers_sync_chainwork_tests.cpp(127): error: in "headers_sync_chainwork_tests/headers_sync_state": check result.pow_validated_headers.size() > 0 has failed [0 <= 0]
    

    marcofleon commented at 3:28 pm on August 13, 2025:
    Same check fails for me too 🤝
  93. in src/arith_uint256.h:40 in d9cf9c7c2a outdated
    39         for (int i = 0; i < WIDTH; i++)
    40             pn[i] = 0;
    41     }
    42 
    43-    base_uint(const base_uint& b)
    44+    constexpr base_uint(const base_uint& b)
    


    l0rinc commented at 7:02 pm on August 12, 2025:
    nit: because of these changes the commit message should likely be adjusted: the refactor is not test-only anymore
  94. in src/test/headers_sync_chainwork_tests.cpp:40 in d9cf9c7c2a outdated
    51         size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
    52         const uint256& merkle_root, const uint32_t nBits)
    53 {
    54     uint256 prev_hash = starting_hash;
    55 
    56     while (headers.size() < count) {
    


    l0rinc commented at 7:24 pm on August 12, 2025:

    d9cf9c7c2a1cbe9e99b3b5e641b3a94a149c8ee8 nit: since we’re editing this already, consider removing the primitive consts:

    0        size_t count, const uint256& starting_hash, int32_t nVersion, uint32_t prev_time,
    1        const uint256& merkle_root, uint32_t nBits)
    

    slightly unrelated: this is a fixed iteration, it implicitly assumes headers is empty when we get here - maybe we can make that clearer here or in another follow-up by explicitly iterating from 0 or headers.size() to count instead:

     0void HeadersGeneratorSetup::GenerateHeaders(
     1    std::vector<CBlockHeader>& headers,
     2    size_t count, const uint256& starting_hash, int32_t nVersion, uint32_t prev_time,
     3    const uint256& merkle_root, uint32_t nBits)
     4{
     5    for (size_t i{0}; i < count; ++i) {
     6        auto prev_hash{i ? headers.back().GetHash() : starting_hash};
     7        auto& next_header{headers.emplace_back()};
     8        next_header.nVersion = nVersion;
     9        next_header.hashPrevBlock = prev_hash;
    10        next_header.hashMerkleRoot = merkle_root;
    11        next_header.nTime = ++prev_time;
    12        next_header.nBits = nBits;
    13
    14        FindProofOfWork(next_header);
    15    }
    16}
    

    Or since we’re already reassigning arguments:

     0void HeadersGeneratorSetup::GenerateHeaders(
     1    std::vector<CBlockHeader>& headers,
     2    size_t count, uint256 prev_hash, int32_t nVersion, uint32_t prev_time,
     3    const uint256& merkle_root, uint32_t nBits)
     4{
     5    for (size_t i{0}; i < count; ++i) {
     6        auto& next_header{headers.emplace_back()};
     7        next_header.nVersion = nVersion;
     8        next_header.hashPrevBlock = prev_hash;
     9        next_header.hashMerkleRoot = merkle_root;
    10        next_header.nTime = ++prev_time;
    11        next_header.nBits = nBits;
    12
    13        FindProofOfWork(next_header);
    14        prev_hash = next_header.GetHash();
    15    }
    16}
    

    What’s changed:

    • fixed for loop, since we know exactly the number of iterations;
    • prev_hash can be localized, we don’t need to update it based on the iteration’s state;
    • or prev_hash arg can be copied and reassigned, like we did with prev_time already;
    • prev_time is basically just incremented, no need to assign it back and forth;
    • no point in emplace_back + back(), let’s just store the result of the emplace;
    • edit: nVersion and nTime should be a int32_t instead.

    Maybe this reduces the need for such a detailed documentation as well (e.g. the advance of 1 second is obvious now - it wasn’t obvious before)


    hodlinator commented at 8:59 pm on August 12, 2025:

    Incorporated many of your suggestions, latest version:

    https://github.com/bitcoin/bitcoin/blob/a4191b57b21a0fea871206f368972c7741ad6437/src/test/headers_sync_chainwork_tests.cpp#L60-L80

    • Return value enables making first_chain/second_chain const from the beginning.
    • Looping over headers means we don’t need the explicit i.
    • Still keeping const on primitive arguments which we do not modify within function body, such as nVersion. (Agree that they don’t belong if we had a separate function declaration without the body).
    • Shortened the comment about advancing 1 second.

    l0rinc commented at 9:44 pm on August 12, 2025:
    hah, that’s even better! I still recommend changing prev_time and nVersion to uint32_t

    hodlinator commented at 8:42 pm on August 13, 2025:
    Adjusted - CBlockHeader::nVersion is int32_t however, so matched that.
  95. in src/test/headers_sync_chainwork_tests.cpp:72 in d9cf9c7c2a outdated
    68@@ -72,27 +69,23 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    69 
    70     std::unique_ptr<HeadersSyncState> hss;
    71 
    72-    const int target_blocks = 15000;
    73-    arith_uint256 chain_work = target_blocks*2;
    74+    const auto genesis{Params().GenesisBlock()};
    


    l0rinc commented at 7:33 pm on August 12, 2025:

    nit:

    0    const auto& genesis{Params().GenesisBlock()};
    

    nit2: we may want to extract its hash as well


    hodlinator commented at 9:07 pm on August 12, 2025:
    Added ampersand but keeping the 3 GetHash() calls for now.
  96. in src/test/headers_sync_chainwork_tests.cpp:54 in d9cf9c7c2a outdated
    31-void HeadersGeneratorSetup::FindProofOfWork(CBlockHeader& starting_header)
    32+constexpr size_t TARGET_BLOCKS{15'000};
    33+constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
    34+
    35+/** Search for a nonce to meet (regtest) proof of work */
    36+static void FindProofOfWork(CBlockHeader& starting_header)
    


    l0rinc commented at 7:45 pm on August 12, 2025:
    not sure what “starting” refers to here, since we’re mutating it, so it will also store all intermediary and final header states as well

    hodlinator commented at 9:04 pm on August 12, 2025:
    Name could be better, but keeping it for now as it helps make the diff somewhat less jarring.
  97. in src/test/headers_sync_chainwork_tests.cpp:221 in 058ebc1b02 outdated
    152+static void TooLittleWork(const CBlockIndex* chain_start,
    153+        const std::vector<CBlockHeader>& second_chain)
    154+{
    155+    std::unique_ptr<HeadersSyncState> hss;
    156+    HeadersSyncState::ProcessingResult result;
    157+    // Verify that just trying to process the second chain would not succeed
    


    l0rinc commented at 7:51 pm on August 12, 2025:
    058ebc1b02ccfc75bbe5178a4191aebdcf9b4d3a: might make sense to phrase the comment like this in the first place to minimize diffs

    hodlinator commented at 9:05 pm on August 12, 2025:
    Postponed updating this and one other comment until the “Improve comments” commit.

    l0rinc commented at 9:45 pm on August 12, 2025:
    Didn’t we just add this comment a few commits before? I meant it may make sense to phrase it such that we don’t need changing later

    hodlinator commented at 8:34 pm on August 13, 2025:

    Hm… (too trigger happy with publishing)

    Didn’t we just add this comment a few commits before?

    No

    Based on your initial comment in this thread, I undid modifying the comment from master until the Improve comments commit (0863bde64e62b040da749161f7bd9c5628759a18, this aspect is unchanged from previous push).

  98. l0rinc changes_requested
  99. l0rinc commented at 7:52 pm on August 12, 2025: contributor
    Code reviewed until 058ebc1b02ccfc75bbe5178a4191aebdcf9b4d3a
  100. test(headerssync): Test returning of pow_validated_headers behavior
    This commit by itself can be used in combination with locally modifying REDOWNLOAD_BUFFER_SIZE in headerssync.cpp to >= 15'000 (the value of target_blocks in headers_sync_chainwork_tests.cpp) to illustrate that we will soon start testing weird behavior unless we implement something like the rest of this PR.
    
    (The release process includes an occasional increase of the REDOWNLOAD_BUFFER_SIZE value, see release-process.md and history of headerssync.cpp).
    fa3d88ac35
  101. in src/test/headers_sync_chainwork_tests.cpp:83 in 14f5c34ece outdated
    51@@ -52,6 +52,14 @@ static void GenerateHeaders(std::vector<CBlockHeader>& headers,
    52     }
    53 }
    54 
    55+static HeadersSyncState CreateState(const CBlockIndex* chain_start)
    


    l0rinc commented at 9:02 pm on August 12, 2025:

    can it ever be null or can we do static HeadersSyncState CreateState(const CBlockIndex& chain_start) by asserting the lookup:

    0    const CBlockIndex& chain_start{*Assert(WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(genesis.GetHash())))};
    

    hodlinator commented at 8:21 am on August 13, 2025:
    Would rather postpone this until #32740 (which also changes it to a reference type in the HeadersSyncState-ctor) has been merged.
  102. in src/test/headers_sync_chainwork_tests.cpp:111 in 058ebc1b02 outdated
    68+static void HappyPath(const CBlockIndex*, const std::vector<CBlockHeader>&);
    69+// 4. Repeat the second set of headers in both phases to demonstrate behavior
    70+//    when the chain a peer provides has too little work.
    71+static void TooLittleWork(const CBlockIndex*, const std::vector<CBlockHeader>&);
    72+
    73 BOOST_AUTO_TEST_CASE(headers_sync_state)
    


    l0rinc commented at 9:16 pm on August 12, 2025:
    Arguably this is where the tests start and the above methods are helpers, consider moving BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, RegTestingSetup) here

    hodlinator commented at 9:53 am on August 13, 2025:
    It’s true that many tests have BOOST_FIXTURE_TEST_SUITE directly followed by BOOST_AUTO_TEST_CASE, but these methods (TooLittleWork etc) are tightly coupled with the headers_sync_state test case. They could almost be their own test cases entirely but I’d rather avoid repeated cycles to generate the chains. (Actually (re?)tested implementing separate test cases but fixtures are re-instantiated for each test case, so runtime went from 0.39 to 0.65, added note in commit message breaking up into functions).
  103. in src/test/headers_sync_chainwork_tests.cpp:67 in 058ebc1b02 outdated
    64-//    phases is successful.
    65+static void SneakyRedownload(const CBlockIndex*, const std::vector<CBlockHeader>&, const std::vector<CBlockHeader>&);
    66+// 3. Verify that repeating with the first set of headers in both phases is
    67+//    successful.
    68+static void HappyPath(const CBlockIndex*, const std::vector<CBlockHeader>&);
    69+// 4. Repeat the second set of headers in both phases to demonstrate behavior
    


    l0rinc commented at 9:17 pm on August 12, 2025:
    nit: as mentioned, we should only move things around here, this comment should be added to where we’ve introduced the happy path

    hodlinator commented at 4:19 pm on August 13, 2025:

    I find it hard to follow, are you referring to a specific commit?

    Hopefully this is the same as #32579 (review)

  104. in src/test/headers_sync_chainwork_tests.cpp:121 in 95de558590 outdated
    87+    // roughly as the coefficient 0x7fffff with the exponent 0x20 (32 bytes).
    88+    assert(genesis.nBits == 0x207fffff);
    89+    // ...which implies around every 2nd hash attempt should succeed...
    90+    assert(GetBlockProof(CBlockIndex(genesis)) == 2);
    91+    // ...and that is where we get our minimum PoW threshold.
    92+    assert(CHAIN_WORK == TARGET_BLOCKS * 2);
    


    l0rinc commented at 9:23 pm on August 12, 2025:

    why do you think we need this whole new setup?

    I’m not sure I understand why we’re checking a constant, we’ve just assigned it to TARGET_BLOCKS * 2 a few lines before


    hodlinator commented at 8:38 pm on August 13, 2025:
    It’s my way of documenting why we are multiplying by 2. :) Comes from the research I did in this thread: #32579 (review)

    l0rinc commented at 10:04 pm on August 13, 2025:

    I don’t think it’s useful, it’s literally just stating:

    0constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
    1...
    2assert(CHAIN_WORK == TARGET_BLOCKS * 2);
    

    hodlinator commented at 6:39 am on August 15, 2025:
    Ideally the genesis-block and the other expressions would be constexpr, so these could be static_asserts right after CHAIN_WORK and TARGET_BLOCKS. But given they are 90+ lines apart, I’d rather keep it like this. Hm.. I could experiment with ways of decreasing that line distance if I re-push (extracting function bodies and putting them later, moving these constants after the initial functions).

    hodlinator commented at 10:34 am on August 19, 2025:
    Moved up the macro above the constants in the latest push, and brought the assert block up closer to the constants - in the process removing the final assert.
  105. in src/test/headers_sync_chainwork_tests.cpp:124 in 95de558590 outdated
    91+    // ...and that is where we get our minimum PoW threshold.
    92+    assert(CHAIN_WORK == TARGET_BLOCKS * 2);
    93+
    94     // Generate headers for two different chains (using differing merkle roots
    95     // to ensure the headers are different).
    96+    // - 1 since the genesis block also contributes work so we just meet the
    


    l0rinc commented at 9:27 pm on August 12, 2025:
    this looks like a list item, starting with -, not sure it helps with understanding the code better. Also, explaining TARGET_BLOCKS - 2 at the TARGET_BLOCKS - 1 line is also a bit confusing.

    marcofleon commented at 2:56 pm on August 13, 2025:
    Agreed, the - 1 and -2 part of comment was confusing to me at first.

    hodlinator commented at 8:38 pm on August 13, 2025:
    Adjusted comment to avoid list item interpretation.
  106. in src/test/headers_sync_chainwork_tests.cpp:33 in 59144d5283 outdated
    24+
    25+// Standard set of checks common to all scenarios. Macro keeps failure lines at the call-site.
    26+#define CHECK_RESULT(result_expression, hss, exp_state, exp_success, exp_request_more,                   \
    27+                     min_headers_size, exp_pow_validated_prev, exp_locator_hash)                         \
    28+    do {                                                                                                 \
    29+        const auto result = result_expression;                                                           \
    


    l0rinc commented at 9:33 pm on August 12, 2025:

    nit: to be consistent with the following lines

    0        const auto result{result_expression};                                                            \
    
  107. hodlinator force-pushed on Aug 12, 2025
  108. hodlinator commented at 9:39 pm on August 12, 2025: contributor
    Pushed updates based off #32579#pullrequestreview-3112310893.
  109. in contrib/devtools/README.md:143 in a4191b57b2 outdated
    136@@ -137,7 +137,7 @@ BUILDDIR=$PWD/my-build-dir contrib/devtools/gen-manpages.py
    137 headerssync-params.py
    138 =====================
    139 
    140-A script to generate optimal parameters for the headerssync module (src/headerssync.cpp). It takes no command-line
    141+A script to generate optimal parameters for the headerssync module (stored in src/kernel/chainparams.cpp). It takes no command-line
    142 options, as all its configuration is set at the top of the file. It runs many times faster inside PyPy. Invocation:
    143 
    144 ```bash
    


    l0rinc commented at 9:52 pm on August 12, 2025:

    👍, it seems to generate to correct values currently:

    pypy3 contrib/devtools/headerssync-params.py

     0Searching configurations:
     1- Initial: period=618, buffer=14672, mem=701.418 KiB
     2- New best: period=629, buffer=14955, mem=701.188 KiB
     3- New best: period=622, buffer=14775, mem=696.908 KiB
     4- New best: period=625, buffer=14852, mem=696.360 KiB
     5- New best: period=623, buffer=14801, mem=695.789 KiB
     6- New best: period=624, buffer=14827, mem=695.189 KiB
     7
     8Given current min chainwork headers of 886157, the optimal parameters for low
     9memory usage on mainchain for release until 2027-10-06 is:
    10
    11        // Generated by headerssync-params.py on 2025-08-12.
    12        m_headers_sync_params = HeadersSyncParams{
    13            .commitment_period = 624,
    14            .redownload_buffer_size = 14827, // 14827/624 = ~23.8 commitments
    15        };
    16
    17Properties:
    18- Per-peer memory for mainchain sync: 695.189 KiB
    19- Per-peer memory for timewarp attack: 694.674 KiB
    20- Attack rate: 31382.7 attacks for 1 header of memory growth
    21  (where each attack costs 68.454 MiB bandwidth)
    
  110. in src/headerssync.cpp:20 in a4191b57b2 outdated
    29 HeadersSyncState::HeadersSyncState(NodeId id, const Consensus::Params& consensus_params,
    30-        const CBlockIndex* chain_start, const arith_uint256& minimum_required_work) :
    31-    m_commit_offset(FastRandomContext().randrange<unsigned>(HEADER_COMMITMENT_PERIOD)),
    32+        const HeadersSyncParams& params, const CBlockIndex* chain_start,
    33+        const arith_uint256& minimum_required_work) :
    34+    m_commit_offset(FastRandomContext().randrange<unsigned>(params.commitment_period)),
    


    l0rinc commented at 9:55 pm on August 12, 2025:

    nit: given size_t commitment_period we might as well:

    0    m_commit_offset(FastRandomContext().randrange<size_t>(params.commitment_period)),
    

    hodlinator commented at 10:18 am on August 13, 2025:
    Changed HeadersSyncState::m_commit_offset from unsigned to size_t as well to remove the explicit template parameter.
  111. in src/headerssync.cpp:37 in a4191b57b2 outdated
    33@@ -41,7 +34,7 @@ HeadersSyncState::HeadersSyncState(NodeId id, const Consensus::Params& consensus
    34     // exceeds this bound, because it's not possible for a consensus-valid
    35     // chain to be longer than this (at the current time -- in the future we
    36     // could try again, if necessary, to sync a longer chain).
    37-    m_max_commitments = 6*(Ticks<std::chrono::seconds>(NodeClock::now() - NodeSeconds{std::chrono::seconds{chain_start->GetMedianTimePast()}}) + MAX_FUTURE_BLOCK_TIME) / HEADER_COMMITMENT_PERIOD;
    38+    m_max_commitments = 6*(Ticks<std::chrono::seconds>(NodeClock::now() - NodeSeconds{std::chrono::seconds{chain_start->GetMedianTimePast()}}) + MAX_FUTURE_BLOCK_TIME) / m_params.commitment_period;
    


    l0rinc commented at 9:57 pm on August 12, 2025:

    hodlinator commented at 11:49 am on August 13, 2025:
    Extracted max_seconds_since_start.
  112. in src/kernel/chainparams.h:71 in a4191b57b2 outdated
    68+    //! Distance in blocks between header commitments.
    69+    size_t commitment_period;
    70+    //! Start feeding back headers into the permanent block index once more than
    71+    //! this number of them have been received and validated against commitments.
    72+    size_t redownload_buffer_size;
    73+};
    


    l0rinc commented at 9:59 pm on August 12, 2025:

    The comment on redownload_buffer_size is quite cumbersome, starts with an instruction and keeps going in circles, please consider:

    0//! Configuration for Headers-Sync memory usage.
    1struct HeadersSyncParams
    2{
    3    //! Distance, in blocks, between header commitments.
    4    size_t commitment_period;
    5    //! Minimum number of validated headers to accumulate in the redownload
    6    //! buffer before feeding them into the permanent block index.
    7    size_t redownload_buffer_size;
    8};
    
  113. in src/headerssync.cpp:192 in a4191b57b2 outdated
    185@@ -193,7 +186,7 @@ bool HeadersSyncState::ValidateAndProcessSingleHeader(const CBlockHeader& curren
    186         return false;
    187     }
    188 
    189-    if (next_height % HEADER_COMMITMENT_PERIOD == m_commit_offset) {
    190+    if (next_height % m_params.commitment_period == m_commit_offset) {
    


    l0rinc commented at 10:03 pm on August 12, 2025:
    a4191b57b21a0fea871206f368972c7741ad6437 is a bit complicated, can you split it up to smaller refactors?

    hodlinator commented at 12:04 pm on August 13, 2025:
    I broke out many of the chainwork test changes into a later commit. Beyond that, I have a hard time untangling things elegantly.
  114. l0rinc changes_requested
  115. l0rinc commented at 10:04 pm on August 12, 2025: contributor
    Finished the review, my main concern is that the last commit is a bit hard to review (or that I got tired by the end). Left a few more nits as well.
  116. in src/kernel/chainparams.cpp:657 in a4191b57b2 outdated
    648@@ -625,6 +649,12 @@ class CRegTestParams : public CChainParams
    649         base58Prefixes[EXT_SECRET_KEY] = {0x04, 0x35, 0x83, 0x94};
    650 
    651         bech32_hrp = "bcrt";
    652+
    653+        // Copied from Testnet4.
    654+        m_headers_sync_params = HeadersSyncParams{
    655+            .commitment_period = 256,
    656+            .redownload_buffer_size = 6586, // 6586/256 = ~25.7 commitments
    657+        };
    


    marcofleon commented at 2:47 pm on August 13, 2025:
    Could change these to zero to better show they aren’t needed.

    hodlinator commented at 4:52 pm on August 13, 2025:

    The commitment_period needs to be non-zero for functional tests. Example:

     0₿ ./build/test/functional/p2p_headers_sync_with_minchainwork.py
     1[1/13] Generating bitcoin-build-info.h
     22025-08-13T19:03:34.693000Z TestFramework (INFO): PRNG seed is: 3053858877177300552
     32025-08-13T19:03:34.694000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_cm19o0ak
     42025-08-13T19:03:35.502000Z TestFramework (INFO): Generate blocks on the node with no required chainwork, and verify nodes 1 and 2 have no new headers in their headers tree
     52025-08-13T19:03:35.580000Z TestFramework (INFO): Check that node3 will sync headers (due to noban permissions)
     62025-08-13T19:03:35.588000Z TestFramework (INFO): Generate more blocks to satisfy node1's minchainwork requirement, and verify node2 still has no new headers in headers tree
     72025-08-13T19:03:35.647000Z TestFramework (INFO): Check that node3 accepted these headers as well
     82025-08-13T19:03:35.649000Z TestFramework (INFO): Generate long chain for node0/node1/node3
     92025-08-13T19:03:36.337000Z TestFramework (INFO): Verify that node2 and node3 will sync the chain when it gets long enough
    102025-08-13T19:03:36.890000Z TestFramework (ERROR): Unexpected exception caught during testing
    11Traceback (most recent call last):
    12  File "/home/hodlinator/bitcoin/test/functional/test_framework/test_framework.py", line 184, in main
    13    self.run_test()
    14  File "/home/hodlinator/bitcoin/./build/test/functional/p2p_headers_sync_with_minchainwork.py", line 164, in run_test
    15    self.test_chains_sync_when_long_enough()
    16  File "/home/hodlinator/bitcoin/./build/test/functional/p2p_headers_sync_with_minchainwork.py", line 108, in test_chains_sync_when_long_enough
    17    self.sync_blocks()
    18  File "/home/hodlinator/bitcoin/test/functional/test_framework/test_framework.py", line 777, in sync_blocks
    19    best_hash = [x.getbestblockhash() for x in rpc_connections]
    20                 ^^^^^^^^^^^^^^^^^^^^
    21  File "/home/hodlinator/bitcoin/test/functional/test_framework/coverage.py", line 50, in __call__
    22    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
    23                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    24  File "/home/hodlinator/bitcoin/test/functional/test_framework/authproxy.py", line 132, in __call__
    25    response, status = self._request('POST', self.__url.path, postdata.encode('utf-8'))
    26                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    27  File "/home/hodlinator/bitcoin/test/functional/test_framework/authproxy.py", line 106, in _request
    28    return self._get_response()
    29           ^^^^^^^^^^^^^^^^^^^^
    30  File "/home/hodlinator/bitcoin/test/functional/test_framework/authproxy.py", line 169, in _get_response
    31    http_response = self.__conn.getresponse()
    32                    ^^^^^^^^^^^^^^^^^^^^^^^^^
    33  File "/nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/lib/python3.12/http/client.py", line 1430, in getresponse
    34    response.begin()
    35  File "/nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/lib/python3.12/http/client.py", line 331, in begin
    36    version, status, reason = self._read_status()
    37                              ^^^^^^^^^^^^^^^^^^^
    38  File "/nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/lib/python3.12/http/client.py", line 292, in _read_status
    39    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    40               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    41  File "/nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/lib/python3.12/socket.py", line 720, in readinto
    42    return self._sock.recv_into(b)
    43           ^^^^^^^^^^^^^^^^^^^^^^^
    44ConnectionResetError: [Errno 104] Connection reset by peer
    

    …and it’s currently a bit tricky to get functional tests to be able to specify values.

    But this prompted me to make the fields default to zero in the struct definition and add an assert in the HeadersSyncState-ctor.

  117. marcofleon commented at 3:26 pm on August 13, 2025: contributor

    Code review ACK a4191b57b21a0fea871206f368972c7741ad6437

    I made sure the new unit test doesn’t omit any of the current test’s coverage. Breaking it into the three functions (sneaky, happy, and low work) makes this fairly easy to confirm. The HappyPath case adds coverage for when we get close to the redownload buffer, making sure that no headers are added to the block index until we cross that threshold.

    Ran headerssync-params.py to verify that the new params added in chainparams.cpp are correct.

    Just some small nits below. I also was thinking that it could be good to have the headers sync fuzz test choose from a reasonable range of different commitment periods and redownload buffers. The current one always uses the mainnet params, which is fine too.

  118. DrahtBot requested review from TheCharlatan on Aug 13, 2025
  119. DrahtBot requested review from l0rinc on Aug 13, 2025
  120. DrahtBot requested review from jonatack on Aug 13, 2025
  121. DrahtBot requested review from danielabrozzoni on Aug 13, 2025
  122. hodlinator force-pushed on Aug 13, 2025
  123. hodlinator commented at 9:14 pm on August 13, 2025: contributor

    Thanks for joining @marcofleon! :)

    Notable changes in latest push:

  124. in src/test/headers_sync_chainwork_tests.cpp:29 in 8fc783a95e outdated
    37         const auto result{result_expression};                                                            \
    38         BOOST_REQUIRE_EQUAL(hss.GetState(), exp_state);                                                  \
    39         BOOST_CHECK_EQUAL(result.success, exp_success);                                                  \
    40         BOOST_CHECK_EQUAL(result.request_more, exp_request_more);                                        \
    41-        BOOST_CHECK_GE(result.pow_validated_headers.size(), min_headers_size);                           \
    42+        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), exp_headers_size);                        \
    


    l0rinc commented at 10:09 pm on August 13, 2025:
    👍 nice
  125. in src/headerssync.cpp:20 in 7016f8aefc outdated
    29 HeadersSyncState::HeadersSyncState(NodeId id, const Consensus::Params& consensus_params,
    30-        const CBlockIndex* chain_start, const arith_uint256& minimum_required_work) :
    31-    m_commit_offset(FastRandomContext().randrange<unsigned>(HEADER_COMMITMENT_PERIOD)),
    32+        const HeadersSyncParams& params, const CBlockIndex* chain_start,
    33+        const arith_uint256& minimum_required_work) :
    34+    m_commit_offset((assert(params.commitment_period > 0), // HeadersSyncParams field must be initialized to non-zero.
    


    l0rinc commented at 10:15 pm on August 13, 2025:
    we already have an Assume(range > 0) inside, isn’t that enough?

    hodlinator commented at 7:42 am on August 14, 2025:
    True, but Assume()s are not always active, and checking it here means we can describe the constraint with a comment as is done in this project.
  126. in src/test/headers_sync_chainwork_tests.cpp:26 in 8fc783a95e outdated
    36+constexpr size_t TARGET_BLOCKS{15'000};
    37+constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
    38+
    39+// Subtract MAX_HEADERS_RESULTS (2000 headers/message) + an arbitrary smaller
    40+// value (123) so our redownload buffer is well below the number of blocks
    41+// required to reach the CHAIN_WORK threshold, to behave similar to like mainnet.
    


    l0rinc commented at 10:25 pm on August 13, 2025:

    nit: the AI gods think this should be:

    0// required to reach the CHAIN_WORK threshold, to behave like mainnet.
    

    or

    0// required to reach the CHAIN_WORK threshold, to behave similarly to mainnet.
    

    hodlinator commented at 7:42 am on August 14, 2025:
    Cheers, fixed in latest push!
  127. l0rinc approved
  128. l0rinc commented at 10:28 pm on August 13, 2025: contributor

    tested ACK 8fc783a95e568c8a5797da1ac6a9922819863175 (edit: rebased locally to check for conflicts with e.g. #33116, but all tests are passing. I also ran a -reindex until 400k blocks and all seems fine)

    Generalized the header sync params for each chain, split the complicated test into smaller, less stateful chunks, updated testing for current mainchain heights, modernized surrounding code - in small, focused, easy to follow, well commented commits.

  129. DrahtBot requested review from marcofleon on Aug 13, 2025
  130. hodlinator force-pushed on Aug 14, 2025
  131. in src/test/fuzz/headerssync.cpp:75 in a91a5f0070 outdated
    67@@ -67,7 +68,11 @@ FUZZ_TARGET(headers_sync_state, .init = initialize_headers_sync_state_fuzz)
    68 
    69     arith_uint256 min_work{UintToArith256(ConsumeUInt256(fuzzed_data_provider))};
    70     FuzzedHeadersSyncState headers_sync(
    71-        /*commit_offset=*/fuzzed_data_provider.ConsumeIntegralInRange<unsigned>(1, 1024),
    72+        /*sync_params=*/HeadersSyncParams{
    73+            .commitment_period = fuzzed_data_provider.ConsumeIntegralInRange<size_t>(1, Params().HeadersSync().commitment_period * 2),
    74+            .redownload_buffer_size = fuzzed_data_provider.ConsumeIntegralInRange<size_t>(0, Params().HeadersSync().redownload_buffer_size * 2),
    75+        },
    76+        /*commit_offset=*/fuzzed_data_provider.ConsumeIntegralInRange<size_t>(1, 1024),
    


    marcofleon commented at 12:08 pm on August 14, 2025:

    Because commit_offset will never be greater than commitment_period:

    0         const HeadersSyncParams params{
    1        .commitment_period = fuzzed_data_provider.ConsumeIntegralInRange<size_t>(1, Params().HeadersSync().commitment_period * 2),
    2        .redownload_buffer_size = fuzzed_data_provider.ConsumeIntegralInRange<size_t>(0, Params().HeadersSync().redownload_buffer_size * 2),
    3    };
    4    const size_t commit_offset = fuzzed_data_provider.ConsumeIntegralInRange<size_t>(0, params.commitment_period - 1);
    5    FuzzedHeadersSyncState headers_sync(
    6        params,
    7        commit_offset,
    8        /*chain_start=*/&start_index,
    9        /*minimum_required_work=*/min_work);
    

    Although the fuzzer will come up with correct values at some point, so this isn’t too critical.


    hodlinator commented at 3:26 pm on August 14, 2025:
    Thanks! Pushed almost the same.
  132. marcofleon commented at 12:12 pm on August 14, 2025: contributor
    Re ACK a91a5f0070129192dc50ed50452a2788d69f2d97
  133. DrahtBot requested review from l0rinc on Aug 14, 2025
  134. hodlinator force-pushed on Aug 14, 2025
  135. l0rinc commented at 3:30 pm on August 14, 2025: contributor

    reACK 342359f9538874a87b86b67dd98f81cb78237a1e

    typo fix + commit_offset was change since last push to depend on commitment_period

  136. DrahtBot requested review from marcofleon on Aug 14, 2025
  137. hodlinator commented at 3:30 pm on August 14, 2025: contributor

    Sorry for invalidating the ACKs. Implemented suggestion in #32579 (review)

    • Fixes fuzzing of commitment_offset to have min value of 0 instead of 1 as on master, and sets the max depending on the fuzzed commitment_period.
  138. marcofleon commented at 4:01 pm on August 14, 2025: contributor

    Re-ACK 342359f9538874a87b86b67dd98f81cb78237a1e

    Nice, everything lgtm. Just some basic cleanups and changing the headers sync fuzz target to choose the params since my original review.

  139. in src/test/headers_sync_chainwork_tests.cpp:84 in 232485d248 outdated
    78@@ -73,6 +79,17 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    79 
    80     const CBlockIndex* chain_start = WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(genesis.GetHash()));
    81 
    82+    SneakyRedownload(chain_start, first_chain, second_chain);
    83+    HappyPath(chain_start, first_chain);
    84+    TooLittleWork(chain_start, second_chain);
    


    achow101 commented at 8:39 pm on August 14, 2025:

    In 232485d2486f08221203224212d5451a47eb4fc8 “refactor(test): Break up headers_sync_chainwork_tests”

    Our preferred pattern for test cases within unit tests is to define them with BOOST_AUTO_TEST_CASE rather than functions like these.


    hodlinator commented at 9:42 am on August 19, 2025:

    Was concerned about increasing the runtime of the tests due to repeatedly re-generating first_chain/second_chain as documented in the commit message:

    The approach to use one BOOST_AUTO_TEST_CASE for each scenario was attempted but the common initialization is repeated for every test case using the same fixture. This made the local runtime go from 0.39s to 0.65s. Would rather avoid global fixtures, so continued with one boost test case for now.

    However, I figured out an okay way to switch back to using a fixture while avoiding to re-generate the chains:

    0    const std::vector<CBlockHeader>& FirstChain()
    1    {
    2        static const auto first_chain{GenerateHeaders(/*count=*/TARGET_BLOCKS - 1, genesis.GetHash(),
    3                genesis.nVersion, genesis.nTime, /*merkle_root=*/uint256::ZERO, genesis.nBits)};
    4        return first_chain;
    5    }
    

    I’m running some final tests on this but thought I’d give a heads up to other reviewers.


    hodlinator commented at 10:40 am on August 19, 2025:
    Pushed version with multiple test cases.
  140. refactor(headerssync): Extract test constants ahead of breakup into functions
    Made arith_uint256 constexpr-constructible.
    
    Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
    b6ee1de777
  141. refactor(headerssync): Process spans of headers
    More lightweight than vectors which needed to be copied in tests. Also good to get rid of headers_batch-vector before breaking up test.
    4aea5fa369
  142. refactor(test): Break up headers_sync_state
    Helps logically separate the scenarios being tested.
    
    Also adds missing comment for part 4.
    
    (unique_ptrs and ProcessingResults will be cleaned up in next commit).
    75ac11810a
  143. refactor(test): Store HeadersSyncState on the stack 79df0372f2
  144. doc(test): Improve comments
    + new assert helping explain why CHAIN_WORK == TARGET_BLOCKS * 2.
    c7b7017122
  145. test(headerssync): headers_sync_chainwork test improvements
    Introduces CHECK_RESULT for consistently validating ProcessingResult.
    * Verifies HeadersSyncState::State directly after ProcessNextHeaders().
    * Uses BOOST_REQUIRE_EQUAL for HeadersSyncState::State - Nicer failure output and prevents continuing test in nonsensical state.
    * Encourages checking Locator and result.pow_validated_headers.
    f7861fd65b
  146. headerssync: Make HeadersSyncState more flexible and move constants
    Move calculated constants from the top of src/headerssync.cpp into src/kernel/chainparams.cpp.
    
    Instead of being hardcoded to mainnet parameters, HeadersSyncState can now vary depending on chain or test.
    
    Signet and testnets got new HeadersSyncParams constants through temporarily altering headerssync-params.py with corresponding GENESIS_TIME and MINCHAINWORK_HEADERS (based off defaultAssumeValid block height comments, corresponding to nMinimumChainWork). Regtest doesn't have a default assume valid block height, so the values are copied from Testnet 4. Since the constants only affect memory usage, and have very low impact unless dealing with a largely malicious chain, it's not that critical to keep updating them for non-mainnet chains.
    1771e59f1a
  147. test(headerssync): Harden checks
    Since headers_sync_chainwork_tests.cpp now has full control over its own REDOWNLOAD_BUFFER_SIZE, it can now know exactly how many pow_validated_headers to expect, meaning we can test for exp_headers_size (expected) instead of min_headers_size (minimum).
    53341ea10d
  148. hodlinator force-pushed on Aug 19, 2025
  149. hodlinator commented at 10:41 am on August 19, 2025: contributor

    Latest push:

  150. marcofleon commented at 3:24 pm on August 19, 2025: contributor

    Re ACK 53341ea10dc2f7df371b416060863bbc094b8773

    Breaking up the three scenarios into their own test cases makes sense. I checked to make sure the test logic is unchanged.

  151. in src/test/headers_sync_chainwork_tests.cpp:146 in 53341ea10d
    165+BOOST_AUTO_TEST_CASE(sneaky_redownload)
    166 {
    167-    std::vector<CBlockHeader> first_chain;
    168-    std::vector<CBlockHeader> second_chain;
    169+    const auto& first_chain{FirstChain()};
    170+    const auto& second_chain{SecondChain()};
    


    davidgumberg commented at 8:32 pm on August 19, 2025:
    nit: The sneaky redownload and insufficient work cases should be differentiated by making the second chain have enough work, e.g.: https://github.com/davidgumberg/bitcoin/commit/1b1b7f2d06e3b11eac0f844002ff4c1bcf497b0d

    hodlinator commented at 9:31 pm on August 19, 2025:

    That might be a cool higher level change, although hopefully a commitment will mismatch before we reach the final header.

    Hm… thinking through the current code - given 15'000 headers and a period of 600, we have 38 commitments. So there’s a 1 in 2^38 chance that all commitments will match. There’s a 1/600 chance that HeadersSyncState::commit_offset will be 599, in which case second_chain would be too short to check the last commitment, making it a 1 in 2^37 chance of test failure for that offset. (In that case, we would not reach the PoW limit from processing the second chain and the CHECK_RESULT() below will explode).

    https://github.com/bitcoin/bitcoin/blob/53341ea10dc2f7df371b416060863bbc094b8773/src/test/headers_sync_chainwork_tests.cpp#L168-L172

    Although this aspect isn’t materially changed by this PR, I agree it would be good to at least document these probabilities in the test. Will do if I re-push.

    (Also note that the chain generation functions use static internally in order to avoid increasing the runtime of the test).


    davidgumberg commented at 9:28 pm on August 20, 2025:
    I don’t think we should be that worried about a $(\frac{599}{600} * \frac{1}{2^{38}}) + (\frac{1}{600} * \frac{1}{2^{37}})$ probability of spurious test failure, but we could just double the number of headers if that’s a concern.
  152. in src/kernel/chainparams.h:72 in 53341ea10d
    67+    size_t commitment_period{0};
    68+    //! Minimum number of validated headers to accumulate in the redownload
    69+    //! buffer before feeding them into the permanent block index.
    70+    size_t redownload_buffer_size{0};
    71+};
    72+
    


    davidgumberg commented at 9:20 pm on August 19, 2025:
    It really feels like the default constructor for this should be deleted, but that’s not possible because CChainParams() default constructs HeadersSyncParams: https://github.com/bitcoin/bitcoin/blob/53341ea10dc2f7df371b416060863bbc094b8773/src/kernel/chainparams.h#L165

    davidgumberg commented at 9:32 pm on August 19, 2025:

    Ah, I missed that one of the params gets checked for not being default-constructed here:

    https://github.com/bitcoin/bitcoin/blob/53341ea10dc2f7df371b416060863bbc094b8773/src/headerssync.cpp#L20


    hodlinator commented at 9:34 pm on August 19, 2025:
    Agreed, would be nice to have each chain’s params be a constexpr expression tree in an ideal world.
  153. in src/test/fuzz/headerssync.cpp:71 in 53341ea10d
    65@@ -65,9 +66,14 @@ FUZZ_TARGET(headers_sync_state, .init = initialize_headers_sync_state_fuzz)
    66     const uint256 genesis_hash = genesis_header.GetHash();
    67     start_index.phashBlock = &genesis_hash;
    68 
    69+    const HeadersSyncParams params{
    70+        .commitment_period = fuzzed_data_provider.ConsumeIntegralInRange<size_t>(1, Params().HeadersSync().commitment_period * 2),
    71+        .redownload_buffer_size = fuzzed_data_provider.ConsumeIntegralInRange<size_t>(0, Params().HeadersSync().redownload_buffer_size * 2),
    


    davidgumberg commented at 9:21 pm on August 19, 2025:
    why limit these to * 2?

    hodlinator commented at 9:35 pm on August 19, 2025:
    Giving us 2x mainnet should give the fuzzers enough time to break things before we reach that level. See 11a2d3a63e90cdc1920ede3c67d52a9c72860e6b for an example of how gradually these values change for 1 release.
  154. in src/test/headers_sync_chainwork_tests.cpp:162 in 53341ea10d
    189-            Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    190-            ArithToUint256(0), Params().GenesisBlock().nBits);
    191+    // Pretend the message is still "full", so we don't abort.
    192+    // This chain should look valid, and we should have met the proof-of-work
    193+    // requirement during PRESYNC and transitioned to REDOWNLOAD.
    194+    CHECK_RESULT(hss.ProcessNextHeaders(std::span{first_chain}.last(first_chain.size() - 1), true),
    


    davidgumberg commented at 9:45 pm on August 19, 2025:

    nit, I think this was wrong in the original test as well, should be:

    0    CHECK_RESULT(hss.ProcessNextHeaders(std::span{first_chain}.last(first_chain.size() - 1), false),
    

    hodlinator commented at 9:55 pm on August 19, 2025:
    I think the idea is to pretend that our peer has longer chain, meaning they are able to send us more headers, it’s just that HeadersSyncState should be detecting that we’ve met the threshold and switch to REDOWNLOAD even though we tempt it that there are more headers in case it erroneously wants to continue PRESYNC.
  155. in src/test/headers_sync_chainwork_tests.cpp:159 in 53341ea10d
    186-    // Generate headers for two different chains (using differing merkle roots
    187-    // to ensure the headers are different).
    188-    GenerateHeaders(first_chain, target_blocks-1, Params().GenesisBlock().GetHash(),
    189-            Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    190-            ArithToUint256(0), Params().GenesisBlock().nBits);
    191+    // Pretend the message is still "full", so we don't abort.
    


    davidgumberg commented at 9:46 pm on August 19, 2025:
    nit: I think this comment was incorrectly placed in the original, it is the ProcessNextHeaders above that needs /*full_headers_message=*/true

    hodlinator commented at 10:00 pm on August 19, 2025:

    davidgumberg commented at 2:46 am on August 20, 2025:

    My bad, what I mean is that this comment:

    0// Pretend the message is still "full", so we don't abort.
    

    Belongs to the ProcessNextHeaders above.


    hodlinator commented at 7:03 am on August 20, 2025:

    Base version for this PR: https://github.com/bitcoin/bitcoin/blob/fa05a726c225dc65dee79367bb67f099ae4f99e6/src/test/headers_sync_chainwork_tests.cpp#L95-L98

    That comment would indeed be more useful before the first ProcessNextHeaders-call. Will correct if I push again.

  156. in src/test/headers_sync_chainwork_tests.cpp:209 in 53341ea10d
    239+        /*exp_headers_size=*/1, /*exp_pow_validated_prev=*/genesis_hash,
    240+        /*exp_locator_hash=*/first_chain[REDOWNLOAD_BUFFER_SIZE].GetHash());
    241+
    242+    // Feed in remaining headers, meeting the work threshold again and
    243+    // completing the REDOWNLOAD phase:
    244+    CHECK_RESULT(hss.ProcessNextHeaders({first_chain.begin() + REDOWNLOAD_BUFFER_SIZE + 1, first_chain.end()}, true),
    


    davidgumberg commented at 9:50 pm on August 19, 2025:

    nit:

    0    CHECK_RESULT(hss.ProcessNextHeaders({first_chain.begin() + REDOWNLOAD_BUFFER_SIZE + 1, first_chain.end()}, false),
    

    hodlinator commented at 10:05 pm on August 19, 2025:
    Agree here, it seems to be a more taxing test to not pretend like there are any more headers, that HeadersSyncState will have to make do. Will consider changing this if I push, although it is the same on master. Maybe making it random or doing a for x in {true, false} loop over it would be even better.
  157. danielabrozzoni commented at 4:36 pm on August 20, 2025: contributor

    Code Review ACK 53341ea10dc2f7df371b416060863bbc094b8773

    I reviewed the code and checked that tests were passing locally, but didn’t do any extensive testing.

    This PR makes the HEADER_COMMITMENT_PERIOD and REDOWNLOAD_BUFFER_SIZE configurable by the tests, instead of using an hardcoded value, to avoid ending up testing an unrealistic behavior once REDOWNLOAD_BUFFER_SIZE surpasses 15_000 in 0.30.

    (I like how we can test what could happen using the first commit and investigating the weird behavior!)

    Additionally, the PR makes it easier to follow the headers_sync_chainwork_test, by cleaning up code and expanding comments. It also makes the test more extensive, by making sure to check every parameter of ProcessingResult using the CHECK_RESULT macro.

  158. davidgumberg commented at 9:50 pm on August 20, 2025: contributor

    crACK 53341ea10dc2f7df3

    The test changes here look good to me. I left some nits that can be addressed if following-up or rebasing. I sanity checked that a fresh node doing headers syncing for mainnet, testnet, and signet work on my machine with this branch.

    I think it would be good to have input from one of the authors/reviewers of #25717 on the non-test changes to headers sync in this PR (https://github.com/bitcoin/bitcoin/pull/32579/commits/4aea5fa3690310bfc2617e3bf31f555dae7d9337 and https://github.com/bitcoin/bitcoin/pull/32579/commits/1771e59f1a1fcaa6a1d79c93bc0c0c3b1003cf82) before merging.

  159. DrahtBot requested review from davidgumberg on Aug 20, 2025
  160. hodlinator commented at 12:09 pm on August 21, 2025: contributor

    @Sjors or @sdaftuar: would you mind doing some light review of 1771e59f1a1fcaa6a1d79c93bc0c0c3b1003cf82? It represents the meat of this PR, making the headers sync state params that affect memory usage able to be configured for tests/other chains instead of hard-coded to mainnet conditions.

    (davidgumberg also suggested (https://github.com/bitcoin/bitcoin/pull/32579#issuecomment-3208186878) reviewing 4aea5fa3690310bfc2617e3bf31f555dae7d9337 which also touches the HeadersSyncState implementation, replacing vectors with spans).

    (Made a list of author+reviewers from #25717 and used a RNG to select 2 to ping).

  161. achow101 commented at 4:15 pm on August 21, 2025: member
    I generally disagree with the approach in this PR. It’s doing a lot of refactoring that feels too much like refactoring for the sake of refactoring. I think this can be dropped from the milestone.
  162. achow101 removed this from the milestone 30.0 on Aug 21, 2025
  163. hodlinator commented at 7:12 pm on August 21, 2025: contributor

    @achow101:

    I generally disagree with the approach in this PR. It’s doing a lot of refactoring that feels too much like refactoring for the sake of refactoring. I think this can be dropped from the milestone.

    Commit order

    I understand that given the current order of commits, the test refactorings seem less motivated. Order used to be refactorings + main commit + illustrative test + additional log test (https://github.com/bitcoin/bitcoin/compare/37405238a8c4fe4dff82781c91fb668391eb51d4~9...37405238a8c4fe4dff82781c91fb668391eb51d4). Having all the refactoring first got the tests in better shape so that later behavior changes could be expressed clearer. @fanquake wrote on IRC 2 weeks ago (https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2025-08-07#1142169;):

    would be good if it’s clearer why #32579 needs to go into 30, or not

    That made me think it would be better to have an independent commit at the beginning of the PR which breaks once the REDOWNLOAD_BUFFER_SIZE is increased. If you prefer the order: all refactorings + illustrative test + main commit, that also makes sense.

    Context of headers sync changes

    I also must admit that the refactoring began in the context of my other WIP changes for HeadersSyncState - adding a cache (https://github.com/l0rinc/bitcoin/pull/3). Before proposing that one on the main repo, I’m also working on another change to make the number of parallel HeadersSyncState instances in net_processing.cpp more predictable, based off an old comment from 2022 (https://github.com/bitcoin/bitcoin/pull/25720#discussion_r936707629). Maybe I should have a tracking issue to show my larger interest in this area, rather than this PR appearing like a random drive-by.

    Refactoring direction

    If you think the test refactorings go in arbitrary directions, I would be curious to hear which parts you still disagree with now that I’ve used multiple test cases as you suggested (https://github.com/bitcoin/bitcoin/pull/32579#discussion_r2277678459).

    Finishing thoughts

    If you think that the changes in the main commit 1771e59f1a1fcaa6a1d79c93bc0c0c3b1003cf82 are also refactoring for the sake of refactoring, rather than parameters ending up where they should be to enable better testing, that’s a different conversation.

  164. hodlinator commented at 7:17 pm on August 21, 2025: contributor
    (As I said on IRC, I respect the decision to drop it from the milestone. There’s enough moving parts in the release process. It would’ve been neat to fix ahead of bumping REDOWNLOAD_BUFFER_SIZE, the tests on master will not fail when it happens though).
  165. l0rinc commented at 1:13 am on August 22, 2025: contributor

    Given that the current PR already demonstrates the production code is still correct, I tend to agree that we don’t have to rush. Would have been cleaner, but it’s not urgent.

    I do think however that the refactorings are necessary - but I’d keep the commits before the final fix, and if other reviewers think it’s too risky, maybe we can do it in a separate PR - unifying it with @danielabrozzoni’s change and your other related proposal either through a tracking issue or pushing the rest as drafts. I am fine with doing the refactorings here.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-08-24 15:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me