headerssync: Preempt unrealistic unit test behavior #32579

pull hodlinator wants to merge 9 commits into bitcoin:master from hodlinator:2025/05/headerssync_params changing 12 files +310 −128
  1. hodlinator commented at 1:48 pm on May 21, 2025: contributor

    Background

    As part of the release process we often run contrib/devtools/headerssync-params.py and increase the values of the constants HEADER_COMMITMENT_PERIOD and REDOWNLOAD_BUFFER_SIZE in src/headerssync.cpp as per doc/release-process.md. This helps fine tune the memory consumption per HeadersSyncState-instance in the face of malicious peers.

    Problem

    In the next (v30) or following release, it is very likely that REDOWNLOAD_BUFFER_SIZE (14827 as of v29) will exceed the target_blocks value used to test Headers Sync (15000, headers_sync_chainwork_tests.cpp).

    Date at time of calculation: 2025-05-21 Current block height: 897'662 Avg block time: ~9.8min (https://mempool.space). Added 6 months to TIME constant in headerssync-params.py: datetime(2028, 4, 6) Increased block height in headerssync-params.py (MINCHAINWORK_HEADERS) to: 920'000 Block height difference: 22'338 Days: 22'338 * 9.8min / (60min * 24h) = ~152 Estimated date for block height 920'000: 2025-10-20 REDOWNLOAD_BUFFER_SIZE value calculated by headerssync-params.py: 15002

    This would result in the HeadersSyncState::m_redownloaded_headers-buffer not reaching the REDOWNLOAD_BUFFER_SIZE-threshold during those unit tests. While not the end of the world, the test would go from testing behavior that resembles mainnet, to behavior that is not possible/expected there.

    Solution

    Preempt the tests from only testing this unrealistic condition of succeeding before reaching REDOWNLOAD_BUFFER_SIZE by making tests able to define their own value for this parameter of HeadersSyncState. Then also add tests that verify the behavior around that boundary threshold.

    Commits

    • Initial commits refactor and improve the unit tests.
    • We then extract the section from headerssync.cpp containing the constants to kernel/chainparams.cpp while making HeadersSyncState less hard-coded.
    • Finally we add tests for the behavior around the threshold as we can now specify it in the tests.
  2. DrahtBot commented at 1:48 pm on May 21, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32579.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK TheCharlatan, jonatack, l0rinc, danielabrozzoni

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #32740 (refactor: Header sync optimisations & simplifications by danielabrozzoni)
    • #31974 (Drop testnet3 by Sjors)
    • #28122 (Silent Payments: Implement BIP352 by josibake)
    • #26812 (test: add end-to-end tests for CConnman and PeerManager by vasild)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. in src/headerssync-params.h:16 in 0c1978b70d outdated
    11+//! Store one header commitment per HEADER_COMMITMENT_PERIOD blocks.
    12+constexpr size_t HEADER_COMMITMENT_PERIOD{624};
    13+
    14+//! Only feed headers to validation once this many headers on top have been
    15+//! received and validated against commitments.
    16+constexpr size_t REDOWNLOAD_BUFFER_SIZE{14827}; // 14827/624 = ~23.8 commitments
    


    l0rinc commented at 2:13 pm on May 21, 2025:
    I know this is just a move, but instead of just bare magic number + magic comment (which also has to be manually updated every time), could we have the something like : Generated by headerssync-params.py (on 2025-03-04)

    hodlinator commented at 12:35 pm on May 30, 2025:

    I like the idea of including the generation date, and was including changes to the Python generator file in an unpublished version of this PR, but decided to keep that separate. Added to that separate WIP branch.

    The magic numbers are not updated manually, they are included in the calculated output: https://github.com/bitcoin/bitcoin/blob/11a2d3a63e90cdc1920ede3c67d52a9c72860e6b/contrib/devtools/headerssync-params.py#L347-L348


    hodlinator commented at 1:43 pm on June 4, 2025:
    Added generation date in latest push as it was making modifications to the Python script anyway.
  4. hodlinator force-pushed on May 21, 2025
  5. DrahtBot added the label CI failed on May 21, 2025
  6. DrahtBot commented at 2:15 pm on May 21, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/runs/42636718296 LLM reason (✨ experimental): The CI failure is due to a missing include guard in the src/headerssync-params.h file, as reported by the lint checks.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  7. DrahtBot removed the label CI failed on May 21, 2025
  8. in src/test/headers_sync_chainwork_tests.cpp:66 in 34e1513efe outdated
    62@@ -63,15 +63,19 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
    63 //    updates to the REDOWNLOAD phase successfully.
    64 // 2. Then we deliver the second set of headers and verify that they fail
    65 //    processing (presumably due to commitments not matching).
    66-// 3. Finally, we verify that repeating with the first set of headers in both
    67-//    phases is successful.
    68+static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
    


    l0rinc commented at 10:02 am on May 27, 2025:

    34e1513efeb193a46d37f184b9ab41bcda974afa: I have inlined the methods and verified that the remaining changes are only comments, reused-and-cleared pointers and vectors 👍

      0diff --git a/src/test/headers_sync_chainwork_tests.cpp b/src/test/headers_sync_chainwork_tests.cpp
      1index 054853bbbf..89bb6682d1 100644
      2--- a/src/test/headers_sync_chainwork_tests.cpp
      3+++ b/src/test/headers_sync_chainwork_tests.cpp
      4@@ -13,29 +13,32 @@
      5 
      6 #include <boost/test/unit_test.hpp>
      7 
      8-struct HeadersGeneratorSetup : public RegTestingSetup {
      9+struct HeadersGeneratorSetup : RegTestingSetup
     10+{
     11     /** Search for a nonce to meet (regtest) proof of work */
     12-    void FindProofOfWork(CBlockHeader& starting_header);
     13+    static void FindProofOfWork(CBlockHeader& starting_header);
     14     /**
     15      * Generate headers in a chain that build off a given starting hash, using
     16      * the given nVersion, advancing time by 1 second from the starting
     17      * prev_time, and with a fixed merkle root hash.
     18      */
     19-    void GenerateHeaders(std::vector<CBlockHeader>& headers, size_t count,
     20-            const uint256& starting_hash, const int nVersion, int prev_time,
     21-            const uint256& merkle_root, const uint32_t nBits);
     22+    static void GenerateHeaders(
     23+        std::vector<CBlockHeader>& headers, size_t count,
     24+        const uint256& starting_hash, int nVersion, int prev_time,
     25+        const uint256& merkle_root, uint32_t nBits);
     26 };
     27 
     28 void HeadersGeneratorSetup::FindProofOfWork(CBlockHeader& starting_header)
     29 {
     30     while (!CheckProofOfWork(starting_header.GetHash(), starting_header.nBits, Params().GetConsensus())) {
     31-        ++(starting_header.nNonce);
     32+        ++starting_header.nNonce;
     33     }
     34 }
     35 
     36-void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     37-        size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     38-        const uint256& merkle_root, const uint32_t nBits)
     39+void HeadersGeneratorSetup::GenerateHeaders(
     40+    std::vector<CBlockHeader>& headers,
     41+    const size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     42+    const uint256& merkle_root, const uint32_t nBits)
     43 {
     44     uint256 prev_hash = starting_hash;
     45 
     46@@ -45,14 +48,13 @@ void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     47         next_header.nVersion = nVersion;
     48         next_header.hashPrevBlock = prev_hash;
     49         next_header.hashMerkleRoot = merkle_root;
     50-        next_header.nTime = prev_time+1;
     51+        next_header.nTime = prev_time + 1;
     52         next_header.nBits = nBits;
     53 
     54         FindProofOfWork(next_header);
     55         prev_hash = next_header.GetHash();
     56         prev_time = next_header.nTime;
     57     }
     58-    return;
     59 }
     60 
     61 BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
     62@@ -63,31 +65,41 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
     63 //    updates to the REDOWNLOAD phase successfully.
     64 // 2. Then we deliver the second set of headers and verify that they fail
     65 //    processing (presumably due to commitments not matching).
     66-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
     67+static void SneakyRedownload(
     68+    const std::vector<CBlockHeader>& first_chain,
     69+    const std::vector<CBlockHeader>& second_chain,
     70+    const CBlockIndex* chain_start,
     71+    const arith_uint256& chain_work);
     72 // 3. Verify that repeating with the first set of headers in both phases is
     73 //    successful.
     74-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
     75+static void HappyPath(
     76+    const std::vector<CBlockHeader>& first_chain,
     77+    const CBlockIndex* chain_start,
     78+    const arith_uint256& chain_work);
     79 // 4. Finally, repeat the second set of headers in both phases to demonstrate
     80 //    behavior when the chain a peer provides has too little work.
     81-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
     82+static void TooLittleWork(
     83+    const std::vector<CBlockHeader>& second_chain,
     84+    const CBlockIndex* chain_start,
     85+    const arith_uint256& chain_work);
     86 
     87 BOOST_AUTO_TEST_CASE(headers_sync_state)
     88 {
     89     std::vector<CBlockHeader> first_chain;
     90     std::vector<CBlockHeader> second_chain;
     91 
     92-    const int target_blocks = 15000;
     93-    arith_uint256 chain_work = target_blocks*2;
     94+    constexpr int target_blocks = 15000;
     95+    const arith_uint256 chain_work = target_blocks * 2;
     96 
     97     // Generate headers for two different chains (using differing merkle roots
     98     // to ensure the headers are different).
     99-    GenerateHeaders(first_chain, target_blocks-1, Params().GenesisBlock().GetHash(),
    100-            Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    101-            ArithToUint256(0), Params().GenesisBlock().nBits);
    102+    GenerateHeaders(first_chain, target_blocks - 1, Params().GenesisBlock().GetHash(),
    103+                    Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    104+                    ArithToUint256(0), Params().GenesisBlock().nBits);
    105 
    106-    GenerateHeaders(second_chain, target_blocks-2, Params().GenesisBlock().GetHash(),
    107-            Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    108-            ArithToUint256(1), Params().GenesisBlock().nBits);
    109+    GenerateHeaders(second_chain, target_blocks - 2, Params().GenesisBlock().GetHash(),
    110+                    Params().GenesisBlock().nVersion, Params().GenesisBlock().nTime,
    111+                    ArithToUint256(1), Params().GenesisBlock().nBits);
    112 
    113     const CBlockIndex* chain_start = WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(Params().GenesisBlock().GetHash()));
    114 
    115@@ -96,7 +108,10 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    116     TooLittleWork(second_chain, chain_start, chain_work);
    117 }
    118 
    119-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work)
    120+static void SneakyRedownload(
    121+    const std::vector<CBlockHeader>& first_chain,
    122+    const std::vector<CBlockHeader>& second_chain,
    123+    const CBlockIndex* chain_start, const arith_uint256& chain_work)
    124 {
    125     std::vector<CBlockHeader> headers_batch;
    126 
    127@@ -109,56 +124,59 @@ static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const
    128     // Pretend the first header is still "full", so we don't abort.
    129     auto result = hss.ProcessNextHeaders(headers_batch, true);
    130 
    131-    // This chain should look valid, and we should have met the proof-of-work
    132-    // requirement.
    133+    // This chain should look valid, and we should have met the proof-of-work requirement.
    134     BOOST_CHECK(result.success);
    135     BOOST_CHECK(result.request_more);
    136-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    137+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    138 
    139     // Try to sneakily feed back the second chain.
    140     result = hss.ProcessNextHeaders(second_chain, true);
    141     BOOST_CHECK(!result.success); // foiled!
    142-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    143+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    144 }
    145 
    146-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work)
    147+static void HappyPath(
    148+    const std::vector<CBlockHeader>& first_chain,
    149+    const CBlockIndex* chain_start,
    150+    const arith_uint256& chain_work)
    151 {
    152     // This time we feed the first chain twice.
    153     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, chain_work};
    154     (void)hss.ProcessNextHeaders(first_chain, true);
    155-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    156+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    157 
    158-    auto result = hss.ProcessNextHeaders(first_chain, true);
    159-    BOOST_CHECK(result.success);
    160-    BOOST_CHECK(!result.request_more);
    161+    const auto [pow_validated_headers, success, request_more] = hss.ProcessNextHeaders(first_chain, true);
    162+    BOOST_CHECK(success);
    163+    BOOST_CHECK(!request_more);
    164     // All headers should be ready for acceptance:
    165-    BOOST_CHECK(result.pow_validated_headers.size() == first_chain.size());
    166+    BOOST_CHECK_EQUAL(pow_validated_headers.size(), first_chain.size());
    167     // Nothing left for the sync logic to do:
    168-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    169+    BOOST_CHECK_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    170 }
    171 
    172-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work)
    173+static void TooLittleWork(
    174+    const std::vector<CBlockHeader>& second_chain,
    175+    const CBlockIndex* chain_start,
    176+    const arith_uint256& chain_work)
    177 {
    178-    // Verify that just trying to process the second chain would not succeed
    179-    // (too little work).
    180+    // Verify that just trying to process the second chain would not succeed (too little work).
    181     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, chain_work};
    182     BOOST_CHECK(hss.GetState() == HeadersSyncState::State::PRESYNC);
    183-     // Pretend just the first message is "full", so we don't abort.
    184+    // Pretend just the first message is "full", so we don't abort.
    185     (void)hss.ProcessNextHeaders({second_chain.front()}, true);
    186     BOOST_CHECK(hss.GetState() == HeadersSyncState::State::PRESYNC);
    187 
    188     std::vector<CBlockHeader> headers_batch;
    189     headers_batch.insert(headers_batch.end(), std::next(second_chain.begin(), 1), second_chain.end());
    190     // Tell the sync logic that the headers message was not full, implying no
    191-    // more headers can be requested. For a low-work-chain, this should causes
    192+    // more headers can be requested. For a low-work-chain, this should cause
    193     // the sync to end with no headers for acceptance.
    194-    auto result = hss.ProcessNextHeaders(headers_batch, false);
    195+    const auto [pow_validated_headers, success, request_more] = hss.ProcessNextHeaders(headers_batch, false);
    196     BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    197-    BOOST_CHECK(result.pow_validated_headers.empty());
    198-    BOOST_CHECK(!result.request_more);
    199-    // Nevertheless, no validation errors should have been detected with the
    200-    // chain:
    201-    BOOST_CHECK(result.success);
    202+    BOOST_CHECK(pow_validated_headers.empty());
    203+    BOOST_CHECK(!request_more);
    204+    // Nevertheless, no validation errors should have been detected with the chain:
    205+    BOOST_CHECK(success);
    206 }
    207 
    208 BOOST_AUTO_TEST_SUITE_END()
    

    I see that some of these were done in next commits, but I’d prefer doing either moves or refactors or adding new checks in the commits to simplify review


    hodlinator commented at 11:13 am on June 2, 2025:
    (Incorporated most of this, breaking out commits, expanding function prototypes into multiple lines, even removing the HeadersGeneratorSetup type in favor of free functions).
  9. in src/test/headers_sync_chainwork_tests.cpp:124 in 13aa0b860e outdated
    147+    // Try to sneakily feed back the second chain during REDOWNLOAD.
    148     result = hss.ProcessNextHeaders(second_chain, true);
    149+    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    150     BOOST_CHECK(!result.success); // foiled!
    151-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    152+    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    


    l0rinc commented at 10:20 am on May 27, 2025:

    we’re reusing the previous result (which was brace initialized), what if we used braces to delimit the two independent code parts (+ a few other nits you might want to consider):

     0// Verify that just trying to process the second chain would not succeed (too little work).
     1HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
     2BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
     3// Pretend just the first message is "full", so we don't abort.
     4{
     5    const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin(), 1}, true)};
     6    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
     7    BOOST_CHECK(success);
     8    BOOST_CHECK(request_more);
     9    BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    10}
    11
    12// Tell the sync logic that the headers message was not full, implying no
    13// more headers can be requested. For a low-work-chain, this should cause
    14// the sync to end with no headers for acceptance.
    15{
    16    const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin() + 1, second_chain.end()}, false)};
    17    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    18    BOOST_CHECK(!request_more);
    19    BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    20    // Nevertheless, no validation errors should have been detected with the chain:
    21    BOOST_CHECK(success);
    22}
    

    hodlinator commented at 8:22 pm on May 30, 2025:
    Took the scoping idea, still not sure about the structured bindings.

    l0rinc commented at 8:39 am on June 2, 2025:

    not sure about the structured bindings

    That’s what helped me in noticing that we weren’t checking request_more


    hodlinator commented at 11:17 am on June 2, 2025:

    Is there anything to catch wrong order?

    0const auto [pow_validated_headers, success, request_more] = ...
    

    vs

    0const auto [pow_validated_headers, request_more, success] = ...
    

    (There’s clang-tidy logic to catch /*paramname=*/true mismatches for function calls).


    l0rinc commented at 2:13 pm on June 2, 2025:
    I don’t think there could/should be, you could name them as const auto [x, y, z]{hss.ProcessNextHeaders(std::span{second_chain.begin() + 1, second_chain.end()}, false)}; as well

    hodlinator commented at 7:27 pm on June 2, 2025:
    Yes, but when breaking apart a struct and two of the new variable names match struct field names with identical types, it would be nice to have some kind of guard rail.
  10. in src/test/headers_sync_chainwork_tests.cpp:123 in 13aa0b860e outdated
    145 
    146-    // Try to sneakily feed back the second chain.
    147+    // Try to sneakily feed back the second chain during REDOWNLOAD.
    148     result = hss.ProcessNextHeaders(second_chain, true);
    149+    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    150     BOOST_CHECK(!result.success); // foiled!
    


    l0rinc commented at 10:25 am on May 27, 2025:

    This seems to be the only place where request_more isn’t validated.

    nit: what’s the meaning of “foiled” here?


    hodlinator commented at 12:40 pm on May 30, 2025:

    If we didn’t succeed, then request_more shouldn’t matter, but agree it’s best to have it be false (just like pow_validated_headers should be empty). It would be better if ProcessNextHeaders returned something like std::optional<ProcessingResult> instead of having a bool field to indicate success, but this PR already feels large enough.

    “foiled” means the attack attempt was detected and therefore failed. Elaborated the comment a bit in next push.


    hodlinator commented at 8:45 pm on May 30, 2025:

    Hm.. my std::optional<ProcessingResult> suggestion would not work not clarify things here:

    https://github.com/bitcoin/bitcoin/blob/a7b9cc4a0af0dbe4031ec5848d403f61dc9c3356/src/test/headers_sync_chainwork_tests.cpp#L127-L136

    Success seems to mean that PRESYNC- and REDOWNLOAD-headers matched, even if the chain is too short.

  11. in src/arith_uint256.h:34 in 13aa0b860e outdated
    32     uint32_t pn[WIDTH];
    33-public:
    34 
    35-    base_uint()
    36+public:
    37+    constexpr base_uint()
    


    l0rinc commented at 10:32 am on May 27, 2025:
    should we make any of these explicit?

    hodlinator commented at 12:23 pm on May 30, 2025:
    One could argue at least the uint64_t-taking ctor here and in arith_uint256 should be explicit, but I’d rather not have to change src/pow.cpp in this PR.
  12. in src/test/headers_sync_chainwork_tests.cpp:16 in 13aa0b860e outdated
    12@@ -13,6 +13,9 @@
    13 
    14 #include <boost/test/unit_test.hpp>
    15 
    16+constexpr int TARGET_BLOCKS{15000};
    


    l0rinc commented at 10:32 am on May 27, 2025:

    size_t might me more fitting here since GenerateHeaders accepts that instead of an int:

    0constexpr size_t TARGET_BLOCKS{15'000};
    
  13. in src/test/headers_sync_chainwork_tests.cpp:68 in 13aa0b860e outdated
    64@@ -63,99 +65,107 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
    65 //    updates to the REDOWNLOAD phase successfully.
    66 // 2. Then we deliver the second set of headers and verify that they fail
    67 //    processing (presumably due to commitments not matching).
    68-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start, const arith_uint256& chain_work);
    69+static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start);
    


    l0rinc commented at 10:39 am on May 27, 2025:
    Could we use const std::span<const CBlockHeader> in these helpers as well? And maybe split these long lines.

    hodlinator commented at 12:42 pm on May 30, 2025:
    Agree on the split but what would be the point of changing the type?

    l0rinc commented at 8:39 am on June 2, 2025:
    • they’re simpler/less specific
    • they’re shorter (matters for these long lines)
    • you’re already changing them in other places

    i.e. what’s the point of keeping them as vectors when we don’t actually need so much power?


    hodlinator commented at 11:21 am on June 2, 2025:
    Using const vector& communicates that we are sending in the entire chain, not some subset. I like the specificity. const vectors feel more neutered than spans IMO.

    l0rinc commented at 2:17 pm on June 2, 2025:
    When we’re only iterating inside, it makes sense to have a fine-grained, segregated interface and only define the narrowest functionality which supports the operations we actually need inside. It announces to the reader that we shouldn’t expect any extra complexity, only the ones that the interface implements. But if you think vector does it better, I’m also fine with that. For me span is not about a subset of teh data but a subset of the functionality - just a narrowed view to rule out certain operations that we definitely won’t need.
  14. in src/test/headers_sync_chainwork_tests.cpp:136 in 13aa0b860e outdated
    163+    auto result{hss.ProcessNextHeaders(first_chain, true)};
    164+    // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    165+    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    166+    BOOST_CHECK(result.success);
    167+    BOOST_CHECK(result.request_more);
    168+    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    


    l0rinc commented at 10:40 am on May 27, 2025:
    👍 for comparing size instead of empty - the failure message would likely be more meaningful
  15. in src/test/headers_sync_chainwork_tests.cpp:72 in 13aa0b860e outdated
    43@@ -41,7 +44,7 @@ void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
    44 
    45     while (headers.size() < count) {
    46         headers.emplace_back();
    47-        CBlockHeader& next_header = headers.back();;
    48+        CBlockHeader& next_header = headers.back();
    


    l0rinc commented at 10:43 am on May 27, 2025:
    I think we could do these in the previous refactor commit instead - to separate low risk changes from ones where we have to may more attention, while keeping the first one about move-only changes + minor ones. Or do the first commit as strictly move-only and add a second one with these tiny refactors (I’d like the latter more).
  16. in src/arith_uint256.h:1 in 13aa0b860e outdated


    l0rinc commented at 10:44 am on May 27, 2025:

    full patch for 13aa0b860e9692715d96b13acacb3f83a537b905

      0diff --git a/src/arith_uint256.h b/src/arith_uint256.h
      1index e021947381..b07e8e9f41 100644
      2--- a/src/arith_uint256.h
      3+++ b/src/arith_uint256.h
      4@@ -31,7 +31,7 @@ protected:
      5     uint32_t pn[WIDTH];
      6 
      7 public:
      8-    constexpr base_uint()
      9+    explicit constexpr base_uint()
     10     {
     11         for (int i = 0; i < WIDTH; i++)
     12             pn[i] = 0;
     13@@ -243,8 +243,8 @@ class arith_uint256 : public base_uint<256>
     14 {
     15 public:
     16     constexpr arith_uint256() = default;
     17-    constexpr arith_uint256(const base_uint<256>& b) : base_uint<256>(b) {}
     18-    constexpr arith_uint256(uint64_t b) : base_uint<256>(b) {}
     19+    constexpr arith_uint256(const base_uint& b) : base_uint(b) {}
     20+    constexpr arith_uint256(const uint64_t b) : base_uint(b) {}
     21 
     22     /**
     23      * The "compact" format is a representation of a whole
     24diff --git a/src/test/headers_sync_chainwork_tests.cpp b/src/test/headers_sync_chainwork_tests.cpp
     25index b696285d0e..264959f8c3 100644
     26--- a/src/test/headers_sync_chainwork_tests.cpp
     27+++ b/src/test/headers_sync_chainwork_tests.cpp
     28@@ -13,32 +13,35 @@
     29 
     30 #include <boost/test/unit_test.hpp>
     31 
     32-constexpr int TARGET_BLOCKS{15000};
     33+constexpr size_t TARGET_BLOCKS{15'000};
     34 constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
     35 
     36-struct HeadersGeneratorSetup : public RegTestingSetup {
     37+struct HeadersGeneratorSetup : RegTestingSetup
     38+{
     39     /** Search for a nonce to meet (regtest) proof of work */
     40-    void FindProofOfWork(CBlockHeader& starting_header);
     41+    static void FindProofOfWork(CBlockHeader& starting_header);
     42     /**
     43      * Generate headers in a chain that build off a given starting hash, using
     44      * the given nVersion, advancing time by 1 second from the starting
     45      * prev_time, and with a fixed merkle root hash.
     46      */
     47-    void GenerateHeaders(std::vector<CBlockHeader>& headers, size_t count,
     48-            const uint256& starting_hash, const int nVersion, int prev_time,
     49-            const uint256& merkle_root, const uint32_t nBits);
     50+    static void GenerateHeaders(
     51+        std::vector<CBlockHeader>& headers, size_t count,
     52+        const uint256& starting_hash, int nVersion, int prev_time,
     53+        const uint256& merkle_root, uint32_t nBits);
     54 };
     55 
     56 void HeadersGeneratorSetup::FindProofOfWork(CBlockHeader& starting_header)
     57 {
     58     while (!CheckProofOfWork(starting_header.GetHash(), starting_header.nBits, Params().GetConsensus())) {
     59-        ++(starting_header.nNonce);
     60+        ++starting_header.nNonce;
     61     }
     62 }
     63 
     64-void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     65-        size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     66-        const uint256& merkle_root, const uint32_t nBits)
     67+void HeadersGeneratorSetup::GenerateHeaders(
     68+    std::vector<CBlockHeader>& headers,
     69+    const size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     70+    const uint256& merkle_root, const uint32_t nBits)
     71 {
     72     uint256 prev_hash = starting_hash;
     73 
     74@@ -48,7 +51,7 @@ void HeadersGeneratorSetup::GenerateHeaders(std::vector<CBlockHeader>& headers,
     75         next_header.nVersion = nVersion;
     76         next_header.hashPrevBlock = prev_hash;
     77         next_header.hashMerkleRoot = merkle_root;
     78-        next_header.nTime = prev_time+1;
     79+        next_header.nTime = prev_time + 1;
     80         next_header.nBits = nBits;
     81 
     82         FindProofOfWork(next_header);
     83@@ -65,13 +68,20 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, HeadersGeneratorSetup)
     84 //    updates to the REDOWNLOAD phase successfully.
     85 // 2. Then we deliver the second set of headers and verify that they fail
     86 //    processing (presumably due to commitments not matching).
     87-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start);
     88+static void SneakyRedownload(
     89+    const std::span<const CBlockHeader> first_chain,
     90+    const std::span<const CBlockHeader> second_chain,
     91+    const CBlockIndex* chain_start);
     92 // 3. Verify that repeating with the first set of headers in both phases is
     93 //    successful.
     94-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start);
     95+static void HappyPath(
     96+    const std::span<const CBlockHeader> first_chain,
     97+    const CBlockIndex* chain_start);
     98 // 4. Finally, repeat the second set of headers in both phases to demonstrate
     99 //    behavior when the chain a peer provides has too little work.
    100-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start);
    101+static void TooLittleWork(
    102+    const std::span<const CBlockHeader> second_chain,
    103+    const CBlockIndex* chain_start);
    104 
    105 BOOST_AUTO_TEST_CASE(headers_sync_state)
    106 {
    107@@ -94,81 +104,101 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    108     TooLittleWork(second_chain, chain_start);
    109 }
    110 
    111-static void SneakyRedownload(const std::vector<CBlockHeader>& first_chain, const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start)
    112+static void SneakyRedownload(
    113+    const std::span<const CBlockHeader> first_chain,
    114+    const std::span<const CBlockHeader> second_chain,
    115+    const CBlockIndex* chain_start)
    116 {
    117     // Feed the first chain to HeadersSyncState, by delivering 1 header
    118     // initially and then the rest.
    119     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
    120-    auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    121-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    122-    BOOST_CHECK(result.success);
    123-    BOOST_CHECK(result.request_more);
    124-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    125-    BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain.front().GetHash());
    126+    {
    127+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({first_chain.begin(), 1}, true)};
    128+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    129+        BOOST_CHECK(success);
    130+        BOOST_CHECK(request_more);
    131+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    132+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain.front().GetHash());
    133+    }
    134 
    135     // Pretend the first header is still "full", so we don't abort.
    136-    result = hss.ProcessNextHeaders(std::span{first_chain.begin() + 1, first_chain.end()}, true);
    137-    // This chain should look valid, and we should have met the proof-of-work
    138-    // requirement during PRESYNC and transitioned to REDOWNLOAD.
    139-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    140-    BOOST_CHECK(result.success);
    141-    BOOST_CHECK(result.request_more);
    142-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    143-    // The locator should reset to genesis.
    144-    BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    145+    {
    146+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({first_chain.begin() + 1, first_chain.end()}, true)};
    147+        // This chain should look valid, and we should have met the proof-of-work
    148+        // requirement during PRESYNC and transitioned to REDOWNLOAD.
    149+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    150+        BOOST_CHECK(success);
    151+        BOOST_CHECK(request_more);
    152+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    153+        // The locator should reset to genesis.
    154+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    155+    }
    156 
    157     // Try to sneakily feed back the second chain during REDOWNLOAD.
    158-    result = hss.ProcessNextHeaders(second_chain, true);
    159-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    160-    BOOST_CHECK(!result.success); // foiled!
    161-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    162+    {
    163+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders(second_chain, true)};
    164+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    165+        BOOST_CHECK(!success); // foiled!
    166+        BOOST_CHECK(!request_more);
    167+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    168+    }
    169 }
    170 
    171-static void HappyPath(const std::vector<CBlockHeader>& first_chain, const CBlockIndex* chain_start)
    172+static void HappyPath(
    173+    const std::span<const CBlockHeader> first_chain,
    174+    const CBlockIndex* chain_start)
    175 {
    176     // This time we feed the first chain twice.
    177     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
    178-    auto result{hss.ProcessNextHeaders(first_chain, true)};
    179-    // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    180-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    181-    BOOST_CHECK(result.success);
    182-    BOOST_CHECK(result.request_more);
    183-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    184-    // The locator should reset to genesis.
    185-    BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    186-
    187-    result = hss.ProcessNextHeaders(first_chain, true);
    188-    // Nothing left for the sync logic to do:
    189-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    190-    BOOST_CHECK(result.success);
    191-    BOOST_CHECK(!result.request_more);
    192-    // All headers should be ready for acceptance:
    193-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), first_chain.size());
    194+    {
    195+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders(first_chain, true)};
    196+        // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    197+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    198+        BOOST_CHECK(success);
    199+        BOOST_CHECK(request_more);
    200+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    201+        // The locator should reset to genesis.
    202+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    203+    }
    204+
    205+    {
    206+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders(first_chain, true)};
    207+        // Nothing left for the sync logic to do:
    208+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    209+        BOOST_CHECK(success);
    210+        BOOST_CHECK(!request_more);
    211+        // All headers should be ready for acceptance:
    212+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), first_chain.size());
    213+    }
    214 }
    215 
    216-static void TooLittleWork(const std::vector<CBlockHeader>& second_chain, const CBlockIndex* chain_start)
    217+static void TooLittleWork(
    218+    const std::span<const CBlockHeader> second_chain,
    219+    const CBlockIndex* chain_start)
    220 {
    221-    // Verify that just trying to process the second chain would not succeed
    222-    // (too little work).
    223+    // Verify that just trying to process the second chain would not succeed (too little work).
    224     HeadersSyncState hss{0, Params().GetConsensus(), chain_start, CHAIN_WORK};
    225     BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    226     // Pretend just the first message is "full", so we don't abort.
    227-    auto result{hss.ProcessNextHeaders(std::span{second_chain.begin(), 1}, true)};
    228-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    229-    BOOST_CHECK(result.success);
    230-    BOOST_CHECK(result.request_more);
    231-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    232+    {
    233+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin(), 1}, true)};
    234+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    235+        BOOST_CHECK(success);
    236+        BOOST_CHECK(request_more);
    237+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    238+    }
    239 
    240     // Tell the sync logic that the headers message was not full, implying no
    241     // more headers can be requested. For a low-work-chain, this should cause
    242     // the sync to end with no headers for acceptance.
    243-    result = hss.ProcessNextHeaders(std::span{second_chain.begin() + 1, second_chain.end()}, false);
    244-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    245-    BOOST_CHECK(!result.request_more);
    246-    BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    247-    // Nevertheless, no validation errors should have been detected with the
    248-    // chain:
    249-    BOOST_CHECK(result.success);
    250+    {
    251+        const auto [pow_validated_headers, success, request_more]{hss.ProcessNextHeaders({second_chain.begin() + 1, second_chain.end()}, false)};
    252+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    253+        BOOST_CHECK(!request_more);
    254+        BOOST_CHECK_EQUAL(pow_validated_headers.size(), 0);
    255+        // Nevertheless, no validation errors should have been detected with the chain:
    256+        BOOST_CHECK(success);
    257+    }
    258 }
    259 
    260 BOOST_AUTO_TEST_SUITE_END()
    

    l0rinc commented at 10:49 am on May 27, 2025:
    For other headers we’ve used an underscore instead of a hyphen in the file name: src/headerssync_params.h

    hodlinator commented at 12:06 pm on May 30, 2025:
    What’s the point of making a constructor that takes no arguments explicit?

    hodlinator commented at 8:18 pm on May 30, 2025:
    Started renaming but then realized that we already have headerssync-params.py. It’s not obvious to me that underscores would be more correct for C++ files. We do have other C++ files with hyphens.

    l0rinc commented at 8:40 am on June 2, 2025:
    Based on the Sonarcloud recommendations I eagerly checked which other constructors can be explicit, but exploring your question in more detail it seems there’s barely any advantage in explicit default constructors.

    l0rinc commented at 8:40 am on June 2, 2025:

    It seems to be a mix indeed:

    0% find . -name "*.h" | grep '_' | wc -l
    1185
    

    vs

    0% find . -name "*.h" | grep '-' | wc -l
    139
    

    and most of those are from the recent ipc changes:

    0% find . -name "*.h" | grep '-' | grep -v ipc | wc -l            
    111
    

    it’s just a nit from my part, but standard lib also uses underscores mostly (e.g. string_view) and in many regexes the \w, i.e. a word character includes _ but usually not - (if I remember it correctly some bash implementations can also trip on -, but that’s likely not a problem here).


    hodlinator commented at 11:08 am on June 2, 2025:

    Ah, the Sonarcloud recommendation was actually for the uint64_t-ctor, similar to my response here: #32579 (review).

    Tested out making it explicit, modifying 13 additional files. It did uncover some implicit casts that went from int64_t Params::nPowTargetTimespan to arith_uint256… But the rest felt a bit forced. I think it is intentional and makes sense for u64s to implicitly cast to u256.


    hodlinator commented at 11:29 am on June 2, 2025:
    Yeah, seems src/ipc/ really went all in on -. Holding off on changing for now to see what others think.

    l0rinc commented at 2:18 pm on June 2, 2025:
    Thanks for checking!

    l0rinc commented at 9:23 am on June 20, 2025:
    typo in commit message of 42e746096b93b8026569817d8925750089d6daf4

    hodlinator commented at 8:22 am on June 23, 2025:
    Fixed.
  17. in doc/release-process.md:56 in e9f4f6a5d9 outdated
    52@@ -53,7 +53,7 @@ Release Process
    53     - Set `MINCHAINWORK_HEADERS` to the height used for the `nMinimumChainWork` calculation above.
    54     - Check that the other variables still look reasonable.
    55   - Run the script. It works fine in CPython, but PyPy is much faster (seconds instead of minutes): `pypy3 contrib/devtools/headerssync-params.py`.
    56-  - Paste the output defining `HEADER_COMMITMENT_PERIOD` and `REDOWNLOAD_BUFFER_SIZE` into the top of [`src/headerssync.cpp`](/src/headerssync.cpp).
    57+  - Paste the output defining `HEADER_COMMITMENT_PERIOD` and `REDOWNLOAD_BUFFER_SIZE` into [`src/headerssync-params.h`](/src/headerssync-params.h).
    


    l0rinc commented at 10:50 am on May 27, 2025:

    hodlinator commented at 12:59 pm on May 30, 2025:
    Thanks, missed that!
  18. in src/headerssync.cpp:14 in e9f4f6a5d9 outdated
    23-constexpr size_t REDOWNLOAD_BUFFER_SIZE{14827}; // 14827/624 = ~23.8 commitments
    24-
    25-// Our memory analysis assumes 48 bytes for a CompressedHeader (so we should
    26-// re-calculate parameters if we compress further)
    27+// Our memory analysis in headerssync-params.py assumes 48 bytes for a
    28+// CompressedHeader (we should re-calculate parameters if we compress further).
    


    l0rinc commented at 10:54 am on May 27, 2025:

    I find the new comment slightly more confusing - the point is that the cpp and py assumptions should align, right?

    0static_assert(sizeof(CompressedHeader) == 48); // align with headerssync-params.py
    

    hodlinator commented at 12:55 pm on May 30, 2025:

    The new comment is replacing the first (mention of python file) and last comments here: https://github.com/bitcoin/bitcoin/blob/11a2d3a63e90cdc1920ede3c67d52a9c72860e6b/src/headerssync.cpp#L12-L25

    Your comment would probably be enough but I’d rather keep some more context.

  19. in src/headerssync-params.h:11 in e9f4f6a5d9 outdated
     6+#define BITCOIN_HEADERSSYNC_PARAMS_H
     7+
     8+// The two constants below are computed using the simulation script in
     9+// contrib/devtools/headerssync-params.py.
    10+
    11+//! Store one header commitment per HEADER_COMMITMENT_PERIOD blocks.
    


    l0rinc commented at 10:56 am on May 27, 2025:

    nit: I find it a bit circular that the comment explaining a constant references the upcoming constant itself

    0//! Commitment period, in blocks
    

    hodlinator commented at 12:49 pm on May 30, 2025:
    Added to separate WIP branch modifying generator script.

    hodlinator commented at 1:44 pm on June 4, 2025:

    Latest push has:

    0struct HeadersSyncParams {
    1    //! Distance in blocks between header commitments.
    2    size_t commitment_period;
    3...
    

    in src/kernel/chainparams.h.

  20. l0rinc changes_requested
  21. l0rinc commented at 11:35 am on May 27, 2025: contributor

    Concept ACK – this PR replaces the imminently outdated hard-coded test target of 15'000 headers with a value derived from the live REDOWNLOAD_BUFFER_SIZE constant. Moving both header-sync tuning constants into a shared header, and making the unit test reference them, keeps the test exercising the ‘buffer-full’ path as the parameters grow each release.

    Recommendations:

    • split the series into more commits, separating low-risk refactors from ones that change behaviour: strict move-only; refactor / cleanup with no behavioural change; extra assertions; wiring the test to the new parameter header;
    • in the test file, wrap each independent ProcessNextHeaders call inside {} blocks and use structured bindings so temporary variables aren’t reused accidentally;
    • rename the new file to headerssync_params.h (underscore instead of -) for consistency with other similar headers;
    • update contrib/devtools/README.md to reference the header instead of the .cpp;
    • Tweak a few comments whose wording is unclear or slightly misleading.
  22. TheCharlatan commented at 10:09 am on May 30, 2025: contributor
    Concept ACK
  23. hodlinator force-pushed on May 30, 2025
  24. hodlinator commented at 8:37 pm on May 30, 2025: contributor
    Thanks for your feedback so far @l0rinc! Took most of your suggestions (https://github.com/bitcoin/bitcoin/pull/32579#pullrequestreview-2857912770).
  25. refactor(test): Extract constants ahead of breakup into functions
    Needed to make arith_uint256 constexpr-constructible.
    
    Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
    fab6f59e2e
  26. hodlinator force-pushed on May 30, 2025
  27. jonatack commented at 5:55 pm on June 2, 2025: member
    Concept ACK
  28. in src/headerssync.cpp:17 in bbe5609b84 outdated
    26-// re-calculate parameters if we compress further)
    27+// Our memory analysis in headerssync-params.py assumes 48 bytes for a
    28+// CompressedHeader (we should re-calculate parameters if we compress further).
    29 static_assert(sizeof(CompressedHeader) == 48);
    30 
    31 HeadersSyncState::HeadersSyncState(NodeId id, const Consensus::Params& consensus_params,
    


    hodlinator commented at 9:04 pm on June 2, 2025:

    Just realized a different possible approach: passing HEADER_COMMITMENT_PERIOD and REDOWNLOAD_BUFFER_SIZE values into the HeadersSyncState-ctor. Right now that aspect of the class is hardcoded for mainnet, despite the ctor accepting Consensus::Params. Having these values be sent in would enable tests to send in different (smaller) variables for regtest, while still testing behavior around the threshold.

    In fact, it would fit nicely with other values updated during the release process to have them written into the corresponding section (mainnet/regtest/etc) of src/kernel/chainparams.cpp.

    Started implementing that approach to see how it pans out.


    hodlinator commented at 1:48 pm on June 4, 2025:
    Implemented in latest push.
  29. hodlinator force-pushed on Jun 4, 2025
  30. hodlinator commented at 2:39 pm on June 4, 2025: contributor

    Latest push (42e746096b93b8026569817d8925750089d6daf4, compare):

    • Relaxes the hardcoding of the header commitment period and the redownload buffer size in HeadersSyncState.
      • This makes things more consistent with the constructor already taking Consensus::Params (implying different chains are supported).
      • Moves specification of the above constants from the top of headerssync.cpp into kernel/chainparams.cpp (similar to many other constants).
      • Returns the chainwork test suite back to only using an integer literal for TARGET_BLOCKS (instead of depending on an #included constant). Also now uses test-local values for commitment period and redownload buffer size.
    • Adds a warning in HeadersSyncState::PopHeadersReadyForAcceptance() for the case when we reach minimum PoW before exceeding the redownload buffer size together with tests.

    (Updated PR description).

  31. hodlinator renamed this:
    headerssync: Keep tests ahead of increasing params
    headerssync: Preempt unrealistic unit test behavior
    on Jun 4, 2025
  32. in src/test/headers_sync_chainwork_tests.cpp:21 in fab6f59e2e outdated
    30-
    31-void HeadersGeneratorSetup::FindProofOfWork(CBlockHeader& starting_header)
    32+constexpr size_t TARGET_BLOCKS{15'000};
    33+constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
    34+
    35+/** Search for a nonce to meet (regtest) proof of work */
    


    l0rinc commented at 1:00 pm on June 17, 2025:
    I don’t think it’s regtest specific - is it?

    hodlinator commented at 7:08 am on June 23, 2025:

    The test is running on RegTest:

    https://github.com/bitcoin/bitcoin/blob/42e746096b93b8026569817d8925750089d6daf4/src/test/headers_sync_chainwork_tests.cpp#L72

    I’m guessing including the comment is meant to assuage fears that this would be an intensive operation.


    l0rinc commented at 2:14 pm on June 24, 2025:
    ah, so it’s jut a reassurance that it doesn’t need a fancy asic, it can mine a block in no time - makes sense
  33. in src/test/headers_sync_chainwork_tests.cpp:86 in 7d45872a05 outdated
    85     // initially and then the rest.
    86-    headers_batch.insert(headers_batch.end(), std::next(first_chain.begin()), first_chain.end());
    87-
    88     hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    89-    (void)hss->ProcessNextHeaders({first_chain.front()}, true);
    90+    (void)hss->ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    


    l0rinc commented at 1:07 pm on June 17, 2025:

    Doc states: by delivering 1 header initially and then the rest. Could we clarify this by code instead? We don’t really have head/tail like in many functional programming languages, but maybe we could do:

    0    const std::span headers_batch{second_chain};
    1    (void)hss->ProcessNextHeaders(headers_batch.first(1), true);
    2...
    3    result = hss->ProcessNextHeaders(headers_batch.last(headers_batch.size() - 1), false);
    

    hodlinator commented at 7:32 am on June 23, 2025:

    Would rather not go back to having an intermediate headers_batch variable. Changed to {{second_chain.front()}} + std::span{second_chain}.last(second_chain.size() - 1).

    Kept the almost the existing style in HappyPath since I couldn’t see an improvement to {first_chain.begin() + REDOWNLOAD_BUFFER_SIZE, 1}.

  34. in src/test/headers_sync_chainwork_tests.cpp:125 in 7d45872a05 outdated
    115@@ -119,15 +116,13 @@ BOOST_AUTO_TEST_CASE(headers_sync_state)
    116     hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    117     BOOST_CHECK(hss->GetState() == HeadersSyncState::State::PRESYNC);
    118      // Pretend just the first message is "full", so we don't abort.
    119-    (void)hss->ProcessNextHeaders({second_chain.front()}, true);
    120+    (void)hss->ProcessNextHeaders(std::span{second_chain.begin(), 1}, true);
    121     BOOST_CHECK(hss->GetState() == HeadersSyncState::State::PRESYNC);
    122 
    123-    headers_batch.clear();
    


    l0rinc commented at 1:24 pm on June 17, 2025:
    👍 for getting rid of this
  35. in src/test/headers_sync_chainwork_tests.cpp:63 in a0cd196d2f outdated
    59@@ -60,15 +60,23 @@ BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, RegTestingSetup)
    60 //    updates to the REDOWNLOAD phase successfully.
    61 // 2. Then we deliver the second set of headers and verify that they fail
    62 //    processing (presumably due to commitments not matching).
    63-// 3. Finally, we verify that repeating with the first set of headers in both
    64-//    phases is successful.
    65+static void SneakyRedownload(const CBlockIndex* chain_start,
    


    l0rinc commented at 1:32 pm on June 17, 2025:
    👍 for splitting these up. Is it important to keep 1) and 2) as a single method, given that we thought it’s best described in two separate list items? Since you grouped them into a “redownload”, it’s probably the comments that should be adjusted.

    hodlinator commented at 7:38 am on June 23, 2025:
    Agreed that 1 & 2 should ideally be combined if this were written from scratch, but I prefer keeping the current approach to make the diff less jarring compared to master. Open to re-evaluate if other reviewers weigh in.

    l0rinc commented at 2:55 pm on June 24, 2025:
    Can we add 2 helpers in that case? I really dislike that the comments don’t follow the code’s structure.

    hodlinator commented at 8:33 pm on June 25, 2025:
    Renumbered in the “Improve comments”-commit.
  36. in src/test/headers_sync_chainwork_tests.cpp:110 in 79be6f7744 outdated
    109-
    110     // Feed the first chain to HeadersSyncState, by delivering 1 header
    111     // initially and then the rest.
    112-    hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    113-    (void)hss->ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    114+    HeadersSyncState hss{CreateState(chain_start, Params())};
    


    l0rinc commented at 1:39 pm on June 17, 2025:
    nit: Maybe mention in the previous commit message that “don’t worry, I’m not gonna’ leave the inits like that” :D
  37. in src/test/headers_sync_chainwork_tests.cpp:149 in 79be6f7744 outdated
    163-    HeadersSyncState::ProcessingResult result;
    164     // Verify that just trying to process the second chain would not succeed
    165     // (too little work).
    166-    hss.reset(new HeadersSyncState(0, Params().GetConsensus(), chain_start, CHAIN_WORK));
    167-    BOOST_CHECK(hss->GetState() == HeadersSyncState::State::PRESYNC);
    168+    HeadersSyncState hss{CreateState(chain_start, Params())};
    


    l0rinc commented at 1:43 pm on June 17, 2025:
    nit: I don’t particularly like the abbreviated hss name - would const auto sync_state{CreateState(chain_start, Params())}; maybe be more descriptive?

    hodlinator commented at 7:48 am on June 23, 2025:
    net_processing.cpp uses m_headers_sync, but I’m totally fine with hss-abbreviation, especially if we keep spelling out HeadersSyncState. Makes for a slightly less stark diff with master.
  38. in src/test/headers_sync_chainwork_tests.cpp:113 in 667fd31e74 outdated
    108@@ -109,24 +109,35 @@ static void SneakyRedownload(const CBlockIndex* chain_start,
    109     // initially and then the rest.
    110     HeadersSyncState hss{CreateState(chain_start, Params())};
    111     {
    112-        (void)hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    113+        // Just feed one header and check state.
    114+        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    


    l0rinc commented at 7:24 am on June 20, 2025:

    nit:

    0        const auto result{hss.ProcessNextHeaders({first_chain.begin(), 1}, /*full_headers_message=*/true)};
    

    hodlinator commented at 8:05 am on June 23, 2025:
    Replaced with CHECK_RESULT() macro from your suggestion in #32579 (review). Sprinkled some less obvious /*full_headers_message=*/ around.
  39. in src/test/headers_sync_chainwork_tests.cpp:114 in 667fd31e74 outdated
    108@@ -109,24 +109,35 @@ static void SneakyRedownload(const CBlockIndex* chain_start,
    109     // initially and then the rest.
    110     HeadersSyncState hss{CreateState(chain_start, Params())};
    111     {
    112-        (void)hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true);
    113+        // Just feed one header and check state.
    114+        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    115+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    


    l0rinc commented at 7:39 am on June 20, 2025:
    HeadersSyncState::State::PRESYNC is very verbose in a file called headers_sync_chainwork_tests - can we using State = HeadersSyncState::State; here instead to reduce the noise somewhat?

    hodlinator commented at 7:52 am on June 23, 2025:
    Done in the same commit as we switch to BOOST_REQUIRE_EQUAL.
  40. in src/test/headers_sync_chainwork_tests.cpp:68 in a0cd196d2f outdated
    65+static void SneakyRedownload(const CBlockIndex* chain_start,
    66+        const std::vector<CBlockHeader>& first_chain,
    67+        const std::vector<CBlockHeader>& second_chain);
    68+// 3. Verify that repeating with the first set of headers in both phases is
    69+//    successful.
    70+static void HappyPath(const CBlockIndex* chain_start,
    


    l0rinc commented at 7:54 am on June 20, 2025:
    do we really need these forward declarations here? Wouldn’t it be simpler to just put the test at the end of the file like we’re doing in other places e.g. https://github.com/bitcoin/bitcoin/blob/master/src/test/addrman_tests.cpp#L23-L61 ? If we do need the prototypes can we maybe remove the arg names to reduce the noise?

    hodlinator commented at 8:19 am on June 23, 2025:

    do we really need these forward declarations here? Wouldn’t it be simpler to just put the test at the end of the file like we’re doing in other places e.g. https://github.com/bitcoin/bitcoin/blob/master/src/test/addrman_tests.cpp#L23-L61 ?

    Did it in order to weave the breakup into the existing comments, while avoiding to re-order chunks of code to minimize diffs.

    If we do need the prototypes can we maybe remove the arg names to reduce the noise?

    Done - not used to this style, but warming up to it. :)


    l0rinc commented at 2:58 pm on June 24, 2025:

    chunks of code to minimize diffs

    We’ve rewritten the tests at this stage, diffs and the function prototypes don’t help in my opinion. But if you insist, I don’t mind, please resolve the comment

  41. in src/test/headers_sync_chainwork_tests.cpp:215 in 42e746096b outdated
    286+        BOOST_CHECK_EQUAL(result.pow_validated_headers.front().hashPrevBlock, first_after_genesis.GetHash());
    287+    }
    288+}
    289+
    290+static void TooLittleWork(const CBlockIndex* chain_start,
    291+        const std::vector<CBlockHeader>& second_chain)
    


    l0rinc commented at 9:12 am on June 20, 2025:
    first_chain/second_chain is a bit verbose given how often we’re using them

    hodlinator commented at 8:21 am on June 23, 2025:
    Would rather keep the names to minimize diffs. sufficient_chain & insufficient_chain would have been more descriptive but longer.
  42. in src/test/headers_sync_chainwork_tests.cpp:92 in 42e746096b outdated
    88@@ -87,6 +89,9 @@ static void HappyPath(const CBlockIndex* chain_start,
    89 static void TooLittleWork(const CBlockIndex* chain_start,
    90         const std::vector<CBlockHeader>& second_chain);
    91 
    92+static void TooBigBuffer(const CBlockIndex* chain_start,
    


    l0rinc commented at 9:24 am on June 20, 2025:
    this scenario could also use a comment like the rest

    hodlinator commented at 9:06 am on June 23, 2025:
    Done.
  43. in src/test/headers_sync_chainwork_tests.cpp:170 in 42e746096b outdated
    165+        if (str == nullptr) {
    166+            return false; // Disable exception for not finding a match.
    167+        } else {
    168+            throw std::runtime_error{strprintf("'%s' should not exist in debug log\n", str)};
    169+        }
    170+    }};
    


    l0rinc commented at 9:35 am on June 20, 2025:

    This is a bit hacky - can we add a negated helper to logging.h?

     0// Test fails if the pattern *DOES NOT* show up.
     1#define ASSERT_DEBUG_LOG(message) DebugLogHelper UNIQUE_NAME(debugloghelper)(message)
     2// Test fails if the pattern *DOES* show up.
     3#define ASSERT_NO_DEBUG_LOG(message)                                              \
     4    DebugLogHelper UNIQUE_NAME(nologhelper){                                      \
     5        message,                                                                  \
     6        [](const std::string* line) {                                             \
     7            if (line) throw std::runtime_error("unexpected log line: " + *line);  \
     8            return false; /* suppress default 'not found' failure */              \
     9        }                                                                         \
    10    }
    

    And use it here as:

    0    // During normal operation we shouldn't get the redownload buffer size warning.
    1    ASSERT_NO_DEBUG_LOG(BUFFER_SIZE_WARNING);
    

    hodlinator commented at 9:20 am on June 23, 2025:
    Originally resisted the impulse since there is only one case, but done now. It mirrors what we do in functional Python tests after all.
  44. in src/test/headers_sync_chainwork_tests.cpp:264 in 42e746096b outdated
    259+                         HeadersSyncParams{
    260+                             .commitment_period = COMMITMENT_PERIOD,
    261+                             .redownload_buffer_size = first_chain.size(),
    262+                         },
    263+                         chain_start,
    264+                         /*minimum_required_work=*/CHAIN_WORK};
    


    l0rinc commented at 9:36 am on June 20, 2025:

    we should be able to reuse CreateState here - if we make HeadersSyncParams configurable:

    0static HeadersSyncState CreateState(const CChainParams& chain_params, const HeadersSyncParams& sync_params, const CBlockIndex* chain_start)
    1{
    2    return {/*id=*/0,
    3            chain_params.GetConsensus(),
    4            sync_params,
    5            chain_start,
    6            /*minimum_required_work=*/CHAIN_WORK};
    7}
    

    and

    0// Scenario that intentionally uses an oversized buffer to trigger the warning.
    1static void TooBigBuffer(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1)
    2{
    3    // Intentionally too big redownload buffer in order to trigger warning.
    4    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, chain1.size()}, chain_start)};
    5    ...
    

    hodlinator commented at 9:31 am on June 23, 2025:
    Taken in broad strokes.
  45. in src/test/headers_sync_chainwork_tests.cpp:265 in 42e746096b outdated
    260+                             .commitment_period = COMMITMENT_PERIOD,
    261+                             .redownload_buffer_size = first_chain.size(),
    262+                         },
    263+                         chain_start,
    264+                         /*minimum_required_work=*/CHAIN_WORK};
    265+    (void)hss.ProcessNextHeaders(first_chain, true);
    


    l0rinc commented at 9:37 am on June 20, 2025:

    can we validate this as well instead of just voiding it?

    0CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    1             state, State::REDOWNLOAD,
    2             /*exp_success=*/true,
    3             /*exp_request_more=*/true,
    4             /*exp_headers_size=*/0);
    
  46. in src/test/headers_sync_chainwork_tests.cpp:272 in 42e746096b outdated
    334+
    335+    {
    336+        ASSERT_DEBUG_LOG(BUFFER_SIZE_WARNING);
    337+        auto result{hss.ProcessNextHeaders(first_chain, true)};
    338+        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    339+        BOOST_CHECK(result.success);
    


    l0rinc commented at 9:39 am on June 20, 2025:

    We could validate request_more and pow_validated_headers here as well:

    0CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    1             state, State::FINAL,
    2             /*exp_success=*/true,
    3             /*exp_request_more=*/false,
    4             /*exp_headers_size=*/chain1.size());
    
  47. in src/test/headers_sync_chainwork_tests.cpp:127 in 875e89c82f outdated
    136+        BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    137+    }
    138+
    139+    {
    140+        // Try to sneakily feed back the second chain.
    141+        auto result{hss.ProcessNextHeaders(second_chain, true)};
    


    l0rinc commented at 9:42 am on June 20, 2025:

    👍 for reducing the scope of each separate check.

    Was hoping we could compare the result object directly as:

    0BOOST_CHECK_EQUAL(result, (HeadersSyncState::ProcessingResult{
    1    .pow_validated_headers = {},
    2    .success = true,
    3    .request_more = true,
    4}));
    

    But unfortunately we’d need to add equality operators to ProcessingResult and CBlockHeader which would defeat the purpose (+ the operator<<(std::ostream& for printing)…

    But if we extract a validation helper function instead, we’d have keep the scope reduction and make the differences between the checks less noisy:

    0// Standard set of checks common to all scenarios; macro keeps failure lines at the call-site.
    1#define CHECK_RESULT(res, state, exp_state, exp_success, exp_request_more, exp_headers_size) \
    2    do {                                                                                     \
    3        const auto& [res_headers, res_success, res_request_more]{res};                       \
    4        BOOST_REQUIRE_EQUAL(state.GetState(), exp_state);                                    \
    5        BOOST_CHECK_EQUAL(res_headers.size(), exp_headers_size);                             \
    6        BOOST_CHECK_EQUAL(res_success, exp_success);                                         \
    7        BOOST_CHECK_EQUAL(res_request_more, exp_request_more);                               \
    8    } while (false)
    

    and using it as e.g.

     0// Repeat the second set of headers in both phases to demonstrate
     1// behavior when the chain a peer provides has too little work.
     2static void TooLittleWork(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain2)
     3{
     4    // Verify that just trying to process the second chain would not succeed (too little work).
     5    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
     6
     7    // Pretend just the first message is "full", so we don't abort.
     8    CHECK_RESULT(state.ProcessNextHeaders({chain2.begin(), 1}, /*full_headers_message=*/true),
     9                 state, State::PRESYNC,
    10                 /*exp_success=*/true,
    11                 /*exp_request_more=*/true,
    12                 /*exp_headers_size=*/0);
    13
    14    // Tell the sync logic that the headers message was not full, implying no
    15    // more headers can be requested. For a low-work-chain, this should cause
    16    // the sync to end with no headers for acceptance.
    17    CHECK_RESULT(state.ProcessNextHeaders({chain2.begin() + 1, chain2.end()}, /*full_headers_message=*/false),
    18                 state, State::FINAL,
    19                 /*exp_success=*/true,
    20                 /*exp_request_more=*/false,
    21                 /*exp_headers_size=*/0);
    22}
    

    hodlinator commented at 12:45 pm on June 23, 2025:
    Taken, added exp_pow_validated_prev and exp_locator_hash-parameters. Feels good to verify all parameters, also always in the same order.
  48. in src/kernel/chainparams.cpp:193 in 6a5914ea5f outdated
    188@@ -189,6 +189,12 @@ class CMainParams : public CChainParams {
    189             .tx_count = 1161875261,
    190             .dTxRate  = 4.620728156243148,
    191         };
    192+
    193+        // Generated by headerssync-params.py on 2025-03-04.
    


    l0rinc commented at 9:48 am on June 20, 2025:
    We’ll update this during the v30 release?

    hodlinator commented at 8:32 am on June 23, 2025:

    As release-process.md documents, it’s not critical to update on every release. It’s mostly about fine-tuning memory usage under adversarial conditions, and the values don’t tend to change that much. But I would not be surprised if it were updated since it was updated during the previous 2 releases:

    • v29: 11a2d3a63e90cdc1920ede3c67d52a9c72860e6b
    • v28: 221809b81cfcecb04050915eebacffda2599da42

    Also, we’re slated to remove checkpoints (#31649) in v30 which might further encourage the release manager to do this.

  49. in contrib/devtools/headerssync-params.py:343 in 6a5914ea5f outdated
    345+    print(f"memory usage on mainchain for release until {TIME:%Y-%m-%d} is:")
    346     print()
    347-    print("//! Only feed headers to validation once this many headers on top have been")
    348-    print("//! received and validated against commitments.")
    349-    print(f"constexpr size_t REDOWNLOAD_BUFFER_SIZE{{{bufsize}}};"
    350+    print(f"        // Generated by headerssync-params.py on {datetime.today():%Y-%m-%d}.")
    


    l0rinc commented at 9:48 am on June 20, 2025:
    👍
  50. in contrib/devtools/headerssync-params.py:347 in 6a5914ea5f outdated
    349-    print(f"constexpr size_t REDOWNLOAD_BUFFER_SIZE{{{bufsize}}};"
    350+    print(f"        // Generated by headerssync-params.py on {datetime.today():%Y-%m-%d}.")
    351+    print( "        m_headers_sync_params = HeadersSyncParams{")
    352+    print(f"            .commitment_period = {period},")
    353+    print(f"            .redownload_buffer_size = {bufsize},"
    354           f" // {bufsize}/{period} = ~{bufsize/period:.1f} commitments")
    


    l0rinc commented at 9:49 am on June 20, 2025:

    nit: I’d join this with the previous line, it’s just confusing this way. And can we round this (given we already have a leading ~) instead of adding a weird decimal (what is 0.8 commitment mean?)

    And why are we repeating the division in the comment, why not just:

    0print(f"            .redownload_buffer_size = {bufsize}}; // ~{round(bufsize / period)} commitments")
    

    resulting in something like:

    0        // Generated by headerssync-params.py on 2025-03-04.
    1        m_headers_sync_params = HeadersSyncParams{
    2            .commitment_period = 624,
    3            .redownload_buffer_size = 14827}; // ~24 commitments
    

    hodlinator commented at 8:54 am on June 23, 2025:

    I’d join this with the previous line, it’s just confusing this way.

    Prefer keeping this line unchanged compared to master (would rather have lines closer to 80 than 120 chars here).

    can we round this (given we already have a leading ~) instead of adding a weird decimal (what is 0.8 commitment mean?)

    Including one decimal makes it more likely that the number changes between releases, indicating some kind of usefulness + precision of simulation. Not sure if that was what the original authors intended.

    And why are we repeating the division in the comment

    I think keeping the division makes it slightly clearer what the number is to readers of the C++ source.


    l0rinc commented at 2:38 pm on June 24, 2025:

    makes it more likely that the number changes between releases

    That sounds like a counter-argument to me, but don’t have strong feelings about it, if you disagree, I don’t mind, please resolve it.

  51. in src/headerssync.cpp:13 in 6a5914ea5f outdated
    20-//! received and validated against commitments.
    21-constexpr size_t REDOWNLOAD_BUFFER_SIZE{14827}; // 14827/624 = ~23.8 commitments
    22-
    23-// Our memory analysis assumes 48 bytes for a CompressedHeader (so we should
    24-// re-calculate parameters if we compress further)
    25+// Our memory analysis in headerssync-params.py assumes 48 bytes for a
    


    l0rinc commented at 9:51 am on June 20, 2025:
    • 48 is already stated in the code, no need to repeat it in the comment
    • we should re-calculate parameters if we compress further - doesn’t the static assert already obviate that?

    hodlinator commented at 12:27 pm on June 23, 2025:

    48 is already stated in the code, no need to repeat it in the comment

    Replaced with “this many”.

    we should re-calculate parameters if we compress further - doesn’t the static assert already obviate that?

    Don’t think so? This is about prompting whoever changes the struct to also alter and re-run the Python script.

  52. in src/test/headers_sync_chainwork_tests.cpp:165 in 04a0b6609f outdated
    161@@ -161,17 +162,45 @@ static void HappyPath(const CBlockIndex* chain_start,
    162         BOOST_CHECK(result.request_more);
    163         BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    164         // The locator should reset to genesis.
    165-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    166+        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    


    l0rinc commented at 9:59 am on June 20, 2025:
    nice, we don’t need the comment anymore after this change

    hodlinator commented at 12:22 pm on June 23, 2025:
    Still think it is good to document why something is expected.

    l0rinc commented at 3:01 pm on June 24, 2025:
    Not sure I understand, especially in the new code which states something like locator == genesis - which is basically the same as the comment
  53. in src/headerssync.cpp:283 in 42e746096b outdated
    277@@ -278,6 +278,14 @@ std::vector<CBlockHeader> HeadersSyncState::PopHeadersReadyForAcceptance()
    278     Assume(m_download_state == State::REDOWNLOAD);
    279     if (m_download_state != State::REDOWNLOAD) return ret;
    280 
    281+    if (m_process_all_remaining_headers && m_chain_start->nHeight == 0 &&
    282+        m_redownload_buffer_first_prev_hash == m_chain_start->GetBlockHash() &&
    283+        m_redownloaded_headers.size() <= m_params.redownload_buffer_size) {
    


    l0rinc commented at 10:01 am on June 20, 2025:
    Q: I can’t immediately tell if this should be triggered for equality as well - please just resolve if it’s a stupid question.

    hodlinator commented at 12:34 pm on June 23, 2025:

    Appreciate feedback on this!

    Are you asking whether m_redownloaded_headers.size() < m_params.redownload_buffer_size might be more correct?

    The while-condition below includes m_redownloaded_headers.size() > m_params.redownload_buffer_size as one of the conditions to start feeding back PoW-validated headers. This means using m_redownloaded_headers.size() < m_params.redownload_buffer_size for the condition above would give us no headers and no warning when the buffer size equals the parameter.


    l0rinc commented at 2:46 pm on June 24, 2025:
    Hmmm, if it has to be the opposite of the while below, would it make sense to extract the common parts to explain it with code (your explanation makes sense, would prefer if the code already made it simpler so that I don’t have to check the code that’s after the current one to understand it).

    hodlinator commented at 8:34 pm on June 25, 2025:
    Broke out bool buffer_exceeded, let me know if you think it’s an improvement.

    l0rinc commented at 2:29 pm on June 27, 2025:
    I don’t like the new repetition - will have to debug it locally to understand why we run the condition outside of the loop in the first place 🤔
  54. l0rinc changes_requested
  55. l0rinc commented at 10:17 am on June 20, 2025: contributor

    I might have gone a bit overboard with the test review, but I think we can make it a lot less noisy - here’s the patch of many of my suggestions for convenience:

      0diff --git a/src/kernel/chainparams.cpp b/src/kernel/chainparams.cpp
      1index d64dd43aba..658ea7f2c3 100644
      2--- a/src/kernel/chainparams.cpp
      3+++ b/src/kernel/chainparams.cpp
      4@@ -193,8 +193,7 @@ public:
      5         // Generated by headerssync-params.py on 2025-03-04.
      6         m_headers_sync_params = HeadersSyncParams{
      7             .commitment_period = 624,
      8-            .redownload_buffer_size = 14827, // 14827/624 = ~23.8 commitments
      9-        };
     10+            .redownload_buffer_size = 14827}; // ~24 commitments
     11     }
     12 };
     13 
     14diff --git a/src/test/headers_sync_chainwork_tests.cpp b/src/test/headers_sync_chainwork_tests.cpp
     15index 1f85c69838..ccbbc1be50 100644
     16--- a/src/test/headers_sync_chainwork_tests.cpp
     17+++ b/src/test/headers_sync_chainwork_tests.cpp
     18@@ -13,10 +13,13 @@
     19 #include <validation.h>
     20 
     21 #include <cstddef>
     22+#include <optional>
     23 #include <vector>
     24 
     25 #include <boost/test/unit_test.hpp>
     26 
     27+using State = HeadersSyncState::State;
     28+
     29 constexpr size_t TARGET_BLOCKS{15'000};
     30 constexpr arith_uint256 CHAIN_WORK{TARGET_BLOCKS * 2};
     31 constexpr size_t REDOWNLOAD_BUFFER_SIZE{TARGET_BLOCKS - (MAX_HEADERS_RESULTS + 123)};
     32@@ -37,18 +40,18 @@ static void FindProofOfWork(CBlockHeader& starting_header)
     33  * prev_time, and with a fixed merkle root hash.
     34  */
     35 static void GenerateHeaders(std::vector<CBlockHeader>& headers,
     36-        size_t count, const uint256& starting_hash, const int nVersion, int prev_time,
     37-        const uint256& merkle_root, const uint32_t nBits)
     38+                            size_t count, const uint256& starting_hash, int nVersion, int prev_time,
     39+                            const uint256& merkle_root, uint32_t nBits)
     40 {
     41-    uint256 prev_hash = starting_hash;
     42+    uint256 prev_hash{starting_hash};
     43 
     44     while (headers.size() < count) {
     45         headers.emplace_back();
     46-        CBlockHeader& next_header = headers.back();
     47+        CBlockHeader& next_header{headers.back()};
     48         next_header.nVersion = nVersion;
     49         next_header.hashPrevBlock = prev_hash;
     50         next_header.hashMerkleRoot = merkle_root;
     51-        next_header.nTime = prev_time+1;
     52+        next_header.nTime = prev_time + 1;
     53         next_header.nBits = nBits;
     54 
     55         FindProofOfWork(next_header);
     56@@ -57,220 +60,167 @@ static void GenerateHeaders(std::vector<CBlockHeader>& headers,
     57     }
     58 }
     59 
     60-static HeadersSyncState CreateState(const CBlockIndex* chain_start, const CChainParams& chain_params)
     61+static HeadersSyncState CreateState(const CChainParams& chain_params, const HeadersSyncParams& sync_params, const CBlockIndex* chain_start)
     62 {
     63     return {/*id=*/0,
     64             chain_params.GetConsensus(),
     65-            HeadersSyncParams{
     66-                .commitment_period = COMMITMENT_PERIOD,
     67-                .redownload_buffer_size = REDOWNLOAD_BUFFER_SIZE,
     68-            },
     69+            sync_params,
     70             chain_start,
     71-            /*minimum_required_work=*/CHAIN_WORK};
     72+           /*minimum_required_work=*/CHAIN_WORK};
     73 }
     74 
     75-BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, RegTestingSetup)
     76-
     77-// In this test, we construct two sets of headers from genesis, one with
     78-// sufficient proof of work and one without.
     79-// 1. We deliver the first set of headers and verify that the headers sync state
     80-//    updates to the REDOWNLOAD phase successfully.
     81-// 2. Then we deliver the second set of headers and verify that they fail
     82-//    processing (presumably due to commitments not matching).
     83-static void SneakyRedownload(const CBlockIndex* chain_start,
     84-        const std::vector<CBlockHeader>& first_chain,
     85-        const std::vector<CBlockHeader>& second_chain);
     86-// 3. Verify that repeating with the first set of headers in both phases is
     87-//    successful.
     88-static void HappyPath(const CBlockIndex* chain_start,
     89-        const std::vector<CBlockHeader>& first_chain);
     90-// 4. Finally, repeat the second set of headers in both phases to demonstrate
     91-//    behavior when the chain a peer provides has too little work.
     92-static void TooLittleWork(const CBlockIndex* chain_start,
     93-        const std::vector<CBlockHeader>& second_chain);
     94-
     95-static void TooBigBuffer(const CBlockIndex* chain_start,
     96-        const std::vector<CBlockHeader>& first_chain);
     97-
     98-BOOST_AUTO_TEST_CASE(headers_sync_state)
     99+// Standard set of checks common to all scenarios; macro keeps failure lines at the call-site.
    100+#define CHECK_RESULT(res, state, exp_state, exp_success, exp_request_more, exp_headers_size) \
    101+    do {                                                                                     \
    102+        const auto& [res_headers, res_success, res_request_more]{res};                       \
    103+        BOOST_REQUIRE_EQUAL(state.GetState(), exp_state);                                    \
    104+        BOOST_CHECK_EQUAL(res_headers.size(), exp_headers_size);                             \
    105+        BOOST_CHECK_EQUAL(res_success, exp_success);                                         \
    106+        BOOST_CHECK_EQUAL(res_request_more, exp_request_more);                               \
    107+    } while (false)
    108+
    109+// Deliver the first set of headers from genesis to reach REDOWNLOAD,
    110+// then deliver the second set and verify that commitments don’t match.
    111+static void SneakyRedownload(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1, const std::vector<CBlockHeader>& chain2)
    112 {
    113-    std::vector<CBlockHeader> first_chain;
    114-    std::vector<CBlockHeader> second_chain;
    115-
    116-    const auto genesis{Params().GenesisBlock()};
    117-
    118-    // Generate headers for two different chains (using differing merkle roots
    119-    // to ensure the headers are different).
    120-    GenerateHeaders(first_chain, TARGET_BLOCKS - 1, genesis.GetHash(), genesis.nVersion,
    121-                    genesis.nTime, /*merkle_root=*/uint256::ZERO, genesis.nBits);
    122-    GenerateHeaders(second_chain, TARGET_BLOCKS - 2, genesis.GetHash(), genesis.nVersion,
    123-                    genesis.nTime, /*merkle_root=*/uint256::ONE, genesis.nBits);
    124-
    125-    const CBlockIndex* chain_start = WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(genesis.GetHash()));
    126+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
    127+    const auto genesis_hash{Params().GenesisBlock().GetHash()};
    128 
    129-    SneakyRedownload(chain_start, first_chain, second_chain);
    130-    HappyPath(chain_start, first_chain);
    131-    TooLittleWork(chain_start, second_chain);
    132-    TooBigBuffer(chain_start, first_chain);
    133+    const std::span headers{chain1};
    134+    CHECK_RESULT(state.ProcessNextHeaders(headers.first(1), /*full_headers_message=*/true),
    135+                 state, State::PRESYNC,
    136+                 /*exp_success=*/true,
    137+                 /*exp_request_more=*/true,
    138+                 /*exp_headers_size=*/0);
    139+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), chain1.front().GetHash());
    140+
    141+    // Pretend the message is still "full", so we don't abort.
    142+    // This chain should look valid, and we should have met the proof-of-work
    143+    // requirement during PRESYNC and transitioned to REDOWNLOAD.
    144+    CHECK_RESULT(state.ProcessNextHeaders(headers.last(headers.size() - 1), /*full_headers_message=*/true),
    145+                 state, State::REDOWNLOAD,
    146+                 /*exp_success=*/true,
    147+                 /*exp_request_more=*/true,
    148+                 /*exp_headers_size=*/0);
    149+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    150+
    151+    // Try to sneakily feed back the second chain during REDOWNLOAD.
    152+    CHECK_RESULT(state.ProcessNextHeaders(chain2, /*full_headers_message=*/true),
    153+                 state, State::FINAL,
    154+                 /*exp_success=*/false,
    155+                 /*exp_request_more=*/false,
    156+                 /*exp_headers_size=*/0);
    157 }
    158 
    159-static void SneakyRedownload(const CBlockIndex* chain_start,
    160-        const std::vector<CBlockHeader>& first_chain,
    161-        const std::vector<CBlockHeader>& second_chain)
    162+// Verify that repeating with the first set of headers in both phases is successful.
    163+static void HappyPath(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1)
    164 {
    165-    // Feed the first chain to HeadersSyncState, by delivering 1 header
    166-    // initially and then the rest.
    167-    HeadersSyncState hss{CreateState(chain_start, Params())};
    168-    {
    169-        // Just feed one header and check state.
    170-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), 1}, true)};
    171-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    172-        BOOST_CHECK(result.success);
    173-        BOOST_CHECK(result.request_more);
    174-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    175-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain.front().GetHash());
    176-    }
    177-
    178-    {
    179-        // Pretend the message is still "full", so we don't abort.
    180-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin() + 1, first_chain.end()}, true)};
    181-        // This chain should look valid, and we should have met the proof-of-work
    182-        // requirement during PRESYNC and transitioned to REDOWNLOAD.
    183-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    184-        BOOST_CHECK(result.success);
    185-        BOOST_CHECK(result.request_more);
    186-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    187-        // The locator should reset to genesis.
    188-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), Params().GenesisBlock().GetHash());
    189-    }
    190+    // Feed the first chain twice.
    191+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
    192+    const auto genesis_hash{Params().GenesisBlock().GetHash()};
    193+    const std::span headers{chain1};
    194 
    195-    {
    196-        // Try to sneakily feed back the second chain during REDOWNLOAD.
    197-        auto result{hss.ProcessNextHeaders(second_chain, true)};
    198-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    199-        BOOST_CHECK(!result.success); // Foiled! We detected mismatching headers.
    200-        BOOST_CHECK(!result.request_more);
    201-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    202-    }
    203+    // During normal operation we shouldn't get the redownload buffer size warning.
    204+    ASSERT_NO_DEBUG_LOG(BUFFER_SIZE_WARNING);
    205+
    206+    // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    207+    CHECK_RESULT(state.ProcessNextHeaders(headers, /*full_headers_message=*/true),
    208+                 state, State::REDOWNLOAD,
    209+                 /*exp_success=*/true,
    210+                 /*exp_request_more=*/true,
    211+                 /*exp_headers_size=*/0);
    212+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    213+
    214+    // Process only so that the internal threshold isn't met, meaning validated headers shouldn't be returned yet:
    215+    CHECK_RESULT(state.ProcessNextHeaders(headers.first(REDOWNLOAD_BUFFER_SIZE), /*full_headers_message=*/true),
    216+                 state, State::REDOWNLOAD,
    217+                 /*exp_success=*/true,
    218+                 /*exp_request_more=*/true,
    219+                 /*exp_headers_size=*/0);
    220+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), chain1[REDOWNLOAD_BUFFER_SIZE - 1].GetHash());
    221+
    222+    // Next header should make us exceed the threshold, but still not be done:
    223+    const auto res_threshold{state.ProcessNextHeaders(headers.subspan(REDOWNLOAD_BUFFER_SIZE, 1), /*full_headers_message=*/true)};
    224+    CHECK_RESULT(res_threshold,
    225+                 state, State::REDOWNLOAD,
    226+                 /*exp_success=*/true,
    227+                 /*exp_request_more=*/true,
    228+                 /*exp_headers_size=*/1);
    229+    BOOST_CHECK_EQUAL(state.NextHeadersRequestLocator().vHave.front(), chain1[REDOWNLOAD_BUFFER_SIZE].GetHash());
    230+
    231+    // Feed in remaining headers, meeting the work threshold again and completing the REDOWNLOAD phase:
    232+    CHECK_RESULT(state.ProcessNextHeaders(headers.subspan(REDOWNLOAD_BUFFER_SIZE + 1), /*full_headers_message=*/true),
    233+                 state, State::FINAL,
    234+                 /*exp_success=*/true,
    235+                 /*exp_request_more=*/false,
    236+                 /*exp_headers_size=*/chain1.size() - 1);
    237+    BOOST_CHECK_EQUAL(res_threshold.pow_validated_headers.front().hashPrevBlock, genesis_hash);
    238 }
    239 
    240-static void HappyPath(const CBlockIndex* chain_start,
    241-        const std::vector<CBlockHeader>& first_chain)
    242+// Repeat the second set of headers in both phases to demonstrate
    243+// behavior when the chain a peer provides has too little work.
    244+static void TooLittleWork(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain2)
    245 {
    246-    // This time we feed the first chain twice.
    247-    HeadersSyncState hss{CreateState(chain_start, Params())};
    248-
    249-    // During normal operation we shouldn't get the redownload buffer size warning.
    250-    DebugLogHelper forbidden{BUFFER_SIZE_WARNING, [](const std::string* str) {
    251-        if (str == nullptr) {
    252-            return false; // Disable exception for not finding a match.
    253-        } else {
    254-            throw std::runtime_error{strprintf("'%s' should not exist in debug log\n", str)};
    255-        }
    256-    }};
    257-
    258-    const auto genesis_hash{Params().GenesisBlock().GetHash()};
    259-    {
    260-        auto result{hss.ProcessNextHeaders(first_chain, true)};
    261-        // Switched from PRESYNC to REDOWNLOAD after reaching sufficient work:
    262-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    263-        BOOST_CHECK(result.success);
    264-        BOOST_CHECK(result.request_more);
    265-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    266-        // The locator should reset to genesis.
    267-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), genesis_hash);
    268-    }
    269-
    270-    {
    271-        // Process only so that the internal threshold isn't met, meaning validated
    272-        // headers shouldn't be returned yet:
    273-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin(), REDOWNLOAD_BUFFER_SIZE}, true)};
    274-        // Not done:
    275-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    276-        BOOST_CHECK(result.success);
    277-        BOOST_CHECK(result.request_more);
    278-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    279-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain[REDOWNLOAD_BUFFER_SIZE - 1].GetHash());
    280-    }
    281-
    282-    CBlockHeader first_after_genesis;
    283-    {
    284-        // Next header should make us exceed the threshold, but still not be done:
    285-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin() + REDOWNLOAD_BUFFER_SIZE, 1}, true)};
    286-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    287-        BOOST_CHECK(result.success);
    288-        BOOST_CHECK(result.request_more);
    289-        BOOST_REQUIRE_EQUAL(result.pow_validated_headers.size(), 1);
    290-        BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.front(), first_chain[REDOWNLOAD_BUFFER_SIZE].GetHash());
    291-        first_after_genesis = result.pow_validated_headers.front();
    292-        BOOST_CHECK_EQUAL(first_after_genesis.hashPrevBlock, genesis_hash);
    293-    }
    294-
    295-    {
    296-        // Feed in remaining headers, meeting the work threshold again and
    297-        // completing the REDOWNLOAD phase.
    298-        auto result{hss.ProcessNextHeaders(std::span{first_chain.begin() + REDOWNLOAD_BUFFER_SIZE + 1, first_chain.end()}, true)};
    299-        // Nothing left for the sync logic to do:
    300-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    301-        BOOST_CHECK(result.success);
    302-        BOOST_CHECK(!result.request_more);
    303-        // All headers except the one already returned above:
    304-        BOOST_REQUIRE_EQUAL(result.pow_validated_headers.size(), first_chain.size() - 1);
    305-        BOOST_CHECK_EQUAL(result.pow_validated_headers.front().hashPrevBlock, first_after_genesis.GetHash());
    306-    }
    307+    // Verify that just trying to process the second chain would not succeed (too little work).
    308+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, REDOWNLOAD_BUFFER_SIZE}, chain_start)};
    309+    const std::span headers{chain2};
    310+
    311+    // Pretend just the first message is "full", so we don't abort.
    312+    CHECK_RESULT(state.ProcessNextHeaders(headers.first(1), /*full_headers_message=*/true),
    313+                 state, State::PRESYNC,
    314+                 /*exp_success=*/true,
    315+                 /*exp_request_more=*/true,
    316+                 /*exp_headers_size=*/0);
    317+
    318+    // Non-full message causes sync to end with no headers accepted.
    319+    CHECK_RESULT(state.ProcessNextHeaders(headers.subspan(1), /*full_headers_message=*/false),
    320+                 state, State::FINAL,
    321+                 /*exp_success=*/true,
    322+                 /*exp_request_more=*/false,
    323+                 /*exp_headers_size=*/0);
    324 }
    325 
    326-static void TooLittleWork(const CBlockIndex* chain_start,
    327-        const std::vector<CBlockHeader>& second_chain)
    328+// Scenario that intentionally uses an oversized buffer to trigger the warning.
    329+static void TooBigBuffer(const CBlockIndex* chain_start, const std::vector<CBlockHeader>& chain1)
    330 {
    331-    // Verify that just trying to process the second chain would not succeed
    332-    // (too little work).
    333-    HeadersSyncState hss{CreateState(chain_start, Params())};
    334-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    335-    {
    336-        // Pretend just the first message is "full", so we don't abort.
    337-        auto result{hss.ProcessNextHeaders(std::span{second_chain.begin(), 1}, true)};
    338-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::PRESYNC);
    339-        BOOST_CHECK(result.success);
    340-        BOOST_CHECK(result.request_more);
    341-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    342-    }
    343-
    344-    {
    345-        // Tell the sync logic that the headers message was not full, implying no
    346-        // more headers can be requested. For a low-work-chain, this should cause
    347-        // the sync to end with no headers for acceptance.
    348-        auto result{hss.ProcessNextHeaders(std::span{second_chain.begin() + 1, second_chain.end()}, false)};
    349-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    350-        BOOST_CHECK(!result.request_more);
    351-        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), 0);
    352-        // Nevertheless, no validation errors should have been detected with the
    353-        // chain:
    354-        BOOST_CHECK(result.success);
    355-    }
    356+    // Intentionally too big redownload buffer in order to trigger warning.
    357+    HeadersSyncState state{CreateState(Params(), HeadersSyncParams{COMMITMENT_PERIOD, chain1.size()}, chain_start)};
    358+
    359+    CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    360+                 state, State::REDOWNLOAD,
    361+                 /*exp_success=*/true,
    362+                 /*exp_request_more=*/true,
    363+                 /*exp_headers_size=*/0);
    364+
    365+    ASSERT_DEBUG_LOG(BUFFER_SIZE_WARNING);
    366+    CHECK_RESULT(state.ProcessNextHeaders(chain1, /*full_headers_message=*/true),
    367+                 state, State::FINAL,
    368+                 /*exp_success=*/true,
    369+                 /*exp_request_more=*/false,
    370+                 /*exp_headers_size=*/chain1.size());
    371 }
    372 
    373-static void TooBigBuffer(const CBlockIndex* chain_start,
    374-        const std::vector<CBlockHeader>& first_chain)
    375+BOOST_FIXTURE_TEST_SUITE(headers_sync_chainwork_tests, RegTestingSetup)
    376+
    377+BOOST_AUTO_TEST_CASE(headers_sync_state)
    378 {
    379-    // Intentionally too big redownload buffer in order to trigger warning.
    380-    HeadersSyncState hss{/*id=*/0,
    381-                         Params().GetConsensus(),
    382-                         HeadersSyncParams{
    383-                             .commitment_period = COMMITMENT_PERIOD,
    384-                             .redownload_buffer_size = first_chain.size(),
    385-                         },
    386-                         chain_start,
    387-                         /*minimum_required_work=*/CHAIN_WORK};
    388-    (void)hss.ProcessNextHeaders(first_chain, true);
    389-    BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::REDOWNLOAD);
    390-
    391-    {
    392-        ASSERT_DEBUG_LOG(BUFFER_SIZE_WARNING);
    393-        auto result{hss.ProcessNextHeaders(first_chain, true)};
    394-        BOOST_REQUIRE_EQUAL(hss.GetState(), HeadersSyncState::State::FINAL);
    395-        BOOST_CHECK(result.success);
    396-    }
    397+    const auto& genesis{Params().GenesisBlock()};
    398+    const uint256 genesis_hash{genesis.GetHash()};
    399+
    400+    // Generate headers for two different chains (using differing merkle roots
    401+    // to ensure the headers are different).
    402+    std::vector<CBlockHeader> chain1{};
    403+    GenerateHeaders(chain1, TARGET_BLOCKS - 1, genesis_hash, genesis.nVersion, genesis.nTime,
    404+                    /*merkle_root=*/uint256::ZERO, genesis.nBits);
    405+    std::vector<CBlockHeader> chain2{};
    406+    GenerateHeaders(chain2, TARGET_BLOCKS - 2, genesis_hash, genesis.nVersion, genesis.nTime,
    407+                    /*merkle_root=*/uint256::ONE, genesis.nBits);
    408+
    409+    const CBlockIndex* chain_start{WITH_LOCK(::cs_main, return m_node.chainman->m_blockman.LookupBlockIndex(genesis_hash))};
    410+    SneakyRedownload(chain_start, chain1, chain2);
    411+    HappyPath(chain_start, chain1);
    412+    TooLittleWork(chain_start, chain2);
    413+    TooBigBuffer(chain_start, chain1);
    414 }
    415 
    416 BOOST_AUTO_TEST_SUITE_END()
    417diff --git a/src/test/util/logging.h b/src/test/util/logging.h
    418index 73ac23825f..622ff58938 100644
    419--- a/src/test/util/logging.h
    420+++ b/src/test/util/logging.h
    421@@ -36,6 +36,16 @@ public:
    422     ~DebugLogHelper() { check_found(); }
    423 };
    424 
    425+// Test fails if the pattern *DOES NOT* show up.
    426 #define ASSERT_DEBUG_LOG(message) DebugLogHelper UNIQUE_NAME(debugloghelper)(message)
    427+// Test fails if the pattern *DOES* show up.
    428+#define ASSERT_NO_DEBUG_LOG(message)                                              \
    429+    DebugLogHelper UNIQUE_NAME(nologhelper){                                      \
    430+        message,                                                                  \
    431+        [](const std::string* line) {                                             \
    432+            if (line) throw std::runtime_error("unexpected log line: " + *line);  \
    433+            return false; /* suppress default 'not found' failure */              \
    434+        }                                                                         \
    435+    }
    436 
    437 #endif // BITCOIN_TEST_UTIL_LOGGING_H
    
  56. refactor(headerssync): Process spans of headers 4c45b5d349
  57. refactor(test): Break up headers_sync_chainwork_tests
    Also adds missing comment for part 4.
    
    (unique_ptr's and ProcessingResults will be cleaned up in next commit).
    da2a414b7c
  58. refactor(test): Use function-scope instances of HeadersSyncState a636070d28
  59. hodlinator force-pushed on Jun 23, 2025
  60. hodlinator force-pushed on Jun 23, 2025
  61. DrahtBot added the label CI failed on Jun 23, 2025
  62. hodlinator commented at 5:49 pm on June 23, 2025: contributor

    Thanks for the extensive review @l0rinc! Implemented many of the suggestions and answered the other points.

    Removed the commit breaking out scopes in favor of the suggested CHECK_RESULT macro (https://github.com/bitcoin/bitcoin/pull/32579#discussion_r2158508969). Broke out the comment improvements into their own commit ahead of that since it gives a “skeleton” for the next commit’s diff to hang off of.

  63. hodlinator force-pushed on Jun 23, 2025
  64. DrahtBot removed the label CI failed on Jun 23, 2025
  65. in src/test/headers_sync_chainwork_tests.cpp:36 in 8c96be3306 outdated
    31+        BOOST_CHECK_EQUAL(result.pow_validated_headers.size(), exp_headers_size);                          \
    32+        std::optional<uint256> locator_hash_opt{exp_locator_hash};                                         \
    33+        if (locator_hash_opt == std::nullopt) {                                                            \
    34+            BOOST_CHECK_EQUAL(exp_state, State::FINAL);                                                    \
    35+        } else {                                                                                           \
    36+            BOOST_REQUIRE_GT(hss.NextHeadersRequestLocator().vHave.size(), 0);                             \
    


    l0rinc commented at 2:33 pm on June 24, 2025:
    would’t an .at(0) suffice?

    hodlinator commented at 7:56 pm on June 25, 2025:
    Slightly more confusing error, but like that it’s more terse. Taken.

    l0rinc commented at 2:28 pm on June 27, 2025:
    I think it applies to BOOST_REQUIRE_GT(result.pow_validated_headers.size(), 0); as well

    hodlinator commented at 7:36 am on June 28, 2025:
    True, sole change in latest push.
  66. in src/test/headers_sync_chainwork_tests.cpp:171 in 8c96be3306 outdated
    176-    // Nothing left for the sync logic to do:
    177-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::FINAL);
    178+    CHECK_RESULT(hss.ProcessNextHeaders(first_chain, /*full_headers_message=*/true),
    179+        hss, /*exp_state=*/State::REDOWNLOAD,
    180+        /*exp_success*/true, /*exp_request_more=*/true,
    181+        // The locator should reset to genesis.
    


    l0rinc commented at 2:35 pm on June 24, 2025:
    this comment is obvious from the code now: /*exp_locator_hash=*/Params().GenesisBlock().GetHash() or /*exp_locator_hash=*/genesis_hash later in 65a96b507c11b3f34efa996919892fa1a2fcf49c
  67. in src/kernel/chainparams.h:71 in 1032f52791 outdated
    66+ */
    67+struct HeadersSyncParams {
    68+    //! Distance in blocks between header commitments.
    69+    size_t commitment_period;
    70+    //! Only feed headers to validation once this many headers on top have been
    71+    //! received and validated against commitments.
    


    l0rinc commented at 2:39 pm on June 24, 2025:
    I’m not sure what “this many headers on top have been received”

    hodlinator commented at 8:11 pm on June 25, 2025:

    Agree. New attempt:

    0//! Only start outputting headers once this many headers have been received
    1//! and validated against commitments.
    
  68. in src/test/headers_sync_chainwork_tests.cpp:189 in 65a96b507c outdated
    187+    // headers shouldn't be returned yet:
    188+    CHECK_RESULT(hss.ProcessNextHeaders({first_chain.begin(), REDOWNLOAD_BUFFER_SIZE}, /*full_headers_message=*/true),
    189+        // Not done:
    190+        hss, /*exp_state=*/State::REDOWNLOAD,
    191+        /*exp_success*/true, /*exp_request_more=*/true,
    192+        // The locator should reset to genesis.
    


    l0rinc commented at 2:42 pm on June 24, 2025:
    is this comment accurate?

    hodlinator commented at 8:01 pm on June 25, 2025:
    No, rebase mishap. Thanks for catching!
  69. in src/test/headers_sync_chainwork_tests.cpp:115 in 14e933ab6a outdated
    111@@ -109,6 +112,9 @@ static void HappyPath(const CBlockIndex*, const std::vector<CBlockHeader>&);
    112 // 4. Repeat the second set of headers in both phases to demonstrate behavior
    113 //    when the chain a peer provides has too little work.
    114 static void TooLittleWork(const CBlockIndex*, const std::vector<CBlockHeader>&);
    115+// 5. Sets the redownload buffer size to be large enough large enough that we
    


    l0rinc commented at 2:47 pm on June 24, 2025:
    0// 5. Sets the redownload buffer size to be large enough that we
    
  70. in src/test/headers_sync_chainwork_tests.cpp:245 in 14e933ab6a outdated
    243@@ -234,4 +244,28 @@ static void TooLittleWork(const CBlockIndex* chain_start,
    244         /*exp_locator_hash=*/std::nullopt);
    245 }
    246 
    247+static void TooBigBuffer(const CBlockIndex* chain_start,
    248+        const std::vector<CBlockHeader>& first_chain)
    249+{
    250+    // Intentionally too big redownload buffer in order to trigger warning.
    


    l0rinc commented at 2:49 pm on June 24, 2025:
    nice, this is exactly how comments should be used to augment what the code cannot easily tell <3
  71. in src/test/headers_sync_chainwork_tests.cpp:261 in 14e933ab6a outdated
    256+
    257+    CHECK_RESULT(hss.ProcessNextHeaders(first_chain, true),
    258+        hss, /*exp_state=*/State::REDOWNLOAD,
    259+        /*exp_success*/true, /*exp_request_more=*/true,
    260+        /*exp_headers_size=*/0, /*exp_pow_validated_prev=*/std::nullopt,
    261+        /*exp_locator_hash=*/Params().GenesisBlock().GetHash());
    


    l0rinc commented at 2:51 pm on June 24, 2025:
    might make sense to extract genesis hash here as well like we did in HappyPath
  72. l0rinc approved
  73. l0rinc commented at 2:53 pm on June 24, 2025: contributor
    I find the current layout of tiny, focused commits a lot easier to follow, thanks! Left a few comments, after that I will test it again locally and I can ACK.
  74. doc(test): Improve comments aa098640ce
  75. hodlinator force-pushed on Jun 25, 2025
  76. test(headerssync): headers_sync_chainwork test improvements
    Introduces CHECK_RESULT for consistently validating ProcessingResult.
    * Verifies HeadersSyncState::State directly after ProcessNextHeaders().
    * Uses BOOST_REQUIRE_EQUAL for HeadersSyncState::State - Nicer failure output and prevents continuing test in nonsensical state.
    * Always checks Locator and result.pow_validated_headers.
    79bdc6a785
  77. headerssync: Make HeadersSyncState more flexible and move constants
    Move calculated constants from the top of src/headerssync.cpp into src/kernel/chainparams.cpp.
    
    Instead of being hardcoded to mainnet parameters, HeadersSyncState can now vary depending on chain or test.
    
    In the chainwork test suite: for REDOWNLOAD_BUFFER_SIZE - subtract MAX_HEADERS_RESULTS (2000) and a smaller arbitrary value (123) from the test constant in order to allow for broader testing.
    ced2ddb37c
  78. test(headerssync): Test returning of pow_validated_headers behavior
    Enabled by increased flexibility of HeadersSyncState in parent commit.
    7e20d5ccd9
  79. headerssync: Add warning for redownload buffer size being too big
    Also test that the warning is logged when we expect it, and not logged when we don't.
    c3bc47cfb6
  80. in src/arith_uint256.h:247 in fab6f59e2e outdated
    246-    arith_uint256() = default;
    247-    arith_uint256(const base_uint<256>& b) : base_uint<256>(b) {}
    248-    arith_uint256(uint64_t b) : base_uint<256>(b) {}
    249+    constexpr arith_uint256() = default;
    250+    constexpr arith_uint256(const base_uint& b) : base_uint(b) {}
    251+    constexpr arith_uint256(uint64_t b) : base_uint(b) {}
    


    danielabrozzoni commented at 3:09 pm on June 27, 2025:
    Just to confirm that I understand this correctly, are you removing the <256> because it’s implicit?

    l0rinc commented at 4:35 pm on July 2, 2025:
    Yes, they’re redundant, can be deduced by the compiler
  81. hodlinator force-pushed on Jun 28, 2025
  82. in src/test/headers_sync_chainwork_tests.cpp:143 in 79bdc6a785 outdated
    145     // This chain should look valid, and we should have met the proof-of-work
    146     // requirement during PRESYNC and transitioned to REDOWNLOAD.
    147-    BOOST_CHECK(result.success);
    148-    BOOST_CHECK(result.request_more);
    149-    BOOST_CHECK(hss.GetState() == HeadersSyncState::State::REDOWNLOAD);
    150+    CHECK_RESULT(hss.ProcessNextHeaders(std::span{first_chain}.last(first_chain.size() - 1), true),
    


    danielabrozzoni commented at 2:37 pm on July 2, 2025:
    nit in 79bdc6a785bca96dababc698336c04b8625b7d1c: in some CHECK_RESULT you are putting /*full_headers_message=*/, in some others you’re not, like this one, the one below, and the ones in TooLittleWork
  83. in src/kernel/chainparams.h:70 in ced2ddb37c outdated
    65+ * Used to configure Headers Sync memory usage.
    66+ */
    67+struct HeadersSyncParams {
    68+    //! Distance in blocks between header commitments.
    69+    size_t commitment_period;
    70+    //! Only start outputting headers once this many headers have been received
    


    danielabrozzoni commented at 3:29 pm on July 2, 2025:

    nit in ced2ddb37c957e53d4a8aa40d3512ea3786511b4: the previous comment (in headerssync.cpp) said: “Only feed headers to validation once this many headers..”

    I think that phrasing was a bit clearer than “outputting”

  84. in src/test/headers_sync_chainwork_tests.cpp:38 in 79bdc6a785 outdated
    33+        if (locator_hash_opt == std::nullopt) {                                                          \
    34+            BOOST_CHECK_EQUAL(exp_state, State::FINAL);                                                  \
    35+        } else {                                                                                         \
    36+            BOOST_CHECK_EQUAL(hss.NextHeadersRequestLocator().vHave.at(0), locator_hash_opt);            \
    37+        }                                                                                                \
    38+        std::optional<uint256> pow_validated_prev_opt{exp_pow_validated_prev};                           \
    


    danielabrozzoni commented at 4:06 pm on July 2, 2025:
    micro nit in 79bdc6a785bca96dababc698336c04b8625b7d1c: you could check exp_pow_validated_prev first and exp_locator_hash second, to maintain the same order as the macro parameters
  85. danielabrozzoni commented at 4:28 pm on July 2, 2025: contributor

    Concept ACK, I’ve left a few questions/nits

    I’ve only done a partial review so far, I still need to go through the last two commits and part of ced2ddb37c957e53d4a8aa40d3512ea3786511b4


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-07-04 00:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me