[DO NOT MERGE] Erlay: bandwidth-efficient transaction relay protocol (Full implementation) #35591

pull sr-gi wants to merge 31 commits into bitcoin:master from sr-gi:2026-05-erlay-recon-only-full-impl changing 26 files +3474 −325
  1. sr-gi commented at 12:40 PM on June 23, 2026: member

    Erlay Project Tracking: #30249 Conceptual Discussion: #34542


    This is a full implementation of Erlay. Its purpose is to check the integrity and correctness of the implementation against changes/additions that may originate from the review process and/or rebases on top of newer functionality.

    This is not to be merged. Functionality will be spread across multiple smaller PRs to ease the review process.


    Approach

    This approach uses Erlay as a fallback mechanism for transaction propagation. Instead of mixing fanout and reconciliation into a single connection type, the current approach leaves the existing connections as they are, and adds additional low-bandwidth connections to be used in case the node is being eclipsed. This connections should have minimal cost under normal circumstances, and only undergo real traffic in case other 8 full-outbound connections are being captured.

    outbound-full-reconciliation connections

    For now, we are adding 4 additional reconciliation-only connections to the node while we test it's impact on real node running the approach. Further analysis may be needed to pick a meaningful value for this. The number of inbound connections should also be scaled based on how many connections we are adding.

  2. DrahtBot commented at 12:40 PM on June 23, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35591.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process. A summary of reviews will appear here.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35587 (Remove boost as a unit test runner by rustaceanrob)
    • #35561 (net: move some CNodeState fields to Peer by Crypt-iQ)
    • #35522 (refactor: Extract per-message helpers from SendMessages() (move-only) by pablomartin4btc)
    • #35511 (RFC: consensus: Make CAmount a class by hodlinator)
    • #35502 (refactor: extract per-message helpers from ProcessMessage (move-only) by w0xlt)
    • #34824 (net: encapsulate TxRelay state and replace recursive mutexes by w0xlt)
    • #34565 (refactor: extract BlockDownloadManager from PeerManagerImpl by w0xlt)
    • #28690 (build: Introduce internal kernel library by sedited)
    • #28463 (p2p: Increase inbound capacity for block-relay only connections by mzumsande)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • src/node/txreconciliation_impl.cpp: // Schedules the a reconciliation request -> // Schedules a reconciliation request [“the a” is a typo]
    • test/functional/p2p_reqtxrcncl.py: self.log.info('REQTXRCNCL sent to an peer 2') -> ...to a peer 2 [“an peer” is misspelled]
    • test/functional/p2p_txrecon_responder.py: # Most of them will be pick for reconciliation. -> ...will be picked... [“pick” should be “picked”]
    • test/functional/p2p_txrecon_responder.py: # ...the phase moves to EXT_REQUESTED, an a subsequent requests are not allowed -> ...and subsequent requests are not allowed [“an a” is a typo]

    Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

    • self.generate_txs(self.wallet, 0, 0, 2) in test/functional/p2p_txrecon_initiator.py
    • self.generate_txs(self.wallet, 0, 10, 0) in test/functional/p2p_txrecon_initiator.py
    • self.test_reconciliation_initiator_no_extension(20, 15, 0) in test/functional/p2p_txrecon_initiator.py
    • self.generate_txs(self.wallet, 0, 1, 0) in test/functional/p2p_txrecon_responder.py

    <sup>2026-06-24 16:06:16</sup>

  3. refactor: redesigns txreconciliation file split and namespace
    Splits the txreconciliation logic in three files instead of two, allowing the
    TxreconciliationState to be properly tested, instead of being internal to
    txreconciliation.cpp.
    
    Also includes everything in the node namespace, instead of being part
    of an anonymous one.
    accbc3a133
  4. refactor: remove legacy comments
    These comments became irrelevant in one of the previous code changes.
    They simply don't make sense anymore.
    be09a3faa5
  5. sr-gi force-pushed on Jun 23, 2026
  6. sr-gi commented at 1:13 PM on June 23, 2026: member

    Rebased on master.

    Opening this so it can be tested against CI and to make sure nothing obvious is missing. Next step will be testing with real nodes (most likely in Warnet).

    Things to consider/add:

    • This is currently using sendtxrcncl as a way, for outbound nodes, to signal they want to establish a full-reconciliation connection, plus adding a limit on the number of full-reconcilition inbounds that will be accepted by a node. This is not the originally intended way of using sendtxrcncl. It may be worth considering an approach based on BIP-434.
    • The extension phase was designed so peers with an ongoing reconciliation that had underpredicted the sketch capacity could still reconcile without having to directly fallback to fanout. Given reconciliation is now used as fallback, and the extra bandwidth may not be as problematic, it may be worth considering overshooting the initial sketch capacity and getting rid of the extension phase.
    • Fuzz tests are missing
  7. DrahtBot added the label CI failed on Jun 23, 2026
  8. sr-gi force-pushed on Jun 23, 2026
  9. sr-gi force-pushed on Jun 23, 2026
  10. refactor: Defines generic error to be used in several reconciliation methods 33bedee44b
  11. refactor: add full stop in existing txreconciliation LogDebug lines 64c79875a7
  12. p2p: Allows inbound reconciliation connections up to a limit
    Set the current limit to 32.
    d7b3c606b0
  13. net, gui, test: adds new connection type (OUTBOUND_FULL_RECONCILIATION)
    Adds a new connection type that will be used for reconciliation only.
    
    Defines the default max number of this type of connections to 4.
    46e0a20008
  14. p2p: send SENDTXRCNCL messages only over OUTBOUND_FULL_RECONCILIATION connection 2e7e258fff
  15. DrahtBot removed the label CI failed on Jun 23, 2026
  16. p2p: Functions to add/remove wtxids to tx reconciliation sets
    They will be used later on.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    0a9d1583e1
  17. sr-gi force-pushed on Jun 23, 2026
  18. sr-gi commented at 11:04 PM on June 23, 2026: member

    Fixed some typos and addressed some of the linter suggestions

    6554707...e63db1a

  19. sr-gi force-pushed on Jun 24, 2026
  20. DrahtBot added the label CI failed on Jun 24, 2026
  21. sr-gi force-pushed on Jun 24, 2026
  22. sr-gi commented at 2:41 PM on June 24, 2026: member

    Fixed several bugs and added a new test for receiving reconciliation messages when reconciliation is not enabled.

    e63db1a...35bcd04

  23. p2p: Add transactions to reconciliation sets
    Transactions are added to the reconciliation sets of reconciling peers, and processed normally for fanout peers.
    26e24a6b6f
  24. p2p: Add helper to compute reconciliation tx short ids and a cache of short ids to wtxids 92362ea119
  25. p2p: Deal with shortid collisions for reconciliation sets
    If a transaction to be added to a peer's recon set has a shot id collisions (a previously
    added wtxid maps to the same short id), both transaction should be fanout, given
    our peer may have added the opposite transaction to our recon set, and these two
    transaction won't be reconciled.
    cc75da584a
  26. p2p: Add peers to reconciliation queue on negotiation
    When we're finalizing negotiation, we should add the peers
    for which we will initiate reconciliations to the queue.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    3c79bd552c
  27. p2p: Track reconciliation requests schedule
    We initiate reconciliation by looking at the queue periodically
    with equal intervals between peers to achieve efficiency.
    
    This will be later used to see whether it's time to initiate.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    3cd454ee09
  28. p2p: Initiate reconciliation round
    When the time comes for the peer, we send a
    reconciliation request with the parameters which
    will help the peer to construct a (hopefully) sufficient
    reconciliation sketch for us. We will then use that
    sketch to find missing transactions.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    82ab9d38fe
  29. test: Functional test for reqtxrcncl
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    c6e7cae54d
  30. p2p: Handle reconciliation request
    Store the parameters the peer sent us inside th reconciliation request.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    8ed766781f
  31. p2p: Add helper to compute sketches for tx reconciliation
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    0a459b3b96
  32. p2p: Respond to a reconciliation request
    When the time comes, we should send a sketch of our
    local reconciliation set to the reconciliation initiator.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    1e97945af6
  33. p2p: Add a function to identify local/remote missing txs
    When the sketches from both sides are combined successfully,
    the diff is produced. Then this diff can (together with the local txs)
    be used to identified which transactions are missing locally and remotely.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    d36c828992
  34. Use txid/uint256 in CompareInvMempoolOrder
    This will help to reuse the code later on in the function to announce transactions.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    688e1011af
  35. p2p: Handle reconciliation sketch and successful decoding
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    c21fbe6844
  36. p2p: Request extension if decoding failed
    If after decoding a reconciliation sketch it turned out
    to be insufficient to find set difference, request extension.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    cfb997098b
  37. p2p: Be ready to receive sketch extension
    Store the initial sketches so that we are able to process
    extension sketch while avoiding transmitting the same data.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    0b9de161fe
  38. p2p: Prepare for sketch extension request
    To be ready to respond to a sketch extension request
    from our peer, we should store a snapshot of our state
    and capacity of the initial sketch, so that we compute
    extension of the same size and over the exact same
    transactions.
    
    Transactions arriving during this reconciliation will
    be instead stored in the regular set.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    53930c2076
  39. p2p: Keep track of announcements during txrcncl extension
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    a82db0abb3
  40. p2p: Handle reconciliation extension request
    If peer failed to reconcile based on our initial response sketch,
    they will ask us for a sketch extension. Store this request to respond later.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    dd4c7d6e29
  41. p2p: Respond to sketch extension request
    Sending an extension may allow the peer to reconcile
    transactions, because now the full sketch has twice
    as much capacity.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    25b173bd80
  42. p2p: Handle sketch extension
    If a peer sent us an extension sketch, we should
    reconstruct a full sketch from it with the snapshot
    we stored initially, and attempt to decode the difference.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    8c133ac067
  43. p2p: Add a finalize incoming reconciliation function
    This currently unused function is supposed to be used once
    a reconciliation round is done. It cleans the state corresponding
    to the passed reconciliation.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    896aaf07c8
  44. p2p: Handle reconciliation finalization message
    Once a peer tells us reconciliation is done, we should behave as follows:
    - if it was successful, just respond them with the transactions they asked
      by short ID.
    - if it was a full failure, respond with all local transactions from the reconciliation
      set snapshot
    - if it was a partial failure (only low or high part was failed after a bisection),
      respond with all transactions which were asked for by short id,
      and announce local txs which belong to the failed chunk.
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    fa7b5404fe
  45. p2p, test: Add tx reconciliation functional tests
    We may still need to add more tests, specially around extensions (if we keep them)
    
    Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
    280d603597
  46. sr-gi force-pushed on Jun 24, 2026
  47. sr-gi commented at 4:06 PM on June 24, 2026: member

    Addressed some issues pointed out by corecheck.

    35bcd04...280d603


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-06-24 17:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me