Replace cluster linearization algorithm with SFL #32545

pull sipa wants to merge 14 commits into bitcoin:master from sipa:202505_sfl changing 4 files +1092 −910
  1. sipa commented at 11:24 pm on May 17, 2025: member

    Part of cluster mempool: #30289

    This replaces the cluster linearization algorithm introduced in #30126 and #30286 (a combination of LIMO with candidate-set search), with a completely different algorithm: spanning-forest linearization, which appears to have much better performance for hard clusters. See this post for a comparison between various linearization algorithms, and this post for benchmarks comparing them. Replaying historical mempool data on it shows that it can effectively linearize every observed cluster up to 64 transactions optimally within tens of microseconds, though pathological examples can be created which take longer.

    The algorithm is effectively a very specialized version of the simplex algorithm to the problem of finding high-feerate topological subsets of clusters, but modified to find all consecutive such subsets concurrently rather than just the first one. See the post above for how it is related.

    It represents the cluster as partitioned into a set of chunks, each with a spanning tree of its internal dependencies connecting the transactions. Randomized improvements are made by selecting dependencies to add and remove to these spanning trees, merging and splitting chunks, until no more improvements are possible. Like simplex, it does not necessarily make progress in every step, and thus has no upper bound on its runtime, but randomization makes long runtimes very unlikely, and additionally makes it hard to adversarially construct clusters in which the algorithm reliably makes bad choices.

  2. DrahtBot commented at 11:24 pm on May 17, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32545.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK jonatack

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #32263 (cluster mempool: add TxGraph work controls by sipa)
    • #30605 (Cluster linearization: separate tests from tests-of-tests by sipa)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • recursee -> recurse [Typo]
    • child’s -> parent’s [Incorrect reference in documentation]
    • stategy -> strategy [Typo]
      • good (in the feerate diagram sense) as old_linearization. [Orphaned documentation fragment, missing context for comprehension]
        • A boolean indicating whether the result is guaranteed to be [Orphaned documentation fragment, missing context for comprehension]
      • optimal. [Orphaned documentation fragment, missing context for comprehension]

    drahtbot_id_0520

  3. sipa force-pushed on May 17, 2025
  4. DrahtBot added the label CI failed on May 17, 2025
  5. DrahtBot commented at 11:40 pm on May 17, 2025: contributor

    🚧 At least one of the CI tasks failed. Task ARM, unit tests, no functional tests: https://github.com/bitcoin/bitcoin/runs/42417371062 LLM reason (✨ experimental): The CI failure is due to a build error during the compilation of txgraph.cpp.o.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  6. sipa force-pushed on May 18, 2025
  7. sipa force-pushed on May 18, 2025
  8. DrahtBot removed the label CI failed on May 18, 2025
  9. sipa force-pushed on May 18, 2025
  10. sipa force-pushed on May 19, 2025
  11. DrahtBot added the label CI failed on May 19, 2025
  12. DrahtBot commented at 3:39 am on May 19, 2025: contributor

    🚧 At least one of the CI tasks failed. Task ARM, unit tests, no functional tests: https://github.com/bitcoin/bitcoin/runs/42447412610 LLM reason (✨ experimental): The CI failure is due to a failed CTest test: cluster_linearize_tests.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  13. sipa force-pushed on May 20, 2025
  14. sipa force-pushed on May 20, 2025
  15. DrahtBot removed the label CI failed on May 20, 2025
  16. sipa added the label Mempool on May 20, 2025
  17. in src/cluster_linearize.h:558 in 23072f2b0e outdated
    554@@ -539,492 +555,651 @@ class LinearizationChunking
    555     }
    556 };
    557 
    558-/** Class encapsulating the state needed to find the best remaining ancestor set.
    559+/** Class to represent the internal state of the spanning-forest linearization algorithm.
    


    jonatack commented at 4:09 pm on May 22, 2025:
    Appreciate the excellent doxygen documentation here.
  18. jonatack commented at 4:09 pm on May 22, 2025: member
    Concept ACK
  19. sipa force-pushed on May 23, 2025
  20. sipa force-pushed on May 23, 2025
  21. clusterlin: add known-correct optimal linearization tests (tests) 93b3d7b83f
  22. clusterlin: replace benchmarks with SFL-hard ones (bench) c168809fe1
  23. sipa force-pushed on May 24, 2025
  24. clusterlin: replace cluster linearization with SFL implementation (feature)
    This replaces the existing LIMO linearization algorithm (which internally uses
    ancestor set finding and candidate set finding) with the much more performant
    spanning-forest linearization algorithm.
    
    See https://delvingbitcoin.org/t/spanning-forest-cluster-linearization/1419
    b607d29c5d
  25. clusterlin: remove unused {Ancestor,Search}CandidateFinder code (cleanup)
    This removes the candidate set finding classes, as well as related tests
    and benchmarks for them.
    53f2855a4c
  26. clusterlin: keep track of the active parents of each transaction (optimization)
    This avoids the need for a loop over all parents of a transaction while walking
    a chunk, and removes the need to store the set of parent dependencies explicitly.
    25f5bad46d
  27. clusterlin: split chunks based on maximum gain (optimization)
    This introduces the notion of gain to the SFL algorithm. Given a chunk c,
    an active dependency d in it, and the chunks (t, b) that c would split into
    if d were deactivated, the gain is defined as either (they are equivalent):
    
      (feerate(t) - feerate(b)) * size(t) * size(b)
      fee(t) * size(b) - fee(b) * size(t)
    
    It happens to also be equal to these:
    
      (feerate(t) - feerate(c)) * size(t) * size(c)
      fee(t) * size(c) - fee(c) * size(t)
    
    Its relevance is that this metric is proportional to a lower bound on the area
    under the fee-size diagram which would be gained IF a deactivation of d does not
    result in a self-merge of t and b again.
    
    This commit adds logic to find, within each chunk, the dependency with the highest
    gain. In benchmarks, this appears to be a very good heuristic for deciding which
    splits are worth making.
    4045f9dea4
  28. clusterlin: use pre-allocated array for dependencies instead of vector (optimization)
    This reduces the number of allocations required inside the SFL algorithm, and works
    because the number of dependencies per transaction is at most n-1.
    
    To minimize the memory usage from this pre-allocation (which might impact memory locality),
    change the data type of DepIdx from uint32_t to uint8_t or uint16_t when possible.
    dc682202d9
  29. clusterlin: keep child dependencies split into active and inactive (optimization)
    Within the per-transaction child dependency list, keep the active ones before all
    inactive ones. This improves the complexity over iterating over active dependencies
    from O(m) to O(n), as at most n-1 dependencies can be active within any given chunk
    at any given time.
    39104c8c06
  30. clusterlin: keep FIFO queue of improvable chunks (optimization)
    This distributes the work over the various chunks fairly, and simultaneously
    avoids retrying chunks over and over again which are already known to be optimal.
    9f1c93c79e
  31. clusterlin: randomize merges/splits in SFL (feature) dae70d6109
  32. clusterlin: use random split in SFL on every 3rd attempt (feature)
    Out of an abundance of caution that adversarially-constructed clusters might
    reliably result in bad chunk split decisions with the maximum-gain strategy,
    make every third consecutive attempt to split the same chunk use a random
    strategy instead.
    6e991ade64
  33. clusterlin: make dependency being active implicit (optimization)
    We do not need to actually keep track of whether a dependency is active
    or not; it is implied by whether or not it appears within the active
    prefix of its parent's child_deps, and its child's parent_deps.
    
    Just remove setting and checking it.
    b5a1bd7d43
  34. clusterlin: add more accurate cost modeling (feature)
    This adds a rough estimate of algorithm runtime, so it can be interrupted if no
    solution is found in time. Due to inherent differences between platforms, this
    will not be extremely accurate, but it is preferable over directly measuring
    time for consistency.
    27db297817
  35. sipa force-pushed on May 25, 2025
  36. clusterlin: minimize chunks (feature)
    After the normal optimization process finishes, and finds an optimal
    spanning forest, run a second process (while computation budget remains)
    to split chunks into minimal equal-feerate chunks.
    ba7464a864
  37. sipa force-pushed on May 25, 2025
  38. DrahtBot added the label CI failed on May 25, 2025
  39. DrahtBot commented at 4:44 pm on May 25, 2025: contributor

    🚧 At least one of the CI tasks failed. Task CentOS, depends, gui: https://github.com/bitcoin/bitcoin/runs/42858330826 LLM reason (✨ experimental): The CI failure is due to assertion failures within the cluster_linearize_tests and bench_sanity_check tests.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  40. DrahtBot removed the label CI failed on May 25, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-05-25 18:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me