cluster mempool: control/optimize TxGraph memory usage #33157

pull sipa wants to merge 6 commits into bitcoin:master from sipa:202508_txgraph_memusage changing 5 files +239 −123
  1. sipa commented at 2:21 pm on August 8, 2025: member

    Part of #30289.

    This adds a few optimizations to reduce TxGraph’s memory usage, and makes sure that dynamic memory it uses doesn’t linger after shrinking clusters. Finally, it exposes a function GetMainMemoryUsage() to compute TxGraph’s approximate memory usage.

    On my 64-bit system, it needs 72 bytes per transaction, 120 bytes per cluster, and 64 bytes per chunk. The code using it also needs a TxGraph::Ref object per transaction, at 16 bytes per transaction. Overall, this means up to 272 bytes per transaction. Experimenting with this integrated into #28676 shows a total per-transaction overhead of ~720 bytes per mempool transaction, while master has around ~548 bytes per transaction. Or expressed as a ratio of memory usage divided by mempool vsize, 5.8 in master and 6.4 in cluster mempool.

    When designing TxGraph, the intent was to make the Cluster type a virtual class, with different instances for different cluster sizes (in particular, for singletons) that could optimize memory usage in a more tailored way. I have a very immature idea about getting rid of DepGraph instances inside Cluster after #32545, which would cut the increase in overhead down, so I’m not including that here.

    Another memory usage optimization that may worthwhile, independently of the idea above, is to special-case singleton clusters (given the large fraction of transactions those are) and just store those as a list of transactions rather than having individual Cluster objects for them. I’m estimating that could save ~96 bytes per singleton-cluster transaction, but I’m leaving that for a potential future improvement too.

  2. DrahtBot commented at 2:21 pm on August 8, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33157.

    Reviews

    See the guideline for information on the review process. A summary of reviews will appear here.

  3. txgraph: Make level of Cluster implicit (optimization)
    This reduces per-Cluster memory usage by making Clusters not aware of their
    own level. Instead, track it either in calling code, or infer it based on
    the transactions in them.
    15277c05a1
  4. txgraph: avoid holes in DepGraph positions (mem optimization) c638faec3a
  5. depgraph: add memory usage control (feature) 614eefb7df
  6. txgraph: keep data structures compact (mem optimization) 65345a3c15
  7. txgraph: keep track of Cluster memory usage (preparation) f0119336a6
  8. txgraph: expose memory usage estimate function (feature) 754775357b
  9. sipa force-pushed on Aug 8, 2025
  10. DrahtBot added the label CI failed on Aug 8, 2025
  11. DrahtBot commented at 2:31 pm on August 8, 2025: contributor

    🚧 At least one of the CI tasks failed. Task macOS-cross, gui, no tests: https://github.com/bitcoin/bitcoin/runs/47684523262 LLM reason (✨ experimental): The CI failure is caused by a compilation error due to ambiguous call to ‘DynamicUsage’ in txgraph.cpp.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  12. DrahtBot removed the label CI failed on Aug 8, 2025


sipa DrahtBot


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-08-13 06:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me