Add Gutter Guard Selector #28977

pull murchandamus wants to merge 2 commits into bitcoin:master from murchandamus:2023-11-gutter-guard-selector changing 4 files +177 −1
  1. murchandamus commented at 9:26 pm on November 30, 2023: contributor

    Please refer to the topic on Delving Bitcoin discussing Gutter Guard/Coingrinder simulation results.

    Gutter Guard Selector bounds the worst case on coin selection outcomes by ensuring that there is at least one candidate input set that exceeds the minimal necessary input count by at most three.

    Does a random selection limited to three more output groups than largest-first selection would select to fund the transaction. If the limit is exceeded during selection, the output group with the lowest effective value is discarded.

    Motivations

    • At high feerates, using unnecessary inputs can significantly increase the fees
    • Users are upset when fees are relatively large compared to the amount sent
    • Minimizing the input set is an NP-hard problem
    • Always minimizing the weight of the input set can lead to fragmentation of the wallet’s UTXO pool

    Approach

    Bitcoin Core uses multiple coin selection algorithms to generate candidate input sets. It then picks the least wasteful input set per the weight metric, but none of the deployed algorithms explicitly look for low-weight input sets. Gutter Guard Selector determines a likely minimum count c of output groups that are necessary to fund the transaction via largest-first selection. It then randomly selects output groups into a lowest-effective-value heap. If the heap exceeds c + 3 output groups, the output group with the lowest-effective value is discarded. It then checks whether the selection can fund the transaction. This approach will always succeed given the effective value available exceeds the selection target.

    Trade-offs

    Largest-first selection (LF) (and to a lesser degree an approach that minimizes the weight of an input set) might grind a wallet’s UTXO pool to dust if overused. Incoming payments grow the wallet’s UTXO pool, while transactions with a single input and a change output result in the same UTXO count as before. LF will additionally decrease range of amounts in the wallet by reducing the amount of the supreme element.

    Branch and Bound looks for a changeless input set. If there are multiple solutions, it prefers the one with the lowest waste, but there might not be a solution, or even the least wasteful one could be composed of many inputs. Knapsack minimizes the distance between target amount and selected amount which is independent from the input set’s weight. Single Random Draw just does a single random drawing from the UTXO pool—if there are a ton of UTXOs in the wallet, this will likely result in a large input set. While we pick the least wasteful among the presented input set candidates, there is no guarantee that any of the three deployed algorithms will produce a thrifty input set.

    In contrast, Gutter Guard Selector is expected to reduce the UTXO pool of a fragmented wallet by at least three UTXOs. When many UTXOs exist that are larger than the target, a solution with fewer inputs and less weight may be found. When transactions with low feerates are being built, the waste metric prefers heavier input sets which may be proposed by other coin selection algorithms anyway. At high feerates, Gutter Guard Selector ensures that there is an input set candidate that isn’t absurdly wasteful and therefore delimits the worst case.

  2. DrahtBot commented at 9:26 pm on November 30, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK kashifs
    Concept ACK achow101

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #28985 (Avoid changeless input sets when SFFO is active by murchandamus)
    • #27877 (wallet: Add CoinGrinder coin selection algorithm by murchandamus)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. murchandamus force-pushed on Nov 30, 2023
  4. DrahtBot added the label CI failed on Nov 30, 2023
  5. murchandamus marked this as a draft on Nov 30, 2023
  6. murchandamus force-pushed on Nov 30, 2023
  7. DrahtBot removed the label CI failed on Nov 30, 2023
  8. in src/wallet/coinselection.cpp:302 in 1e19f925bb outdated
    303+        while (!heap.empty() && heap.size() > gg_group_limit) {
    304+            const OutputGroup& to_remove_group = heap.top();
    305+            selected_eff_value -= to_remove_group.GetSelectionAmount();
    306+            weight -= to_remove_group.m_weight;
    307+            heap.pop();
    308+        }
    


    achow101 commented at 9:36 pm on December 4, 2023:

    In 1e19f925bb3d2b74933d8cb6d3628477cdd58c32 “Add Gutter Guard Selector”

    These two loops could probably be consolidated into a single one since they do the same thing, just under different conditions.


    murchandamus commented at 6:54 pm on December 22, 2023:
    I combined the two loops
  9. achow101 commented at 9:37 pm on December 4, 2023: member
    Concept ACK
  10. Add Gutter Guard Selector
    Does a random selection limited to three more output groups than
    largest-first selection would select to fund the transaction. If the
    limit is exceeded during selection, the output group with the lowest
    effective value is discarded.
    a264cd2a27
  11. Add minimal Gutter Guard tests 4a94fc8d82
  12. murchandamus force-pushed on Dec 22, 2023
  13. murchandamus commented at 6:54 pm on December 22, 2023: contributor
    Addressed comments by @achow101
  14. kashifs commented at 12:09 pm on December 24, 2023: contributor

    tested ACK 4a94fc8

    I read through the conversation, understood the problem and use case for this solution, compiled from source, wrote tests similar to those included

  15. DrahtBot requested review from achow101 on Dec 24, 2023
  16. achow101 commented at 6:16 pm on January 10, 2024: member

    Is there a benefit to using SRD here over something oldest first? We are already doing SRD so it might be better to use a different underlying strategy. Having a selection based on oldest first would also be useful for consuming negative EV UTXOs at very low feerates, e.g. less than 3 s/vb.

    It also seems like having the limit be inversely scaled by feerate would be better than the fixed limit. That way, as the feerates increase, we get less consolidatory instead of current behavior where it switches at a particular threshold.

  17. murchandamus commented at 11:14 pm on January 10, 2024: contributor

    Yeah, oldest first might be better for soaking up tiny UTXOs at low feerates that have been piling up during bouts of high feerates. @achow101: What do you think about one of the two like the following progression

    <2s/vB <3s/vB <4s/vB <5s/vB <6s/vB <7s/vB <8s/vB <9s/vB <10s/vB >=10s/vB
    [A:] UTXO Limit +21 +18 +15 +12 +10 +8 +6 +5 +4 +3
    [B:] UTXO Limit +48 +39 +31 +24 +18 +13 +9 +6 +4 +3

    allow negative effective value inputs

    Additionally the unneeded UTXOs should be bounded dependent on the count of the wallet UTXO pool such as the minimum of this progress and e.g. 10% of the wallet’s UTXO count.

    I guess I’ll have to simulate this.

  18. DrahtBot added the label CI failed on Jan 16, 2024
  19. DrahtBot added the label Needs rebase on Feb 9, 2024
  20. DrahtBot commented at 10:26 pm on February 9, 2024: contributor

    🐙 This pull request conflicts with the target branch and needs rebase.

  21. achow101 removed review request from achow101 on Apr 16, 2024
  22. achow101 commented at 6:02 pm on April 24, 2024: member
    Closing as this is going to be superseded by Sand Compactor
  23. achow101 closed this on Apr 24, 2024


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-06-29 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me