wallet: fix BnB selection upper bound #28395

pull furszy wants to merge 2 commits into bitcoin:master from furszy:2023_coinselection_fix_bnb_upper_bound changing 5 files +48 −9
  1. furszy commented at 10:57 pm on September 2, 2023: member

    As BnB is an algorithm specialized in seeking changeless solutions, the final selection amount (the sum of all the selected UTXOs amount) can be as high as one unit below the amount where the transaction creation process starts creating a change output. Which is: target + min_viable_change - 1.

    In other words, the range of the possible solutions is: target <= solution_amount < target + min_viable_change - 1.

    Code level description: We have a discrepancy on the threshold at which we create, or not, “change” between the BnB algorithm and the, higher level, transaction creation process.

    BnB selection upper bound uses cost_of_change, while the transaction creation process uses the min_viable_change variable to obtain the resulting change amount from the coin selection solution (here).

    Essentially, this means that under certain circumstances; when the BnB solution excess is in-between min_viable_change and cost_of_change, the BnB algorithm returns a successful solution which, under the transaction creation process perspective, requires to create change output. This, isn’t an expected behavior as BnB is specialized to only return a changeless solution.

    This can happen when min_viable_change <= BnB_solution_amount - target < cost_of_change.

    Note: This should solve the issue presented in #28372.

    Testing Note: I have decoupled the test in a standalone commit so it can be easily cherry-picked on top of master to verify the test failure vs the test passing here after the changes.

  2. DrahtBot commented at 10:57 pm on September 2, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK brunoerg

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #28366 (Fix waste calculation in SelectionResult by murchandamus)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Wallet on Sep 2, 2023
  4. brunoerg commented at 11:35 pm on September 2, 2023: contributor
    Concept ACK
  5. brunoerg commented at 11:54 pm on September 2, 2023: contributor

    Cherry-picked this commit to test it with #28372 and did the changes to the harness to fit it. Previous crashes don’t fail anymore, will leave it running overnight!

     0diff --git a/src/wallet/test/fuzz/coinselection.cpp b/src/wallet/test/fuzz/coinselection.cpp
     1index 07bb0f001c..2c96c79ea2 100644
     2--- a/src/wallet/test/fuzz/coinselection.cpp
     3+++ b/src/wallet/test/fuzz/coinselection.cpp
     4@@ -125,9 +125,9 @@ FUZZ_TARGET(coinselection)
     5     }
     6 
     7     // Run coinselection algorithms
     8-    auto result_bnb = SelectCoinsBnB(group_pos, target, coin_params.m_cost_of_change, MAX_STANDARD_TX_WEIGHT);
     9+    auto result_bnb = SelectCoinsBnB(group_pos, target, coin_params.min_viable_change, MAX_STANDARD_TX_WEIGHT);
    10     if (result_bnb) {
    11-        assert(result_bnb->GetChange(coin_params.m_cost_of_change, CAmount{0}) == 0);
    12+        assert(result_bnb->GetChange(coin_params.min_viable_change, CAmount{0}) == 0);
    13         assert(result_bnb->GetSelectedValue() >= target);
    14         (void)result_bnb->GetShuffledInputVector();
    15         (void)result_bnb->GetInputSet();
    
  6. brunoerg commented at 0:23 am on September 3, 2023: contributor
  7. DrahtBot added the label CI failed on Sep 3, 2023
  8. maflcko added this to the milestone 26.0 on Sep 3, 2023
  9. maflcko commented at 7:52 am on September 3, 2023: member
    0/bip324_ecdh.cpp
    1clang-tidy-16 -p=/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu -quiet -load=/tidy-build/libbitcoin-tidy.so /ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/coins.cpp
    2clang-tidy-16 -p=/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu -quiet -load=/tidy-build/libbitcoin-tidy.so /ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/wallet/test/coinselector_tests.cpp
    3wallet/test/coinselector_tests.cpp:434:63: error: argument name 'cost_of_change' in comment does not match parameter name 'min_viable_change' [bugprone-argument-comment,-warnings-as-errors]
    4                                            selection_target, /*cost_of_change=*/0, MAX_STANDARD_TX_WEIGHT);
    5                                                              ^
    6./wallet/coinselection.h:412:131: note: 'min_viable_change' declared here
    7util::Result<SelectionResult> SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& selection_target, const CAmount& min_viable_change,
    8                                                                                                                                  ^
    9--
    
  10. brunoerg commented at 10:21 am on September 3, 2023: contributor
    It needs to change unit tests and fuzz.
  11. furszy force-pushed on Sep 3, 2023
  12. furszy renamed this:
    wallet: bugfix, use min viable change on BnB selection upper bound
    wallet: fix BnB selection upper bound
    on Sep 3, 2023
  13. coinselection: bugfix, use min_viable_change on BnB selection upper bound
    As BnB is an algorithm specialized in seeking changeless solutions, the
    final selection amount (the sum of all the selected UTXOs amount) can be
    as high as one unit below the amount where the transaction creation
    process starts creating a change output. Which is: target + min_viable_change - 1.
    
    In other words, the range of the possible solutions is:
    target <= solution_amount < target + min_viable_change - 1.
    1e83af161f
  14. test: coinselection, add coverage for BnB upper bound selection
    The following cases are covered:
    1) The only available solution contain an excess equal to min_viable_change.
       The expected behavior here is BnB search to fail. Because, if BnB would return a valid solution, the transaction
       creation process would create a change output (which is what we don't expect from a BnB solution).
    
    2) The only available solution contain an excess one unit below the min_viable_change.
       The expected behavior here is BnB search returning a successful solution.
    f81047b4ec
  15. furszy force-pushed on Sep 3, 2023
  16. furszy commented at 4:26 pm on September 3, 2023: member

    Updated the PR and its description. Added test coverage in a standalone commit so it can easily be cherry-picked on top of master to verify the failure.

    Also, about the changes. I wouldn’t be against continue using cost_of_change on the BnB upper bound. I just didn’t do it because that would require further modifications at the transaction creation process level rather than at the coin selection algorithm level. But, happy to discuss this possible approach too.

  17. DrahtBot removed the label CI failed on Sep 5, 2023
  18. achow101 commented at 7:27 pm on September 7, 2023: member
    I don’t think this is correct. The upper bound is the cost of change, not the minimum viable change. The minimum viable change is the minimum we’re willing to make and this is not inherently tied to feerates. Cost of change is tied to the current feerate and the long term feerate. The minimum viable change is generally much higher than the cost of change. This PR will result in BnB solutions throwing away far more money.
  19. in src/wallet/coinselection.cpp:110 in 1e83af161f outdated
    101@@ -102,12 +102,19 @@ util::Result<SelectionResult> SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool
    102     bool is_feerate_high = utxo_pool.at(0).fee > utxo_pool.at(0).long_term_fee;
    103     bool max_tx_weight_exceeded = false;
    104 
    105+    // We seek for a solution where the process does not create change, so we can go as high as one unit below
    106+    // the amount where the transaction creation process starts creating a change output.
    107+    // In other words, the range of the possible solutions is:
    108+    // target <= solution_amount < target + min_viable_change - 1.
    109+    // Note: when 'min_viable_change=0', the upper bound is 'selection_target'.
    110+    CAmount selection_upper_bound = selection_target + (min_viable_change != 0 ? min_viable_change - 1 : 0);
    


    murchandamus commented at 7:38 pm on September 7, 2023:

    In GetChange, min_viable_change refers to the minimum amount of the change output (after deducting the fee), so I think this here should at least be selection_target + min_viable_change + change_fee. And min_viable_change should be at least change_spend_fee + 1.

    Since cost_of_change is change_fee + changes_spend_fee, that would make min_viable_change exactly 1 bigger than cost_of_change.

    Generally, I think we should be careful to distinguish the min_viable_change from the “budget necessary to create a change output with at least an amount of min_viable_change.

    I think this PR may also overlap with https://github.com/bitcoin/bitcoin/pull/28366


    furszy commented at 2:01 pm on September 9, 2023:

    In GetChange, min_viable_change refers to the minimum amount of the change output (after deducting the fee), so I think this here should at least be selection_target + min_viable_change + change_fee. And min_viable_change should be at least change_spend_fee + 1.

    Since cost_of_change is change_fee + changes_spend_fee, that would make min_viable_change exactly 1 bigger than cost_of_change.

    Great comment. This made me go further over #28395#pullrequestreview-1618045956. It seems that we had different views of the process. You were watching the typical flow, while I was looking at the SFFO one.

    Will work on it. Thanks!.

  20. murchandamus commented at 7:41 pm on September 7, 2023: contributor

    Cost of change is tied to the current feerate and the long term feerate.

    If I recall correctly, the Bitcoin Core definition of cost_of_change is tied to the discard_feerate for the input, rather than the LTFRE

  21. furszy commented at 6:24 pm on September 8, 2023: member

    I don’t think this is correct. The upper bound is the cost of change, not the minimum viable change. The minimum viable change is the minimum we’re willing to make and this is not inherently tied to feerates. Cost of change is tied to the current feerate and the long term feerate. The minimum viable change is generally much higher than the cost of change. This PR will result in BnB solutions throwing away far more money.

    Ok, will expand. Thats partially correct. Check the following. It was all taken from the sources.

    “cost of change” is computed by (link):

    0change_fee = effective_feerate.GetFee(change_output_size);
    1change_spend_fee = discard_feerate.GetFee(change_spend_size)
    2cost_of_change = change_spend_fee + change_fee;
    

    And, the “minimum viable change” is computed by (link):

    0dust = GetDustThreshold(change_prototype_txout, discard_feerate);
    1change_spend_fee = discard_feerate.GetFee(change_spend_size);
    2min_viable_change = std::max(change_spend_fee + 1, dust);
    

    So, as generally change_spend_fee > dust. We can say that

    0min_viable_change = change_spend_fee + 1
    

    Which means that, the minimum viable change is lower than cost of change. Exactly by change_fee - 1:

    0cost_of_change = min_viable_change + change_fee - 1
    

    Therefore, conclusion is that cost_of_change > min_viable_change.

    Now, let’s look at the two possible GetChange() outcomes. Using the outputs’ effective value vs SFFO.

    1. Using the outputs’ effective value: GetChange() expands min_viable_change in change_fee, which makes min_viable_change > cost_of_change. Exactly by 1.

    2. Using SFFO: GetChange() uses min_viable_change as is. Keeping the cost_of_change > min_viable_change inequality. The diff is exactly change_fee - 1.

    So, it means that.. we are both correct. The first scenario describes what you are saying, while the second one describes what I’m saying. Essentially, BnB, in master, can currently throw up to change_fee - 1 to the change output during SFFO.

    Will work on this. Thanks.

  22. murchandamus commented at 4:05 pm on September 11, 2023: contributor

    I am confused by your conclusion

    So, as generally change_spend_fee > dust.

    According to GetDustThreshold(…), dust = (output_size + input_size) × discard_feerate while change_spend_fee is only change_spend_size × discard_feerate. So shouldn’t that rather be:

    change_spend_fee < dust

  23. furszy commented at 10:48 pm on September 11, 2023: member

    Thanks Murch. Summary of the off-github convo:

    I am confused by your conclusion

    So, as generally change_spend_fee > dust.

    According to GetDustThreshold(…), dust = (output_size + input_size) × discard_feerate while change_spend_fee is only change_spend_size × discard_feerate. So shouldn’t that rather be:

    change_spend_fee < dust

    Yeah. Thats true for the common change output types (p2pkh and p2wpkh), dust is greater than change_spend_fee. But, for p2sh and p2wsh the change output redeem script size can turn around the inequality. GetDustThreshold input size calculation is fixed, while change_spend_fee uses a dummy input mimicking the real input size.

    So.. the conclusion that I had above is not accurate. As regular output types are mostly used in the change output, the bare min_viable_change is usually higher than cost_of_change. Unless the effective feerate is high enough. But, that is a no-prob, because GetChange expands min_viable_change by change_fee which covers for the difference (change_fee is the field linked to the effective feerate).

    As far as I can deduce now, the only scenario where min_viable_change could be lower than cost_of_change is when SFFO is enabled and the effective feerate is high enough. Because BnB upper bound includes cost_of_change, which includes change_fee (the cost of the output). And.. we don’t extend min_viable_change in ´GetChange´ when SFFO is enabled. So, change_fee is not part of the min_viable_change expansion. Which ends up with a BnB result requiring a change output.

    Apologies if this looks a bit intricate; it’s a subject that really drills down into a niche within a niche..

  24. murchandamus commented at 2:41 pm on September 12, 2023: contributor

    Given that we had so much confusion on the exact details of this, maybe we should add tests that confirm our understanding of all the edge cases here.


    Yet again, it is revealed that SFFO is the root of all evil in Bitcoin Core wallet. ;)

  25. furszy commented at 4:23 pm on September 12, 2023: member

    Given that we had so much confusion on the exact details of this, maybe we should add tests that confirm our understanding of all the edge cases here.

    Sure. Made a test for this here https://github.com/furszy/bitcoin-core/commit/23333b47e4f8f15fe97bc12d444f76cd8597845c. As the test is on top of master. It is failing outputting: “Error: BnB selection produced a change output!”.

    Now.. will spend some time thinking on an elegant way to solve this. The simplest one, and the more targeted one, would be to decrease the cost of change by change_fee for the BnB selection function argument. But there might be other ways to fix it.

  26. furszy commented at 10:30 pm on October 1, 2023: member
    Closing per discussion. Instead of modifying BnB upper bound (as is proposed here), there was consensus on disabling BnB when SFFO is enabled. @murchandamus should be pushing a new PR shortly. The unit test from my previous comment can still be used there. Feel free to cherry-pick the following branch commits (containing a slightly different version of my previous test, using the logs instead of adding a new return value). This test branch fails without the bugfix and will pass with it.
  27. furszy closed this on Oct 1, 2023


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-01 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me