Mempool util: Add RBF diagram checks for single chunks against clusters of size 2 #29242

pull instagibbs wants to merge 9 commits into bitcoin:master from instagibbs:2024-01-diagram-checks changing 13 files +1310 −41
  1. instagibbs commented at 6:07 pm on January 12, 2024: member

    This is a smaller piece of #28984 broken off for easier review.

    Up to date explanation of diagram checks are here: https://delvingbitcoin.org/t/mempool-incentive-compatibility/553

    This infrastructure has two near term applications prior to cluster mempool:

    1. Limited Package RBF(https://github.com/bitcoin/bitcoin/pull/28984): We want to allow package RBF only when we know it improves the mempool. This narrowly scoped functionality allows use with v3-like topologies, and will be expanded at some point post-cluster mempool when diagram checks can be done efficiently against bounded cluster sizes.
    2. Replacement for single tx RBF(in a cluster size of up to two) against conflicts of up to cluster size two. ImprovesFeerateDiagram interface will have to change for this use-case, which is a future direction to solve certain pins and improve mempool incentive compatibility: https://delvingbitcoin.org/t/ephemeral-anchors-and-mev/383#diagram-checks-fix-this-3

    And longer-term, this would be the proposed way we would compute incentive compatibility for all conflicts, post-cluster mempool.

  2. DrahtBot commented at 6:07 pm on January 12, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK sipa, glozow, murchandamus, ismaelsadeeq, willcl-ark, sdaftuar
    Concept ACK hebasto

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    No conflicts as of last run.

  3. instagibbs force-pushed on Jan 12, 2024
  4. DrahtBot commented at 6:29 pm on January 12, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/20435922692

  5. DrahtBot added the label CI failed on Jan 12, 2024
  6. instagibbs force-pushed on Jan 12, 2024
  7. instagibbs force-pushed on Jan 12, 2024
  8. DrahtBot removed the label CI failed on Jan 12, 2024
  9. instagibbs commented at 9:38 pm on January 12, 2024: member

    ready for review

    cc @glozow @ismaelsadeeq @achow101

  10. in src/util/feefrac.h:91 in ec3cf62361 outdated
    86+    {
    87+        fee += other.fee;
    88+        size += other.size;
    89+    }
    90+
    91+    /** Subtrack size and size of another FeeFrac from this one. */
    


    ismaelsadeeq commented at 2:09 pm on January 16, 2024:
    0    /** Add fee and size of another FeeFrac to this one. */
    1    void inline operator+=(const FeeFrac& other) noexcept
    2    {
    3        fee += other.fee;
    4        size += other.size;
    5    }
    6
    7    /** Subtrack fee and size of another FeeFrac from this one. */
    

    instagibbs commented at 1:14 pm on January 17, 2024:
    also, Subtrack isn’t a word…

    instagibbs commented at 4:01 pm on January 19, 2024:
    done
  11. in src/util/feefrac.cpp:12 in ec3cf62361 outdated
    0@@ -0,0 +1,21 @@
    1+// Copyright (c) The Bitcoin Core developers
    2+// Distributed under the MIT software license, see the accompanying
    3+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
    4+
    5+#include <util/feefrac.h>
    6+
    7+void BuildDiagramFromUnsortedChunks(std::vector<FeeFrac>& chunks, std::vector<FeeFrac>& diagram)
    8+{
    9+    diagram.clear();
    


    ismaelsadeeq commented at 2:09 pm on January 16, 2024:

    Reserve here?

    0    diagram.clear();
    1    diagram.reserve(chunks.size() + 1);
    

    instagibbs commented at 4:01 pm on January 19, 2024:
    done
  12. in src/util/feefrac.cpp:20 in ec3cf62361 outdated
    15+    for (auto& chunk : chunks) {
    16+        FeeFrac& last = diagram.back();
    17+        diagram.emplace_back(last.fee+chunk.fee, last.size+chunk.size);
    18+    }
    19+    return;
    20+}
    


    ismaelsadeeq commented at 2:10 pm on January 16, 2024:

    No need for this return here

    0}
    

    instagibbs commented at 4:01 pm on January 19, 2024:
    done
  13. in src/util/feefrac.h:150 in ec3cf62361 outdated
    151+        std::swap(a.fee, b.fee);
    152+        std::swap(a.size, b.size);
    153+    }
    154+};
    155+
    156+void BuildDiagramFromUnsortedChunks(std::vector<FeeFrac>& chunks, std::vector<FeeFrac>& diagram);
    


    ismaelsadeeq commented at 2:14 pm on January 16, 2024:
    Maybe add a docstring on what BuildDiagramFromUnsortedChunks does in and out params description that are doxygen compatible?

    instagibbs commented at 4:01 pm on January 19, 2024:
    added comment

    sipa commented at 4:20 pm on February 21, 2024:

    The correctness of the result of BuildDiagramFromUnsortedChunks very much depends on the assumption that the chunks can be reordered w.r.t. one another, which seems to be the case in the current PR, but will very much not be the case going forward in a general cluster mempool world.

    I would suggest either:

    • Removing the sorting step in this function (moving it to the caller). The function can then take Span<const FeeFrac> as input too, and perhaps in a further iteration (in a later PR) this function could then be turned into one that actually performs chunking too.
    • Adding a big comment on the function definition that it needs independent chunks.

    sdaftuar commented at 4:13 pm on February 22, 2024:

    The correctness of the result of BuildDiagramFromUnsortedChunks very much depends on the assumption that the chunks can be reordered w.r.t. one another, which seems to be the case in the current PR, but will very much not be the case going forward in a general cluster mempool world.

    No objection to your suggested change here, but I don’t follow this comment – in a general cluster mempool world, unless we were to bound chunk sizes to something smaller than a cluster size (resulting in the possibility of chunk feerates not being monotonically decreasing), I think for the purposes of a feerate diagram it is fine to sort by chunk feerate and permit equal-feerate-chunks to be reordered? From the perspective of a feerate diagram, two such diagrams are equivalent.

    If you’re suggesting that we want to preserve the ability in the future to impose a chunk size limit, then yes I agree that we should rewrite this… Is there another case that I’m overlooking?


    instagibbs commented at 5:57 pm on February 22, 2024:

    @sdaftuar that matches my understanding.

    Removing the sorting step in this function (moving it to the caller). The function can then take Span as input too, and perhaps in a further iteration (in a later PR) this function could then be turned into one that actually performs chunking too.

    Took this suggestion

  14. in src/test/feefrac_tests.cpp:17 in 8d74a28a62 outdated
    12+BOOST_AUTO_TEST_CASE(feefrac_operators)
    13+{
    14+    FeeFrac p1{1000, 100}, p2{500, 300};
    15+    FeeFrac sum{1500, 400};
    16+    FeeFrac diff{500, -200};
    17+    FeeFrac empty{0, 0};
    


    ismaelsadeeq commented at 2:24 pm on January 16, 2024:

    nit: we can just use the empty constructor

    0    FeeFrac empty{};
    

    instagibbs commented at 12:19 pm on January 17, 2024:
    Even better, I’ll add a test that they’re the same

    instagibbs commented at 4:01 pm on January 19, 2024:
    done
  15. in src/test/feefrac_tests.cpp:16 in 8d74a28a62 outdated
    11+
    12+BOOST_AUTO_TEST_CASE(feefrac_operators)
    13+{
    14+    FeeFrac p1{1000, 100}, p2{500, 300};
    15+    FeeFrac sum{1500, 400};
    16+    FeeFrac diff{500, -200};
    


    ismaelsadeeq commented at 2:25 pm on January 16, 2024:
    why can we have negative size ?

    instagibbs commented at 12:26 pm on January 17, 2024:
    this is testing that subtraction works correctly, not that an individual chunk can have negative size

    ismaelsadeeq commented at 1:19 pm on January 17, 2024:
    Yes, My question was… whats the rationale for FeeFrac to generally have negative size ?

    instagibbs commented at 4:01 pm on January 19, 2024:
    We use subtraction, and should try to handle that?

    sipa commented at 4:18 pm on January 19, 2024:

    I can think of one use case where negative sizes are actually useful, though it’s pretty far fetched.

    Currently the code hardcodes a “tail feerate” (the feerate of which we assume there is an infinite supply) of 0 sat/vB for mempool improvement checks, but in theory this can be configurable. In that case, in addition to the diagram check between old and new package, we want the property that fee_new >= fee_old + (size_new - size_old) * feerate_tail. If old, new, and tail are CFeeFrac objects, this condition is exactly !((new - old) << tail). If new is smaller than old, the new - old object has negative size.

    A more pragmatic answer is that there is no real reason to restrict it to just positive sizes, and not enforcing it is simpler.


    sdaftuar commented at 4:54 pm on February 14, 2024:

    If old, new, and tail are CFeeFrac objects, this condition is exactly !((new - old) << tail). If new is smaller than old, the new - old object has negative size. @sipa With the various Assume() calls that check that if the size is 0, the fee must also be zero, doesn’t that mean that we can’t really write code like this example you gave? Unless we checked that new != old first – otherwise you might create a FeeFrac with 0 size and non-zero fee.


    sipa commented at 2:25 pm on February 15, 2024:
    @sdaftuar Indeed. It wouldn’t require anything more than dropping those Assume() calls, but right now FeeFrac objects represent (the aggregate fee and size of) sets of transactions, rather differences between such aggregates (as would be needed for this use case).

    sdaftuar commented at 3:42 pm on February 17, 2024:
    I think we should drop the Assume() calls, as otherwise I think we ought to do more to ensure that using operator- on FeeFrac’s is safe everywhere that it might be invoked.
  16. in src/txmempool.cpp:1305 in ab9a952bdd outdated
    1300+
    1301+    // new_diagram will have chunks that consist of each ancestor of
    1302+    // direct_conflicts that is at its own fee/size, along with the replacement
    1303+    // tx/package at its own fee/size
    1304+
    1305+    // old diagram will consist of each element of direct_conflicts either at
    


    ismaelsadeeq commented at 3:36 pm on January 16, 2024:

    I think you mean all_conflicts here

    0    // old diagram will consist of each element of all_conflicts either at
    

    instagibbs commented at 4:01 pm on January 19, 2024:
    done
  17. in src/txmempool.cpp:1327 in ab9a952bdd outdated
    1322+        // Does this transaction have ancestors?
    1323+        FeeFrac individual{txiter->GetModifiedFee(), txiter->GetTxSize()};
    1324+        if (txiter->GetCountWithAncestors() > 1) {
    1325+            // We'll add chunks for either the ancestor by itself and this tx
    1326+            // by itself, or for a combined package.
    1327+            FeeFrac package{txiter->GetModFeesWithAncestors(), int32_t(txiter->GetSizeWithAncestors())};
    


    ismaelsadeeq commented at 3:42 pm on January 16, 2024:

    use static cast?

    0            FeeFrac package{txiter->GetModFeesWithAncestors(), static_cast<int32_t>(txiter->GetSizeWithAncestors())};
    

    instagibbs commented at 4:01 pm on January 19, 2024:
    done
  18. in src/txmempool.cpp:1363 in ab9a952bdd outdated
    1360+                new_chunks.emplace_back(parent.GetModifiedFee(), parent.GetTxSize());
    1361+            }
    1362+        }
    1363+    }
    1364+    // + CNK
    1365+    new_chunks.emplace_back(replacement_fees, int32_t(replacement_vsize));
    


    ismaelsadeeq commented at 3:52 pm on January 16, 2024:

    I understand what happens here but its a bit confusing to me the short forms CON, CNK DIAGRAM Will prefer something like the compute in computing the OLD chunk

     0    // Step 2: build the new diagram
     1
     2    // Add any parents of direct conflicts that are not conflicted themselves into the NEW chunk
     3    for (auto direct_conflict : direct_conflicts) {
     4        // If a direct conflict has an ancestor that is not in all_conflicts,
     5        // it can be affected by the replacement of the child.
     6        if (direct_conflict->GetMemPoolParentsConst().size() > 0) {
     7            // Grab the parent.
     8            const CTxMemPoolEntry& parent = direct_conflict->GetMemPoolParentsConst().begin()->get();
     9            if (!all_conflicts.count(mapTx.iterator_to(parent))) {
    10                // This transaction would be left over, so add to the NEW
    11                // chunk.
    12                new_chunks.emplace_back(parent.GetModifiedFee(), parent.GetTxSize());
    13            }
    14        }
    15    }
    16    // Add the replacement package to NEW
    17    new_chunks.emplace_back(replacement_fees, int32_t(replacement_vsize));
    

    instagibbs commented at 12:21 pm on January 17, 2024:

    These are shorthand labels from https://delvingbitcoin.org/t/post-clustermempool-package-rbf-per-chunk-processing/190 which helped me think through preciesely how it maps onto the concepts laid out.

    I’d like to keep track of these ideas somehow, maybe by defining the labels?


    ismaelsadeeq commented at 1:17 pm on January 17, 2024:
    Yeah, maybe to be consistent define the labels and do the same in the OLD chunk computation loop above. Though the explicit comments in the OLD chunk computation loop is more easier to parse for me :).

    instagibbs commented at 4:01 pm on January 19, 2024:
    took some of hte language, and explicitly defined each term
  19. in src/policy/rbf.cpp:204 in ab9a952bdd outdated
    301+        return strprintf("unable to compute mining score");
    302+    }
    303+
    304+    if (!CompareFeerateDiagram(old_diagram, new_diagram)) {
    305+        return strprintf("insufficient feerate");
    306+    }
    


    ismaelsadeeq commented at 4:23 pm on January 16, 2024:
    Should forward the detailed error message also?

    instagibbs commented at 12:23 pm on January 17, 2024:
    can you be more explicit in suggestion? The errors are being returned?

    ismaelsadeeq commented at 1:11 pm on January 17, 2024:
    I meant what is being returned by err_string and CompareFeerateDiagram was not used. why were you unable to compute mining score, same for CompareFeerateDiagram why is the fee rate insufficient.

    instagibbs commented at 2:08 pm on January 17, 2024:
    CompareFeerateDiagram won’t return much very interesting, but you’re right for the err_string, will return it

    instagibbs commented at 4:01 pm on January 19, 2024:
    done
  20. instagibbs force-pushed on Jan 18, 2024
  21. DrahtBot added the label CI failed on Jan 18, 2024
  22. DrahtBot commented at 8:29 pm on January 18, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/20632055548

  23. instagibbs force-pushed on Jan 18, 2024
  24. instagibbs force-pushed on Jan 19, 2024
  25. instagibbs force-pushed on Jan 19, 2024
  26. in src/policy/rbf.h:128 in a4b92cfd64 outdated
    123+                                                CAmount replacement_fees,
    124+                                                int64_t replacement_vsize)
    125+                                                EXCLUSIVE_LOCKS_REQUIRED(pool.cs);
    126+
    127+/** Compares two feerate diagrams. The shorter one is padded with a horizonal line. */
    128+std::partial_ordering CompareFeerateDiagram(std::vector<FeeFrac>& dia0, std::vector<FeeFrac>& dia1);
    


    sipa commented at 4:09 pm on January 19, 2024:
    Nit: perhaps this function belongs more in util/feefrac (as the building of the diagram also lives there)

    instagibbs commented at 8:25 pm on January 19, 2024:
    makes sense, moved
  27. in src/test/fuzz/feefrac.cpp:47 in a4b92cfd64 outdated
    42+    add_fn((a & 0xffffffff) * (b >> 32), 1);
    43+    add_fn((a >> 32) * (b >> 32), 2);
    44+    return ret;
    45+}
    46+
    47+/* comparison helper for std::vector */
    


    sipa commented at 4:20 pm on January 19, 2024:
    Nit: std::array.

    instagibbs commented at 8:25 pm on January 19, 2024:
    done
  28. in src/test/fuzz/feefrac.cpp:50 in a4b92cfd64 outdated
    45+}
    46+
    47+/* comparison helper for std::vector */
    48+std::strong_ordering compare_arrays(const std::array<uint32_t, 4>& a, const std::array<uint32_t, 4>& b) {
    49+    for (size_t i = 0; i < a.size(); ++i) {
    50+        if (a[i] < b[i]) return std::strong_ordering::less;
    


    sipa commented at 4:21 pm on January 19, 2024:

    Nit: EMBRACE THE SPACESHIP

    0if (a[i] != b[i]) return a[i] <=> b[i];
    

    instagibbs commented at 8:25 pm on January 19, 2024:
    done
  29. in src/test/fuzz/feefrac.cpp:39 in a4b92cfd64 outdated
    34+        }
    35+        // Make sure no overflow.
    36+        assert(carry == 0);
    37+    };
    38+
    39+    // Multiply the 4 individual limbs (schoolbook multiply, with base 2^64).
    


    sipa commented at 4:21 pm on January 19, 2024:
    Self-nit: 2^64 -> 2^32

    instagibbs commented at 8:25 pm on January 19, 2024:
    done
  30. in src/test/fuzz/feefrac.cpp:22 in a4b92cfd64 outdated
    17+std::array<uint32_t, 4> Mul128(uint64_t a, uint64_t b)
    18+{
    19+    std::array<uint32_t, 4> ret{0, 0, 0, 0};
    20+
    21+    /** Perform ret += v << (32 * pos), at 128-bit precision. */
    22+    auto add_fn = [&](uint64_t v, int pos) {
    


    sipa commented at 4:36 pm on January 19, 2024:

    Suggested change that (I believe) avoids the need for suppressions (and may be more readable?):

     0    auto add_fn = [&](uint64_t v, int pos) {
     1        uint64_t accum{0};
     2        for (int i = 0; i + pos < 4; ++i) {
     3            // Add current value at limb pos in ret.
     4            accum += ret[3 - pos - i];
     5            // Add low or high half of v.
     6            if (i == 0) accum += v & 0xffffffff;
     7            if (i == 1) accum += v >> 32;
     8            // Store lower half of result in limb pos in ret.
     9            ret[3 - pos - i] = accum & 0xffffffff;
    10            // Leave carry in accum.
    11            accum >>= 32;
    12        }
    13        // Make sure no overflow.
    14        assert(accum == 0);
    15    };
    

    instagibbs commented at 8:25 pm on January 19, 2024:
    looks good, done
  31. in src/test/fuzz/feefrac.cpp:62 in a4b92cfd64 outdated
    58+    // Compute and compare signs.
    59+    int sign_a = (a1 == 0 ? 0 : a1 < 0 ? -1 : 1) * (a2 == 0 ? 0 : a2 < 0 ? -1 : 1);
    60+    int sign_b = (b1 == 0 ? 0 : b1 < 0 ? -1 : 1) * (b2 == 0 ? 0 : b2 < 0 ? -1 : 1);
    61+    if (sign_a != sign_b) return sign_a <=> sign_b;
    62+
    63+    // Compute absolute values.
    


    sipa commented at 4:38 pm on January 19, 2024:

    Avoid the need for a suppression:

    0// Compute absolute values.
    1uint64_t abs_a1 = static_cast<uint64_t>(a1), abs_a2 = static_cast<uint64_t>(a2);
    2uint64_t abs_b1 = static_cast<uint64_t>(b1), abs_b2 = static_cast<uint64_t>(b2);
    3// Use (~x + 1) instead of the equivalent (-x) to silence the linter; mod 2^64 behavior is
    4// intentional here.
    5if (a1 < 0) abs_a1 = ~abs_a1 + 1;
    6if (a2 < 0) abs_a2 = ~abs_a2 + 1;
    7if (b1 < 0) abs_b1 = ~abs_b1 + 1;
    8if (b2 < 0) abs_b2 = ~abs_b2 + 1;
    

    (The same trick is used in CScriptNum::serialize, it could be abstracted out)


    instagibbs commented at 8:25 pm on January 19, 2024:
    done
  32. in src/util/feefrac.h:13 in a4b92cfd64 outdated
     8+#include <stdint.h>
     9+#include <compare>
    10+#include <vector>
    11+#include <util/check.h>
    12+
    13+namespace {
    


    sipa commented at 4:39 pm on January 19, 2024:
    Self-nit: no need for this namespace anymore.

    instagibbs commented at 8:25 pm on January 19, 2024:
    done
  33. in src/util/feefrac.h:125 in a4b92cfd64 outdated
    120+    {
    121+        return a.fee == b.fee && a.size == b.size;
    122+    }
    123+
    124+    /** Check if two FeeFrac objects are different (either fee or size differs). */
    125+    friend inline bool operator!=(const FeeFrac& a, const FeeFrac& b) noexcept
    


    sipa commented at 4:40 pm on January 19, 2024:
    Self-nit: this operator can be dropped (C++20 will automatically generate it as negation of operator==).

    instagibbs commented at 8:25 pm on January 19, 2024:
    done
  34. in src/util/feefrac.h:160 in a4b92cfd64 outdated
    155+        if (cross_a == cross_b) return b.size <=> a.size;
    156+        return cross_a <=> cross_b;
    157+    }
    158+
    159+    /** Check if a FeeFrac object is worse than another. */
    160+    friend inline bool operator<(const FeeFrac& a, const FeeFrac& b) noexcept
    


    sipa commented at 4:41 pm on January 19, 2024:
    Self-nit: operator<, operator>, operator<= and operator>= can be dropped (C++20 will autogenerate them using operator<=> (inspecting the compiled code, it’s marginally less efficient, but all the actually performance-critical uses use operator<<, operator>>, or FeeRateCompare anyway).

    instagibbs commented at 8:25 pm on January 19, 2024:
    done
  35. DrahtBot removed the label CI failed on Jan 19, 2024
  36. instagibbs force-pushed on Jan 19, 2024
  37. instagibbs force-pushed on Jan 19, 2024
  38. instagibbs force-pushed on Jan 19, 2024
  39. DrahtBot added the label CI failed on Jan 19, 2024
  40. DrahtBot commented at 8:32 pm on January 19, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/20672153966

  41. instagibbs commented at 9:42 pm on January 19, 2024: member
    test failure appears to be spurious wallet failure, ready for review
  42. DrahtBot removed the label CI failed on Jan 19, 2024
  43. sipa commented at 2:00 pm on January 21, 2024: member
    In https://github.com/sipa/bitcoin/commits/pr29242 I pushed another commit which makes CompareFeerateDiagram not modify the diagrams in-place.
  44. instagibbs force-pushed on Jan 22, 2024
  45. instagibbs commented at 5:30 pm on January 22, 2024: member

    In https://github.com/sipa/bitcoin/commits/pr29242 I pushed another commit which makes CompareFeerateDiagram not modify the diagrams in-place.

    Taken only with minor comment changes, and added another test case or two to cover the iterative nature of the check

  46. instagibbs force-pushed on Jan 22, 2024
  47. in src/policy/rbf.cpp:22 in 9257f33d25 outdated
    18@@ -19,6 +19,8 @@
    19 #include <limits>
    20 #include <vector>
    21 
    22+#include <compare>
    


    murchandamus commented at 7:42 pm on January 22, 2024:

    In the commit message of “Implement ImprovesFeerateDiagram” (9257f33d259bf68f1df1cc41b7b3caaea341782f):

    This new function takes the populated sets of direct and all conflicts computed in the current mempool, assuming the replacements are a single chunk, and computes a diagram check.

    The first sentence is confusing to me. Could you perhaps clarify “the populated sets of direct and all conflicts” and split the sentence up?


    instagibbs commented at 1:26 pm on January 23, 2024:
    direct_conflicts and all_conflicts. Does that help?

    murchandamus commented at 7:05 pm on January 23, 2024:
    Yeah, that would improve it
  48. in src/txmempool.cpp:1264 in 9257f33d25 outdated
    1270+        } else if (has_ancestor && has_descendant) {
    1271+            return strprintf("%s has both ancestor and descendant", txid_string);
    1272+        }
    1273+        // Additionally enforce that:
    1274+        // If we have a parent, we are its only child.
    1275+        // If we have a child,  we are its only parent.
    


    murchandamus commented at 7:55 pm on January 22, 2024:
    In “Implement ImprovesFeerateDiagram” (9257f33d259bf68f1df1cc41b7b3caaea341782f): Nit: The documentation is in opposite order as the checks. Perhaps switch these two lines.

    instagibbs commented at 1:52 pm on January 23, 2024:
    done
  49. in src/txmempool.cpp:1271 in 9257f33d25 outdated
    1266+        if (ancestor_count > 2) {
    1267+            return strprintf("%s has %u ancestors, max 1 allowed", txid_string, ancestor_count - 1);
    1268+        } else if (descendant_count > 2) {
    1269+            return strprintf("%s has %u descendants, max 1 allowed", txid_string, descendant_count - 1);
    1270+        } else if (has_ancestor && has_descendant) {
    1271+            return strprintf("%s has both ancestor and descendant", txid_string);
    


    murchandamus commented at 7:57 pm on January 22, 2024:

    In “Implement ImprovesFeerateDiagram” (9257f33d259bf68f1df1cc41b7b3caaea341782f):

    0            return strprintf("%s has both ancestor and descendant, exceeding cluster limit of 2", txid_string);
    

    instagibbs commented at 1:52 pm on January 23, 2024:
    taken
  50. in src/policy/rbf.h:122 in 9257f33d25 outdated
    114@@ -106,4 +115,20 @@ std::optional<std::string> PaysForRBF(CAmount original_fees,
    115                                       CFeeRate relay_fee,
    116                                       const uint256& txid);
    117 
    118+/**
    119+ * The replacement transaction must improve the feerate diagram of the mempool.
    120+ * @param[in]   pool                The mempool.
    121+ * @param[in]   direct_conflicts    Set of txids corresponding to the direct conflicts
    122+ * @param[in]   all_conflicts       Set of mempool entries corresponding to all conflicts
    


    murchandamus commented at 8:12 pm on January 22, 2024:
    In “Implement ImprovesFeerateDiagram” (9257f33d259bf68f1df1cc41b7b3caaea341782f): Maybe that’s my unfamiliarity with the mempool code, but it is not obvious to me what “direct_conflicts” and “all_conflicts” are, and the description here is self-referential. Do these transactions belong to original, replacement, or both, etc.?

    instagibbs commented at 1:30 pm on January 23, 2024:

    direct_conflicts -> the set of transactions that have at least one input conflicting with a proposed transaction. all_conflicts -> Everything that would be evicted by the proposed transaction

    I’ll touch this up a bit but this is the nomenclature elsewhere

  51. in src/txmempool.cpp:1335 in 9257f33d25 outdated
    1330+                // therefore higher than the parent's fee. Chunk these
    1331+                // together.
    1332+                old_chunks.emplace_back(package);
    1333+            } else {
    1334+                // Add two points, one for the parent and one for this child.
    1335+                old_chunks.emplace_back(package-individual);
    


    murchandamus commented at 8:34 pm on January 22, 2024:
    In “Implement ImprovesFeerateDiagram” (9257f33d259bf68f1df1cc41b7b3caaea341782f): Nit: Spaces around that minus? It took me a moment to realize that dashes aren’t a thing in variables.

    instagibbs commented at 1:51 pm on January 23, 2024:
    fixed
  52. in src/txmempool.h:744 in 9257f33d25 outdated
    739+     * Calculate the old and new mempool feerate diagrams relating to the
    740+     * clusters that would be affected by a potential replacement transaction.
    741+     *
    742+     * @param[in] replacement_fees    Package fees
    743+     * @param[in] replacement_vsize   Package size
    744+     * @param[in] direct_conflicts    All transactions that would be removed directly (not via descendants)
    


    murchandamus commented at 8:55 pm on January 22, 2024:

    In “Implement ImprovesFeerateDiagram” (9257f33d259bf68f1df1cc41b7b3caaea341782f):

    0     * [@param](/bitcoin-bitcoin/contributor/param/)[in] direct_conflicts    All transactions that would be removed directly (not by being descendants of conflicting transactions)
    

    instagibbs commented at 1:51 pm on January 23, 2024:
    tried my own explanation
  53. in src/test/rbf_tests.cpp:296 in b3d415fe84 outdated
    294+    old_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 400}};
    295+    new_diagram = {FeeFrac{0, 0}, FeeFrac{1100, 300}};
    296+
    297+    BOOST_CHECK(std::is_lt(CompareFeerateDiagram(old_diagram, new_diagram)));
    298+
    299+    // Feerate of first chunk is sufficiently better, but second chunk is worse.
    


    murchandamus commented at 9:40 pm on January 22, 2024:
    In “test: Add tests for CompareFeerateDiagram and CheckConflictTopology” (b3d415fe84be7edfbed567a79a25d406b438622b): Nit: I found the comment here confusing. How about: “new_diagram is strictly better due to the first chunk, while the second chunk is worse.”

    instagibbs commented at 1:51 pm on January 23, 2024:
    tried my own re-explanation. I agree as-is is confusing
  54. in src/test/rbf_tests.cpp:304 in b3d415fe84 outdated
    302+
    303+    BOOST_CHECK(std::is_lt(CompareFeerateDiagram(old_diagram, new_diagram)));
    304+
    305+    // Feerate of first chunk is better, but second chunk is worse
    306+    old_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 400}};
    307+    new_diagram = {FeeFrac{0, 0}, FeeFrac{750, 100}, FeeFrac{999, 350}, FeeFrac{1150, 500}};
    


    murchandamus commented at 9:44 pm on January 22, 2024:

    In “test: Add tests for CompareFeerateDiagram and CheckConflictTopology” (b3d415fe84be7edfbed567a79a25d406b438622b):

    Nit: in the new diagram, the last two chunks should be one chunk because the last chunk is better (151 sats, 150 vB) than the prior (249 sats, 250 vB).


    instagibbs commented at 1:51 pm on January 23, 2024:
    ah hmm, I’m not sure that matters for correctness of test, but jacked up the last chunk’s size just to make it clearer what we’re testing
  55. in src/test/rbf_tests.cpp:328 in b3d415fe84 outdated
    326+
    327+    // No change in evaluation when tail check needed.
    328+    BOOST_CHECK(std::is_gt(CompareFeerateDiagram(old_diagram, new_diagram)));
    329+
    330+    old_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 399}};
    331+
    


    murchandamus commented at 9:51 pm on January 22, 2024:

    In “test: Add tests for CompareFeerateDiagram and CheckConflictTopology” (b3d415fe84be7edfbed567a79a25d406b438622b):

    This is the same old_diagram. Did you mean to repeat the same transactions,

    0
    1    old_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 399}};
    2    new_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 400}};
    

    or just test in both directions?


    instagibbs commented at 1:51 pm on January 23, 2024:
    I can’t recall honestly, removed that line
  56. murchandamus commented at 10:06 pm on January 22, 2024: contributor
    Concept ACK, tentative crACK f094a0370db7085b8a89842de0b6d12272d826cb
  57. instagibbs force-pushed on Jan 23, 2024
  58. instagibbs commented at 1:56 pm on January 23, 2024: member
    @murchandamus comments should be addressed or taken
  59. in src/txmempool.h:745 in a14b95129d outdated
    740+     * clusters that would be affected by a potential replacement transaction.
    741+     *
    742+     * @param[in] replacement_fees    Package fees
    743+     * @param[in] replacement_vsize   Package size
    744+     * @param[in] direct_conflicts    All transactions that would be removed directly by
    745+     *                                having a conflicting input wih a proposed transaction
    


    murchandamus commented at 7:09 pm on January 23, 2024:
    0     *                                having a conflicting input with a proposed transaction
    

    instagibbs commented at 1:37 pm on January 25, 2024:
    fixed
  60. in src/test/rbf_tests.cpp:509 in a14b95129d outdated
    326+
    327+    BOOST_CHECK(std::is_lt(CompareFeerateDiagram(old_diagram, new_diagram)));
    328+
    329+    // Incomparable diagrams
    330+    old_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 400}};
    331+    new_diagram = {FeeFrac{0, 0}, FeeFrac{1000, 300}, FeeFrac{1000, 400}};
    


    murchandamus commented at 7:12 pm on January 23, 2024:
    If all diagrams always start with FeeFrac{0, 0}, why do we need to mention that as a first element?

    instagibbs commented at 1:37 pm on January 25, 2024:
    I’ll leave this as an ergonomics issue for later debate
  61. murchandamus commented at 7:14 pm on January 23, 2024: contributor

    ACK a14b95129d3a2894b7a41ce919a426bb60f62e35

    Note: I am fairly familiar with the concept of feerate diagram checks, but not particularly familiar with the mempool code.

  62. dergoegge commented at 9:07 pm on January 23, 2024: member
    0$ echo "BAIBAgICAgIABfcN/f11BwAAAPsAAAICAgICAgICAi0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwb3J0AgL+//////8hAQICAooBATIBAR4AAgICAgICAgICAgICAgICAgICAgICAgIC/v///////wECAgKKAQEyAQEeAgIiAgICAgICHgACAgIDAgICAgICAiICAgICAExLQAICAgICAgICAv//////////////////////////////////////////////////////AQEBAQEByQEBAQEBAQEB/P8AAC8AGvsAXAEAAgLcAgICAgICAlSbtqZ56uTNNhvmIHMu83ifTNtXiVGR9z3J4We8dnhZegAAAAAAAAD/AAAAAAAHAAAAAAAAXP96AAAAAgICAgAF9w39/QAAAAEAAAABAgICAgICAgICLZCPcnQCAgIC3AICAgICAgJUm7ameerkzRvmIHMu83ifTNtXiVGR9z3J4We8dnhZegAAAAAAAAAAAAAAAAcABgEAAAAAAAAAAAAAAPsCIlwBAAAHAAAAAQAAAAmLXAEVAAAAAAAAAAAAAAAAAAAAXP+B/////////wAAAAAABwAAAAAAAAAAAAAABwAALAACAgIATEtAAgICAgICAgIC//////////////////////////////////////////////////////8BAQEBAQHJAQEBAQEBAQH8/wAALwAa+wBcAQACAtwCAgICAgICVJu2pnnq5M02G+Ygcy7zeJ9M21eJUZH3PcnhZ7x2eFl6AAAAAAAAAP8AAAAAAAcAAAAAAABc/3oAAAACAgICAAX3Df39AAAAAQAAAAECAgICAgICAgItkI9ydAICAgLcAgICAgICAlSbtqZ56uTNG+Ygcy7zeJ9M21eJUZH3PcnhZ7x2eFl6AAAAAAAAAAAAAAAABwAGAQAAAAAAAAAAAAAA+wIiXAEAAAcAAAABAAAACYtcARUAAAAAAAAAAAAAAAAAAABc/4H/////////AAAAAAAHAAAAAAAAAAAAAAAHAAAsAAAAAPsCAlwBAAAHAAAAAQAAAAmLXAEDAwMDAwMD/729vb29vb29vb29vb29vb29vb29vb29vT+9/wUtYmluvb29vb29vb29vb29vb29vb29vb29/wUtYmluZC0U/////wAAAAAAAAAAHAAAAAAAAP///////////////////////4pVyFwAAAD7AgJcAQAABwAAAAEAAAAJi1wBAwMDAwMDA/+9vb29vb29vb29vb29vb29vb29vb29vb0/vf8FLWJpbr29vb29vb29vb29vb29vb29vb29vf8FLWJpbmQtFP////8AAAAAAAAAABwAAAAAAAD///////////////////////+KVchcToownpjPpiYHL06aRtJ04wyb/sdpyYUpxMKugP////////////////////////9t////AQAAAEZFRUVFRUVFRUVFRUVFRUVFRUVFRUVF/////////////////////3r//////////wMDAwP//////w==" | base64 --decode > package_rbf-7d643e62d75f4ac67d3bdeaf9f5fbfd67675806a.crash
    1$ FUZZ=package_rbf ./src/test/fuzz/fuzz package_rbf-7d643e62d75f4ac67d3bdeaf9f5fbfd67675806a.crash
    2fuzz_libfuzzer: test/fuzz/rbf.cpp:147: void package_rbf_fuzz_target(FuzzBufferType): Assertion `old_diagram.back().fee - replaced_fee + replacement_fees == new_diagram.back().fee' failed.
    
  63. instagibbs commented at 3:07 pm on January 24, 2024: member
    @dergoegge fuzzer created a circular reference which caused incongruities between descendant counts and actually walking the descendants in mempool. Please run again?
  64. dergoegge commented at 4:03 pm on January 24, 2024: member

    Please run again?

    Running now.

    Some coverage reports from the previous runs (one per harness): http://bitcoind-fuzz.dergoegge.de/instagibbs/bitcoin/2024-01-diagram-checks/coverage/

  65. willcl-ark added the label TX fees and policy on Jan 24, 2024
  66. willcl-ark added the label Mempool on Jan 24, 2024
  67. instagibbs force-pushed on Jan 25, 2024
  68. instagibbs force-pushed on Jan 25, 2024
  69. instagibbs commented at 1:38 pm on January 25, 2024: member
    didn’t see any fuzzer crashes locally, all comments addressed
  70. in src/util/feefrac.h:64 in c5e63487c5 outdated
    61+#endif
    62+
    63+    /** Fee. */
    64+    int64_t fee;
    65+    /** Size. */
    66+    int32_t size;
    


    ismaelsadeeq commented at 4:37 pm on January 29, 2024:
    0    int64_t fee;
    1    int32_t size;
    

    Should their be more information that will describe the transaction size and fee, base / modified fee, vsize in bytes or something of that sort. If not I think its okay like this.


    Unrelated just asking to learn. Why are’nt we using CAmount here for the fee?

    In some places I see transaction size as uint32_t while some places its int32_t.

    Should we have a type for size just like CAmount?


    instagibbs commented at 3:54 pm on February 12, 2024:

    If not I think its okay like this.

    removed comments, how it’s precisely used is an implementation detail outside of this code

  71. murchandamus commented at 6:22 pm on February 2, 2024: contributor

    ACK a1414b3388a870aeb463e9cc293d47cbd86e7f0e

    via rangediff a14b95129d3a2894b7a41ce919a426bb60f62e35..a1414b3388a870aeb463e9cc293d47cbd86e7f0e

    The changes since my prior ACK appear to only pertain to improvements in the fuzz test to avoid circular packages and a typo fix on a comment in the code.

  72. DrahtBot added the label Needs rebase on Feb 10, 2024
  73. instagibbs force-pushed on Feb 12, 2024
  74. instagibbs commented at 3:56 pm on February 12, 2024: member
    rebased
  75. DrahtBot removed the label Needs rebase on Feb 12, 2024
  76. in src/util/feefrac.h:30 in d200d30431 outdated
    25+ * - fee=2 size=2 (feerate 1)
    26+ * - fee=1 size=1 (feerate 1)
    27+ * - fee=3 size=2 (feerate 1.5)
    28+ * - fee=2 size=1 (feerate 2)
    29+ * - fee=0 size=0 (undefined feerate)
    30+ *
    


    ismaelsadeeq commented at 5:16 pm on February 13, 2024:

    In implementation they are sorted in increasing fee rate and the chunks in the comments should be ordered as is? and also the 0 fee and size chucks sorts first.

     0 * by decreasing size. The empty FeeFrac (fee and size both 0) sorts first. So for example, the
     1 * following FeeFracs are in sorted order:
     2 *
     3 * - fee=0 size=0 (undefined feerate)
     4 * - fee=2 size=1 (feerate 2)
     5 * - fee=3 size=2 (feerate 1.5)
     6 * - fee=1 size=1 (feerate 1)
     7 * - fee=2 size=2 (feerate 1)
     8 * - fee=2 size=3 (feerate 0.667...)
     9 * - fee=1 size=2 (feerate 0.5)
    10 * - fee=0 size=1 (feerate 0)
    

    instagibbs commented at 8:44 pm on February 14, 2024:
    I don’t understand your comment, sorry. They’re listed in increasing feerate, with decreasing size as a tie breaker.

    ismaelsadeeq commented at 1:50 pm on February 15, 2024:

    Yes but in reverse I think. Suppose to be the other way around no?

    A sorted FeeFracs should have undefined chunks first, the highest fee rate chunk second continuously… If their is a tie, the chunk with lower size comes first. Hence the are sorted in increasing fee rates, and then by decreasing size.

    I’ve added test to verify this here https://github.com/ismaelsadeeq/bitcoin/commit/8ce89b132f4304a7feda067a88fd0a72330044a6, this test passed on this branch.


    sipa commented at 1:55 pm on February 15, 2024:
    @ismaelsadeeq The comparator you provide to std::sort is supposed to return whether the first argument should sort before the second one (if no comparator is provided, operator< is used). With that, you should see lowest feerate first, highest feerate last, and the undefined feefrac at the very end. Within equal-feerate groups, larger size comes first.

    ismaelsadeeq commented at 2:00 pm on February 15, 2024:
    Yes thanks for clarification 👍🏾

    ismaelsadeeq commented at 2:05 pm on February 15, 2024:
    I looked at it the other way around, using the result from BuildDiagramFromUnsortedChunks sort which uses custom > operator @instagibbs.

    instagibbs commented at 4:53 pm on February 15, 2024:
    I added some additional feefrac fuzzing for the +/- operations, marking as resolved
  77. in src/txmempool.cpp:1283 in 802a104d0e outdated
    1275+        }
    1276+    }
    1277+    return std::nullopt;
    1278+}
    1279+
    1280+std::optional<std::string> CTxMemPool::CalculateFeerateDiagramsForRBF(CAmount replacement_fees, int64_t replacement_vsize, const setEntries& direct_conflicts, const setEntries& all_conflicts, std::vector<FeeFrac>& old_diagram, std::vector<FeeFrac>& new_diagram)
    


    ismaelsadeeq commented at 5:18 pm on February 13, 2024:
    Maybe add unit test for CalculateFeerateDiagramsForRBF also?

    instagibbs commented at 4:52 pm on February 15, 2024:
    good point; added
  78. in src/test/fuzz/feefrac.cpp:98 in ff96f3a7a6 outdated
    93+    FeeFrac fr1(f1, s1);
    94+    assert(fr1.IsEmpty() == (s1 == 0));
    95+
    96+    int64_t f2 = provider.ConsumeIntegral<int64_t>();
    97+    //int64_t f2 = provider.ConsumeIntegralInRange(std::numeric_limits<int64_t>::min() + 1,
    98+    //                            std::numeric_limits<int64_t>::max() - 1);
    


    ismaelsadeeq commented at 5:21 pm on February 13, 2024:

    Why not delete the commented version of f1 and f2.

    0    int32_t s1 = provider.ConsumeIntegral<int32_t>();
    1    if (s1 == 0) f1 = 0;
    2    FeeFrac fr1(f1, s1);
    3    assert(fr1.IsEmpty() == (s1 == 0));
    4
    5    int64_t f2 = provider.ConsumeIntegral<int64_t>();
    6      
    

    instagibbs commented at 4:52 pm on February 15, 2024:
    mistakenly left at some point; removed
  79. ismaelsadeeq approved
  80. ismaelsadeeq commented at 5:34 pm on February 13, 2024: member

    Code review ACK 3a2c66cc4f0afad9aff300110b18a39981055d63

    I’ve reviewed the following commits

    And a light review of the fuzzing commits.

    The code looks good to me, I did some local testing and mess around with the implementation and unit tests.

    I have just a few comments for potential further improvements.

  81. DrahtBot requested review from murchandamus on Feb 13, 2024
  82. instagibbs force-pushed on Feb 15, 2024
  83. instagibbs commented at 4:56 pm on February 15, 2024: member
    addressed all comments
  84. instagibbs force-pushed on Feb 15, 2024
  85. DrahtBot added the label CI failed on Feb 15, 2024
  86. DrahtBot removed the label CI failed on Feb 15, 2024
  87. in src/test/rbf_tests.cpp:354 in d57fbda350 outdated
    349+    BOOST_CHECK(old_diagram.size() == 2);
    350+    BOOST_CHECK(new_diagram.size() == 2);
    351+    BOOST_CHECK(old_diagram[0] == FeeFrac(0, 0));
    352+    BOOST_CHECK(old_diagram[1] == FeeFrac(low_fee, low_size));
    353+    BOOST_CHECK(new_diagram[0] == FeeFrac(0, 0));
    354+    BOOST_CHECK(new_diagram[1] == FeeFrac(0, 0));
    


    instagibbs commented at 5:14 pm on February 16, 2024:
    this is odd, should we just Assume() replacement_vsize>0 in CalculateFeerateDiagramsForRBF?

    ismaelsadeeq commented at 7:41 pm on February 16, 2024:
    Yes, I can’t think of a scenario where that will happen!

    instagibbs commented at 5:10 pm on February 20, 2024:
    Added Assume for this and remove test
  88. in src/test/rbf_tests.cpp:326 in d57fbda350 outdated
    321+    const CAmount low_fee{CENT/100};
    322+    const CAmount normal_fee{CENT/10};
    323+    const CAmount high_fee{CENT};
    324+
    325+    // low -> high -> medium fee transactions that would result in two chunks together
    326+    const auto low_tx= make_tx(/*inputs=*/ {m_coinbase_txns[0]}, /*output_values=*/ {10 * COIN});
    


    ismaelsadeeq commented at 7:42 pm on February 16, 2024:

    nit:

    0    const auto low_tx = make_tx(/*inputs=*/ {m_coinbase_txns[0]}, /*output_values=*/ {10 * COIN});
    

    ismaelsadeeq commented at 10:44 am on February 17, 2024:
    here an another place below!

    instagibbs commented at 5:10 pm on February 20, 2024:
    fixed
  89. in src/test/rbf_tests.cpp:495 in d57fbda350 outdated
    429+    BOOST_CHECK(old_diagram[2] == FeeFrac(low_fee + high_fee, low_size_2 + high_size_2));
    430+    BOOST_CHECK(new_diagram[0] == FeeFrac(0, 0));
    431+    BOOST_CHECK(new_diagram[1] == FeeFrac(high_fee, low_size_2));
    432+    old_diagram.clear();
    433+    new_diagram.clear();
    434 }
    


    ismaelsadeeq commented at 10:43 am on February 17, 2024:

    Maybe weshould also check that we can have more than two directly conflicting transactions as long as they are all in a cluster size <=2.

     0diff --git a/src/test/rbf_tests.cpp b/src/test/rbf_tests.cpp
     1index e15fd29da3..561ebd3881 100644
     2--- a/src/test/rbf_tests.cpp
     3+++ b/src/test/rbf_tests.cpp
     4@@ -431,6 +431,53 @@ BOOST_FIXTURE_TEST_CASE(calc_feerate_diagram_rbf, TestChain100Setup)
     5     BOOST_CHECK(new_diagram[1] == FeeFrac(high_fee, low_size_2));
     6     old_diagram.clear();
     7     new_diagram.clear();
     8+
     9+    // You can more than two direct conflicts,  if all the directly conflicting transactions are in a cluster size < 2
    10+    const auto conflict_1 = make_tx(/*inputs=*/ {m_coinbase_txns[2]}, /*output_values=*/ {10 * COIN});
    11+    pool.addUnchecked(entry.Fee(low_fee).FromTx(conflict_1));
    12+    const auto conflict_1_entry = pool.GetIter(conflict_1->GetHash()).value();
    13+
    14+    const auto conflict_2 = make_tx(/*inputs=*/ {m_coinbase_txns[3]}, /*output_values=*/ {10 * COIN});
    15+    pool.addUnchecked(entry.Fee(low_fee).FromTx(conflict_2));
    16+    const auto conflict_2_entry = pool.GetIter(conflict_2->GetHash()).value();
    17+
    18+    const auto conflict_3 = make_tx(/*inputs=*/ {m_coinbase_txns[4]}, /*output_values=*/ {10 * COIN});
    19+    pool.addUnchecked(entry.Fee(low_fee).FromTx(conflict_3));
    20+    const auto conflict_3_entry = pool.GetIter(conflict_3->GetHash()).value();
    21+
    22+    const auto err_string8{pool.CalculateFeerateDiagramsForRBF(/*replacement_fees=*/high_fee, /*replacement_vsize=*/low_size, {conflict_1_entry, conflict_2_entry, conflict_3_ent
    23ry}, {conflict_1_entry, conflict_2_entry, conflict_3_entry}, old_diagram, new_diagram)};
    24+
    25+    BOOST_CHECK(!err_string8.has_value());
    26+    BOOST_CHECK(old_diagram.size() == 4);
    27+    BOOST_CHECK(new_diagram.size() == 2);
    28+    old_diagram.clear();
    29+    new_diagram.clear();
    30+
    31+    // Add a child transaction to conflict_1 and make it cluster size 2
    32+    const auto conflict_1_child = make_tx(/*inputs=*/{conflict_1}, /*output_values=*/ {995 * CENT});
    33+    pool.addUnchecked(entry.Fee(low_fee).FromTx(conflict_1_child));
    34+    const auto conflict_1_child_entry = pool.GetIter(conflict_1_child->GetHash()).value();
    35+
    36+    const auto err_string9{pool.CalculateFeerateDiagramsForRBF(/*replacement_fees=*/high_fee, /*replacement_vsize=*/low_size, {conflict_1_entry, conflict_2_entry, conflict_3_ent
    37ry}, {conflict_1_entry, conflict_2_entry, conflict_3_entry, conflict_1_child_entry}, old_diagram, new_diagram)};
    38+
    39+    BOOST_CHECK(!err_string8.has_value());
    40+    BOOST_CHECK(old_diagram.size() == 4);
    41+    BOOST_CHECK(new_diagram.size() == 2);
    42+    old_diagram.clear();
    43+    new_diagram.clear();
    44+
    45+
    46+    // Add another descendant to conflict_1, making the cluster size > 2 should fail at this point.
    47+    const auto conflict_1_grand_child = make_tx(/*inputs=*/{conflict_1_child}, /*output_values=*/ {995 * CENT});
    48+    pool.addUnchecked(entry.Fee(high_fee).FromTx(conflict_1_grand_child));
    49+    const auto conflict_1_grand_child_entry = pool.GetIter(conflict_1_child->GetHash()).value();
    50+
    51+    const auto err_string10{pool.CalculateFeerateDiagramsForRBF(/*replacement_fees=*/high_fee, /*replacement_vsize=*/low_size, {conflict_1_entry, conflict_2_entry, conflict_3_en
    52try}, {conflict_1_entry, conflict_2_entry, conflict_3_entry, conflict_1_child_entry, conflict_1_grand_child_entry}, old_diagram, new_diagram)};
    53+
    54+    BOOST_CHECK(err_string10.has_value());
    55+    BOOST_CHECK(err_string10.value() == strprintf("%s has 2 descendants, max 1 allowed", conflict_1->GetHash().GetHex()));
    56+    old_diagram.clear();
    57+    new_diagram.clear();
    58 }
    59 
    60 BOOST_AUTO_TEST_CASE(feerate_diagram_utilities)
    

    instagibbs commented at 5:09 pm on February 20, 2024:
    taken with only light modifications, thanks!
  90. ismaelsadeeq commented at 10:46 am on February 17, 2024: member
    Thanks for adding the test! re-ACK d57fbda350ee9051931d9a0dad4beb55f6b2e574
  91. in src/util/feefrac.cpp:1 in d200d30431 outdated
    0@@ -0,0 +1,93 @@
    1+// Copyright (c) The Bitcoin Core developers
    


    sdaftuar commented at 3:44 pm on February 17, 2024:
    nit: As the functions here don’t relate to the FeeFrac implementation, perhaps this should be in a different file? feerate_diagram.cpp maybe?

    ismaelsadeeq commented at 11:22 pm on February 17, 2024:
    Curious to know the full meaning of FeeFrac is it Fee Fraction? Maybe Chunk will be a better name?

    sipa commented at 11:46 am on February 20, 2024:

    Yeah, it means “fee fraction”, though the name isn’t perfect - it really represents any ratio where the numerator is an int64_t and the denominator is an int32_t, with a sort order that tie-breaks by putting larger denominators last.

    I don’t think Chunk is a good name, as we’re using that term for a subset of transactions. A FeeFrac is related, but represents the aggregate feerate of a chunk (or really of any set of transactions, or even the difference between the fees/sizes of two sets of transactions).


    instagibbs commented at 5:09 pm on February 20, 2024:
    @sdaftuar going to hold off on this for now if that’s ok
  92. in src/util/feefrac.cpp:51 in d200d30431 outdated
    46+        const int unproc_side = next_point(0).size > next_point(1).size;
    47+
    48+        // let `P` be the next point on diagram unproc_side, and `A` and `B` the previous and next points
    49+        // on the other diagram. We want to know if P lies above or below the line AB. To determine this, we
    50+        // compute the direction coefficients of line AB and of line AP, and compare them. These
    51+        // direction coefficients are fee per size, and can thus be expressed as FeeFracs.
    


    sdaftuar commented at 3:48 pm on February 17, 2024:
    nit: “direction coefficient” is not a term I’m familiar with (and from googling it doesn’t seem like a super common term); perhaps it would be clearer to just use the word “slope” here?

    sipa commented at 9:35 pm on February 17, 2024:
    Oh, I seem to have translated literally from Dutch. “Slope” is indeed the proper English translation.

    instagibbs commented at 5:09 pm on February 20, 2024:
    replaced, even though I thought it sounded pretty smart
  93. in src/util/feefrac.h:159 in d200d30431 outdated
    154+
    155+/** Takes the pre-computed chunks and generates a fee diagram which starts at FeeFrac of (0, 0) */
    156+void BuildDiagramFromUnsortedChunks(std::vector<FeeFrac>& chunks, std::vector<FeeFrac>& diagram);
    157+
    158+/** Compares two feerate diagrams (which must both start at size=0). The shorter one is implicitly
    159+ * extended with a horizontal straight line. */
    


    sdaftuar commented at 7:20 pm on February 17, 2024:
    nit: Could add a comment here more precisely defining the preconditions for this function and what definition we are using for feerate diagram: specifically that a feerate diagram consists of a list of (fee, size) points with the property that size is strictly increasing and that the first entry is (0, 0). (If these conditions are violated then the function can fail with an Assume() failure.)

    instagibbs commented at 5:09 pm on February 20, 2024:
    taken with some liberties
  94. in src/policy/rbf.cpp:193 in 802a104d0e outdated
    188+                                                const CTxMemPool::setEntries& direct_conflicts,
    189+                                                const CTxMemPool::setEntries& all_conflicts,
    190+                                                CAmount replacement_fees,
    191+                                                int64_t replacement_vsize)
    192+{
    193+    // Require that the replacement strictly improve the mempool's fee vs. size diagram.
    


    sdaftuar commented at 7:25 pm on February 17, 2024:
    nit: “fee vs. size diagram” – maybe we should standardize the terminology to “feerate diagram”? I was also thinking about adding some documentation to doc/policy/mempool-replacements.md explaining what we’re doing now.

    instagibbs commented at 5:09 pm on February 20, 2024:
    changed to standard lingo
  95. in src/txmempool.h:757 in 802a104d0e outdated
    745+     * @param[in] all_conflicts       All transactions that would be removed
    746+     * @param[out] old_diagram        The feerate diagram of the relevant clusters before accepting the new tx
    747+     * @param[out] new_diagram        The feerate diagram of the relevant clusters after accepting the new tx
    748+     * @return An optional error string if the conflicts don't match a calculable topology
    749+     */
    750+    std::optional<std::string> CalculateFeerateDiagramsForRBF(CAmount replacement_fees, int64_t replacement_vsize, const setEntries& direct_conflicts, const setEntries& all_conflicts, std::vector<FeeFrac>& old_diagram, std::vector<FeeFrac>& new_diagram) EXCLUSIVE_LOCKS_REQUIRED(cs);
    


    sdaftuar commented at 8:43 pm on February 17, 2024:
    nit: Maybe update the comment to be more explicit that this function only works for the situation where (replacement_fees, replacement_size) corresponds to a transaction package that has no in-mempool dependences.

    instagibbs commented at 5:09 pm on February 20, 2024:
    updated to be more explicit on what those values represent
  96. sdaftuar commented at 8:59 pm on February 17, 2024: member

    Still reviewing all the tests – leaving some nits that I’ve noticed so far.

    In addition:

    • I commented here that I think we should drop the Assume(size != 0 || fee == 0) calls in feefrac.h, so that we don’t have to police all the places we invoke operator- to make sure we can never accidentally violate that.
    • Another nit: in the commit message for commit 860823fb93212fa78ab928589bd403da462be222, it should say “FeeFrac” and not “FeeFrace”

    Finally, as a note to other reviewers, I recently wrote up more explanation of the logic for using feerate diagram comparisons in our RBF policy here: https://delvingbitcoin.org/t/mempool-incentive-compatibility/553/1.

  97. in src/util/feefrac.cpp:73 in d200d30431 outdated
    68+        if (point_b.size == point_p.size) ++next_index[!unproc_side];
    69+    }
    70+
    71+    // Tail check at 0 feerate: Compare the remaining area. Use similar logic as in the loop above,
    72+    // except we use a horizontal line instead of AB, as no point B exists anymore.
    73+    const int long_side = next_index[1] != dias[1].size();
    


    sipa commented at 1:36 pm on February 20, 2024:

    Here is a patch that merges the two processing loops into one (deduplicating some logic between them), and also makes the tail feerate explicit (even though only FeeFrac{0, 1} is used for now).

    No need to take this, but in case people find it simpler to review a generic approach, perhaps it’s useful.

      0diff --git a/src/util/feefrac.cpp b/src/util/feefrac.cpp
      1index a970adedc30..569f32f4615 100644
      2--- a/src/util/feefrac.cpp
      3+++ b/src/util/feefrac.cpp
      4@@ -22,7 +22,7 @@ void BuildDiagramFromUnsortedChunks(std::vector<FeeFrac>& chunks, std::vector<Fe
      5     }
      6 }
      7 
      8-std::partial_ordering CompareFeerateDiagram(Span<const FeeFrac> dia0, Span<const FeeFrac> dia1)
      9+std::partial_ordering CompareFeerateDiagram(Span<const FeeFrac> dia0, Span<const FeeFrac> dia1, FeeFrac tail_feerate)
     10 {
     11     /** Array to allow indexed access to input diagrams. */
     12     const std::array<Span<const FeeFrac>, 2> dias = {dia0, dia1};
     13@@ -40,51 +40,47 @@ std::partial_ordering CompareFeerateDiagram(Span<const FeeFrac> dia0, Span<const
     14     Assert(prev_point(0).IsEmpty());
     15     Assert(prev_point(1).IsEmpty());
     16 
     17-    // Compare the overlapping area of the diagrams.
     18-    while (next_index[0] < dias[0].size() && next_index[1] < dias[1].size()) {
     19-        // Determine which diagram has the first unprocessed point.
     20-        const int unproc_side = next_point(0).size > next_point(1).size;
     21+    do {
     22+        bool done_0 = next_index[0] == dias[0].size();
     23+        bool done_1 = next_index[1] == dias[1].size();
     24+        if (done_0 && done_1) break;
     25+
     26+        // Determine which diagram has the first unprocessed point. If one side is finished, use the
     27+        // other one.
     28+        const int unproc_side = (done_0 || done_1) ? done_0 : next_point(0).size > next_point(1).size;
     29 
     30         // let `P` be the next point on diagram unproc_side, and `A` and `B` the previous and next points
     31         // on the other diagram. We want to know if P lies above or below the line AB. To determine this, we
     32-        // compute the direction coefficients of line AB and of line AP, and compare them. These
     33-        // direction coefficients are fee per size, and can thus be expressed as FeeFracs.
     34+        // compute the slopes of line AB and of line AP, and compare them. These slopes are fee per size,
     35+        // and can thus be expressed as FeeFracs.
     36         const FeeFrac& point_p = next_point(unproc_side);
     37         const FeeFrac& point_a = prev_point(!unproc_side);
     38-        const FeeFrac& point_b = next_point(!unproc_side);
     39-        const auto coef_ab = point_b - point_a;
     40-        const auto coef_ap = point_p - point_a;
     41-        Assume(coef_ap.size > 0);
     42-        Assume(coef_ab.size >= coef_ap.size);
     43-        // Perform the comparison. If P lies above AB, unproc_side is better in P. If P lies below
     44-        // AB, then !unproc_side is better in P.
     45-        const auto cmp = FeeRateCompare(coef_ap, coef_ab);
     46+        const auto slope_ap = point_p - point_a;
     47+        Assume(slope_ap.size > 0);
     48+        std::weak_ordering cmp = std::weak_ordering::equivalent;
     49+        if (!(done_0 || done_1)) {
     50+            // If both sides have points left, compute B, and the slope of AB explicitly.
     51+            const FeeFrac& point_b = next_point(!unproc_side);
     52+            const auto slope_ab = point_b - point_a;
     53+            Assume(slope_ab.size >= slope_ap.size);
     54+            cmp = FeeRateCompare(slope_ap, slope_ab);
     55+
     56+            // If B and P have the same size, B can be marked as processed (in addition to P, see
     57+            // below), as we've already performed a comparison at this size.
     58+            if (point_b.size == point_p.size) ++next_index[!unproc_side];
     59+        } else {
     60+            // If one of the sides has no points left, act as if AB has slope tail_feerate.
     61+            cmp = FeeRateCompare(slope_ap, tail_feerate);
     62+        }
     63+
     64+        // If P lies above AB, unproc_side is better in P. If P lies below AB, then !unproc_side is
     65+        // better in P.
     66         if (std::is_gt(cmp)) better_somewhere[unproc_side] = true;
     67         if (std::is_lt(cmp)) better_somewhere[!unproc_side] = true;
     68 
     69-        // Mark P as processed. If B and P have the same size, B can also be marked as processed as
     70-        // we've already performed a comparison at this size.
     71-        ++next_index[unproc_side];
     72-        if (point_b.size == point_p.size) ++next_index[!unproc_side];
     73-    }
     74-
     75-    // Tail check at 0 feerate: Compare the remaining area. Use similar logic as in the loop above,
     76-    // except we use a horizontal line instead of AB, as no point B exists anymore.
     77-    const int long_side = next_index[1] != dias[1].size();
     78-    Assume(next_index[!long_side] == dias[!long_side].size());
     79-    // The point A now remains fixed: the last point of the shorter diagram.
     80-    const FeeFrac& point_a = prev_point(!long_side);
     81-    while (next_index[long_side] < dias[long_side].size()) {
     82-        // Compare AP (where P is the next unprocessed point on the longer diagram) with a horizontal line
     83-        // extending infinitely from A. This is equivalent to checking the sign of the fee of P-A.
     84-        const FeeFrac& point_p = next_point(long_side);
     85-        const auto coef_ap = point_p - point_a;
     86-        const auto cmp = coef_ap.fee <=> 0;
     87-        if (std::is_gt(cmp)) better_somewhere[long_side] = true;
     88-        if (std::is_lt(cmp)) better_somewhere[!long_side] = true;
     89         // Mark P as processed.
     90-        ++next_index[long_side];
     91-    }
     92+        ++next_index[unproc_side];
     93+    } while(true);
     94 
     95     // If both diagrams are better somewhere, they are incomparable.
     96     if (better_somewhere[0] && better_somewhere[1]) return std::partial_ordering::unordered;
     97diff --git a/src/util/feefrac.h b/src/util/feefrac.h
     98index 12698a4a653..706facc34b0 100644
     99--- a/src/util/feefrac.h
    100+++ b/src/util/feefrac.h
    101@@ -156,7 +156,7 @@ struct FeeFrac
    102 void BuildDiagramFromUnsortedChunks(std::vector<FeeFrac>& chunks, std::vector<FeeFrac>& diagram);
    103 
    104 /** Compares two feerate diagrams (which must both start at size=0). The shorter one is implicitly
    105- * extended with a horizontal straight line. */
    106-std::partial_ordering CompareFeerateDiagram(Span<const FeeFrac> dia0, Span<const FeeFrac> dia1);
    107+ *  extended with an infinite line whose slope equals tail_feefrac (by default: horizontal). */
    108+std::partial_ordering CompareFeerateDiagram(Span<const FeeFrac> dia0, Span<const FeeFrac> dia1, FeeFrac tail_feerate={0, 1});
    109 
    110 #endif // BITCOIN_UTIL_FEEFRAC_H
    

    instagibbs commented at 5:09 pm on February 20, 2024:
    took in a modified form since it does seem to read cleaner to me, left the direct exposure of tail_feerate as a YNGNI extension
  98. in src/txmempool.h:760 in 802a104d0e outdated
    748+     * @return An optional error string if the conflicts don't match a calculable topology
    749+     */
    750+    std::optional<std::string> CalculateFeerateDiagramsForRBF(CAmount replacement_fees, int64_t replacement_vsize, const setEntries& direct_conflicts, const setEntries& all_conflicts, std::vector<FeeFrac>& old_diagram, std::vector<FeeFrac>& new_diagram) EXCLUSIVE_LOCKS_REQUIRED(cs);
    751+
    752+    /* Check that all direct conflicts are in a cluster size of two or less. */
    753+    std::optional<std::string> CheckConflictTopology(const setEntries& direct_conflicts);
    


    glozow commented at 2:58 pm on February 20, 2024:
    Question: is the idea that in the future we’ll use the same function but loosen it beyond cluster <=2? Otherwise, I wonder why it isn’t called LimitTopology1P1C or LimitClusterSize2 or something.

    instagibbs commented at 2:58 pm on March 12, 2024:
    Potentially yes. I suspect it will become fully unrestricted next post-cluster mempool, but we don’t know anything for sure.
  99. instagibbs force-pushed on Feb 20, 2024
  100. instagibbs commented at 5:10 pm on February 20, 2024: member

    I commented #29242 (review) that I think we should drop the Assume(size != 0 || fee == 0) calls in feefrac.h, so that we don’t have to police all the places we invoke operator- to make sure we can never accidentally violate that. Another nit: in the commit message for commit https://github.com/bitcoin/bitcoin/commit/860823fb93212fa78ab928589bd403da462be222, it should say “FeeFrac” and not “FeeFrace”

    Addressed these by removing Assumes and updating the commit text

  101. in src/test/feefrac_tests.cpp:102 in 947693bc8e outdated
     97+    chunks.push_back(empty);
     98+    chunks.push_back(oversized_1);
     99+    chunks.push_back(oversized_2);
    100+
    101+    FastRandomContext rng{/*fDeterministic=*/true};
    102+    std::shuffle(chunks.begin(), chunks.end(), rng);
    


    sipa commented at 4:27 pm on February 21, 2024:
    We have Shuffle() in random.h, which is more efficient than std::shuffle (but only works with FastRandomContext).

    instagibbs commented at 5:48 pm on February 22, 2024:
    removed, since I stripped out the sorting function from BuildDiagramFromUnsortedChunks
  102. in src/util/feefrac.cpp:21 in 4f050b23f3 outdated
    16+
    17+    // And now build the diagram for these chunks.
    18+    diagram.emplace_back(0, 0);
    19+    for (auto& chunk : chunks) {
    20+        FeeFrac& last = diagram.back();
    21+        diagram.emplace_back(last.fee+chunk.fee, last.size+chunk.size);
    


    sipa commented at 4:49 pm on February 21, 2024:

    Nit: no need to reimplement FeeFrac addition:

    0diagram.push_back(diagram.back() + chunk);
    

    instagibbs commented at 5:48 pm on February 22, 2024:
    done
  103. in src/util/feefrac.cpp:56 in 4f050b23f3 outdated
    54+        // compute the slopes of line AB and of line AP, and compare them. These slopes are fee per size,
    55+        // and can thus be expressed as FeeFracs.
    56+        const FeeFrac& point_p = next_point(unproc_side);
    57+        const FeeFrac& point_a = prev_point(!unproc_side);
    58+
    59+        // Slope of AP can be negative, unlike AB
    


    sdaftuar commented at 3:49 pm on February 22, 2024:
    I think the slope of AB can be negative, because a user could use the prioritisetransaction to give a transaction a negative fee?

    instagibbs commented at 3:53 pm on February 22, 2024:
    one weird trick! will remove the comment

    willcl-ark commented at 10:40 am on March 20, 2024:
    Suhas’ comment is still present.

    instagibbs commented at 12:07 pm on March 25, 2024:
    un-resolved comment(oops), will add to follow-up

    instagibbs commented at 4:14 pm on March 25, 2024:
    opened in follow-up
  104. sipa commented at 3:54 pm on February 22, 2024: member
    Partial review.
  105. hebasto commented at 5:14 pm on February 22, 2024: member
    Concept ACK on introducing feerate diagram checks for a subset of RBF transactions.
  106. instagibbs force-pushed on Feb 22, 2024
  107. instagibbs force-pushed on Feb 22, 2024
  108. in src/test/rbf_tests.cpp:423 in 55eae36cfb outdated
    418+    BOOST_CHECK(new_diagram[0] == FeeFrac(0, 0));
    419+    BOOST_CHECK(new_diagram[1] == FeeFrac(high_fee, low_size_2));
    420+    old_diagram.clear();
    421+    new_diagram.clear();
    422+
    423+    // You can have more than two direct conflicts if the therea re multiple effected clusters, all of size 2 or less
    


    ismaelsadeeq commented at 12:03 pm on March 4, 2024:
    0    // You can have more than two direct conflicts if the there are multiple effected clusters, all of size 2 or less
    

    instagibbs commented at 1:49 pm on March 11, 2024:
    done
  109. in src/test/feefrac_tests.cpp:24 in cc7b730683 outdated
    19+
    20+    BOOST_CHECK(empty == FeeFrac{}); // same as no-args
    21+
    22+    BOOST_CHECK(p1 == p1);
    23+    BOOST_CHECK(p1+p2 == sum);
    24+    BOOST_CHECK(p1-p2 == diff);
    


    ismaelsadeeq commented at 12:29 pm on March 4, 2024:

    nit: inconsistent use of the spaces between operators and operands here and other places

    0    BOOST_CHECK(p1 + p2 == sum);
    1    BOOST_CHECK(p1 - p2 == diff);
    

    instagibbs commented at 1:49 pm on March 11, 2024:
    done
  110. in src/test/feefrac_tests.cpp:71 in cc7b730683 outdated
    66+    BOOST_CHECK(oversized_1 <= oversized_2);
    67+    BOOST_CHECK(oversized_1 << oversized_2);
    68+    BOOST_CHECK(oversized_1 != oversized_2);
    69+
    70+    // Tests paths that use double arithmetic
    71+    FeeFrac busted{((int64_t) INT32_MAX) + 1, INT32_MAX};
    


    ismaelsadeeq commented at 12:30 pm on March 4, 2024:

    nit: static cast

    0    FeeFrac busted{static_cast<int64_t>(INT32_MAX) + 1, INT32_MAX};
    

    instagibbs commented at 1:49 pm on March 11, 2024:
    done
  111. ismaelsadeeq approved
  112. ismaelsadeeq commented at 1:33 pm on March 4, 2024: member

    Re-ACK 55eae36cfbd829eabc0772fc1b5312ec1f747e2c Reviewed Diff

    Left some nits

  113. DrahtBot requested review from hebasto on Mar 4, 2024
  114. instagibbs commented at 2:03 pm on March 4, 2024: member
    @ismaelsadeeq thanks, will fix up if I touch commits again
  115. DrahtBot added the label Needs rebase on Mar 9, 2024
  116. instagibbs force-pushed on Mar 11, 2024
  117. instagibbs force-pushed on Mar 11, 2024
  118. instagibbs commented at 1:49 pm on March 11, 2024: member
    took all suggestions and rebased
  119. DrahtBot removed the label Needs rebase on Mar 11, 2024
  120. in src/test/feefrac_tests.cpp:1 in eebe434383 outdated
    0@@ -0,0 +1,124 @@
    1+// Copyright (c) 2016-2021 The Bitcoin Core developers
    


    glozow commented at 1:00 pm on March 12, 2024:
    nit: wrong dates

    instagibbs commented at 3:33 pm on March 13, 2024:
    fixed
  121. in src/test/rbf_tests.cpp:324 in 70ffb25852 outdated
    294+    // Now test ImprovesFeerateDiagram with various levels of "package rbf" feerates
    295+
    296+    // It doesn't improve itself
    297+    const auto res1 = ImprovesFeerateDiagram(pool, {entry1}, {entry1, entry2}, tx1_fee + tx2_fee, tx1_size + tx2_size);
    298+    BOOST_CHECK(res1.has_value());
    299+    BOOST_CHECK(res1.value().first == DiagramCheckError::FAILURE);
    


    glozow commented at 1:25 pm on March 12, 2024:
    nit: I see you use BOOST_CHECK(x == y) a lot. BOOST_CHECK_EQUAL is nice since it prints more helpfully

    instagibbs commented at 4:31 pm on March 12, 2024:
    seems I have to implement overloaded operator std::ostream& operator<<(std::ostream& os, const FeeFrac& type) to support it, and seems somewhat rare in the codebase. I can do it of course. Thoughts?

    glozow commented at 5:38 pm on March 12, 2024:
    Just a nit, so feel free to ignore.

    instagibbs commented at 3:33 pm on March 13, 2024:
    ignoring for now
  122. in src/test/rbf_tests.cpp:347 in 8c77f2552a outdated
    342+    BOOST_CHECK(new_diagram[1] == FeeFrac(0, 1));
    343+    old_diagram.clear();
    344+    new_diagram.clear();
    345+
    346+    // Non-zero replacement fee/size
    347+    const auto err_string3{pool.CalculateFeerateDiagramsForRBF(/*replacement_fees=*/high_fee, /*replacement_vsize=*/low_size, {entry_low}, {entry_low}, old_diagram, new_diagram)};
    


    glozow commented at 1:44 pm on March 12, 2024:
    nit: you skipped err_string2, maybe you could give them descriptive names like result_cpfp_replace_cpfp, result_replace_3gen result_replace_3independent, etc.

    instagibbs commented at 3:33 pm on March 13, 2024:
    an attempt was made!
  123. in src/txmempool.h:759 in 5f6ddba263 outdated
    754+     * @param[out] new_diagram        The feerate diagram of the relevant clusters after accepting the new tx
    755+     * @return An optional error string if the conflicts don't match a calculable topology
    756+     */
    757+    std::optional<std::string> CalculateFeerateDiagramsForRBF(CAmount replacement_fees, int64_t replacement_vsize, const setEntries& direct_conflicts, const setEntries& all_conflicts, std::vector<FeeFrac>& old_diagram, std::vector<FeeFrac>& new_diagram) EXCLUSIVE_LOCKS_REQUIRED(cs);
    758+
    759+    /* Check that all direct conflicts are in a cluster size of two or less. */
    


    glozow commented at 1:46 pm on March 12, 2024:
    5f6ddba2635ca2100ed46560f338b64a58f61b73 Maybe mention can contain multiple clusters?

    instagibbs commented at 3:33 pm on March 13, 2024:
    done
  124. in src/txmempool.h:757 in 5f6ddba263 outdated
    752+     * @param[in] all_conflicts       All transactions that would be removed
    753+     * @param[out] old_diagram        The feerate diagram of the relevant clusters before accepting the new tx
    754+     * @param[out] new_diagram        The feerate diagram of the relevant clusters after accepting the new tx
    755+     * @return An optional error string if the conflicts don't match a calculable topology
    756+     */
    757+    std::optional<std::string> CalculateFeerateDiagramsForRBF(CAmount replacement_fees, int64_t replacement_vsize, const setEntries& direct_conflicts, const setEntries& all_conflicts, std::vector<FeeFrac>& old_diagram, std::vector<FeeFrac>& new_diagram) EXCLUSIVE_LOCKS_REQUIRED(cs);
    


    glozow commented at 2:43 pm on March 12, 2024:
    Did you consider using util::Result for this instead? Seems like you want std::pair<std::vector<FeeFrac>, std::vector<FeeFrac>> on success or a std::string on failure.

    instagibbs commented at 3:33 pm on March 13, 2024:
    done
  125. in src/util/feefrac.h:150 in d2025b37cd outdated
    145+        std::swap(a.size, b.size);
    146+    }
    147+};
    148+
    149+/** Takes the pre-computed and topologically-valid chunks and generates a fee diagram which starts at FeeFrac of (0, 0) */
    150+std::vector<FeeFrac> BuildDiagramFromChunks(const Span<const FeeFrac> chunks);
    


    glozow commented at 2:44 pm on March 12, 2024:
    Would you consider creating an alias using FeerateDiagram = std::vector<FeeFrac>? Could be a bit easier to write/read.

    instagibbs commented at 3:33 pm on March 13, 2024:
    doesn’t seem to save much typing/reading if we’re using Span, leaving as is for now

    sipa commented at 7:54 pm on March 15, 2024:

    In commit “Add FeeFrac utils”

    Nit: the const before Span<const FeeFrac> doesn’t do anything (it’s ignored in the declaration of by-value function arguments). You can have it be const in the function definition (in feefrac.cpp) without having it const here (where it does have the effect that the function body cannot modify the copy it receives).


    instagibbs commented at 2:26 pm on March 18, 2024:
    I probably should just be passing a reference here anyways? Changed to pass by reference

    sipa commented at 2:29 pm on March 18, 2024:
    The general advice is to pass arguments not larger than 2 pointers by value, which is the case for spans.

    instagibbs commented at 2:35 pm on March 18, 2024:
    got it, double checked, taking your suggestion to remove const from declaration instead
  126. in src/txmempool.cpp:1278 in 5f6ddba263 outdated
    1273+            const auto& our_parent = direct_conflict->GetMemPoolParentsConst().begin();
    1274+            if (our_parent->get().GetCountWithDescendants() > 2) {
    1275+                return strprintf("%s is not the only child of parent %s",
    1276+                                 txid_string, our_parent->get().GetSharedTx()->GetHash().ToString());
    1277+            }
    1278+        }
    


    glozow commented at 2:49 pm on March 12, 2024:
    Nothing failed when I commented out this chunk, maybe need some 1p2c or 2p1c CheckConflictTopology tests?

    instagibbs commented at 3:33 pm on March 13, 2024:
    oops, bad test setup was missing this. Fixed and added string assertions to check what it’s actually catching.

    glozow commented at 12:17 pm on March 19, 2024:
    I see you’ve added coverage for the “not the only parent” case, but I don’t see a test for “not the only child” case. Also nothing failed when I commented it out.

    instagibbs commented at 3:34 pm on March 19, 2024:
    I presume you mean the multi-child case didn’t trigger failure when removed? multi-parent should
  127. glozow commented at 2:50 pm on March 12, 2024: member
    The code looks good, I’m still reading through the tests
  128. in src/test/rbf_tests.cpp:281 in 70ffb25852 outdated
    276+    TestMemPoolEntryHelper entry;
    277+
    278+    const CAmount low_fee{CENT/100};
    279+    const CAmount normal_fee{CENT/10};
    280+
    281+    // One low feerate txn followed be normal feerate
    


    glozow commented at 5:35 pm on March 12, 2024:

    typo: s/be/by

    If you’re taking suggestions,

    0    // low feerate parent with normal feerate child
    

    instagibbs commented at 3:33 pm on March 13, 2024:
    taken
  129. in src/test/rbf_tests.cpp:305 in 70ffb25852 outdated
    300+    BOOST_CHECK(res1.value().second == "insufficient feerate: does not improve feerate diagram");
    301+
    302+    // With one more satoshi it does
    303+    BOOST_CHECK(ImprovesFeerateDiagram(pool, {entry1}, {entry1, entry2}, tx1_fee + tx2_fee + 1, tx1_size + tx2_size) == std::nullopt);
    304+
    305+    // Make conflict un-calculable(for now)
    


    glozow commented at 5:37 pm on March 12, 2024:

    What does “for now” mean in this context?

    0    // Adding a grandchild makes the cluster size 3, which is uncalculable
    

    instagibbs commented at 3:33 pm on March 13, 2024:
    we could expand acceptable topologies later. Took language.
  130. in src/test/fuzz/rbf.cpp:107 in 5661fbaec3 outdated
    102+    if (!mtx) return;
    103+
    104+    CTxMemPool pool{MemPoolOptionsForTest(g_setup->m_node)};
    105+
    106+    // Add a bunch of parent-child pairs to the mempool, and remember them.
    107+    std::vector<CTransaction> txs;
    


    glozow commented at 5:42 pm on March 12, 2024:

    nit 5661fbaec3cb2f8d8790e8c347889a6dea29f324

    mempool_txs would be a helpful name


    instagibbs commented at 3:34 pm on March 13, 2024:
    done
  131. in src/test/fuzz/rbf.cpp:121 in 5661fbaec3 outdated
    116+        assert(iter <= g_outpoints.size());
    117+        parent->vin.resize(1);
    118+        parent->vin[0].prevout = g_outpoints[iter++];
    119+
    120+        txs.emplace_back(*parent);
    121+        LOCK2(cs_main, pool.cs);
    


    glozow commented at 5:49 pm on March 12, 2024:
    Is there any reason to unlock and lock in the loop? Why not just hold throughout?

    instagibbs commented at 3:34 pm on March 13, 2024:
    changed to hold it throughout
  132. in src/test/fuzz/rbf.cpp:43 in 5661fbaec3 outdated
    38+    static const auto testing_setup = MakeNoLogFileContext<>();
    39+    g_setup = testing_setup.get();
    40+
    41+    // Create a fixed set of unique "UTXOs" to source parents from
    42+    // to avoid fuzzer giving circular references
    43+    for (int i = 0; i < NUM_ITERS * 2; ++i) {
    


    glozow commented at 5:52 pm on March 12, 2024:
    I don’t think you need * 2

    instagibbs commented at 3:34 pm on March 13, 2024:
    removed
  133. in src/test/fuzz/rbf.cpp:177 in 5661fbaec3 outdated
    172+        assert(old_diagram.back().size - replaced_size + replacement_vsize == new_diagram.back().size);
    173+    }
    174+
    175+    // If internals report error, wrapper should too
    176+    auto err_tuple{ImprovesFeerateDiagram(pool, direct_conflicts, all_conflicts, replacement_fees, replacement_vsize)};
    177+    if (err_string.has_value()) assert(err_tuple.has_value());
    


    glozow commented at 5:56 pm on March 12, 2024:
    It seems like the 10000 mempool setup could be reused for multiple iterations of replacement tests?

    instagibbs commented at 3:34 pm on March 13, 2024:
    I guess the question is if unique mempools per run are worth the overhead. I think it’s pretty fast still?
  134. in src/test/fuzz/rbf.cpp:102 in 5661fbaec3 outdated
     97+{
     98+    FuzzedDataProvider fuzzed_data_provider(buffer.data(), buffer.size());
     99+    SetMockTime(ConsumeTime(fuzzed_data_provider));
    100+
    101+    std::optional<CMutableTransaction> mtx = ConsumeDeserializable<CMutableTransaction>(fuzzed_data_provider, TX_WITH_WITNESS);
    102+    if (!mtx) return;
    


    glozow commented at 6:01 pm on March 12, 2024:
    Why make this ahead of time instead of just creating a new CMutableTransaction for each child?

    instagibbs commented at 3:34 pm on March 13, 2024:
    Aping the current rbf tests, probably. Picks a valid transaction structure, then sticks with it for the run. Suggestions for follow-up improvements maybe?
  135. in src/test/fuzz/feeratediagram.cpp:92 in f4e7c2748d outdated
    124+    }
    125+}
    126+
    127+FUZZ_TARGET(build_and_compare_feerate_diagram)
    128+{
    129+    // Generate a random set of chunks
    


    glozow commented at 6:02 pm on March 12, 2024:
    Maybe “same as above, but start with chunks instead of diagrams” ? Can any of the code maybe be helper funcitoned?

    instagibbs commented at 3:34 pm on March 13, 2024:
    removed the more limited fuzz test, as I don’t think it adds additional coverage
  136. glozow commented at 6:04 pm on March 12, 2024: member
    Reviewed tests, just have some nits
  137. instagibbs force-pushed on Mar 13, 2024
  138. instagibbs commented at 4:26 pm on March 13, 2024: member
    I believe I addressed all comments. (and failure unrelated?)
  139. instagibbs force-pushed on Mar 13, 2024
  140. DrahtBot added the label CI failed on Mar 13, 2024
  141. dergoegge commented at 11:19 pm on March 13, 2024: member
  142. DrahtBot removed the label CI failed on Mar 14, 2024
  143. in src/txmempool.cpp:1305 in e88d6ceff0 outdated
    1300+
    1301+    std::vector<FeeFrac> old_chunks;
    1302+    // Step 1: build the old diagram.
    1303+
    1304+    // The above clusters are all trivially linearized;
    1305+    // they have a strict topology of 1 or two connected transactions.
    


    murchandamus commented at 7:19 pm on March 14, 2024:

    Nit:

    0    // they have a strict topology of one or two connected transactions.
    

    or

    0    // they have a strict topology of 1 or 2 connected transactions.
    

    instagibbs commented at 4:14 pm on March 25, 2024:
    opened in follow-up
  144. in src/policy/rbf.cpp:198 in e88d6ceff0 outdated
    193+    // Require that the replacement strictly improve the mempool's feerate diagram.
    194+    std::vector<FeeFrac> old_diagram, new_diagram;
    195+
    196+    const auto diagram_results{pool.CalculateFeerateDiagramsForRBF(replacement_fees, replacement_vsize, direct_conflicts, all_conflicts)};
    197+
    198+    if (!diagram_results.has_value()) {
    


    sipa commented at 8:36 pm on March 15, 2024:

    In commit “Implement ImprovesFeerateDiagram”

    I’m confused about this. If diagram_results has no value, does it make sense to pass it to util::ErrorString below?


    instagibbs commented at 2:04 pm on March 18, 2024:

    util::Result lets you pass an error string in lieu of a std::nullopt, and based on the docs it seems I’m using this right. @glozow ?

    0//! `std::optional<T>` can be updated to return `util::Result<T>` and return
    1//! error strings usually just replacing `return std::nullopt;` with `return
    2//! util::Error{error_string};`.
    

    sipa commented at 2:10 pm on March 18, 2024:

    But you aren’t passing it an error string, you’re passing it diagram_results, which is at this stage known to be std::nullopt?

    Nevermind, I missed that diagram_results is an util::Result, not an std::optional, so the !diagram_results.has_value() means it does contain an error. Sorry!


    instagibbs commented at 2:15 pm on March 18, 2024:

    this is what it’s returning instead of a std::nullopt: https://github.com/bitcoin/bitcoin/pull/29242/files#diff-c065d4cd2398ad0dbcef393c5dfc53f465bf44723348892395fffd2fb3bac522R1290

    (we might be talking past each other)


    sipa commented at 2:17 pm on March 18, 2024:
    Right, just discovered I missed what type this was all along. See update above.
  145. in src/txmempool.cpp:1336 in e88d6ceff0 outdated
    1331+            old_chunks.emplace_back(individual);
    1332+        }
    1333+    }
    1334+
    1335+    // No topology restrictions post-chunking; sort
    1336+    std::sort(old_chunks.begin(), old_chunks.end(), [](const FeeFrac& a, const FeeFrac& b) { return a > b; });
    


    sipa commented at 9:01 pm on March 15, 2024:

    In commit “Implement ImprovesFeerateDiagram”

    Nit: shorter as:

    0std::sort(old_chunks.begin(), old_chunks.end(), std::greater{});
    

    (here and below)


    instagibbs commented at 2:26 pm on March 18, 2024:
    done
  146. sipa commented at 1:45 pm on March 17, 2024: member

    Here is a commit that removes the need to explicitly compute the diagram, and instead makes the diagram comparison operate directly on a (sorted) list of chunks: https://github.com/sipa/bitcoin/commits/pr29242

    Feel free to ignore, postpone, include, or squash.

  147. instagibbs commented at 2:07 pm on March 18, 2024: member
    @sipa to keep things moving (muh package rbf) I’ll postpone/ promise to review your PR for that commit as a follow-up.
  148. instagibbs force-pushed on Mar 18, 2024
  149. Add FeeFrac utils
    Co-authored-by: Suhas Daftuar <sdaftuar@chaincode.com>
    Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
    ce8e22542e
  150. Add FeeFrac unit tests
    Co-authored-by: Suhas Daftuar <sdaftuar@chaincode.com>
    66d966dcfa
  151. Implement ImprovesFeerateDiagram
    This new function takes the populated sets of
    direct and all conflicts computed in the current
    mempool, assuming the replacements are a single
    chunk, and computes a diagram check.
    
    The diagram check only works against cluster
    sizes of 2 or less, and fails if it encounters
    a different topology.
    
    Co-authored-by: Suhas Daftuar <sdaftuar@chaincode.com>
    2079b80854
  152. fuzz: Add fuzz target for ImprovesFeerateDiagram
    Co-authored-by: Suhas Daftuar <sdaftuar@chaincode.com>
    588a98dccc
  153. test: Add tests for CompareFeerateDiagram and CheckConflictTopology e9c5aeb11d
  154. fuzz: fuzz diagram creation and comparison
    Co-authored-by: Suhas Daftuar <sdaftuar@chaincode.com>
    Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
    4d6528a3d6
  155. Add fuzz test for FeeFrac 7e89b659e1
  156. test: unit test for ImprovesFeerateDiagram b767e6bd47
  157. Unit tests for CalculateFeerateDiagramsForRBF 7295986778
  158. instagibbs force-pushed on Mar 18, 2024
  159. in src/util/feefrac.h:34 in ce8e22542e outdated
    29+ * - fee=0 size=0 (undefined feerate)
    30+ *
    31+ * A FeeFrac is considered "better" if it sorts after another, by this ordering. All standard
    32+ * comparison operators (<=>, ==, !=, >, <, >=, <=) respect this ordering.
    33+ *
    34+ * The CompareFeeFrac, and >> and << operators only compare feerate and treat equal feerate but
    


    sipa commented at 2:59 pm on March 18, 2024:

    In commit “Add FeeFrac utils”:

    Self-nit: CompareFeeFrac -> FeeRateCompare


    instagibbs commented at 4:13 pm on March 25, 2024:
    opened in follow-up
  160. in src/policy/rbf.cpp:193 in 2079b80854 outdated
    188+                                                const CTxMemPool::setEntries& direct_conflicts,
    189+                                                const CTxMemPool::setEntries& all_conflicts,
    190+                                                CAmount replacement_fees,
    191+                                                int64_t replacement_vsize)
    192+{
    193+    // Require that the replacement strictly improve the mempool's feerate diagram.
    


    sipa commented at 3:04 pm on March 18, 2024:

    In commit “Implement ImprovesFeerateDiagram”

    Nit: improves


    instagibbs commented at 4:13 pm on March 25, 2024:
    opened in follow-up
  161. sipa commented at 4:34 pm on March 18, 2024: member

    utACK 72959867784098137a50c34f86deca8235eef4f8

    Some non-blocking nits:

  162. DrahtBot requested review from ismaelsadeeq on Mar 18, 2024
  163. in src/test/rbf_tests.cpp:343 in e9c5aeb11d outdated
    341+    // Identical diagrams, cannot be strictly better
    342+    old_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 400}};
    343+    new_diagram = {FeeFrac{0, 0}, FeeFrac{950, 300}, FeeFrac{1050, 400}};
    344+
    345+    BOOST_CHECK(std::is_eq(CompareFeerateDiagram(old_diagram, new_diagram)));
    346+
    


    glozow commented at 12:00 pm on March 19, 2024:

    e9c5aeb11d641b8cae373452339760809625021d could do the reciprocal checks everywhere i.e.

    0BOOST_CHECK(std::is_lt(CompareFeerateDiagram(old_diagram, new_diagram)));
    1BOOST_CHECK(std::is_gt(CompareFeerateDiagram(new_diagram, old_diagram)));
    2
    3...
    4BOOST_CHECK(std::is_eq(CompareFeerateDiagram(old_diagram, new_diagram)));
    5BOOST_CHECK(std::is_eq(CompareFeerateDiagram(new_diagram, old_diagram)));
    

    instagibbs commented at 4:13 pm on March 25, 2024:
    opened in follow-up
  164. in src/test/rbf_tests.cpp:130 in e9c5aeb11d outdated
    147+    const auto entry7_high = pool.GetIter(tx7->GetHash()).value();
    148+    const auto entry8_high = pool.GetIter(tx8->GetHash()).value();
    149+    const auto entry9_unchained = pool.GetIter(tx9->GetHash()).value();
    150+    const auto entry10_unchained = pool.GetIter(tx10->GetHash()).value();
    151+    const auto entry11_unchained = pool.GetIter(tx11->GetHash()).value();
    152+    const auto entry12_unchained = pool.GetIter(tx12->GetHash()).value();
    


    glozow commented at 12:10 pm on March 19, 2024:

    e9c5aeb11d641b8cae373452339760809625021d could have been 3 commits

    • rename entry{1…8} to entry{1…8}_{normal,low,high, etc} in rbf_tests
    • unit tests for CheckConflictTopology
    • unit tests for CompareFeerateDiagram
  165. in src/test/rbf_tests.cpp:264 in e9c5aeb11d outdated
    262+    BOOST_CHECK(HasNoNewUnconfirmed(*spends_conflicting_confirmed.get(), pool, {entry1_normal, entry3_low}) == std::nullopt);
    263+
    264+    // Tests for CheckConflictTopology
    265+
    266+    // Tx4 has 23 descendants
    267+    BOOST_CHECK(pool.CheckConflictTopology(set_34_cpfp).has_value());
    


    glozow commented at 12:11 pm on March 19, 2024:
    Check the value of the string as well, i.e. “%s has 23 descendants, max 1 allowed”

    instagibbs commented at 4:13 pm on March 25, 2024:
    opened in follow-up
  166. in src/test/rbf_tests.cpp:308 in b767e6bd47 outdated
    303+    const CAmount low_fee{CENT/100};
    304+    const CAmount normal_fee{CENT/10};
    305+
    306+    // low feerate parent with normal feerate child
    307+    const auto tx1 = make_tx(/*inputs=*/ {m_coinbase_txns[0]}, /*output_values=*/ {10 * COIN});
    308+    pool.addUnchecked(entry.Fee(low_fee).FromTx(tx1));
    


    glozow commented at 12:25 pm on March 19, 2024:
    b767e6bd47cb0fb8f7aea3fb10c597e59a35bf74 This is perhaps the place to add coverage for the fact that modified feerate is used, e.g. PrioritiseTransaction here and check that using a replacement_fee between modified and base makes a difference in the result.

    instagibbs commented at 4:15 pm on March 25, 2024:
    opened in follow-up
  167. in src/test/fuzz/rbf.cpp:124 in 588a98dccc outdated
    119+        assert(iter <= g_outpoints.size());
    120+        parent->vin.resize(1);
    121+        parent->vin[0].prevout = g_outpoints[iter++];
    122+
    123+        mempool_txs.emplace_back(*parent);
    124+        pool.addUnchecked(ConsumeTxMemPoolEntry(fuzzed_data_provider, mempool_txs.back()));
    


    glozow commented at 12:27 pm on March 19, 2024:

    Similar to previous comment, could do a random PrioritiseTransaction here

    0if (fuzzed_data_provider.ConsumeBool()) {
    1    pool.PrioritiseTransaction(mempool_txs.back().GetHash().ToUint256(), fuzzed_data_provider.ConsumeIntegralInRange<int32_t>(-100000, 100000));
    2}
    

    instagibbs commented at 4:15 pm on March 25, 2024:
    opened in follow-up
  168. in src/test/rbf_tests.cpp:369 in 7295986778
    364+    BOOST_CHECK(old_diagram.size() == 2);
    365+    BOOST_CHECK(new_diagram.size() == 2);
    366+    BOOST_CHECK(old_diagram[0] == FeeFrac(0, 0));
    367+    BOOST_CHECK(old_diagram[1] == FeeFrac(low_fee, low_size));
    368+    BOOST_CHECK(new_diagram[0] == FeeFrac(0, 0));
    369+    BOOST_CHECK(new_diagram[1] == FeeFrac(0, 1));
    


    glozow commented at 12:53 pm on March 19, 2024:

    72959867784098137a50c34f86deca8235eef4f8 Nit, but this old_diagram, new_diagram assignment setup is

    • making a copies of everything
    • brittle / easy for the tests to bleed into each other, e.g. if you forgot to reset the variables (and, previously, if you forgot to clear them between tests)
    • repetitive
    • checking size + each element of a vector when you could just compare it to an expected vector

    Something cleaner might e.g. look like:

    0{
    1    const auto replace_one{pool.CalculateFeerateDiagramsForRBF(/*replacement_fees=*/0, /*replacement_vsize=*/1, {entry_low}, {entry_low})};
    2    BOOST_CHECK(replace_one.has_value());
    3    std::vector<FeeFrac> expected_old_diagram{FeeFrac(0, 0), FeeFrac(low_fee, low_size)};
    4    BOOST_CHECK(replace_one->first == expected_old_diagram);
    5    std::vector<FeeFrac> expected_new_diagram{FeeFrac(0, 0), FeeFrac(0, 1)};
    6    BOOST_CHECK(replace_one->second == expected_new_diagram);
    7}
    

    instagibbs commented at 4:15 pm on March 25, 2024:
    opened in follow-up
  169. in src/test/rbf_tests.cpp:329 in b767e6bd47 outdated
    324+    BOOST_CHECK(res1.value().first == DiagramCheckError::FAILURE);
    325+    BOOST_CHECK(res1.value().second == "insufficient feerate: does not improve feerate diagram");
    326+
    327+    // With one more satoshi it does
    328+    BOOST_CHECK(ImprovesFeerateDiagram(pool, {entry1}, {entry1, entry2}, tx1_fee + tx2_fee + 1, tx1_size + tx2_size) == std::nullopt);
    329+
    


    glozow commented at 2:12 pm on March 19, 2024:
    0    // With one less vB it does
    1    BOOST_CHECK(ImprovesFeerateDiagram(pool, {entry1}, {entry1, entry2}, tx1_fee + tx2_fee, tx1_size + tx2_size - 1) == std::nullopt);
    

    instagibbs commented at 4:15 pm on March 25, 2024:
    opened in follow-up
  170. in src/test/fuzz/rbf.cpp:177 in 588a98dccc outdated
    172+        assert(calc_results->first.back().size - replaced_size + replacement_vsize == calc_results->second.back().size);
    173+    }
    174+
    175+    // If internals report error, wrapper should too
    176+    auto err_tuple{ImprovesFeerateDiagram(pool, direct_conflicts, all_conflicts, replacement_fees, replacement_vsize)};
    177+    if (!calc_results.has_value()) assert(err_tuple.has_value());
    


    glozow commented at 2:40 pm on March 19, 2024:
    I think we can be a bit more specific than this, i.e. if !calc_results.has_value() then err_tuple.value().first == DiagramCheckError::UNCALCULABLE

    instagibbs commented at 4:14 pm on March 25, 2024:
    opened in follow-up
  171. in src/test/fuzz/rbf.cpp:172 in 588a98dccc outdated
    167+            replaced_size += txiter->GetTxSize();
    168+        }
    169+        // The total fee of the new diagram should be the total fee of the old
    170+        // diagram - replaced_fee + replacement_fees
    171+        assert(calc_results->first.back().fee - replaced_fee + replacement_fees == calc_results->second.back().fee);
    172+        assert(calc_results->first.back().size - replaced_size + replacement_vsize == calc_results->second.back().size);
    


    glozow commented at 2:50 pm on March 19, 2024:

    This test for ImprovesFeerateDiagram doesn’t seem very strict beyond checking NEW = OLD - CON + replacement and that internal errors = wrapper errors. For example, if I swap !is_gt for a !is_eq in ImprovesFeerateDiagram, this fuzzer is still happy with everything. There is coverage for that in other places, but it seems like we could be writing stronger assertions?

    Can we perhaps add a check that the result is consistent with what CompareFeerateDiagram says when comparing the diagrams returned in calc_results? Or some kind of sanity check that when the new diagram ends with lower fees, we should be getting DiagramCheckError::FAILURE?


    instagibbs commented at 4:14 pm on March 25, 2024:
    I added a couple more checks where I could clearly add them in followup
  172. glozow commented at 2:57 pm on March 19, 2024: member

    code review ACK 72959867784098137a50c34f86deca8235eef4f8

    Various non-blocking comments about the tests, can be saved for a followup.

  173. instagibbs commented at 3:36 pm on March 19, 2024: member

    Thanks for the review!

    As long as it’s nits only, I will compile a list of :+1: I will accomplish on a follow-up PR.

  174. in src/util/feefrac.h:63 in ce8e22542e outdated
    58+    }
    59+#else
    60+    static constexpr auto Mul = MulFallback;
    61+#endif
    62+
    63+    int64_t fee;
    


    murchandamus commented at 6:28 pm on March 19, 2024:

    In “Add FeeFrac utils” (ce8e22542ed0b4fa5794d3203207146418d59473):

    Is there a special reason why this is an int64_t instead of a CAmount?


    sipa commented at 7:59 pm on March 19, 2024:
    I’ll answer because this is my code: I consider this class to be a low-level utility class for representing monetary fractions, without needing a dependency on consensus code.
  175. in src/txmempool.cpp:1335 in 2079b80854 outdated
    1330+        } else {
    1331+            old_chunks.emplace_back(individual);
    1332+        }
    1333+    }
    1334+
    1335+    // No topology restrictions post-chunking; sort
    


    murchandamus commented at 6:51 pm on March 19, 2024:

    In “Implement ImprovesFeerateDiagram” (2079b80854e2595f6f696e7c13a56c7f2a7da9f4):

    I’m not sure I understand why there are “No topology restrictions post-chunking”. I am guessing that this is alluding to all affected clusters at most having two transactions and therefore each having only two possible configurations: child has higher feerate and the cluster is one chunk, or child has lower feerate and the cluster has two chunks, but the chunk with the child naturally sorts after the chunk with the parent. If that’s what is being alluded to here, perhaps you could elaborate in the comment if you touch this again.

    Also, I have one open question here. It is my understanding that we do not combine two transactions into one chunk if they tie in feerate. If we have a parent and a child that have the same feerate, and the child is smaller, would that not mean that the chunk with the child could be sorted in front of the chunk with the parent incorrectly?


    Thinking more about this, I think this does not matter here, because even if the child were sorted in front of the parent, given them having the same feerate, it would result in the same feerate diagram.

    Also, it seems to me that this problem could be sidestepped by chunking together a parent and a child on equal feerate for the purpose of the diagram, if you used if (individual >= package) { in line 1320 instead of if (individual > package) {.


    sipa commented at 7:49 pm on March 19, 2024:
    It’d need to be if (!(individual << package)) {, because >= and > use size as tie breaker when comparing FeeFracs. I think that’s generally an improvement, actually, because size shouldn’t matter here.

    instagibbs commented at 8:32 pm on March 19, 2024:

    If that’s what is being alluded to here, perhaps you could elaborate in the comment if you touch this again.

    Will elaborate a bit on the chunk construction comments in a follow-up.

    I think that’s generally an improvement, actually, because size shouldn’t matter here.

    Agreed, at a minimum I’d change to individual >> package to get rid of the tie-breaking behavior which is confusing to have here. Will take on a chosen change in follow-up.


    instagibbs commented at 4:14 pm on March 25, 2024:
    opened in follow-up with >> as noted
  176. in src/util/feefrac.h:64 in ce8e22542e outdated
    59+#else
    60+    static constexpr auto Mul = MulFallback;
    61+#endif
    62+
    63+    int64_t fee;
    64+    int32_t size;
    


    murchandamus commented at 7:04 pm on March 19, 2024:

    In “Add FeeFrac utils” (ce8e22542ed0b4fa5794d3203207146418d59473):

    Nit: I noticed that in CalculateFeerateDiagramsForRBF(…) you explicitly use vsize, but here you use size. Perhaps vsize should be used consistently? Using vsize for the FeeFrac member would also have the added benefit that the member name doesn’t collide with the use of size referring to the count of objects in the diagram.

    0    int32_t vsize;
    

    sipa commented at 8:00 pm on March 19, 2024:

    Hmm, I was hoping that this class could be used in contexts where the size is WU rather than vsize too. I do agree with the confusion w.r.t. the size() member functions.

    Should they perhaps just be called numerator and denominator?


    instagibbs commented at 8:32 pm on March 19, 2024:

    Should they perhaps just be called numerator and denominator?

    I’ll mull this over a bit


    murchandamus commented at 8:33 pm on March 19, 2024:
    If it were weight that would also differentiate well.

    instagibbs commented at 8:38 pm on March 19, 2024:
    I think that would be even more confusing; if we’re going to change it we need it to be unit-neutral (punting for this PR)
  177. in src/test/fuzz/feeratediagram.cpp:29 in 4d6528a3d6 outdated
    24+{
    25+    assert(diagram.size() > 0);
    26+    unsigned not_above = 0;
    27+    unsigned not_below = diagram.size() - 1;
    28+    // If outside the range of diagram, extend begin/end.
    29+    if (size < diagram[not_above].size) return {diagram[not_above].fee, 1};
    


    murchandamus commented at 7:15 pm on March 19, 2024:

    In “fuzz: fuzz diagram creation and comparison” (4d6528a3d6bf3821c216c68f99170e2faab5d63c):

    Nit: The use of vsize throughout would help differentiate the mixed occurrences of diagram.size() and diagram[i].size


    instagibbs commented at 4:14 pm on March 25, 2024:
    leaving as is
  178. in src/test/rbf_tests.cpp:448 in 7295986778
    443+    BOOST_CHECK(old_diagram[1] == FeeFrac(high_fee, high_size_2));
    444+    BOOST_CHECK(old_diagram[2] == FeeFrac(low_fee + high_fee, low_size_2 + high_size_2));
    445+    BOOST_CHECK(new_diagram[0] == FeeFrac(0, 0));
    446+    BOOST_CHECK(new_diagram[1] == FeeFrac(high_fee, low_size_2));
    447+
    448+    // You can have more than two direct conflicts if the there are multiple effected clusters, all of size 2 or less
    


    murchandamus commented at 7:26 pm on March 19, 2024:

    Nit: In “Unit tests for CalculateFeerateDiagramsForRBF” (72959867784098137a50c34f86deca8235eef4f8):

    0    // You can have more than two direct conflicts if the there are multiple affected clusters, all of size 2 or less
    

    instagibbs commented at 4:14 pm on March 25, 2024:
    opened in follow-up
  179. in src/test/rbf_tests.cpp:350 in 7295986778
    345+
    346+    const CAmount low_fee{CENT/100};
    347+    const CAmount normal_fee{CENT/10};
    348+    const CAmount high_fee{CENT};
    349+
    350+    // low -> high -> medium fee transactions that would result in two chunks together
    


    murchandamus commented at 7:40 pm on March 19, 2024:

    In “Unit tests for CalculateFeerateDiagramsForRBF” (72959867784098137a50c34f86deca8235eef4f8):

    Nit: this comment appears to make the unstated assumption that the three transactions are of similar vsize


    instagibbs commented at 4:14 pm on March 25, 2024:
    opened in follow-up
  180. murchandamus commented at 7:42 pm on March 19, 2024: contributor

    utACK 72959867784098137a50c34f86deca8235eef4f8

    I also only have non-blocking nits

  181. in src/policy/rbf.cpp:194 in 2079b80854 outdated
    189+                                                const CTxMemPool::setEntries& all_conflicts,
    190+                                                CAmount replacement_fees,
    191+                                                int64_t replacement_vsize)
    192+{
    193+    // Require that the replacement strictly improve the mempool's feerate diagram.
    194+    std::vector<FeeFrac> old_diagram, new_diagram;
    


    willcl-ark commented at 11:33 am on March 20, 2024:

    In “Implement ImprovesFeerateDiagram”: 2079b80854e2595f6f696e7c13a56c7f2a7da9f4

    0    std::vector<FeeFrac> old_diagram, new_diagram;
    

    These vectors at src/policy/rbf.cpp:L194 appear to be unused and the diagrams are built inside of CalculateFeerateDiagramsForRBF.


    instagibbs commented at 12:05 pm on March 25, 2024:
    leftovers, will remove on touchup/followup

    instagibbs commented at 4:14 pm on March 25, 2024:
    opened in follow-up
  182. willcl-ark approved
  183. willcl-ark commented at 1:43 pm on March 20, 2024: contributor

    crACK 72959867784098137a50c34f86deca8235eef4f8

    This looks good on a first pass of the code, and was a good introduction (to me) for how we are approaching implementation of the Diagrammatic fee rate checks, which matches my current understanding of the proposed concept.

    Left only two small non-blocking nits.

    In addition, while checking each commit I did notice that clang-format could be run on most (all?) of them too if wanted (but I’d guess it’s not wanted at this stage of review 😄 ).

  184. achow101 assigned glozow on Mar 20, 2024
  185. instagibbs commented at 4:14 pm on March 25, 2024: member

    I see you’ve added coverage for the “not the only parent” case, but I don’t see a test for “not the only child” case. Also nothing failed when I commented it out. @glozow can’t figure out github interface, I added a test in followup

  186. in src/util/feefrac.cpp:80 in ce8e22542e outdated
    75+        // If P lies above AB, unproc_side is better in P. If P lies below AB, then !unproc_side is
    76+        // better in P.
    77+        if (std::is_gt(cmp)) better_somewhere[unproc_side] = true;
    78+        if (std::is_lt(cmp)) better_somewhere[!unproc_side] = true;
    79+        ++next_index[unproc_side];
    80+    } while(true);
    


    sdaftuar commented at 7:24 pm on March 25, 2024:

    Just noticed that I think we could change this to short circuit in the case where we’ve detected that both diagrams are “better somewhere” and hence incomparable, eg:

    0} while (!(better_somewhere[0] && better_somewhere[1]));
    

    instagibbs commented at 12:40 pm on March 26, 2024:

    I think slightly better is to pull the check a few lines below into the loop, just after updating the better_somewhere array:

    0if (better_somewhere[0] && better_somewhere[1]) return std::partial_ordering::unordered;
    

    ?


  187. in src/txmempool.cpp:1299 in 2079b80854 outdated
    1294+    // direct_conflicts that is at its own fee/size, along with the replacement
    1295+    // tx/package at its own fee/size
    1296+
    1297+    // old diagram will consist of each element of all_conflicts either at
    1298+    // its own feerate (followed by any descendant at its own feerate) or as a
    1299+    // single chunk at its descendant's ancestor feerate.
    


    sdaftuar commented at 7:48 pm on March 25, 2024:

    I think this comment is wrong – it should say something like:

    0// old diagram will consist of the ancestors and descendants of each element of 
    1// all_conflicts.  every such transaction will either be at its own feerate (followed 
    2// by any descendant at its own feerate), or as a single chunk at the descendant's 
    3// ancestor feerate.
    

    instagibbs commented at 12:42 pm on March 26, 2024:
    taken, indeed my definition missed the ancestors
  188. sdaftuar approved
  189. sdaftuar commented at 7:54 pm on March 25, 2024: member

    ACK 72959867784098137a50c34f86deca8235eef4f8

    Left some nits that can be addressed (or not) in a followup.

  190. glozow merged this on Mar 26, 2024
  191. glozow closed this on Mar 26, 2024

  192. glozow unassigned glozow on Mar 26, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-01 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me