validation: re-delegate absurd fee checking from mempool to clients #19339

pull glozow wants to merge 3 commits into bitcoin:master from glozow:mempool-remove-absurdfee changing 15 files +63 −47
  1. glozow commented at 7:34 pm on June 20, 2020: member

    Picked up from #15810. Add separate fee-checking logic for clients that need to enforce max fee rates, then remove the absurdFee logic from ATMP.

    ATMP’s nAbsurdFee argument is used to enforce user-specific behavior (it is not policy since it isn’t applied consistently: it is only ever used in RPC and wallet, and set to 0 everywhere else internally). It should be removed from AcceptToMemoryPool because (1) validation results/mempool behavior should not be user-specific and (2) enforcing a max fee rate should be the responsibility of the client instead of the mempool.

    Note: this PR does not intend to remove protection from high fees, just re-delegate the responsibility to clients.

  2. luke-jr commented at 7:39 pm on June 20, 2020: member
    Absurd fees aren’t even close to the only policy being enforced in ATMP… It doesn’t make sense to split just this one out.
  3. glozow commented at 7:42 pm on June 20, 2020: member

    Absurd fees aren’t even close to the only policy being enforced in ATMP… It doesn’t make sense to split just this one out. @luke-jr Sorry 😅 I’m a beginner so I don’t know very much. Do you mean that there are other things that should be taken out too, or that it’s not worth doing so at all?

  4. luke-jr commented at 7:45 pm on June 20, 2020: member
    It might be worth doing, but I’m not sure if the costs outweigh the benefits, and if/when done, it should be splitting all policy from all consensus checks, not just one policy from the rest.
  5. luke-jr commented at 7:50 pm on June 20, 2020: member

    Check out https://github.com/bitcoin/bitcoin/compare/0.20...luke-jr:sendraw_force-0.20+knots#diff-24efdb00bfbe56b140fb006b562cc70bR578

    This changes all the policy checks to conditionals, while leaving the non-policy checks as strictly enforced. So a split would be on the same criteria. Note that reordering stuff may not always be a good idea, however…

  6. DrahtBot added the label GUI on Jun 20, 2020
  7. DrahtBot added the label P2P on Jun 20, 2020
  8. DrahtBot added the label RPC/REST/ZMQ on Jun 20, 2020
  9. DrahtBot added the label Tests on Jun 20, 2020
  10. DrahtBot added the label Validation on Jun 20, 2020
  11. DrahtBot added the label Wallet on Jun 20, 2020
  12. DrahtBot commented at 7:54 pm on June 20, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #19753 (p2p: don’t add AlreadyHave transactions to recentRejects by troygiorshev)
    • #19645 (Allow updating mempool-txn with cheaper witnesses by ariard)
    • #16546 (External signer support - Wallet Box edition by Sjors)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  13. glozow commented at 7:57 pm on June 20, 2020: member
    @luke-jr okay, I see what you’re saying. Agree that refactoring consensus/policy should be as a whole + may or may not be worth it. I could be completely mistaken, but I thought that this was more… a user’s preference rather than node’s fee policy, since it’s pretty much only used in RPC and wallet. It seems to be set to 0 in other places?
  14. JeremyRubin commented at 8:59 pm on June 20, 2020: contributor

    I think you’re correct in the abstract on this one @gzhao408, but concretely this behavior is probably desirable as it’s a belt-and-suspenders check of preventing things getting into the mempool that are almost certainly a mistake.

    Even though mining/accepting these things should be done, and our node shouldn’t really do ‘deep transaction inspection’, the case of absurd fees is pretty clearly accidental, so we add this redundant check (which can be disabled afaict via allowhighfees) even though clients should ultimately be responsible for this and not rely on absurd fees being rejected.

    I think if you wanted to move the logic somewhere else (e.g., to just RPC layer) you could do that too, but the most convenient place for that logic is in ATMP as we load the required context to check fees there already.

    Client side absurd fee problems should generally improve post-taproot & with the segwit fixes as signers should be more aware of how much fees their txn is spending as they can be made aware of all the inputs values they are signing for.

  15. luke-jr commented at 9:21 pm on June 20, 2020: member
    @gzhao408 You may be right - it’s a good point that we don’t enforce it on other stuff. Maybe we should really be checking it before even calling ATMP. @JeremyRubin This is needed for GUI transactions too, at least.
  16. JeremyRubin commented at 9:31 pm on June 20, 2020: contributor
    One approach might be to introduce a redundant layer of logic to all the places we actually want to perform the absurdfee check, and then in a follow up PR (or commit if you want to do it in one), remove the now redundant ATMP check. I agree that ATMP is later than ideal for checking this property.
  17. achow101 commented at 10:08 pm on June 20, 2020: member
    I agree with removing the absurd fee check from ATMP, but we should still have it somewhere instead of removing it completely. Since it only is used for transactions sent by the node, the ideal place to move this to would be BroadcastTransaction in src/node/transaction.cpp. The RPC and wallet (GUI goes through the wallet) both use BroadcastTransaction.
  18. jnewbery commented at 11:08 pm on June 22, 2020: member
    Concept ACK! This isn’t policy, it’s wallet/RPC behaviour (and as such, belongs in the wallet/RPC).
  19. NicolasDorier commented at 0:49 am on June 23, 2020: contributor
    Concept ACK
  20. glozow renamed this:
    [WIP] mempool: Remove absurd fee logic from ATMP
    [WIP] mempool: re-delegate absurd fee checking from mempool to clients
    on Jul 1, 2020
  21. ariard commented at 3:11 pm on July 24, 2020: member

    Concept ACK,

    This check should be part of a consistent wallet tx-sanitization policy, i.e a set of checks protecting users funds against harmful actions before authorizing spend. Some signers already do this like Coincard. We could envision Core implementing the same kind of logic behind ScriptPubKeyMan::SignTransaction, with user settings. Now do this policy should apply to transaction flowing through our node but not originating from our wallet (sendrawtransaction) ? That’s harder to decide on because in some cases an absurd fee may have a game-theory rational (a LN justice transaction burning everything in fees before CSV delay expiration).

    Overall, absurd fee protection is already enforced on its own in CreateTransaction (grep m_default_max_tx_fee) but blindly removing from mempool checks would be a behavior change as it won’t be applied anymore to sendrawtransaction/GUI transactions. So going forward with this PR check should be moved inside BroadcastTransaction to conserve behavior IMO.

    With regards to untangling policy/consensus checks I think that should be a distinct conversation, order of them matter for anti-DoS reasons.

  22. glozow force-pushed on Jul 25, 2020
  23. glozow commented at 8:48 pm on July 25, 2020: member

    This builds on #19093. If that one is merged first then I’ll rebase this; if this one gets more review attention, @rajarshimaitra is listed as the author of the base commit here so hopefully that’s ok.

    Per y’all’s wonderful advice, I moved the absurd fee checking into BroadcastTransaction and as an extra step in testmempoolaccept. Calling this a TransactionError::MAX_FEE_EXCEEDED seemed more appropriate, since that’s what the wallet uses. I still don’t think it’s policy since it’s empty for transactions not originating from the node’s own wallet. Whether it should be policy would a separate and interesting discussion 👀 .

    There’s a couple intermediate stages to (1) test_accept and check fee before submitting (https://github.com/bitcoin/bitcoin/pull/19339/commits/7dc55a5f4c2bc3ba3c301338ce75b25412365dd7) and (2) keep absurd fee but ignore it in validation (https://github.com/bitcoin/bitcoin/pull/19339/commits/40703b1d5185cd26626d9da1fb76c3babdcdd550) to show that the behavior is the same.

  24. glozow force-pushed on Jul 25, 2020
  25. glozow marked this as ready for review on Jul 26, 2020
  26. glozow renamed this:
    [WIP] mempool: re-delegate absurd fee checking from mempool to clients
    validation: re-delegate absurd fee checking from mempool to clients
    on Jul 26, 2020
  27. jnewbery commented at 10:58 am on July 27, 2020: member
    Approach ACK. This looks great. Thanks Gloria!
  28. jnewbery removed the label GUI on Aug 15, 2020
  29. jnewbery removed the label P2P on Aug 15, 2020
  30. jnewbery added the label Review club on Aug 15, 2020
  31. glozow force-pushed on Aug 15, 2020
  32. glozow commented at 4:51 pm on August 16, 2020: member
    Rebased and refactored some of the commits to make it cleaner + easier to review in preparation for PR Review Club (Notes). Now’s a perfect time to review if anyone has time :) thanks everybody!
  33. in src/node/transaction.cpp:58 in 67bb7148ad outdated
    45+        if (test_accepted) {
    46+            if (max_tx_fee && fee > max_tx_fee) {
    47+                return TransactionError::MAX_FEE_EXCEEDED;
    48+            }
    49+        }
    50+        // Transaction fee is acceptable. Submit the transaction.
    


    pinheadmz commented at 6:19 pm on August 19, 2020:
    Instead of calling ATMP twice, can we trust the test_accepted value from the first call and skip redundant checks in the second call? Or maybe to put it another way, if all we need is the fee here, can we compute that quickly and test it against max_tx_fee before checking anything else about the tx? I think if that’s easy enough to do, you don’t have to add an argument to ATMP. But I guess you’d still have to remove nAbsurdFee everywhere it’s called.

    glozow commented at 1:08 pm on August 22, 2020:

    Instead of calling ATMP twice, can we trust the test_accepted value from the first call and skip redundant checks in the second call?

    From what I understand, we do this somewhat (i.e. for expensive parts of validation) - the signature cache and script cache are filled in PolicyScriptChecks and ConsensusScriptChecks during the test accept and thus those parts don’t need to be redone the second time around.

    if all we need is the fee here, can we compute that quickly and test it against max_tx_fee before checking anything else about the tx?

    I definitely thought about doing it this way! Seemed simple enough, but without any tx sanitization (PreChecks) or state to go off of, it started getting really complicated 😅 . Test-accepting it through ATMP did exactly what I needed, and we kind of get to do it for… I don’t want to say “free,” but it’s way cheaper than 2x work.


    glozow commented at 1:53 pm on August 22, 2020:

    I’ve made a branch that adds logging to the signature and script caches. Running the truncated version of rpc_rawtransaction.py shows how much reuse we get. Here’s a sample of a sendrawtransaction which does the 2 ATMPs:

     0 node0 2020-08-22T13:50:00.085822Z [httpworker.1] @@@ In BroadcastTransaction() 
     1 node0 2020-08-22T13:50:00.085883Z [httpworker.1] @@@ Calling ATMP with testaccept=1 ! 
     2 node0 2020-08-22T13:50:00.086011Z [httpworker.1] @@@ Calling PolicyScriptChecks! 
     3 node0 2020-08-22T13:50:00.086049Z [httpworker.1] @@@ Verifying Signature 
     4 node0 2020-08-22T13:50:00.086674Z [httpworker.1] @@@ Adding entry to Signature Cache! 
     5 node0 2020-08-22T13:50:00.086740Z [httpworker.1] @@@ Calling ConsensusScriptChecks! 
     6 node0 2020-08-22T13:50:00.086785Z [httpworker.1] @@@ Verifying Signature 
     7 node0 2020-08-22T13:50:00.086830Z [httpworker.1] @@@ Signature Cache already filled! 
     8 node0 2020-08-22T13:50:00.086856Z [httpworker.1] @@@ Adding entry to Script Execution cache! 
     9 node0 2020-08-22T13:50:00.086890Z [httpworker.1] @@@ Accepted, checking max fee 
    10 node0 2020-08-22T13:50:00.086917Z [httpworker.1] @@@ Calling ATMP with testaccept=0 ! 
    11 node0 2020-08-22T13:50:00.086992Z [httpworker.1] @@@ Calling PolicyScriptChecks! 
    12 node0 2020-08-22T13:50:00.087023Z [httpworker.1] @@@ Verifying Signature 
    13 node0 2020-08-22T13:50:00.087067Z [httpworker.1] @@@ Signature Cache already filled! 
    14 node0 2020-08-22T13:50:00.087094Z [httpworker.1] @@@ Calling ConsensusScriptChecks! 
    15 node0 2020-08-22T13:50:00.087119Z [httpworker.1] @@@ Script Execution cache is already filled! 
    

    LarryRuane commented at 4:55 pm on September 22, 2020:

    Nonblocking suggestion: You could simplify the code branches and reduce nesting:

     0        if (max_tx_fee) {
     1            CAmount fee{0};
     2            if (!GetTransactionFee(*node.mempool, state, tx, &fee)) {
     3                err_string = state.ToString();
     4                if (!state.IsInvalid()) {
     5                    return TransactionError::MEMPOOL_ERROR;
     6                }
     7                if (state.GetResult() == TxValidationResult::TX_MISSING_INPUTS) {
     8                    return TransactionError::MISSING_INPUTS;
     9                }
    10                return TransactionError::MEMPOOL_REJECTED;
    11            }
    12            if (fee > max_tx_fee) {
    13                return TransactionError::MAX_FEE_EXCEEDED;
    14            }
    15        }
    
  34. in doc/release-notes-19093.md:1 in 67bb7148ad outdated
    0@@ -0,0 +1,7 @@
    1+RPC changes
    


    pinheadmz commented at 6:48 pm on August 19, 2020:
    Should the filename have the PR number? (19339) I didn’t know about this convention yet, just looked it up: https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#release-notes

    glozow commented at 1:10 pm on August 22, 2020:
  35. pinheadmz commented at 6:49 pm on August 19, 2020: member
    Concept ACK! Makes a lot of sense. Builds on OSX 10.14, unit tests passed but I had one functional test tool_wallet not pass every time (not sure why, investigating…)
  36. robot-dreams commented at 6:12 pm on August 22, 2020: contributor

    Concept ACK, Approach ACK, unit tests + all updated functional tests pass locally

    I think the discussion of “what are the performance implications of calling ATMP twice” is very interesting. I looked up where BroadcastTransaction could be called, and here’s what I found:

    • sendrawtransaction RPC
    • SubmitMemoryPoolAndRelay in the wallet
    • PSBTOperationsDialog in the UI

    In particular, I’m guessing BroadcastTransaction calls aren’t too frequent (e.g. compared to relaying transactions in the P2P network), so the overhead of calling ATMP twice (especially since @gzhao408 showed that many of the verification results are cached) is OK.

    Follow-ups for review club discussion:

    What would be the tradeoffs / considerations involved with either of these alternative approaches?

    1. Replace the nAbsurdFee parameter with some sort of generic clientSafetyChecks object that ATMP can call without knowing the client-specific details

    2. Split up ATMP into two parts: (i) TestForMemoryPool performs all checks and returns a fee together with some context, (ii) AddToMemoryPool takes the context and finishes the process. This way, we can remove the test_accept parameter. The testmempoolaccept RPC could just call TestForMemoryPool, and then sendrawtransaction could do something like this:

    0context = TestForMemoryPool(...);
    1if (max_tx_fee && context.fee > max_tx_fee) {
    2	return TransactionError::MAX_FEE_EXCEEDED;
    3}
    4
    5if (!AddToMemoryPool(*node.mempool, context, ...)) {
    6	// return error result
    7}
    

    How will this change interact with taproot?

    Given the “Client side absurd fee problems should generally improve post-taproot” comment above, will this code need to change again soon, or will “post-taproot” have multiple code paths, of which this is one of them?

  37. glozow commented at 7:11 pm on August 23, 2020: member

    Thanks for the review @robot-dreams :)

    In particular, I’m guessing BroadcastTransaction calls aren’t too frequent (e.g. compared to relaying transactions in the P2P network)

    Nice observation; I’d like to highlight that BroadcastTransaction is used for transactions coming from clients as opposed to peers. Even if the slowdown were significant, it’s very different from an inefficiency that may be exploited by random network peers.

    Replace the nAbsurdFee parameter with some sort of generic clientSafetyChecks object that ATMP can call without knowing the client-specific details

    The idea here is that we want to separate client-specific details from ATMP. In general, the validation result returned by ATMP shouldn’t depend on which client submitted it (my opinion).

    Split up ATMP into two parts: (i) TestForMemoryPool performs all checks and returns a fee together with some context, (ii) AddToMemoryPool takes the context and finishes the process.

    I’m not quite sure I fully understand this idea, but it seems pretty similar to what happens right now in ATMP. Most of it can be seen in AcceptSingleTransaction which has 4 components: the 3 *Checks (validation and fill in the fee), and Finalize (try to add it to the mempool). A test-accept returns before Finalize; a “real” accept runs through everything (and uses what’s cached if possible). Other than repeating checks (which is the safe thing to do), this seems equivalent to what you’re suggesting.

  38. in src/node/transaction.cpp:42 in 67bb7148ad outdated
    35@@ -36,10 +36,19 @@ TransactionError BroadcastTransaction(NodeContext& node, const CTransactionRef t
    36         if (!existingCoin.IsSpent()) return TransactionError::ALREADY_IN_CHAIN;
    37     }
    38     if (!node.mempool->exists(hashTx)) {
    39-        // Transaction is not already in the mempool. Submit it.
    40+        // Transaction is not already in the mempool.
    41         TxValidationState state;
    42+        CAmount fee;
    43+        bool test_accepted = AcceptToMemoryPool(*node.mempool, state, std::move(tx),
    


    jnewbery commented at 11:57 am on August 24, 2020:

    Both this ATMP call and the one below should have their std::move(tx) argument changed to tx. In general, you should never use a variable after it’s been used with std::move in a function call, since it might have been moved away. In this case, if tx has been moved from, it’ll be null, and any attempt to dereference it will segfault.

    However, because ATMP takes a const reference to CTransactionRef, it can’t be moved away from, so if ATMP wants to retain shared ownership of the transaction, it must copy rather than move to the new shared_ptr. That means that technically it is safe to use tx after this call.

    Let’s just make it explicit though: ATMP takes a const ref and has to copy if it wants to retain shared ownership, so just pass it tx.

    See R30, R34, R35 and R36 here: http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#r30-take-smart-pointers-as-parameters-only-to-explicitly-express-lifetime-semantics for more background on passing raw/smart pointers.


    robot-dreams commented at 3:18 pm on August 24, 2020:

    Thanks for noting this and posting the link, @jnewbery ! After looking into this, I have two follow-up questions for you if that’s OK:

    Question 1 (historical context)

    Previously, this call always used std::move(tx) because tx did not appear later in the function. However, use of tx after move appeared in c7eb6b4f1fe5bd76388a023529977674534334a7, which was merged as part of #18044. Do you happen to have any ideas about why it wasn’t a concern then?

    Question 2 (C++ guidelines)

    The linked guidelines mention the const shared_ptr<widget>& case, but not the const shared_ptr<widget> case that’s used in BroadcastTransaction:

    Do you know if we should think of the const shared_ptr<widget> case any differently?


    jnewbery commented at 4:57 pm on August 24, 2020:

    However, use of tx after move appeared in c7eb6b4, which was merged as part of #18044. Do you happen to have any ideas about why it wasn’t a concern then?

    Because it was missed in review. In this case, it’s not going to cause any issues because tx isn’t actually moved from, but we should still never be doing anything with a variable after it’s had std::move used on it.

    Do you know if we should think of the const shared_ptr case any differently?

    I don’t think there’s any good reason to pass a shared_ptr by const value. I may be mistaken here, but I think that means you need to increment the refcount when you enter the function (because the parameter is copy-constructed from the argument), and if you want to retain the refcount, you need to increment it again (because you can’t move from the const parameter). Move semantics with smart pointers is a deep rabbit hole. There’s lots more here: https://herbsutter.com/2013/06/05/gotw-91-solution-smart-pointer-parameters/ and http://scottmeyers.blogspot.com/2014/07/should-move-only-types-ever-be-passed.html.


    glozow commented at 2:39 pm on August 25, 2020:
    done! thanks for the explanation @jnewbery :)
  39. jnewbery commented at 12:04 pm on August 24, 2020: member
    In the last commit log, there’s no need to state which commit hash the commit was cherry-picked from.
  40. DrahtBot added the label Needs rebase on Aug 24, 2020
  41. robot-dreams commented at 3:57 pm on August 24, 2020: contributor

    @gzhao408 I did some cleanup and pushed it to https://github.com/robot-dreams/bitcoin/commits/mempool-remove-absurdfee ; feel free to grab any of the updated commits if you’d find them helpful, but of course no pressure to do so!

    • Rebase and fix merge conflict
    • Change std::move(tx) to tx
    • Change PR number in release notes
    • Change release notes about testmempoolaccept from “if transaction is accepted” to “if transaction would be accepted”
    • Fix typo in commit message (udpate -> update)
    • Remove “cherry-picked from” from commit message
  42. in test/functional/rpc_psbt.py:24 in c763343c00 outdated
    20@@ -21,6 +21,7 @@
    21 import os
    22 
    23 MAX_BIP125_RBF_SEQUENCE = 0xfffffffd
    24+MAX_FEE_EXCEEDED_MSG = 'Fee exceeds maximum configured by user (eg -maxtxfee, maxfeerate)'
    


    fjahr commented at 9:37 pm on August 24, 2020:
    If you use a global namespace constant for the error message it would make sense to share it across files as well. Otherwise local variables would make more sense to me. I think c763343c0064fe88aeef15eecd6273ccf2746fcc could also just be a clean scripted-diff without the need to introduce the constants.

    glozow commented at 2:39 pm on August 25, 2020:
    you’re totally right, I changed it to a scripted diff
  43. in src/node/transaction.cpp:42 in a872d8d31b outdated
    35@@ -36,8 +36,17 @@ TransactionError BroadcastTransaction(NodeContext& node, const CTransactionRef t
    36         if (!existingCoin.IsSpent()) return TransactionError::ALREADY_IN_CHAIN;
    37     }
    38     if (!node.mempool->exists(hashTx)) {
    39-        // Transaction is not already in the mempool. Submit it.
    40+        // Transaction is not already in the mempool.
    41         TxValidationState state;
    42+        CAmount fee;
    43+        bool test_accepted = AcceptToMemoryPool(*node.mempool, state, std::move(tx),
    


    fjahr commented at 9:49 pm on August 24, 2020:

    IMO you don’t need the intermediate return value and can just move the call into the if() but if you want to keep it:

    0        const bool test_accepted = AcceptToMemoryPool(*node.mempool, state, std::move(tx),
    

    glozow commented at 2:39 pm on August 25, 2020:
    moved it into the if(), thanks!
  44. fjahr commented at 9:58 pm on August 24, 2020: member

    Concept ACK

    Typo in c763343c0064fe88aeef15eecd6273ccf2746fcc commit message: “udpate”

    I think this would be easier to review if the renaming commit was a scripted-diff and all the other commits (except the one from #19093) would be squashed. I found this easier to review looking at the final diff than going through the commits.

  45. glozow force-pushed on Aug 25, 2020
  46. glozow force-pushed on Aug 25, 2020
  47. DrahtBot removed the label Needs rebase on Aug 25, 2020
  48. glozow commented at 4:30 pm on August 25, 2020: member

    Rebased and addressed comments. Apologies, I’m wrestling with the scripted-diff linter :'(

    I also edited the first commit so that I’m only picking the ATMP changes in #19093 (which is more RPC-focused) - these two PRs are now uncoupled.

  49. glozow force-pushed on Aug 25, 2020
  50. fanquake commented at 2:26 am on August 26, 2020: member

    Rebased and addressed comments. Apologies, I’m wrestling with the scripted-diff linter :'(

    The issue is GNU vs BSD sed. I’m assuming you are on macOS (BSD sed), and you’ve written a command that is incompatible with GNU sed (used by Travis). The fix here is to drop the ’’ after -i, which is the part that’s incompatible. Unfortunately portability when using -i isn’t trivial.

    When testing sed commands on macOS I’d suggest brew installing and using gsed.

  51. jnewbery commented at 7:24 am on August 26, 2020: member

    The issue is GNU vs BSD sed. I’m assuming you are on macOS (BSD sed), and you’ve written a command that is incompatible with GNU sed (used by Travis). The fix here is to drop the ’’ after -i, which is the part that’s incompatible. Unfortunately portability when using -i isn’t trivial.

    This issue has tripped lots of people up (including me). Perhaps commit-script-check.sh should detect if a scripted diff uses the BSD sed syntax and print a more helpful error message.

  52. in src/node/transaction.cpp:41 in 330b0ba112 outdated
    35@@ -36,10 +36,18 @@ TransactionError BroadcastTransaction(NodeContext& node, const CTransactionRef t
    36         if (!existingCoin.IsSpent()) return TransactionError::ALREADY_IN_CHAIN;
    37     }
    38     if (!node.mempool->exists(hashTx)) {
    39-        // Transaction is not already in the mempool. Submit it.
    40+        // Transaction is not already in the mempool.
    41         TxValidationState state;
    42-        if (!AcceptToMemoryPool(*node.mempool, state, std::move(tx),
    43-                nullptr /* plTxnReplaced */, false /* bypass_limits */, max_tx_fee)) {
    44+        CAmount fee;
    


    jonatack commented at 3:25 pm on August 26, 2020:

    baab966 suggestion, as CAmount is of type int64_t per src/amount.h, provide a default value

    0        CAmount fee{0};
    
  53. in src/rpc/rawtransaction.cpp:927 in 330b0ba112 outdated
    923@@ -924,10 +924,20 @@ static UniValue testmempoolaccept(const JSONRPCRequest& request)
    924 
    925     TxValidationState state;
    926     bool test_accept_res;
    927+    CAmount fee; // To return transaction fee from AcceptToMemoryPool
    


    jonatack commented at 3:26 pm on August 26, 2020:

    bcb85f4

    0    CAmount fee{0}; // To return transaction fee from AcceptToMemoryPool
    
  54. in src/util/error.cpp:33 in 330b0ba112 outdated
    29@@ -30,7 +30,7 @@ bilingual_str TransactionErrorString(const TransactionError err)
    30         case TransactionError::SIGHASH_MISMATCH:
    31             return Untranslated("Specified sighash value does not match value stored in PSBT");
    32         case TransactionError::MAX_FEE_EXCEEDED:
    33-            return Untranslated("Fee exceeds maximum configured by -maxtxfee");
    34+            return Untranslated("Fee exceeds maximum configured by user (eg -maxtxfee, maxfeerate)");
    


    jonatack commented at 3:32 pm on August 26, 2020:

    d81090e8a s/eg/e.g./ or

    0- return Untranslated("Fee exceeds maximum configured by user (eg -maxtxfee, maxfeerate)");
    1+ return Untranslated("Fee exceeds maximum configured by user in -maxtxfee or maxfeerate");
    
  55. in src/rpc/rawtransaction.cpp:961 in 330b0ba112 outdated
    935+    // Check that fee does not exceed maxfee
    936+    if (test_accept_res && max_raw_tx_fee && fee > max_raw_tx_fee) {
    937+        result_0.pushKV("allowed", false);
    938+        result_0.pushKV("reject-reason", "max-fee-exceeded");
    939+        result_0.pushKV("fee", fee);
    940+        result.push_back(std::move(result_0));
    


    jonatack commented at 3:48 pm on August 26, 2020:

    a5b33914

    • Should these keys also be returned with relevant values for the happy case as well, and not only if fee exceeds maxfee? (I suspect they should.)

    • These additions would need to be documented in the RPCHelpMan help a few lines above, as well as with an Updated RPCs release note.


    jonatack commented at 4:37 pm on August 26, 2020:

    Oops, I was confused by the early and duplicate result.push_back(std::move(result_0)) and return, which I wasn’t expecting. Perhaps use an else instead and have all the cases fall through to the single push_back and return.

    Consider harmonising the cases to both return the fee or neither, and if fee is added, then update the help + write a release note + add fee test cases.


    glozow commented at 1:06 pm on August 30, 2020:
    Actually I shouldn’t even return the fee - it was leftover from #19093. Thanks for catching that!
  56. jonatack commented at 4:05 pm on August 26, 2020: member

    Concept ACK

    Feel free to ignore these, but they came to mind while reviewing:

    • with the current structure, reviewing commit-by-commit is less clear/more laborious than reviewing the overall diff; would suggest combining some of the steps

    • consider harmonising the commit titles between e.g. [rpc] and validation:

    • if a field (fee) is added to an RPC, you’ll want to update the RPCHelpman, write a release note, and add fee test assertions.

  57. in src/validation.h:204 in 330b0ba112 outdated
    199@@ -200,10 +200,11 @@ void UnlinkPrunedFiles(const std::set<int>& setFilesToPrune);
    200 void PruneBlockFilesManual(int nManualPruneHeight);
    201 
    202 /** (try to) add transaction to memory pool
    203- * plTxnReplaced will be appended to with all transactions replaced from mempool **/
    204+ * plTxnReplaced will be appended to with all transactions replaced from mempool
    205+ * optionally takes an argument to return tx fee to the caller **/
    


    jonatack commented at 4:45 pm on August 26, 2020:

    bcb85f4 perhaps document this added argument per https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#coding-style-doxygen-compatible-comments

    0- * optionally takes an argument to return tx fee to the caller **/
    1+ * [@param](/bitcoin-bitcoin/contributor/param/)[out] fee_out   optional argument to return tx fee to the caller **/
    

    glozow commented at 1:17 pm on August 30, 2020:
    done!
  58. michaelfolkson commented at 7:50 pm on August 26, 2020: contributor

    Concept ACK, Approach ACK

    It might be worth doing, but I’m not sure if the costs outweigh the benefits, and if/when done, it should be splitting all policy from all consensus checks, not just one policy from the rest.

    As was pointed out during the Bitcoin Core PR review club session and John earlier here absurd fee checking is not policy and so splitting policy from consensus checks is not the goal in this PR.

    (Using @gzhao408 definition of policy here as “a node’s extra standards for validation in addition to consensus”)

  59. glozow force-pushed on Aug 30, 2020
  60. glozow force-pushed on Aug 30, 2020
  61. glozow commented at 5:09 pm on August 30, 2020: member

    Rebased, addressed @jonatack’s review comments, fixed the sed problem. I also squashed the commits that add the client checking so hopefully the commits are more clear. This is ready for another look!

    I’m not sure what the appveyor failure is…? It doesn’t seem related to this PR’s changes 🤔

  62. jonatack commented at 5:19 pm on August 30, 2020: member

    I’m not sure what the appveyor failure is…? It doesn’t seem related to this PR’s changes 🤔

    Yes, appveyor is down ATM, #19839 was opened today as a fix.

  63. in src/rpc/rawtransaction.cpp:962 in d0f10e25af outdated
    935+    // Check that fee does not exceed maxfee
    936+    if (test_accept_res && max_raw_tx_fee && fee > max_raw_tx_fee) {
    937+        result_0.pushKV("allowed", false);
    938+        result_0.pushKV("reject-reason", "max-fee-exceeded");
    939+        result.push_back(std::move(result_0));
    940+        return result;
    


    jonatack commented at 8:02 pm on August 31, 2020:

    ea8eb397 and 3721417 suggestion to keep the final std::move and return separate from the processing

    git diff -w

     0-    // Check that fee does not exceed maxfee
     1     if (test_accept_res && max_raw_tx_fee && fee > max_raw_tx_fee) {
     2+        // Accepted, but disallow because fee exceeds maxfee
     3         result_0.pushKV("allowed", false);
     4         result_0.pushKV("reject-reason", "max-fee-exceeded");
     5-        result.push_back(std::move(result_0));
     6-        return result;
     7-    }
     8+    } else {
     9         result_0.pushKV("allowed", test_accept_res);
    10@@ -950,6 +948,7 @@ static UniValue testmempoolaccept(const JSONRPCRequest& request)
    11                 result_0.pushKV("reject-reason", state.GetRejectReason());
    12             }
    13         }
    14+    }
    15 
    16     result.push_back(std::move(result_0));
    17     return result;
    
     0    if (test_accept_res && max_raw_tx_fee && fee > max_raw_tx_fee) {
     1        // Accepted, but disallow because fee exceeds maxfee
     2        result_0.pushKV("allowed", false);
     3        result_0.pushKV("reject-reason", "max-fee-exceeded");
     4    } else {
     5        result_0.pushKV("allowed", test_accept_res);
     6        if (!test_accept_res) {
     7            if (state.IsInvalid()) {
     8                if (state.GetResult() == TxValidationResult::TX_MISSING_INPUTS) {
     9                    result_0.pushKV("reject-reason", "missing-inputs");
    10                } else {
    11                    result_0.pushKV("reject-reason", strprintf("%s", state.GetRejectReason()));
    12                }
    13            } else {
    14                result_0.pushKV("reject-reason", state.GetRejectReason());
    15            }
    16        }
    17    }
    18
    19    result.push_back(std::move(result_0));
    20    return result;
    21}
    

    jnewbery commented at 7:49 am on September 3, 2020:
    Both seem fine. I have a preference for how it is right now to avoid the deeply nested if statements.

    glozow commented at 4:00 pm on September 3, 2020:
    Absurdfee is kind of a “special case” where test_accept_res is true, but we still say not allowed… maybe it’s good to keep it in a separate block?

    jonatack commented at 2:23 pm on September 4, 2020:

    I think (a) keep processing separate from move-and-return, which is why it is separated by an empty line at the end of the function, (b) don’t repeat yourself, and (c) the principle of least surprise (at least, I found it surprising) outweigh the also-valid (d) avoid nested conditionals, but it’s a detail that I won’t hold against anyone :smiling_face_with_three_hearts:

    OTOH, I find a bit odd the logic of test_accept_res being true but being possibly disallowed due to the maxfee. Perhaps rename it (it’s also not named like a boolean, e.g. is/has/verb_{some_state}, or mention it in the code like my doc suggestion (which could be improved). I think it’s good to optimise the overall section for clarity and coherence, even if it changes a few more lines of code.

    But these are all code style opinions, feel free to ignore if you disagree.


    LarryRuane commented at 4:11 pm on September 15, 2020:

    Another suggestion (but it’s also good as-is):

     0--- a/src/rpc/rawtransaction.cpp
     1+++ b/src/rpc/rawtransaction.cpp
     2@@ -923,34 +923,36 @@ static UniValue testmempoolaccept(const JSONRPCRequest& request)
     3     result_0.pushKV("txid", tx_hash.GetHex());
     4 
     5     TxValidationState state;
     6-    bool test_accept_res;
     7+    bool allowed;
     8     CAmount fee{0}; // To return transaction fee from AcceptToMemoryPool
     9     {
    10         LOCK(cs_main);
    11-        test_accept_res = AcceptToMemoryPool(mempool, state, std::move(tx),
    12+        allowed = AcceptToMemoryPool(mempool, state, std::move(tx),
    13             nullptr /* plTxnReplaced */, false /* bypass_limits */, /* test_accept */ true, &fee);
    14     }
    15+    std::string reject_reason;
    16 
    17     // Check that fee does not exceed maxfee
    18-    if (test_accept_res && max_raw_tx_fee && fee > max_raw_tx_fee) {
    19-        result_0.pushKV("allowed", false);
    20-        result_0.pushKV("reject-reason", "max-fee-exceeded");
    21-        result.push_back(std::move(result_0));
    22-        return result;
    23+    if (allowed) {
    24+        if (max_raw_tx_fee && fee > max_raw_tx_fee) {
    25+            allowed = false;
    26+            reject_reason = "max-fee-exceeded";
    27         }
    28-    result_0.pushKV("allowed", test_accept_res);
    29-    if (!test_accept_res) {
    30+    } else {
    31         if (state.IsInvalid()) {
    32             if (state.GetResult() == TxValidationResult::TX_MISSING_INPUTS) {
    33-                result_0.pushKV("reject-reason", "missing-inputs");
    34+                reject_reason = "missing-inputs";
    35             } else {
    36-                result_0.pushKV("reject-reason", strprintf("%s", state.GetRejectReason()));
    37+                reject_reason = strprintf("%s", state.GetRejectReason());
    38             }
    39         } else {
    40-            result_0.pushKV("reject-reason", state.GetRejectReason());
    41+            reject_reason = state.GetRejectReason();
    42         }
    43     }
    44-
    45+    result_0.pushKV("allowed", allowed);
    46+    if (!reject_reason.empty()) {
    47+        result_0.pushKV("reject-reason", reject_reason);
    48+    }
    49     result.push_back(std::move(result_0));
    50     return result;
    51 }
    
     0    bool allowed;
     1    CAmount fee{0}; // To return transaction fee from AcceptToMemoryPool
     2    {
     3        LOCK(cs_main);
     4        allowed = AcceptToMemoryPool(mempool, state, std::move(tx),
     5            nullptr /* plTxnReplaced */, false /* bypass_limits */, /* test_accept */ true, &fee);
     6    }
     7    std::string reject_reason;
     8
     9    // Check that fee does not exceed maxfee
    10    if (allowed) {
    11        if (max_raw_tx_fee && fee > max_raw_tx_fee) {
    12            allowed = false;
    13            reject_reason = "max-fee-exceeded";
    14        }
    15    } else {
    16        if (state.IsInvalid()) {
    17            if (state.GetResult() == TxValidationResult::TX_MISSING_INPUTS) {
    18                reject_reason = "missing-inputs";
    19            } else {
    20                reject_reason = strprintf("%s", state.GetRejectReason());
    21            }
    22        } else {
    23            reject_reason = state.GetRejectReason();
    24        }
    25    }
    26    result_0.pushKV("allowed", allowed);
    27    if (!reject_reason.empty()) {
    28        result_0.pushKV("reject-reason", reject_reason);
    29    }
    30    result.push_back(std::move(result_0));
    31    return result;
    
  64. DrahtBot added the label Needs rebase on Sep 2, 2020
  65. jnewbery commented at 7:48 am on September 3, 2020: member
    utACK d0f10e25afbbbcfc2c46300cf7112ce4f303987d (but needs rebase)
  66. glozow force-pushed on Sep 3, 2020
  67. glozow commented at 4:19 pm on September 3, 2020: member
    Rebased
  68. jnewbery commented at 4:57 pm on September 3, 2020: member

    utACK 6bbda0e5d4

    Verified git range-diff 4afff49265~..d0f10e25af 9ec325e2e9~..6bbda0e5d4

  69. DrahtBot removed the label Needs rebase on Sep 3, 2020
  70. in src/node/transaction.cpp:42 in 6bbda0e5d4 outdated
    40+        // Transaction is not already in the mempool.
    41         TxValidationState state;
    42-        if (!AcceptToMemoryPool(*node.mempool, state, std::move(tx),
    43-                nullptr /* plTxnReplaced */, false /* bypass_limits */, max_tx_fee)) {
    44+        CAmount fee{0};
    45+        if (AcceptToMemoryPool(*node.mempool, state, tx,
    


    ariard commented at 11:42 pm on September 3, 2020:

    I think this approach is okay for now but it would be better to refactor further in the future. For now the mempool is guaranteed not to change due to tacking cs_main lock above but if we remove this lock requirement from ATMP in the future you might have different views of the mempool in the future, thus being rejected at first try and not being evaluated against max_tx_fee but succeeding at second throw and effectively bypassing the protection.

    So technically the comment “Transaction fee is acceptable” is misleading as fee check might have been skipped.

    You could add a new function GetTransactionFee composed of both a mempool lock tacking, a outpoints fetching logic (src/validation.cpp L649) and Consensus::CheckTxInputs.

  71. in src/rpc/rawtransaction.cpp:954 in 6bbda0e5d4 outdated
    923@@ -924,10 +924,19 @@ static UniValue testmempoolaccept(const JSONRPCRequest& request)
    924 
    925     TxValidationState state;
    926     bool test_accept_res;
    927+    CAmount fee{0}; // To return transaction fee from AcceptToMemoryPool
    928     {
    929         LOCK(cs_main);
    930         test_accept_res = AcceptToMemoryPool(mempool, state, std::move(tx),
    931-            nullptr /* plTxnReplaced */, false /* bypass_limits */, max_raw_tx_fee, /* test_accept */ true);
    932+            nullptr /* plTxnReplaced */, false /* bypass_limits */, /* test_accept */ true, &fee);
    


    ariard commented at 11:46 pm on September 3, 2020:
    A follow-up could be to modify testmempoolaccept signature to return the computed fee/feerate, do we already have a RPC to evaluate-fee-but-don’t-submit-to-mempool ?

    glozow commented at 0:05 am on September 4, 2020:
    testmempoolaccept would be the best place for this imo, see #19940 :)
  72. ariard commented at 11:46 pm on September 3, 2020: member
    Code Review ACK 6bbda0e
  73. jonatack commented at 2:26 pm on September 4, 2020: member
    utACK 6bbda0e5d4c63d4923ddc5ef6fcf4d8145a5e8c7 modulo #19339 (review)
  74. jonatack commented at 2:31 pm on September 4, 2020: member
    The CI failures seem unrelated–the travis job logs no error afaict and cirrus timed out.
  75. sdaftuar commented at 0:03 am on September 5, 2020: member
    I agree with the goal of this PR, but invoking ATMP twice really feels like the wrong solution. Could we just add a function to validation.cpp that returns the fee of a transaction (similar to what @ariard suggested in #19339 (review))? That would be helpful more generally, for instance so that our wallet could someday tell us what the feerate is of incoming transactions.
  76. jnewbery commented at 9:48 am on September 5, 2020: member

    invoking ATMP twice really feels like the wrong solution

    Can you give a bit more concrete reason why this feels like the wrong solution to you? It seems like a very elegant way to utilize the existing code and avoid repetition. Note that this approach was suggested by @MarcoFalke in #15810 and got approach ACKs there from @sipa, @gmaxwell and me.

  77. sdaftuar commented at 10:54 am on September 5, 2020: member

    Note that this approach was suggested by @MarcoFalke in #15810 and got approach ACKs there from @sipa, @gmaxwell and me.

    Thanks for the context of #15810. I agree that this PR’s approach is better in some ways than the one there, by making the check in one place, at the time we try to call ATMP, rather than spread across multiple places in our rpc/wallet code.

    However ATMP does so much more than just calculate the fee of a transaction, so if that’s the only reason we’re invoking it twice it seems like overkill. How much slower does this make the rpc call, particularly if a user is working with large transactions? Re-validating a transaction, even when its signature checks have been cached, can still be slow for large transactions (especially non-segwit transactions) due to needing to rehash the data for those signature checks again.

    If all we need is to look up inputs in the utxo set or in the mempool that is much quicker; so that aspect of #15810 I think is preferable to what we have here. I think the best outcome would be to add a function to validation that does the very simple thing of returning the fee of a given transaction (if calculable from the utxo and mempool, otherwise return an error); and where we call ATMP twice in this patch we could instead call the fee calculator and, on success and passing the absurd fee check, invoke ATMP.

    This would result in the cleanest outcome for ATMP (slight improvement over what is proposed here is to get rid of the optional fee_out argument), and would be most performant. The downside is the small amount of logic duplication in having a way to calculate fee in a single place outside of ATMP, but I think the performance benefit would outweigh that. (By contrast, the version of #15810 that I reviewed just now had this check reimplemented in 3 places through the rpc/wallet code, so I wonder if that aspect is what reviewers were objecting to?)

  78. glozow commented at 2:34 pm on September 5, 2020: member

    Thanks for your input @sdaftuar! I agree that the fee check should be minimal.

    Some concerns I’ve heard about the 2x ATMP and will try to summarize here:

    • Performance: I did some bookkeeping by adding some logs to see how much use we get out of the signature and script caches (comment). We get quite a bit of reuse although, as you said, the hashing is also expensive. I assumed that a small performance hit wouldn’t be too egregious in a function only used by clients.
    • Unnecessary ATMP validation logic when all we need is the fee: I definitely see this. Adding another function would probably have to do many of the sanitization steps as PreChecks, which isn’t too expensive but would also be a little redundant. I think that just calling PreChecks would be sufficient if we could do that, but perhaps we would want to uncache the coins as well (handled in ATMPWithTime afterward)? As I thought about the “minimum amount” we need, it started to look more and more similar to ATMP, even if it’s just adding it in 1 place.

    Some reasons for using 2x ATMP:

    • Minimal new code
    • In the case where a transaction has both, say, an invalid script and absurd fee, I think it would make more sense to return the more serious validation failure rather than max-fee-exceeded.
    • If the minimal fee-checking function returns an error, should we try again with ATMP or trust its result and return failure? If the latter, unless it’s comprehensive, I was worried I’d accidentally introduce a bug where BroadcastTransaction is often failing due to imperfect fee-checker rather than real invalidity. If I were to do it this way, I’d probably reuse PreChecks.

    This would result in the cleanest outcome for ATMP (slight improvement over what is proposed here is to get rid of the optional fee_out argument)

    For some context, #19093 added fee_out to return the fee to the user in testmempoolaccept. I built on top of that since it suited my needs as well.

    I followed this approach under others’ guidance (as @jnewbery mentioned), but also thought about the case of adding a separate check. Since you mentioned a separate fee-checking function could be helpful in other cases as well, I’m open/interested in implementing this. I just think that many of the things we do in ATMP would need to be repeated there as well. Would you agree that this is the case, and would it still be appropriate to add this?

  79. ariard commented at 4:29 pm on September 6, 2020: member

    EDIT: Just to be clear, I maintain my Code Review ACK. That said it would be great if we open an issue or else to discuss and track further improvements in this area of the code, there is room for it.


    If the problem solves by this PR is “How to disentangle wallet sanity max fee from mempool policy checks ?”, I’m not sure the current approach achieves it as types of checks are still blurred, i.e a policy failure will mask the wallet max fee.

    In the case where a transaction has both, say, an invalid script and absurd fee, I think it would make more sense to return the more serious validation failure rather than max-fee-exceeded.

    A failure is a failure, what heuristic do you follow to sort the severity of them ? Your transaction can still succeed all mempool rules, but still not broadcast due to its feerate inferior to all your peer’s feefilter. Gathering checks by component which are responsible of them a) should ease review reasoning b) should enhance performance.

    I was worried I’d accidentally introduce a bug where BroadcastTransaction is often failing due to imperfect fee-checker rather than real invalidity. If I were to do it this way, I’d probably reuse PreChecks.

    Note that this approach isn’t bug-free, see above if we refactor the main lock tacking. Code duplication is a concern. The fee computation job is partially done by Consensus::CheckTxInputs which can be reused. What we can do is refactor internally PreChecks to extract the outpoints gathering logic in its own method thus a new FeeCalculator just have to wrap both and should be consistent with ATMP.

    Also, any wallet check shouldn’t belong to BroadcastTransaction but on the other side of the interface in src/wallet/. IIRC the design goal of BroadcastTransaction is just to be a wrapper between nodes components.

    The current PR is worthy on its own to at least separate logically wallet from mempool checks and the reason I gave my ACK. But more I think about it more I guess we would like to have a clean FeeCalculator method. It’s more costly in reviewing time but we could reuse it for exacting feerate of Taproot witnesses (which may come with their own feerate accounting rules, hard to reimplement correctly by client) or build an automatic RBF logic in the wallet (and you need to cache feerate across attempts).

  80. glozow commented at 5:14 pm on September 7, 2020: member

    Thanks @ariard! I appreciate that your ACK is not being withdrawn.

    A failure is a failure, what heuristic do you follow to sort the severity of them ?

    We have a few defined categories that are relevant here, afaik:

    • Validation/Consensus failures (e.g. an invalid script or signature, etc)
    • Policy fails (e.g. minimum feerate for relay)
    • Client preference (e.g. absurd fee)

    I’ve [arbitrarily] sorted them in severity, based on the idea that consensus failure = nobody will accept, policy failure = this node won’t accept, client preference = this user won’t accept. So I meant that any validation failure would be more serious than a client preference not being met. That being said, I could be misguided to say that higher severity should take priority. But this PR doesn’t hinge on this idea.

    If the problem solves by this PR is “How to disentangle wallet sanity max fee from mempool policy checks ?“, I’m not sure the current approach achieves it as types of checks are still blurred, i.e a policy failure will mask the wallet max fee.

    Please let me know if I misunderstand your comment, but I don’t think the checks are “still blurred.” I would interpret blurring to be when code should only be responsible for checking mempool/policy also has a client check mixed in there, which was the case before this PR. Also confusing, it would return TX_NOT_STANDARD (should = policy error) for absurd fee. In terms of failure cases, let me enumerate them:

    policy absurd fee ATMP error (before this PR) ATMP error (after this PR)
    fail fail (depends on which one comes first) TxValidationResult::(policy)
    fail pass TxValidationResult::(policy) TxValidationResult::(policy)
    pass fail TxValidationResult::TX_NOT_STANDARD none. client returns MAX_FEE_EXCEEDED

    I’d say this is less blurred than before. Further, the wallet (before and after this PR) checks against max fee in CreateTransaction. Thus, afaik it catches absurd fees before ever trying to submit to mempool, and case 1 and 3 don’t actually happen. This PR is more relevant to the RPCs testmempoolaccept and sendrawtransaction, where ATMP was actually shouldering the fee checking responsibility.

    I think we should discuss/implement a FeeCalculator method on a separate issue or PR if it isn’t a blocker here (although still waiting to hear from @sdaftuar on whether it is) :slightly_smiling_face:

  81. DrahtBot added the label Needs rebase on Sep 7, 2020
  82. ariard commented at 1:19 pm on September 8, 2020: member

    I’ve [arbitrarily] sorted them in severity, based on the idea that consensus failure = nobody will accept, policy failure = this node won’t accept, client preference = this user won’t accept.

    Sure, the set of parties interested to the rule violation is one axis of analysis. The one I would favor is ordering them by tightness, namely a set A is more tight than set B if transaction is accepted by B but not by A. So wallet’s absurd fee is more strict than any consensus rules violation, even if local user is the only one interested. Concretely that gives you a heuristic to organize code in a way that if your transaction is rejected by set B (policy) it won’t mask rejection by set A (client preference), as current code after the PR is doing by reusing ATMP.

    I would interpret blurring to be when code should only be responsible for checking mempool/policy also has a client check mixed in there, which was the case before this PR.

    Correct if I’m wrong but before/after this PR, if your transaction violates MIN_STANDARD_TX_NONWITNESS_SIZEand absurd fee, a user will only see reject message for the first violation and might still fail its client preferences after correcting the policy violation. Layering cleanly errors means a user is less confused about what to do in case of failure.

    This PR is more relevant to the RPCs testmempoolaccept and sendrawtransaction, where ATMP was actually shouldering the fee checking responsibility.

    Yep and encapsulating the code further to be reused across RPCs, wallets or even as a library sounds like future steps :)

  83. sdaftuar commented at 2:28 pm on September 9, 2020: member

    I think this patch is basically what the fee calculator would need to look like:

     0diff --git a/src/validation.cpp b/src/validation.cpp
     1index 58af8744d90..87685b44f6b 100644
     2--- a/src/validation.cpp
     3+++ b/src/validation.cpp
     4@@ -1059,6 +1059,30 @@ bool MemPoolAccept::AcceptSingleTransaction(const CTransactionRef& ptx, ATMPArgs
     5 
     6 } // anon namespace
     7 
     8+
     9+// Calculate the fee of a transaction, by looking up inputs from the mempool if
    10+// necessary. Return true if the fee was successfully calculated, or false if
    11+// an error with the transaction was discovered.
    12+// Success does not imply that the transaction is valid -- but if it is valid,
    13+// then the fee returned is correct.
    14+bool GetTransactionFee(CTxMemPool& pool, TxValidationState &state, const CTransactionRef &tx, CAmount *fee_out)
    15+{
    16+    LOCK2(cs_main, pool.cs);
    17+
    18+    CCoinsViewMemPool viewmempool(&::ChainstateActive().CoinsTip(), pool);
    19+    CCoinsViewCache view(&viewmempool);
    20+    view.GetBestBlock();
    21+
    22+    *fee_out = 0;
    23+
    24+    // Fees don't make sense for coinbase transactions
    25+    if (tx->IsCoinBase()) return false;
    26+    // Re-use the fee calculation in CheckTxInputs, which also validates that
    27+    // the fee calculation is correct (ie that we don't overflow)
    28+    if (!Consensus::CheckTxInputs(*tx, state, view, GetSpendHeight(view), *fee_out)) return false;
    29+    return true;
    30+}
    31+
    32 /** (try to) add transaction to memory pool with a specified acceptance time **/
    33 static bool AcceptToMemoryPoolWithTime(const CChainParams& chainparams, CTxMemPool& pool, TxValidationState &state, const CTransactionRef &tx,
    34                         int64_t nAcceptTime, std::list<CTransactionRef>* plTxnReplaced,
    35diff --git a/src/validation.h b/src/validation.h
    36index cac9473c7a5..9833b9d9e05 100644
    37--- a/src/validation.h
    38+++ b/src/validation.h
    39@@ -199,6 +199,8 @@ void UnlinkPrunedFiles(const std::set<int>& setFilesToPrune);
    40 /** Prune block files up to a given height */
    41 void PruneBlockFilesManual(int nManualPruneHeight);
    42 
    43+bool GetTransactionFee(CTxMemPool& pool, TxValidationState &state, const CTransactionRef &tx, CAmount *fee_out);
    44+
    45 /** (try to) add transaction to memory pool
    46  * plTxnReplaced will be appended to with all transactions replaced from mempool **/
    47 bool AcceptToMemoryPool(CTxMemPool& pool, TxValidationState &state, const CTransactionRef &tx,
    

    Well it should probably fill in the TxValidationState for a coinbase transaction rather than just return false, but aside from that I don’t believe we need more than this. The semantics I suggest are: if the transaction is valid, then the fee calculated here is definitely correct; but if the transaction is invalid you don’t know. I think this is sufficient for many purposes, including the proposal here.

    I was giving this more thought and I can also see a philosophical argument for fees only being meaningful if a transaction is valid, but validity is a tricky idea, because mempool acceptance couples various policy and anti-DoS ideas in with consensus validity. So something like #19093 is reasonable to me for that reason, and building on that to do the absurd fee check for testmempoolaccept seems reasonable. However for other use cases in our wallet/rpc code, I think pulling out the fee calculation separately would be preferable to running ATMP twice. And it seems to me that this kind of function should be more generally useful (for example I could imagine wanting a way to calculate the fee of a rawtransaction, without having to run it through ATMP).

    I think the way you’d use this is, if you’re doing the maximum fee check, invoke this function and see if it succeeds; if not you give up on the transaction. If it succeeds and the fee is too high, again give up. Otherwise, invoke ATMP() and return the success or failure from that.

    In the case where a transaction has both, say, an invalid script and absurd fee, I think it would make more sense to return the more serious validation failure rather than max-fee-exceeded.

    I think this is hard to say; I suspect it’s way more likely to get a max-fee-exceeded than an actual script or policy failure! It seems perfectly reasonable to me that a workflow for submitting a transaction is to first check the fees are sane, and then validate if that passes, so if our software does that too I think it’s fine.

    If the minimal fee-checking function returns an error, should we try again with ATMP or trust its result and return failure? If the latter, unless it’s comprehensive, I was worried I’d accidentally introduce a bug where BroadcastTransaction is often failing due to imperfect fee-checker rather than real invalidity. If I were to do it this way, I’d probably reuse PreChecks.

    With my proposed semantics, we wouldn’t continue if there’s an error – the fee calculator I propose is only correct if there are no errors, otherwise all bets are off. I don’t think using PreChecks() would be problematic, but I think it’s easier and more concise to just use CheckTxInputs() as I suggest.

  84. in src/node/transaction.cpp:63 in 6bbda0e5d4 outdated
    45+        if (AcceptToMemoryPool(*node.mempool, state, tx,
    46+                nullptr /* plTxnReplaced */, false /* bypass_limits */, /* test_accept */ true, &fee)) {
    47+            if (max_tx_fee && fee > max_tx_fee) {
    48+                return TransactionError::MAX_FEE_EXCEEDED;
    49+            }
    50+        }
    


    LarryRuane commented at 6:09 pm on September 15, 2020:
    0        if (max_tx_fee && AcceptToMemoryPool(*node.mempool, state, tx,
    1                nullptr /* plTxnReplaced */, false /* bypass_limits */, /* test_accept */ true, &fee)) {
    2            if (fee > max_tx_fee) {
    3                return TransactionError::MAX_FEE_EXCEEDED;
    4            }
    5        }
    

    Performance: Call AcceptToMemoryPool() only if necessary.


    glozow commented at 4:07 pm on September 18, 2020:
    yep will do! :)
  85. LarryRuane commented at 3:06 pm on September 16, 2020: contributor
    This PR updates several of the tests (both unit and functional), but, unless I’m missing it, it doesn’t look like there’s any verification of the modified behavior (asserting that transactions with high fees are forwarded with the PR but not without the PR). This would probably require a brand new test or tests. Have you considered doing that? I may be able to help with this, but not until next week at the earliest. What do you think?
  86. glozow force-pushed on Sep 18, 2020
  87. glozow commented at 8:20 pm on September 18, 2020: member

    Just pushed with a rebase and GetTransactionFee. Hope that you can take a look at it @sdaftuar; it’s largely based on your comment. And @jnewbery @ariard @jonatack - diff of the first 5 commits is the same so I hope it’s easier to re-review. Thank you so much :pray:

    Thank you for your review @larryruane! Incorporated your only if(max_tx_fee) suggestion in https://github.com/bitcoin/bitcoin/pull/19339/commits/8c8b936dca571e4fd631c9a8a1306549072b449c

    it doesn’t look like there’s any verification of the modified behavior (asserting that transactions with high fees are forwarded with the PR but not without the PR).

    This behavior hasn’t been changed, high fee transactions are still not forwarded if from client and are forwarded if from peers :)

  88. DrahtBot removed the label Needs rebase on Sep 18, 2020
  89. DrahtBot added the label Needs rebase on Sep 19, 2020
  90. glozow force-pushed on Sep 22, 2020
  91. glozow commented at 0:23 am on September 22, 2020: member
    Thanks for your feedback @DrahtBot, addressed.
  92. DrahtBot removed the label Needs rebase on Sep 22, 2020
  93. glozow force-pushed on Sep 22, 2020
  94. MarcoFalke removed the label Tests on Sep 22, 2020
  95. in src/node/transaction.cpp:66 in 5c5fd76ad1 outdated
    57+            }
    58+        }
    59+        // Transaction fee is acceptable. Submit the transaction.
    60         if (!AcceptToMemoryPool(*node.mempool, state, tx,
    61-                                nullptr /* plTxnReplaced */, false /* bypass_limits */, max_tx_fee)) {
    62+                nullptr /* plTxnReplaced */, false /* bypass_limits */)) {
    


    LarryRuane commented at 5:03 pm on September 22, 2020:

    Similar nonblocking suggestion as above – You may even want to consider refactoring this common error processing into a TransactionError method.

     0        if (!AcceptToMemoryPool(*node.mempool, state, tx,
     1                nullptr /* plTxnReplaced */, false /* bypass_limits */)) {
     2            err_string = state.ToString();
     3            if (!state.IsInvalid()) {
     4                return TransactionError::MEMPOOL_ERROR;
     5            }
     6            if (state.GetResult() == TxValidationResult::TX_MISSING_INPUTS) {
     7                return TransactionError::MISSING_INPUTS;
     8            }
     9            return TransactionError::MEMPOOL_REJECTED;
    10        }
    

    glozow commented at 10:21 pm on September 24, 2020:
    Good point, made a helper function :)
  96. in src/validation.cpp:1117 in 5c5fd76ad1 outdated
    1114+    // Re-use the fee calculation in CheckTxInputs, which also validates that
    1115+    // the fee calculation is correct (i.e. that we don't overflow)
    1116+    if (!Consensus::CheckTxInputs(*tx, state, view, GetSpendHeight(view), *fee_out)) {
    1117+        return false; // state filled in by CheckTxInputs
    1118+    }
    1119+    return true;
    


    LarryRuane commented at 5:05 pm on September 22, 2020:

    Simplification suggestion:

    0    return Consensus::CheckTxInputs(*tx, state, view, GetSpendHeight(view), *fee_out);
    
  97. LarryRuane commented at 5:35 pm on September 22, 2020: contributor
    ACK 5c5fd76ad1ee68c3cd77436bad49041969cf1cb1 (three nonblocking suggestions)
  98. in src/validation.cpp:1093 in 5c5fd76ad1 outdated
    1089@@ -1090,6 +1090,33 @@ bool AcceptToMemoryPool(CTxMemPool& pool, TxValidationState &state, const CTrans
    1090     return AcceptToMemoryPoolWithTime(chainparams, pool, state, tx, GetTime(), plTxnReplaced, bypass_limits, test_accept, fee_out);
    1091 }
    1092 
    1093+// Calculate the fee of a transaction by looking up inputs from the mempool if
    


    jnewbery commented at 10:41 am on September 23, 2020:
    This would be better as a doxygen comment next to the declaration in the header file.
  99. in src/validation.cpp:1098 in 5c5fd76ad1 outdated
    1089@@ -1090,6 +1090,33 @@ bool AcceptToMemoryPool(CTxMemPool& pool, TxValidationState &state, const CTrans
    1090     return AcceptToMemoryPoolWithTime(chainparams, pool, state, tx, GetTime(), plTxnReplaced, bypass_limits, test_accept, fee_out);
    1091 }
    1092 
    1093+// Calculate the fee of a transaction by looking up inputs from the mempool if
    1094+// necessary. Return true if the fee was successfully calculated, or false if
    1095+// an error with the transaction was discovered.
    1096+// If the transaction is valid, then the fee returned is correct.
    1097+// However, success does not imply that the transaction is valid.
    1098+bool GetTransactionFee(CTxMemPool& pool, TxValidationState &state, const CTransactionRef &tx, CAmount *fee_out)
    


    jnewbery commented at 10:43 am on September 23, 2020:

    CTransactionRef is a shared_ptr. shared_ptrs should only be passed by const ref to express that the called function might retain a ref count to the object, which never happens here. There are no lifetime semantics involved, so this should be passed as a pointer or reference. Const reference (const CTransaction&) would be most appropriate here.

    (C++ Core guidelines on passing shared_ptrs: http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rr-sharedptrparam-const)


    jnewbery commented at 10:45 am on September 23, 2020:
    s/TxValidationState &state/TxValidationState& state/ and s/Amount *fee_out/Amount* fee_out/

    jnewbery commented at 10:50 am on September 23, 2020:
    I think an Optional<CAmount> is the most natural return type here, rather than a ‘success’ bool and an out-param. Unset would mean “transaction invalid - fee can’t be calculated” and set would mean “fee calculated (but transaction not necessarily valid)”.
  100. in src/validation.cpp:1100 in 5c5fd76ad1 outdated
    1095+// an error with the transaction was discovered.
    1096+// If the transaction is valid, then the fee returned is correct.
    1097+// However, success does not imply that the transaction is valid.
    1098+bool GetTransactionFee(CTxMemPool& pool, TxValidationState &state, const CTransactionRef &tx, CAmount *fee_out)
    1099+{
    1100+    LOCK2(cs_main, pool.cs);
    


    jnewbery commented at 10:46 am on September 23, 2020:
    This function is only ever called in one place, where cs_main is already being held. Rather than lock recursively, it would be better to annotate the function to require cs_main.
  101. in src/validation.cpp:1104 in 5c5fd76ad1 outdated
    1099+{
    1100+    LOCK2(cs_main, pool.cs);
    1101+
    1102+    CCoinsViewMemPool viewmempool(&::ChainstateActive().CoinsTip(), pool);
    1103+    CCoinsViewCache view(&viewmempool);
    1104+    view.GetBestBlock();
    


    jnewbery commented at 10:47 am on September 23, 2020:
    What is this line for? GetBestBlock() is a const method, and the return value is being discarded here, so this appears to be a no-op.

    sdaftuar commented at 11:04 pm on September 25, 2020:

    GetBestBlock() is const, but hashBlock is mutable, and what this function does is load the block hash associated with the utxo set into the CoinsViewCache object.

    The reason it is here is so that when we invoke GetSpendHeight(m_view) in the call to Consensus::CheckTxInputs(), we’re doing so with the right height of the next block, which is necessary for the coinbase maturity check that happens in that function.

    I think omitting this line would probably cause the code to crash, due to a null dereference in GetSpendHeight()?


    glozow commented at 2:47 pm on September 26, 2020:

    I tried removing the line and it looks okay. I might be reading this wrong but I think when we call CheckTxInputs with the GetSpendHeight(view):

    0if (!Consensus::CheckTxInputs(tx, state, view, GetSpendHeight(view), *fee_out)) {
    

    GetSpendHeight() calls GetBestBlock() for view as well:

    0int GetSpendHeight(const CCoinsViewCache& inputs)
    1{
    2    LOCK(cs_main);
    3    CBlockIndex* pindexPrev = LookupBlockIndex(inputs.GetBestBlock());
    4    return pindexPrev->nHeight + 1;
    5}
    

    sdaftuar commented at 3:08 pm on September 26, 2020:
    Good point!
  102. in src/validation.cpp:1110 in 5c5fd76ad1 outdated
    1105+
    1106+    *fee_out = 0;
    1107+
    1108+    // Fees don't make sense for coinbase transactions
    1109+    if (tx->IsCoinBase()) {
    1110+        return state.Invalid(TxValidationResult::TX_CONSENSUS, "coinbase");
    


    jnewbery commented at 10:54 am on September 23, 2020:
    It feels weird to be setting a TxValidationResult in a function that purports to simply be a utility helper, and duplicating validity checks in this function feels like code duplication with the risk of divergence in the future. I’d much prefer it if this function was only checking/returning the fee, and all validity state was handled by ATMP.
  103. in src/node/transaction.cpp:52 in 5c5fd76ad1 outdated
    50+                        return TransactionError::MISSING_INPUTS;
    51+                    }
    52+                    return TransactionError::MEMPOOL_REJECTED;
    53+                } else {
    54+                    return TransactionError::MEMPOOL_ERROR;
    55+                }
    


    jnewbery commented at 10:58 am on September 23, 2020:
    It’s a shame to duplicate the code a few lines below. If we used GetTransactionFee() just to get the transaction fee (and not return any validation state) and relied on ATMP to return validation state, we wouldn’t have this code duplication.

    glozow commented at 10:40 pm on September 24, 2020:
    I made a helper function for this so it doesn’t duplicate the code The question of “if GetFee fails, still run ATMP or fail immediately?” is still up in the air for me…

    sdaftuar commented at 11:11 pm on September 25, 2020:
    @gzhao408 What is the benefit of running ATMP if GetTransactionFee() fails? It seems to me that if you get a failure, you should abort – particularly because the way it’s being used here is to prevent an absurd fee from getting through. If a transient failure prevented the fee from being calculated (say, a race condition with an input being confirmed in a block or arriving in the mempool from another transaction), then continuing after the fee check fails to return a value risks actually sending out a transaction in the call to ATMP. @jnewbery I think GetTransactionFee() should return a reason for failure. For instance, I could imagine that we might want to expose a getrawtransactionfee RPC where we just return the fee of a rawtransaction, and being able to return a meaningful error for why the fee couldn’t be calculated seems useful in that context.

    glozow commented at 2:08 pm on September 26, 2020:

    @sdaftuar I definitely agree that if GetTransactionFee fails, we shouldn’t try to submit the transaction. My issue with using GetTransactionFee to return a TxValidationResult is that, in my opinion, ATMP should be the authority on validation results.

    At the end of the day, the fee is not meaningful unless validation succeeds. If GetTransactionFee fails due to the transaction being invalid, yes it fails, but it’s more useful to return the validation error than “sorry, couldn’t get the fee.” But I don’t think GetTransactionFee should be relied upon to return a validation error. It would make sense to me to run a test_accept of ATMP to get the proper validation error if GetTransactionFee fails (but then with this idea… why not just dry-run ATMP haha).


    glozow commented at 2:15 pm on September 26, 2020:
    Also @sdaftuar the testmempoolaccept RPC already returns the fee for a raw transaction (#19940)!
  104. jnewbery commented at 10:58 am on September 23, 2020: member
    I think the last commit is unnecessary, and I have a few questions/comments about the new code.
  105. glozow force-pushed on Sep 24, 2020
  106. glozow commented at 4:26 pm on September 26, 2020: member

    I felt that we had a lot of open questions regarding GetTransactionFee so I’ve opened a separate PR for review, #20025 (addressed @jnewbery’s comments there as well).

    This PR just re-delegates absurdfee checking to clients. The branch as it currently stands is pretty much the same as what was ACK’ed before, with a few small fixes.

    For everyone who has reviewed this PR, I really appreciate your time so far! If you’re still comfortable with these changes, please let me know. If this approach is unacceptable please let me know as well, I can always close this in favor of #20025 since it’s based on this branch :) in general I’d just like to discuss GetTransactionFee there instead of here.

  107. in src/node/transaction.cpp:54 in 22ebb7fda0 outdated
    60-                }
    61-                return TransactionError::MEMPOOL_REJECTED;
    62-            } else {
    63-                return TransactionError::MEMPOOL_ERROR;
    64+        CAmount fee{0};
    65+        if (max_tx_fee) {
    


    LarryRuane commented at 8:37 pm on September 26, 2020:
    0        if (max_tx_fee) {
    1            CAmount fee{0};
    

    instagibbs commented at 2:04 am on October 7, 2020:
    0        if (max_tx_fee > 0) {
    
  108. in src/node/transaction.cpp:16 in 22ebb7fda0 outdated
    12@@ -13,6 +13,18 @@
    13 
    14 #include <future>
    15 
    16+TransactionError HandleATMPError(TxValidationState state, std::string& err_string) {
    


    LarryRuane commented at 8:40 pm on September 26, 2020:
    0TransactionError HandleATMPError(TxValidationState state, std::string& err_string)
    1{
    

    jnewbery commented at 8:15 am on October 5, 2020:

    Minor style suggestions (no need to change these unless you’re retouching the branch anyway):

    • Make the TxValidationState state argument a const reference to avoid a copy on the stack.
    • This function is only used internally in this translation unit. Make it static or put it in an unnamed namespace to avoid external linkage.

    MarcoFalke commented at 8:38 am on October 5, 2020:

    style nit in commit f907f4b9389b86fc655f45917367be5838b79684:

    • Newline before {
    • Could suffix return parameters with _out
    0TransactionError HandleATMPError(TxValidationState state, std::string& err_string_out)
    1{
    

    glozow commented at 12:00 pm on October 5, 2020:
    done, thanks!
  109. in src/node/transaction.cpp:58 in 22ebb7fda0 outdated
    64+        CAmount fee{0};
    65+        if (max_tx_fee) {
    66+            // First, call ATMP with test_accept and check the fee. If ATMP
    67+            // fails here, return error immediately.
    68+            if (!AcceptToMemoryPool(*node.mempool, state, tx,
    69+                nullptr /* plTxnReplaced */, false /* bypass_limits */, /* test_accept */ true, &fee)) {
    


    LarryRuane commented at 8:46 pm on September 26, 2020:
    0            if (!AcceptToMemoryPool(
    1                    *node.mempool, state, tx,
    2                    nullptr /* plTxnReplaced */, false /* bypass_limits */, /* test_accept */ true, &fee)) {
    
  110. in src/node/transaction.cpp:66 in 22ebb7fda0 outdated
    72+                return TransactionError::MAX_FEE_EXCEEDED;
    73             }
    74         }
    75+        // Try to submit the transaction to the mempool.
    76+        if (!AcceptToMemoryPool(*node.mempool, state, tx,
    77+                nullptr /* plTxnReplaced */, false /* bypass_limits */)) {
    


    LarryRuane commented at 8:48 pm on September 26, 2020:
    0        if (!AcceptToMemoryPool(
    1                *node.mempool, state, tx,
    2                nullptr /* plTxnReplaced */, false /* bypass_limits */)) {
    

    glozow commented at 11:54 pm on September 30, 2020:
    🤔 seems different from normal
  111. in src/rpc/rawtransaction.cpp:950 in 22ebb7fda0 outdated
    928@@ -929,11 +929,19 @@ static UniValue testmempoolaccept(const JSONRPCRequest& request)
    929 
    930     TxValidationState state;
    931     bool test_accept_res;
    932-    CAmount fee;
    933+    CAmount fee{0}; // To return transaction fee from AcceptToMemoryPool
    


    LarryRuane commented at 8:49 pm on September 26, 2020:
    0    CAmount fee{0};
    
  112. LarryRuane commented at 9:19 pm on September 26, 2020: contributor

    utACK 22ebb7fda03fd18361066b4ce92eeed009542976 This PR removes six instances of 0 /* nAbsurdFee */ which, just by itself, is a nice cleanup! (In addition to the other more important reasons.) Thanks for the PR!

    I ran the unit tests but didn’t do any careful manual testing.

    A few more non-blocking mostly-formatting suggestions (most of these were from running contrib/devtools/clang-format-diff.py):

  113. DrahtBot added the label Needs rebase on Sep 30, 2020
  114. glozow force-pushed on Oct 1, 2020
  115. glozow commented at 11:12 am on October 1, 2020: member
    Rebased
  116. DrahtBot removed the label Needs rebase on Oct 1, 2020
  117. LarryRuane commented at 6:57 pm on October 4, 2020: contributor

    utACK 8784930

     01:  b1ea8a00f ! 1:  f907f4b93 [rpc/node] check for high fee before ATMP in clients
     1    @@ src/node/transaction.cpp: TransactionError BroadcastTransaction(NodeContext& nod
     2      
     3     
     4      ## src/rpc/rawtransaction.cpp ##
     5    -@@ src/rpc/rawtransaction.cpp: static UniValue testmempoolaccept(const JSONRPCRequest& request)
     6    +@@ src/rpc/rawtransaction.cpp: static RPCHelpMan testmempoolaccept()
     7      
     8          TxValidationState state;
     9          bool test_accept_res;
    102:  4568fa37f = 2:  b0ec04de0 scripted-diff: update max-fee-exceeded error message to include RPC
    113:  b635b5d51 ! 3:  6a4a7c642 [validation] ignore absurdfee arg in ATMP
    12    @@ Commit message
    13         [validation] ignore absurdfee arg in ATMP
    14     
    15      ## src/rpc/rawtransaction.cpp ##
    16    -@@ src/rpc/rawtransaction.cpp: static UniValue testmempoolaccept(const JSONRPCRequest& request)
    17    +@@ src/rpc/rawtransaction.cpp: static RPCHelpMan testmempoolaccept()
    18                  nullptr /* plTxnReplaced */, false /* bypass_limits */, max_raw_tx_fee, /* test_accept */ true, &fee);
    19          }
    20      
    214:  22ebb7fda ! 4:  8784930df [validation] Remove absurdfee from accepttomempool
    22    @@ src/bench/block_assemble.cpp: static void AssembleBlock(benchmark::Bench& bench)
    23          }
    24     
    25      ## src/net_processing.cpp ##
    26    -@@ src/net_processing.cpp: void PeerManager::ProcessOrphanTx(std::set<uint256>& orphan_work_set, std::list<
    27    -         TxValidationState orphan_state;
    28    +@@ src/net_processing.cpp: void PeerManager::ProcessOrphanTx(std::set<uint256>& orphan_work_set)
    29    +         TxValidationState state;
    30    +         std::list<CTransactionRef> removed_txn;
    31      
    32    -         if (setMisbehaving.count(fromPeer)) continue;
    33    --        if (AcceptToMemoryPool(m_mempool, orphan_state, porphanTx, &removed_txn, false /* bypass_limits */, 0 /* nAbsurdFee */)) {
    34    -+        if (AcceptToMemoryPool(m_mempool, orphan_state, porphanTx, &removed_txn, false /* bypass_limits */)) {
    35    +-        if (AcceptToMemoryPool(m_mempool, state, porphanTx, &removed_txn, false /* bypass_limits */, 0 /* nAbsurdFee */)) {
    36    ++        if (AcceptToMemoryPool(m_mempool, state, porphanTx, &removed_txn, false /* bypass_limits */)) {
    37                  LogPrint(BCLog::MEMPOOL, "   accepted orphan tx %s\n", orphanHash.ToString());
    38                  RelayTransaction(orphanHash, porphanTx->GetWitnessHash(), m_connman);
    39    -             for (unsigned int i = 0; i < orphanTx.vout.size(); i++) {
    40    +             for (unsigned int i = 0; i < porphanTx->vout.size(); i++) {
    41     @@ src/net_processing.cpp: void PeerManager::ProcessMessage(CNode& pfrom, const std::string& msg_type, CDat
    42              // (older than our recency filter) if trying to DoS us, without any need
    43              // for witness malleation.
    44    @@ src/node/transaction.cpp: TransactionError BroadcastTransaction(NodeContext& nod
    45      
    46     
    47      ## src/rpc/rawtransaction.cpp ##
    48    -@@ src/rpc/rawtransaction.cpp: static UniValue testmempoolaccept(const JSONRPCRequest& request)
    49    +@@ src/rpc/rawtransaction.cpp: static RPCHelpMan testmempoolaccept()
    50          {
    51              LOCK(cs_main);
    52              test_accept_res = AcceptToMemoryPool(mempool, state, std::move(tx),
    
  118. jnewbery commented at 8:19 am on October 5, 2020: member

    utACK 8784930dfaf4b981be6a3670c4298de2226a3989

    I left a couple of style suggestions inline, which you should ignore unless you touch this branch again.

  119. in test/functional/rpc_rawtransaction.py:461 in f907f4b938 outdated
    455@@ -456,9 +456,9 @@ def run_test(self):
    456         # Thus, testmempoolaccept should reject
    457         testres = self.nodes[2].testmempoolaccept([rawTxSigned['hex']], 0.00001000)[0]
    458         assert_equal(testres['allowed'], False)
    459-        assert_equal(testres['reject-reason'], 'absurdly-high-fee')
    460+        assert_equal(testres['reject-reason'], 'max-fee-exceeded')
    461         # and sendrawtransaction should throw
    462-        assert_raises_rpc_error(-26, "absurdly-high-fee", self.nodes[2].sendrawtransaction, rawTxSigned['hex'], 0.00001000)
    463+        assert_raises_rpc_error(-25, 'Fee exceeds maximum configured by -maxtxfee', self.nodes[2].sendrawtransaction, rawTxSigned['hex'], 0.00001000)
    


    MarcoFalke commented at 8:51 am on October 5, 2020:

    bug in commit f907f4b:

    This is wrong. maxtxfee doesn’t influence sendrawtransaction at all. See help sendrawtransaction


    MarcoFalke commented at 8:57 am on October 5, 2020:

    Ah, I see this is fixed in the next commit. Maybe it could make sense to put the scripted diff first, so that this line doesn’t have to change twice in the same pull request?

    Also, the scripted diff could use git grep -l "bla" instead of the hardcoded list of file names. This way it is easier to see that it is complete. Also, rebasing will be easier if a filename changes.


    glozow commented at 12:48 pm on October 5, 2020:

    Also, the scripted diff could use git grep -l “bla” instead of the hardcoded list of file names.

    I will definitely do that in the future 😅

    Maybe it could make sense to put the scripted diff first, so that this line doesn’t have to change twice in the same pull request?

    I think the line would still need to change, sine the rpc error code changes from -26 to -25 as well

  120. in test/functional/rpc_rawtransaction.py:459 in f907f4b938 outdated
    455@@ -456,9 +456,9 @@ def run_test(self):
    456         # Thus, testmempoolaccept should reject
    457         testres = self.nodes[2].testmempoolaccept([rawTxSigned['hex']], 0.00001000)[0]
    458         assert_equal(testres['allowed'], False)
    459-        assert_equal(testres['reject-reason'], 'absurdly-high-fee')
    460+        assert_equal(testres['reject-reason'], 'max-fee-exceeded')
    


    MarcoFalke commented at 8:51 am on October 5, 2020:

    question in commit f907f4b:

    Any reason to change the reject-reason?


    glozow commented at 11:09 am on October 5, 2020:
    I wanted to use the same reject reason that wallet gives when a fee is too high, which is TransactionError::MAX_FEE_EXCEEDED. Also a little sanity check for all the absurdly-high-fee tests.

    MarcoFalke commented at 1:21 pm on October 5, 2020:

    Not sure if I understand this. MAX_FEE_EXCEEDED returns the string "Fee exceeds maximum configured by user", not "max-fee-exceeded".

    As there is no reject code, RPC users might match on the reject reason, which seems brittle, but if there is no convincing argument to change, I’d prefer to leave it as is.

    edit: probably not too much of a deal


    glozow commented at 2:55 pm on October 5, 2020:
    You’re right, yeah. I guess this is an arbitrary change to make the error messages more similar 😅
  121. in src/rpc/rawtransaction.cpp:957 in f907f4b938 outdated
    953         LOCK(cs_main);
    954         test_accept_res = AcceptToMemoryPool(mempool, state, std::move(tx),
    955             nullptr /* plTxnReplaced */, false /* bypass_limits */, max_raw_tx_fee, /* test_accept */ true, &fee);
    956     }
    957+
    958+    // TODO: move ATMP absurdFee check to here
    


    MarcoFalke commented at 9:05 am on October 5, 2020:

    in commit f907f4b9389b86fc655f45917367be5838b79684:

    Any reason to not squash commit 6a4a7c64246c826fed6b2bdf97ad91a5f864621b into this? Seems odd to add an TODO in one commit and then fix it in the next commit in the same pull.


    glozow commented at 11:10 am on October 5, 2020:
    Yeah, I had this odd idea that people might review commit-by-commit and that documenting what to expect might be helpful, but perhaps it’s just weird and confusing 😅

    glozow commented at 11:50 am on October 5, 2020:
    will squash, that is a better way 👍
  122. in src/validation.cpp:733 in 6a4a7c6424 outdated
    729@@ -730,8 +730,7 @@ bool MemPoolAccept::PreChecks(ATMPArgs& args, Workspace& ws)
    730     if (!bypass_limits && !CheckFeeRate(nSize, nModifiedFees, state)) return false;
    731 
    732     if (nAbsurdFee && nFees > nAbsurdFee)
    733-        return state.Invalid(TxValidationResult::TX_NOT_STANDARD,
    734-                "absurdly-high-fee", strprintf("%d > %d", nFees, nAbsurdFee));
    735+        LogPrintf("Ignoring Absurdfee\n");
    


    MarcoFalke commented at 9:06 am on October 5, 2020:

    in commit 6a4a7c64246c826fed6b2bdf97ad91a5f864621b:

    Any reason to change this? The next commit removes the line anyway.


    glozow commented at 11:14 am on October 5, 2020:
    I wanted to make it really obvious that the behavior is identical if ATMP ignores high fees at this point, that’s all :)
  123. MarcoFalke commented at 9:08 am on October 5, 2020: member

    Concept ACK 8784930dfaf4b981be6a3670c4298de2226a3989, left some questions 😈

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3Concept ACK 8784930dfaf4b981be6a3670c4298de2226a3989, left some questions 😈
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUjoWAwAwUFIuJljEL0JfJpPE2ETBPrIL0soXgmLHNBRHXgwPZ10WH2y6TrXAR7t
     8Xo9NESbW4Qp17O/PJWn9GZ/Oxl6UjbE9KF6XMqptXaUC0Jt2eGbNdvoHkVyj2Mqc
     9+pp76rgrKd+AU0G6YObPkDy0e8RCG66PX8u2JfLmCYoQA+cRcblg4GEFqWlBTZGw
    1043jQ9uGLCC3ShF2tbFfFJVeTRF4ywnyUGnmWno/vtqlWxGRtYSBdf8WSseUMHJz7
    11OzsWnvPyOajSCIFEHI2rJ7rutEXJQ4RXQMwCwC4RMkLTL5rsm+E3QHXmHUXOoT3r
    12chFpzmP7XqiEaPDEJE4zhmM8+aL3JUzGa6bBoc4EZyWLIf1imwHVREhVW9BZYV4d
    13AydaxhQ96c0uK5sIcfUWGZb4h8cv6tgSl/glM9gmUDewAg3JczHmwDCZv22cvcy6
    14ONYwgiB/0S8BpWx4OsIGVQvAjfBXtfpmkjD+ya5VoPTSCqIW+mYPpVWkGyRDjInm
    15EVIDY2IH
    16=1moD
    17-----END PGP SIGNATURE-----
    

    Timestamp of file with hash 41bb2c8bffc4df4e4705ffe7775dabf9aa5e03d062a86e3f08624699d44592ad -

  124. [rpc/node] check for high fee before ATMP in clients
    Check absurd fee in BroadcastTransaction and RPC,
    return TransactionError::MAX_FEE_EXCEEDED instead
    of TxValidationResult::TX_NOT_STANDARD because this
    is client preference, not a node-wide policy.
    8f1290c601
  125. scripted-diff: update max-fee-exceeded error message to include RPC
    -BEGIN VERIFY SCRIPT-
    sed -i 's/Fee exceeds maximum configured by \-\maxtxfee/Fee exceeds maximum configured by user (e.g. -maxtxfee, maxfeerate)/g' src/util/error.cpp test/functional/rpc_fundrawtransaction.py test/functional/rpc_psbt.py test/functional/rpc_rawtransaction.py test/functional/wallet_bumpfee.py test/functional/wallet_create_tx.py
    -END VERIFY SCRIPT-
    932564b9cf
  126. [validation] Remove absurdfee from accepttomempool
    Mempool behavior should not be user-specific.
    Checking that txfee is acceptable should be
    the responsibility of the wallet or client, not
    the mempool.
    b048b275d9
  127. glozow force-pushed on Oct 5, 2020
  128. jnewbery commented at 12:49 pm on October 5, 2020: member
    utACK b048b275d9711f70847afaea5450f17a0f7e673a
  129. MarcoFalke commented at 1:10 pm on October 5, 2020: member

    re-ACK b048b275d9 , only change is squashing one commit 🏦

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3re-ACK b048b275d9 , only change is squashing one commit 🏦
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUgq2gv8DSuQInycB5w6VMTMI7aBtPBoUDAX9yY9Z4Ruix1Im9Bf2fcJts6d9TxY
     8Abgr2NlSDVGkF4wNvOi3SnXkcYSto+Yt7vNat1HCORsRSzKiyshjKbXjJg/Rvi7K
     9S11vfpiA5YI87HoYgP/+QGUM912cLBt7DH78lnuskTVeN7B94PiyycU+WHgZw9Qm
    108WvFmyft5Tn5iEhOJufn7kB1B4Zdc4CbwTRsT1FpDyxoS+JQ2rvcpJgsXRf4l4Yn
    119wmZt637cYe4r5Nh4qyi56CgClbQvyyW1jgXZr3InFozkzPfXt9flbQJSKGGBDKy
    12c8sVErF1g2QYC5jjkvhJ6c5fpkCt9lvrudf5ol8Ouv/KVhGE9HBqtK/05GVjUefw
    13FfJaatnobEBXGE8JXVGvHWIazgjEd70MF1GSkZjxIyGHvvWU6IELJ2um0WP/1t4B
    14SDSUbDFECh4hhZQupjXmTKF5dxl2h3RgkAot+tQc4EgIt9mAg10+vL/jTWTaE1B6
    15l9ujyThH
    16=rq5V
    17-----END PGP SIGNATURE-----
    

    Timestamp of file with hash ef381a3560ba63d06ec8d836ebc9b3379cdaab5ad0d2d7c5ddbfb38d235c2c69 -

  130. LarryRuane commented at 6:46 pm on October 5, 2020: contributor
    re-ACK b048b275d9711f70847afaea5450f17a0f7e673a
  131. instagibbs commented at 1:55 am on October 7, 2020: member
    concept ack, this will require release notes since people tend to use sending failure codes/text in fine-grained ways and it may break downstream things
  132. fanquake added the label Needs release note on Oct 7, 2020
  133. instagibbs approved
  134. instagibbs commented at 2:10 am on October 7, 2020: member

    utACK https://github.com/bitcoin/bitcoin/pull/19339/commits/b048b275d9711f70847afaea5450f17a0f7e673a

    please don’t address my nit unless you have to rebase or something.

  135. fanquake merged this on Oct 7, 2020
  136. fanquake closed this on Oct 7, 2020

  137. fanquake commented at 3:00 am on October 7, 2020: member

    please don’t address my nit unless you have to rebase or something. @glozow can follow up with some release notes and address any remaining nits. Either in a new PR, or possibly as part of #20025.

  138. MarcoFalke commented at 7:15 am on October 7, 2020: member

    Instead of mentioning that absurdly-high-fee changed into max-fee-exceeded, I’d rather just change it back and prevent the release notes from bloating.

    Though, the reason (my reason) testmempoolaccept doesn’t return an error code because we can’t really guarantee stability of failure reasons across versions of Bitcoin Core. Similarly the string can’t be expected to be stable. So either we assume that and don’t mention the string change in the release notes, or we mention in the RPC help doc that the strings shouldn’t be assumed to be stable. In both cases, the release notes don’t mention the change.

  139. instagibbs commented at 12:15 pm on October 7, 2020: member

    Might be worth opening an issue on the topic because apparently the codes aren’t enough for downstream users. Recent breaks include LND, probably more.

    On Wed, Oct 7, 2020, 3:15 AM MarcoFalke notifications@github.com wrote:

    Instead of mentioning that absurdly-high-fee changed into max-fee-exceeded, I’d rather just change it back and prevent the release notes from bloating.

    Though, the reason (my reason) testmempoolaccept doesn’t return an error code because we can’t really guarantee stability of failure reasons across versions of Bitcoin Core. Similarly the string can’t be expected to be stable. So either we assume that and don’t mention the string change in the release notes, or we mention in the RPC help doc that the strings shouldn’t be assumed to be stable. In both cases, the release notes don’t mention the change.

    — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/pull/19339#issuecomment-704744784, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMAFU67TYMCCYJAG63SDSTSJQIRVANCNFSM4ODQNOOQ .

  140. MarcoFalke commented at 12:17 pm on October 7, 2020: member
    Mind sharing a link on further information how LND was broken by this?
  141. instagibbs commented at 12:22 pm on October 7, 2020: member

    Not this but text changes in general for mempool rpc messages. I’ll dig through my notes sometime when I have the time

    On Wed, Oct 7, 2020, 8:17 AM MarcoFalke notifications@github.com wrote:

    Mind sharing a link on further information how LND was broken by this?

    — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/pull/19339#issuecomment-704895935, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMAFU3CRQUFMVDXYTTEOBDSJRL7JANCNFSM4ODQNOOQ .

  142. glozow commented at 9:09 pm on October 8, 2020: member
    fwiw I think max-fee-exceeded is a more accurate error message - we’re checking the maxfeerate passed in by the user, absurd or not. I’m not sure how to determine whether something should be in release notes or not :/ I’ll just open a PR and we can talk there
  143. MarcoFalke removed the label Needs release note on Oct 9, 2020
  144. fanquake referenced this in commit e21b824386 on Oct 14, 2020
  145. sidhujag referenced this in commit 6d323e260d on Oct 15, 2020
  146. Fabcien referenced this in commit 664bafdc08 on Nov 16, 2021
  147. Fabcien referenced this in commit 3789536961 on Nov 16, 2021
  148. Fabcien referenced this in commit 497282c479 on Nov 16, 2021
  149. MarcoFalke locked this on Aug 14, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-06-29 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me