include verbose “reject-details” field in testmempoolaccept response #28121

pull pinheadmz wants to merge 4 commits into bitcoin:master from pinheadmz:tma-debug changing 5 files +60 −16
  1. pinheadmz commented at 2:47 pm on July 21, 2023: member

    Adds a new field reject-details in testmempoolaccept responses to include m_debug_message from ValidationState. This string is part of the complete error message thrown by the mempool in response to sendrawtransaction.

    The extra verbosity is helpful to consumers of testmempoolaccept, which is sort of a debug tool anyway.

    example:

    0{
    1  "txid": "07d7a59a7bdad4c3a5070659ea04147c9b755ad9e173c52b6a38e017abf0f5b8",
    2  "wtxid": "5dc243b1b92ee2f5a43134eb3e23449be03d1abb3d7f3c03c836ed0f13c50185",
    3  "allowed": false,
    4  "reject-reason": "insufficient fee",
    5  "reject-details": "rejecting replacement 07d7a59a7bdad4c3a5070659ea04147c9b755ad9e173c52b6a38e017abf0f5b8; new feerate 0.00300000 BTC/kvB <= old feerate 0.00300000 BTC/kvB"
    6}
    
  2. DrahtBot commented at 2:47 pm on July 21, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK glozow, instagibbs, kristapsk, jonatack

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #30964 (Disable RBF Rule 2 by arik-so)
    • #28676 ([WIP] Cluster mempool implementation by sdaftuar)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. pinheadmz marked this as ready for review on Jul 21, 2023
  4. glozow added the label RPC/REST/ZMQ on Jul 24, 2023
  5. luke-jr commented at 6:35 pm on July 25, 2023: member
    Why not pass the details in m_debug_message and add that as a new field?
  6. pinheadmz commented at 1:48 pm on July 26, 2023: member

    Why not pass the details in m_debug_message and add that as a new field?

    My thinking was to match exactly the single combined string returned by sendrawtransaction. But if adding the new field gets your ACK @luke-jr I’ll make the change ;-)

  7. glozow commented at 9:26 am on August 30, 2023: member

    Concept ACK to providing the debug string, I’m sure it’d be helpful for users. Would be nice to do this for submitpackage as well.

    I’m not sure if we try to keep strings for reject-reason stable at all. If we do, I’d be in favor of adding a new field error_msg with the full string instead of a separate field with just the debug message.

  8. pinheadmz commented at 3:54 pm on September 1, 2023: member

    I’m not sure if we try to keep strings for reject-reason stable at all.

    You mean like how block reject reasons are defined in BIP22?

  9. in test/functional/feature_rbf.py:126 in bcdc3646c3 outdated
    122-        assert_raises_rpc_error(-26, "insufficient fee", self.nodes[0].sendrawtransaction, tx.serialize().hex(), 0)
    123+        debug_message = f"insufficient fee, rejecting replacement {tx.hash}; new feerate"
    124+        assert_raises_rpc_error(-26, debug_message, self.nodes[0].sendrawtransaction, tx.serialize().hex(), 0)
    125+        res = self.nodes[0].testmempoolaccept(rawtxs=[tx.serialize().hex()])[0]
    126+        assert not res['allowed']
    127+        assert debug_message in res['reject-reason']
    


    jonatack commented at 9:45 pm on September 7, 2023:

    In commit bafd175ca898c3ff74ae7927a3e9e3607ea3c8cd, can make a helper for this.

     0--- a/test/functional/feature_rbf.py
     1+++ b/test/functional/feature_rbf.py
     2@@ -107,6 +107,11 @@ class ReplaceByFeeTest(BitcoinTestFramework):
     3 
     4         return self.wallet.get_utxo(txid=tx["txid"], vout=tx["sent_vout"])
     5 
     6+    def assert_rejected_by_testmempoolaccept(self, tx_hex, msg):
     7+        result = self.nodes[0].testmempoolaccept(rawtxs=[tx_hex])[0]
     8+        assert not result["allowed"]
     9+        assert msg in result["reject-reason"]
    10+
    11     def test_simple_doublespend(self):
    12         """Simple doublespend"""
    13         # we use MiniWallet to create a transaction template with inputs correctly set,
    14@@ -121,9 +126,7 @@ class ReplaceByFeeTest(BitcoinTestFramework):
    15         # This will raise an exception due to insufficient fee
    16         debug_message = f"insufficient fee, rejecting replacement {tx.hash}; new feerate"
    17         assert_raises_rpc_error(-26, debug_message, self.nodes[0].sendrawtransaction, tx.serialize().hex(), 0)
    18-        res = self.nodes[0].testmempoolaccept(rawtxs=[tx.serialize().hex()])[0]
    19-        assert not res['allowed']
    20-        assert debug_message in res['reject-reason']
    21+        self.assert_rejected_by_testmempoolaccept(tx.serialize().hex(), debug_message)
    22 
    23         # Extra 0.1 BTC fee
    24         tx.vout[0].nValue -= int(0.1 * COIN)
    25@@ -168,9 +171,7 @@ class ReplaceByFeeTest(BitcoinTestFramework):
    26         # This will raise an exception due to insufficient fee
    27         debug_message = f"insufficient fee, rejecting replacement {dbl_tx['txid']}; less fees than conflicting txs"
    28         assert_raises_rpc_error(-26, debug_message, self.nodes[0].sendrawtransaction, dbl_tx["hex"], 0)
    29-        res = self.nodes[0].testmempoolaccept(rawtxs=[dbl_tx["hex"]])[0]
    30-        assert not res['allowed']
    31-        assert debug_message in res['reject-reason']
    32+        self.assert_rejected_by_testmempoolaccept(dbl_tx["hex"], debug_message)
    33 
    34         # Accepted with sufficient fee
    35         dbl_tx["tx"].vout[0].nValue = int(0.1 * COIN)
    36@@ -307,9 +308,7 @@ class ReplaceByFeeTest(BitcoinTestFramework):
    37         # This will raise an exception
    38         debug_message = f"bad-txns-spends-conflicting-tx, {tx2['txid']} spends conflicting transaction {tx1a['txid']}"
    39         assert_raises_rpc_error(-26, debug_message, self.nodes[0].sendrawtransaction, tx2["hex"], 0)
    40-        res = self.nodes[0].testmempoolaccept(rawtxs=[tx2["hex"]])[0]
    41-        assert not res['allowed']
    42-        assert debug_message in res['reject-reason']
    43+        self.assert_rejected_by_testmempoolaccept(tx2["hex"], debug_message)
    44 
    45         # Spend tx1a's output to test the indirect case.
    46         tx1b_utxo = self.wallet.send_self_transfer(
    47@@ -349,9 +348,7 @@ class ReplaceByFeeTest(BitcoinTestFramework):
    48         # This will raise an exception
    49         debug_message = f"replacement-adds-unconfirmed, replacement {tx2['txid']} adds unconfirmed input, idx"
    50         assert_raises_rpc_error(-26, debug_message, self.nodes[0].sendrawtransaction, tx2["hex"], 0)
    51-        res = self.nodes[0].testmempoolaccept(rawtxs=[tx2["hex"]])[0]
    52-        assert not res['allowed']
    53-        assert debug_message in res['reject-reason']
    54+        self.assert_rejected_by_testmempoolaccept(tx2["hex"], debug_message)
    55 
    56     def test_too_many_replacements(self):
    57         """Replacements that evict too many transactions are rejected"""
    58@@ -394,9 +391,7 @@ class ReplaceByFeeTest(BitcoinTestFramework):
    59         # This will raise an exception
    60         debug_message = f"too many potential replacements, rejecting replacement {double_tx['txid']}; too many potential replacements"
    61         assert_raises_rpc_error(-26, debug_message, self.nodes[0].sendrawtransaction, double_tx["hex"], 0)
    62-        res = self.nodes[0].testmempoolaccept(rawtxs=[double_tx["hex"]])[0]
    63-        assert not res['allowed']
    64-        assert debug_message in res['reject-reason']
    65+        self.assert_rejected_by_testmempoolaccept(double_tx["hex"], debug_message)
    66 
    67         # If we remove an input, it should pass
    68         double_tx["tx"].vin.pop()
    69@@ -720,9 +715,7 @@ class ReplaceByFeeTest(BitcoinTestFramework):
    70         tx.rehash()
    71         debug_message = f"insufficient fee, rejecting replacement {tx.hash}; not enough additional fees to relay"
    72         assert_raises_rpc_error(-26, debug_message, self.nodes[0].sendrawtransaction, tx.serialize().hex())
    73-        res = self.nodes[0].testmempoolaccept(rawtxs=[tx.serialize().hex()])[0]
    74-        assert not res['allowed']
    75-        assert debug_message in res['reject-reason']
    76+        self.assert_rejected_by_testmempoolaccept(tx.serialize().hex(), debug_message)
    77 
    78     def test_fullrbf(self):
    
  10. jonatack commented at 9:50 pm on September 7, 2023: member
    lgtm ACK bcdc3646c3e24930433e30b6363ca2d6f6fb9198 modulo the question of a new field vs the appending to the reject_reason output.
  11. DrahtBot added the label CI failed on Sep 15, 2023
  12. glozow commented at 12:16 pm on September 26, 2023: member

    I’m not sure if we try to keep strings for reject-reason stable at all.

    You mean like how block reject reasons are defined in BIP22?

    Yeah exactly. Like if somebody has a fee bumper that’s triggered by reject-reason == "mempool min fee not met" or something.

  13. pinheadmz commented at 8:13 pm on September 26, 2023: member

    Yeah exactly. Like if somebody has a fee bumper that’s triggered by reject-reason == "mempool min fee not met" or something.

    Yeah ok so, the detailed message is appended to the original so “somebody” would just need to change their == to a “starts with” or “includes” or regex checker. In my opinion, this API-breaking change is why we have major version numbers and release notes. So I’m not sure how we decide whats precious or not. Either way is fine with me.

  14. DrahtBot commented at 6:38 am on September 27, 2023: contributor
    Needs rebase if still relevant
  15. pinheadmz force-pushed on Oct 17, 2023
  16. pinheadmz commented at 7:48 pm on October 17, 2023: member

    Needs rebase if still relevant

    Done 🙏

  17. pinheadmz force-pushed on Oct 18, 2023
  18. DrahtBot removed the label CI failed on Oct 18, 2023
  19. instagibbs commented at 2:50 pm on October 19, 2023: member

    concept ACK

    unfortunately I know people do use reject-reason strings for various informational purposes. Would be a good project to gather those uses and maybe think of better error codes to capture these?

  20. pinheadmz commented at 2:57 pm on October 19, 2023: member
    @instagibbs thanks for taking a look! Not sure what you mean about gathering uses though. Are you agreeing with @luke-jr that the debug messages should have their own field in TMA response object and leave reject-reason alone?
  21. instagibbs commented at 2:58 pm on October 19, 2023: member

    Not sure what you mean about gathering uses though.

    Looking at why error codes aren’t enough for production uses

  22. pinheadmz commented at 3:08 pm on October 19, 2023: member

    Looking at why error codes aren’t enough for production uses

    I’m working on an RBF-batcher with CardCoins and we were getting insufficient fee errors in test from testmempoolaccept(), which could be reported from two different places. Inspiration for this PR came from that ambiguity.

    https://github.com/bitcoin/bitcoin/blob/6e721c923c87abdb8d99674093d08d8c17bc52c2/src/validation.cpp#L983

    https://github.com/bitcoin/bitcoin/blob/6e721c923c87abdb8d99674093d08d8c17bc52c2/src/validation.cpp#L1004

  23. kristapsk commented at 7:41 am on December 20, 2023: contributor
    Concept ACK
  24. DrahtBot added the label CI failed on Jan 7, 2024
  25. achow101 requested review from 0xB10C on Apr 9, 2024
  26. achow101 requested review from theStack on Apr 9, 2024
  27. 0xB10C commented at 3:08 pm on April 9, 2024: contributor
    I have a slight preference to keep the reject-reason string as is, but like the idea of having more information on rejections. In https://github.com/0xB10C/find-non-standard-tx and for https://bitcoin-data.github.io/non-standard-transactions/ I use the short reject-reason. Maybe add a new field for details?
  28. pinheadmz renamed this:
    include verbose debug messages in testmempoolaccept reject-reason
    include verbose "debug-message" field in testmempoolaccept response
    on Apr 16, 2024
  29. pinheadmz force-pushed on Apr 16, 2024
  30. pinheadmz commented at 7:59 pm on April 16, 2024: member
    Thanks for taking a look, @0xB10C. I’ve refactored the PR to add a new field instead of changing anything already in place. Edited PR title and description. I made the tests super clear that the complete string is returned by submitrawtransaction whereas its split into two fields in testmempoolaccept
  31. pinheadmz force-pushed on Apr 16, 2024
  32. DrahtBot removed the label CI failed on Apr 16, 2024
  33. 0xB10C referenced this in commit 087866aab2 on Apr 18, 2024
  34. 0xB10C commented at 10:21 am on May 5, 2024: contributor

    I’ve played around a bit with this in https://github.com/0xB10C/find-non-standard-tx/pull/2.

    I’ve seen output for the following reasons:

    • too-long-mempool-chain: e.g.
      • exceeds descendant size limit for tx 9f95e53da39ece93ceba3cc78a443a1aef0fc656bf921bbd2aee7f6533f0604f [limit: 101000]
      • exceeds ancestor size limit [limit: 101000]
      • 2256302dedb181f83bc51248215e51f680f77d07e11ad7e9e34c2e483ad10e7b [limit: 26]
    • min relay fee not met: e.g. 0 < 200

    Other reasons like e.g. scriptpubkey, tx-size, dust, missing-inputs, multi-op-return have an empty debug-message.

    I haven’t yet taken a closer look at the code.

  35. pinheadmz commented at 3:42 pm on May 6, 2024: member

    Other reasons like e.g. scriptpubkey, tx-size, dust, missing-inputs, multi-op-return have an empty debug-message.

    debug-message was empty like ""? I’d expect the key to not be present at all:

    0if (!state.GetDebugMessage().empty()) result_inner.pushKV("debug-message", state.GetDebugMessage());
    
  36. 0xB10C commented at 4:31 pm on May 6, 2024: contributor

    debug-message was empty like ""? I’d expect the key to not be present at all:

    You’re right. I should have been clearer in my comment: the field wasn’t present. My tooling handles the missing field by setting the value to “”.

  37. DrahtBot added the label Needs rebase on May 13, 2024
  38. pinheadmz force-pushed on May 14, 2024
  39. pinheadmz commented at 7:08 pm on May 14, 2024: member
    force-push to fix merge conflict which also looked like a git mistake on my part in the first place 😬
  40. DrahtBot removed the label Needs rebase on May 14, 2024
  41. in src/rpc/mempool.cpp:144 in d2917e7e0b outdated
    140@@ -141,6 +141,7 @@ static RPCHelpMan testmempoolaccept()
    141                         }},
    142                     }},
    143                     {RPCResult::Type::STR, "reject-reason", /*optional=*/true, "Rejection string (only present when 'allowed' is false)"},
    144+                    {RPCResult::Type::STR, "debug-message", /*optional=*/true, "Rejection details (only present when 'allowed' is false and a message is provided)"},
    


    jonatack commented at 10:48 pm on May 16, 2024:

    c1be9d4

    • it might be more user-friendly to call the field reject-details?

      edit: (also, per #28121 (comment), the rationale for adding this field isn’t only for debugging)

    • maybe just me, but a message is provided is potentially confusing, as it might sound like user input

    0                    {RPCResult::Type::STR, "reject-details", /*optional=*/true, "Rejection details (only present when 'allowed' is false and rejection details exist)"},
    
  42. in src/rpc/mempool.cpp:143 in d2917e7e0b outdated
    140@@ -141,6 +141,7 @@ static RPCHelpMan testmempoolaccept()
    141                         }},
    142                     }},
    143                     {RPCResult::Type::STR, "reject-reason", /*optional=*/true, "Rejection string (only present when 'allowed' is false)"},
    


    jonatack commented at 10:48 pm on May 16, 2024:

    https://github.com/bitcoin/bitcoin/commit/c1be9d40f8c3044587d1d3480ccbedfc427951fe while touching the next line, string here might be a relic from when the field type wasn’t auto-generated in the help, ISTM reason would be clearer

    0                    {RPCResult::Type::STR, "reject-reason", /*optional=*/true, "Rejection reason (only present when 'allowed' is false)"},
    
  43. in doc/release-notes-28121.md:1 in d2917e7e0b outdated
    0@@ -0,0 +1,5 @@
    1+RPC
    


    jonatack commented at 10:54 pm on May 16, 2024:
    d2917e7e0b342509d50325cf7f00da15d81b3c65 unsure if Updated RPCs would be better here, feel free to ignore
  44. in doc/release-notes-28121.md:4 in d2917e7e0b outdated
    0@@ -0,0 +1,5 @@
    1+RPC
    2+---
    3+
    4+`testmempoolaccept` response will now include verbose field "debug-message"
    


    jonatack commented at 10:57 pm on May 16, 2024:

    d2917e7e0b342509d50325cf7f00da15d81b3c65 the new field won’t always be included (and why “verbose”?)

    0The RPC `testmempoolaccept` response now includes a "reject-details" field in some cases
    
  45. jonatack commented at 11:01 pm on May 16, 2024: member
    Concept ACK d2917e7e0b342509d50325cf7f00da15d81b3c65
  46. pinheadmz force-pushed on Jun 6, 2024
  47. pinheadmz commented at 8:04 pm on June 6, 2024: member
    Thanks for reviewing @jonatack I like the idea of “reject-details” and addressed your other comments as well
  48. DrahtBot added the label CI failed on Jul 4, 2024
  49. DrahtBot commented at 6:14 pm on July 10, 2024: contributor
    0�[0m�[0;31mmempool_package_rbf.py                                   | ✖ Failed  | 14 s
    
  50. rpc: include verbose debug-message field in testmempoolaccept response 0959273869
  51. rbf: remove unecessary newline at end of error string 76159062b9
  52. test: cover testmempoolaccept debug-message in RBF test 375aa5c301
  53. doc: add release note about testmempoolaccept debug-message 64953df796
  54. pinheadmz renamed this:
    include verbose "debug-message" field in testmempoolaccept response
    include verbose "reject-details" field in testmempoolaccept response
    on Jul 12, 2024
  55. pinheadmz force-pushed on Jul 12, 2024
  56. pinheadmz commented at 6:52 pm on July 12, 2024: member

    Thanks @DrahtBot good bot, have a biscuit.

    Rebased on master to fix silent conflict that failed functional test. Also updated PR title and description since the name of the new field has changed

  57. DrahtBot removed the label CI failed on Jul 12, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-28 22:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me