Use RPCHelpMan for all RPCs #14726

MarcoFalke commented at 2:09 AM on November 14, 2018: member

The resulting documentation should not change unless the type in the oneline-summary was previously incorrect. (E.g. string vs bool)

MarcoFalke added the label Docs on Nov 14, 2018

MarcoFalke added the label RPC/REST/ZMQ on Nov 14, 2018

DrahtBot commented at 2:16 AM on November 14, 2018: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#14707 ([RPC] Include coinbase transactions in receivedby RPCs by andrewtoth)
#14670 (http: Fix HTTP server shutdown by promag)
#14602 (Bugfix: Correctly calculate balances when min_conf is used, and for getbalance("*") by luke-jr)
#14459 (More RPC help description fixes by ch4ot1c)
#13756 (wallet: "avoid_reuse" wallet flag for improved privacy by kallewoof)
#13541 (wallet/rpc: sendrawtransaction maxfeerate by kallewoof)
#12096 ([rpc] [wallet] Allow specifying the output index when using bumpfee by kallewoof)
#11413 ([wallet] [rpc] sendtoaddress/sendmany: Add explicit feerate option by kallewoof)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

laanwj commented at 10:23 AM on November 14, 2018: member

Concept ACK!

in test/lint/lint-rpc-help.sh:6 in fa20188aa1 outdated

   0 | @@ -0,0 +1,17 @@
   1 | +#!/usr/bin/env bash
   2 | +#
   3 | +# Copyright (c) 2018 The Bitcoin Core developers
   4 | +# Distributed under the MIT software license, see the accompanying
   5 | +# file COPYING or http://www.opensource.org/licenses/mit-license.php.
   6 | +

practicalswift commented at 12:18 PM on November 14, 2018:

Add a description of what the linter does :-)

in test/lint/lint-rpc-help.sh:10 in fa20188aa1 outdated

   5 | +# file COPYING or http://www.opensource.org/licenses/mit-license.php.
   6 | +
   7 | +export LC_ALL=C
   8 | +
   9 | +EXIT_CODE=0
  10 | +non_autogenerated_help=$(grep --perl-regexp --null-data --only-matching 'runtime_error\(\n\s*".*\\n"\n' $(git ls-files -- "*.cpp"))

practicalswift commented at 12:18 PM on November 14, 2018:

I'm afraid BSD grep does not support --perl-regexp and --null-data :-\

ryanofsky commented at 3:58 PM on November 15, 2018:

This linter doesn't seem like very reliable. Maybe we can just keep it temporarily (add a comment to remove in 6 months)?

If we're are going to keep this, I would suggest adding a comment that describes what this regex is looking for. It seems like it just looking for std::runtime_error constructions with a specifically formatted type of multiline string, which could miss non-uses of RPCHelpMan, and also get in the way of providing detailed error messages in other cases.

practicalswift commented at 9:11 PM on November 15, 2018:

Perhaps the linter can just exit 0 with an information message if BSD grep is used?

practicalswift commented at 3:02 PM on November 14, 2018: contributor

Concept ACK

Extra credit for adding a linting which makes sure we'll never have to think about this again :-)

promag commented at 3:50 PM on November 14, 2018: member

Concept ACK.

MarcoFalke added this to the milestone 0.18.0 on Nov 14, 2018

MarcoFalke commented at 4:43 PM on November 14, 2018: member

For reference the diff in the resulting rpc doc is:

diff --git a/bumpfee b/bumpfee
index b47ef5b..892a1e6 100644
--- a/bumpfee
+++ b/bumpfee
@@ -1,4 +1,4 @@
-bumpfee "txid" ( options ) 
+bumpfee "txid" ( options )
 
 Bumps the fee of an opt-in-RBF transaction T, replacing it with a new transaction B.
 An opt-in RBF transaction with the given txid must be in the wallet.
diff --git a/estimaterawfee b/estimaterawfee
index 16f57ca..3b99c08 100644
--- a/estimaterawfee
+++ b/estimaterawfee
@@ -1,4 +1,4 @@
-estimaterawfee conf_target (threshold)
+estimaterawfee conf_target ( threshold )
 
 WARNING: This interface is unstable and may disappear or change!
 
diff --git a/estimatesmartfee b/estimatesmartfee
index fc53c26..af7dd35 100644
--- a/estimatesmartfee
+++ b/estimatesmartfee
@@ -1,4 +1,4 @@
-estimatesmartfee conf_target ("estimate_mode")
+estimatesmartfee conf_target ( "estimate_mode" )
 
 Estimates the approximate fee per kilobyte needed for a transaction to begin
 confirmation within conf_target blocks if possible and return the number of blocks
diff --git a/generatetoaddress b/generatetoaddress
index 4506040..7713819 100644
--- a/generatetoaddress
+++ b/generatetoaddress
@@ -1,4 +1,4 @@
-generatetoaddress nblocks address (maxtries)
+generatetoaddress nblocks "address" ( maxtries )
 
 Mine blocks immediately to a specified address (before the RPC call returns)
 
diff --git a/getblock b/getblock
index 5f94e32..a6e6654 100644
--- a/getblock
+++ b/getblock
@@ -1,4 +1,4 @@
-getblock "blockhash" ( verbosity ) 
+getblock "blockhash" ( verbosity )
 
 If verbosity is 0, returns a string that is serialized, hex-encoded data for block 'hash'.
 If verbosity is 1, returns an Object with information about block <hash>.
diff --git a/getmemoryinfo b/getmemoryinfo
index ab184e3..515a275 100644
--- a/getmemoryinfo
+++ b/getmemoryinfo
@@ -1,4 +1,4 @@
-getmemoryinfo ("mode")
+getmemoryinfo ( "mode" )
 Returns an object containing information about memory usage.
 Arguments:
 1. "mode" determines what kind of information is returned. This argument is optional, the default mode is "stats".
diff --git a/getmempoolancestors b/getmempoolancestors
index 406f659..9382b12 100644
--- a/getmempoolancestors
+++ b/getmempoolancestors
@@ -1,4 +1,4 @@
-getmempoolancestors txid ( verbose )
+getmempoolancestors "txid" ( verbose )
 
 If txid is in the mempool, returns all in-mempool ancestors.
 
diff --git a/getmempooldescendants b/getmempooldescendants
index 1198324..aacabbe 100644
--- a/getmempooldescendants
+++ b/getmempooldescendants
@@ -1,4 +1,4 @@
-getmempooldescendants txid ( verbose )
+getmempooldescendants "txid" ( verbose )
 
 If txid is in the mempool, returns all in-mempool descendants.
 
diff --git a/getmempoolentry b/getmempoolentry
index 468470a..cf95910 100644
--- a/getmempoolentry
+++ b/getmempoolentry
@@ -1,4 +1,4 @@
-getmempoolentry txid
+getmempoolentry "txid"
 
 Returns mempool data for given transaction
 
diff --git a/importprivkey b/importprivkey
index 6ee2531..cd361d5 100644
--- a/importprivkey
+++ b/importprivkey
@@ -1,4 +1,4 @@
-importprivkey "privkey" ( "label" ) ( rescan )
+importprivkey "privkey" ( "label" rescan )
 
 Adds a private key (as returned by dumpprivkey) to your wallet. Requires a new wallet backup.
 Hint: use importmulti to import more than one private key.
diff --git a/listreceivedbyaddress b/listreceivedbyaddress
index 95a4a26..baa068d 100644
--- a/listreceivedbyaddress
+++ b/listreceivedbyaddress
@@ -1,4 +1,4 @@
-listreceivedbyaddress ( minconf include_empty include_watchonly address_filter )
+listreceivedbyaddress ( minconf include_empty include_watchonly "address_filter" )
 
 List balances by receiving address.
 
diff --git a/listreceivedbylabel b/listreceivedbylabel
index 23961c3..40ecc52 100644
--- a/listreceivedbylabel
+++ b/listreceivedbylabel
@@ -1,4 +1,4 @@
-listreceivedbylabel ( minconf include_empty include_watchonly)
+listreceivedbylabel ( minconf include_empty include_watchonly )
 
 List received transactions by label.
 
diff --git a/listunspent b/listunspent
index bc3baf8..84235b4 100644
--- a/listunspent
+++ b/listunspent
@@ -1,4 +1,4 @@
-listunspent ( minconf maxconf ["address",...] include_unsafe {"minimumAmount":amount,"maximumAmount":amount,"maximumCount":n,"minimumSumAmount":amount} )
+listunspent ( minconf maxconf ["address",...] include_unsafe query_options )
 
 Returns array of unspent transaction outputs
 with between minconf and maxconf (inclusive) confirmations.
diff --git a/logging b/logging
index 43d6505..56bd6c5 100644
--- a/logging
+++ b/logging
@@ -1,4 +1,4 @@
-logging ( <include> <exclude> )
+logging ( "include" "exclude" )
 Gets and sets the logging configuration.
 When called without an argument, returns the list of categories with status that are currently being debug logged or not.
 When called with arguments, adds or removes categories from debug logging and return the lists above.
diff --git a/rescanblockchain b/rescanblockchain
index 8971533..fcd8024 100644
--- a/rescanblockchain
+++ b/rescanblockchain
@@ -1,4 +1,4 @@
-rescanblockchain ("start_height") ("stop_height")
+rescanblockchain ( start_height stop_height )
 
 Rescan the local blockchain for wallet related transactions.
 
diff --git a/sendmany b/sendmany
index ead3d41..79f54cb 100644
--- a/sendmany
+++ b/sendmany
@@ -1,4 +1,4 @@
-sendmany "" {"address":amount,...} ( minconf "comment" ["address",...] replaceable conf_target "estimate_mode")
+sendmany "" {"address":amount} ( minconf "comment" ["address",...] replaceable conf_target "estimate_mode" )
 
 Send multiple times. Amounts are double-precision floating point numbers.
 
diff --git a/sendtoaddress b/sendtoaddress
index 5b04077..3ca8683 100644
--- a/sendtoaddress
+++ b/sendtoaddress
@@ -1,4 +1,4 @@
-sendtoaddress "address" amount ( "comment" "comment_to" subtractfeefromamount replaceable conf_target "estimate_mode")
+sendtoaddress "address" amount ( "comment" "comment_to" subtractfeefromamount replaceable conf_target "estimate_mode" )
 
 Send an amount to a given address.
 
diff --git a/sethdseed b/sethdseed
index 6b4b16f..b2be7d2 100644
--- a/sethdseed
+++ b/sethdseed
@@ -1,4 +1,4 @@
-sethdseed ( "newkeypool" "seed" )
+sethdseed ( newkeypool "seed" )
 
 Set or generate a new HD wallet seed. Non-HD wallets will not be upgraded to being a HD wallet. Wallets that are already
 HD will have a new HD seed set so that new keys added to the keypool will be derived from this new seed.
diff --git a/submitblock b/submitblock
index d33307a..cdf8d0d 100644
--- a/submitblock
+++ b/submitblock
@@ -1,4 +1,4 @@
-submitblock "hexdata"  ( "dummy" )
+submitblock "hexdata" ( "dummy" )
 
 Attempts to submit new block to network.
 See https://en.bitcoin.it/wiki/BIP_0022 for full specification.
diff --git a/waitforblock b/waitforblock
index 20889b4..45de4eb 100644
--- a/waitforblock
+++ b/waitforblock
@@ -1,4 +1,4 @@
-waitforblock <blockhash> (timeout)
+waitforblock "blockhash" ( timeout )
 
 Waits for a specific new block and returns useful info about it.
 
diff --git a/waitfornewblock b/waitfornewblock
index ea184e3..f66d804 100644
--- a/waitfornewblock
+++ b/waitfornewblock
@@ -1,4 +1,4 @@
-waitfornewblock (timeout)
+waitfornewblock ( timeout )
 
 Waits for a specific new block and returns useful info about it.
 
diff --git a/walletcreatefundedpsbt b/walletcreatefundedpsbt
index 25e77bd..2260770 100644
--- a/walletcreatefundedpsbt
+++ b/walletcreatefundedpsbt
@@ -1,4 +1,4 @@
-walletcreatefundedpsbt [{"txid":"hex","vout":n,"sequence":n},...] [{"address":amount},{"data":"hex"},...] ( locktime {"changeAddress":"hex","changePosition":n,"change_type":"str","includeWatching":bool,"lockUnspents":bool,"feeRate":n,"subtractFeeFromOutputs":[int,...],"replaceable":bool,"conf_target":n,"estimate_mode":"str"} bip32derivs )
+walletcreatefundedpsbt [{"txid":"hex","vout":n,"sequence":n},...] [{"address":amount},{"data":"hex"},...] ( locktime options bip32derivs )
 
 Creates and funds a transaction in the Partially Signed Transaction format. Inputs will be added if supplied inputs are not enough
 Implements the Creator and Updater roles.

lint: Must use RPCHelpMan to generate the RPC docs fa520e72f7

MarcoFalke force-pushed on Nov 14, 2018

DrahtBot added the label Needs rebase on Nov 14, 2018

DrahtBot removed the label Needs rebase on Nov 14, 2018

in src/rpc/mining.cpp:341 in 777758a79d outdated

 329 | +                                },
 330 | +                                true},
 331 | +                        },
 332 | +                        true},
 333 | +                }}
 334 | +                .ToString() +

ryanofsky commented at 5:24 PM on November 14, 2018:

I probably should have asked this in #14530. But is there a plan for:

{"template_request", RPCArg::Type::OBJ,
    {
        {"mode", RPCArg::Type::STR, true},
        {"capabilities", RPCArg::Type::ARR,
            {
                {"support", RPCArg::Type::STR, true},
            },
            true},
        {"rules", RPCArg::Type::ARR,
            {
                {"support", RPCArg::Type::STR, true},
            },
            true},
    },
    true},

to not duplicate:

"1. template_request         (json object, optional) A json object in the following spec\n"
"     {\n"
"       \"mode\":\"template\"    (string, optional) This must be set to \"template\", \"proposal\" (see BIP 23), or omitted\n"
"       \"capabilities\":[     (array, optional) A list of strings\n"
"           \"support\"          (string) client side supported feature, 'longpoll', 'coinbasetxn', 'coinbasevalue', 'proposal', 'serverlist', 'workid'\n"
"           ,...\n"
"       ],\n"
"       \"rules\":[            (array, optional) A list of strings\n"
"           \"support\"          (string) client side supported softfork deployment\n"
"           ,...\n"
"       ]\n"
"     }\n"
"\n"

below?

MarcoFalke commented at 5:33 PM on November 14, 2018:

@ryanofsky You simply pass through the description string for each arg. Though the resulting diff will be larger because in the verbose description the types are not named in the same way across all docs, so I'd like to do that as a follow up pull request.

promag commented at 5:50 PM on November 14, 2018:

I had the same doubt as @ryanofsky. Before merging it would be great to see a POC of that @MarcoFalke.

MarcoFalke commented at 4:51 PM on November 15, 2018:

@ryanofsky @promag See the code in 92377d37cc0e59713874d5af406abfd84a7d972a, which produces

1. template_request    (json object, optional, default=m_default) m_description
     {                 (json object)
       "mode":"str"    (string, optional, default=m_default) m_description
       [               (json array)
         "support"     (string, optional, default=m_default) m_description
         ,...          
       ]               
       [               (json array)
         "support"     (string, optional, default=m_default) m_description
         ,...          
       ]               
       ,...            
     }

Obviously needs someone to pass in the members (m_description and m_default), like I mentioned above.

ryanofsky commented at 5:52 PM on November 14, 2018: member

I think in some of these cases, the previous output might be more readable than the new output. Examples:

-scantxoutset "action" [scanobjects,...]
+scantxoutset "action" [{"desc":"str","range":n},...]

-sendmany "" {"address":amount,...} ( minconf "comment" ["address",...] replaceable conf_target "estimate_mode")
+sendmany "dummy" {"address":amount} ( minconf "comment" ["address",...] replaceable conf_target "estimate_mode" )

In the first case instead of showing name of the argument, this now shows the structure of the argument without any context that would help understand it. Losing the name also seems like an inconvenience if you want to pass named arguments.

In the second case, the change goes in the opposite direction. Instead of showing the right dummy value to pass ("") it now shows the string "dummy" which would get rejected.

I don't know if these changes are desirable, but I think it would be desirable to add some basic flags or options to RPCArg to avoid making these changes in this PR. (I think they'd be fine for a much smaller followup PR.)

MarcoFalke force-pushed on Nov 14, 2018

Use RPCHelpMan for all RPCs fa91e8eda5

MarcoFalke force-pushed on Nov 14, 2018

MarcoFalke commented at 8:47 PM on November 14, 2018: member

Restored the inconsistencies. Resulting changes should be formatting-only.

fanquake commented at 8:32 AM on November 15, 2018: member

Concept ACK

in src/rpc/misc.cpp:464 in fa91e8eda5 outdated

 464 | -            "\nThe difference between echo and echojson is that echojson has argument conversion enabled in the client-side table in"
 465 | -            "bitcoin-cli and the GUI. There is no server-side difference."
 466 | -        );
 467 | +            RPCHelpMan{"echo|echojson ...",
 468 | +                "\nSimply echo back the input arguments. This command is for testing.\n"
 469 | +                "\nThe difference between echo and echojson is that echojson has argument conversion enabled in the client-side table in"

ryanofsky commented at 4:15 PM on November 15, 2018:

Not directly related (space was missing previously) but maybe add space between "in" and "bitcoin-cli"

ryanofsky approved

ryanofsky commented at 5:04 PM on November 15, 2018: member

ACK fa91e8eda541acdb78ca481b74605639f319c108

I dumped and diffed help with: <details><summary>Code</summary>

diff --git a/src/init.cpp b/src/init.cpp
index 3ab97be329..bc119ab1ca 100644
--- a/src/init.cpp
+++ b/src/init.cpp
@@ -46,6 +46,7 @@
 #include <util/system.h>
 #include <util/moneystr.h>
 #include <validationinterface.h>
+#include <wallet/wallet.h>
 #include <warnings.h>
 #include <walletinitinterface.h>
 #include <stdint.h>
@@ -1240,6 +1241,29 @@ bool AppInitMain(InitInterfaces& interfaces)
     for (const auto& client : interfaces.chain_clients) {
         client->registerRpcs();
     }
+    AddWallet(std::make_shared<CWallet>(*interfaces.chain, WalletLocation(), WalletDatabase::CreateDummy()));
+    JSONRPCRequest jreq;
+    jreq.fHelp = true;
+    for (const auto& entry : tableRPC.mapCommands) {
+        try {
+            entry.second->actor(jreq);
+        } catch (const std::exception& e) {
+            printf("===\n%s\n", e.what());
+            continue;
+        } catch (const UniValue& e) {
+            printf("===\n%s\n", e.write().c_str());
+            continue;
+        } catch (...) {
+            printf("===\n%s\n", entry.second->name.c_str());
+            fflush(stdout);
+            assert(0);
+        }
+        printf("===\n%s\n", entry.second->name.c_str());
+        fflush(stdout);
+        assert(0);
+    }
+    fflush(stdout);
+    abort();
     g_rpc_interfaces = &interfaces;
 #if ENABLE_ZMQ
     RegisterZMQRPCCommands(tableRPC);
diff --git a/src/rpc/server.h b/src/rpc/server.h
index 2d62a76f3c..6fbb35ec57 100644
--- a/src/rpc/server.h
+++ b/src/rpc/server.h
@@ -142,7 +142,7 @@ public:
  */
 class CRPCTable
 {
-private:
+public:
     std::map<std::string, const CRPCCommand*> mapCommands;
 public:
     CRPCTable();

</details>

<br/>Complete output ignoring whitespace was: <details><summary>Output</summary>

@@ -263 +263 @@
-createpsbt [{"txid":"id","vout":n},...] [{"address":amount},{"data":"hex"},...] ( locktime ) ( replaceable )
+createpsbt [{"txid":"hex","vout":n,"sequence":n},...] [{"address":amount},{"data":"hex"},...] ( locktime replaceable )
@@ -2022 +2022 @@
-gettxoutproof ["txid",...] ( blockhash )
+gettxoutproof ["txid",...] ( "blockhash" )
@@ -2537 +2537 @@
-listunspent ( minconf maxconf  ["addresses",...] [include_unsafe] [query_options])
+listunspent ( minconf maxconf ["address",...] include_unsafe {"minimumAmount":amount,"maximumAmount":amount,"maximumCount":n,"minimumSumAmount":amount} )
@@ -2640 +2640 @@
-lockunspent unlock ([{"txid":"txid","vout":n},...])
+lockunspent unlock ( [{"txid":"hex","vout":n},...] )
@@ -3131 +3131 @@
-signrawtransactionwithkey "hexstring" ["privatekey1",...] ( [{"txid":"id","vout":n,"scriptPubKey":"hex","redeemScript":"hex"},...] sighashtype )
+signrawtransactionwithkey "hexstring" ["privatekey",...] ( [{"txid":"hex","vout":n,"scriptPubKey":"hex","redeemScript":"hex","amount":amount},...] "sighashtype" )
@@ -3186 +3186 @@
-signrawtransactionwithwallet "hexstring" ( [{"txid":"id","vout":n,"scriptPubKey":"hex","redeemScript":"hex"},...] sighashtype )
+signrawtransactionwithwallet "hexstring" ( [{"txid":"hex","vout":n,"scriptPubKey":"hex","redeemScript":"hex","amount":amount},...] "sighashtype" )
@@ -3484 +3484 @@
-walletcreatefundedpsbt [{"txid":"id","vout":n},...] [{"address":amount},{"data":"hex"},...] ( locktime ) ( replaceable ) ( options bip32derivs )
+walletcreatefundedpsbt [{"txid":"hex","vout":n,"sequence":n},...] [{"address":amount},{"data":"hex"},...] ( locktime {"changeAddress":"hex","changePosition":n,"change_type":"str","includeWatching":bool,"lockUnspents":bool,"feeRate":n,"subtractFeeFromOutputs":[int,...],"replaceable":bool,"conf_target":n,"estimate_mode":"str"} bip32derivs )

</details>

rpc: Documentation fixups fa5e0452e8

MarcoFalke commented at 5:19 PM on November 15, 2018: member

Added a separate commit for the documentation fixups requested by @practicalswift and @ryanofsky

ryanofsky approved

ryanofsky commented at 5:32 PM on November 15, 2018: member

utACK fa5e0452e875a7ca6bf6fe61fdd652d341eece40. Only change is new commit with whitespace/comments.

laanwj commented at 8:42 AM on November 23, 2018: member

utACK fa5e0452e875a7ca6bf6fe61fdd652d341eece40

This is a nice step forward in the consistency of RPC help.

laanwj merged this on Nov 23, 2018

laanwj closed this on Nov 23, 2018

laanwj referenced this in commit a0d86815cd on Nov 23, 2018

MarcoFalke deleted the branch on Nov 23, 2018

Sjors commented at 3:16 PM on November 27, 2018: member

Nice! Does this allow us to get rid of CRPCConvertParam, which is quite easy to forget?

MarcoFalke commented at 7:47 PM on December 3, 2018: member

@Sjors Not so quick. It could potentially help with those, but I'd suggest to first focus on the formatting and getting all parameters in correctly, which is being done in #14796. I have some fun changes piled up locally, but I can only do them after #14796.

deadalnix referenced this in commit 3a84ae8b0c on Mar 25, 2020

ftrader referenced this in commit d21e79aa47 on Dec 1, 2020

MarcoFalke locked this on Sep 8, 2021