cli: Handle arguments that can be either JSON or string #33230

pull achow101 wants to merge 2 commits into bitcoin:master from achow101:cli-strong-or-json changing 8 files +37 −32
  1. achow101 commented at 9:20 pm on August 20, 2025: member
    There are some RPCs where the argument can be either JSON that needs to be parsed, or a string that we can pass straight through. However, bitcoin-cli would always parse those arguments as JSON which makes for some cumbersome argument passing when using those RPCs. Notably, hash_or_height in getblockstats and gettxoutsetinfo do this, and results in a more cumbersome command of bitcoin-cli getblockstats '"<hash>"'. Otherwise, using a normal invocation of bitcoin-cli getblockstats <hash> results in error: Error parsing JSON. This PR marks those particular options as also being a string so that when bitcoin-cli fails to parse the argument as JSON, it will assume that the argument is a string and pass it straight through.
  2. DrahtBot added the label Scripts and tools on Aug 20, 2025
  3. DrahtBot commented at 9:20 pm on August 20, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33230.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK rkrux, ryanofsky, mzumsande
    Approach ACK w0xlt
    Stale ACK pablomartin4btc, kannapoix, kevkevinpal, enirox001, nervana21

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #33192 (refactor: unify container presence checks (without PR conflicts) by l0rinc)
    • #32928 (test: add logging to mock external signers by Sjors)
    • #32821 (rpc: Handle -named argument parsing where ‘=’ character is used by zaidmstrr)
    • #32784 (wallet: derivehdkey RPC to get xpub at arbitrary path by Sjors)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. achow101 force-pushed on Aug 20, 2025
  5. DrahtBot added the label CI failed on Aug 20, 2025
  6. DrahtBot commented at 9:39 pm on August 20, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/runs/48530907189 LLM reason (✨ experimental): The CI failure is caused by a linting error detected by ruff due to an unused import in the Python test code.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  7. achow101 force-pushed on Aug 20, 2025
  8. ryanofsky commented at 9:53 pm on August 20, 2025: contributor
    Seems like a reasonable change. I wonder if an alternate solution might have the client treat hash_or_height as a string parameter instead of a JSON parameter, and update the server to accept height strings? But I guess that might not be backwards compatible if there are old scripts calling bitcoin-cli with extra quotes around the hash string.
  9. polespinasa commented at 10:00 pm on August 20, 2025: contributor

    I wonder if an alternate solution might have the client treat hash_or_height as a string parameter instead of a JSON parameter, and update the server to accept height strings?

    Even if there is, there are other use cases taking lists and strings as parameters. See: https://github.com/bitcoin/bitcoin/pull/32468

  10. w0xlt commented at 11:28 pm on August 20, 2025: contributor
    Approach ACK
  11. DrahtBot removed the label CI failed on Aug 20, 2025
  12. pablomartin4btc commented at 6:32 pm on August 21, 2025: member

    ACK aabf1f6093892d372544c8b5d55d260699dab69c

    This is quite practical, went thru these errors.

  13. DrahtBot requested review from w0xlt on Aug 21, 2025
  14. in test/functional/rpc_help.py:35 in aabf1f6093 outdated
    31@@ -32,7 +32,7 @@ def process_mapping(fname):
    32                 if line.startswith('};'):
    33                     in_rpcs = False
    34                 elif '{' in line and '"' in line:
    35-                    m = re.search(r'{ *("[^"]*"), *([0-9]+) *, *("[^"]*") *},', line)
    36+                    m = re.search(r'{ *("[^"]*"), *([0-9]+) *, *("[^"]*") *(, \/\*also_string=\*\/true *)?},', line)
    


    luke-jr commented at 11:50 am on August 22, 2025:
    Might be a good idea to actually test this is correct?

    achow101 commented at 7:32 pm on August 22, 2025:
    This is here because the test fails without it.

    ryanofsky commented at 5:34 pm on September 22, 2025:

    re: #33230 (review)

    This is here because the test fails without it.

    It seems like the test could be improved, but it would expand the PR a little. I think ideally type of hash_or_height rollback parameters server side would not be NUM, but a more accurate type like NUM_OR_STR. (#32468 could use ARR_OR_STR.)

    The server could return this information back to the test either by modifying the current type column:

    https://github.com/bitcoin/bitcoin/blob/34fefb633584ecd803b01209756f2bef412f1cb1/src/rpc/util.cpp#L846-L847

    Or by adding an extra column. And the test could confirm also_string is set when it should be set and not otherwise, based on the server type.

    I think this would not be a big change, and it could also be useful for making RPC documentation clearer for these parameters.


    pablomartin4btc commented at 2:51 pm on September 24, 2025:
    Following your suggestion @ryanofsky… If a new type is added (NUM_OR_STR or even ARR_OR_STR from #32468), is also_string still needed?

    ryanofsky commented at 3:19 pm on September 24, 2025:

    re: #33230 (review)

    Following your suggestion @ryanofsky… If a new type is added (NUM_OR_STR or even ARR_OR_STR from #32468), is also_string still needed?

    Yes my suggestion above is just a way of improving this python test, which makes sure the client-side type table in src/rpc/client.cpp is consistent with type information in the server RPC code. The other file src/rpc/util.cpp is a server-side file while provides information about the server parameter types to the python test. So an also_string column (or equivalent) is still needed in src/rpc/client.cpp even if the test is improved.


    achow101 commented at 6:11 pm on September 24, 2025:
    I think there are a number of other type checking improvements that we can do at the same time, but I would prefer all of that separately.
  15. luke-jr commented at 11:55 am on August 22, 2025: member

    I guess this basically works for the existing params, since a block hash is never valid JSON (which forbids leading zeros), but could be dangerous to use for new parameters that are ambiguous.

    With that in mind, it might be worth instead having a new RPCResult::Type::BLOCK_HEIGHT_OR_HASH type to sanity check this. Or at least a warning comment (but I don’t know where you’d put it to ensure it’s seen).

  16. luke-jr referenced this in commit 111938ce2c on Aug 22, 2025
  17. achow101 commented at 7:39 pm on August 22, 2025: member

    I guess this basically works for the existing params, since a block hash is never valid JSON (which forbids leading zeros), but could be dangerous to use for new parameters that are ambiguous.

    I don’t think we have any types where that would be the case. But this is also something that needs to be turned on for a parameter explicitly, so I don’t expect this to be problematic.

    With that in mind, it might be worth instead having a new RPCResult::Type::BLOCK_HEIGHT_OR_HASH type to sanity check this.

    Part of the impetus for this is #32468 where the parameter can be either is a JSON array or a string, so something that is more generic is better.

    Or at least a warning comment (but I don’t know where you’d put it to ensure it’s seen).

    I think with it being default off makes that not really necessary.

  18. nervana21 commented at 2:34 am on August 23, 2025: contributor
    Concept ACK
  19. kannapoix commented at 4:17 am on August 23, 2025: none

    ACK aabf1f6093892d372544c8b5d55d260699dab69c

    nit: It may be a good idea to also update the example command in the help message of getblockstats. Given that this PR eliminates the need for quotes around arguments, it would be good to update the example accordingly by using plain text instead of the quoted format. https://github.com/bitcoin/bitcoin/blob/73220fc0f958f9b65f66cf0cf042af220b312fc6/src/rpc/blockchain.cpp#L1943

  20. kevkevinpal commented at 1:19 pm on August 26, 2025: contributor

    ACK 111938c

    This change makes sense. I’ve definitely run into this issue, and being able to use a regular string is less confusing

    I checked that with the change to src/rpc/client.cpp, the old tests using convert_to_json_for_cli still passed.

    I was wondering if it makes sense to keep one test using convert_to_json_for_cli to ensure that these parameters can still be used as JSON for backward compatibility.

  21. in src/rpc/client.cpp:195 in 009c972c09 outdated
    188@@ -188,10 +189,10 @@ static const CRPCConvertParam vRPCConvertParams[] =
    189     { "gettxout", 1, "n" },
    190     { "gettxout", 2, "include_mempool" },
    191     { "gettxoutproof", 0, "txids" },
    192-    { "gettxoutsetinfo", 1, "hash_or_height" },
    193+    { "gettxoutsetinfo", 1, "hash_or_height", /*also_string=*/true },
    194     { "gettxoutsetinfo", 2, "use_index"},
    195     { "dumptxoutset", 2, "options" },
    196-    { "dumptxoutset", 2, "rollback" },
    197+    { "dumptxoutset", 2, "rollback", /*also_string=*/true },
    


    nervana21 commented at 2:07 am on August 29, 2025:

    Will this entry have an effect? In the positional map members, the key ("dumptxoutset", 2) is first set to false. My understanding is that subsequent encounters of the same key are ignored by std::map::emplace. Is this the desired behavior, or should the subsequent value overwrite the first?

    0{ "dumptxoutset", 2, "options", /*also_string=*/false }, // inserted
    1{ "dumptxoutset", 2, "rollback", /*also_string=*/true }, // ignored or used to overwrite?
    

    achow101 commented at 8:05 pm on September 9, 2025:

    Yes, the effect it has is on the named parameter. rollback cannot actually be passed as a positional parameter since it is actually part of the options object which would have to be passed in position 2. However, if rollback is passed by name, then we need to apply the conversion.

    Order does matter here in that the options parameter needs to come first so that the positional is not interpreted as a string.

  22. DrahtBot requested review from nervana21 on Aug 29, 2025
  23. enirox001 commented at 10:55 pm on August 30, 2025: contributor
    ACK aabf1f6 — improves CLI ergonomics for mixed JSON/string RPC params
  24. nervana21 commented at 11:57 pm on September 10, 2025: contributor
    tACK aabf1f6093892d372544c8b5d55d260699dab69c
  25. DrahtBot requested review from nervana21 on Sep 10, 2025
  26. ryanofsky approved
  27. ryanofsky commented at 6:18 pm on September 22, 2025: contributor

    Code review ACK aabf1f6093892d372544c8b5d55d260699dab69c. This should be a nice usability improvement. I left a suggestion for making the test stronger and confirming client and server types are consistent, and I think it would not be hard to implement, but it’s not critical.

    I do think this approach is a pretty reasonable way to avoid the need for calling bitcoin-cli with extra quotes around ParseHashOrHeight parameters here, and around addresses in #32468. And it’s probably the best approach, though other approaches are possible:

    1. As suggested earlier, another approach could be to make the client just always send these parameters as strings, instead of attempting to parse them as json, and leave it up to the server to convert the strings to desired types. This solution might not be appealing, though, because it would allow clients that do supports types like python and rust to pass arguments as strings when more properly they should be typed.

    2. Conversely, the client could use the specific types of these parameters and parse them strictly as hash or height values instead of generic json. But this would add more complexity to the client, especially if this approach was extended to handle other parameters like #32468.

  28. glozow commented at 7:28 pm on September 23, 2025: member
    Needs rebase for #33031, which was merged first per the author’s request.
  29. cli: Allow arguments to be both strings and json 44a493e150
  30. test: Remove convert_to_json_for_cli df67bb6fd8
  31. achow101 force-pushed on Sep 23, 2025
  32. achow101 commented at 7:58 pm on September 23, 2025: member
    Rebased
  33. rkrux approved
  34. rkrux commented at 1:44 pm on September 24, 2025: contributor

    Light code review, lgtm ACK df67bb6fd84c393eaf00f19074085ee080546bd3

    Adds convenience in CLI, cleans up tests too slightly.

    I do remember facing this issue earlier while using the CLI.

  35. DrahtBot requested review from ryanofsky on Sep 24, 2025
  36. DrahtBot requested review from pablomartin4btc on Sep 24, 2025
  37. DrahtBot requested review from enirox001 on Sep 24, 2025
  38. ryanofsky commented at 2:34 pm on September 24, 2025: contributor
    Code review ACK df67bb6fd84c393eaf00f19074085ee080546bd3, just rebased since last review. I do still think it would be good to improve the test (https://github.com/bitcoin/bitcoin/pull/33230#discussion_r2369570345)
  39. mzumsande commented at 9:27 pm on September 24, 2025: contributor

    Code Review ACK df67bb6fd84c393eaf00f19074085ee080546bd3

    Possible additional changes or follow-ups:

    I am not really a fan of multi-type args that require workarounds like this - e.g. I see no reason getblockstats would need to support a height while getblock doesn’t - but that ship has sailed.

  40. glozow merged this on Sep 25, 2025
  41. glozow closed this on Sep 25, 2025


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-10-10 15:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me