RPC: Reject RPC requests with same named parameter specified multiple times #26628

pull ryanofsky wants to merge 4 commits into bitcoin:master from ryanofsky:pr/nmult changing 5 files +25 −4
  1. ryanofsky commented at 0:00 am on December 3, 2022: contributor

    Make the JSON-RPC server reject requests with the same named parameter specified multiple times, instead of silently overwriting earlier parameter values with later ones.

    Generally JSON keys are supposed to unique, and their order isn’t supposed to be significant, so having the server silently discard duplicate keys is error-prone. Most likely if an RPC client is sending a request with duplicate keys it means something is wrong with the request and there should be an error.

    After this change, named parameters are still allowed to specified multiple times on the bitcoin-cli command line, since bitcoin-cli automatically replaces earlier values with later values before sending the JSON-RPC request. This makes sense, since it’s not unusual for the order of command line options to be significant or for later command line options to override earlier ones.

  2. test: Add RPC tests for same named parameter specified more than once
    Current behavior isn't ideal and will be changed in upcoming commits, but it's
    useful to have test coverage regardless.
    
    MarcoFalke reported the case of bitcoin-cli positional arguments overwriting
    the named "args" parameter in
    https://github.com/bitcoin/bitcoin/pull/19762#discussion_r1035761471
    e2c3b18e67
  3. rpc: Make it an error server-side to specify same named parameter multiple times
    Specifying same named parameter multiple times is still allowed by bitcoin-cli.
    The client implementation overwrites earlier option values with later ones
    before sending to server. This is tested by interface_bitcoin_cli.py
    
    Rationale for allowing client parameters to be specified multiple times in
    bitcoin-cli is that this behavior has been supported for a long time, and that
    when using the command line interactively, it can be convenient to override
    earlier option values with new values without having to go back and remove the
    old value.
    
    But for the RPC server, there isn't really a good use-case for earlier values
    to be discarded if multiple values are specified. JSON keys are generally
    supposed to be unique and if they aren't it's probably an indication of some
    problem generating the RPC request.
    6bd1d20b8c
  4. bitcoin-cli: Make it an error to specify the "args" parameter two different ways
    MarcoFalke reported the case of positional arguments silently overwriting the
    named "args" parameter in bitcoin-cli
    https://github.com/bitcoin/bitcoin/pull/19762#discussion_r1035761471 and this
    behavior is confusing and was not intended when support for "args" parameters
    was added to bitcoin-cli in #19762.
    
    Instead of letting one "args" value overwrite the other in the client, just
    pass the values to the server verbatim, and let the error be handled server
    side.
    d1ca563825
  5. DrahtBot commented at 0:00 am on December 3, 2022: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK MarcoFalke, kristapsk, stickies-v
    Concept ACK luke-jr
    Stale ACK promag

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #26612 ([WIP] refactor: RPC: pass named argument as string_view by stickies-v)
    • #26506 (refactor: rpc: use convenience fn to auto parse non-string parameters by stickies-v)
    • #26485 (RPC: Add support for name-only parameters by ryanofsky)
    • #19949 (cli: Parse and allow hash value by fjahr)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  6. DrahtBot added the label RPC/REST/ZMQ on Dec 3, 2022
  7. in src/test/rpc_tests.cpp:98 in 763d3947d3 outdated
    93+    BOOST_CHECK_EXCEPTION(TransformParams(JSON(R"({"arg2": 2, "arg2": 4})"), arg_names), UniValue,
    94+                          HasJSON(R"({"code":-8,"message":"Parameter arg2 specified multiple times"})"));
    95+
    96     // Make sure named and positional arguments can be combined.
    97     BOOST_CHECK_EQUAL(TransformParams(JSON(R"({"arg5": 5, "args": [1, 2], "arg4": 4})"), arg_names).write(), "[1,2,null,4,5]");
    98 
    


    maflcko commented at 11:27 am on December 3, 2022:
    nit: Could fix #19762 (review) here as well?

    ryanofsky commented at 3:30 pm on December 9, 2022:

    re: #26628 (review)

    nit: Could fix #19762 (comment) here as well?

    Thanks, added commit with fixups suggested in #19762

  8. in doc/release-notes-xxxxx.md:4 in 763d3947d3 outdated
    0@@ -0,0 +1,4 @@
    1+JSON-RPC
    2+---
    3+
    4+The JSON-RPC server now rejects requests where same named parameter is specified multiple times, instead of silently overwriting earlier parameter values with later ones.
    


    maflcko commented at 11:28 am on December 3, 2022:
    0The JSON-RPC server now rejects requests where same named parameter is specified multiple times, instead of silently overwriting earlier parameter values with later ones. (#26628)
    

    stickies-v commented at 11:43 am on December 6, 2022:
    0The JSON-RPC server now rejects requests where a parameter is specified multiple times with the same name, instead of silently overwriting earlier parameter values with later ones.
    

    stickies-v commented at 12:30 pm on December 6, 2022:
    File name still has to be updated too

    ryanofsky commented at 3:41 pm on December 9, 2022:

    re: #26628 (review)

    File name still has to be updated too

    Thanks, applied both suggestions


    ryanofsky commented at 3:42 pm on December 9, 2022:

    re: #26628 (review)

    Thanks, applied suggestion

  9. maflcko approved
  10. maflcko commented at 11:28 am on December 3, 2022: member
    lgtm, left two nits
  11. maflcko added this to the milestone 25.0 on Dec 3, 2022
  12. stickies-v commented at 4:48 am on December 5, 2022: contributor
    Concept ACK
  13. in test/functional/interface_bitcoin_cli.py:95 in 763d3947d3 outdated
    89@@ -90,6 +90,10 @@ def run_test(self):
    90         assert_raises_rpc_error(-8, "Parameter arg1 specified twice both as positional and named argument", self.nodes[0].cli.echo, 0, 1, arg1=1)
    91         assert_raises_rpc_error(-8, "Parameter arg1 specified twice both as positional and named argument", self.nodes[0].cli.echo, 0, None, 2, arg1=1)
    92 
    93+        self.log.info("Test that later cli named arguments values silently overwrite earlier ones")
    94+        assert_equal(self.nodes[0].cli("-named", "echo", "arg0=0", "arg1=1", "arg2=2", "arg1=3").send_cli(), ['0', '3', '2'])
    95+        assert_raises_rpc_error(-8, "Parameter args specified multiple times", self.nodes[0].cli("-named", "echo", "args=[0,1,2,3]", "4", "5", "6", ).send_cli)
    


    stickies-v commented at 12:03 pm on December 6, 2022:
    I’m not sure if it’s worth changing, but seeing this test made me realize this error message could be confusing feedback to a user? They’re not specifying args multiple times explicitly - the fact that it’s labelled args internally may not be immediately obvious? But also maybe it’s a bit unnecessary to add special error handling for the args case.

    ryanofsky commented at 3:46 pm on December 9, 2022:

    re: #26628 (review)

    I’m not sure if it’s worth changing, but seeing this test made me realize this error message could be confusing feedback to a user? They’re not specifying args multiple times explicitly - the fact that it’s labelled args internally may not be immediately obvious? But also maybe it’s a bit unnecessary to add special error handling for the args case.

    Practically speaking, I don’t think this is an error case any human is going to see. It’s a corner case Marco found in #19762 (review) that was triggering unintended behavior in a newly added feature. Even if a bitcoin-cli user does manage to trigger this, I think the error message is accurate and helpful enough for finding the problem. I wouldn’t object to making a change here if there is a specific suggestion, but I also think the current behavior is ok, and there is room to add a special message later if the generic message turns out to be confusing.

  14. in src/rpc/server.cpp:403 in 763d3947d3 outdated
    398@@ -399,7 +399,10 @@ static inline JSONRPCRequest transformNamedArguments(const JSONRPCRequest& in, c
    399     const std::vector<UniValue>& values = in.params.getValues();
    400     std::unordered_map<std::string, const UniValue*> argsIn;
    401     for (size_t i=0; i<keys.size(); ++i) {
    402-        argsIn[keys[i]] = &values[i];
    403+        auto inserted = argsIn.emplace(keys[i], &values[i]);
    404+        if (!inserted.second) {
    


    stickies-v commented at 12:29 pm on December 6, 2022:

    nit: I think it’s reasonable enough to skip using the intermediate inserted variable entirely, but if you think it’s better for readability I’d suggest going all the way and make inserted the actual boolean directly?

    0        auto [_, inserted] = argsIn.emplace(keys[i], &values[i]);
    1        if (!inserted) {
    

    ryanofsky commented at 3:45 pm on December 9, 2022:

    re: #26628 (review)

    nit: I think it’s reasonable enough to skip using the intermediate inserted variable entirely, but if you think it’s better for readability I’d suggest going all the way and make inserted the actual boolean directly?

    Thanks, applied suggestion. Agree it’s better to avoid less descriptive “first” and “second” variables.

  15. stickies-v commented at 12:32 pm on December 6, 2022: contributor
    Approach ACK 763d3947d34fa71f1a0816c95fb6e206507e2ece
  16. luke-jr commented at 9:04 am on December 7, 2022: member

    Multiple keys with the same name is IIRC a supported feature of JSON, but concept ACK on rejecting them rather than ignoring all but one.

    I think bitcoin-cli should also reject duplicate parameters, though. It’s not obvious to a user that later ones will override earlier ones - they might expect to be able to specify some parameters twice and get the behaviour of both somehow. For example, getblock blockhash=... blockhash=... would logically make more sense as a batch call (though I don’t think it’s practical to support that, so an error is best IMO).

  17. in src/rpc/client.cpp:308 in 763d3947d3 outdated
    298@@ -299,7 +299,7 @@ UniValue RPCConvertNamedValues(const std::string &strMethod, const std::vector<s
    299     }
    300 
    301     if (!positional_args.empty()) {
    302-        params.pushKV("args", positional_args);
    303+        params.__pushKV("args", positional_args);
    


    promag commented at 10:16 pm on December 7, 2022:

    763d3947d34fa71f1a0816c95fb6e206507e2ece

    Using args= is fine but not when used along with positional arguments. Agree with @stickies-v, it’s not that clear from the error message what the user should do.

    An alternative is to fail here if params already contain args.


    ryanofsky commented at 3:43 pm on December 9, 2022:

    763d394

    Using args= is fine but not when used along with positional arguments. Agree with @stickies-v, it’s not that clear from the error message what the user should do.

    An alternative is to fail here if params already contain args.

    See other thread #26628 (review), but I think the message is accurate and not likely in practice to confuse anyone. If this is wrong, there is room to add a special message like the one you’re describing later.

  18. promag approved
  19. promag commented at 10:18 pm on December 7, 2022: member
    Code review ACK 763d3947d34fa71f1a0816c95fb6e206507e2ece.
  20. promag commented at 10:20 pm on December 7, 2022: member
    @luke-jr suggestion to fail on duplicate named parameters also makes sense to me, either implemented here or in a different branch.
  21. test: Suggested cleanups for rpc_namedparams test
    No changes in behavior, just implements review suggestions from
    
    https://github.com/bitcoin/bitcoin/pull/19762#discussion_r1025573943
    https://github.com/bitcoin/bitcoin/pull/19762#discussion_r1035955247
    https://github.com/bitcoin/bitcoin/pull/26628#discussion_r1038765894
    8c3ff7d52a
  22. ryanofsky force-pushed on Dec 9, 2022
  23. ryanofsky commented at 4:54 pm on December 9, 2022: contributor

    Thanks for the reviews!

    Updated 763d3947d34fa71f1a0816c95fb6e206507e2ece -> 8c3ff7d52ae3314959e1e66da8718a3f0d30abaa (pr/nmult.1 -> pr/nmult.2, compare) with suggestions.

    re: #26628 (comment)

    I think bitcoin-cli should also reject duplicate parameters, though. It’s not obvious to a user that later ones will override earlier ones - they might expect to be able to specify some parameters twice and get the behaviour of both somehow. For example, getblock blockhash=... blockhash=... would logically make more sense as a batch call (though I don’t think it’s practical to support that, so an error is best IMO).

    I think it’s pretty normal for command line tools to let later option values override earlier ones. Our binaries bitcoind, bitcoin-qt, bitcoin-wallet and bitcoin-cli all support this for options which don’t accept multiple values. In general I think it’s good for command line tools to support shortcuts that make scripting easier and remove the need for unnecessary typing and editing, but I can understand the opposite point of view. In any case, i think it would be better to follow up on this in another PR. This PR is trying to avoid the need for a decision by keeping the current behavior unchanged.

  24. maflcko commented at 10:09 am on December 12, 2022: member

    review ACK 8c3ff7d52ae3314959e1e66da8718a3f0d30abaa 🗂

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3review ACK 8c3ff7d52ae3314959e1e66da8718a3f0d30abaa 🗂
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUjCxwwAvbXzoDqw5hQHSmtV1qviWaF9zdsfXYeGfXy4TlIyT5jk/pSAmpMj6vqc
     8gry65aiuJiZ1mP3ywqMXw7+Fs6m/8wJCXl1cBHt43HDLyD1GDMJusVD3A5FsrtPv
     91zfZ4zt24cuDnEjImYdf+aWmszO1/PWeXs9n7WJ9AxhV7lE/740lHgjDj8or/yLz
    10KwuHwmGE1Z1pwV8t9aM0VLgPB6VWcAfXREoxeRbftF84UZ2/tTwER3YNw9Oh2Mbs
    11hOh1dCDUyzizziKSWlpzXUjPBLeGoe70W0W2CP9iukl/sRpLgEcDooPwqI9DQ1wv
    12htJqNJrXN/SDtnSulGSNC57500g00J/Jy/Xp+flridDtdZpqQB9LPx0KE+U+0oYz
    13vRaF9DsrOWtR6QSLqheFwjusosOKm4g7OV8DRrCIdbZVwiwZDeNFW/bfEBrTRvvk
    14T+efue7d+zc8Gq/oWQ+hRMUfj0GkESfWDlUZok/SqWvKWQnA82bvMi/qmb7BhPB9
    15swtWkjI9
    16=ECGn
    17-----END PGP SIGNATURE-----
    
  25. kristapsk approved
  26. kristapsk commented at 11:38 am on December 13, 2022: contributor
    ACK 8c3ff7d52ae3314959e1e66da8718a3f0d30abaa
  27. maflcko requested review from promag on Dec 13, 2022
  28. maflcko requested review from stickies-v on Dec 13, 2022
  29. stickies-v approved
  30. stickies-v commented at 4:54 pm on December 13, 2022: contributor

    ACK 8c3ff7d52

    No strong view on the bitcoin-cli behaviour. Intuitively I’d have implemented it to fail on duplicated names, but I think both approaches have merit. Happy to start with the current behaviour and potentially change in the future.

  31. maflcko merged this on Dec 13, 2022
  32. maflcko closed this on Dec 13, 2022

  33. sidhujag referenced this in commit 411a6c6f51 on Dec 14, 2022
  34. LarryRuane commented at 10:36 pm on January 2, 2023: contributor

    I think it’s pretty normal for command line tools to let later option values override earlier ones. Our binaries bitcoind, bitcoin-qt, bitcoin-wallet and bitcoin-cli all support this for options which don’t accept multiple values. In general I think it’s good for command line tools to support shortcuts that make scripting easier and remove the need for unnecessary typing and editing, but I can understand the opposite point of view.

    FWIW, I also favor the behavior of later arguments overriding earlier ones (when multiple don’t make sense). I’m not sure if the following is what you’re referring to, but I think it’s helpful to be able to change a default in a shell script or an alias, but also allow the user to override the new default, something like:

    0alias printblock='bitcoin-cli -named getblock verbose=true'
    

    Then you can do: printblock -blockhash=00...9 to get verbose output, or printblock -verbose=false -blockhash=00...9 to get a non-verbose result.

  35. janus referenced this in commit 86ae49b974 on Jan 20, 2023
  36. bitcoin locked this on Jan 2, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-05 22:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me