Utility to replace RPC calls that don't need wallet or chain context

Sjors commented at 3:22 pm on November 6, 2018: member

There’s a number of RPC calls currently that don’t need to know anything about the blockchain or the users wallet, and also don’t need p2p network access:

createmultisig
signmessagewithprivkey
verifymessage
validateaddress
combinepsbt
converttopsbt
decodepsbt
decodescript
finalizepsbt

Arguably also:

decoderawtransaction (but createrawtransaction needs the node)

There’s also a few being proposed:

deriveaddress #14667
encodescript #14476

We could call the new thing bitcoin-util. The deprecation notice for bitcoin-cli method would simply be Use "bitcoin-util method" instead.

A bigger issue is that some application may depend on calling bitcoind via RPC, perhaps on a different machine. A few things we could do:

take a very long time to deprecate
add a parameter to bitcoind that adds all utility functions via RPC, like e.g. bitcoin-cli util-deriveaddress
let bitcoin-util run its own RPC server on a different port

However (2 & 3): maintaining RPC wrappers around utility functions is some overhead, and extra complexity.

Ideally bitcoin-util comes with a library that’s easy to include in ruby gems, npm modules, etc.

One additional benefit of a separate utility is that you can switch between testnet and mainnet without restarting the node.

rodentrabies commented at 3:37 pm on November 6, 2018: contributor

let bitcoin-util run its own RPC server on a different port

This would allow implementing of a wallet-like transaction watcher that exposes APIs like ’listsentbyaddress’ that do not quite fit the current wallet model, but may be useful for some use cases (see #14498 and @sipa’s comment).

sipa commented at 3:44 pm on November 6, 2018: member

@mrwhythat Huh? No, this utility wouldn’t have a wallet or network access even.

rodentrabies commented at 3:55 pm on November 6, 2018: contributor

If I understood correctly, by separate transaction watcher in #14498 you meant something that can store addresses and watch transactions related to those addresses, hence a wallet-like, but of course it should not have access to the wallet. As for the network, proposal to run an RPC server

let bitcoin-util run its own RPC server on a different port

implies network access.

sipa commented at 4:14 pm on November 6, 2018: member

@mrwhythat I was talking about a separate module inside bitcoind, which acts in a wallet-like way.

That’s completely the opposite thing of what this issue is about (moving things that don’t need any context and are pure utilities out of bitcoind).

Yes this could offer an RPC interface. We’re not going to create a completely separate Bitcoin implementation though; we already have one.

rodentrabies commented at 4:43 pm on November 6, 2018: contributor

Ok, I guess I mistook “own RPC server” to be another instance of bitcoind with separate configuration for bircoin-util to use.

MarcoFalke added the label Brainstorming on Nov 6, 2018

MarcoFalke added the label Utils/log/libs on Nov 6, 2018

promag commented at 11:30 pm on November 7, 2018: member

Is there a reason to deprecate/remove and not add new utility to the RPC interface? If a client already uses the RPC interface for other calls then having these is a plus.

Sjors commented at 8:25 am on November 8, 2018: member

@promag that is a good question. There seems to be some reluctance about adding more RPC methods in the two PR’s I linked to above. There’s some overhead in maintaining a list of RPC boiler plate, but as long as it’s well tested, I think it’s manageable.

If the concern is having a never ending number of new RPC methods to review, then having a separate utility would be easier to eventually move into its own repository to reduce review burden here.

sipa commented at 8:32 am on November 8, 2018: member

I don’t think it’s so much the review burden. In a separate tool we also have the review burden, as it would be built from the same source.

Sending queries over RPC to bitcoind is just fundamentally stupid for context-free utlity operations. They can be computed by a library/binary running on the client side, or be performed by simple services that are orders of magnitude cheaper to run than bitcoind.

promag commented at 9:41 am on November 8, 2018: member

@sipa but what if a client already depends on the RPC interface and it wants to be as light as possible? There are also cons of using libraries and external binaries.

MarcoFalke commented at 6:54 pm on November 8, 2018: member

I think many applications don’t care about the json-serialization overhead and would prefer to use the convenient RPC over some library or bitcoin-tx like binary. Especially in case they have the bitcoind running for other reasons already.

If someone really cares about performance, they can write the util functions in their native language or use a library. I am not sure if our repo is the right place for a high performance library, but providing such a library seems partly orthogonal to providing the utility functions on a JSON RPC server.

MarcoFalke commented at 8:04 pm on November 8, 2018: member

There has been an IRC discussion recently: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-11-08-19.00.html

One topic was utility calls in general (http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-11-08-19.00.log.html#l-188) and the conclusion was that it should be decided case-by-case if they are in scope for Bitcoin Core.

The other topic was where to put them (library/util-binary/RPC), with no clear conclusion.

Sjors commented at 8:34 am on October 17, 2020: member

It’s happening! #19937 introduces bitcoin-util with a grind command for signet blocks.

Sjors commented at 12:50 pm on September 6, 2022: member

bitcoin-util now exists. Consider moving more things to it up for grabs.

Sjors closed this on Sep 6, 2022

bitcoin locked this on Sep 6, 2023

Utility to replace RPC calls that don’t need wallet or chain context #14671