[Feature] Unspent Transaction Outputs Index

kyledrake commented at 3:03 pm on April 5, 2014: none

As per the conversation started in #3383, I am submitting a feature proposal for bitcoind.

The proposal is pretty straightforward (to describe, anyways): Implement an unspent transaction outputs index, and allow users to query it via the RPC API.

This is very similar in design to the txindex. It would be disabled by default, and enabled via the config file with something like utxoindex=1.

Then the user could query the daemon for all the unspent transactions available on any address at any time, on demand:

0getutxo [minconf=1] [maxconf=9999999] ["address",...]

And receive an array response of the utxo objects (perhaps with a few more niceties for people building wallets, such as the date the utxo was received):

0[
1    {
2        "txid" : "00003b2f320f9618162d209c67d2c29a812a8644f321093adc6b8f4a925f430e",
3        "vout" : 0,
4        "address" : "mhZFsQJddG5jGNEmwQgwzeKeiqVT2XgX84",
5        "scriptPubKey" : "2102224f4fd0ce5aeb19b7212218f6fdf86f6cd0cef870537294d277014bcc4bcf09ac",
6        "amount" : 50.00000000,
7        "confirmations" : 5332
8    }
9]

I have spent a lot of time promoting the adoption of the watchonly address patch, but I believe this would provide essentially the same resource that don’t-trust-the-server wallet developers need to implement. The transaction information can easily be stored in a ledger on the wallet developer’s end, and this complements the goal of separating bitcoind from the wallet code so that it can focus on its primary goal as a listener node that provides information to developers from the network.

I think this is potentially a very important feature to add. Right now, developers solve this by building their own custom blockchain parser, that duplicates (often poorly) a lot of the same business logic that’s already in bitcoind. It’s really slow to have to do this, it’s easy to screw things up, and it’s more stuff you need to install/manage.

With this functionality, developers could easily spawn a pool of bitcoind servers to request data and not have to worry about “priming” the daemons with the needed addresses to listen on. It also (relatively) increases privacy/security, because an attacker cannot use bitcoind to find which addresses are associated with it (sans a packet sniffer or something like that).

luke-jr commented at 9:17 pm on April 5, 2014: member

There is already a UTXO index, just not an address index. Also, UTXOs are not associated to addresses, so a “UTXO address index” doesn’t really make sense as a reasonable feature.

gmaxwell commented at 9:22 pm on April 5, 2014: contributor

@kyledrake forcing it to only work with ‘addresses’ is unnecessarily constraining. The more general approach in the addrindex patch is superior.

dangershony commented at 11:55 am on May 5, 2014: contributor

This suggestion is absolutely useful, and I wonder how it was not proposed earlier. I can currently use the listunspent rpc call but that returns all unspent address in my wallet only, and also if I have thousands of addresses it gets messy.

laanwj added the label Improvement on May 6, 2014

laanwj added the label Priority Medium on May 6, 2014

laanwj commented at 8:09 am on May 6, 2014: member

I think this is a good idea. I’d like more ways to query the UTXO set through RPC.

An index on the UTXO set would also be comparatively small and low-overhead compared to an index on all transactions.

laanwj added the label UTXO Db and Indexes on May 6, 2014

laanwj added the label Feature on May 13, 2014

laanwj removed the label Improvement on May 13, 2014

btcdrak commented at 10:09 pm on August 24, 2014: contributor

I would also like to see this feature implemented.

GrynnSan commented at 11:14 pm on August 24, 2014: none

OK, I will start on this feature. I would like to suggest that anyone with a stake in this feature to please comment on how you think it should be done and/or what information you can use in the result set, so that I end up with something useful for those who are going to actually be the consumers of it. Thanks! (I’ve read kyledrake’s proposal and pull 3383 already)

cozz commented at 2:25 am on August 25, 2014: contributor

I suggest it to be called gettxoutsbyaddress and to be similar to the gettxout call in rpcblockchain.cpp with the parameters:

minconf, default=1, if 0 then include the mempool.
list of scripts or addresses (like listunspent). You can look into the importaddress rpc call, its possible to import a script or address there. We automatically detect, whether a script or an address is given by the user, same can be done here.

Return values, same as for gettxout, but as an array of those, because an address can have multiple outputs and the user can pass multiple addresses. In addition every element in the array should have:

txid
vout

Preferably information about the block the output is in:

blockhash
blockheight
blockdate

We store the height in CCoins, with that you can just retrieve the blockinformation from the blockindex in chainActive.

laanwj removed the label Priority Medium on Apr 25, 2017

MarcoFalke commented at 2:02 pm on April 25, 2020: member

This has been implemented in the following RPCs: scantxoutset. (Alternatively, dumptxoutset with #18689)

Because address index was also mentioned in this thread (but seems unrelated to utxoset index), the address index pull request is here: #14053

Let us know if there is a feature for a use case we are still missing.

MarcoFalke closed this on Apr 25, 2020

pierreN commented at 7:47 am on April 28, 2020: contributor

Yes it seems that scantxoutset and dumptxoutset with #18689 fill that purpose.

If anyone sees additional features to add to #18689 I’d be happy to add them to the PR.

mikemilla commented at 9:30 pm on May 4, 2020: none

@MarcoFalke @pierreN Apologies if I am missing something here, but will this constantly maintain an index of all the utxos? For example if I run this once, after it completes, do I ever need to run this again to get the latest utxos for an address?

MarcoFalke commented at 10:39 pm on May 4, 2020: member

scantxoutset and dumptxoutset do not retain state in Bitcoin Core after the call has finished. Those calls operate by iterating over the current unspent transaction outputs. This set changes with every newly connected or disconnected block. txouts can be added or removed, and only unspent txouts are iterated over.

I think there was also discussion about an index of all addresses used in the chain. Assuming no reorgs, such an index is append-only. We have a txindex in current master. There are also plans to have an address index #14053

pierreN commented at 5:06 am on May 5, 2020: contributor

Note also that #18689 should be fast enough on decent hardware to dump the UTXO set every block mined (for most blocks..) A (really non-optimal) way could be to dump the UTXO set into tmpfs every time a block is mined and then diff the file.

mikemilla commented at 3:17 am on May 6, 2020: none

@pierreN that would work, but the use case I’m working on currently needs the outputs after a transaction is broadcasted and may not be confirmed at all (similar to apis like https://blockchair.com/ and https://bitaps.com/).

Does bitcoin-core have something similar to this or is #18689 as close as it gets?

If this doesn’t exist, to anyone that sees this, it would be nice if this could be added easily by having a bitcoin.conf property like indexoutputs=1 and a simple rpc like getoutputs address (Kinda like @kyledrake mentioned above!)

pierreN commented at 1:05 pm on May 6, 2020: contributor

I guess that if you want to scan the UTXO set for elements that match an address you should use scantxoutset (and then parse the JSON since the output shouldn’t be that big)?

dumptxoutset use case is if you want to dump all elements in the UTXO set, and then if you want filter elements with at least A confs and no more than B confs (which seems to be the original question? took 3mn on my machine):

0$ bitcoin-cli dumptxoutset utxo.dat '["height","txid"]' false ' '
1$ awk -v cur_height=$(bitcoin-cli getblockcount) '{num_conf=cur_height-$1; if(num_conf>A && num_conf<B) print num_conf,$2}' utxo.dat

MarcoFalke commented at 5:53 pm on May 6, 2020: member

outputs after a transaction is broadcasted and may not be confirmed at all

mempool entries are not in the utxo set (via scanttxoutset etc), since they are not yet included in a block and confirmed in the chain. You will need to get them with the mempool related RPCs

pierreN commented at 1:12 pm on May 7, 2020: contributor

(related to OP’s request:) Note that we could also add a field addresses here but then why not also add all the outputs of ScriptPubKeyToUniv… this looks like spam to me (and you could easily script that calling gettxout) so I’m not sure it’s a great idea. I guess it’s best to leave it up to the user.

DrahtBot locked this on Feb 15, 2022

[Feature] Unspent Transaction Outputs Index #4007