listunspent is really slow #10492

1. 
   RHavar commented at 10:32 pm on May 31, 2017: contributor

   Even with a small (~50) amount of inputs, calling bitcoin-cli listunspent is rather slow taking between 7 and 15 seconds on a high powered machine with lots of memory. This also makes a bunch of other things rather painful (for instance sending funds).

   No doubt the issue is exacerbated by my large (~1GB) wallet file containing about ~520k transactions; but I assume it means there’s some indexes or something missing.

2. fanquake added the label RPC/REST/ZMQ on Jun 1, 2017
3. 
   jonasschnelli commented at 5:51 am on June 1, 2017: contributor

   The current implementation is not optimised for large “enterprise” size wallets. I guess listunspent mainly suffers from the fact that it goes over the complete mapWallet (in memory map of all transactions), in your case ~520k transactions.

   There is a little optimisation in master that hasn’t been deployed (will be in 0.15): #8952

4. laanwj added the label Wallet on Jun 1, 2017
5. 
   laanwj commented at 6:35 am on June 1, 2017: member

   The current process for finding unspents is inefficient. These PRs aim or aimed at improving listunspent performance, I’d recommend investigating them:

   • Add index to wallet UTXO #7823 (closed)
   • Cache vout IsMine() value on CWallet::AvailableCoins() #9939
6. 
   MarcoFalke commented at 10:22 am on June 6, 2017: member

   Related: #7475
7. MarcoFalke added the label Resource usage on Jun 6, 2017
8. MarcoFalke removed the label RPC/REST/ZMQ on Jun 6, 2017
9. 
   gmaxwell commented at 9:59 pm on June 6, 2017: contributor

   Even with a small (~50) amount of inputs, calling bitcoin-cli listunspent is rather slow taking between 7 and 15 seconds on a high powered machine with lots of memory.

   To be clear, this is a wallet with tens of thousands of transactions in it– right? Listunspent iterates over all the transactions, not just the ones with unspent outputs.

10. 
    RHavar commented at 10:02 pm on June 6, 2017: contributor

    @gmaxwell Sorry, I meant a small amount of unspent outputs. (as in, the listunspent return result is quite small) but yeah the wallet itself has a lot of transactions
11. 
    promag commented at 2:15 pm on December 7, 2017: member

    Indeed for large wallets the current implementation is slow and @joaopaulofonseca tried to improve parts of the algorithm. In the past I’ve submitted #7649 which improved a bit. It is hard to improve taking the current wallet implementation.

    If you must create a lot of transactions then I can suggest:

    • cache externally: you periodically retrieve unspents and then try to first spend those when creating new transactions. With that approach you have to deal with change output and fee calculation;
    • create a new “send wallet” when the current has more than N transactions. Just transfer the balance to an address of the new wallet.
12. 
    MarcoFalke commented at 12:32 pm on May 9, 2020: member

    The feature request didn’t seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

    Closing due to lack of interest. Pull requests with improvements are always welcome.

13. MarcoFalke closed this on May 9, 2020

    ---

14. DrahtBot locked this on Feb 15, 2022

Contributors
RHavar jonasschnelli laanwj MarcoFalke gmaxwell promag

Labels
Wallet Resource usage