listunspent, fundrawtransaction, getwalletinfo locks wallet for any other operation

gituser commented at 11:31 pm on January 30, 2023: none

Actual behavior

If you run a listunspent, fundrawtransaction, getwalletinfo (maybe other commands) on a big wallet (e.g. with wallet with at least 8000 transactions or more) all other operations conducted in parallel (e.g. getnewaddress) will be stuck until that command finishes.

Because of this issue when our wallet is sending transactions using listunspent and fundrawtransactions there is no way to get a new address via getnewaddress RPC call (as bitcoin waits for those calls to finish).

I’ve got bitcoin’s datadir and wallet both on NVME drives, but running listunspent and sometimes getwalletinfo is painfully slow.

The wallet itself is not that big and was created recently to migrate from bdb to the new descriptors wallet:

0$ du -sh .bitcoin/desc/
178M	.bitcoin/desc/

 0~$ ./bitcoin-cli -rpcwallet=desc getwalletinfo
 1{
 2  "walletname": "desc",
 3  "walletversion": 169900,
 4  "format": "sqlite",
 5  "balance": xxx,
 6  "unconfirmed_balance": xxx,
 7  "immature_balance": 0.00000000,
 8  "txcount": 8482,
 9  "keypoolsize": 4000,
10  "keypoolsize_hd_internal": 4000,
11  "paytxfee": 0.00000000,
12  "private_keys_enabled": true,
13  "avoid_reuse": false,
14  "scanning": false,
15  "descriptors": true,
16  "external_signer": false
17}

0$ time ./bitcoin-cli -rpcwallet=desc listunspent|grep txid|wc -l
1224
2
3real	1m15.596s
4user	0m0.008s
5sys	0m0.000s

Expected behavior

listunspent or fundrawtransaction shouldn’t lock the wallet at least for getting new address, maybe other operations as well?

To reproduce

create a new descriptors wallet and generate 17K addresses there
create at least 8K transactions in and out
try to run listunspent or fundrawtransaction and in parallel getnewaddress
getnewaddress will take 60 seconds or more to return an address until either of listunspent or fundrawtransaction finishes

System information

I’ve tried latest bitcoin release v24.0.1, master branch (built 77a36033b5ecbf8dedb917d680f4116786fd7375 commit) issue reproduces.

Self-built for Ubuntu 18.04 LTS.

I’ve got bitcoin’s datadir and wallet both on NVME drives, but running listunspent is painfully slow.

gituser added the label Bug on Jan 30, 2023

willcl-ark commented at 0:30 am on January 31, 2023: contributor

These commands intentionally lock the wallet (and in the case of getnewaddress the script pubkey manager being called) in order that their results are guarenteed to be atomic and correct.

listunspent must lock the wallet so that when it returns your list of unspent utxos these are certain to still be unspent by the time the function returns. In fact it will lock the wallet twice, once while it fetches all available coins as outputs, and again as it parses each output to provide context on it.

My understanding is that getnewaddress locks the wallet when generating new addresses to avoid e.g. a race between two calls to fetch a new address which could result in the same address being returned twice (or written to the db multiple times).

I don’t think performing either of these operations (nor fundrawtransaction) without blocking is avoidable. The remaining issue therefore is whether any of these operations can be sped up in bitcoin core descriptor wallets, to which I defer to others.

As an alternative, perhaps you could consider implementing some client side caching either on the getnewaddress or the listunspent calls to try and circumvent this?

gituser commented at 7:49 am on January 31, 2023: none

@willcl-ark thanks for the quick reply.

However, If I run getnewaddress in parallel (e.g. using parallel) it fetches 1000 unique addresses without any issues in less than 5 secs:

e.g. little script like this

 0#!/bin/bash
 1WALLET=test
 2USER=yyy
 3PASS=xxx
 4THREADS=1000
 5
 6if [ -z "$1" ]; then
 7 echo "Usage: `basename $0` [threads]"
 8 exit 1
 9fi
10
11THREADS=$1
12ulimit -n 65535
13
14seq $THREADS|parallel -j$THREADS -n0 "curl -s --user $USER:$PASS --data-binary '{\"jsonrpc\": \"1.0\", \"id\":\"curltest\", \"method\": \"getnewaddress\", \"params\": [] }' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/$WALLET"

Results in:

0$ time ./gen_addresses.sh 1000 >/dev/null
1real	0m4.495s
2user	0m4.745s
3sys	0m8.019s

So that makes me wonder why getnewaddress isn’t blocking fetching another address in parallel, whilst listunspent blocks entirety of the wallet.

How do I implement client side caching of getnewaddress or listunspent, if listunspent still locks wallet for at least 1 minute? That means at least 1 minute bitcoin wallet won’t be able to respond to any other calls.

Currently my app is very simple:

it fetches new address by using getnewaddress
and if the user requests withdrawal it collects UTXO and combines them into transaction

So my problem is with 2) as it locks the wallet and 1) stops working resulting in service unavailability.

Of course the best is to generate addresses completely offline, but I’ve wanted (for now) to use bitcoin wallet’s functionality for that.

willcl-ark commented at 2:20 pm on January 31, 2023: contributor

I don’t think you’ll easily hit the contention lock on getnewaddress as it returns so fast, i.e. it does not block for long enough on any call. For me the entire RPC call takes about 10 ms or less, with aonly a fraction of that locking the wallet lock.

To cache getnewaddress you could perhaps call it some number of times on startup so that you have a cache of unused addresses in your client app, and periodically after that (rather than on demand).

Are you calling listunspent RPC each time user wants to make a withdrawal, and if so I am curious as to why? If user funds are pooled, then could you not let Bitcoin Core’s coin selection algorithms handle constructing transactions of any amount for you using available UTXOs, as an alternative (without knowing any more about how you are using the output of listunspent)?

gituser commented at 4:39 pm on January 31, 2023: none

Are you calling listunspent RPC each time user wants to make a withdrawal, and if so I am curious as to why? If user funds are pooled, then could you not let Bitcoin Core’s coin selection algorithms handle constructing transactions of any amount for you using available UTXOs, as an alternative (without knowing any more about how you are using the output of listunspent)?

Yes I do call listunspent every time I want to:

accumulate small (close to dust) payments (e.g. < 0.01 BTC) into 1 large transaction (via crontab every hour or so) to limit number of unspent payments
for user withdrawals I collect requests and construct 1 large transaction (every 5 minutes) with multiple withdrawal outputs with fundrawtransaction

In both cases there will be a lock on my system, I can’t cache listunspent or fundrawtransaction because the UTXO are changing all the time (new transactions are arriving).

willcl-ark commented at 4:49 pm on January 31, 2023: contributor

I see. Well I don’t think the wallet locks on these functions will be going anywhere, so I think best to consider how you can work around it…

For example, there are query_options in listunspent (see bitcoin-cli help listunspent):

05. query_options                             (json object, optional) JSON with query options
1     {
2       "minimumAmount": amount,              (numeric or string, optional, default="0.00") Minimum value of each UTXO in BTC
3       "maximumAmount": amount,              (numeric or string, optional, default=unlimited) Maximum value of each UTXO in BTC
4       "maximumCount": n,                    (numeric, optional, default=unlimited) Maximum number of UTXOs
5       "minimumSumAmount": amount,           (numeric or string, optional, default=unlimited) Minimum sum value of all UTXOs in BTC
6       "include_immature_coinbase": bool,    (boolean, optional, default=false) Include immature coinbase UTXOs
7     }

…which might help reduce your query time if you only want to collect low-value UTXOs, by specifying max amount, count or maximum sumAmount for example?

achow101 commented at 4:53 pm on January 31, 2023: member

listunspent, fundrawtransaction, and the send* RPCs all require figuring out the UTXOs that belong to the wallet. Currently, that means iterating through every single output of every transaction in the wallet, including received, sent, and unrelated, all while holding the wallet lock to ensure that atomicity of the operation. This can be very slow for large wallets (I think your wallets could be classified as such). We are working on improving the wallet so that it does not do this and is more performant, but it requires rewriting a large portion of the transaction handling code, so it’s going pretty slowly. There unfortunately is not much that can be done about this other than using multiple smaller wallets rather than one very big one.

gituser commented at 5:04 pm on January 31, 2023: none

I see. Well I don’t think the wallet locks on these functions will be going anywhere, so I think best to consider how you can work around it…

For example, there are query_options in listunspent (see bitcoin-cli help listunspent):
05. query_options                             (json object, optional) JSON with query options
1     {
2       "minimumAmount": amount,              (numeric or string, optional, default="0.00") Minimum value of each UTXO in BTC
3       "maximumAmount": amount,              (numeric or string, optional, default=unlimited) Maximum value of each UTXO in BTC
4       "maximumCount": n,                    (numeric, optional, default=unlimited) Maximum number of UTXOs
5       "minimumSumAmount": amount,           (numeric or string, optional, default=unlimited) Minimum sum value of all UTXOs in BTC
6       "include_immature_coinbase": bool,    (boolean, optional, default=false) Include immature coinbase UTXOs
7     }
…which might help reduce your query time if you only want to collect low-value UTXOs, by specifying max amount, count or maximum sumAmount for example?

Thanks for the tip, but I’m already using only confirmed UTXOs and also got a filter on maximumAmount, but it’s still slow..

I’m also thinking about creating a copy of the wallet (identical) where I will be running slow operations like listunspent or fundrawtransaction.

gituser commented at 9:10 pm on January 31, 2023: none

It seems to me slow listunspent is some kind of bitcoin’s regression (might be?), as I have some test litecoin wallet (though it’s bdb wallet v130000 not descriptors) with more than 19K unspent outputs and more than 80K transactions and listunspent works instantly in few ms.

achow101 commented at 9:54 pm on January 31, 2023: member

If you’re able to, could you try profiling with perf to see where the time is being spent during listunspent?

gituser commented at 3:10 am on February 1, 2023: none

@achow101 there you go https://pastebin.com/raw/fSzSr6NF

achow101 commented at 4:00 am on February 1, 2023: member

Hmm, it looks like it actually spends most of the time in IsMine, not in iterating the transactions. Around how many addresses does the wallet have?

Could you try this branch? I’ve changed the map used in IsMine lookups to be an unordered map so it should be faster, but based on the profiling, I’m not sure if this is the main source of the slowdown.

I’m working on generating a big wallet for testing, should be ready to reproduce this myself tomorrow.

gituser commented at 9:52 am on February 1, 2023: none

Hmm, it looks like it actually spends most of the time in IsMine, not in iterating the transactions. Around how many addresses does the wallet have?

Could you try this branch? I’ve changed the map used in IsMine lookups to be an unordered map so it should be faster, but based on the profiling, I’m not sure if this is the main source of the slowdown.

I’ve tried your branch and had similar issue with slow listunspent it even took more than 90 seconds:

0 time ./bitcoin-cli -rpcwallet=desc listunspent|grep txid|wc -l
161
2
3real	2m53.541s
4user	0m0.005s
5sys	0m0.000s

and perf report is similar, I’m pasting a snip here:

 0# Total Lost Samples: 0
 1#
 2# Samples: 12K of event 'cycles'
 3# Event count (approx.): 404461969778
 4#
 5# Children      Self  Command         Shared Object        Symbol                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
 6# ........  ........  ..............  ...................  ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 7#
 8    99.85%     0.00%  b-httpworker.1  [unknown]            [.] 0xffffffffffffffff
 9            |
10            ---0xffffffffffffffff
11               RPCHelpMan::HandleRequest
12               std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), wallet::listunspent()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)#1}>::_M_invoke
13               wallet::listunspent()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)#1}::operator()
14               wallet::AvailableCoinsListUnspent
15               wallet::AvailableCoins
16               |          
17               |--97.39%--wallet::CWallet::IsMine
18               |          |          
19               |          |--60.54%--wallet::DescriptorScriptPubKeyMan::IsMine
20               |          |          |          
21               |          |          |--47.50%--std::_Hashtable<CScript, std::pair<CScript const, int>, std::allocator<std::pair<CScript const, int> >, std::__detail::_Select1st, std::equal_to<CScript>, SaltedSipHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::count
22               |          |          |          |          
23               |          |          |           --30.67%--SaltedSipHasher::operator()
24               |          |          |                     |          
25               |          |          |                     |--18.71%--CSipHasher::Write
26               |          |          |                     |          
27               |          |          |                     |--8.78%--CSipHasher::Finalize
28               |          |          |                     |          
29               |          |          |                      --2.03%--CSipHasher::CSipHasher
30               |          |          |          
31               |          |          |--5.72%--std::unique_lock<std::recursive_mutex>::unlock
32               |          |          |          |          
33               |          |          |           --4.33%--__GI___pthread_mutex_unlock (inlined)
34               |          |          |          
35               |          |           --4.62%--UniqueLock<AnnotatedMixin<std::recursive_mutex> >::UniqueLock
36               |          |                     |          
37               |          |                      --3.09%--__GI___pthread_mutex_lock (inlined)
38               |          |          
39               |           --22.32%--std::_Rb_tree_increment
40               |          
41                --2.34%--wallet::CachedTxIsFromMe
42                          wallet::CachedTxGetDebit
43                          wallet::GetCachableAmount
44                          wallet::CWallet::GetDebit
45                          wallet::CWallet::GetDebit
46                          |          
47                           --2.30%--wallet::CWallet::IsMine
48                                     |          
49                                      --1.51%--wallet::DescriptorScriptPubKeyMan::IsMine
50                                                |          
51                                                 --1.13%--std::_Hashtable<CScript, std::pair<CScript const, int>, std::allocator<std::pair<CScript const, int> >, std::__detail::_Select1st, std::equal_to<CScript>, SaltedSipHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::count
52                                                           |          
53                                                            --0.77%--SaltedSipHasher::operator()

I’m working on generating a big wallet for testing, should be ready to reproduce this myself tomorrow.

Great! Thank you.

gituser commented at 3:23 pm on February 1, 2023: none

Maybe this PR will help https://github.com/bitcoin/bitcoin/pull/9939

achow101 commented at 4:53 pm on February 1, 2023: member

I’ve generated a wallet that has ~7500 UTXOs, ~91000 addresses, and ~333000 transactions, listunspent takes ~2 seconds to run, and listtransactions ~20 seconds. I think I’ve produced a wallet with similar parameters as yours, but am not seeing nearly as big of a slowdown.

Have you imported additional descriptors into your wallet?

furszy commented at 5:20 pm on February 1, 2023: member

Are you calling listunspent RPC each time user wants to make a withdrawal, and if so I am curious as to why? If user funds are pooled, then could you not let Bitcoin Core’s coin selection algorithms handle constructing transactions of any amount for you using available UTXOs, as an alternative (without knowing any more about how you are using the output of listunspent)?

Yes I do call listunspent every time I want to:

accumulate small (close to dust) payments (e.g. < 0.01 BTC) into 1 large transaction (via crontab every hour or so) to limit number of unspent payments

for user withdrawals I collect requests and construct 1 large transaction (every 5 minutes) with multiple withdrawal outputs with fundrawtransaction

In both cases there will be a lock on my system, I can’t cache listunspent or fundrawtransaction because the UTXO are changing all the time (new transactions are arriving).

As fundrawtransaction doesn’t allow you to pre-select inputs. How are you doing (1)?

If you are manually creating the tx by calling listunspent + createrawtransaction, then you could skip fundrawtransaction step and just call to signrawtransactionwithwallet (or, disable the fundrawtransaction “add_inputs” option which will command the wallet to skip the entire available coins fetching and selection process, which you don’t need to execute as you are manually selecting the UTXOs that will be spent).

For (2), stuff like #25806 + what will come after it, will speedup the tx creation process (on big wallets the difference should be very noticeable).

gituser commented at 6:28 pm on February 1, 2023: none

I’ve generated a wallet that has ~7500 UTXOs, ~91000 addresses, and ~333000 transactions, listunspent takes ~2 seconds to run, and listtransactions ~20 seconds. I think I’ve produced a wallet with similar parameters as yours, but am not seeing nearly as big of a slowdown.

Have you imported additional descriptors into your wallet?

Yes I have imported descriptors for single addresses from my old legacy wallet (about 15K addresses).

gituser commented at 6:33 pm on February 1, 2023: none

Are you calling listunspent RPC each time user wants to make a withdrawal, and if so I am curious as to why? If user funds are pooled, then could you not let Bitcoin Core’s coin selection algorithms handle constructing transactions of any amount for you using available UTXOs, as an alternative (without knowing any more about how you are using the output of listunspent)?

Yes I do call listunspent every time I want to:

accumulate small (close to dust) payments (e.g. < 0.01 BTC) into 1 large transaction (via crontab every hour or so) to limit number of unspent payments

for user withdrawals I collect requests and construct 1 large transaction (every 5 minutes) with multiple withdrawal outputs with fundrawtransaction

In both cases there will be a lock on my system, I can’t cache listunspent or fundrawtransaction because the UTXO are changing all the time (new transactions are arriving).

As fundrawtransaction doesn’t allow you to pre-select inputs. How are you doing (1)?

If you are manually creating the tx by calling listunspent + createrawtransaction, then you could skip fundrawtransaction step and just call to signrawtransactionwithwallet (or, disable the fundrawtransaction “add_inputs” option which will command the wallet to skip the entire available coins fetching and selection process, which you don’t need to execute as you are manually selecting the UTXOs that will be spent).

I do use fundrawtransaction() in order to determine fee as it’s complicated to calculate it manually because of all the different BTC type addresses. Yes, I create transaction with createrawtransaction() with txids from listunspent, then call fundrawtransaction() calc fee, minus that fee from overall amount of outputs and then call again createrawtransaction() + fundrawtransaction() just to be sure that there will be a single address in the outputs.

Also another solution is to pre-determine fee for hardcoded number of input UTXOs and minus it.

If you have a better solution I’d love to hear it.

For (2), stuff like #25806 + what will come after it, will speedup the tx creation process (on big wallets the difference should be very noticeable).

Nice!

achow101 commented at 6:38 pm on February 1, 2023: member

Yes I have imported descriptors for single addresses from my old legacy wallet (about 15K addresses).

Ahh, I think that’s the issue then. Based on the profiling, what’s happening is that most of the time is being spent iterating through each imported descriptor. I believe #26008 should fix this for you, can you try that?

gituser commented at 6:54 pm on February 1, 2023: none

Yes I have imported descriptors for single addresses from my old legacy wallet (about 15K addresses).

Ahh, I think that’s the issue then. Based on the profiling, what’s happening is that most of the time is being spent iterating through each imported descriptor. I believe #26008 should fix this for you, can you try that?

Yes, this is it! Version of bitcoind from this #26008 pull request works just fine:

0$ time ./bitcoin-cli -rpcwallet=desc listunspent|grep txid|wc -l
1314
2
3real	0m0.464s
4user	0m0.009s
5sys	0m0.000s

@achow101 you’re genius, thanks a lot!

Crypto2 commented at 2:55 am on May 29, 2023: none

listunspent, fundrawtransaction, and the send* RPCs all require figuring out the UTXOs that belong to the wallet. Currently, that means iterating through every single output of every transaction in the wallet, including received, sent, and unrelated, all while holding the wallet lock to ensure that atomicity of the operation.

I’ve suggested before just having a second map of only unspents, should speed it up massively at the expense of some memory. Could be default off and have to enable in the config file/command line if that’s a worry.

listunspent, fundrawtransaction, getwalletinfo locks wallet for any other operation #27002