Not able to load multiple wallet more than X number of wallet from RPC

adneerav commented at 2:21 pm on December 21, 2018: none

Hello

I am using Bitcoin core for the multiple wallet functionality. All the wallet access and things I am doing using RPC.

Issue : I am not able to load wallet more than 240(may b depends on system or db limit,Please correct me if I am wrong).

How can I increase the size of the DB or limit so I can load multiple wallet more compare to current limit.

Or is there any way where I can load multiple wallet more compare to current ?

Daemon getting stopped after 240 wallets.

What I’ve tried so far ? I’ve added below things by creating DB_CONFIG file in datadirectory set_lk_max_lockers 400000 set_lg_regionmax 1048576

Reference link of above : https://bitcoin.stackexchange.com/a/70737

Result same issue

If anyone face this type of issue please share or guide on this.

fanquake added the label RPC/REST/ZMQ on Dec 22, 2018

promag commented at 4:45 pm on November 7, 2019: member

Do you still have the issue with latest release?

Also, looks like all wallets are in the same Berkeley DB environment. Is that a requirement?

adneerav commented at 2:44 pm on November 11, 2019: none

Hello @promag

Yes I am facing same issue with latest build. I’ve mentioned entire test cases as below.

I’ve tried with latest release(Branch : master) and build the core ,cli and all.

I kept observing memory utilization by creating , loading and unloading 100 wallets. Below are the screenshots of memory utilization ( free -m )

Step 1 (Before Creating or loading any wallet) :

Before creating wallet memory usage states are as below (createwallet by default loading wallets)

Before_loading_wallet

Step 2 (After creating or loaded 100 wallets):

After creating(loaded wallet) memory utilization states are as below (also check bitcoin-cli listwallets and in response its showing all 100 wallets are in response as a loaded wallet)

After_loading_100_wallet

lsof -p pidof bitcoind

above command shows all 100 loaded details and processes in this step(after creating wallet) and memory usage increases.

Step 3 (After unloading all loaded wallets(100 wallets):

After unloading all loaded 100 wallets memory utilization are almost same as loaded wallet. (bitcoin-cli listwallets showing all wallets as unloaded empty jsonarray response) .

After_unloading_100_wallets

lsof -p pidof bitcoind Above command not showing any wallets loaded in steps(after unloading wallet) But memory usage are almost same like loaded wallet, so I guess its not releasing memory

Based on above test I found still RAM usage is same even after unloading wallet.I’ve tried with 300+ wallets also but the result were same.

Core and cli built as per mentioned step on this link

command and config file has below configuration

0rpcuser=bitcoin
1rpcpassword=bitcoinrpcpassword
2rpcport=2051
3mempoolexpiry=2
4daemon=1
5rpcworkqueue=128
6rpcthreads=128
7rpctimeout=220

Note : I’ve removed DB_CONFIG file and all extra setup and tested this things. As I’ve not found any major change with this DB_CONFIG configuration.

If anyone face this issue and resolved please help/guide me to resolve this.

promag commented at 2:52 pm on November 11, 2019: member

#15017 (comment)

Please reply to this.

so I guess its not releasing memory

I’m sure someone can explain this better but memory management doesn’t work that way - memory isn’t claimed after free() or delete.

Have you tried load -> unload -> load -> … if there’s a leak then the memory should grow consistently.

adneerav commented at 3:03 pm on November 11, 2019: none

@promag

Have you tried load -> unload -> load -> … if there’s a leak then the memory should grow consistently.

Yes, tested above scenario. load -> unload -> load

Result : memory usage are almost same when i loaded first time not grown too much.

So is it like by unloading wallet will never release memory ?

daemon core stop and start is the only solution to release the memory ?

promag commented at 3:06 pm on November 11, 2019: member

See this http://www.thecrowned.org/c-delete-operator-really-frees-memory

adneerav commented at 3:17 pm on November 11, 2019: none

thanks for the link @promag

So is there any way where I can increase the limit for loading multiple wallet ?

promag commented at 3:27 pm on November 11, 2019: member

Also, looks like all wallets are in the same Berkeley DB environment. Is that a requirement?

In https://bitcoin.stackexchange.com/questions/70734/bitcoin-core-multi-wallet-limit-to-333-wallets/70737#70737 you say that multiple wallets are configured as

0wallet=wallet-1.dat

Can you list the contents of <datadir>/regtest/wallets (if regtest, otherwise adjust the path for your case)?

adneerav commented at 3:36 pm on November 11, 2019: none

@promag I am loading and unloading wallet using RPC not using configuration file.

Testing with mainnet and path for the wallets is .bitcoin/wallets

Screenshot from 2019-11-11 21-07-02

promag commented at 3:39 pm on November 11, 2019: member

Where are the wallets?

adneerav commented at 3:48 pm on November 11, 2019: none

All 1 to 100 wallets are in wallets dir

adneerav commented at 5:36 am on November 12, 2019: none

@promag

.bitcoin directory

Screenshot from 2019-11-12 11-03-56

wallets folder contains all the wallets which are created using RPC

Screenshot from 2019-11-12 11-05-45

willcl-ark commented at 9:33 pm on November 24, 2022: contributor

@adneerav Is this still an issue for you? I have just tried to reproduce this using v24.0 and was unable to make bitcoind crash creating and loading 241 legacy (non-descriptor) BDB wallets. While creating the wallets memory usage increased from 127MB to 5958MB, but bitcoind remained stable.

-t-15017_001

I then unloaded all wallets which dropped memory usage back to 188MB, before reloading them via rpc using a loop in fish shell:

0$ for wallet in (exa $pwd)
1      /home/will/src/bitcoin/src/bitcoin-cli -signet loadwallet /tmp/15017/$wallet
2      sleep 0.5
3  end

…which also did not crash bitcoind.

-t-15017_002

Is there something that you are doing differently to me?

I did not change the following settings from default which I notice you tried.

0rpcworkqueue=128
1rpcthreads=128
2rpctimeout=220

willcl-ark commented at 9:40 am on March 27, 2023: contributor

@adneerav As there has been no update on this since 2019 (aside from my own follow-up) and I was unable to reproduce this issue with the current version of Bitcoin Core, I’m going to close this for now.

Feel free to comment here if you are still experiencing this problem and want this re-opened to be investigated again.

willcl-ark closed this on Mar 27, 2023

bitcoin locked this on Mar 26, 2024

Not able to load multiple wallet more than X number of wallet from RPC #15017

Step 1 (Before Creating or loading any wallet) :

Step 2 (After creating or loaded 100 wallets):

Step 3 (After unloading all loaded wallets(100 wallets):