wallet: bitcoind RAM usage issue

dzmitrykliapkou commented at 2:36 pm on June 23, 2021: none

We are running bitcoin full-node on AWS EC2 r5.xlarge instance and the problem is that bitcoind uses almost all available memory - 24GB. If we try to use smaller instance types - bitcoind consumes all the memory and the instance itself becomes unavailable.

Expected behavior bitcoind memory usage is limited by dbcache and maxmempool parameters

Actual behavior bitcoind uses 24GB of memory despite configuration

To reproduce Install bitcoind v0.21.1 on Amazon Linux 2 r5.xlarge instance. Get it fully synced. Check memory usage

System information Bitcoin Core version v0.21.1 OS - Amazon Linux 2 Instance type - r5.xlarge - 4 vCPU, 32GB RAM Disk for blockchain data - 500GB general purpose SSD

bitcoind start command:

0/usr/local/bin/bitcoind -daemon -pid=/run/bitcoind/bitcoind.pid -conf=/etc/bitcoin/bitcoin.conf -datadir=/data/bitcoind -txindex=1 -sysperms=true

/etc/bitcoin/bitcoin.conf:

 0# [core]
 1daemon=1
 2datadir=/data/bitcoind
 3dbbatchsize=16000000
 4sysperms=1
 5
 6# [rpc]
 7server=1
 8rpcbind=0.0.0.0
 9rpcport=8332
10rpcallowip=0.0.0.0
11rpcuser=******
12rpcpassword=******
13
14[main]
15[test]
16[regtest]

dzmitrykliapkou added the label Bug on Jun 23, 2021

MarcoFalke added the label Resource usage on Jun 23, 2021

rebroad commented at 3:25 pm on June 23, 2021: contributor

Why are you setting dbbatchsize to an decimal based number? What happens if you remove this setting?

dzmitrykliapkou commented at 4:10 pm on June 23, 2021: none

Commented this option. Restarted bitcoind. Nothing changed

hebasto commented at 4:13 pm on June 23, 2021: member

@dzmitrykliapkou

bitcoind uses 24GB of memory despite configuration

Which tool is used to measure bitcoind RAM usage?

dzmitrykliapkou commented at 4:18 pm on June 23, 2021: none

[ec2-user@ip-******** ~]$ free total used free shared buff/cache available Mem: 32465184 23673052 240036 180 8552096 8333304 Swap: 8388604 539884 7848720

Here is a moment of bitcoind restart^

sipa commented at 4:21 pm on June 23, 2021: member

how much RSS is bitcoind itself using? System memory may be distorted by the OS choosing to use some of the available memory to cache disk data.

hebasto commented at 4:22 pm on June 23, 2021: member

“buff/cache” is used by OS – e.g., https://www.redhat.com/sysadmin/dissecting-free-command

dzmitrykliapkou commented at 4:24 pm on June 23, 2021: none

sipa commented at 4:25 pm on June 23, 2021: member

Is that the full bitcoin.conf you pasted above? No dbcache line, in particular?

dzmitrykliapkou commented at 4:27 pm on June 23, 2021: none

Yes, it is full conf file. I decided to use default value for dbcache

sipa commented at 4:29 pm on June 23, 2021: member

can you report the beginning of your debug.log file, the first few dozen lines after the last restart?

MarcoFalke commented at 4:30 pm on June 23, 2021: member

Also, does it consume that much memory when you run with -txindex=0?

dzmitrykliapkou commented at 4:34 pm on June 23, 2021: none

bebug.log after restart:

 02021-06-23T15:59:41Z Bitcoin Core version v0.21.1 (release build)
 12021-06-23T15:59:41Z Assuming ancestors of block 0000000000000000000b9d2ec5a352ecba0592946514a92f14319dc2b367fc72 have valid signatures.
 22021-06-23T15:59:41Z Setting nMinimumChainWork=00000000000000000000000000000000000000001533efd8d716a517fe2c5008
 32021-06-23T15:59:41Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
 42021-06-23T15:59:41Z Using RdSeed as additional entropy source
 52021-06-23T15:59:41Z Using RdRand as an additional entropy source
 62021-06-23T15:59:41Z Default data directory /data/.bitcoin
 72021-06-23T15:59:41Z Using data directory /data/bitcoind
 82021-06-23T15:59:41Z Config file: /etc/bitcoin/bitcoin.conf
 92021-06-23T15:59:41Z Config file arg: daemon="1"
102021-06-23T15:59:41Z Config file arg: datadir="/data/bitcoind"
112021-06-23T15:59:41Z Config file arg: rpcallowip="0.0.0.0"
122021-06-23T15:59:41Z Config file arg: rpcbind=****
132021-06-23T15:59:41Z Config file arg: rpcpassword=****
142021-06-23T15:59:41Z Config file arg: rpcport="8332"
152021-06-23T15:59:41Z Config file arg: rpcuser=****
162021-06-23T15:59:41Z Config file arg: server="1"
172021-06-23T15:59:41Z Config file arg: sysperms="1"
182021-06-23T15:59:41Z Command-line arg: conf="/etc/bitcoin/bitcoin.conf"
192021-06-23T15:59:41Z Command-line arg: daemon=""
202021-06-23T15:59:41Z Command-line arg: datadir="/data/bitcoind"
212021-06-23T15:59:41Z Command-line arg: pid="/run/bitcoind/bitcoind.pid"
222021-06-23T15:59:41Z Command-line arg: sysperms="true"
232021-06-23T15:59:41Z Command-line arg: txindex="1"
242021-06-23T15:59:41Z Using at most 125 automatic connections (1024 file descriptors available)
252021-06-23T15:59:41Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
262021-06-23T15:59:41Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
272021-06-23T15:59:41Z Script verification uses 3 additional threads
282021-06-23T15:59:41Z scheduler thread start
292021-06-23T15:59:41Z WARNING: the RPC server is not safe to expose to untrusted networks such as the public internet
302021-06-23T15:59:41Z HTTP: creating work queue of depth 16
312021-06-23T15:59:41Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-bas
32ed auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
332021-06-23T15:59:41Z HTTP: starting 4 worker threads
342021-06-23T15:59:41Z Using wallet directory /data/bitcoind
352021-06-23T15:59:41Z init message: Verifying wallet(s)...
362021-06-23T15:59:41Z Using BerkeleyDB version Berkeley DB 4.8.30: (April 9, 2010)
372021-06-23T15:59:41Z Using wallet /data/bitcoind/wallet.dat
382021-06-23T15:59:41Z BerkeleyEnvironment::Open: LogDir=/data/bitcoind/database ErrorFile=/data/bitcoind/db.log
392021-06-23T16:05:41Z init message: Loading banlist...
402021-06-23T16:05:41Z SetNetworkActive: true
412021-06-23T16:05:41Z Using /16 prefix for IP bucketing
422021-06-23T16:05:41Z Cache configuration:
432021-06-23T16:05:41Z * Using 2.0 MiB for block index database
442021-06-23T16:05:41Z * Using 56.0 MiB for transaction index database
452021-06-23T16:05:41Z * Using 8.0 MiB for chain state database
462021-06-23T16:05:41Z * Using 384.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
472021-06-23T16:05:41Z init message: Loading block index...
482021-06-23T16:05:41Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
492021-06-23T16:05:41Z Opening LevelDB in /data/bitcoind/blocks/index
502021-06-23T16:05:41Z Opened LevelDB successfully
512021-06-23T16:05:41Z Using obfuscation key for /data/bitcoind/blocks/index: 0000000000000000
522021-06-23T16:05:47Z LoadBlockIndexDB: last block file = 2635
532021-06-23T16:05:47Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=39, size=54950192, heights=688559...688597, time=2021-06-23.
54..2021-06-23)
552021-06-23T16:05:47Z Checking all blk files are present...
562021-06-23T16:05:48Z Opening LevelDB in /data/bitcoind/chainstate
572021-06-23T16:05:48Z Opened LevelDB successfully
582021-06-23T16:05:48Z Using obfuscation key for /data/bitcoind/chainstate: 91cafe70bba20bc0
592021-06-23T16:05:48Z Loaded best chain: hashBestChain=000000000000000000082b4e5e90e45bba65c64e1353915b7831d077a54af5b4 height=688597 date=2021-
6006-23T15:28:40Z progress=0.999988
612021-06-23T16:05:48Z init message: Rewinding blocks...
622021-06-23T16:05:48Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
632021-06-23T16:05:48Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
642021-06-23T16:05:48Z init message: Verifying blocks...
652021-06-23T16:05:48Z Verifying last 6 blocks at level 3
662021-06-23T16:05:48Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
672021-06-23T16:05:51Z No coin database inconsistencies in last 6 blocks (13250 transactions)
682021-06-23T16:05:51Z block index          10295ms
692021-06-23T16:05:51Z Opening LevelDB in /data/bitcoind/indexes/txindex
702021-06-23T16:05:51Z Opened LevelDB successfully
712021-06-23T16:05:51Z Using obfuscation key for /data/bitcoind/indexes/txindex: 0000000000000000
722021-06-23T16:05:51Z txindex thread start
732021-06-23T16:05:51Z txindex is enabled at height 688597
742021-06-23T16:05:51Z txindex thread exit
752021-06-23T16:05:51Z init message: Loading wallet...
762021-06-23T16:05:51Z BerkeleyEnvironment::Open: LogDir=/data/bitcoind/database ErrorFile=/data/bitcoind/db.log
772021-06-23T16:13:20Z [default wallet] Wallet File Version = 169900
782021-06-23T16:13:20Z [default wallet] Keys: 2001 plaintext, 0 encrypted, 3001 w/ metadata, 2001 total. Unknown wallet records: 0
792021-06-23T16:13:20Z [default wallet] Wallet completed loading in         449117ms
802021-06-23T16:13:20Z [default wallet] setKeyPool.size() = 2000
812021-06-23T16:13:20Z [default wallet] mapWallet.size() = 2946276
822021-06-23T16:13:20Z [default wallet] m_address_book.size() = 1000
832021-06-23T16:13:20Z block tree size = 688631
842021-06-23T16:13:20Z nBestHeight = 688597
852021-06-23T16:13:20Z loadblk thread start
862021-06-23T16:13:20Z Bound to [::]:8333
872021-06-23T16:13:20Z torcontrol thread start
882021-06-23T16:13:20Z Bound to 0.0.0.0:8333
892021-06-23T16:13:20Z Bound to 127.0.0.1:8334
902021-06-23T16:13:20Z init message: Loading P2P addresses...
912021-06-23T16:13:20Z Leaving InitialBlockDownload (latching to false)

Let me check with -txindex=0

MarcoFalke commented at 4:40 pm on June 23, 2021: member

If -txindex=0 doesn’t give any results, can you try disabling the public RPC? There might be an (accidental) DoS over RPC happening.

dzmitrykliapkou commented at 5:17 pm on June 23, 2021: none

Neither txindex nor server have helped. RAM usage is the same

MarcoFalke commented at 6:35 am on June 24, 2021: member

2021-06-23T16:13:20Z [default wallet] mapWallet.size() = 2946276

Can you try with the wallet disabled?

MarcoFalke added the label Wallet on Jun 24, 2021

dzmitrykliapkou commented at 7:00 am on June 24, 2021: none

Looks like disabling wallet has solved the issue What could be the reason of such behavior?

MarcoFalke removed the label Bug on Jun 24, 2021

MarcoFalke renamed this:
~~bitcoind RAM usage issue~~
wallet: bitcoind RAM usage issue
on Jun 24, 2021

MarcoFalke commented at 7:24 am on June 24, 2021: member

I am not familiar with the wallet, but I think all txs are kept in memory. Your wallet seems to have a lot of transactions.

This is the current (expected) behaviour. Maybe @achow101 know more about the long term plan to scale the wallet.

achow101 commented at 5:55 pm on June 24, 2021: member

Everything in the wallet is currently loaded into memory, so if your wallet file is large, so will the RAM usage for that wallet. There is some long term and ongoing work to load things on demand, but there is no expected time for when that might be complete.

dzmitrykliapkou closed this on Sep 7, 2021

ghost commented at 10:43 am on September 21, 2021: none

I had a similar problem some time ago, probably after upgrading to 0.21 but I cannot really recall when I first noticed bitcoind was consuming all RAM in my Linux box. It was running in an Arch system before I started running it in a Manjaro server at version 0.21 thereabouts, and it took me a while before noticing excessive RAM usage, so I cannot pinpoint when that problem started from memory.

I asked in bitcointalk https://bitcointalk.org/index.php?topic=5331310.msg56810036#msg56810036 where it was said RAM usage was not normal indeed.

After finding this Issue here, I tried the following steps: renamed wallet.dat (which just contained test stuff, anyways), restarted. It did not improved. Next, I disabled wallet, restarted, no chance. Then I disabled txindex. Restarted and saw an improvement!

Then I decided to re-enable each feature again to check precisely what was the culprit. I enabled wallet and then txindex, and it is working fine now (still did not create a new wallet.dat)..

Virtual memory usage is very high, but being cached memory, the kernel can do OK with it, pretty normal now, IMO. I still did not test the new bitcoind release..

But what I wanted to write here is that perhaps effect does not take until multiple restarts happen. Cheers!

ghost commented at 3:19 pm on September 21, 2021: none

@mountaineerbr Is it possible to run bitcoin core with this wallet and config on a different OS? Maybe in VM

I am interested to know the difference in memory usage because I had writen a simple script which had huge difference between Linux and Windows memory usage:

https://github.com/prayank23/bitcoin-ps-scripts/blob/main/Scripts/Wallet/CPU_RAM_sendtoaddress.ps1

ghost commented at 5:41 pm on September 21, 2021: none

@prayank23 You know you gave out a great idea on how to run bitcoind in my personal server and limit RAM usage easily.. Specially now that I am tentatively setting up a VM with FreeBSD but the VM is not properly configured nor even running from the server yet.. This procedure should warrant some benchmarking commentary from personal usage, however unfortunately (or fortunately) there are more urgent things for me to do right dow and in fact, today I just shut down my server.

I should also remove that bad wallet file (<5MB) as it was really random stuff, I reckon I might have tried one thing or two that could screw up wallet file itself long time ago.. I just remembered, for example, I set to it to watch some public addresses, and a blockchain rescan was needed, I don’t know if I killed the rescan at that time but maybe it was still rescanning..

I might add some notes here if I do it in the following 6 months or so but that is not promise. Hopefully you would have had enough benchmarking by then. Cheers!

DrahtBot locked this on Oct 30, 2022

wallet: bitcoind RAM usage issue #22329