bitcoind 'compacts' chainstate database almost every time I run it

dooglus commented at 1:22 am on February 4, 2017: contributor

I’m running the current master branch version (1c2edd9f) and find that most times when I start bitcoind it spends about 10 minutes thrashing my hard drive doing a ‘compaction’ of the chainstate database.

The relevant stack trace:

[#9](/bitcoin-bitcoin/9/)  0x000055555596ce1f in leveldb::DBImpl::DoCompactionWork (this=0x7fffd8619e90, compact=0x7fffa8015860) at leveldb/db/db_impl.cc:1006
[#10](/bitcoin-bitcoin/10/) 0x000055555596d69f in leveldb::DBImpl::BackgroundCompaction (this=0x7fffd8619e90) at leveldb/db/db_impl.cc:736
[#11](/bitcoin-bitcoin/11/) 0x000055555596e192 in leveldb::DBImpl::BackgroundCall (this=0x7fffd8619e90) at leveldb/db/db_impl.cc:674
[#12](/bitcoin-bitcoin/12/) 0x00005555559951a3 in BGThread (this=0x7fffd847c490) at leveldb/util/env_posix.cc:582
[#13](/bitcoin-bitcoin/13/) leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7fffd847c490) at leveldb/util/env_posix.cc:521
[#14](/bitcoin-bitcoin/14/) 0x00007ffff4cc30a4 in start_thread (arg=0x7fffb73bc700) at pthread_create.c:309

I tried restarting bitcoind 10 times and kept a copy of the LOG file each time. Here are the file sizes:

-rw------- 1 chris chris 184366 Feb  3 11:19 LOG.00
-rw------- 1 chris chris 182024 Feb  3 11:30 LOG.01
-rw------- 1 chris chris    344 Feb  3 12:11 LOG.02
-rw------- 1 chris chris 194758 Feb  3 12:27 LOG.03
-rw------- 1 chris chris    344 Feb  3 12:27 LOG.04
-rw------- 1 chris chris 187248 Feb  3 12:36 LOG.05
-rw------- 1 chris chris    571 Feb  3 15:28 LOG.06
-rw------- 1 chris chris 189728 Feb  3 15:45 LOG.07
-rw------- 1 chris chris 194434 Feb  3 16:00 LOG.08
-rw------- 1 chris chris 188599 Feb  3 16:16 LOG.09

7 out of 10 times it did a full recompact, causing the computer to almost come to a standstill while it kept the harddrive busy. One of the 10 times bitcoind crashed hard almost immediately, but I’m sure that’s a separate issue (#9683).

Here’s the LOG file from the ~/.bitcoin/chainstate/ folder.

Edit: note that I use:

./configure --with-incompatible-bdb

And see “Using BerkeleyDB version Berkeley DB 5.3.28: (September 9, 2013)” in the debug.log.

If I run configure without that flag I see an error:

configure: error: Found Berkeley DB other than 4.8, required for portable wallets (--with-incompatible-bdb to ignore or --disable-wallet to disable wallet functionality)

TheBlueMatt commented at 2:02 am on February 4, 2017: member

Is this a regression? LevelDB has always done compaction at startup, AFAIU.

On February 3, 2017 8:23:00 PM EST, Chris Moore notifications@github.com wrote:

I’m running the current master branch version (1c2edd9f) and find that most times when I start bitcoind it spends about 10 minutes thrashing my hard drive doing a ‘compaction’ of the chainstate database.

The relevant stack trace:

#9 0x000055555596ce1f in leveldb::DBImpl::DoCompactionWork (this=0x7fffd8619e90, compact=0x7fffa8015860) at leveldb/db/db_impl.cc:1006 #10 0x000055555596d69f in leveldb::DBImpl::BackgroundCompaction (this=0x7fffd8619e90) at leveldb/db/db_impl.cc:736 #11 0x000055555596e192 in leveldb::DBImpl::BackgroundCall (this=0x7fffd8619e90) at leveldb/db/db_impl.cc:674 #12 0x00005555559951a3 in BGThread (this=0x7fffd847c490) at leveldb/util/env_posix.cc:582 #13 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7fffd847c490) at leveldb/util/env_posix.cc:521 #14 0x00007ffff4cc30a4 in start_thread (arg=0x7fffb73bc700) at pthread_create.c:309

I tried restarting bitcoind 10 times and kept a copy of the LOG file each time. Here are the file sizes:

-rw——- 1 chris chris 184366 Feb 3 11:19 LOG.00 -rw——- 1 chris chris 182024 Feb 3 11:30 LOG.01 -rw——- 1 chris chris 344 Feb 3 12:11 LOG.02 -rw——- 1 chris chris 194758 Feb 3 12:27 LOG.03 -rw——- 1 chris chris 344 Feb 3 12:27 LOG.04 -rw——- 1 chris chris 187248 Feb 3 12:36 LOG.05 -rw——- 1 chris chris 571 Feb 3 15:28 LOG.06 -rw——- 1 chris chris 189728 Feb 3 15:45 LOG.07 -rw——- 1 chris chris 194434 Feb 3 16:00 LOG.08 -rw——- 1 chris chris 188599 Feb 3 16:16 LOG.09

7 out of 10 times it did a full recompact, causing the computer to almost come to a standstill while it kept the harddrive busy. One of the 10 times bitcoind crashed hard almost immediately, but I’m sure that’s a separate issue (#9683).

Here’s the LOG file from the ~/.bitcoin/chainstate/ folder.

– You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/9684

dooglus commented at 2:15 am on February 4, 2017: contributor

I used to be able to start bitcoind without it thrashing my hard drive for 10 minutes. And sometimes I still can (3 out of 10 times in my recent tests). Did it always read and rewrite most of the chainstate db on startup?

sipa commented at 2:23 am on February 4, 2017: member

I think it’s related to how much has been written to the database before shutdown, and how long that was. With larger dbcache values in recent versions (in 0.14/master it can be up to 600M by default, if the mempool is empty), there may be a large dump to disk of dirty entries at shutdown, which need compaction at startup.

sipa commented at 2:24 am on February 4, 2017: member

@dooglus What version/options were you using last where you did not notice this behaviour?

dooglus commented at 8:02 am on February 4, 2017: contributor

I think it’s related to how much has been written to the database before shutdown, and how long that was.

I was seeing a full recompact even when I had a fully synced blockchain and restarted shortly after starting up. ie. I’d start bitcoind, wait 10 minutes for the recompact to finish, then stop and restart it, and I’d see another full recompact immediately.

What version/options were you using last where you did not notice this behaviour?

I don’t know. I didn’t know this was a behavior that could happen until I first noticed it happening. These are the settings I use now, when it does happen:

keypool=10
logips=1
minrelaytxfee=0.0001
paytxfee=0.0001
upnp=0
spendzeroconfchange=0
logips=1
bind=127.0.0.1
maxconnections=32
testnet=0
txindex=1
checkblocks=1
listen=0
server=1
daemon=1

I changed to daemon=0 when testing with bitcoind instead of bitcoin-cli, but it didn’t seem to change the behavior.

When I first noticed the behavior I was running from commit 453bda63. In the past I have run bitcored from bitpay, which comes with its own bitcoind executable, Bitcoin version v0.11.2.0-bc99ef3 (2016-04-08 10:42:17 -0400). I suppose that could be statically linked with a different leveldb version, and may be contributing to the behavior I’m seeing, since I run it on the same ~/.bitcoin/ directory.

I’ll can rebuilding from older commits and see if I can determine which commit introduced the behavior if that would be helpful.

sipa commented at 8:21 am on February 4, 2017: member

Well the compaction will always run at startup, I believe, and always has. The question is whether it takes a long time.

v0.11.2 had a much smaller dbcache, and did much more frequent writes to disk. Also, txindex may affect performance (though that’s in the block index db, not the chainstate).

laanwj added the label UTXO Db and Indexes on Feb 6, 2017

eklitzke commented at 7:20 am on March 10, 2018: contributor

There are some things we can do to improve this but they all require modifying our in-tree copy of the LevelDB source code. That’s something I’d like to do anyway, but I’m not sure everyone else is convinced that’s a good idea. @dooglus In the meantime you can use ionice to deprioritize the compaction thread. On Linux you can set this to a specific thread id. Usually the LevelDB compaction thread is the highest numbered thread id in the output of ps -eL. (Renaming that thread is another thing I’d like to change in the LevelDB source code).

MarcoFalke commented at 7:21 pm on May 8, 2020: member

Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce?

MarcoFalke closed this on May 8, 2020

DrahtBot locked this on Feb 15, 2022

bitcoind ‘compacts’ chainstate database almost every time I run it #9684