libdb_cxx-4.8 leaks? #5380

I just stumbled over some (many) mem leaks from libdb_cxx-4.8. Local build of current master. Tested on OSX 10.10.

Does someone has a valgrind setup ready and can test this on a non-OSX system?

So are these just one-time leaks or do they accumulate as the application is running?

I wasn't looking very deep into this. But it seams that the it's leaking more and more... I will re-test, re-analyze it soon. I also tried in valgrind in a Ubuntu VM... horrible performance, but also detected some things. Need to create a better setup and will re-analyze soon.

Just did a valgrind session on ubuntu. LevelDB or the bitcoind usage of it seams to produce multiple leaks (not to underestimate!). I assume bitcoind produces some serious mem leaks.

Here is a session dump: this is a -regtest only session with a empty regtest dir with no further block creations. No nodes connected. http://paste.ubuntu.com/9379050/

Neither of those in the pasted log seems related to leveldb. Just BerkeleyDB and the wallet. What sizes are we talking about here? Do they grow over time? Leaking some one-time setup structures is unclean, but hardly a critical issue.

Sorry. The pastbin is somehow useless. Here is a valgrind output of a very short testnet session with downloading some 100 blocks (see below). It seams to be not a huge problem on ubuntu.

The definitive lost come from the wallet db (bdb) and might be fixed without huge effort. The still reachable blocks (which grow largely over time) are probably coming out of leveldb. But could be because of leveldb's internals and might be okay.

==20446== LEAK SUMMARY:
==20446==    definitely lost: 96 bytes in 1 blocks
==20446==    indirectly lost: 0 bytes in 0 blocks
==20446==      possibly lost: 152 bytes in 1 blocks
==20446==    still reachable: 1,708 bytes in 11 blocks
==20446==         suppressed: 0 bytes in 0 blocks
==20446== Reachable blocks (those to which a pointer was found) are not shown.
==20446== To see them, rerun with: --leak-check=full --show-leak-kinds=all

Sorry. Mix it up! LevelDB is clean. It's just the BDB Wallet who is leaking. I try to go behind it and fix those.

After a few minutes of running bitcoind on testnet, with wallet (-disablewallet -> no leaks):

xxx $ leaks bitcoind | grep "Call stack" | sort | uniq -c | sort -rn
80895   Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:791 | CDBEnv::Open(boost::filesystem::path const&) db.cpp:94 | DbEnv::open(char const*, unsigned int, int) | __env_open | __env_attach_regions | __lock_open | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
  44    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1103 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CWalletDB::LoadWallet(CWallet*) walletdb.cpp:635 | CDB::ReadAtCursor(Dbc*, CDataStream&, CDataStream&, unsigned int) .db.h:244 | Dbc::get(Dbt*, Dbt*, unsigned int) | __dbc_get_pp | __dbc_iget | __bamc_get | __bamc_next | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   7    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:150 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_common | __db_vrfy_putpageinfo | __db_put | __dbc_put | __dbc_iput | __bamc_put | __bam_split | __db_new | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   4    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:150 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_dbinfo_create | __db_open | __db_new_file | __bam_new_file | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   3    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:150 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_dbinfo_create | __db_open | __env_setup | __env_mpool | __memp_fopen | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   2    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:150 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_dbinfo_create | __db_vrfy_pgset | __db_open | __db_new_file | __bam_new_file | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   2    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:150 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __bam_vrfy_structure | __bam_vrfy_subtree | __bam_vrfy_subtree | __db_vrfy_pgset_inc | __db_put | __dbc_put | __dbc_iput | __bamc_put | __bam_split | __db_new | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:791 | CDBEnv::Open(boost::filesystem::path const&) db.cpp:94 | DbEnv::open(char const*, unsigned int, int) | __env_open | __env_attach_regions | __memp_open | __memp_init | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1103 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CWalletDB::LoadWallet(CWallet*) walletdb.cpp:635 | CDB::ReadAtCursor(Dbc*, CDataStream&, CDataStream&, unsigned int) .db.h:244 | Dbc::get(Dbt*, Dbt*, unsigned int) | __dbc_get_pp | __dbc_iget | __db_goff | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
xxx $

I have tried bitcoind with db-6.1.19.NC and --with-incompatible-bdb and it is much better (the first type of leaks is resolved, but two new "invented"):

  49    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1104 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CWalletDB::LoadWallet(CWallet*) walletdb.cpp:635 | CDB::ReadAtCursor(Dbc*, CDataStream&, CDataStream&, unsigned int) .db.h:244 | Dbc::get(Dbt*, Dbt*, unsigned int) | __dbc_get_pp | __dbc_iget | __bamc_get | __bamc_next | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   7    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:151 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_common | __db_vrfy_putpageinfo | __db_put | __dbc_put | __dbc_iput | __bamc_put | __bam_split | __db_new | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   4    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:151 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_dbinfo_create | __db_open | __bam_new_file | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   4    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1104 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CWalletDB::LoadWallet(CWallet*) walletdb.cpp:635 | CDB::ReadAtCursor(Dbc*, CDataStream&, CDataStream&, unsigned int) .db.h:244 | Dbc::get(Dbt*, Dbt*, unsigned int) | __dbc_get_pp | __dbc_iget | __db_goff | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   3    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:151 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_dbinfo_create | __db_vrfy_pgset | __db_open | __env_setup | __env_mpool | __memp_fopen | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   3    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:151 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_dbinfo_create | __db_open | __env_setup | __env_mpool | __memp_fopen | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   2    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:151 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __db_vrfy_dbinfo_create | __db_vrfy_pgset | __db_open | __bam_new_file | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   2    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:820 | CDBEnv::Verify(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool (*)(CDBEnv&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)) db.cpp:151 | Db::verify(char const*, char const*, std::__1::basic_ostream<char, std::__1::char_traits<char> >*, unsigned int) | __db_verify_internal | __db_verify | __bam_vrfy_structure | __bam_vrfy_subtree | __bam_vrfy_subtree | __db_vrfy_pgset_inc | __db_put | __dbc_put | __dbc_iput | __bamc_put | __bam_split | __db_new | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:791 | CDBEnv::Open(boost::filesystem::path const&) db.cpp:94 | DbEnv::open(char const*, unsigned int, int) | __env_open | __env_attach_regions | __memp_open | __memp_init | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1104 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CWalletDB::LoadWallet(CWallet*) walletdb.cpp:635 | CDB::ReadAtCursor(Dbc*, CDataStream&, CDataStream&, unsigned int) .db.h:244 | Dbc::get(Dbt*, Dbt*, unsigned int) | __dbc_get_pp | __dbc_iget | __bamc_get | __bamc_search | __bam_search | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1104 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CDB::CDB(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) db.cpp:266 | bool CDB::Exists<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) .db.h:211 | Db::exists(DbTxn*, Dbt*, unsigned int) | __db_exists | __db_get_pp | __db_get | __dbc_iget | __bamc_get | __bamc_search | __bam_search | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1104 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CDB::CDB(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) db.cpp:266 | bool CDB::Exists<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) .db.h:211 | Db::exists(DbTxn*, Dbt*, unsigned int) | __db_exists | __db_get_pp | __db_get | __dbc_iget | __bamc_get | __bamc_search | __bam_search | __bam_get_root | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x7fff771b0300]: | start | main bitcoind.cpp:185 | AppInit(int, char**) bitcoind.cpp:148 | AppInit2(boost::thread_group&) init.cpp:1104 | CWallet::LoadWallet(bool&) wallet.cpp:1652 | CDB::CDB(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) db.cpp:251 | Db::open(DbTxn*, char const*, char const*, DBTYPE, unsigned int, int) | __db_open_pp | __db_open | __bam_read_root | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x11492e000]: | thread_start | _pthread_body | _pthread_body | boost::(anonymous namespace)::thread_proxy(void*) | ThreadFlushWalletDB(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) walletdb.cpp:840 | DbEnv::lsn_reset(char const*, unsigned int) | __env_lsn_reset_pp | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 
   1    Call stack: [thread 0x11492e000]: | thread_start | _pthread_body | _pthread_body | boost::(anonymous namespace)::thread_proxy(void*) | ThreadFlushWalletDB(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) walletdb.cpp:840 | DbEnv::lsn_reset(char const*, unsigned int) | __env_lsn_reset_pp | __db_open | __bam_read_root | __memp_fget | __memp_alloc | __env_alloc | __os_malloc | malloc | malloc_zone_malloc 

10s snapshots:

Process 63781: 81027 leaks for 9409888 total leaked bytes.
Process 63781: 81027 leaks for 9409888 total leaked bytes.
Process 63781: 81027 leaks for 9409888 total leaked bytes.

10MB constantly.

As all leaks are coming from libdb, I'd suggest closing this issue.

I think as long as bitcoin core produces leaks (even because of leaking subsystems) we should consider keeping this issue open. My debuggings lead me to possible forgotten database closing (but didn't had a deep look)

But for sure: it's low priority.

I don't think there is anything we can do to prevent leaks in the unmaintained bdb other than moving away from it. See #18916