Regular crash+autorestart after loading/unloading many wallets #19232

issue fscemama opened this issue on June 10, 2020

fscemama commented at 9:05 AM on June 10, 2020: none


Hi everyone, We use bitcoin core with hundreds of wallets. The only efficient way we have found to do so is :
1. wait for any bitcoin-cli command to terminate before issuing another one
2. never issue any bitcoin-cli command unless tip update is 1
3. always unload a wallet after loading and using it
Now, we observ that bitcoin core systematically crashes every 20-30 minutes, then automatically restarts. And the problem is, it has to re-sync from about 0.997, which easily takes 15-20 minutes. During that time, the daemon cannot be used.

If we don't load/unload wallets, the daemon is stable and keeps to 1, no issue.

System information Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd Compiled and run on Debian 9, on a VM with 16 mo memory. Launched that way : /bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1

bitcoin.conf : server=1 dbcache=16384 par=1 blocksonly=1 maxuploadtarget=137 maxconnections=16 rpcuser=bitcoinrpc rpcpassword=xxxxxx rpcallowip=127.0.0.1 testnet=0 txindex=1 walletbroadcast=1 txconfirmtarget=1

Please note that the same behaviour was happening with older releases (from 0.17.x).

Thanks for any help. Please ask for any needed information / testing. Regards,
fscemama added the label Bug on Jun 10, 2020
MarcoFalke commented at 10:15 AM on June 10, 2020: member

Are there any debug log excerpts at the time of the crash? Or stderr/stdout?

fscemama commented at 10:18 AM on June 10, 2020: none

Are there any debug log excerpts at the time of the crash? Or stderr/stdout?

Hi Marco, typical sequence of debug.log below. Thanks !

2020-06-10T09:39:55Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2020-06-10T09:39:55Z Using wallet /root/wallets/xxxxx-2019-10-15-18-48-48-02
2020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
2020-06-10T09:39:55Z init message: Loading wallet...
2020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
2020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet File Version = 169900
2020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Keys: 2027 plaintext, 0 encrypted, 2027 w/ metadata, 2027 total. Unknown wallet records: 0
2020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet completed loading in              61ms
2020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] setKeyPool.size() = 2000
2020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] mapWallet.size() = 11
2020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] m_address_book.size() = 26
2020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Releasing wallet
2020-06-10T09:39:55Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2020-06-10T09:39:55Z Using wallet /root/wallets/xxxxx-2019-10-15-18-48-48-02
2020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
2020-06-10T09:39:55Z init message: Loading wallet...
2020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet File Version = 169900
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Keys: 2027 plaintext, 0 encrypted, 2027 w/ metadata, 2027 total. Unknown wallet records: 0
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet completed loading in              43ms
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] setKeyPool.size() = 2000
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] mapWallet.size() = 11
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] m_address_book.size() = 26
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Releasing wallet
2020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-11-27-12-00-18-01] Releasing wallet
2020-06-10T09:39:57Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2020-06-10T09:39:57Z Using wallet /root/wallets/xxxxx-2020-04-10-23-21-36-02
2020-06-10T09:39:57Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2020-04-10-23-21-36-02/database ErrorFile=/root/wallets/xxxxx-2020-04-10-23-21-36-02/db.log
2020-06-10T09:39:57Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2020-06-10T09:39:57Z Using wallet /root/wallets/xxxxx-2020-04-10-23-21-36-02
2020-06-10T09:39:57Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2020-04-10-23-21-36-02/database ErrorFile=/root/wallets/xxxxx-2020-04-10-23-21-36-02/db.log
2020-06-10T09:39:57Z init message: Loading wallet...





2020-06-10T09:40:02Z Bitcoin Core version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd (release build)
2020-06-10T09:40:02Z InitParameterInteraction: parameter interaction: -blocksonly=1 -> setting -whitelistrelay=0
2020-06-10T09:40:02Z Assuming ancestors of block 0000000000000000000f2adce67e49b0b6bdeb9de8b7c3d7e93b21e7fc1e819d have valid signatures.
2020-06-10T09:40:02Z Setting nMinimumChainWork=00000000000000000000000000000000000000000e1ab5ec9348e9f4b8eb8154
2020-06-10T09:40:02Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2020-06-10T09:40:02Z Using RdSeed as additional entropy source
2020-06-10T09:40:02Z Using RdRand as an additional entropy source
2020-06-10T09:40:02Z Default data directory /root/.bitcoin
2020-06-10T09:40:02Z Using data directory /root/.bitcoin
2020-06-10T09:40:02Z Config file: /root/.bitcoin/bitcoin.conf
2020-06-10T09:40:02Z Config file arg: blocksonly="1"
2020-06-10T09:40:02Z Config file arg: dbcache="16384"
2020-06-10T09:40:02Z Config file arg: maxconnections="16"
2020-06-10T09:40:02Z Config file arg: maxuploadtarget="137"
2020-06-10T09:40:02Z Config file arg: par="1"
2020-06-10T09:40:02Z Config file arg: rpcallowip="127.0.0.1"
2020-06-10T09:40:02Z Config file arg: rpcpassword=****
2020-06-10T09:40:02Z Config file arg: rpcuser=****
2020-06-10T09:40:02Z Config file arg: server="1"
2020-06-10T09:40:02Z Config file arg: testnet="0"
2020-06-10T09:40:02Z Config file arg: txconfirmtarget="1"
2020-06-10T09:40:02Z Config file arg: txindex="1"
2020-06-10T09:40:02Z Config file arg: walletbroadcast="1"
2020-06-10T09:40:02Z Command-line arg: conf="/root/.bitcoin/bitcoin.conf"
2020-06-10T09:40:02Z Command-line arg: daemon=""
2020-06-10T09:40:02Z Command-line arg: fallbackfee="0.00001"
2020-06-10T09:40:02Z Command-line arg: rpcworkqueue="128"
2020-06-10T09:40:02Z Command-line arg: txindex="1"
2020-06-10T09:40:02Z Command-line arg: walletrbf=""
2020-06-10T09:40:02Z Using at most 16 automatic connections (1024 file descriptors available)
2020-06-10T09:40:02Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2020-06-10T09:40:02Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2020-06-10T09:40:02Z Script verification uses 0 additional threads
2020-06-10T09:40:02Z scheduler thread start
2020-06-10T09:40:02Z WARNING: option -rpcallowip was specified without -rpcbind; this doesn't usually make sense
2020-06-10T09:40:02Z HTTP: creating work queue of depth 128
2020-06-10T09:40:02Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
2020-06-10T09:40:02Z HTTP: starting 4 worker threads
2020-06-10T09:40:02Z Using wallet directory /root/.bitcoin/wallets
2020-06-10T09:40:02Z init message: Verifying wallet(s)...
2020-06-10T09:40:02Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2020-06-10T09:40:02Z Using wallet /root/.bitcoin/wallets
2020-06-10T09:40:02Z BerkeleyEnvironment::Open: LogDir=/root/.bitcoin/wallets/database ErrorFile=/root/.bitcoin/wallets/db.log
2020-06-10T09:40:03Z init message: Loading banlist...
2020-06-10T09:40:03Z Using /16 prefix for IP bucketing
2020-06-10T09:40:03Z Cache configuration:
2020-06-10T09:40:03Z * Using 2.0 MiB for block index database
2020-06-10T09:40:03Z * Using 1024.0 MiB for transaction index database
2020-06-10T09:40:03Z * Using 8.0 MiB for chain state database
2020-06-10T09:40:03Z * Using 15350.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2020-06-10T09:40:03Z init message: Loading block index...
2020-06-10T09:40:03Z Opening LevelDB in /root/.bitcoin/blocks/index
2020-06-10T09:40:03Z Opened LevelDB successfully
2020-06-10T09:40:03Z Using obfuscation key for /root/.bitcoin/blocks/index: 0000000000000000
2020-06-10T09:40:10Z LoadBlockIndexDB: last block file = 2106
2020-06-10T09:40:10Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=134, size=132209876, heights=633875...634008, time=2020-06-09...2020-06-10)
2020-06-10T09:40:10Z Checking all blk files are present...
2020-06-10T09:40:11Z Opening LevelDB in /root/.bitcoin/chainstate
2020-06-10T09:40:11Z Opened LevelDB successfully
2020-06-10T09:40:11Z Using obfuscation key for /root/.bitcoin/chainstate: 4501f5bf5624cc54
2020-06-10T09:40:11Z Loaded best chain: hashBestChain=0000000000000000000fc56a5b8b7c1c3ce9ba39acc75cc13db25fb5bc8c62e0 height=633123 date=2020-06-05T03:46:00Z progress=0.997213
2020-06-10T09:40:11Z init message: Rewinding blocks...
2020-06-10T09:40:11Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
2020-06-10T09:40:11Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
2020-06-10T09:40:11Z init message: Verifying blocks...
2020-06-10T09:40:11Z Verifying last 6 blocks at level 3
2020-06-10T09:40:11Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2020-06-10T09:43:08Z No coin database inconsistencies in last 6 blocks (13097 transactions)
2020-06-10T09:43:08Z  block index          185680ms
2020-06-10T09:43:08Z Opening LevelDB in /root/.bitcoin/indexes/txindex
2020-06-10T09:43:17Z Opened LevelDB successfully
2020-06-10T09:43:17Z Using obfuscation key for /root/.bitcoin/indexes/txindex: 0000000000000000
2020-06-10T09:43:17Z txindex thread start
2020-06-10T09:43:17Z txindex is enabled at height 633123
2020-06-10T09:43:17Z txindex thread exit
2020-06-10T09:43:17Z init message: Loading wallet...
2020-06-10T09:43:17Z BerkeleyEnvironment::Open: LogDir=/root/.bitcoin/wallets/database ErrorFile=/root/.bitcoin/wallets/db.log
2020-06-10T09:43:17Z [default wallet] Wallet File Version = 169900
2020-06-10T09:43:17Z [default wallet] Keys: 2001 plaintext, 0 encrypted, 2001 w/ metadata, 2001 total. Unknown wallet records: 0
2020-06-10T09:43:18Z [default wallet] Wallet completed loading in             556ms
2020-06-10T09:43:18Z [default wallet] setKeyPool.size() = 2000
2020-06-10T09:43:18Z [default wallet] mapWallet.size() = 0
2020-06-10T09:43:18Z [default wallet] m_address_book.size() = 0
2020-06-10T09:43:18Z block tree size = 634017
2020-06-10T09:43:18Z nBestHeight = 633123
2020-06-10T09:43:18Z torcontrol thread start
2020-06-10T09:43:18Z Bound to [::]:8333
2020-06-10T09:43:18Z Bound to 0.0.0.0:8333
2020-06-10T09:43:18Z init message: Loading P2P addresses...
2020-06-10T09:43:19Z Loaded 58642 addresses from peers.dat  880ms
2020-06-10T09:43:19Z init message: Starting network threads...
2020-06-10T09:43:19Z net thread start
2020-06-10T09:43:19Z dnsseed thread start
2020-06-10T09:43:19Z addcon thread start
2020-06-10T09:43:19Z opencon thread start
2020-06-10T09:43:19Z init message: Done loading
2020-06-10T09:43:19Z msghand thread start
2020-06-10T09:43:19Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2020-06-10T09:43:19Z Using wallet /root/wallets/keplerk
2020-06-10T09:43:19Z BerkeleyEnvironment::Open: LogDir=/root/wallets/keplerk/database ErrorFile=/root/wallets/keplerk/db.log
2020-06-10T09:43:27Z init message: Loading wallet...
2020-06-10T09:43:30Z Loading addresses from DNS seed seed.bitcoin.jonasschnelli.ch
2020-06-10T09:43:38Z UpdateTip: new best=0000000000000000000891768c73b989c11ab68ef59a7272d4953e1e40110e50 height=633124 version=0x3fff0000 log2_work=92.004565 tx=536058118 date='2020-06-05T03:49:48Z' progress=0.997213 cache=0.9MiB(7204txo)
2020-06-10T09:43:40Z Loading addresses from DNS seed dnsseed.emzy.de
2020-06-10T09:43:50Z Loading addresses from DNS seed dnsseed.bluematt.me
2020-06-10T09:43:55Z UpdateTip: new best=000000000000000000109465e408d8cac29bf79001ab12b352344186f3f59171 height=633125 version=0x20000000 log2_work=92.004582 tx=536059715 date='2020-06-05T03:59:30Z' progress=0.997216 cache=1.9MiB(14138txo)
2020-06-10T09:44:11Z Loading addresses from DNS seed seed.btc.petertodd.org
2020-06-10T09:44:16Z Loading addresses from DNS seed seed.bitcoin.sprovoost.nl
2020-06-10T09:44:21Z UpdateTip: new best=000000000000000000141612a0f0dd110e5fd29db90bcd9a2f103684ca6e669d height=633126 version=0x37ffe000 log2_work=92.0046 tx=536061851 date='2020-06-05T04:12:56Z' progress=0.997221 cache=3.0MiB(22542txo)
2020-06-10T09:44:23Z UpdateTip: new best=00000000000000000007afd66af6119e5568df8b72d31d3b8699a7011225c69c height=633127 version=0x20400000 log2_work=92.004617 tx=536062401 date='2020-06-05T04:15:16Z' progress=0.997222 cache=3.1MiB(23691txo)
2020-06-10T09:44:26Z Loading addresses from DNS seed dnsseed.bitcoin.dashjr.org
2020-06-10T09:44:41Z UpdateTip: new best=000000000000000000091f1a4746918881ff3afc6475c9fb92a9b63c4be9347e height=633128 version=0x20c00000 log2_work=92.004634 tx=536064469 date='2020-06-05T04:28:18Z' progress=0.997227 cache=4.3MiB(30860txo)
2020-06-10T09:44:47Z Loading addresses from DNS seed seed.bitcoin.sipa.be
2020-06-10T09:44:56Z UpdateTip: new best=00000000000000000013fa5d77449e82bf43ff6db20794752ae5fe07e11a6537 height=633129 version=0x20000000 log2_work=92.004651 tx=536066188 date='2020-06-05T04:39:39Z' progress=0.997231 cache=5.0MiB(36837txo)

fscemama commented at 2:16 PM on June 10, 2020: none

Currently, it's been up and running for one hour without crashing. I can't tell why nor when it does accurately. There's something behind.
MarcoFalke commented at 2:20 PM on June 10, 2020: member

Given that this is somewhat reproducible, could you try running the binary in gdb or valgrind?
fscemama commented at 2:40 PM on June 10, 2020: none

I'll do it with valgrind tonight. Must unfortunately be run in live, so I need a quiet period...
fscemama commented at 3:13 PM on June 10, 2020: none
In our dev environment, I got this crash. It seems that the creation of wallets (we do use bitcoin core for this too) might be the cause, and it may be worth testing in the live environment if crashes still happen if no wallet is never created. I'll give this a try.
```
root# valgrind --leak-check=yes  /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128
==32342== Memcheck, a memory error detector
==32342== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==32342== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==32342== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128
==32342==
Bitcoin Core starting
==32342==
==32342== HEAP SUMMARY:
==32342==     in use at exit: 2,232,114 bytes in 2,271 blocks
==32342==   total heap usage: 3,234 allocs, 963 frees, 2,462,760 bytes allocated
==32342==
==32342== LEAK SUMMARY:
==32342==    definitely lost: 0 bytes in 0 blocks
==32342==    indirectly lost: 0 bytes in 0 blocks
==32342==      possibly lost: 0 bytes in 0 blocks
==32342==    still reachable: 2,232,114 bytes in 2,271 blocks
==32342==         suppressed: 0 bytes in 0 blocks
==32342== Reachable blocks (those to which a pointer was found) are not shown.
==32342== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==32342==
==32342== For lists of detected and suppressed errors, rerun with: -s
==32342== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==32364==
==32364== Process terminating with default action of signal 6 (SIGABRT)
==32364==    at 0x6350FFF: raise (raise.c:51)
==32364==    by 0x6352429: abort (abort.c:89)
==32364==    by 0x6349E66: __assert_fail_base (assert.c:92)
==32364==    by 0x6349F11: __assert_fail (assert.c:101)
==32364==    by 0x49085A: BerkeleyDatabase (db.h:125)
==32364==    by 0x49085A: std::unique_ptr<BerkeleyDatabase, std::default_delete<BerkeleyDatabase> > MakeUnique<BerkeleyDatabase, std::shared_ptr<BerkeleyEnvironment>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::shared_ptr<BerkeleyEnvironment>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (memory.h:16)
==32364==    by 0x49091F: BerkeleyDatabase::Create(boost::filesystem::path const&) (db.h:139)
==32364==    by 0x47E2C1: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3805)
==32364==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
==32364==    by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626)
==32364==    by 0x22BFEB: operator() (server.h:104)
==32364==    by 0x22BFEB: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
==32364==    by 0x14A95E: operator() (functional:2127)
==32364==    by 0x14A95E: operator() (chain.cpp:208)
==32364==    by 0x14A95E: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
==32364==    by 0x2B32DB: operator() (functional:2127)
==32364==    by 0x2B32DB: ExecuteCommand (server.cpp:453)
==32364==    by 0x2B32DB: CRPCTable::execute(JSONRPCRequest const&) const (server.cpp:436)
==32364==
```
edit: Comment shortened by @MarcoFalke

<!-- ==32364== HEAP SUMMARY: ==32364== in use at exit: 463,096,888 bytes in 3,727,077 blocks ==32364== total heap usage: 9,714,945 allocs, 5,987,868 frees, 1,242,011,790 bytes allocated ==32364== ==32364== 12 bytes in 1 blocks are possibly lost in loss record 87 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C69F7: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== ==32364== 12 bytes in 1 blocks are possibly lost in loss record 88 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C69F7: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== ==32364== 13 bytes in 1 blocks are possibly lost in loss record 89 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53971FE: __dbreg_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5366AF9: __env_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53865E3: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B8ACE: Db::open(DbTxn*, char const*, char const*, DBTYPE, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DCA76: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:550) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 19 bytes in 1 blocks are possibly lost in loss record 187 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C17CE: __memp_fopen (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53666A4: __env_mpool (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5366A59: __env_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53865E3: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53678EA: __db_master_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53AE11E: __fop_subdb_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5386874: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 19 bytes in 1 blocks are possibly lost in loss record 188 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539716B: __dbreg_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5366AF9: __env_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53865E3: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B8ACE: Db::open(DbTxn*, char const*, char const*, DBTYPE, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DCA76: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:550) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 24 bytes in 1 blocks are possibly lost in loss record 420 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C210A: __memp_fopen (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53666A4: __env_mpool (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5366A59: __env_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53865E3: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53678EA: __db_master_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53AE11E: __fop_subdb_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5386874: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 28 bytes in 1 blocks are possibly lost in loss record 428 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C1FCD: __memp_fopen (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53666A4: __env_mpool (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5366A59: __env_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53865E3: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53678EA: __db_master_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53AE11E: __fop_subdb_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5386874: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 89 bytes in 1 blocks are possibly lost in loss record 1,005 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B0CB: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 96 bytes in 1 blocks are possibly lost in loss record 1,035 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535E7CF: __mutex_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A1B90: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 96 bytes in 1 blocks are possibly lost in loss record 1,036 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53970D9: __dbreg_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5366AF9: __env_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53865E3: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B8ACE: Db::open(DbTxn*, char const*, char const*, DBTYPE, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DCA76: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:550) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 96 bytes in 1 blocks are possibly lost in loss record 1,037 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535E7CF: __mutex_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A1B90: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626) ==32364== ==32364== 120 bytes in 1 blocks are definitely lost in loss record 1,073 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C68E5: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== ==32364== 120 bytes in 1 blocks are definitely lost in loss record 1,074 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C68E5: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== ==32364== 160 bytes in 1 blocks are definitely lost in loss record 1,314 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x3EBDEF: VerifyWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:58) ==32364== by 0x15E76D: AppInitMain(NodeContext&) (init.cpp:1334) ==32364== ==32364== 160 bytes in 1 blocks are definitely lost in loss record 1,315 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626) ==32364== ==32364== 192 bytes in 1 blocks are possibly lost in loss record 1,344 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53DF793: __txn_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A219C: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 216 bytes in 1 blocks are possibly lost in loss record 1,358 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C1668: __memp_fopen (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53666A4: __env_mpool (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5366A59: __env_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53865E3: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53678EA: __db_master_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53AE11E: __fop_subdb_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5386874: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,404 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x4E54478: boost::thread::start_thread_noexcept() (in /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0) ==32364== by 0x15E5AE: start_thread (thread.hpp:178) ==32364== by 0x15E5AE: thread<std::_Bind<void (*(char const*, std::function<void()>))(char const*, std::function<void()>)>&> (thread.hpp:265) ==32364== by 0x15E5AE: create_thread<std::_Bind<void (*(char const*, std::function<void()>))(char const*, std::function<void()>)> > (thread_group.hpp:79) ==32364== by 0x15E5AE: AppInitMain(NodeContext&) (init.cpp:1293) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,405 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x3926D0: thread<bool (&)(event_base*), event_base*&> (thread:137) ==32364== by 0x3926D0: StartHTTPServer() (httpserver.cpp:431) ==32364== by 0x15F6ED: AppInitServers (init.cpp:777) ==32364== by 0x15F6ED: AppInitMain(NodeContext&) (init.cpp:1328) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,406 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x392786: thread<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (thread:137) ==32364== by 0x392786: construct<std::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (new_allocator.h:120) ==32364== by 0x392786: construct<std::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (alloc_traits.h:475) ==32364== by 0x392786: emplace_back<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (vector.tcc:96) ==32364== by 0x392786: StartHTTPServer() (httpserver.cpp:434) ==32364== by 0x15F6ED: AppInitServers (init.cpp:777) ==32364== by 0x15F6ED: AppInitMain(NodeContext&) (init.cpp:1328) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,407 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x624F7D: thread<void (&)(leveldb::(anonymous namespace)::PosixEnv*), leveldb::(anonymous namespace)::PosixEnv*> (thread:137) ==32364== by 0x624F7D: leveldb::(anonymous namespace)::PosixEnv::Schedule(void (*)(void*), void*) (env_posix.cc:802) ==32364== by 0x5FA187: leveldb::DBImpl::RecordReadSample(leveldb::Slice) (db_impl.cc:1173) ==32364== by 0x6060F4: ParseKey (db_iter.cc:128) ==32364== by 0x6060F4: leveldb::(anonymous namespace)::DBIter::FindNextUserEntry(bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (db_iter.cc:183) ==32364== by 0x2D808A: CBlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>) (txdb.cpp:281) ==32364== by 0x324B9E: BlockManager::LoadBlockIndex(Consensus::Params const&, CBlockTreeDB&, std::set<CBlockIndex*, CBlockIndexWorkComparator, std::allocator<CBlockIndex*> >&) (validation.cpp:4049) ==32364== by 0x3251C0: LoadBlockIndexDB(CChainParams const&) (validation.cpp:4112) ==32364== by 0x325BB4: LoadBlockIndex(CChainParams const&) (validation.cpp:4573) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,408 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x3A12C6: thread<void (*)(char const*, std::function<void()>), char const*, std::_Bind<std::_Mem_fn<void (BaseIndex::*)()>(BaseIndex*)> > (thread:137) ==32364== by 0x3A12C6: BaseIndex::Start() (base.cpp:311) ==32364== by 0x161F69: AppInitMain(NodeContext&) (init.cpp:1712) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,409 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x4E54478: boost::thread::start_thread_noexcept() (in /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0) ==32364== by 0x172AD3: start_thread (thread.hpp:178) ==32364== by 0x172AD3: thread<std::_Bind<void (*(std::vector<boost::filesystem::path>))(std::vector<boost::filesystem::path>)>&> (thread.hpp:265) ==32364== by 0x172AD3: boost::thread* boost::thread_group::create_thread<std::_Bind<void (*(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >))(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >)> >(std::_Bind<void (*(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >))(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >)>) (thread_group.hpp:79) ==32364== by 0x162347: AppInitMain(NodeContext&) (init.cpp:1776) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,410 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x2C7EF5: thread<std::_Bind<void (*(char const*, void (*)()))(char const*, void (*)())> > (thread:137) ==32364== by 0x2C7EF5: StartTorControl() (torcontrol.cpp:755) ==32364== by 0x162B72: AppInitMain(NodeContext&) (init.cpp:1807) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,411 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x196E6F: std::thread::thread<void (*)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()> >(void (*&&)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()>&&) (thread:137) ==32364== by 0x18D2EB: CConnman::Start(CScheduler&, CConnman::Options const&) (net.cpp:2321) ==32364== by 0x163149: AppInitMain(NodeContext&) (init.cpp:1866) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,412 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x196E6F: std::thread::thread<void (*)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()> >(void (*&&)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()>&&) (thread:137) ==32364== by 0x18D868: CConnman::Start(CScheduler&, CConnman::Options const&) (net.cpp:2326) ==32364== by 0x163149: AppInitMain(NodeContext&) (init.cpp:1866) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,413 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x196E6F: std::thread::thread<void (*)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()> >(void (*&&)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()>&&) (thread:137) ==32364== by 0x18D3D1: CConnman::Start(CScheduler&, CConnman::Options const&) (net.cpp:2329) ==32364== by 0x163149: AppInitMain(NodeContext&) (init.cpp:1866) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,414 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x196E6F: std::thread::thread<void (*)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()> >(void (*&&)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()>&&) (thread:137) ==32364== by 0x18D532: CConnman::Start(CScheduler&, CConnman::Options const&) (net.cpp:2340) ==32364== by 0x163149: AppInitMain(NodeContext&) (init.cpp:1866) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 304 bytes in 1 blocks are possibly lost in loss record 1,415 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x196E6F: std::thread::thread<void (*)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()> >(void (*&&)(char const*, std::function<void ()>), char const (&) [7], std::function<void ()>&&) (thread:137) ==32364== by 0x18D5D5: CConnman::Start(CScheduler&, CConnman::Options const&) (net.cpp:2343) ==32364== by 0x163149: AppInitMain(NodeContext&) (init.cpp:1866) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 360 bytes in 1 blocks are possibly lost in loss record 1,430 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A46DD: __env_attach (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A1B06: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 360 bytes in 1 blocks are possibly lost in loss record 1,431 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A46DD: __env_attach (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A1B06: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626) ==32364== ==32364== 424 bytes in 1 blocks are possibly lost in loss record 1,458 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C6703: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== ==32364== 424 bytes in 1 blocks are possibly lost in loss record 1,459 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C6703: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== ==32364== 456 bytes in 1 blocks are possibly lost in loss record 1,467 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535AF0B: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 536 bytes in 1 blocks are possibly lost in loss record 1,497 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53B2406: __log_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 536 bytes in 1 blocks are possibly lost in loss record 1,498 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53B2406: __log_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626) ==32364== ==32364== 912 bytes in 3 blocks are possibly lost in loss record 1,529 of 1,691 ==32364== at 0x4030A0C: calloc (vg_replace_malloc.c:762) ==32364== by 0x4011E31: allocate_dtv (dl-tls.c:322) ==32364== by 0x40127BD: _dl_allocate_tls (dl-tls.c:539) ==32364== by 0x5073189: allocate_stack (allocatestack.c:584) ==32364== by 0x5073189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663) ==32364== by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22) ==32364== by 0x39777F: thread<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (thread:137) ==32364== by 0x39777F: construct<std::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (new_allocator.h:120) ==32364== by 0x39777F: construct<std::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (alloc_traits.h:475) ==32364== by 0x39777F: void std::vector<std::thread, std::allocator<std::thread> >::_M_emplace_back_aux<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&) (vector.tcc:416) ==32364== by 0x39280A: emplace_back<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&> (vector.tcc:101) ==32364== by 0x39280A: StartHTTPServer() (httpserver.cpp:434) ==32364== by 0x15F6ED: AppInitServers (init.cpp:777) ==32364== by 0x15F6ED: AppInitMain(NodeContext&) (init.cpp:1328) ==32364== by 0x144110: AppInit(int, char**) (bitcoind.cpp:146) ==32364== by 0x13C86E: main (bitcoind.cpp:176) ==32364== ==32364== 960 bytes in 1 blocks are possibly lost in loss record 1,530 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C6A41: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== ==32364== 960 bytes in 1 blocks are possibly lost in loss record 1,531 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C6A41: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== ==32364== 1,448 bytes in 1 blocks are possibly lost in loss record 1,558 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B1C4: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 4,184 bytes in 1 blocks are possibly lost in loss record 1,593 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52D3B38: __bam_read_root (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x538681D: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53678EA: __db_master_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53AE11E: __fop_subdb_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5386874: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B8ACE: Db::open(DbTxn*, char const*, char const*, DBTYPE, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 4,184 bytes in 1 blocks are possibly lost in loss record 1,594 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52E0B0B: __bam_get_root (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52E1159: __bam_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CBF1A: __bamc_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CE301: __bamc_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537363E: __dbc_iget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5365FD9: __db_master_update (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53AE189: __fop_subdb_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 4,184 bytes in 1 blocks are possibly lost in loss record 1,595 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5387030: __db_init_subdb (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53AE2C3: __fop_subdb_setup (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5386874: __db_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x5380F86: __db_open_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B8ACE: Db::open(DbTxn*, char const*, char const*, DBTYPE, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DCA76: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:550) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== ==32364== 4,184 bytes in 1 blocks are possibly lost in loss record 1,596 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52E0B0B: __bam_get_root (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52E1159: __bam_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CBF1A: __bamc_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CE301: __bamc_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537363E: __dbc_iget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537CE1F: __db_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x538068B: __db_get_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 4,184 bytes in 1 blocks are possibly lost in loss record 1,597 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52E1A3A: __bam_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CBF1A: __bamc_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CE301: __bamc_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537363E: __dbc_iget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537EFFC: __dbc_get_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B9FF0: Dbc::get(Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x4AA9A3: BerkeleyBatch::ReadAtCursor(Dbc*, CDataStream&, CDataStream&) (db.h:355) ==32364== ==32364== 4,184 bytes in 1 blocks are definitely lost in loss record 1,598 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52E1A3A: __bam_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CBF1A: __bamc_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CE301: __bamc_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537363E: __dbc_iget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537EFFC: __dbc_get_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B9FF0: Dbc::get(Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x4AA9A3: BerkeleyBatch::ReadAtCursor(Dbc*, CDataStream&, CDataStream&) (db.h:355) ==32364== ==32364== 7,344 bytes in 1 blocks are possibly lost in loss record 1,609 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C67D5: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== ==32364== 7,344 bytes in 1 blocks are possibly lost in loss record 1,610 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C67D5: __memp_init (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53C7080: __memp_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A20DC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== ==32364== 8,368 bytes in 2 blocks are possibly lost in loss record 1,616 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52E1A3A: __bam_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CBF1A: __bamc_search (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CE301: __bamc_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537363E: __dbc_iget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537CE1F: __db_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x538068B: __db_get_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537CBBB: __db_exists (in /usr/lib/libdb_cxx-4.8.so) ==32364== ==32364== 12,320 bytes in 77 blocks are possibly lost in loss record 1,621 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 16,504 bytes in 1 blocks are possibly lost in loss record 1,624 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B30D: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 65,544 bytes in 1 blocks are possibly lost in loss record 1,648 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53B253E: __log_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 65,544 bytes in 1 blocks are possibly lost in loss record 1,649 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53B253E: __log_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626) ==32364== ==32364== 139,216 bytes in 1,582 blocks are possibly lost in loss record 1,660 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B455: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 147,680 bytes in 923 blocks are definitely lost in loss record 1,661 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 200,832 bytes in 48 blocks are definitely lost in loss record 1,666 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CB4EA: __bamc_next (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CE81B: __bamc_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537363E: __dbc_iget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537EFFC: __dbc_get_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B9FF0: Dbc::get(Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x4AA9A3: BerkeleyBatch::ReadAtCursor(Dbc*, CDataStream&, CDataStream&) (db.h:355) ==32364== by 0x4A86B2: WalletBatch::LoadWallet(CWallet*) (walletdb.cpp:460) ==32364== ==32364== 315,264 bytes in 2,463 blocks are possibly lost in loss record 1,669 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B550: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 1,048,600 bytes in 1 blocks are possibly lost in loss record 1,674 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B117: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 1,079,472 bytes in 258 blocks are possibly lost in loss record 1,675 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CB4EA: __bamc_next (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52CE81B: __bamc_get (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537363E: __dbc_iget (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x537EFFC: __dbc_get_pp (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52B9FF0: Dbc::get(Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x4AA9A3: BerkeleyBatch::ReadAtCursor(Dbc*, CDataStream&, CDataStream&) (db.h:355) ==32364== by 0x4A86B2: WalletBatch::LoadWallet(CWallet*) (walletdb.cpp:460) ==32364== ==32364== 3,378,496 bytes in 38,392 blocks are definitely lost in loss record 1,678 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B455: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 4,800,384 (4,800,256 direct, 128 indirect) bytes in 37,502 blocks are definitely lost in loss record 1,679 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B550: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 5,767,264 bytes in 1 blocks are possibly lost in loss record 1,681 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535B166: __lock_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A21D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 6,209,212 bytes in 1 blocks are possibly lost in loss record 1,682 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535E91B: __mutex_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A1B90: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533) ==32364== by 0x46C0AE: WalletBatch (walletdb.h:207) ==32364== by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123) ==32364== by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806) ==32364== by 0x3EB062: LoadWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:73) ==32364== ==32364== 6,209,212 bytes in 1 blocks are possibly lost in loss record 1,683 of 1,691 ==32364== at 0x402E7BD: malloc (vg_replace_malloc.c:309) ==32364== by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x535E91B: __mutex_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A1B90: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x53A231A: __env_open (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x52BBFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so) ==32364== by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199) ==32364== by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405) ==32364== by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756) ==32364== by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152) ==32364== by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626) ==32364== ==32364== LEAK SUMMARY: ==32364== definitely lost: 8,532,008 bytes in 76,870 blocks ==32364== indirectly lost: 128 bytes in 1 blocks ==32364== possibly lost: 20,984,064 bytes in 4,434 blocks ==32364== still reachable: 433,580,688 bytes in 3,645,772 blocks ==32364== of which reachable via heuristic: ==32364== newarray : 72 bytes in 1 blocks ==32364== suppressed: 0 bytes in 0 blocks ==32364== Reachable blocks (those to which a pointer was found) are not shown. ==32364== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==32364== ==32364== For lists of detected and suppressed errors, rerun with: -s ==32364== ERROR SUMMARY: 64 errors from 64 contexts (suppressed: 0 from 0)

MarcoFalke commented at 4:03 PM on June 10, 2020: member

Assertion happens here:

    /** Create DB handle to real database */
    BerkeleyDatabase(std::shared_ptr<BerkeleyEnvironment> env, std::string filename) :
        nUpdateCounter(0), nLastSeen(0), nLastFlushed(0), nLastWalletUpdate(0), env(std::move(env)), strFile(std::move(filename))
    {
        auto inserted = this->env->m_databases.emplace(strFile, std::ref(*this));
        assert(inserted.second); // <-- here
    }

MarcoFalke added the label Wallet on Jun 10, 2020
fscemama commented at 4:21 PM on June 10, 2020: none

Please let me know if you need any further testing, including patching + recompilation.

fscemama commented at 9:35 AM on June 11, 2020: none

Hi, got this tonight in the live env. Unfortunately, it didn't crash and restart to 0.997 as expected, I'll have to try again. So slow :/

root# valgrind /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
==25768== Memcheck, a memory error detector
==25768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25768== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
==25768== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
==25768==
Bitcoin Core starting
==25768==
==25768== HEAP SUMMARY:
==25768==     in use at exit: 2,246,434 bytes in 2,285 blocks
==25768==   total heap usage: 3,243 allocs, 958 frees, 2,477,009 bytes allocated
==25768==
==25768== LEAK SUMMARY:
==25768==    definitely lost: 0 bytes in 0 blocks
==25768==    indirectly lost: 0 bytes in 0 blocks
==25768==      possibly lost: 0 bytes in 0 blocks
==25768==    still reachable: 2,246,434 bytes in 2,285 blocks
==25768==         suppressed: 0 bytes in 0 blocks
==25768== Rerun with --leak-check=full to see details of leaked memory
==25768==
==25768== For lists of detected and suppressed errors, rerun with: -s
==25768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
root@backprod2:/home/psb/bin# ==25771== Thread 5 b-httpworker.1:
==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
==25771==    at 0x541B983: ??? (syscall-template.S:84)
==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A004C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A0408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519F31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512B408: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509C026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x3DF7E5: BerkeleyEnvironment::Verify(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:277)
==25771==    by 0x3DFB4A: BerkeleyBatch::VerifyDatabaseFile(boost::filesystem::path const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)) (db.cpp:422)
==25771==    by 0x472D48: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3778)
==25771==  Address 0xb2dbb968 is 120 bytes inside a block of size 1,112 alloc'd
==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519EBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x50B7063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5169387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5169906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512E2B8: __db_vrfy_pgset (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512E4BE: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512B056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
==25771==
==25771== Thread 6 b-httpworker.2:
==25771== Conditional jump or move depends on uninitialised value(s)
==25771==    at 0x519B6C4: __log_putr.isra.1 (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519CD74: __log_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x514D384: __db_addrem_log (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x515DD0B: __db_pitem (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x50B86F2: __bam_iitem (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x50B3DBC: __bamc_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5156F5B: __dbc_iput (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x514B5E5: __db_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x516449C: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
==25771==
==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
==25771==    at 0x541B983: ??? (syscall-template.S:84)
==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519B3BD: __log_write (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519D031: __log_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51BD96B: __txn_regop_log (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51BC444: __txn_commit (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5164431: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
==25771==    by 0x47BD26: CWallet::AddToWalletIfInvolvingMe(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:953)
==25771==    by 0x47C1A2: CWallet::SyncTransaction(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:1087)
==25771==  Address 0x74af62c9 is 1,785 bytes inside a block of size 65,544 alloc'd
==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519553E: __log_open (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51851EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
==25771==    by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533)
==25771==    by 0x46C0AE: WalletBatch (walletdb.h:207)
==25771==    by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123)
==25771==    by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806)
==25771==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
==25771==
==25771==
==25771== HEAP SUMMARY:
==25771==     in use at exit: 4,185 bytes in 26 blocks
==25771==   total heap usage: 616,239,856 allocs, 616,239,830 frees, 62,380,430,976 bytes allocated
==25771==
==25771== LEAK SUMMARY:
==25771==    definitely lost: 2,400 bytes in 15 blocks
==25771==    indirectly lost: 0 bytes in 0 blocks
==25771==      possibly lost: 304 bytes in 1 blocks
==25771==    still reachable: 1,481 bytes in 10 blocks
==25771==         suppressed: 0 bytes in 0 blocks
==25771== Rerun with --leak-check=full to see details of leaked memory
==25771==
==25771== Use --track-origins=yes to see where uninitialised values come from
==25771== For lists of detected and suppressed errors, rerun with: -s
==25771== ERROR SUMMARY: 2976 errors from 3 contexts (suppressed: 0 from 0)

fscemama commented at 4:43 PM on June 11, 2020: none

Today, I've run valgrind on bitcoind in our live env for a few hours. Incredibly enough, only small crashes before valgrind was launched (small because reindexing from 0.99998, not 0.997).

Valgrind reported the following, but there were no crash at all during its monitoring the bitcoind process. I don't know why it was so unstable yesterday and days before, and not today. The only system difference is the installation of Debian libc6-dbg package, needed by Valgrind itself ; also, we scheduled creation of wallets to occur less often (once per hour).

More testing seems to be needed.

bin# valgrind /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
==25768== Memcheck, a memory error detector
==25768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25768== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
==25768== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
==25768==
Bitcoin Core starting
==25768==
==25768== HEAP SUMMARY:
==25768==     in use at exit: 2,246,434 bytes in 2,285 blocks
==25768==   total heap usage: 3,243 allocs, 958 frees, 2,477,009 bytes allocated
==25768==
==25768== LEAK SUMMARY:
==25768==    definitely lost: 0 bytes in 0 blocks
==25768==    indirectly lost: 0 bytes in 0 blocks
==25768==      possibly lost: 0 bytes in 0 blocks
==25768==    still reachable: 2,246,434 bytes in 2,285 blocks
==25768==         suppressed: 0 bytes in 0 blocks
==25768== Rerun with --leak-check=full to see details of leaked memory
==25768==
==25768== For lists of detected and suppressed errors, rerun with: -s
==25768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
root@backprod2:/home/psb/bin# ==25771== Thread 5 b-httpworker.1:
==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
==25771==    at 0x541B983: ??? (syscall-template.S:84)
==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A004C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A0408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519F31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512B408: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509C026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x3DF7E5: BerkeleyEnvironment::Verify(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:277)
==25771==    by 0x3DFB4A: BerkeleyBatch::VerifyDatabaseFile(boost::filesystem::path const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)) (db.cpp:422)
==25771==    by 0x472D48: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3778)
==25771==  Address 0xb2dbb968 is 120 bytes inside a block of size 1,112 alloc'd
==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519EBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x50B7063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5169387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5169906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512E2B8: __db_vrfy_pgset (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512E4BE: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512B056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
==25771==
==25771== Thread 6 b-httpworker.2:
==25771== Conditional jump or move depends on uninitialised value(s)
==25771==    at 0x519B6C4: __log_putr.isra.1 (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519CD74: __log_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x514D384: __db_addrem_log (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x515DD0B: __db_pitem (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x50B86F2: __bam_iitem (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x50B3DBC: __bamc_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5156F5B: __dbc_iput (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x514B5E5: __db_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x516449C: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
==25771==
==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
==25771==    at 0x541B983: ??? (syscall-template.S:84)
==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519B3BD: __log_write (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519D031: __log_put (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51BD96B: __txn_regop_log (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51BC444: __txn_commit (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x5164431: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
==25771==    by 0x47BD26: CWallet::AddToWalletIfInvolvingMe(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:953)
==25771==    by 0x47C1A2: CWallet::SyncTransaction(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:1087)
==25771==  Address 0x74af62c9 is 1,785 bytes inside a block of size 65,544 alloc'd
==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x519553E: __log_open (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x51851EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
==25771==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
==25771==    by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533)
==25771==    by 0x46C0AE: WalletBatch (walletdb.h:207)
==25771==    by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123)
==25771==    by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806)
==25771==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
==25771==
==25771==
==25771== HEAP SUMMARY:
==25771==     in use at exit: 4,185 bytes in 26 blocks
==25771==   total heap usage: 616,239,856 allocs, 616,239,830 frees, 62,380,430,976 bytes allocated
==25771==
==25771== LEAK SUMMARY:
==25771==    definitely lost: 2,400 bytes in 15 blocks
==25771==    indirectly lost: 0 bytes in 0 blocks
==25771==      possibly lost: 304 bytes in 1 blocks
==25771==    still reachable: 1,481 bytes in 10 blocks
==25771==         suppressed: 0 bytes in 0 blocks
==25771== Rerun with --leak-check=full to see details of leaked memory
==25771==
==25771== Use --track-origins=yes to see where uninitialised values come from
==25771== For lists of detected and suppressed errors, rerun with: -s
==25771== ERROR SUMMARY: 2976 errors from 3 contexts (suppressed: 0 from 0)
root@backprod2:/home/psb/bin# valgrind --leak-check=full /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
root@backprod2:/home/psb/bin# valgrind --leak-check=full /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
==24289== Memcheck, a memory error detector
==24289== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24289== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
==24289== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
==24289==
Bitcoin Core starting
==24289==
==24289== HEAP SUMMARY:
==24289==     in use at exit: 2,246,434 bytes in 2,285 blocks
==24289==   total heap usage: 3,243 allocs, 958 frees, 2,477,009 bytes allocated
==24289==
==24289== LEAK SUMMARY:
==24289==    definitely lost: 0 bytes in 0 blocks
==24289==    indirectly lost: 0 bytes in 0 blocks
==24289==      possibly lost: 0 bytes in 0 blocks
==24289==    still reachable: 2,246,434 bytes in 2,285 blocks
==24289==         suppressed: 0 bytes in 0 blocks
==24289== Reachable blocks (those to which a pointer was found) are not shown.
==24289== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24289==
==24289== For lists of detected and suppressed errors, rerun with: -s
==24289== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
root@backprod2:/home/psb/bin# ==24359== Thread 7 b-httpworker.3:
==24359== Syscall param pwrite64(buf) points to uninitialised byte(s)
==24359==    at 0x541B983: ??? (syscall-template.S:84)
==24359==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x51A004C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x51A0408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x519F31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x5166A48: __db_new (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x50C7CAE: __bam_split (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x50B3EA1: __bamc_put (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x5156F5B: __dbc_iput (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x514B5E5: __db_put (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x512E0C1: __db_vrfy_putpageinfo (in /usr/lib/libdb_cxx-4.8.so)
==24359==  Address 0x2370b958 is 120 bytes inside a block of size 1,112 alloc'd
==24359==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
==24359==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x519EBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x50B7063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x5169387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x5169906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x512E415: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x512B056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x509C026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==24359==
==24359==
==24359== HEAP SUMMARY:
==24359==     in use at exit: 8,985 bytes in 56 blocks
==24359==   total heap usage: 314,206,840 allocs, 314,206,784 frees, 39,827,904,698 bytes allocated
==24359==
==24359== Thread 1 b-shutoff:
==24359== 160 bytes in 1 blocks are definitely lost in loss record 9 of 13
==24359==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
==24359==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x513E5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x51851D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
==24359==    by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405)
==24359==    by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756)
==24359==    by 0x3EBDEF: VerifyWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:58)
==24359==    by 0x15E76D: AppInitMain(NodeContext&) (init.cpp:1334)
==24359==
==24359== 304 bytes in 1 blocks are possibly lost in loss record 10 of 13
==24359==    at 0x4030A0C: calloc (vg_replace_malloc.c:760)
==24359==    by 0x4011E31: allocate_dtv (dl-tls.c:322)
==24359==    by 0x40127BD: _dl_allocate_tls (dl-tls.c:539)
==24359==    by 0x5413189: allocate_stack (allocatestack.c:584)
==24359==    by 0x5413189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663)
==24359==    by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22)
==24359==    by 0x624F7D: thread<void (&)(leveldb::(anonymous namespace)::PosixEnv*), leveldb::(anonymous namespace)::PosixEnv*> (thread:137)
==24359==    by 0x624F7D: leveldb::(anonymous namespace)::PosixEnv::Schedule(void (*)(void*), void*) (env_posix.cc:802)
==24359==    by 0x5F9586: leveldb::DBImpl::Get(leveldb::ReadOptions const&, leveldb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (db_impl.cc:1150)
==24359==    by 0x2D881E: Read<std::pair<char, int>, CBlockFileInfo> (dbwrapper.h:238)
==24359==    by 0x2D881E: CBlockTreeDB::ReadBlockFileInfo(int, CBlockFileInfo&) (txdb.cpp:153)
==24359==    by 0x3252A9: LoadBlockIndexDB(CChainParams const&) (validation.cpp:4121)
==24359==    by 0x325BB4: LoadBlockIndex(CChainParams const&) (validation.cpp:4573)
==24359==    by 0x160AC9: AppInitMain(NodeContext&) (init.cpp:1553)
==24359==    by 0x144110: AppInit(int, char**) (bitcoind.cpp:146)
==24359==
==24359== 7,040 bytes in 44 blocks are definitely lost in loss record 13 of 13
==24359==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
==24359==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x513E5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x51851D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
==24359==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
==24359==    by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405)
==24359==    by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756)
==24359==    by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152)
==24359==    by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626)
==24359==
==24359== LEAK SUMMARY:
==24359==    definitely lost: 7,200 bytes in 45 blocks
==24359==    indirectly lost: 0 bytes in 0 blocks
==24359==      possibly lost: 304 bytes in 1 blocks
==24359==    still reachable: 1,481 bytes in 10 blocks
==24359==         suppressed: 0 bytes in 0 blocks
==24359== Reachable blocks (those to which a pointer was found) are not shown.
==24359== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24359==
==24359== Use --track-origins=yes to see where uninitialised values come from
==24359== For lists of detected and suppressed errors, rerun with: -s
==24359== ERROR SUMMARY: 3167 errors from 4 contexts (suppressed: 0 from 0)

MarcoFalke commented at 4:54 PM on June 11, 2020: member

You might want to start valgrind with the suppressions file --suppressions= and point it to the file contrib/valgrind.supp.

But it seems we already found the location of the crash: #19232 (comment)
ryanofsky commented at 12:06 AM on June 17, 2020: member
re: #19232 (comment)

Assert is strange, but seems like maybe it could happen if a wallet loaded more than once at the same time by different threads. What is strange is that the assert that's failing runs twice in a row but somehow succeeds the first time and fails the second time:
```
loadwallet
  LoadWallet
    CWallet::Verify
      WalletDatabase::Create
        BerkeleyDatabase::BerkeleyDatabase
           m_databases.emplace                <--- apparently succeeds
        BerkeleyDatabase::~BerkeleyDatabase
           m_databases.erase
    CWallet::CreateWalletFromFile
      WalletDatabase::Create
         BerkeleyDatabase::BerkeleyDatabase
           m_databases.emplace                <--- fails & aborts according to trace
```
I don't see how the crash would happen if the application is "[waiting] for any bitcoin-cli command to terminate before issuing another one" as described #19232#issue-636073520.

But if it's possible the application is loading the same wallet twice from different threads, then the crash makes more sense. I think we would probably have to fix it by introducing something like g_loading_wallet_set similar to existing g_unloading_wallet_set to track and avoid concurrently loading wallets.
fscemama commented at 12:09 AM on June 17, 2020: none

Interesting enough. I should probably make sure the wait-for-any-bitcoin-cli-command-to-terminate is working well. Your idea that it should be prevented from the daemon itself does make sense. Thanks.
promag commented at 1:00 AM on June 17, 2020: member

But if it's possible the application is loading the same wallet twice from different threads, then the crash makes more sense.

Right, there's nothing handling this. Should it prevent concurrent loads by env? Or simply any concurrent loads?

BTW, concurrent unload can throw "Requested wallet does not exist or is not loaded" or "Requested wallet already unloaded". I wonder if with 2 concurrent loads, both should succeed - one actually loads, the other waits for the first - so both clients work as expected.
ryanofsky commented at 2:48 AM on June 17, 2020: member

Right, there's nothing handling this. Should it prevent concurrent loads by env? Or simply any concurrent loads?

The idea with the g_loading_wallet_set suggestion is that it would just track wallet loads using given wallet names, the same way g_unloading_wallet_set tracks wallet unloads. Environments don't actually have much to do with this bug and are already protected with .walletlock files. Also environments aren't visible at the wallet.cpp layer, only the db.cpp layer.

BTW, concurrent unload can throw "Requested wallet does not exist or is not loaded" or "Requested wallet already unloaded".

That's interesting. I think more ideally they would both return a simple "Wallet is not currently loaded" error so applications trying to handle this don't have to deal with two different error messages, one more frequent and the other more rare.

I wonder if with 2 concurrent loads, both should succeed - one actually loads, the other waits for the first - so both clients work as expected.

I think that would just add complexity without use-cases to justify it. It could also be confusing and break backwards compatibility with existing applications expecting to see the "Duplicate -wallet filename specified" error when a wallet's already loaded.
fscemama commented at 9:18 AM on June 17, 2020: none

Dear all, I can now confirm that after using procmail's lockfile to make absolutely sure no concurrent calls to bitcoin-cli are made, the stability has greatly improved. It can be dealt with that way, at least. Your help is precious.
promag commented at 10:16 AM on June 17, 2020: member

@fscemama it would be great if you can test #19300.

Note that loadwallet can fail (RPC error) if the wallet is already being loaded so you have to deal with this error.

But honestly you should rethink your approach. The server doesn't count how many clients "load/want" a given wallet so when client C1 unloads the wallet W1 it can mess some other client C2 that is also using W1. I think you should track/handle this outside unless there's agreement in supporting this use case here (which I don't support :trollface:).

fscemama commented at 11:18 AM on June 17, 2020: none

Lol, we do track and handle, this at least works well :) BTW, I'm glad to see you again, Promag. Right advice. About #19300, did the following with no error.

# Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd
wallet=xxxxxx
for i in {1..1000}
do
  echo $i
  bitcoin-cli loadwallet $wallet
  bitcoin-cli unloadwallet $wallet
done

promag commented at 11:25 AM on June 17, 2020: member
did the following with no error.

You must run that script multiple times in parallel, like
```
test.sh &
test.sh
```
fscemama commented at 11:28 AM on June 17, 2020: none

Sorry. Did it, immediate crash.
promag commented at 11:30 AM on June 17, 2020: member

Sorry. Did it, immediate crash.

Was this with #19300 branch? What was the error?

fscemama commented at 11:31 AM on June 17, 2020: none

...
error code: -18
error message:
Requested wallet does not exist or is not loaded
5 26133
error code: -4
error message:
Wallet file verification failed: Error loading wallet /root/wallets/0033665502992-2018-08-09-15-25-18-653299-00. Duplicate -wallet filename specified.
error code: -18
error message:
Requested wallet does not exist or is not loaded
6 26133
error code: -4
error message:
Wallet file verification failed: Error loading wallet /root/wallets/0033665502992-2018-08-09-15-25-18-653299-00. Duplicate -wallet filename specified.
error code: -18
error message:
Requested wallet does not exist or is not loaded
7 26133
5 26232
error: Could not connect to the server 127.0.0.1:18332 (error code 1 - "EOF reached")

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
error: Could not connect to the server 127.0.0.1:18332 (error code 1 - "EOF reached")

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
error: Could not connect to the server 127.0.0.1:18332
...

fscemama commented at 11:32 AM on June 17, 2020: none

root# cat a.sh
#!/bin/sh
wallet=xxxxx
for i in 0 1 2 3 4 5 6 7 8 9
do
  echo $i $$
  bitcoin-cli loadwallet $wallet
  bitcoin-cli unloadwallet $wallet
done

Crash after launching a few times in background.

fscemama commented at 11:34 AM on June 17, 2020: none

This probably deserves a valgrind. Please give 10 minutes.
promag commented at 11:54 AM on June 17, 2020: member

Can you repeat this with #19300 branch?

fscemama commented at 12:06 PM on June 17, 2020: none

Sorry, removed much pollution (not needed processes). Useful info:

=30964== Memcheck, a memory error detector
==30964== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30964== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==30964== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128
==30964==
Bitcoin Core starting
==30964==
==30964== HEAP SUMMARY:
==30964==     in use at exit: 2,232,114 bytes in 2,271 blocks
==30964==   total heap usage: 3,234 allocs, 963 frees, 2,462,760 bytes allocated
==30964==
==30964== LEAK SUMMARY:
==30964==    definitely lost: 0 bytes in 0 blocks
==30964==    indirectly lost: 0 bytes in 0 blocks
==30964==      possibly lost: 0 bytes in 0 blocks
==30964==    still reachable: 2,232,114 bytes in 2,271 blocks
==30964==         suppressed: 0 bytes in 0 blocks
==30964== Reachable blocks (those to which a pointer was found) are not shown.
==30964== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30964==
==30964== For lists of detected and suppressed errors, rerun with: -s
==30964== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
root@backdev2:/home/psb/bin# ==30973== Syscall param pwrite64(buf) points to uninitialised byte(s)
==30973==    at 0x507B983: ??? (syscall-template.S:84)
==30973==    by 0x53CF188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x53BD04C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x53BD408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x53BC31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x5348408: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x534A846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x52B9026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x3DF7E5: BerkeleyEnvironment::Verify(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:277)
==30973==    by 0x3DFB4A: BerkeleyBatch::VerifyDatabaseFile(boost::filesystem::path const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)) (db.cpp:422)
==30973==    by 0x472D48: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3778)
==30973==  Address 0xd63ee08 is 120 bytes inside a block of size 1,112 alloc'd
==30973==    at 0x402E7BD: malloc (vg_replace_malloc.c:309)
==30973==    by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x52D4063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x5386387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x5386906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x534B2B8: __db_vrfy_pgset (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x534B4BE: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x5348056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
==30973==    by 0x534A846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
==30973==
==30973==
==30973== Process terminating with default action of signal 6 (SIGABRT)
==30973==    at 0x6350FFF: raise (raise.c:51)
==30973==    by 0x6352429: abort (abort.c:89)
==30973==    by 0x6349E66: __assert_fail_base (assert.c:92)
==30973==    by 0x6349F11: __assert_fail (assert.c:101)
==30973==    by 0x49085A: BerkeleyDatabase (db.h:125)
==30973==    by 0x49085A: std::unique_ptr<BerkeleyDatabase, std::default_delete<BerkeleyDatabase> > MakeUnique<BerkeleyDatabase, std::shared_ptr<BerkeleyEnvironment>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::shared_ptr<BerkeleyEnvironment>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (memory.h:16)
==30973==    by 0x49091F: BerkeleyDatabase::Create(boost::filesystem::path const&) (db.h:139)
==30973==    by 0x47E2C1: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3805)
==30973==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
==30973==    by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626)
==30973==    by 0x22BFEB: operator() (server.h:104)
==30973==    by 0x22BFEB: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
==30973==    by 0x14A95E: operator() (functional:2127)
==30973==    by 0x14A95E: operator() (chain.cpp:208)
==30973==    by 0x14A95E: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
==30973==    by 0x2B32DB: operator() (functional:2127)
==30973==    by 0x2B32DB: ExecuteCommand (server.cpp:453)
==30973==    by 0x2B32DB: CRPCTable::execute(JSONRPCRequest const&) const (server.cpp:436)
==30973==
==30973== HEAP SUMMARY:
==30973==     in use at exit: 475,082,336 bytes in 3,760,553 blocks
==30973==   total heap usage: 9,888,764 allocs, 6,128,211 frees, 1,385,505,603 bytes allocated
==30973==

MarcoFalke closed this on Jun 29, 2020
MarcoFalke locked this on Feb 15, 2022

Contributors

Labels

Linked (view graph)

#19300 wallet: Handle concurrent wallet loading