Regular crash+autorestart after loading/unloading many wallets

fscemama commented at 9:05 am on June 10, 2020: none

Hi everyone, We use bitcoin core with hundreds of wallets. The only efficient way we have found to do so is :

wait for any bitcoin-cli command to terminate before issuing another one
never issue any bitcoin-cli command unless tip update is 1
always unload a wallet after loading and using it

Now, we observ that bitcoin core systematically crashes every 20-30 minutes, then automatically restarts. And the problem is, it has to re-sync from about 0.997, which easily takes 15-20 minutes. During that time, the daemon cannot be used.

If we don’t load/unload wallets, the daemon is stable and keeps to 1, no issue.

System information Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd Compiled and run on Debian 9, on a VM with 16 mo memory. Launched that way : /bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1

bitcoin.conf : server=1 dbcache=16384 par=1 blocksonly=1 maxuploadtarget=137 maxconnections=16 rpcuser=bitcoinrpc rpcpassword=xxxxxx rpcallowip=127.0.0.1 testnet=0 txindex=1 walletbroadcast=1 txconfirmtarget=1

Please note that the same behaviour was happening with older releases (from 0.17.x).

Thanks for any help. Please ask for any needed information / testing. Regards,

fscemama added the label Bug on Jun 10, 2020

MarcoFalke commented at 10:15 am on June 10, 2020: member

Are there any debug log excerpts at the time of the crash? Or stderr/stdout?

fscemama commented at 10:18 am on June 10, 2020: none

Are there any debug log excerpts at the time of the crash? Or stderr/stdout?

Hi Marco, typical sequence of debug.log below. Thanks !

  02020-06-10T09:39:55Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
  12020-06-10T09:39:55Z Using wallet /root/wallets/xxxxx-2019-10-15-18-48-48-02
  22020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
  32020-06-10T09:39:55Z init message: Loading wallet...
  42020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
  52020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet File Version = 169900
  62020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Keys: 2027 plaintext, 0 encrypted, 2027 w/ metadata, 2027 total. Unknown wallet records: 0
  72020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet completed loading in              61ms
  82020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] setKeyPool.size() = 2000
  92020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] mapWallet.size() = 11
 102020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] m_address_book.size() = 26
 112020-06-10T09:39:55Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Releasing wallet
 122020-06-10T09:39:55Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
 132020-06-10T09:39:55Z Using wallet /root/wallets/xxxxx-2019-10-15-18-48-48-02
 142020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
 152020-06-10T09:39:55Z init message: Loading wallet...
 162020-06-10T09:39:55Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2019-10-15-18-48-48-02/database ErrorFile=/root/wallets/xxxxx-2019-10-15-18-48-48-02/db.log
 172020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet File Version = 169900
 182020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Keys: 2027 plaintext, 0 encrypted, 2027 w/ metadata, 2027 total. Unknown wallet records: 0
 192020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Wallet completed loading in              43ms
 202020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] setKeyPool.size() = 2000
 212020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] mapWallet.size() = 11
 222020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] m_address_book.size() = 26
 232020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-10-15-18-48-48-02] Releasing wallet
 242020-06-10T09:39:56Z [/root/wallets/xxxxx-2019-11-27-12-00-18-01] Releasing wallet
 252020-06-10T09:39:57Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
 262020-06-10T09:39:57Z Using wallet /root/wallets/xxxxx-2020-04-10-23-21-36-02
 272020-06-10T09:39:57Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2020-04-10-23-21-36-02/database ErrorFile=/root/wallets/xxxxx-2020-04-10-23-21-36-02/db.log
 282020-06-10T09:39:57Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
 292020-06-10T09:39:57Z Using wallet /root/wallets/xxxxx-2020-04-10-23-21-36-02
 302020-06-10T09:39:57Z BerkeleyEnvironment::Open: LogDir=/root/wallets/xxxxx-2020-04-10-23-21-36-02/database ErrorFile=/root/wallets/xxxxx-2020-04-10-23-21-36-02/db.log
 312020-06-10T09:39:57Z init message: Loading wallet...
 32
 33
 34
 35
 36
 372020-06-10T09:40:02Z Bitcoin Core version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd (release build)
 382020-06-10T09:40:02Z InitParameterInteraction: parameter interaction: -blocksonly=1 -> setting -whitelistrelay=0
 392020-06-10T09:40:02Z Assuming ancestors of block 0000000000000000000f2adce67e49b0b6bdeb9de8b7c3d7e93b21e7fc1e819d have valid signatures.
 402020-06-10T09:40:02Z Setting nMinimumChainWork=00000000000000000000000000000000000000000e1ab5ec9348e9f4b8eb8154
 412020-06-10T09:40:02Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
 422020-06-10T09:40:02Z Using RdSeed as additional entropy source
 432020-06-10T09:40:02Z Using RdRand as an additional entropy source
 442020-06-10T09:40:02Z Default data directory /root/.bitcoin
 452020-06-10T09:40:02Z Using data directory /root/.bitcoin
 462020-06-10T09:40:02Z Config file: /root/.bitcoin/bitcoin.conf
 472020-06-10T09:40:02Z Config file arg: blocksonly="1"
 482020-06-10T09:40:02Z Config file arg: dbcache="16384"
 492020-06-10T09:40:02Z Config file arg: maxconnections="16"
 502020-06-10T09:40:02Z Config file arg: maxuploadtarget="137"
 512020-06-10T09:40:02Z Config file arg: par="1"
 522020-06-10T09:40:02Z Config file arg: rpcallowip="127.0.0.1"
 532020-06-10T09:40:02Z Config file arg: rpcpassword=****
 542020-06-10T09:40:02Z Config file arg: rpcuser=****
 552020-06-10T09:40:02Z Config file arg: server="1"
 562020-06-10T09:40:02Z Config file arg: testnet="0"
 572020-06-10T09:40:02Z Config file arg: txconfirmtarget="1"
 582020-06-10T09:40:02Z Config file arg: txindex="1"
 592020-06-10T09:40:02Z Config file arg: walletbroadcast="1"
 602020-06-10T09:40:02Z Command-line arg: conf="/root/.bitcoin/bitcoin.conf"
 612020-06-10T09:40:02Z Command-line arg: daemon=""
 622020-06-10T09:40:02Z Command-line arg: fallbackfee="0.00001"
 632020-06-10T09:40:02Z Command-line arg: rpcworkqueue="128"
 642020-06-10T09:40:02Z Command-line arg: txindex="1"
 652020-06-10T09:40:02Z Command-line arg: walletrbf=""
 662020-06-10T09:40:02Z Using at most 16 automatic connections (1024 file descriptors available)
 672020-06-10T09:40:02Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
 682020-06-10T09:40:02Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
 692020-06-10T09:40:02Z Script verification uses 0 additional threads
 702020-06-10T09:40:02Z scheduler thread start
 712020-06-10T09:40:02Z WARNING: option -rpcallowip was specified without -rpcbind; this doesn't usually make sense
 722020-06-10T09:40:02Z HTTP: creating work queue of depth 128
 732020-06-10T09:40:02Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
 742020-06-10T09:40:02Z HTTP: starting 4 worker threads
 752020-06-10T09:40:02Z Using wallet directory /root/.bitcoin/wallets
 762020-06-10T09:40:02Z init message: Verifying wallet(s)...
 772020-06-10T09:40:02Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
 782020-06-10T09:40:02Z Using wallet /root/.bitcoin/wallets
 792020-06-10T09:40:02Z BerkeleyEnvironment::Open: LogDir=/root/.bitcoin/wallets/database ErrorFile=/root/.bitcoin/wallets/db.log
 802020-06-10T09:40:03Z init message: Loading banlist...
 812020-06-10T09:40:03Z Using /16 prefix for IP bucketing
 822020-06-10T09:40:03Z Cache configuration:
 832020-06-10T09:40:03Z * Using 2.0 MiB for block index database
 842020-06-10T09:40:03Z * Using 1024.0 MiB for transaction index database
 852020-06-10T09:40:03Z * Using 8.0 MiB for chain state database
 862020-06-10T09:40:03Z * Using 15350.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
 872020-06-10T09:40:03Z init message: Loading block index...
 882020-06-10T09:40:03Z Opening LevelDB in /root/.bitcoin/blocks/index
 892020-06-10T09:40:03Z Opened LevelDB successfully
 902020-06-10T09:40:03Z Using obfuscation key for /root/.bitcoin/blocks/index: 0000000000000000
 912020-06-10T09:40:10Z LoadBlockIndexDB: last block file = 2106
 922020-06-10T09:40:10Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=134, size=132209876, heights=633875...634008, time=2020-06-09...2020-06-10)
 932020-06-10T09:40:10Z Checking all blk files are present...
 942020-06-10T09:40:11Z Opening LevelDB in /root/.bitcoin/chainstate
 952020-06-10T09:40:11Z Opened LevelDB successfully
 962020-06-10T09:40:11Z Using obfuscation key for /root/.bitcoin/chainstate: 4501f5bf5624cc54
 972020-06-10T09:40:11Z Loaded best chain: hashBestChain=0000000000000000000fc56a5b8b7c1c3ce9ba39acc75cc13db25fb5bc8c62e0 height=633123 date=2020-06-05T03:46:00Z progress=0.997213
 982020-06-10T09:40:11Z init message: Rewinding blocks...
 992020-06-10T09:40:11Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
1002020-06-10T09:40:11Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
1012020-06-10T09:40:11Z init message: Verifying blocks...
1022020-06-10T09:40:11Z Verifying last 6 blocks at level 3
1032020-06-10T09:40:11Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
1042020-06-10T09:43:08Z No coin database inconsistencies in last 6 blocks (13097 transactions)
1052020-06-10T09:43:08Z  block index          185680ms
1062020-06-10T09:43:08Z Opening LevelDB in /root/.bitcoin/indexes/txindex
1072020-06-10T09:43:17Z Opened LevelDB successfully
1082020-06-10T09:43:17Z Using obfuscation key for /root/.bitcoin/indexes/txindex: 0000000000000000
1092020-06-10T09:43:17Z txindex thread start
1102020-06-10T09:43:17Z txindex is enabled at height 633123
1112020-06-10T09:43:17Z txindex thread exit
1122020-06-10T09:43:17Z init message: Loading wallet...
1132020-06-10T09:43:17Z BerkeleyEnvironment::Open: LogDir=/root/.bitcoin/wallets/database ErrorFile=/root/.bitcoin/wallets/db.log
1142020-06-10T09:43:17Z [default wallet] Wallet File Version = 169900
1152020-06-10T09:43:17Z [default wallet] Keys: 2001 plaintext, 0 encrypted, 2001 w/ metadata, 2001 total. Unknown wallet records: 0
1162020-06-10T09:43:18Z [default wallet] Wallet completed loading in             556ms
1172020-06-10T09:43:18Z [default wallet] setKeyPool.size() = 2000
1182020-06-10T09:43:18Z [default wallet] mapWallet.size() = 0
1192020-06-10T09:43:18Z [default wallet] m_address_book.size() = 0
1202020-06-10T09:43:18Z block tree size = 634017
1212020-06-10T09:43:18Z nBestHeight = 633123
1222020-06-10T09:43:18Z torcontrol thread start
1232020-06-10T09:43:18Z Bound to [::]:8333
1242020-06-10T09:43:18Z Bound to 0.0.0.0:8333
1252020-06-10T09:43:18Z init message: Loading P2P addresses...
1262020-06-10T09:43:19Z Loaded 58642 addresses from peers.dat  880ms
1272020-06-10T09:43:19Z init message: Starting network threads...
1282020-06-10T09:43:19Z net thread start
1292020-06-10T09:43:19Z dnsseed thread start
1302020-06-10T09:43:19Z addcon thread start
1312020-06-10T09:43:19Z opencon thread start
1322020-06-10T09:43:19Z init message: Done loading
1332020-06-10T09:43:19Z msghand thread start
1342020-06-10T09:43:19Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
1352020-06-10T09:43:19Z Using wallet /root/wallets/keplerk
1362020-06-10T09:43:19Z BerkeleyEnvironment::Open: LogDir=/root/wallets/keplerk/database ErrorFile=/root/wallets/keplerk/db.log
1372020-06-10T09:43:27Z init message: Loading wallet...
1382020-06-10T09:43:30Z Loading addresses from DNS seed seed.bitcoin.jonasschnelli.ch
1392020-06-10T09:43:38Z UpdateTip: new best=0000000000000000000891768c73b989c11ab68ef59a7272d4953e1e40110e50 height=633124 version=0x3fff0000 log2_work=92.004565 tx=536058118 date='2020-06-05T03:49:48Z' progress=0.997213 cache=0.9MiB(7204txo)
1402020-06-10T09:43:40Z Loading addresses from DNS seed dnsseed.emzy.de
1412020-06-10T09:43:50Z Loading addresses from DNS seed dnsseed.bluematt.me
1422020-06-10T09:43:55Z UpdateTip: new best=000000000000000000109465e408d8cac29bf79001ab12b352344186f3f59171 height=633125 version=0x20000000 log2_work=92.004582 tx=536059715 date='2020-06-05T03:59:30Z' progress=0.997216 cache=1.9MiB(14138txo)
1432020-06-10T09:44:11Z Loading addresses from DNS seed seed.btc.petertodd.org
1442020-06-10T09:44:16Z Loading addresses from DNS seed seed.bitcoin.sprovoost.nl
1452020-06-10T09:44:21Z UpdateTip: new best=000000000000000000141612a0f0dd110e5fd29db90bcd9a2f103684ca6e669d height=633126 version=0x37ffe000 log2_work=92.0046 tx=536061851 date='2020-06-05T04:12:56Z' progress=0.997221 cache=3.0MiB(22542txo)
1462020-06-10T09:44:23Z UpdateTip: new best=00000000000000000007afd66af6119e5568df8b72d31d3b8699a7011225c69c height=633127 version=0x20400000 log2_work=92.004617 tx=536062401 date='2020-06-05T04:15:16Z' progress=0.997222 cache=3.1MiB(23691txo)
1472020-06-10T09:44:26Z Loading addresses from DNS seed dnsseed.bitcoin.dashjr.org
1482020-06-10T09:44:41Z UpdateTip: new best=000000000000000000091f1a4746918881ff3afc6475c9fb92a9b63c4be9347e height=633128 version=0x20c00000 log2_work=92.004634 tx=536064469 date='2020-06-05T04:28:18Z' progress=0.997227 cache=4.3MiB(30860txo)
1492020-06-10T09:44:47Z Loading addresses from DNS seed seed.bitcoin.sipa.be
1502020-06-10T09:44:56Z UpdateTip: new best=00000000000000000013fa5d77449e82bf43ff6db20794752ae5fe07e11a6537 height=633129 version=0x20000000 log2_work=92.004651 tx=536066188 date='2020-06-05T04:39:39Z' progress=0.997231 cache=5.0MiB(36837txo)

fscemama commented at 2:16 pm on June 10, 2020: none

Currently, it’s been up and running for one hour without crashing. I can’t tell why nor when it does accurately. There’s something behind.

MarcoFalke commented at 2:20 pm on June 10, 2020: member

Given that this is somewhat reproducible, could you try running the binary in gdb or valgrind?

fscemama commented at 2:40 pm on June 10, 2020: none

I’ll do it with valgrind tonight. Must unfortunately be run in live, so I need a quiet period…

fscemama commented at 3:13 pm on June 10, 2020: none

In our dev environment, I got this crash. It seems that the creation of wallets (we do use bitcoin core for this too) might be the cause, and it may be worth testing in the live environment if crashes still happen if no wallet is never created. I’ll give this a try.

 0root# valgrind --leak-check=yes  /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128
 1==32342== Memcheck, a memory error detector
 2==32342== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
 3==32342== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
 4==32342== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128
 5==32342==
 6Bitcoin Core starting
 7==32342==
 8==32342== HEAP SUMMARY:
 9==32342==     in use at exit: 2,232,114 bytes in 2,271 blocks
10==32342==   total heap usage: 3,234 allocs, 963 frees, 2,462,760 bytes allocated
11==32342==
12==32342== LEAK SUMMARY:
13==32342==    definitely lost: 0 bytes in 0 blocks
14==32342==    indirectly lost: 0 bytes in 0 blocks
15==32342==      possibly lost: 0 bytes in 0 blocks
16==32342==    still reachable: 2,232,114 bytes in 2,271 blocks
17==32342==         suppressed: 0 bytes in 0 blocks
18==32342== Reachable blocks (those to which a pointer was found) are not shown.
19==32342== To see them, rerun with: --leak-check=full --show-leak-kinds=all
20==32342==
21==32342== For lists of detected and suppressed errors, rerun with: -s
22==32342== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
23==32364==
24==32364== Process terminating with default action of signal 6 (SIGABRT)
25==32364==    at 0x6350FFF: raise (raise.c:51)
26==32364==    by 0x6352429: abort (abort.c:89)
27==32364==    by 0x6349E66: __assert_fail_base (assert.c:92)
28==32364==    by 0x6349F11: __assert_fail (assert.c:101)
29==32364==    by 0x49085A: BerkeleyDatabase (db.h:125)
30==32364==    by 0x49085A: std::unique_ptr<BerkeleyDatabase, std::default_delete<BerkeleyDatabase> > MakeUnique<BerkeleyDatabase, std::shared_ptr<BerkeleyEnvironment>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::shared_ptr<BerkeleyEnvironment>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (memory.h:16)
31==32364==    by 0x49091F: BerkeleyDatabase::Create(boost::filesystem::path const&) (db.h:139)
32==32364==    by 0x47E2C1: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3805)
33==32364==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
34==32364==    by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626)
35==32364==    by 0x22BFEB: operator() (server.h:104)
36==32364==    by 0x22BFEB: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
37==32364==    by 0x14A95E: operator() (functional:2127)
38==32364==    by 0x14A95E: operator() (chain.cpp:208)
39==32364==    by 0x14A95E: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
40==32364==    by 0x2B32DB: operator() (functional:2127)
41==32364==    by 0x2B32DB: ExecuteCommand (server.cpp:453)
42==32364==    by 0x2B32DB: CRPCTable::execute(JSONRPCRequest const&) const (server.cpp:436)
43==32364==

edit: Comment shortened by @MarcoFalke

MarcoFalke commented at 4:03 pm on June 10, 2020: member

Assertion happens here:

0    /** Create DB handle to real database */
1    BerkeleyDatabase(std::shared_ptr<BerkeleyEnvironment> env, std::string filename) :
2        nUpdateCounter(0), nLastSeen(0), nLastFlushed(0), nLastWalletUpdate(0), env(std::move(env)), strFile(std::move(filename))
3    {
4        auto inserted = this->env->m_databases.emplace(strFile, std::ref(*this));
5        assert(inserted.second); // <-- here
6    }

MarcoFalke added the label Wallet on Jun 10, 2020

fscemama commented at 4:21 pm on June 10, 2020: none

Please let me know if you need any further testing, including patching + recompilation.

fscemama commented at 9:35 am on June 11, 2020: none

Hi, got this tonight in the live env. Unfortunately, it didn’t crash and restart to 0.997 as expected, I’ll have to try again. So slow :/

  0root# valgrind /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
  1==25768== Memcheck, a memory error detector
  2==25768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
  3==25768== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
  4==25768== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
  5==25768==
  6Bitcoin Core starting
  7==25768==
  8==25768== HEAP SUMMARY:
  9==25768==     in use at exit: 2,246,434 bytes in 2,285 blocks
 10==25768==   total heap usage: 3,243 allocs, 958 frees, 2,477,009 bytes allocated
 11==25768==
 12==25768== LEAK SUMMARY:
 13==25768==    definitely lost: 0 bytes in 0 blocks
 14==25768==    indirectly lost: 0 bytes in 0 blocks
 15==25768==      possibly lost: 0 bytes in 0 blocks
 16==25768==    still reachable: 2,246,434 bytes in 2,285 blocks
 17==25768==         suppressed: 0 bytes in 0 blocks
 18==25768== Rerun with --leak-check=full to see details of leaked memory
 19==25768==
 20==25768== For lists of detected and suppressed errors, rerun with: -s
 21==25768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
 22root@backprod2:/home/psb/bin# ==25771== Thread 5 b-httpworker.1:
 23==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
 24==25771==    at 0x541B983: ??? (syscall-template.S:84)
 25==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
 26==25771==    by 0x51A004C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
 27==25771==    by 0x51A0408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
 28==25771==    by 0x519F31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
 29==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
 30==25771==    by 0x512B408: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
 31==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
 32==25771==    by 0x509C026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
 33==25771==    by 0x3DF7E5: BerkeleyEnvironment::Verify(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:277)
 34==25771==    by 0x3DFB4A: BerkeleyBatch::VerifyDatabaseFile(boost::filesystem::path const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)) (db.cpp:422)
 35==25771==    by 0x472D48: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3778)
 36==25771==  Address 0xb2dbb968 is 120 bytes inside a block of size 1,112 alloc'd
 37==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
 38==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
 39==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
 40==25771==    by 0x519EBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
 41==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
 42==25771==    by 0x50B7063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
 43==25771==    by 0x5169387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
 44==25771==    by 0x5169906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
 45==25771==    by 0x512E2B8: __db_vrfy_pgset (in /usr/lib/libdb_cxx-4.8.so)
 46==25771==    by 0x512E4BE: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
 47==25771==    by 0x512B056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
 48==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
 49==25771==
 50==25771== Thread 6 b-httpworker.2:
 51==25771== Conditional jump or move depends on uninitialised value(s)
 52==25771==    at 0x519B6C4: __log_putr.isra.1 (in /usr/lib/libdb_cxx-4.8.so)
 53==25771==    by 0x519CD74: __log_put (in /usr/lib/libdb_cxx-4.8.so)
 54==25771==    by 0x514D384: __db_addrem_log (in /usr/lib/libdb_cxx-4.8.so)
 55==25771==    by 0x515DD0B: __db_pitem (in /usr/lib/libdb_cxx-4.8.so)
 56==25771==    by 0x50B86F2: __bam_iitem (in /usr/lib/libdb_cxx-4.8.so)
 57==25771==    by 0x50B3DBC: __bamc_put (in /usr/lib/libdb_cxx-4.8.so)
 58==25771==    by 0x5156F5B: __dbc_iput (in /usr/lib/libdb_cxx-4.8.so)
 59==25771==    by 0x514B5E5: __db_put (in /usr/lib/libdb_cxx-4.8.so)
 60==25771==    by 0x516449C: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
 61==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
 62==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
 63==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
 64==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
 65==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
 66==25771==
 67==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
 68==25771==    at 0x541B983: ??? (syscall-template.S:84)
 69==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
 70==25771==    by 0x519B3BD: __log_write (in /usr/lib/libdb_cxx-4.8.so)
 71==25771==    by 0x519D031: __log_put (in /usr/lib/libdb_cxx-4.8.so)
 72==25771==    by 0x51BD96B: __txn_regop_log (in /usr/lib/libdb_cxx-4.8.so)
 73==25771==    by 0x51BC444: __txn_commit (in /usr/lib/libdb_cxx-4.8.so)
 74==25771==    by 0x5164431: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
 75==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
 76==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
 77==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
 78==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
 79==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
 80==25771==    by 0x47BD26: CWallet::AddToWalletIfInvolvingMe(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:953)
 81==25771==    by 0x47C1A2: CWallet::SyncTransaction(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:1087)
 82==25771==  Address 0x74af62c9 is 1,785 bytes inside a block of size 65,544 alloc'd
 83==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
 84==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
 85==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
 86==25771==    by 0x519553E: __log_open (in /usr/lib/libdb_cxx-4.8.so)
 87==25771==    by 0x51851EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
 88==25771==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
 89==25771==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
 90==25771==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
 91==25771==    by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533)
 92==25771==    by 0x46C0AE: WalletBatch (walletdb.h:207)
 93==25771==    by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123)
 94==25771==    by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806)
 95==25771==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
 96==25771==
 97==25771==
 98==25771== HEAP SUMMARY:
 99==25771==     in use at exit: 4,185 bytes in 26 blocks
100==25771==   total heap usage: 616,239,856 allocs, 616,239,830 frees, 62,380,430,976 bytes allocated
101==25771==
102==25771== LEAK SUMMARY:
103==25771==    definitely lost: 2,400 bytes in 15 blocks
104==25771==    indirectly lost: 0 bytes in 0 blocks
105==25771==      possibly lost: 304 bytes in 1 blocks
106==25771==    still reachable: 1,481 bytes in 10 blocks
107==25771==         suppressed: 0 bytes in 0 blocks
108==25771== Rerun with --leak-check=full to see details of leaked memory
109==25771==
110==25771== Use --track-origins=yes to see where uninitialised values come from
111==25771== For lists of detected and suppressed errors, rerun with: -s
112==25771== ERROR SUMMARY: 2976 errors from 3 contexts (suppressed: 0 from 0)

fscemama commented at 4:43 pm on June 11, 2020: none

Today, I’ve run valgrind on bitcoind in our live env for a few hours. Incredibly enough, only small crashes before valgrind was launched (small because reindexing from 0.99998, not 0.997).

Valgrind reported the following, but there were no crash at all during its monitoring the bitcoind process. I don’t know why it was so unstable yesterday and days before, and not today. The only system difference is the installation of Debian libc6-dbg package, needed by Valgrind itself ; also, we scheduled creation of wallets to occur less often (once per hour).

More testing seems to be needed.

  0bin# valgrind /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
  1==25768== Memcheck, a memory error detector
  2==25768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
  3==25768== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
  4==25768== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
  5==25768==
  6Bitcoin Core starting
  7==25768==
  8==25768== HEAP SUMMARY:
  9==25768==     in use at exit: 2,246,434 bytes in 2,285 blocks
 10==25768==   total heap usage: 3,243 allocs, 958 frees, 2,477,009 bytes allocated
 11==25768==
 12==25768== LEAK SUMMARY:
 13==25768==    definitely lost: 0 bytes in 0 blocks
 14==25768==    indirectly lost: 0 bytes in 0 blocks
 15==25768==      possibly lost: 0 bytes in 0 blocks
 16==25768==    still reachable: 2,246,434 bytes in 2,285 blocks
 17==25768==         suppressed: 0 bytes in 0 blocks
 18==25768== Rerun with --leak-check=full to see details of leaked memory
 19==25768==
 20==25768== For lists of detected and suppressed errors, rerun with: -s
 21==25768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
 22root@backprod2:/home/psb/bin# ==25771== Thread 5 b-httpworker.1:
 23==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
 24==25771==    at 0x541B983: ??? (syscall-template.S:84)
 25==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
 26==25771==    by 0x51A004C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
 27==25771==    by 0x51A0408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
 28==25771==    by 0x519F31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
 29==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
 30==25771==    by 0x512B408: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
 31==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
 32==25771==    by 0x509C026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
 33==25771==    by 0x3DF7E5: BerkeleyEnvironment::Verify(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:277)
 34==25771==    by 0x3DFB4A: BerkeleyBatch::VerifyDatabaseFile(boost::filesystem::path const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)) (db.cpp:422)
 35==25771==    by 0x472D48: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3778)
 36==25771==  Address 0xb2dbb968 is 120 bytes inside a block of size 1,112 alloc'd
 37==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
 38==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
 39==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
 40==25771==    by 0x519EBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
 41==25771==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
 42==25771==    by 0x50B7063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
 43==25771==    by 0x5169387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
 44==25771==    by 0x5169906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
 45==25771==    by 0x512E2B8: __db_vrfy_pgset (in /usr/lib/libdb_cxx-4.8.so)
 46==25771==    by 0x512E4BE: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
 47==25771==    by 0x512B056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
 48==25771==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
 49==25771==
 50==25771== Thread 6 b-httpworker.2:
 51==25771== Conditional jump or move depends on uninitialised value(s)
 52==25771==    at 0x519B6C4: __log_putr.isra.1 (in /usr/lib/libdb_cxx-4.8.so)
 53==25771==    by 0x519CD74: __log_put (in /usr/lib/libdb_cxx-4.8.so)
 54==25771==    by 0x514D384: __db_addrem_log (in /usr/lib/libdb_cxx-4.8.so)
 55==25771==    by 0x515DD0B: __db_pitem (in /usr/lib/libdb_cxx-4.8.so)
 56==25771==    by 0x50B86F2: __bam_iitem (in /usr/lib/libdb_cxx-4.8.so)
 57==25771==    by 0x50B3DBC: __bamc_put (in /usr/lib/libdb_cxx-4.8.so)
 58==25771==    by 0x5156F5B: __dbc_iput (in /usr/lib/libdb_cxx-4.8.so)
 59==25771==    by 0x514B5E5: __db_put (in /usr/lib/libdb_cxx-4.8.so)
 60==25771==    by 0x516449C: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
 61==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
 62==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
 63==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
 64==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
 65==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
 66==25771==
 67==25771== Syscall param pwrite64(buf) points to uninitialised byte(s)
 68==25771==    at 0x541B983: ??? (syscall-template.S:84)
 69==25771==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
 70==25771==    by 0x519B3BD: __log_write (in /usr/lib/libdb_cxx-4.8.so)
 71==25771==    by 0x519D031: __log_put (in /usr/lib/libdb_cxx-4.8.so)
 72==25771==    by 0x51BD96B: __txn_regop_log (in /usr/lib/libdb_cxx-4.8.so)
 73==25771==    by 0x51BC444: __txn_commit (in /usr/lib/libdb_cxx-4.8.so)
 74==25771==    by 0x5164431: __db_put_pp (in /usr/lib/libdb_cxx-4.8.so)
 75==25771==    by 0x509BC4E: Db::put(DbTxn*, Dbt*, Dbt*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
 76==25771==    by 0x4A5E8D: Write<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (db.h:299)
 77==25771==    by 0x4A5E8D: WriteIC<std::pair<std::__cxx11::basic_string<char>, uint256>, CWalletTx> (walletdb.h:181)
 78==25771==    by 0x4A5E8D: WalletBatch::WriteTx(CWalletTx const&) (walletdb.cpp:76)
 79==25771==    by 0x47AA9C: CWallet::AddToWallet(CWalletTx const&, bool) (wallet.cpp:837)
 80==25771==    by 0x47BD26: CWallet::AddToWalletIfInvolvingMe(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:953)
 81==25771==    by 0x47C1A2: CWallet::SyncTransaction(std::shared_ptr<CTransaction const> const&, CWalletTx::Confirmation, bool) (wallet.cpp:1087)
 82==25771==  Address 0x74af62c9 is 1,785 bytes inside a block of size 65,544 alloc'd
 83==25771==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
 84==25771==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
 85==25771==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
 86==25771==    by 0x519553E: __log_open (in /usr/lib/libdb_cxx-4.8.so)
 87==25771==    by 0x51851EC: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
 88==25771==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
 89==25771==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
 90==25771==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
 91==25771==    by 0x3DC975: BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase&, char const*, bool) (db.cpp:533)
 92==25771==    by 0x46C0AE: WalletBatch (walletdb.h:207)
 93==25771==    by 0x46C0AE: CWallet::LoadWallet(bool&) (wallet.cpp:3123)
 94==25771==    by 0x47E346: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3806)
 95==25771==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
 96==25771==
 97==25771==
 98==25771== HEAP SUMMARY:
 99==25771==     in use at exit: 4,185 bytes in 26 blocks
100==25771==   total heap usage: 616,239,856 allocs, 616,239,830 frees, 62,380,430,976 bytes allocated
101==25771==
102==25771== LEAK SUMMARY:
103==25771==    definitely lost: 2,400 bytes in 15 blocks
104==25771==    indirectly lost: 0 bytes in 0 blocks
105==25771==      possibly lost: 304 bytes in 1 blocks
106==25771==    still reachable: 1,481 bytes in 10 blocks
107==25771==         suppressed: 0 bytes in 0 blocks
108==25771== Rerun with --leak-check=full to see details of leaked memory
109==25771==
110==25771== Use --track-origins=yes to see where uninitialised values come from
111==25771== For lists of detected and suppressed errors, rerun with: -s
112==25771== ERROR SUMMARY: 2976 errors from 3 contexts (suppressed: 0 from 0)
113root@backprod2:/home/psb/bin# valgrind --leak-check=full /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
114root@backprod2:/home/psb/bin# valgrind --leak-check=full /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
115==24289== Memcheck, a memory error detector
116==24289== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
117==24289== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
118==24289== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1
119==24289==
120Bitcoin Core starting
121==24289==
122==24289== HEAP SUMMARY:
123==24289==     in use at exit: 2,246,434 bytes in 2,285 blocks
124==24289==   total heap usage: 3,243 allocs, 958 frees, 2,477,009 bytes allocated
125==24289==
126==24289== LEAK SUMMARY:
127==24289==    definitely lost: 0 bytes in 0 blocks
128==24289==    indirectly lost: 0 bytes in 0 blocks
129==24289==      possibly lost: 0 bytes in 0 blocks
130==24289==    still reachable: 2,246,434 bytes in 2,285 blocks
131==24289==         suppressed: 0 bytes in 0 blocks
132==24289== Reachable blocks (those to which a pointer was found) are not shown.
133==24289== To see them, rerun with: --leak-check=full --show-leak-kinds=all
134==24289==
135==24289== For lists of detected and suppressed errors, rerun with: -s
136==24289== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
137root@backprod2:/home/psb/bin# ==24359== Thread 7 b-httpworker.3:
138==24359== Syscall param pwrite64(buf) points to uninitialised byte(s)
139==24359==    at 0x541B983: ??? (syscall-template.S:84)
140==24359==    by 0x51B2188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
141==24359==    by 0x51A004C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
142==24359==    by 0x51A0408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
143==24359==    by 0x519F31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
144==24359==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
145==24359==    by 0x5166A48: __db_new (in /usr/lib/libdb_cxx-4.8.so)
146==24359==    by 0x50C7CAE: __bam_split (in /usr/lib/libdb_cxx-4.8.so)
147==24359==    by 0x50B3EA1: __bamc_put (in /usr/lib/libdb_cxx-4.8.so)
148==24359==    by 0x5156F5B: __dbc_iput (in /usr/lib/libdb_cxx-4.8.so)
149==24359==    by 0x514B5E5: __db_put (in /usr/lib/libdb_cxx-4.8.so)
150==24359==    by 0x512E0C1: __db_vrfy_putpageinfo (in /usr/lib/libdb_cxx-4.8.so)
151==24359==  Address 0x2370b958 is 120 bytes inside a block of size 1,112 alloc'd
152==24359==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
153==24359==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
154==24359==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
155==24359==    by 0x519EBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
156==24359==    by 0x51A1D27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
157==24359==    by 0x50B7063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
158==24359==    by 0x5169387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
159==24359==    by 0x5169906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
160==24359==    by 0x512E415: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
161==24359==    by 0x512B056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
162==24359==    by 0x512D846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
163==24359==    by 0x509C026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
164==24359==
165==24359==
166==24359== HEAP SUMMARY:
167==24359==     in use at exit: 8,985 bytes in 56 blocks
168==24359==   total heap usage: 314,206,840 allocs, 314,206,784 frees, 39,827,904,698 bytes allocated
169==24359==
170==24359== Thread 1 b-shutoff:
171==24359== 160 bytes in 1 blocks are definitely lost in loss record 9 of 13
172==24359==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
173==24359==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
174==24359==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
175==24359==    by 0x513E5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so)
176==24359==    by 0x51851D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
177==24359==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
178==24359==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
179==24359==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
180==24359==    by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405)
181==24359==    by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756)
182==24359==    by 0x3EBDEF: VerifyWallets(interfaces::Chain&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (load.cpp:58)
183==24359==    by 0x15E76D: AppInitMain(NodeContext&) (init.cpp:1334)
184==24359==
185==24359== 304 bytes in 1 blocks are possibly lost in loss record 10 of 13
186==24359==    at 0x4030A0C: calloc (vg_replace_malloc.c:760)
187==24359==    by 0x4011E31: allocate_dtv (dl-tls.c:322)
188==24359==    by 0x40127BD: _dl_allocate_tls (dl-tls.c:539)
189==24359==    by 0x5413189: allocate_stack (allocatestack.c:584)
190==24359==    by 0x5413189: pthread_create@@GLIBC_2.2.5 (pthread_create.c:663)
191==24359==    by 0x5B3B164: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22)
192==24359==    by 0x624F7D: thread<void (&)(leveldb::(anonymous namespace)::PosixEnv*), leveldb::(anonymous namespace)::PosixEnv*> (thread:137)
193==24359==    by 0x624F7D: leveldb::(anonymous namespace)::PosixEnv::Schedule(void (*)(void*), void*) (env_posix.cc:802)
194==24359==    by 0x5F9586: leveldb::DBImpl::Get(leveldb::ReadOptions const&, leveldb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (db_impl.cc:1150)
195==24359==    by 0x2D881E: Read<std::pair<char, int>, CBlockFileInfo> (dbwrapper.h:238)
196==24359==    by 0x2D881E: CBlockTreeDB::ReadBlockFileInfo(int, CBlockFileInfo&) (txdb.cpp:153)
197==24359==    by 0x3252A9: LoadBlockIndexDB(CChainParams const&) (validation.cpp:4121)
198==24359==    by 0x325BB4: LoadBlockIndex(CChainParams const&) (validation.cpp:4573)
199==24359==    by 0x160AC9: AppInitMain(NodeContext&) (init.cpp:1553)
200==24359==    by 0x144110: AppInit(int, char**) (bitcoind.cpp:146)
201==24359==
202==24359== 7,040 bytes in 44 blocks are definitely lost in loss record 13 of 13
203==24359==    at 0x402E7BD: malloc (vg_replace_malloc.c:307)
204==24359==    by 0x51AF964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
205==24359==    by 0x517E4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
206==24359==    by 0x513E5F7: __lock_open (in /usr/lib/libdb_cxx-4.8.so)
207==24359==    by 0x51851D0: __env_attach_regions (in /usr/lib/libdb_cxx-4.8.so)
208==24359==    by 0x518531A: __env_open (in /usr/lib/libdb_cxx-4.8.so)
209==24359==    by 0x509EFA3: DbEnv::open(char const*, unsigned int, int) (in /usr/lib/libdb_cxx-4.8.so)
210==24359==    by 0x3DAC27: BerkeleyEnvironment::Open(bool) (db.cpp:199)
211==24359==    by 0x3DBE7F: BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:405)
212==24359==    by 0x472B3F: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3756)
213==24359==    by 0x482359: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:152)
214==24359==    by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626)
215==24359==
216==24359== LEAK SUMMARY:
217==24359==    definitely lost: 7,200 bytes in 45 blocks
218==24359==    indirectly lost: 0 bytes in 0 blocks
219==24359==      possibly lost: 304 bytes in 1 blocks
220==24359==    still reachable: 1,481 bytes in 10 blocks
221==24359==         suppressed: 0 bytes in 0 blocks
222==24359== Reachable blocks (those to which a pointer was found) are not shown.
223==24359== To see them, rerun with: --leak-check=full --show-leak-kinds=all
224==24359==
225==24359== Use --track-origins=yes to see where uninitialised values come from
226==24359== For lists of detected and suppressed errors, rerun with: -s
227==24359== ERROR SUMMARY: 3167 errors from 4 contexts (suppressed: 0 from 0)

MarcoFalke commented at 4:54 pm on June 11, 2020: member

You might want to start valgrind with the suppressions file --suppressions= and point it to the file contrib/valgrind.supp.

But it seems we already found the location of the crash: #19232 (comment)

ryanofsky commented at 0:06 am on June 17, 2020: member

re: #19232 (comment)

Assert is strange, but seems like maybe it could happen if a wallet loaded more than once at the same time by different threads. What is strange is that the assert that’s failing runs twice in a row but somehow succeeds the first time and fails the second time:

 0loadwallet
 1  LoadWallet
 2    CWallet::Verify
 3      WalletDatabase::Create
 4        BerkeleyDatabase::BerkeleyDatabase
 5           m_databases.emplace                <--- apparently succeeds
 6        BerkeleyDatabase::~BerkeleyDatabase
 7           m_databases.erase
 8    CWallet::CreateWalletFromFile
 9      WalletDatabase::Create
10         BerkeleyDatabase::BerkeleyDatabase
11           m_databases.emplace                <--- fails & aborts according to trace

I don’t see how the crash would happen if the application is “[waiting] for any bitcoin-cli command to terminate before issuing another one” as described #19232#issue-636073520.

But if it’s possible the application is loading the same wallet twice from different threads, then the crash makes more sense. I think we would probably have to fix it by introducing something like g_loading_wallet_set similar to existing g_unloading_wallet_set to track and avoid concurrently loading wallets.

fscemama commented at 0:09 am on June 17, 2020: none

Interesting enough. I should probably make sure the wait-for-any-bitcoin-cli-command-to-terminate is working well. Your idea that it should be prevented from the daemon itself does make sense. Thanks.

promag commented at 1:00 am on June 17, 2020: member

But if it’s possible the application is loading the same wallet twice from different threads, then the crash makes more sense.

Right, there’s nothing handling this. Should it prevent concurrent loads by env? Or simply any concurrent loads?

BTW, concurrent unload can throw “Requested wallet does not exist or is not loaded” or “Requested wallet already unloaded”. I wonder if with 2 concurrent loads, both should succeed - one actually loads, the other waits for the first - so both clients work as expected.

ryanofsky commented at 2:48 am on June 17, 2020: member

Right, there’s nothing handling this. Should it prevent concurrent loads by env? Or simply any concurrent loads?

The idea with the g_loading_wallet_set suggestion is that it would just track wallet loads using given wallet names, the same way g_unloading_wallet_set tracks wallet unloads. Environments don’t actually have much to do with this bug and are already protected with .walletlock files. Also environments aren’t visible at the wallet.cpp layer, only the db.cpp layer.

BTW, concurrent unload can throw “Requested wallet does not exist or is not loaded” or “Requested wallet already unloaded”.

That’s interesting. I think more ideally they would both return a simple “Wallet is not currently loaded” error so applications trying to handle this don’t have to deal with two different error messages, one more frequent and the other more rare.

I wonder if with 2 concurrent loads, both should succeed - one actually loads, the other waits for the first - so both clients work as expected.

I think that would just add complexity without use-cases to justify it. It could also be confusing and break backwards compatibility with existing applications expecting to see the “Duplicate -wallet filename specified” error when a wallet’s already loaded.

fscemama commented at 9:18 am on June 17, 2020: none

Dear all, I can now confirm that after using procmail’s lockfile to make absolutely sure no concurrent calls to bitcoin-cli are made, the stability has greatly improved. It can be dealt with that way, at least. Your help is precious.

promag commented at 10:16 am on June 17, 2020: member

@fscemama it would be great if you can test #19300.

Note that loadwallet can fail (RPC error) if the wallet is already being loaded so you have to deal with this error.

But honestly you should rethink your approach. The server doesn’t count how many clients “load/want” a given wallet so when client C1 unloads the wallet W1 it can mess some other client C2 that is also using W1. I think you should track/handle this outside unless there’s agreement in supporting this use case here (which I don’t support :trollface:).

fscemama commented at 11:18 am on June 17, 2020: none

Lol, we do track and handle, this at least works well :) BTW, I’m glad to see you again, Promag. Right advice. About #19300, did the following with no error.

0# Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd
1wallet=xxxxxx
2for i in {1..1000}
3do
4  echo $i
5  bitcoin-cli loadwallet $wallet
6  bitcoin-cli unloadwallet $wallet
7done

promag commented at 11:25 am on June 17, 2020: member

did the following with no error.

You must run that script multiple times in parallel, like

0test.sh &
1test.sh

fscemama commented at 11:28 am on June 17, 2020: none

Sorry. Did it, immediate crash.

promag commented at 11:30 am on June 17, 2020: member

Sorry. Did it, immediate crash.

Was this with #19300 branch? What was the error?

fscemama commented at 11:31 am on June 17, 2020: none

 0...
 1error code: -18
 2error message:
 3Requested wallet does not exist or is not loaded
 45 26133
 5error code: -4
 6error message:
 7Wallet file verification failed: Error loading wallet /root/wallets/0033665502992-2018-08-09-15-25-18-653299-00. Duplicate -wallet filename specified.
 8error code: -18
 9error message:
10Requested wallet does not exist or is not loaded
116 26133
12error code: -4
13error message:
14Wallet file verification failed: Error loading wallet /root/wallets/0033665502992-2018-08-09-15-25-18-653299-00. Duplicate -wallet filename specified.
15error code: -18
16error message:
17Requested wallet does not exist or is not loaded
187 26133
195 26232
20error: Could not connect to the server 127.0.0.1:18332 (error code 1 - "EOF reached")
21
22Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
23error: Could not connect to the server 127.0.0.1:18332 (error code 1 - "EOF reached")
24
25Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
26error: Could not connect to the server 127.0.0.1:18332
27...

fscemama commented at 11:32 am on June 17, 2020: none

0root# cat a.sh
1#!/bin/sh
2wallet=xxxxx
3for i in 0 1 2 3 4 5 6 7 8 9
4do
5  echo $i $$
6  bitcoin-cli loadwallet $wallet
7  bitcoin-cli unloadwallet $wallet
8done

Crash after launching a few times in background.

fscemama commented at 11:34 am on June 17, 2020: none

This probably deserves a valgrind. Please give 10 minutes.

promag commented at 11:54 am on June 17, 2020: member

Can you repeat this with #19300 branch?

fscemama commented at 12:06 pm on June 17, 2020: none

Sorry, removed much pollution (not needed processes). Useful info:

 0=30964== Memcheck, a memory error detector
 1==30964== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
 2==30964== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
 3==30964== Command: /usr/local/bin/bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128
 4==30964==
 5Bitcoin Core starting
 6==30964==
 7==30964== HEAP SUMMARY:
 8==30964==     in use at exit: 2,232,114 bytes in 2,271 blocks
 9==30964==   total heap usage: 3,234 allocs, 963 frees, 2,462,760 bytes allocated
10==30964==
11==30964== LEAK SUMMARY:
12==30964==    definitely lost: 0 bytes in 0 blocks
13==30964==    indirectly lost: 0 bytes in 0 blocks
14==30964==      possibly lost: 0 bytes in 0 blocks
15==30964==    still reachable: 2,232,114 bytes in 2,271 blocks
16==30964==         suppressed: 0 bytes in 0 blocks
17==30964== Reachable blocks (those to which a pointer was found) are not shown.
18==30964== To see them, rerun with: --leak-check=full --show-leak-kinds=all
19==30964==
20==30964== For lists of detected and suppressed errors, rerun with: -s
21==30964== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
22root@backdev2:/home/psb/bin# ==30973== Syscall param pwrite64(buf) points to uninitialised byte(s)
23==30973==    at 0x507B983: ??? (syscall-template.S:84)
24==30973==    by 0x53CF188: __os_io (in /usr/lib/libdb_cxx-4.8.so)
25==30973==    by 0x53BD04C: __memp_pgwrite.part.0 (in /usr/lib/libdb_cxx-4.8.so)
26==30973==    by 0x53BD408: __memp_bhwrite (in /usr/lib/libdb_cxx-4.8.so)
27==30973==    by 0x53BC31A: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
28==30973==    by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
29==30973==    by 0x5348408: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
30==30973==    by 0x534A846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
31==30973==    by 0x52B9026: Db::verify(char const*, char const*, std::ostream*, unsigned int) (in /usr/lib/libdb_cxx-4.8.so)
32==30973==    by 0x3DF7E5: BerkeleyEnvironment::Verify(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:277)
33==30973==    by 0x3DFB4A: BerkeleyBatch::VerifyDatabaseFile(boost::filesystem::path const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool (*)(boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)) (db.cpp:422)
34==30973==    by 0x472D48: CWallet::Verify(interfaces::Chain&, WalletLocation const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:3778)
35==30973==  Address 0xd63ee08 is 120 bytes inside a block of size 1,112 alloc'd
36==30973==    at 0x402E7BD: malloc (vg_replace_malloc.c:309)
37==30973==    by 0x53CC964: __os_malloc (in /usr/lib/libdb_cxx-4.8.so)
38==30973==    by 0x539B4F3: __env_alloc (in /usr/lib/libdb_cxx-4.8.so)
39==30973==    by 0x53BBBC0: __memp_alloc (in /usr/lib/libdb_cxx-4.8.so)
40==30973==    by 0x53BED27: __memp_fget (in /usr/lib/libdb_cxx-4.8.so)
41==30973==    by 0x52D4063: __bam_new_file (in /usr/lib/libdb_cxx-4.8.so)
42==30973==    by 0x5386387: __db_new_file (in /usr/lib/libdb_cxx-4.8.so)
43==30973==    by 0x5386906: __db_open (in /usr/lib/libdb_cxx-4.8.so)
44==30973==    by 0x534B2B8: __db_vrfy_pgset (in /usr/lib/libdb_cxx-4.8.so)
45==30973==    by 0x534B4BE: __db_vrfy_dbinfo_create (in /usr/lib/libdb_cxx-4.8.so)
46==30973==    by 0x5348056: __db_verify (in /usr/lib/libdb_cxx-4.8.so)
47==30973==    by 0x534A846: __db_verify_internal (in /usr/lib/libdb_cxx-4.8.so)
48==30973==
49==30973==
50==30973== Process terminating with default action of signal 6 (SIGABRT)
51==30973==    at 0x6350FFF: raise (raise.c:51)
52==30973==    by 0x6352429: abort (abort.c:89)
53==30973==    by 0x6349E66: __assert_fail_base (assert.c:92)
54==30973==    by 0x6349F11: __assert_fail (assert.c:101)
55==30973==    by 0x49085A: BerkeleyDatabase (db.h:125)
56==30973==    by 0x49085A: std::unique_ptr<BerkeleyDatabase, std::default_delete<BerkeleyDatabase> > MakeUnique<BerkeleyDatabase, std::shared_ptr<BerkeleyEnvironment>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::shared_ptr<BerkeleyEnvironment>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (memory.h:16)
57==30973==    by 0x49091F: BerkeleyDatabase::Create(boost::filesystem::path const&) (db.h:139)
58==30973==    by 0x47E2C1: CWallet::CreateWalletFromFile(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, unsigned long) (wallet.cpp:3805)
59==30973==    by 0x4823E6: LoadWallet(interfaces::Chain&, WalletLocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (wallet.cpp:157)
60==30973==    by 0x40A7FD: loadwallet(JSONRPCRequest const&) (rpcwallet.cpp:2626)
61==30973==    by 0x22BFEB: operator() (server.h:104)
62==30973==    by 0x22BFEB: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
63==30973==    by 0x14A95E: operator() (functional:2127)
64==30973==    by 0x14A95E: operator() (chain.cpp:208)
65==30973==    by 0x14A95E: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (functional:1717)
66==30973==    by 0x2B32DB: operator() (functional:2127)
67==30973==    by 0x2B32DB: ExecuteCommand (server.cpp:453)
68==30973==    by 0x2B32DB: CRPCTable::execute(JSONRPCRequest const&) const (server.cpp:436)
69==30973==
70==30973== HEAP SUMMARY:
71==30973==     in use at exit: 475,082,336 bytes in 3,760,553 blocks
72==30973==   total heap usage: 9,888,764 allocs, 6,128,211 frees, 1,385,505,603 bytes allocated
73==30973==

MarcoFalke closed this on Jun 29, 2020

MarcoFalke locked this on Feb 15, 2022

Regular crash+autorestart after loading/unloading many wallets #19232