crash in CMerkleTx::GetDepthInMainChainINTERNAL() #3997

issue dooglus opened this issue on April 2, 2014
  1. dooglus commented at 8:16 PM on April 2, 2014: contributor

    I've been running bitcoin-qt inside gdb for over a year now, because it used to crash pretty often. It just crashed for the first time in over 6 months, a week or two after upgrading to the recent release. I would guess this is a new issue introduced in 0.9.

    I built the code myself, from the git v0.9.0 tag, on a 64 bit Linux system.

    Here's what gdb said:

    Program received signal SIGSEGV, Segmentation fault.
CMerkleTx::GetDepthInMainChainINTERNAL (this=this@entry=0x7fffb77d7ac0, 
    pindexRet=@0x7fffffffd148: 0x555556ca0120) at main.cpp:884
884     if (!pindex || !chainActive.Contains(pindex))
(gdb) where
[#0](/bitcoin-bitcoin/0/)  CMerkleTx::GetDepthInMainChainINTERNAL (this=this@entry=0x7fffb77d7ac0, 
    pindexRet=@0x7fffffffd148: 0x555556ca0120) at main.cpp:884
[#1](/bitcoin-bitcoin/1/)  0x000055555571dca6 in CMerkleTx::GetDepthInMainChain (
    this=this@entry=0x7fffb77d7ac0, pindexRet=@0x7fffffffd148: 0x555556ca0120)
    at main.cpp:901
[#2](/bitcoin-bitcoin/2/)  0x00005555556b632d in GetDepthInMainChain (this=0x7fffb77d7ac0)
    at ../../src/main.h:466
[#3](/bitcoin-bitcoin/3/)  CWalletTx::IsTrusted (this=this@entry=0x7fffb77d7ac0)
    at ../../src/wallet.h:665
[#4](/bitcoin-bitcoin/4/)  0x00005555556b40cd in TransactionRecord::updateStatus (
    this=this@entry=0x555556c00ca0, wtx=...) at transactionrecord.cpp:167
[#5](/bitcoin-bitcoin/5/)  0x00005555556933cd in TransactionTablePriv::index (this=<optimized out>, 
    idx=idx@entry=6314) at transactiontablemodel.cpp:201
[#6](/bitcoin-bitcoin/6/)  0x000055555569231e in TransactionTableModel::index (this=0x555556a5b340, 
    row=6314, column=0, parent=...) at transactiontablemodel.cpp:615
[#7](/bitcoin-bitcoin/7/)  0x000055555568f398 in TransactionFilterProxy::filterAcceptsRow (this=
    0x55555625ee30, sourceRow=6314, sourceParent=...)
    at transactionfilterproxy.cpp:33

    and here's debug.log (it looks like bitcoin-qt was only just getting started):

    2014-03-31 04:41:51 Bitcoin version v0.9.0-1-gd824858-beta (2014-03-20 09:19:05 -0700)
2014-03-31 04:41:51 Using OpenSSL version OpenSSL 1.0.1c 10 May 2012
2014-03-31 04:41:51 Default data directory /home/chris/.bitcoin
2014-03-31 04:41:51 Using data directory /home/chris/.bitcoin
2014-03-31 04:41:51 Using at most 125 connections (1024 file descriptors available)
2014-03-31 04:41:51 Using 2 threads for script verification
2014-03-31 04:41:51 Using wallet wallet.dat
2014-03-31 04:41:51 init message: Verifying wallet...
2014-03-31 04:41:51 dbenv.open LogDir=/home/chris/.bitcoin/database ErrorFile=/home/chris/.bitcoin/db.log
2014-03-31 04:42:16 Bound to [::]:8333
2014-03-31 04:42:16 Bound to 0.0.0.0:8333
2014-03-31 04:42:16 init message: Loading block index...
2014-03-31 04:42:16 Opening LevelDB in /home/chris/.bitcoin/blocks/index
2014-03-31 04:42:17 Opened LevelDB successfully
2014-03-31 04:42:17 Opening LevelDB in /home/chris/.bitcoin/chainstate
2014-03-31 04:42:18 Opened LevelDB successfully
2014-03-31 04:42:36 LoadBlockIndexDB(): last block file = 89
2014-03-31 04:42:36 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=338, size=67010312, heights=292948...293280, time=2014-03-28...2014-03-30)
2014-03-31 04:42:36 LoadBlockIndexDB(): transaction index disabled
2014-03-31 04:42:36 LoadBlockIndexDB(): hashBestChain=000000000000000052d11e4b468cf8e66152f4687ccf6f7b24a4eb8c7c183bbb height=293280 date=2014-03-30 20:05:47 progress=0.998126
2014-03-31 04:42:36 init message: Verifying blocks...
2014-03-31 04:42:36 Verifying last 288 blocks at level 2
2014-03-31 04:43:30 No coin database inconsistencies in last 0 blocks (0 transactions)
2014-03-31 04:43:30  block index           74479ms
2014-03-31 04:43:30 init message: Loading wallet...
2014-03-31 04:44:50 nFileVersion = 90000
2014-03-31 04:44:50 Keys: 36667 plaintext, 0 encrypted, 0 w/ metadata, 36667 total
2014-03-31 04:44:50  wallet                80181ms
2014-03-31 04:44:50 init message: Loading addresses...
2014-03-31 04:44:50 Loaded 18722 addresses from peers.dat  118ms
2014-03-31 04:44:50 mapBlockIndex.size() = 293606
2014-03-31 04:44:50 nBestHeight = 293280
2014-03-31 04:44:50 setKeyPool.size() = 4996
2014-03-31 04:44:50 mapWallet.size() = 30432
2014-03-31 04:44:50 mapAddressBook.size() = 19485
2014-03-31 04:44:50 ext-ip thread start
2014-03-31 04:44:50 dnsseed thread start
2014-03-31 04:44:50 net thread start
2014-03-31 04:44:50 Loading addresses from DNS seeds (could take a while)
2014-03-31 04:44:50 addcon thread start
2014-03-31 04:44:50 msghand thread start
2014-03-31 04:44:50 opencon thread start
2014-03-31 04:44:50 dumpaddr thread start
2014-03-31 04:44:50 init message: Done loading
2014-03-31 04:44:51 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-03-31 04:44:51 GetMyExternalIP() received [a.b.c.d] a.b.c.d:0
2014-03-31 04:44:51 GetMyExternalIP() returned a.b.c.d
2014-03-31 04:44:51 AddLocal(a.b.c.d:8333,4)
2014-03-31 04:44:51 ext-ip thread exit
2014-03-31 04:44:51 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-03-31 04:44:51 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-03-31 04:44:51 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-03-31 04:44:51 receive version message: /Satoshi:0.8.6/: version 70001, blocks=293333, us=a.b.c.d:9089, them=e.f.g.h:8333, peer=e.f.g.h:8333
2014-03-31 04:44:51 Added time data, samples 2, offset -7 (+0 minutes)
2014-03-31 04:44:51 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-03-31 04:44:51 Initialization result: 1
2014-03-31 04:44:53 Pre-allocating up to position 0x5000000 in blk00089.dat
2014-03-31 04:45:08 118 addresses found from DNS seeds
2014-03-31 04:45:08 dnsseed thread exit
  2. laanwj added the label Bug on Apr 2, 2014
  3. laanwj commented at 8:30 PM on April 2, 2014: member

    This looks weird. Could be a synchronization problem.

  4. dooglus commented at 8:33 PM on April 2, 2014: contributor

    I also ran a:

    (gdb) thread apply all bt full

    but the output contains a lot of binary data, so I'm not sure it's safe to post it here.

  5. dooglus commented at 3:59 AM on April 3, 2014: contributor

    The wallet.dat is an old "hot wallet" from the Just-Dice server. I retired it when it got too big. It's about 157MB. I don't know if any of that is relevant.

    Restarting bitcoin-qt went fine however.

  6. dooglus commented at 6:55 PM on April 10, 2014: contributor

    I just had the same crash again.

    Here's the backtrace:

    [New Thread 0x7fff86f5d700 (LWP 20786)]
[Thread 0x7fffbccc7700 (LWP 20776) exited]
[New Thread 0x7fff8670c700 (LWP 20787)]

Program received signal SIGSEGV, Segmentation fault.
CMerkleTx::GetDepthInMainChainINTERNAL (this=this@entry=0x7fffab600f90, 
    pindexRet=@0x7fffffffd448: 0x555556bb0120) at main.cpp:884
884     if (!pindex || !chainActive.Contains(pindex))
(gdb) where
[#0](/bitcoin-bitcoin/0/)  CMerkleTx::GetDepthInMainChainINTERNAL (this=this@entry=0x7fffab600f90, 
    pindexRet=@0x7fffffffd448: 0x555556bb0120) at main.cpp:884
[#1](/bitcoin-bitcoin/1/)  0x000055555571dca6 in CMerkleTx::GetDepthInMainChain (
    this=this@entry=0x7fffab600f90, pindexRet=@0x7fffffffd448: 0x555556bb0120)
    at main.cpp:901
[#2](/bitcoin-bitcoin/2/)  0x00005555556b632d in GetDepthInMainChain (this=0x7fffab600f90)
    at ../../src/main.h:466
[#3](/bitcoin-bitcoin/3/)  CWalletTx::IsTrusted (this=this@entry=0x7fffab600f90)
    at ../../src/wallet.h:665
[#4](/bitcoin-bitcoin/4/)  0x00005555556b40cd in TransactionRecord::updateStatus (
    this=this@entry=0x5555571f9720, wtx=...) at transactionrecord.cpp:167
[#5](/bitcoin-bitcoin/5/)  0x00005555556933cd in TransactionTablePriv::index (this=<optimized out>, 
    idx=idx@entry=29366) at transactiontablemodel.cpp:201
[#6](/bitcoin-bitcoin/6/)  0x000055555569231e in TransactionTableModel::index (this=0x555556a62f70, 
    row=29366, column=0, parent=...) at transactiontablemodel.cpp:615
[#7](/bitcoin-bitcoin/7/)  0x000055555568f398 in TransactionFilterProxy::filterAcceptsRow (this=
    0x5555572e0940, sourceRow=29366, sourceParent=...)
    at transactionfilterproxy.cpp:33

    and here's the debug.log:

    2014-04-10 18:43:25 Bitcoin version v0.9.0-1-gd824858-beta (2014-03-20 09:19:05 -0700)
2014-04-10 18:43:25 Using OpenSSL version OpenSSL 1.0.1e 11 Feb 2013
2014-04-10 18:43:25 Default data directory /home/chris/.bitcoin
2014-04-10 18:43:25 Using data directory /home/chris/.bitcoin
2014-04-10 18:43:25 Using at most 125 connections (1024 file descriptors available)
2014-04-10 18:43:25 Using 4 threads for script verification
2014-04-10 18:43:25 Using wallet wallet.dat
2014-04-10 18:43:25 init message: Verifying wallet...
2014-04-10 18:43:25 dbenv.open LogDir=/home/chris/.bitcoin/database ErrorFile=/home/chris/.bitcoin/db.log
2014-04-10 18:43:31 Bound to [::]:8333
2014-04-10 18:43:31 Bound to 0.0.0.0:8333
2014-04-10 18:43:31 init message: Loading block index...
2014-04-10 18:43:31 Opening LevelDB in /home/chris/.bitcoin/blocks/index
2014-04-10 18:43:31 Opened LevelDB successfully
2014-04-10 18:43:31 Opening LevelDB in /home/chris/.bitcoin/chainstate
2014-04-10 18:43:33 Opened LevelDB successfully
2014-04-10 18:43:39 LoadBlockIndexDB(): last block file = 91
2014-04-10 18:43:39 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=557, size=134128178, heights=294301...294855, time=2014-04-05...2014-04-08)
2014-04-10 18:43:39 LoadBlockIndexDB(): transaction index disabled
2014-04-10 18:43:39 LoadBlockIndexDB(): hashBestChain=00000000000000001ace9b38d37f391b04e9e094dd22e2f141a6d01b955907b0 height=294855 date=2014-04-08 23:28:27 progress=0.991102
2014-04-10 18:43:39 init message: Verifying blocks...
2014-04-10 18:43:39 Verifying last 288 blocks at level 2
2014-04-10 18:43:51 No coin database inconsistencies in last 0 blocks (0 transactions)
2014-04-10 18:43:51  block index           20269ms
2014-04-10 18:43:51 init message: Loading wallet...
2014-04-10 18:46:18 nFileVersion = 90000
2014-04-10 18:46:18 Keys: 36667 plaintext, 0 encrypted, 0 w/ metadata, 36667 total
2014-04-10 18:46:19  wallet               147566ms
2014-04-10 18:46:19 init message: Loading addresses...
2014-04-10 18:46:19 Loaded 18216 addresses from peers.dat  151ms
2014-04-10 18:46:19 mapBlockIndex.size() = 295183
2014-04-10 18:46:19 nBestHeight = 294855
2014-04-10 18:46:19 setKeyPool.size() = 4996
2014-04-10 18:46:19 mapWallet.size() = 30434
2014-04-10 18:46:19 mapAddressBook.size() = 19485
2014-04-10 18:46:19 ext-ip thread start
2014-04-10 18:46:19 dnsseed thread start
2014-04-10 18:46:19 Loading addresses from DNS seeds (could take a while)
2014-04-10 18:46:19 net thread start
2014-04-10 18:46:19 addcon thread start
2014-04-10 18:46:19 opencon thread start
2014-04-10 18:46:19 msghand thread start
2014-04-10 18:46:19 dumpaddr thread start
2014-04-10 18:46:19 init message: Done loading
2014-04-10 18:46:19 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-10 18:46:20 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-10 18:46:20 GetMyExternalIP() received [a.b.c.d] a.b.c.d:0
2014-04-10 18:46:20 GetMyExternalIP() returned a.b.c.d
2014-04-10 18:46:20 AddLocal(a.b.c.d:8333,4)
2014-04-10 18:46:20 ext-ip thread exit
2014-04-10 18:46:20 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-10 18:46:20 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-10 18:46:20 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-10 18:46:20 Initialization result: 1
2014-04-10 18:46:21 receive version message: /Satoshi:0.8.5/: version 70001, blocks=295149, us=a.b.c.d:41277, them=e.f.g.h:8333, peer=e.f.g.h:8333
2014-04-10 18:46:21 Added time data, samples 2, offset +2 (+0 minutes)
2014-04-10 18:46:26 Leaving block file 91: CBlockFileInfo(blocks=557, size=134128178, heights=294301...294855, time=2014-04-05...2014-04-08)
2014-04-10 18:46:26 Pre-allocating up to position 0x1000000 in blk00092.dat
2014-04-10 18:46:28 Pre-allocating up to position 0x100000 in rev00092.dat
2014-04-10 18:46:28 UpdateTip: new best=00000000000000005bc86c2eb08ffb68d420c895b59cc98019cad366e54ad945  height=294856  log2_work=77.845013  tx=36485528  date=2014-04-08 23:55:49 progress=0.991186
Requesting initialize
Running AppInit2 in thread
2014-04-10 18:47:21

    There's mention of dust transactions again.

  7. laanwj added the label Priority High on Apr 11, 2014
  8. laanwj commented at 6:00 AM on April 11, 2014: member

    Bumping this to high priority.

    Given the backtrace I doubt it has anything to do with dust transactions, though this is of course not impossible (possible scenario: wallet tries to accept unconfirmed dust transaction to mempool, this fails, wallet keeps some pointer at NULL. Gdb is not very helpful there, note that the value of 'this' is conveniently not shown in the backtrace).

    But more likely still is that there is a small time at which chainActive is invalid. I wonder if we acquire the right mutexes there.

  9. dooglus commented at 6:15 AM on April 12, 2014: contributor

    Is there anything I can usefully type at the (gdb) prompt at the point of the crash? I have the prompt in a terminal now, having seen the crash for a 3rd time.

    I tried a few things, but I've no idea what I'm doing (as is apparent from the log below):

    Program received signal SIGSEGV, Segmentation fault.
CMerkleTx::GetDepthInMainChainINTERNAL (this=this@entry=0x7fffaf4ca830, 
    pindexRet=@0x7fffffffd348: 0x7fffffff0120) at main.cpp:884
884     if (!pindex || !chainActive.Contains(pindex))
(gdb) where
[#0](/bitcoin-bitcoin/0/)  CMerkleTx::GetDepthInMainChainINTERNAL (this=this@entry=0x7fffaf4ca830, 
    pindexRet=@0x7fffffffd348: 0x7fffffff0120) at main.cpp:884
[#1](/bitcoin-bitcoin/1/)  0x00005555556d2d84 in CMerkleTx::GetDepthInMainChain (
    this=this@entry=0x7fffaf4ca830, pindexRet=@0x7fffffffd348: 0x7fffffff0120)
    at main.cpp:901
(gdb) p pindex
$1 = (CBlockIndex *) 0x7fffce1d27c0
(gdb) p *pindex
$2 = {phashBlock = 0x7fffce1d2870, pprev = 0x7fffce76e280, nHeight = 278865, 
  nFile = 67, nDataPos = 35780181, nUndoPos = 4688225, 
  nChainWork = {<base_uint<256u>> = {pn = {3983914728, 3086537670, 2818, 0, 
        0, 0, 0, 0}}, <No data fields>}, nTx = 657, nChainTx = 30494343, 
  nStatus = 29, nVersion = 2, hashMerkleRoot = {<base_uint<256u>> = {pn = {
        2507042385, 3851559055, 2920226300, 340783779, 2588707176, 
        2242370894, 447711919, 1858711587}}, <No data fields>}, 
  nTime = 1388979213, nBits = 419628831, nNonce = 3562446643, nSequenceId = 0}
(gdb) p this
$3 = (const CMerkleTx * const) 0x7fffaf4ca830
(gdb) p *this
$4 = {<CTransaction> = {static nMinTxFee = 10000, 
    static nMinRelayTxFee = 1000, static CURRENT_VERSION = 1, nVersion = 1, 
    vin = {<std::_Vector_base<CTxIn, std::allocator<CTxIn> >> = {
        _M_impl = {<std::allocator<CTxIn>> = {<__gnu_cxx::new_allocator<CTxIn>> = {<No data fields>}, <No data fields>}, _M_start = 0x7fffaf4ca960, 
          _M_finish = 0x7fffaf4ca9f0, 
          _M_end_of_storage = 0x7fffaf4ca9f0}}, <No data fields>}, 
    vout = {<std::_Vector_base<CTxOut, std::allocator<CTxOut> >> = {
        _M_impl = {<std::allocator<CTxOut>> = {<__gnu_cxx::new_allocator<CTxOut>> = {<No data fields>}, <No data fields>}, _M_start = 0x7fffaf4caa70, 
          _M_finish = 0x7fffaf4caa90, 
          _M_end_of_storage = 0x7fffaf4caa90}}, <No data fields>}, 
    nLockTime = 0}, hashBlock = {<base_uint<256u>> = {pn = {1286607707, 
        3584892090, 1270973470, 1740750181, 949783574, 2764214055, 0, 
        0}}, <No data fields>}, 
  vMerkleBranch = {<std::_Vector_base<uint256, std::allocator<uint256> >> = {
      _M_impl = {<std::allocator<uint256>> = {<__gnu_cxx::new_allocator<uint256>> = {<No data fields>}, <No data fields>}, _M_start = 0x7fffaf4caaa0, 
        _M_finish = 0x7fffaf4cabe0, 
        _M_end_of_storage = 0x7fffaf4cabe0}}, <No data fields>}, 
  nIndex = 587, fMerkleVerified = true}
(gdb) p chainActive
$5 = {
  vChain = {<std::_Vector_base<CBlockIndex*, std::allocator<CBlockIndex*> >> = {
      _M_impl = {<std::allocator<CBlockIndex*>> = {<__gnu_cxx::new_allocator<CBlockIndex*>> = {<No data fields>}, <No data fields>}, _M_start = 0x7fff959fe900, 
        _M_finish = 0x7fff95c3f6c8, 
        _M_end_of_storage = 0x7fff95e80480}}, <No data fields>}}
(gdb) p chainActive.Contains(pindex)
Cannot evaluate function -- may be inlined
(gdb) p pindex->nHeight
$6 = 278865
(gdb) p (*chainActive)[278865]
Cannot resolve function operator* to any overloaded instance
(gdb) p (chainActive)[278865]
Could not find operator[].
(gdb) p chainActive[278865]
Could not find operator[].
(gdb) p *chainActive[278865]
Could not find operator[].
(gdb) p chainActive.vChain
$7 = {<std::_Vector_base<CBlockIndex*, std::allocator<CBlockIndex*> >> = {
    _M_impl = {<std::allocator<CBlockIndex*>> = {<__gnu_cxx::new_allocator<CBlockIndex*>> = {<No data fields>}, <No data fields>}, 
      _M_start = 0x7fff959fe900, _M_finish = 0x7fff95c3f6c8, 
      _M_end_of_storage = 0x7fff95e80480}}, <No data fields>}
(gdb) p chainActive.vChain[278865]
Could not find operator[].
(gdb) p *(chainActive.vChain)
Cannot resolve function operator* to any overloaded instance
(gdb) p chainActive.vChain.size()
Cannot evaluate function -- may be inlined
(gdb) p chainActive.vChain->size()
Cannot resolve method std::vector<CBlockIndex*, std::allocator<CBlockIndex*> >::size to any overloaded instance
(gdb) p chainActive.vChain
$8 = {<std::_Vector_base<CBlockIndex*, std::allocator<CBlockIndex*> >> = {
    _M_impl = {<std::allocator<CBlockIndex*>> = {<__gnu_cxx::new_allocator<CBlockIndex*>> = {<No data fields>}, <No data fields>}, 
      _M_start = 0x7fff959fe900, _M_finish = 0x7fff95c3f6c8, 
      _M_end_of_storage = 0x7fff95e80480}}, <No data fields>}
(gdb) p chainActive.vChain.empty()
Cannot evaluate function -- may be inlined
  10. laanwj added this to the milestone 0.9.2 on Apr 12, 2014
  11. laanwj commented at 1:53 PM on April 12, 2014: member

    Someone on IRC also reported crashes at startup with 0.9.0, at the first updateTip with a large wallet. No traceback, but is likely to be the same issue.

  12. laanwj commented at 2:05 PM on April 12, 2014: member

    @dooglus as you're compiling the source yourself, can you try replacing line 196 in transactiontablemodel.cpp from:

    LOCK(wallet->cs_wallet);

    to

    LOCK2(cs_main, wallet->cs_wallet);

    I'm fairly confident that that will solve this race. (if this works, I'm going to add lockheld assertions over the place to see if there are other cases where, indirectly, main chainActive.* is called without the main lock)

  13. dooglus commented at 4:40 PM on April 12, 2014: contributor

    I sure can.

    Should I just quit the previous bitcoin-qt? It's currently at a (gdb) prompt waiting for you to give me stuff to ask it.

    Also, note that it took me maybe 10 attempts yesterday to get the crash to happen, so if it doesn't happen today it doesn't necessarily mean the bug is fixed.

  14. laanwj commented at 4:47 PM on April 12, 2014: member

    Yes, let's quit and re-try. I wouldn't know how to check this out on the gdb prompt. Likely the chainActive data structure is inconsistent in some way, but it takes knowledge about the internal working of libstdc++ to find out exactly what. It's nothing obvious, and if gdb hasn't optimized all the debugging info away...

  15. dooglus commented at 5:05 PM on April 12, 2014: contributor

    I made the change you suggested, rebuilt, and it crashed the very first time I ran it.

    The backtrace is a little different. Whereas before, CWalletTx::IsTrusted (wallet.h:665, #3 in the trace) was being called from TransactionRecord::updateStatus (transactionrecord.cpp:167), now it is called from CWallet::GetBalance (walletmodel.cpp:66)

    Here's the new backtrace:

    [#0](/bitcoin-bitcoin/0/)  CMerkleTx::GetDepthInMainChainINTERNAL (this=this@entry=0x7fffb198b8f0, 
    pindexRet=@0x7fffffffd628: 0x7fffda16cd90) at main.cpp:884
[#1](/bitcoin-bitcoin/1/)  0x00005555556d2c94 in CMerkleTx::GetDepthInMainChain (
    this=this@entry=0x7fffb198b8f0, pindexRet=@0x7fffffffd628: 0x7fffda16cd90)
    at main.cpp:901
[#2](/bitcoin-bitcoin/2/)  0x0000555555676f0d in GetDepthInMainChain (this=0x7fffb198b8f0)
    at ../../src/main.h:466
[#3](/bitcoin-bitcoin/3/)  CWalletTx::IsTrusted (this=0x7fffb198b8f0) at ../../src/wallet.h:665
[#4](/bitcoin-bitcoin/4/)  0x00005555557bbebc in CWallet::GetBalance (this=0x7fffb0f3ec80)
    at wallet.cpp:966
[#5](/bitcoin-bitcoin/5/)  0x000055555560bf19 in WalletModel::getBalance (this=this@entry=
    0x5555567590b0, coinControl=coinControl@entry=0x0) at walletmodel.cpp:66
[#6](/bitcoin-bitcoin/6/)  0x000055555564ce91 in SendCoinsDialog::setModel (this=0x5555571d7850, 
    model=model@entry=0x5555567590b0) at sendcoinsdialog.cpp:93
[#7](/bitcoin-bitcoin/7/)  0x000055555561ebb1 in WalletView::setWalletModel (this=0x5555571982a0, 
    walletModel=0x5555567590b0) at walletview.cpp:113
[#8](/bitcoin-bitcoin/8/)  0x000055555560ae17 in WalletFrame::addWallet (this=0x555556279540, 
    name=..., walletModel=walletModel@entry=0x5555567590b0)
    at walletframe.cpp:48
[#9](/bitcoin-bitcoin/9/)  0x00005555555d0129 in BitcoinGUI::addWallet (this=<optimized out>, 
    name=..., walletModel=walletModel@entry=0x5555567590b0)
    at bitcoingui.cpp:428
[#10](/bitcoin-bitcoin/10/) 0x00005555555bf4d7 in BitcoinApplication::initializeResult (
    this=this@entry=0x7fffffffe2e0, retval=1) at bitcoin.cpp:400
[#11](/bitcoin-bitcoin/11/) 0x00005555555bf88f in BitcoinApplication::qt_static_metacall (
    _o=0x7fffffffe2e0, _id=<optimized out>, _a=0x7fffb02f0170, 
    _c=<optimized out>) at bitcoin.moc:169
[#12](/bitcoin-bitcoin/12/) 0x00007ffff61f3dce in QObject::event(QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#13](/bitcoin-bitcoin/13/) 0x00007ffff6a58ea3 in QApplication::event(QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#14](/bitcoin-bitcoin/14/) 0x00007ffff6a53dfc in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#15](/bitcoin-bitcoin/15/) 0x00007ffff6a5a470 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#16](/bitcoin-bitcoin/16/) 0x00007ffff61db8bd in QCoreApplication::notifyInternal(QObject*, QEvent*)
    () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#17](/bitcoin-bitcoin/17/) 0x00007ffff61dee1f in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
  16. laanwj commented at 3:24 PM on April 13, 2014: member

    It is promising that the crash now happens in another place where the same problem exists.

    Would be safest to replace all places where the GUI locks the wallet by LOCK2(main ,wallet).

    Edit: hm, changing in it the GUI is not enough: CWallet::GetBalance itself locks the wallet, but not main, it should be changed as well...

  17. laanwj referenced this in commit b45e335dbf on Apr 15, 2014
  18. laanwj commented at 3:43 PM on April 15, 2014: member

    #4058 likely solves this issue

  19. laanwj referenced this in commit 55a1db4fa2 on Apr 18, 2014
  20. laanwj closed this on Apr 22, 2014
  21. dooglus commented at 7:03 PM on April 23, 2014: contributor

    I'm still running the version that you mentioned 8 days ago:

    #4058 likely solves this issue

    and just had it lock up:

    chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
5735c6b22004430f5e2f62933153b5a821417389b89fcdf7bf3c50f4a8a01fd8
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
01d409dac77df5a9ba0e6793dac1e77109d23985eec0d68a638190862645a1c9
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
error: {"code":-4,"message":"Insufficient funds"}
chris@chris:~/.bitcoin$ bc listunspent
[lots of output]
chris@chris:~/.bitcoin$ bc listunspent
[no output, just hung]

    The end of debug.log shows:

    2014-04-23 19:00:50 ERROR: AcceptToMemoryPool : inputs already spent
2014-04-23 19:02:11 ResendWalletTransactions()
2014-04-23 19:02:11 Relaying wtx 0f2b67658e43d518ad37e2d569fcbc414d90487f2ed7e10b442f8c9a2bee9b63
2014-04-23 19:02:11 Relaying wtx 3cc79db066abc7a5582669fbd5b0c855975c4f2537ece52370c2c6eb099f5b85
2014-04-23 19:02:11 Relaying wtx d416993c56c2e7d9e753f902ce04ce1b8c31b8cfc14fe8e17603b36cdfadeedc

    I don't know if that means we're acquiring too many locks now, and trying to acquire the same one twice, causing a deadlock.

    I hit control-c in the gdb window, and got:

    Program received signal SIGINT, Interrupt.
__lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) where
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x0000555555664d54 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=196, 
    pszFile=0x555555afb29d "transactiontablemodel.cpp", 
    pszName=<optimized out>, mutexIn=..., this=<synthetic pointer>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  TransactionTablePriv::index (this=0x5555566c70d0, idx=idx@entry=23040)
    at transactiontablemodel.cpp:196
[#9](/bitcoin-bitcoin/9/)  0x0000555555663a70 in TransactionTableModel::index (this=0x555556859770, 
    row=23040, column=0, parent=...) at transactiontablemodel.cpp:615
[#10](/bitcoin-bitcoin/10/) 0x0000555555660d5d in TransactionFilterProxy::filterAcceptsRow (this=
    0x555557aad020, sourceRow=23040, sourceParent=...)
    at transactionfilterproxy.cpp:33

    I ran a 'where' on all threads, and got this. Hopefully it will tell you why it's frozen:

    (gdb) thread apply all where

Thread 25 (Thread 0x7fff7d799700 (LWP 6128)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x0000555555849bfb in BGThread (this=0x7fffc81614e0)
    at util/env_posix.cc:563
[#2](/bitcoin-bitcoin/2/)  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (
    arg=0x7fffc81614e0) at util/env_posix.cc:510
[#3](/bitcoin-bitcoin/3/)  0x00007ffff58c9182 in start_thread (arg=0x7fff7d799700)
    at pthread_create.c:312
[#4](/bitcoin-bitcoin/4/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 24 (Thread 0x7fff82da3700 (LWP 6126)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x00005555557e19d9 in MilliSleep (n=500) at util.h:97
[#6](/bitcoin-bitcoin/6/)  ThreadFlushWalletDB (strFile=...) at walletdb.cpp:782
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fff82da3700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 23 (Thread 0x7fff8b7ae700 (LWP 6125)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555573b1a6 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=852, 
    pszFile=0x555555b4f980 "rpcserver.cpp", pszName=<optimized out>, 
    mutexIn=..., this=<synthetic pointer>) at sync.h:139
[#8](/bitcoin-bitcoin/8/)  CRPCTable::execute (this=this@entry=0x555555ebd9e0 <tableRPC>, 
    strMethod=..., params=...) at rpcserver.cpp:852
[#9](/bitcoin-bitcoin/9/)  0x000055555573dd78 in ServiceConnection (conn=conn@entry=0x7fff9c0018f0)
    at rpcserver.cpp:797
[#10](/bitcoin-bitcoin/10/) 0x000055555573eb50 in RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > (acceptor=..., context=..., 
    fUseSSL=<optimized out>, conn=0x7fff9c0018f0, error=...)
    at rpcserver.cpp:495
[#11](/bitcoin-bitcoin/11/) 0x000055555574383a in operator()<void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, const boost::system::error_code&), boost::_bi::list1<const boost::system::error_code&> > (
    f=@0x7fff8b7adbf0: 0x55555573e940 <RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> >(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&)>, a=<synthetic pointer>, 
    this=0x7fff8b7adbf8) at /usr/include/boost/bind/bind.hpp:525
[#12](/bitcoin-bitcoin/12/) operator()<boost::system::error_code> (a1=..., this=0x7fff8b7adbf0)
    at /usr/include/boost/bind/bind_template.hpp:47
[#13](/bitcoin-bitcoin/13/) operator() (this=0x7fff8b7adbf0)
    at /usr/include/boost/asio/detail/bind_handler.hpp:47
[#14](/bitcoin-bitcoin/14/) asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code> > (
    function=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
[#15](/bitcoin-bitcoin/15/) invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
[#16](/bitcoin-bitcoin/16/) boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ip::tcp, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > >::do_complete (owner=0x7fffa91a1c30, base=<optimized out>)
    at /usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:123
[#17](/bitcoin-bitcoin/17/) 0x000055555574518d in complete (bytes_transferred=0, ec=..., owner=..., 
    this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#18](/bitcoin-bitcoin/18/) boost::asio::detail::epoll_reactor::descriptor_state::do_complete (
    owner=0x7fffa91a1c30, base=0x7fffa91b7030, ec=..., 
    bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
[#19](/bitcoin-bitcoin/19/) 0x000055555574678e in complete (bytes_transferred=1, ec=..., owner=..., 
    this=0x7fffa91b7030)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#20](/bitcoin-bitcoin/20/) do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa91a1c30)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
[#21](/bitcoin-bitcoin/21/) boost::asio::detail::task_io_service::run (this=0x7fffa91a1c30, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#22](/bitcoin-bitcoin/22/) 0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf8dda10)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#23](/bitcoin-bitcoin/23/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#24](/bitcoin-bitcoin/24/) 0x00007ffff58c9182 in start_thread (arg=0x7fff8b7ae700)
    at pthread_create.c:312
[#25](/bitcoin-bitcoin/25/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 22 (Thread 0x7fff8bfff700 (LWP 6124)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555573b1a6 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=852, 
    pszFile=0x555555b4f980 "rpcserver.cpp", pszName=<optimized out>, 
    mutexIn=..., this=<synthetic pointer>) at sync.h:139
[#8](/bitcoin-bitcoin/8/)  CRPCTable::execute (this=this@entry=0x555555ebd9e0 <tableRPC>, 
    strMethod=..., params=...) at rpcserver.cpp:852
[#9](/bitcoin-bitcoin/9/)  0x000055555573dd78 in ServiceConnection (conn=conn@entry=0x7fff9c01e030)
    at rpcserver.cpp:797
[#10](/bitcoin-bitcoin/10/) 0x000055555573eb50 in RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > (acceptor=..., context=..., 
    fUseSSL=<optimized out>, conn=0x7fff9c01e030, error=...)
    at rpcserver.cpp:495
[#11](/bitcoin-bitcoin/11/) 0x000055555574383a in operator()<void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, const boost::system::error_code&), boost::_bi::list1<const boost::system::error_code&> > (
    f=@0x7fff8bffebf0: 0x55555573e940 <RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> >(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&)>, a=<synthetic pointer>, 
    this=0x7fff8bffebf8) at /usr/include/boost/bind/bind.hpp:525
[#12](/bitcoin-bitcoin/12/) operator()<boost::system::error_code> (a1=..., this=0x7fff8bffebf0)
    at /usr/include/boost/bind/bind_template.hpp:47
[#13](/bitcoin-bitcoin/13/) operator() (this=0x7fff8bffebf0)
    at /usr/include/boost/asio/detail/bind_handler.hpp:47
[#14](/bitcoin-bitcoin/14/) asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code> > (
    function=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
[#15](/bitcoin-bitcoin/15/) invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
[#16](/bitcoin-bitcoin/16/) boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ip::tcp, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > >::do_complete (owner=0x7fffa91a1c30, base=<optimized out>)
    at /usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:123
[#17](/bitcoin-bitcoin/17/) 0x000055555574518d in complete (bytes_transferred=0, ec=..., owner=..., 
    this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#18](/bitcoin-bitcoin/18/) boost::asio::detail::epoll_reactor::descriptor_state::do_complete (
    owner=0x7fffa91a1c30, base=0x7fffa91b7030, ec=..., 
    bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
[#19](/bitcoin-bitcoin/19/) 0x000055555574678e in complete (bytes_transferred=1, ec=..., owner=..., 
    this=0x7fffa91b7030)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#20](/bitcoin-bitcoin/20/) do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa91a1c30)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
[#21](/bitcoin-bitcoin/21/) boost::asio::detail::task_io_service::run (this=0x7fffa91a1c30, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#22](/bitcoin-bitcoin/22/) 0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf8dda10)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#23](/bitcoin-bitcoin/23/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#24](/bitcoin-bitcoin/24/) 0x00007ffff58c9182 in start_thread (arg=0x7fff8bfff700)
    at pthread_create.c:312
[#25](/bitcoin-bitcoin/25/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 21 (Thread 0x7fffa0c13700 (LWP 6123)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555573b1a6 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=852, 
    pszFile=0x555555b4f980 "rpcserver.cpp", pszName=<optimized out>, 
    mutexIn=..., this=<synthetic pointer>) at sync.h:139
[#8](/bitcoin-bitcoin/8/)  CRPCTable::execute (this=this@entry=0x555555ebd9e0 <tableRPC>, 
    strMethod=..., params=...) at rpcserver.cpp:852
[#9](/bitcoin-bitcoin/9/)  0x000055555573dd78 in ServiceConnection (conn=conn@entry=0x7fff74000960)
    at rpcserver.cpp:797
[#10](/bitcoin-bitcoin/10/) 0x000055555573eb50 in RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > (acceptor=..., context=..., 
    fUseSSL=<optimized out>, conn=0x7fff74000960, error=...)
    at rpcserver.cpp:495
[#11](/bitcoin-bitcoin/11/) 0x000055555574383a in operator()<void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, const boost::system::error_code&), boost::_bi::list1<const boost::system::error_code&> > (
    f=@0x7fffa0c12bf0: 0x55555573e940 <RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> >(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&)>, a=<synthetic pointer>, 
    this=0x7fffa0c12bf8) at /usr/include/boost/bind/bind.hpp:525
[#12](/bitcoin-bitcoin/12/) operator()<boost::system::error_code> (a1=..., this=0x7fffa0c12bf0)
    at /usr/include/boost/bind/bind_template.hpp:47
[#13](/bitcoin-bitcoin/13/) operator() (this=0x7fffa0c12bf0)
    at /usr/include/boost/asio/detail/bind_handler.hpp:47
[#14](/bitcoin-bitcoin/14/) asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code> > (
    function=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
[#15](/bitcoin-bitcoin/15/) invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
[#16](/bitcoin-bitcoin/16/) boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ip::tcp, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > >::do_complete (owner=0x7fffa91a1c30, base=<optimized out>)
    at /usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:123
[#17](/bitcoin-bitcoin/17/) 0x000055555574518d in complete (bytes_transferred=0, ec=..., owner=..., 
    this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#18](/bitcoin-bitcoin/18/) boost::asio::detail::epoll_reactor::descriptor_state::do_complete (
    owner=0x7fffa91a1c30, base=0x7fffa91b7030, ec=..., 
    bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
[#19](/bitcoin-bitcoin/19/) 0x000055555574678e in complete (bytes_transferred=1, ec=..., owner=..., 
    this=0x7fffa91b7030)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#20](/bitcoin-bitcoin/20/) do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa91a1c30)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
[#21](/bitcoin-bitcoin/21/) boost::asio::detail::task_io_service::run (this=0x7fffa91a1c30, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#22](/bitcoin-bitcoin/22/) 0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf8dda10)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#23](/bitcoin-bitcoin/23/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#24](/bitcoin-bitcoin/24/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa0c13700)
    at pthread_create.c:312
[#25](/bitcoin-bitcoin/25/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 20 (Thread 0x7fffa166a700 (LWP 6122)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555573b1a6 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=852, 
    pszFile=0x555555b4f980 "rpcserver.cpp", pszName=<optimized out>, 
    mutexIn=..., this=<synthetic pointer>) at sync.h:139
[#8](/bitcoin-bitcoin/8/)  CRPCTable::execute (this=this@entry=0x555555ebd9e0 <tableRPC>, 
    strMethod=..., params=...) at rpcserver.cpp:852
[#9](/bitcoin-bitcoin/9/)  0x000055555573dd78 in ServiceConnection (conn=conn@entry=0x7fff700149c0)
    at rpcserver.cpp:797
[#10](/bitcoin-bitcoin/10/) 0x000055555573eb50 in RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > (acceptor=..., context=..., 
    fUseSSL=<optimized out>, conn=0x7fff700149c0, error=...)
    at rpcserver.cpp:495
[#11](/bitcoin-bitcoin/11/) 0x000055555574383a in operator()<void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, const boost::system::error_code&), boost::_bi::list1<const boost::system::error_code&> > (
    f=@0x7fffa1669bf0: 0x55555573e940 <RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> >(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&)>, a=<synthetic pointer>, 
    this=0x7fffa1669bf8) at /usr/include/boost/bind/bind.hpp:525
[#12](/bitcoin-bitcoin/12/) operator()<boost::system::error_code> (a1=..., this=0x7fffa1669bf0)
    at /usr/include/boost/bind/bind_template.hpp:47
[#13](/bitcoin-bitcoin/13/) operator() (this=0x7fffa1669bf0)
    at /usr/include/boost/asio/detail/bind_handler.hpp:47
[#14](/bitcoin-bitcoin/14/) asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code> > (
    function=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
[#15](/bitcoin-bitcoin/15/) invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
[#16](/bitcoin-bitcoin/16/) boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ip::tcp, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > >::do_complete (owner=0x7fffa91a1c30, base=<optimized out>)
    at /usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:123
[#17](/bitcoin-bitcoin/17/) 0x000055555574518d in complete (bytes_transferred=0, ec=..., owner=..., 
    this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#18](/bitcoin-bitcoin/18/) boost::asio::detail::epoll_reactor::descriptor_state::do_complete (
    owner=0x7fffa91a1c30, base=0x7fffa91b7030, ec=..., 
    bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
[#19](/bitcoin-bitcoin/19/) 0x000055555574678e in complete (bytes_transferred=1, ec=..., owner=..., 
    this=0x7fffa91b7030)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#20](/bitcoin-bitcoin/20/) do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa91a1c30)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
[#21](/bitcoin-bitcoin/21/) boost::asio::detail::task_io_service::run (this=0x7fffa91a1c30, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#22](/bitcoin-bitcoin/22/) 0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf8dda10)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#23](/bitcoin-bitcoin/23/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#24](/bitcoin-bitcoin/24/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa166a700)
    at pthread_create.c:312
[#25](/bitcoin-bitcoin/25/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 19 (Thread 0x7fffa1ebb700 (LWP 6121)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x000055555571e28d in MilliSleep (n=900000) at util.h:97
[#6](/bitcoin-bitcoin/6/)  LoopForever<void (*)()> (name=0x555555b38008 "dumpaddr", 
    func=0x555555711a70 <DumpAddresses()>, msecs=900000) at util.h:550
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa1ebb700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 18 (Thread 0x7fffa270c700 (LWP 6120)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (mutex=0x7fffad215600)
    at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555561c0ad in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=0x7fffa270aa60) at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=0x7fffa270aa60) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock<AnnotatedMixin<boost::recursive_mutex> >::CMutexLock (
    this=0x7fffa270aa60, mutexIn=..., pszName=<optimized out>, 
    pszFile=<optimized out>, nLine=<optimized out>, fTry=<optimized out>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  0x00005555557ce266 in CWallet::SyncTransaction (this=0x7fffad2154b0, 
    hash=..., tx=..., pblock=0x0) at wallet.cpp:609
[#9](/bitcoin-bitcoin/9/)  0x00005555556ff0d5 in operator() (a2=<optimized out>, a1=..., a0=..., 
    this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
[#10](/bitcoin-bitcoin/10/) m_invoke (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:363
[#11](/bitcoin-bitcoin/11/) operator() (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:345
[#12](/bitcoin-bitcoin/12/) dereference (this=0x7fffa270ab60)
    at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
[#13](/bitcoin-bitcoin/13/) dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal3_impl<void, const uint256&, const CTransaction&, const CBlock*, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&, const CTransaction&, const CBlock*)>, boost::function<void(const boost::signals2::connection&, const uint256&, const CTransaction&, const CBlock*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot3<void, const uint256&, const CTransaction&, const CBlock*, boost::function<void(const uint256&, const CTransaction&, const CBlock*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot3<void, const uint256&, const CTransaction&, const CBlock*, boost::function<void(const uint256&, const CTransaction&, const CBlock*)> >, boost::signals2::mutex> > > (f=...)
    at /usr/include/boost/iterator/iterator_facade.hpp:514
[#14](/bitcoin-bitcoin/14/) operator* (this=0x7fffa270ab60)
    at /usr/include/boost/iterator/iterator_facade.hpp:639
[#15](/bitcoin-bitcoin/15/) operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal3_impl<void, const uint256&, const CTransaction&, const CBlock*, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&, const CTransaction&, const CBlock*)>, boost::function<void(const boost::signals2::connection&, const uint256&, const CTransaction&, const CBlock*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot3<void, const uint256&, const CTransaction&, const CBlock*, boost::function<void(const uint256&, const CTransaction&, const CBlock*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot3<void, const uint256&, const CTransaction&, const CBlock*, boost::function<void(const uint256&, const CTransaction&, const CBlock*)> >, boost::signals2::mutex> > > (
    this=<optimized out>, first=..., last=...)
    at /usr/include/boost/signals2/optional_last_value.hpp:55
[#16](/bitcoin-bitcoin/16/) operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal3_impl<void, const uint256&, const CTransaction&, const CBlock*, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&, const CTransaction&, const CBlock*)>, boost::function<void(const boost::signals2::connection&, const uint256&, const CTransaction&, const CBlock*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot3<void, const uint256&, const CTransaction&, const CBlock*, boost::function<void(const uint256&, const CTransaction&, const CBlock*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot3<void, const uint256&, const CTransaction&, const CBlock*, boost::function<void(const uint256&, const CTransaction&, const CBlock*)> >, boost::signals2::mutex> > > (this=<optimized out>, combiner=..., first=..., 
    last=...) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
[#17](/bitcoin-bitcoin/17/) boost::signals2::detail::signal3_impl<void, uint256 const&, CTransaction const&, CBlock const*, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (uint256 const&, CTransaction const&, CBlock const*)>, boost::function<void (boost::signals2::connection const&, uint256 const&, CTransaction const&, CBlock const*)>, boost::signals2::mutex>::operator()(uint256 const&, CTransaction const&, CBlock const*) (this=<optimized out>, 
    arg1=..., arg2=..., arg3=arg3@entry=0x0)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
[#18](/bitcoin-bitcoin/18/) 0x00005555556c63dc in operator() (arg3=0x0, arg2=..., arg1=..., 
    this=0x555555ebce20 <(anonymous namespace)::g_signals>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
[#19](/bitcoin-bitcoin/19/) AcceptToMemoryPool (pool=..., state=..., tx=..., 
    fLimitFree=fLimitFree@entry=true, 
    pfMissingInputs=pfMissingInputs@entry=0x7fffa270b0e5, 
    fRejectInsaneFee=fRejectInsaneFee@entry=false) at main.cpp:955
[#20](/bitcoin-bitcoin/20/) 0x00005555556cdae1 in ProcessMessage (pfrom=pfrom@entry=0x7fff8c002b00, 
    strCommand=..., vRecv=...) at main.cpp:3721
[#21](/bitcoin-bitcoin/21/) 0x00005555556cfde2 in ProcessMessages (pfrom=0x7fff8c002b00)
    at main.cpp:4124
[#22](/bitcoin-bitcoin/22/) 0x000055555572b524 in operator() (a0=<optimized out>, 
    this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
[#23](/bitcoin-bitcoin/23/) m_invoke (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:368
[#24](/bitcoin-bitcoin/24/) operator() (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:345
[#25](/bitcoin-bitcoin/25/) dereference (this=0x7fffa270bb30)
    at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
[#26](/bitcoin-bitcoin/26/) dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (
    f=...) at /usr/include/boost/iterator/iterator_facade.hpp:514
[#27](/bitcoin-bitcoin/27/) operator* (this=0x7fffa270bb30)
    at /usr/include/boost/iterator/iterator_facade.hpp:639
[#28](/bitcoin-bitcoin/28/) operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (
    last=..., first=..., this=<optimized out>)
    at /usr/include/boost/signals2/optional_last_value.hpp:34
[#29](/bitcoin-bitcoin/29/) operator()<boost::signals2::optional_last_value<bool>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (this=<optimized out>, last=..., 
    first=..., combiner=...)
    at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:53
[#30](/bitcoin-bitcoin/30/) boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool (CNode*)>, boost::function<bool (boost::signals2::connection const&, CNode*)>, boost::signals2::mutex>::operator()(CNode*) (this=<optimized out>, 
    arg1=arg1@entry=0x7fff8c002b00)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
[#31](/bitcoin-bitcoin/31/) 0x0000555555716836 in operator() (arg1=0x7fff8c002b00, 
    this=0x555555ebd738 <g_signals+24>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
[#32](/bitcoin-bitcoin/32/) ThreadMessageHandler () at net.cpp:1544
[#33](/bitcoin-bitcoin/33/) 0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b38000 "msghand", 
    func=0x555555716620 <ThreadMessageHandler()>) at util.h:576
[#34](/bitcoin-bitcoin/34/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#35](/bitcoin-bitcoin/35/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa270c700)
    at pthread_create.c:312
[#36](/bitcoin-bitcoin/36/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7fffa2f5d700 (LWP 6119)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x00005555556b1611 in lock (this=0x555555ebcb80 <cs_main>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=0x555555ebcb80 <cs_main>) at sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=237, pszFile=0x555555b1a81e "main.cpp", 
    pszName=0x555555880e59 "cs_main", mutexIn=..., this=<synthetic pointer>)
    at sync.h:139
[#8](/bitcoin-bitcoin/8/)  (anonymous namespace)::GetHeight () at main.cpp:237
[#9](/bitcoin-bitcoin/9/)  0x0000555555729479 in operator() (this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
[#10](/bitcoin-bitcoin/10/) m_invoke (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:368
[#11](/bitcoin-bitcoin/11/) operator() (this=<optimized out>, connectionBody=...)
    at /usr/include/boost/signals2/detail/signal_template.hpp:345
[#12](/bitcoin-bitcoin/12/) dereference (this=0x7fffa2f5c350)
    at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
[#13](/bitcoin-bitcoin/13/) dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal0_impl<int, boost::signals2::optional_last_value<int>, int, std::less<int>, boost::function<int()>, boost::function<int(const boost::signals2::connection&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot0<int, boost::function<int()> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot0<int, boost::function<int()> >, boost::signals2::mutex> > > (f=...) at /usr/include/boost/iterator/iterator_facade.hpp:514
[#14](/bitcoin-bitcoin/14/) operator* (this=0x7fffa2f5c350)
    at /usr/include/boost/iterator/iterator_facade.hpp:639
[#15](/bitcoin-bitcoin/15/) operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal0_impl<int, boost::signals2::optional_last_value<int>, int, std::less<int>, boost::function<int()>, boost::function<int(const boost::signals2::connection&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot0<int, boost::function<int()> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot0<int, boost::function<int()> >, boost::signals2::mutex> > > (last=..., first=..., this=<optimized out>)
    at /usr/include/boost/signals2/optional_last_value.hpp:34
[#16](/bitcoin-bitcoin/16/) operator()<boost::signals2::optional_last_value<int>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal0_impl<int, boost::signals2::optional_last_value<int>, int, std::less<int>, boost::function<int()>, boost::function<int(const boost::signals2::connection&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot0<int, boost::function<int()> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot0<int, boost::function<int()> >, boost::signals2::mutex> > > (
    this=<optimized out>, last=..., first=..., combiner=...)
    at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:53
[#17](/bitcoin-bitcoin/17/) boost::signals2::detail::signal0_impl<int, boost::signals2::optional_last_value<int>, int, std::less<int>, boost::function<int ()>, boost::function<int (boost::signals2::connection const&)>, boost::signals2::mutex>::operator()() (
    this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
[#18](/bitcoin-bitcoin/18/) 0x00005555557122e7 in operator() (this=0x555555ebd720 <g_signals>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
[#19](/bitcoin-bitcoin/19/) CNode::PushVersion (this=this@entry=0x7fff8c004120) at net.cpp:548
[#20](/bitcoin-bitcoin/20/) 0x000055555572a1f2 in CNode::CNode (this=0x7fff8c004120, 
    hSocketIn=<optimized out>, addrIn=..., addrNameIn=..., 
    fInboundIn=<optimized out>) at net.h:312
[#21](/bitcoin-bitcoin/21/) 0x0000555555712c6e in ConnectNode (addrConnect=..., 
    pszDest=pszDest@entry=0x0) at net.cpp:504
[#22](/bitcoin-bitcoin/22/) 0x00005555557131c2 in OpenNetworkConnection (addrConnect=..., 
    grantOutbound=grantOutbound@entry=0x7fffa2f5ccc0, 
    strDest=strDest@entry=0x0, fOneShot=fOneShot@entry=false) at net.cpp:1457
[#23](/bitcoin-bitcoin/23/) 0x0000555555713f34 in ThreadOpenConnections () at net.cpp:1366
[#24](/bitcoin-bitcoin/24/) 0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b37ff8 "opencon", 
    func=0x555555713640 <ThreadOpenConnections()>) at util.h:576
[#25](/bitcoin-bitcoin/25/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#26](/bitcoin-bitcoin/26/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa2f5d700)
    at pthread_create.c:312
[#27](/bitcoin-bitcoin/27/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7fffa37ae700 (LWP 6118)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x0000555555714773 in MilliSleep (n=120000) at util.h:97
[#6](/bitcoin-bitcoin/6/)  ThreadOpenAddedConnections () at net.cpp:1438
[#7](/bitcoin-bitcoin/7/)  0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b37ff1 "addcon", 
    func=0x555555713f80 <ThreadOpenAddedConnections()>) at util.h:576
[#8](/bitcoin-bitcoin/8/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#9](/bitcoin-bitcoin/9/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa37ae700)
    at pthread_create.c:312
[#10](/bitcoin-bitcoin/10/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7fffa3fff700 (LWP 6117)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43d0c33 in select () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x0000555555715b6e in ThreadSocketHandler () at net.cpp:896
[#2](/bitcoin-bitcoin/2/)  0x000055555571e373 in TraceThread<void (*)()> (name=0x555555b06813 "net", 
    func=0x5555557151e0 <ThreadSocketHandler()>) at util.h:576
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa3fff700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7fffc118d700 (LWP 6095)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffc118d700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7fffc19de700 (LWP 6094)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffc19de700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7fffc222f700 (LWP 6093)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffc222f700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7fffe2f61700 (LWP 6087)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff62067be in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#4](/bitcoin-bitcoin/4/)  0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#5](/bitcoin-bitcoin/5/)  0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#6](/bitcoin-bitcoin/6/)  0x00007ffff60d4c5f in QThread::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#7](/bitcoin-bitcoin/7/)  0x00007ffff60d732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe2f61700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7fffe1ebf700 (LWP 6086)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#4](/bitcoin-bitcoin/4/)  0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#5](/bitcoin-bitcoin/5/)  0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#6](/bitcoin-bitcoin/6/)  0x00007ffff60d4c5f in QThread::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#7](/bitcoin-bitcoin/7/)  0x00007ffff60d732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe1ebf700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fffe37b2700 (LWP 6082)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff39c3129 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe37b2700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fffe95de700 (LWP 6081)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c330a in g_main_loop_run ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007fffed561e16 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe95de700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fffe9e2f700 (LWP 6080)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007fffe9e371ad in ?? ()
   from /usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe9e2f700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fc77c0 (LWP 6079)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x0000555555664d54 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=196, 
    pszFile=0x555555afb29d "transactiontablemodel.cpp", 
    pszName=<optimized out>, mutexIn=..., this=<synthetic pointer>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  TransactionTablePriv::index (this=0x5555566c70d0, idx=idx@entry=23040)
    at transactiontablemodel.cpp:196
[#9](/bitcoin-bitcoin/9/)  0x0000555555663a70 in TransactionTableModel::index (this=0x555556859770, 
    row=23040, column=0, parent=...) at transactiontablemodel.cpp:615
[#10](/bitcoin-bitcoin/10/) 0x0000555555660d5d in TransactionFilterProxy::filterAcceptsRow (
    this=0x555557aad020, sourceRow=23040, sourceParent=...)
    at transactionfilterproxy.cpp:33
[#11](/bitcoin-bitcoin/11/) 0x00007ffff6fccfd6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#12](/bitcoin-bitcoin/12/) 0x00007ffff6fcd1cf in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#13](/bitcoin-bitcoin/13/) 0x00007ffff61ed87a in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#14](/bitcoin-bitcoin/14/) 0x00007ffff6237664 in QAbstractItemModel::rowsInserted(QModelIndex const&, int, int) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#15](/bitcoin-bitcoin/15/) 0x00007ffff61d35ce in QAbstractItemModel::endInsertRows() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#16](/bitcoin-bitcoin/16/) 0x00005555556659fc in TransactionTablePriv::updateWallet (
    this=0x5555566c70d0, hash=..., status=status@entry=0)
    at transactiontablemodel.cpp:156
[#17](/bitcoin-bitcoin/17/) 0x000055555566405d in TransactionTableModel::updateTransaction (
    this=0x555556859770, hash=..., status=0) at transactiontablemodel.cpp:255
[#18](/bitcoin-bitcoin/18/) 0x0000555555614566 in WalletModel::updateTransaction (
    this=this@entry=0x55555617d5c0, hash=..., status=<optimized out>)
    at walletmodel.cpp:127
[#19](/bitcoin-bitcoin/19/) 0x000055555562cfdc in WalletModel::qt_static_metacall (_o=0x55555617d5c0, 
    _c=<optimized out>, _id=<optimized out>, _a=0x7fff70017270)
    at moc_walletmodel.cpp:80
[#20](/bitcoin-bitcoin/20/) 0x00007ffff61f1c1e in QObject::event(QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#21](/bitcoin-bitcoin/21/) 0x00007ffff6a52e2c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#22](/bitcoin-bitcoin/22/) 0x00007ffff6a594a0 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#23](/bitcoin-bitcoin/23/) 0x00007ffff61d94dd in QCoreApplication::notifyInternal(QObject*, QEvent*)
    () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#24](/bitcoin-bitcoin/24/) 0x00007ffff61dcb3d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#25](/bitcoin-bitcoin/25/) 0x00007ffff6206f83 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#26](/bitcoin-bitcoin/26/) 0x00007ffff39c2e04 in g_main_context_dispatch ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#27](/bitcoin-bitcoin/27/) 0x00007ffff39c3048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#28](/bitcoin-bitcoin/28/) 0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#29](/bitcoin-bitcoin/29/) 0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#30](/bitcoin-bitcoin/30/) 0x00007ffff6af4bb6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#31](/bitcoin-bitcoin/31/) 0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#32](/bitcoin-bitcoin/32/) 0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#33](/bitcoin-bitcoin/33/) 0x00007ffff61ddb79 in QCoreApplication::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#34](/bitcoin-bitcoin/34/) 0x00005555555b47e9 in main (argc=1, argv=<optimized out>)
    at bitcoin.cpp:580
  22. dooglus commented at 7:12 PM on April 23, 2014: contributor

    I restarted bitcoind, and pretty quickly it hung again. This seems quite easily reproducible:

    chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
ed9fc3069b5db658d8be4a54c9707c228242da9f49aacb96e0299f115e24a993
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
f6a9c90d4c613cf0e76d41d012dc8277c8ebf5c6b83309c93d9087975704c946
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
5718101a513ba18d902f7a543590edd82dc0e0bd61d7327c2c16bd3a5544c913
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
f114730e2470d74a9289d454ae9726976f6fe6f565683ed00cb39e0a1c1a9b26
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
838a56f87e9bb3f8690b19268b3d83dd7311cc53858e5c10bb683416ee6cbfc8
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
41902e0c90cd155bad447904922fa359b5bf4d01ee32006c853d2121000e0fe0
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
67e29ed40704c79392d4d82fa619c78fc0f6aade31d2ee741f961e4597c7cb91
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 50
error: {"code":-4,"message":"Insufficient funds"}
chris@chris:~/.bitcoin$ bc listunspent 0
[too much output]
chris@chris:~/.bitcoin$ bc listunspent 0 | grep amount
        "amount" : 0.01000059,
        "amount" : 0.00090000,
        "amount" : 0.02000000,
        "amount" : 0.11016945,
        "amount" : 0.18438206,
        "amount" : 0.01390000,
        "amount" : 0.00047493,
        "amount" : 0.52000000,
        "amount" : 4.37022011,
        "amount" : 0.01900000,
        "amount" : 0.09990000,
        "amount" : 0.01406309,
        "amount" : 0.00259388,
        "amount" : 0.00016425,
        "amount" : 0.00049679,
        "amount" : 9.90000000,
chris@chris:~/.bitcoin$ bc sendtoaddress 16TBqpEFpivLtDRAtmNnCxEdJW1coodiia 14
[hung]

    debug.log:

    2014-04-23 19:03:58 keypool keep 14209
2014-04-23 19:03:58 AddToWallet 41902e0c90cd155bad447904922fa359b5bf4d01ee32006c853d2121000e0fe0  new
2014-04-23 19:03:58 AddToWallet 41902e0c90cd155bad447904922fa359b5bf4d01ee32006c853d2121000e0fe0  
2014-04-23 19:03:58 Relaying wtx 41902e0c90cd155bad447904922fa359b5bf4d01ee32006c853d2121000e0fe0
2014-04-23 19:04:03 keypool reserve 14210
2014-04-23 19:04:03 CommitTransaction:
CTransaction(hash=67e29ed407, ver=1, vin.size=3, vout.size=2, nLockTime=0)
    CTxIn(COutPoint(41902e0c90, 0), scriptSig=3045022100b41b59985e6945)
    CTxIn(COutPoint(5718101a51, 1), scriptSig=3045022100d9bd04c0806130)
    CTxIn(COutPoint(cbf9d5530d, 0), scriptSig=3044022003ddfdb6ae632423)
    CTxOut(nValue=4.37022011, scriptPubKey=OP_DUP OP_HASH160 0cbc367a1158)
    CTxOut(nValue=50.00000000, scriptPubKey=OP_DUP OP_HASH160 3bcc7e7aa19b)
2014-04-23 19:04:03 keypool keep 14210
2014-04-23 19:04:03 AddToWallet 67e29ed40704c79392d4d82fa619c78fc0f6aade31d2ee741f961e4597c7cb91  new
2014-04-23 19:04:03 AddToWallet 67e29ed40704c79392d4d82fa619c78fc0f6aade31d2ee741f961e4597c7cb91  
2014-04-23 19:04:03 Relaying wtx 67e29ed40704c79392d4d82fa619c78fc0f6aade31d2ee741f961e4597c7cb91

    gdb:

      C-c C-c
Program received signal SIGINT, Interrupt.
__lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) where
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x0000555555664d54 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=196, 
    pszFile=0x555555afb29d "transactiontablemodel.cpp", 
    pszName=<optimized out>, mutexIn=..., this=<synthetic pointer>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  TransactionTablePriv::index (this=0x555556613f20, idx=idx@entry=64625)
    at transactiontablemodel.cpp:196
[#9](/bitcoin-bitcoin/9/)  0x0000555555663a70 in TransactionTableModel::index (this=0x55555685d8f0, 
    row=64625, column=0, parent=...) at transactiontablemodel.cpp:615
[#10](/bitcoin-bitcoin/10/) 0x0000555555660d5d in TransactionFilterProxy::filterAcceptsRow (this=
    0x555557ab0890, sourceRow=64625, sourceParent=...)
    at transactionfilterproxy.cpp:33
[#11](/bitcoin-bitcoin/11/) 0x00007ffff6fccfd6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#12](/bitcoin-bitcoin/12/) 0x00007ffff6fcd1cf in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#13](/bitcoin-bitcoin/13/) 0x00007ffff61ed87a in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#14](/bitcoin-bitcoin/14/) 0x00007ffff6237664 in QAbstractItemModel::rowsInserted(QModelIndex const&, int, int) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#15](/bitcoin-bitcoin/15/) 0x00007ffff61d35ce in QAbstractItemModel::endInsertRows() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#16](/bitcoin-bitcoin/16/) 0x00005555556659fc in TransactionTablePriv::updateWallet (
    this=0x555556613f20, hash=..., status=status@entry=0)
    at transactiontablemodel.cpp:156
[#17](/bitcoin-bitcoin/17/) 0x000055555566405d in TransactionTableModel::updateTransaction (
    this=0x55555685d8f0, hash=..., status=0) at transactiontablemodel.cpp:255
[#18](/bitcoin-bitcoin/18/) 0x0000555555614566 in WalletModel::updateTransaction (
    this=this@entry=0x555556332bd0, hash=..., status=<optimized out>)
    at walletmodel.cpp:127
[#19](/bitcoin-bitcoin/19/) 0x000055555562cfdc in WalletModel::qt_static_metacall (_o=0x555556332bd0, 
    _c=<optimized out>, _id=<optimized out>, _a=0x7fff70017270)
    at moc_walletmodel.cpp:80
[#20](/bitcoin-bitcoin/20/) 0x00007ffff61f1c1e in QObject::event(QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#21](/bitcoin-bitcoin/21/) 0x00007ffff6a52e2c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#22](/bitcoin-bitcoin/22/) 0x00007ffff6a594a0 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#23](/bitcoin-bitcoin/23/) 0x00007ffff61d94dd in QCoreApplication::notifyInternal(QObject*, QEvent*)
    () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#24](/bitcoin-bitcoin/24/) 0x00007ffff61dcb3d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#25](/bitcoin-bitcoin/25/) 0x00007ffff6206f83 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#26](/bitcoin-bitcoin/26/) 0x00007ffff39c2e04 in g_main_context_dispatch ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#27](/bitcoin-bitcoin/27/) 0x00007ffff39c3048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#28](/bitcoin-bitcoin/28/) 0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#29](/bitcoin-bitcoin/29/) 0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#30](/bitcoin-bitcoin/30/) 0x00007ffff6af4bb6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#31](/bitcoin-bitcoin/31/) 0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#32](/bitcoin-bitcoin/32/) 0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#33](/bitcoin-bitcoin/33/) 0x00007ffff61ddb79 in QCoreApplication::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#34](/bitcoin-bitcoin/34/) 0x00005555555b47e9 in main (argc=1, argv=<optimized out>)
    at bitcoin.cpp:580
(gdb) thread apply all where

Thread 25 (Thread 0x7fff7fae6700 (LWP 6268)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x0000555555849bfb in BGThread (this=0x7fffd40e1ee0)
    at util/env_posix.cc:563
[#2](/bitcoin-bitcoin/2/)  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (
    arg=0x7fffd40e1ee0) at util/env_posix.cc:510
[#3](/bitcoin-bitcoin/3/)  0x00007ffff58c9182 in start_thread (arg=0x7fff7fae6700)
    at pthread_create.c:312
[#4](/bitcoin-bitcoin/4/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 24 (Thread 0x7fff855e3700 (LWP 6267)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x00005555557e19d9 in MilliSleep (n=500) at util.h:97
[#6](/bitcoin-bitcoin/6/)  ThreadFlushWalletDB (strFile=...) at walletdb.cpp:782
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fff855e3700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 23 (Thread 0x7fff87fff700 (LWP 6266)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43d99a3 in epoll_wait ()
    at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x000055555574258b in boost::asio::detail::epoll_reactor::run (
    this=0x7fffa91aefe0, block=<optimized out>, ops=...)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:392
[#2](/bitcoin-bitcoin/2/)  0x0000555555746551 in do_run_one (ec=..., this_thread=..., lock=..., 
    this=0x7fffa91ae190)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:368
[#3](/bitcoin-bitcoin/3/)  boost::asio::detail::task_io_service::run (this=0x7fffa91ae190, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#4](/bitcoin-bitcoin/4/)  0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf4f74f0)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#5](/bitcoin-bitcoin/5/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#6](/bitcoin-bitcoin/6/)  0x00007ffff58c9182 in start_thread (arg=0x7fff87fff700)
    at pthread_create.c:312
[#7](/bitcoin-bitcoin/7/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 22 (Thread 0x7fffa0a10700 (LWP 6265)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x000055555574671d in wait<boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex> > (lock=..., this=0x7fffa0a0fdb0)
    at /usr/include/boost/asio/detail/posix_event.hpp:80
[#2](/bitcoin-bitcoin/2/)  do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa91ae190)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:395
[#3](/bitcoin-bitcoin/3/)  boost::asio::detail::task_io_service::run (this=0x7fffa91ae190, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#4](/bitcoin-bitcoin/4/)  0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf4f74f0)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#5](/bitcoin-bitcoin/5/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#6](/bitcoin-bitcoin/6/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa0a10700)
    at pthread_create.c:312
[#7](/bitcoin-bitcoin/7/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 21 (Thread 0x7fffa1261700 (LWP 6264)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x000055555574671d in wait<boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex> > (lock=..., this=0x7fffa1260db0)
    at /usr/include/boost/asio/detail/posix_event.hpp:80
[#2](/bitcoin-bitcoin/2/)  do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa91ae190)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:395
[#3](/bitcoin-bitcoin/3/)  boost::asio::detail::task_io_service::run (this=0x7fffa91ae190, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#4](/bitcoin-bitcoin/4/)  0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf4f74f0)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#5](/bitcoin-bitcoin/5/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#6](/bitcoin-bitcoin/6/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa1261700)
    at pthread_create.c:312
[#7](/bitcoin-bitcoin/7/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 20 (Thread 0x7fffa1ab2700 (LWP 6263)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555573b1a6 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=852, 
    pszFile=0x555555b4f980 "rpcserver.cpp", pszName=<optimized out>, 
    mutexIn=..., this=<synthetic pointer>) at sync.h:139
[#8](/bitcoin-bitcoin/8/)  CRPCTable::execute (this=this@entry=0x555555ebd9e0 <tableRPC>, 
    strMethod=..., params=...) at rpcserver.cpp:852
[#9](/bitcoin-bitcoin/9/)  0x000055555573dd78 in ServiceConnection (conn=conn@entry=0x7fff70000cc0)
    at rpcserver.cpp:797
[#10](/bitcoin-bitcoin/10/) 0x000055555573eb50 in RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > (acceptor=..., context=..., 
    fUseSSL=<optimized out>, conn=0x7fff70000cc0, error=...)
    at rpcserver.cpp:495
[#11](/bitcoin-bitcoin/11/) 0x000055555574383a in operator()<void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, const boost::system::error_code&), boost::_bi::list1<const boost::system::error_code&> > (
    f=@0x7fffa1ab1bf0: 0x55555573e940 <RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> >(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&)>, a=<synthetic pointer>, 
    this=0x7fffa1ab1bf8) at /usr/include/boost/bind/bind.hpp:525
[#12](/bitcoin-bitcoin/12/) operator()<boost::system::error_code> (a1=..., this=0x7fffa1ab1bf0)
    at /usr/include/boost/bind/bind_template.hpp:47
[#13](/bitcoin-bitcoin/13/) operator() (this=0x7fffa1ab1bf0)
    at /usr/include/boost/asio/detail/bind_handler.hpp:47
[#14](/bitcoin-bitcoin/14/) asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code> > (
    function=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
[#15](/bitcoin-bitcoin/15/) invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
[#16](/bitcoin-bitcoin/16/) boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ip::tcp, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > >::do_complete (owner=0x7fffa91ae190, base=<optimized out>)
    at /usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:123
[#17](/bitcoin-bitcoin/17/) 0x000055555574518d in complete (bytes_transferred=0, ec=..., owner=..., 
    this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#18](/bitcoin-bitcoin/18/) boost::asio::detail::epoll_reactor::descriptor_state::do_complete (
    owner=0x7fffa91ae190, base=0x7fffa91c3590, ec=..., 
    bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
[#19](/bitcoin-bitcoin/19/) 0x000055555574678e in complete (bytes_transferred=1, ec=..., owner=..., 
    this=0x7fffa91c3590)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#20](/bitcoin-bitcoin/20/) do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa91ae190)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
[#21](/bitcoin-bitcoin/21/) boost::asio::detail::task_io_service::run (this=0x7fffa91ae190, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#22](/bitcoin-bitcoin/22/) 0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf4f74f0)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#23](/bitcoin-bitcoin/23/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#24](/bitcoin-bitcoin/24/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa1ab2700)
    at pthread_create.c:312
[#25](/bitcoin-bitcoin/25/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 19 (Thread 0x7fffa2509700 (LWP 6262)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x000055555571e28d in MilliSleep (n=900000) at util.h:97
[#6](/bitcoin-bitcoin/6/)  LoopForever<void (*)()> (name=0x555555b38008 "dumpaddr", 
    func=0x555555711a70 <DumpAddresses()>, msecs=900000) at util.h:550
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa2509700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 18 (Thread 0x7fffa2f5d700 (LWP 6261)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (mutex=0x7fffad23e540)
    at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555561c0ad in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=0x7fffa2f5bdb0) at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=0x7fffa2f5bdb0) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock<AnnotatedMixin<boost::recursive_mutex> >::CMutexLock (
    this=0x7fffa2f5bdb0, mutexIn=..., pszName=<optimized out>, 
    pszFile=<optimized out>, nLine=<optimized out>, fTry=<optimized out>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  0x00005555557d295d in CWallet::Inventory (this=0x7fffad23e3f0, hash=...)
    at wallet.h:354
[#9](/bitcoin-bitcoin/9/)  0x00005555557004cc in operator() (a0=..., this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
[#10](/bitcoin-bitcoin/10/) m_invoke (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:363
[#11](/bitcoin-bitcoin/11/) operator() (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:345
[#12](/bitcoin-bitcoin/12/) dereference (this=0x7fffa2f5be80)
    at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
[#13](/bitcoin-bitcoin/13/) dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<void, const uint256&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&)>, boost::function<void(const boost::signals2::connection&, const uint256&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > > (f=...)
    at /usr/include/boost/iterator/iterator_facade.hpp:514
[#14](/bitcoin-bitcoin/14/) operator* (this=0x7fffa2f5be80)
    at /usr/include/boost/iterator/iterator_facade.hpp:639
[#15](/bitcoin-bitcoin/15/) operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<void, const uint256&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&)>, boost::function<void(const boost::signals2::connection&, const uint256&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > > (this=<optimized out>, 
    first=..., last=...)
    at /usr/include/boost/signals2/optional_last_value.hpp:55
[#16](/bitcoin-bitcoin/16/) operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<void, const uint256&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&)>, boost::function<void(const boost::signals2::connection&, const uint256&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > > (this=<optimized out>, combiner=..., first=..., last=...)
    at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
[#17](/bitcoin-bitcoin/17/) boost::signals2::detail::signal1_impl<void, uint256 const&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (uint256 const&)>, boost::function<void (boost::signals2::connection const&, uint256 const&)>, boost::signals2::mutex>::operator()(uint256 const&) (
    this=<optimized out>, arg1=...)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
[#18](/bitcoin-bitcoin/18/) 0x00005555556cca02 in operator() (arg1=..., 
    this=0x555555ebce80 <(anonymous namespace)::g_signals+96>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
[#19](/bitcoin-bitcoin/19/) ProcessMessage (pfrom=pfrom@entry=0x7fff940046c0, strCommand=..., 
    vRecv=...) at main.cpp:3606
[#20](/bitcoin-bitcoin/20/) 0x00005555556cfde2 in ProcessMessages (pfrom=0x7fff940046c0)
    at main.cpp:4124
[#21](/bitcoin-bitcoin/21/) 0x000055555572b524 in operator() (a0=<optimized out>, 
    this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
[#22](/bitcoin-bitcoin/22/) m_invoke (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:368
[#23](/bitcoin-bitcoin/23/) operator() (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:345
[#24](/bitcoin-bitcoin/24/) dereference (this=0x7fffa2f5cb30)
    at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
[#25](/bitcoin-bitcoin/25/) dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (
    f=...) at /usr/include/boost/iterator/iterator_facade.hpp:514
[#26](/bitcoin-bitcoin/26/) operator* (this=0x7fffa2f5cb30)
    at /usr/include/boost/iterator/iterator_facade.hpp:639
[#27](/bitcoin-bitcoin/27/) operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (
    last=..., first=..., this=<optimized out>)
    at /usr/include/boost/signals2/optional_last_value.hpp:34
[#28](/bitcoin-bitcoin/28/) operator()<boost::signals2::optional_last_value<bool>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (this=<optimized out>, last=..., 
    first=..., combiner=...)
    at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:53
[#29](/bitcoin-bitcoin/29/) boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool (CNode*)>, boost::function<bool (boost::signals2::connection const&, CNode*)>, boost::signals2::mutex>::operator()(CNode*) (this=<optimized out>, 
    arg1=arg1@entry=0x7fff940046c0)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
[#30](/bitcoin-bitcoin/30/) 0x0000555555716836 in operator() (arg1=0x7fff940046c0, 
    this=0x555555ebd738 <g_signals+24>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
[#31](/bitcoin-bitcoin/31/) ThreadMessageHandler () at net.cpp:1544
[#32](/bitcoin-bitcoin/32/) 0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b38000 "msghand", 
    func=0x555555716620 <ThreadMessageHandler()>) at util.h:576
[#33](/bitcoin-bitcoin/33/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#34](/bitcoin-bitcoin/34/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa2f5d700)
    at pthread_create.c:312
[#35](/bitcoin-bitcoin/35/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7fffa37ae700 (LWP 6260)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x0000555555709c43 in boost::condition_variable::wait (
    this=0x7fffa91ab810, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  0x000055555571390b in wait (this=0x7fffa91ab810) at sync.h:185
[#3](/bitcoin-bitcoin/3/)  Acquire (this=0x7fffa37adcc0) at sync.h:218
[#4](/bitcoin-bitcoin/4/)  CSemaphoreGrant (fTry=false, sema=..., this=0x7fffa37adcc0) at sync.h:249
[#5](/bitcoin-bitcoin/5/)  ThreadOpenConnections () at net.cpp:1299
[#6](/bitcoin-bitcoin/6/)  0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b37ff8 "opencon", 
    func=0x555555713640 <ThreadOpenConnections()>) at util.h:576
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa37ae700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7fffa3fff700 (LWP 6259)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x0000555555714773 in MilliSleep (n=120000) at util.h:97
[#6](/bitcoin-bitcoin/6/)  ThreadOpenAddedConnections () at net.cpp:1438
[#7](/bitcoin-bitcoin/7/)  0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b37ff1 "addcon", 
    func=0x555555713f80 <ThreadOpenAddedConnections()>) at util.h:576
[#8](/bitcoin-bitcoin/8/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#9](/bitcoin-bitcoin/9/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa3fff700)
    at pthread_create.c:312
[#10](/bitcoin-bitcoin/10/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7fffb09c8700 (LWP 6258)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43d0c33 in select () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x0000555555715b6e in ThreadSocketHandler () at net.cpp:896
[#2](/bitcoin-bitcoin/2/)  0x000055555571e373 in TraceThread<void (*)()> (name=0x555555b06813 "net", 
    func=0x5555557151e0 <ThreadSocketHandler()>) at util.h:576
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffb09c8700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7fffc118d700 (LWP 6234)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffc118d700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7fffc19de700 (LWP 6233)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffc19de700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7fffc222f700 (LWP 6232)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffc222f700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7fffe2710700 (LWP 6230)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff62067be in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#4](/bitcoin-bitcoin/4/)  0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#5](/bitcoin-bitcoin/5/)  0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#6](/bitcoin-bitcoin/6/)  0x00007ffff60d4c5f in QThread::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#7](/bitcoin-bitcoin/7/)  0x00007ffff60d732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe2710700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7fffe1ebf700 (LWP 6229)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#4](/bitcoin-bitcoin/4/)  0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#5](/bitcoin-bitcoin/5/)  0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#6](/bitcoin-bitcoin/6/)  0x00007ffff60d4c5f in QThread::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#7](/bitcoin-bitcoin/7/)  0x00007ffff60d732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe1ebf700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fffe37b2700 (LWP 6225)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff39c3129 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe37b2700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fffe95de700 (LWP 6224)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c330a in g_main_loop_run ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007fffed561e16 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe95de700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fffe9e2f700 (LWP 6223)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007fffe9e371ad in ?? ()
   from /usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe9e2f700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fc77c0 (LWP 6219)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x0000555555664d54 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=196, 
    pszFile=0x555555afb29d "transactiontablemodel.cpp", 
    pszName=<optimized out>, mutexIn=..., this=<synthetic pointer>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  TransactionTablePriv::index (this=0x555556613f20, idx=idx@entry=64625)
    at transactiontablemodel.cpp:196
[#9](/bitcoin-bitcoin/9/)  0x0000555555663a70 in TransactionTableModel::index (this=0x55555685d8f0, 
    row=64625, column=0, parent=...) at transactiontablemodel.cpp:615
[#10](/bitcoin-bitcoin/10/) 0x0000555555660d5d in TransactionFilterProxy::filterAcceptsRow (
    this=0x555557ab0890, sourceRow=64625, sourceParent=...)
    at transactionfilterproxy.cpp:33
[#11](/bitcoin-bitcoin/11/) 0x00007ffff6fccfd6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#12](/bitcoin-bitcoin/12/) 0x00007ffff6fcd1cf in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#13](/bitcoin-bitcoin/13/) 0x00007ffff61ed87a in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#14](/bitcoin-bitcoin/14/) 0x00007ffff6237664 in QAbstractItemModel::rowsInserted(QModelIndex const&, int, int) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#15](/bitcoin-bitcoin/15/) 0x00007ffff61d35ce in QAbstractItemModel::endInsertRows() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#16](/bitcoin-bitcoin/16/) 0x00005555556659fc in TransactionTablePriv::updateWallet (
    this=0x555556613f20, hash=..., status=status@entry=0)
    at transactiontablemodel.cpp:156
[#17](/bitcoin-bitcoin/17/) 0x000055555566405d in TransactionTableModel::updateTransaction (
    this=0x55555685d8f0, hash=..., status=0) at transactiontablemodel.cpp:255
[#18](/bitcoin-bitcoin/18/) 0x0000555555614566 in WalletModel::updateTransaction (
    this=this@entry=0x555556332bd0, hash=..., status=<optimized out>)
    at walletmodel.cpp:127
[#19](/bitcoin-bitcoin/19/) 0x000055555562cfdc in WalletModel::qt_static_metacall (_o=0x555556332bd0, 
    _c=<optimized out>, _id=<optimized out>, _a=0x7fff70017270)
    at moc_walletmodel.cpp:80
[#20](/bitcoin-bitcoin/20/) 0x00007ffff61f1c1e in QObject::event(QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#21](/bitcoin-bitcoin/21/) 0x00007ffff6a52e2c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#22](/bitcoin-bitcoin/22/) 0x00007ffff6a594a0 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#23](/bitcoin-bitcoin/23/) 0x00007ffff61d94dd in QCoreApplication::notifyInternal(QObject*, QEvent*)
    () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#24](/bitcoin-bitcoin/24/) 0x00007ffff61dcb3d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#25](/bitcoin-bitcoin/25/) 0x00007ffff6206f83 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#26](/bitcoin-bitcoin/26/) 0x00007ffff39c2e04 in g_main_context_dispatch ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#27](/bitcoin-bitcoin/27/) 0x00007ffff39c3048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#28](/bitcoin-bitcoin/28/) 0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#29](/bitcoin-bitcoin/29/) 0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#30](/bitcoin-bitcoin/30/) 0x00007ffff6af4bb6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#31](/bitcoin-bitcoin/31/) 0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#32](/bitcoin-bitcoin/32/) 0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#33](/bitcoin-bitcoin/33/) 0x00007ffff61ddb79 in QCoreApplication::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#34](/bitcoin-bitcoin/34/) 0x00005555555b47e9 in main (argc=1, argv=<optimized out>)
    at bitcoin.cpp:580
(gdb)
  23. laanwj commented at 7:26 PM on April 23, 2014: member

    Can you compile with -DDEBUG_LOCKORDER? If it is a lock ordering issue it should come to light then.

  24. dooglus commented at 7:28 PM on April 23, 2014: contributor

    Sure. Is it better to build from the trunk, or from the version I'm already using?

    I don't know if you've made other related changes in the last 8 days.

  25. laanwj commented at 7:39 PM on April 23, 2014: member

    Well I merged #4085 today, which fixes a hanging GUI in some cases, but I'm not sure it would affect this.

    Edit: May be better not to upgrade to make sure that this problem can still be reproduced.

  26. laanwj commented at 7:51 PM on April 23, 2014: member

    Looking at the traceback of the GUI thread there is, in both cases:

    [#8](/bitcoin-bitcoin/8/)  TransactionTablePriv::index (this=0x555556613f20, idx=idx@entry=64625)
    at transactiontablemodel.cpp:196
....
[#16](/bitcoin-bitcoin/16/) 0x00005555556659fc in TransactionTablePriv::updateWallet (
    this=0x555556613f20, hash=..., status=status@entry=0)
    at transactiontablemodel.cpp:156

    TransactionTablePriv::updateWallet acquires both the cs_main and cs_wallet lock using LOCK2. Then, with the locks held, it calls endInsertRows() which eventually deeper in the code ends up in TransactionTablePriv::index().

    This function, in turn, also does a LOCK2(cs_main, cs_wallet). But weirdly enough, here it hangs. With recursive mutexes re-acquiring a lock should never hang should it? It already has both of these locks.

  27. dooglus commented at 8:02 PM on April 23, 2014: contributor

    I don't know.

    It sounds like a workaround for this would be to just use bitcoind instead of bitcoin-qt, since I'm only using the RPC interface anyway and always keep the QT window minimised.

    Also, how do I add the -D compiler flag in the autotools build system?

  28. laanwj commented at 8:04 PM on April 23, 2014: member

    Like this:

    ./configure CPPFLAGS="-DDEBUG_LOCKORDER"
  29. dooglus commented at 8:10 PM on April 23, 2014: contributor

    I'm rebuilding with the debug flag, but in the mean time it hung up again, and this time I hadn't been making any transactions at all:

      C-c C-c
Program received signal SIGINT, Interrupt.
__lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) where
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x0000555555664d54 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=196, 
    pszFile=0x555555afb29d "transactiontablemodel.cpp", 
    pszName=<optimized out>, mutexIn=..., this=<synthetic pointer>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  TransactionTablePriv::index (this=0x555556729f20, idx=idx@entry=40217)
    at transactiontablemodel.cpp:196
[#9](/bitcoin-bitcoin/9/)  0x0000555555663a70 in TransactionTableModel::index (this=0x555556852420, 
    row=40217, column=0, parent=...) at transactiontablemodel.cpp:615
[#10](/bitcoin-bitcoin/10/) 0x0000555555660d5d in TransactionFilterProxy::filterAcceptsRow (
    this=0x555557aa5960, sourceRow=40217, sourceParent=...)
    at transactionfilterproxy.cpp:33
[#11](/bitcoin-bitcoin/11/) 0x00007ffff6fccfd6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#12](/bitcoin-bitcoin/12/) 0x00007ffff6fcd1cf in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#13](/bitcoin-bitcoin/13/) 0x00007ffff61ed87a in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#14](/bitcoin-bitcoin/14/) 0x00007ffff6237664 in QAbstractItemModel::rowsInserted(QModelIndex const&, int, int) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#15](/bitcoin-bitcoin/15/) 0x00007ffff61d35ce in QAbstractItemModel::endInsertRows() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#16](/bitcoin-bitcoin/16/) 0x00005555556659fc in TransactionTablePriv::updateWallet (
    this=0x555556729f20, hash=..., status=status@entry=0)
    at transactiontablemodel.cpp:156
[#17](/bitcoin-bitcoin/17/) 0x000055555566405d in TransactionTableModel::updateTransaction (
    this=0x555556852420, hash=..., status=0) at transactiontablemodel.cpp:255
[#18](/bitcoin-bitcoin/18/) 0x0000555555614566 in WalletModel::updateTransaction (
    this=this@entry=0x555556226790, hash=..., status=<optimized out>)
    at walletmodel.cpp:127
[#19](/bitcoin-bitcoin/19/) 0x000055555562cfdc in WalletModel::qt_static_metacall (_o=0x555556226790, 
    _c=<optimized out>, _id=<optimized out>, _a=0x7fff9158d220)
    at moc_walletmodel.cpp:80
[#20](/bitcoin-bitcoin/20/) 0x00007ffff61f1c1e in QObject::event(QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#21](/bitcoin-bitcoin/21/) 0x00007ffff6a52e2c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#22](/bitcoin-bitcoin/22/) 0x00007ffff6a594a0 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#23](/bitcoin-bitcoin/23/) 0x00007ffff61d94dd in QCoreApplication::notifyInternal(QObject*, QEvent*)
    () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#24](/bitcoin-bitcoin/24/) 0x00007ffff61dcb3d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#25](/bitcoin-bitcoin/25/) 0x00007ffff6206f83 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#26](/bitcoin-bitcoin/26/) 0x00007ffff39c2e04 in g_main_context_dispatch ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#27](/bitcoin-bitcoin/27/) 0x00007ffff39c3048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#28](/bitcoin-bitcoin/28/) 0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#29](/bitcoin-bitcoin/29/) 0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#30](/bitcoin-bitcoin/30/) 0x00007ffff6af4bb6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#31](/bitcoin-bitcoin/31/) 0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#32](/bitcoin-bitcoin/32/) 0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#33](/bitcoin-bitcoin/33/) 0x00007ffff61ddb79 in QCoreApplication::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#34](/bitcoin-bitcoin/34/) 0x00005555555b47e9 in main (argc=1, argv=<optimized out>)
    at bitcoin.cpp:580
(gdb) thread apply all where

Thread 25 (Thread 0x7fff7d9ce700 (LWP 6502)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x0000555555849bfb in BGThread (this=0x7fffc81fa820)
    at util/env_posix.cc:563
[#2](/bitcoin-bitcoin/2/)  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (
    arg=0x7fffc81fa820) at util/env_posix.cc:510
[#3](/bitcoin-bitcoin/3/)  0x00007ffff58c9182 in start_thread (arg=0x7fff7d9ce700)
    at pthread_create.c:312
[#4](/bitcoin-bitcoin/4/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 24 (Thread 0x7fff890c8700 (LWP 6501)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x00005555557e19d9 in MilliSleep (n=500) at util.h:97
[#6](/bitcoin-bitcoin/6/)  ThreadFlushWalletDB (strFile=...) at walletdb.cpp:782
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fff890c8700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 23 (Thread 0x7fff8b7ae700 (LWP 6500)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43d99a3 in epoll_wait ()
    at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x000055555574258b in boost::asio::detail::epoll_reactor::run (
    this=0x7fffa8f6df70, block=<optimized out>, ops=...)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:392
[#2](/bitcoin-bitcoin/2/)  0x0000555555746551 in do_run_one (ec=..., this_thread=..., lock=..., 
    this=0x7fffa8f6d120)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:368
[#3](/bitcoin-bitcoin/3/)  boost::asio::detail::task_io_service::run (this=0x7fffa8f6d120, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#4](/bitcoin-bitcoin/4/)  0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf269300)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#5](/bitcoin-bitcoin/5/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#6](/bitcoin-bitcoin/6/)  0x00007ffff58c9182 in start_thread (arg=0x7fff8b7ae700)
    at pthread_create.c:312
[#7](/bitcoin-bitcoin/7/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 22 (Thread 0x7fff8bfff700 (LWP 6499)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x000055555574671d in wait<boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex> > (lock=..., this=0x7fff8bffedb0)
    at /usr/include/boost/asio/detail/posix_event.hpp:80
[#2](/bitcoin-bitcoin/2/)  do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa8f6d120)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:395
[#3](/bitcoin-bitcoin/3/)  boost::asio::detail::task_io_service::run (this=0x7fffa8f6d120, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#4](/bitcoin-bitcoin/4/)  0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf269300)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#5](/bitcoin-bitcoin/5/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#6](/bitcoin-bitcoin/6/)  0x00007ffff58c9182 in start_thread (arg=0x7fff8bfff700)
    at pthread_create.c:312
[#7](/bitcoin-bitcoin/7/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 21 (Thread 0x7fffa0e19700 (LWP 6498)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x000055555574671d in wait<boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex> > (lock=..., this=0x7fffa0e18db0)
    at /usr/include/boost/asio/detail/posix_event.hpp:80
[#2](/bitcoin-bitcoin/2/)  do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa8f6d120)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:395
[#3](/bitcoin-bitcoin/3/)  boost::asio::detail::task_io_service::run (this=0x7fffa8f6d120, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#4](/bitcoin-bitcoin/4/)  0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf269300)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#5](/bitcoin-bitcoin/5/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#6](/bitcoin-bitcoin/6/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa0e19700)
    at pthread_create.c:312
[#7](/bitcoin-bitcoin/7/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 20 (Thread 0x7fffa166a700 (LWP 6497)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555573b1a6 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=852, 
    pszFile=0x555555b4f980 "rpcserver.cpp", pszName=<optimized out>, 
    mutexIn=..., this=<synthetic pointer>) at sync.h:139
[#8](/bitcoin-bitcoin/8/)  CRPCTable::execute (this=this@entry=0x555555ebd9e0 <tableRPC>, 
    strMethod=..., params=...) at rpcserver.cpp:852
[#9](/bitcoin-bitcoin/9/)  0x000055555573dd78 in ServiceConnection (conn=conn@entry=0x7fff700148b0)
    at rpcserver.cpp:797
[#10](/bitcoin-bitcoin/10/) 0x000055555573eb50 in RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > (acceptor=..., context=..., 
    fUseSSL=<optimized out>, conn=0x7fff700148b0, error=...)
    at rpcserver.cpp:495
[#11](/bitcoin-bitcoin/11/) 0x000055555574383a in operator()<void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, const boost::system::error_code&), boost::_bi::list1<const boost::system::error_code&> > (
    f=@0x7fffa1669bf0: 0x55555573e940 <RPCAcceptHandler<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> >(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&)>, a=<synthetic pointer>, 
    this=0x7fffa1669bf8) at /usr/include/boost/bind/bind.hpp:525
[#12](/bitcoin-bitcoin/12/) operator()<boost::system::error_code> (a1=..., this=0x7fffa1669bf0)
    at /usr/include/boost/bind/bind_template.hpp:47
[#13](/bitcoin-bitcoin/13/) operator() (this=0x7fffa1669bf0)
    at /usr/include/boost/asio/detail/bind_handler.hpp:47
[#14](/bitcoin-bitcoin/14/) asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code> > (
    function=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
[#15](/bitcoin-bitcoin/15/) invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp> > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
[#16](/bitcoin-bitcoin/16/) boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ip::tcp, boost::_bi::bind_t<void, void (*)(boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > >, boost::asio::ssl::context&, bool, AcceptedConnection*, boost::system::error_code const&), boost::_bi::list5<boost::_bi::value<boost::shared_ptr<boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > > >, boost::reference_wrapper<boost::asio::ssl::context>, boost::_bi::value<bool>, boost::_bi::value<AcceptedConnectionImpl<boost::asio::ip::tcp>*>, boost::arg<1> (*)()> > >::do_complete (owner=0x7fffa8f6d120, base=<optimized out>)
    at /usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:123
[#17](/bitcoin-bitcoin/17/) 0x000055555574518d in complete (bytes_transferred=0, ec=..., owner=..., 
    this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#18](/bitcoin-bitcoin/18/) boost::asio::detail::epoll_reactor::descriptor_state::do_complete (
    owner=0x7fffa8f6d120, base=0x7fffa8f82520, ec=..., 
    bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
[#19](/bitcoin-bitcoin/19/) 0x000055555574678e in complete (bytes_transferred=1, ec=..., owner=..., 
    this=0x7fffa8f82520)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
[#20](/bitcoin-bitcoin/20/) do_run_one (ec=..., this_thread=..., lock=..., this=0x7fffa8f6d120)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
[#21](/bitcoin-bitcoin/21/) boost::asio::detail::task_io_service::run (this=0x7fffa8f6d120, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
[#22](/bitcoin-bitcoin/22/) 0x0000555555747006 in boost::asio::io_service::run (this=0x7fffaf269300)
    at /usr/include/boost/asio/impl/io_service.ipp:59
[#23](/bitcoin-bitcoin/23/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#24](/bitcoin-bitcoin/24/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa166a700)
    at pthread_create.c:312
[#25](/bitcoin-bitcoin/25/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 19 (Thread 0x7fffa1ebb700 (LWP 6496)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x000055555571e28d in MilliSleep (n=900000) at util.h:97
[#6](/bitcoin-bitcoin/6/)  LoopForever<void (*)()> (name=0x555555b38008 "dumpaddr", 
    func=0x555555711a70 <DumpAddresses()>, msecs=900000) at util.h:550
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa1ebb700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 18 (Thread 0x7fffa270c700 (LWP 6495)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (mutex=0x7fffad00b000)
    at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x000055555561c0ad in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=0x7fffa270adb0) at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=0x7fffa270adb0) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock<AnnotatedMixin<boost::recursive_mutex> >::CMutexLock (
    this=0x7fffa270adb0, mutexIn=..., pszName=<optimized out>, 
    pszFile=<optimized out>, nLine=<optimized out>, fTry=<optimized out>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  0x00005555557d295d in CWallet::Inventory (this=0x7fffad00aeb0, hash=...)
    at wallet.h:354
[#9](/bitcoin-bitcoin/9/)  0x00005555557004cc in operator() (a0=..., this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
[#10](/bitcoin-bitcoin/10/) m_invoke (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:363
[#11](/bitcoin-bitcoin/11/) operator() (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:345
[#12](/bitcoin-bitcoin/12/) dereference (this=0x7fffa270ae80)
    at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
[#13](/bitcoin-bitcoin/13/) dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<void, const uint256&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&)>, boost::function<void(const boost::signals2::connection&, const uint256&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > > (f=...)
    at /usr/include/boost/iterator/iterator_facade.hpp:514
[#14](/bitcoin-bitcoin/14/) operator* (this=0x7fffa270ae80)
    at /usr/include/boost/iterator/iterator_facade.hpp:639
[#15](/bitcoin-bitcoin/15/) operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<void, const uint256&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&)>, boost::function<void(const boost::signals2::connection&, const uint256&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > > (this=<optimized out>, 
    first=..., last=...)
    at /usr/include/boost/signals2/optional_last_value.hpp:55
[#16](/bitcoin-bitcoin/16/) operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<void, const uint256&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void(const uint256&)>, boost::function<void(const boost::signals2::connection&, const uint256&)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<void, const uint256&, boost::function<void(const uint256&)> >, boost::signals2::mutex> > > (this=<optimized out>, combiner=..., first=..., last=...)
    at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
[#17](/bitcoin-bitcoin/17/) boost::signals2::detail::signal1_impl<void, uint256 const&, boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (uint256 const&)>, boost::function<void (boost::signals2::connection const&, uint256 const&)>, boost::signals2::mutex>::operator()(uint256 const&) (
    this=<optimized out>, arg1=...)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
[#18](/bitcoin-bitcoin/18/) 0x00005555556cca02 in operator() (arg1=..., 
    this=0x555555ebce80 <(anonymous namespace)::g_signals+96>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
[#19](/bitcoin-bitcoin/19/) ProcessMessage (pfrom=pfrom@entry=0x7fff8c004dd0, strCommand=..., 
    vRecv=...) at main.cpp:3606
[#20](/bitcoin-bitcoin/20/) 0x00005555556cfde2 in ProcessMessages (pfrom=0x7fff8c004dd0)
    at main.cpp:4124
[#21](/bitcoin-bitcoin/21/) 0x000055555572b524 in operator() (a0=<optimized out>, 
    this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
[#22](/bitcoin-bitcoin/22/) m_invoke (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:368
[#23](/bitcoin-bitcoin/23/) operator() (connectionBody=..., this=<optimized out>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:345
[#24](/bitcoin-bitcoin/24/) dereference (this=0x7fffa270bb30)
    at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
[#25](/bitcoin-bitcoin/25/) dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (
    f=...) at /usr/include/boost/iterator/iterator_facade.hpp:514
[#26](/bitcoin-bitcoin/26/) operator* (this=0x7fffa270bb30)
    at /usr/include/boost/iterator/iterator_facade.hpp:639
[#27](/bitcoin-bitcoin/27/) operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (
    last=..., first=..., this=<optimized out>)
    at /usr/include/boost/signals2/optional_last_value.hpp:34
[#28](/bitcoin-bitcoin/28/) operator()<boost::signals2::optional_last_value<bool>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool(CNode*)>, boost::function<bool(const boost::signals2::connection&, CNode*)>, boost::signals2::mutex>::slot_invoker, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot1<bool, CNode*, boost::function<bool(CNode*)> >, boost::signals2::mutex> > > (this=<optimized out>, last=..., 
    first=..., combiner=...)
    at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:53
[#29](/bitcoin-bitcoin/29/) boost::signals2::detail::signal1_impl<bool, CNode*, boost::signals2::optional_last_value<bool>, int, std::less<int>, boost::function<bool (CNode*)>, boost::function<bool (boost::signals2::connection const&, CNode*)>, boost::signals2::mutex>::operator()(CNode*) (this=<optimized out>, 
    arg1=arg1@entry=0x7fff8c004dd0)
    at /usr/include/boost/signals2/detail/signal_template.hpp:246
[#30](/bitcoin-bitcoin/30/) 0x0000555555716836 in operator() (arg1=0x7fff8c004dd0, 
    this=0x555555ebd738 <g_signals+24>)
    at /usr/include/boost/signals2/detail/signal_template.hpp:695
[#31](/bitcoin-bitcoin/31/) ThreadMessageHandler () at net.cpp:1544
[#32](/bitcoin-bitcoin/32/) 0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b38000 "msghand", 
    func=0x555555716620 <ThreadMessageHandler()>) at util.h:576
[#33](/bitcoin-bitcoin/33/) 0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#34](/bitcoin-bitcoin/34/) 0x00007ffff58c9182 in start_thread (arg=0x7fffa270c700)
    at pthread_create.c:312
[#35](/bitcoin-bitcoin/35/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 17 (Thread 0x7fffa2f5d700 (LWP 6494)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x0000555555709c43 in boost::condition_variable::wait (
    this=0x7fffa8f6a7a0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  0x000055555571390b in wait (this=0x7fffa8f6a7a0) at sync.h:185
[#3](/bitcoin-bitcoin/3/)  Acquire (this=0x7fffa2f5ccc0) at sync.h:218
[#4](/bitcoin-bitcoin/4/)  CSemaphoreGrant (fTry=false, sema=..., this=0x7fffa2f5ccc0) at sync.h:249
[#5](/bitcoin-bitcoin/5/)  ThreadOpenConnections () at net.cpp:1299
[#6](/bitcoin-bitcoin/6/)  0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b37ff8 "opencon", 
    func=0x555555713640 <ThreadOpenConnections()>) at util.h:576
[#7](/bitcoin-bitcoin/7/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa2f5d700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7fffa37ae700 (LWP 6493)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
[#1](/bitcoin-bitcoin/1/)  0x00007ffff754bacc in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff75484ba in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#3](/bitcoin-bitcoin/3/)  0x0000555555709924 in sleep_for (ns=<synthetic pointer>)
    at /usr/include/boost/thread/pthread/thread_data.hpp:235
[#4](/bitcoin-bitcoin/4/)  boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...)
    at /usr/include/boost/thread/v2/thread.hpp:53
[#5](/bitcoin-bitcoin/5/)  0x0000555555714773 in MilliSleep (n=120000) at util.h:97
[#6](/bitcoin-bitcoin/6/)  ThreadOpenAddedConnections () at net.cpp:1438
[#7](/bitcoin-bitcoin/7/)  0x000055555571e373 in TraceThread<void (*)()> (
    name=0x555555b37ff1 "addcon", 
    func=0x555555713f80 <ThreadOpenAddedConnections()>) at util.h:576
[#8](/bitcoin-bitcoin/8/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#9](/bitcoin-bitcoin/9/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa37ae700)
    at pthread_create.c:312
[#10](/bitcoin-bitcoin/10/) 0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7fffa3fff700 (LWP 6492)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43d0c33 in select () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x0000555555715b6e in ThreadSocketHandler () at net.cpp:896
[#2](/bitcoin-bitcoin/2/)  0x000055555571e373 in TraceThread<void (*)()> (name=0x555555b06813 "net", 
    func=0x5555557151e0 <ThreadSocketHandler()>) at util.h:576
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffa3fff700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7fffbd1b4700 (LWP 6471)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffbd1b4700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7fffbda05700 (LWP 6470)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffbda05700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7fffe2710700 (LWP 6469)):
[#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
[#1](/bitcoin-bitcoin/1/)  0x00005555556f7033 in wait (m=..., 
    this=0x555555ebcce8 <scriptcheckqueue+40>)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:73
[#2](/bitcoin-bitcoin/2/)  CCheckQueue<CScriptCheck>::Loop (this=0x555555ebccc0 <scriptcheckqueue>, 
    fMaster=false) at checkqueue.h:95
[#3](/bitcoin-bitcoin/3/)  0x00007ffff7547a4a in ?? ()
   from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe2710700)
    at pthread_create.c:312
[#5](/bitcoin-bitcoin/5/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7fffcf2dd700 (LWP 6463)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff62067be in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#4](/bitcoin-bitcoin/4/)  0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#5](/bitcoin-bitcoin/5/)  0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#6](/bitcoin-bitcoin/6/)  0x00007ffff60d4c5f in QThread::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#7](/bitcoin-bitcoin/7/)  0x00007ffff60d732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffcf2dd700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7fffe1ebf700 (LWP 6462)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#4](/bitcoin-bitcoin/4/)  0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#5](/bitcoin-bitcoin/5/)  0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#6](/bitcoin-bitcoin/6/)  0x00007ffff60d4c5f in QThread::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#7](/bitcoin-bitcoin/7/)  0x00007ffff60d732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#8](/bitcoin-bitcoin/8/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe1ebf700)
    at pthread_create.c:312
[#9](/bitcoin-bitcoin/9/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fffe37b2700 (LWP 6458)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007ffff39c3129 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe37b2700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fffe95de700 (LWP 6457)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c330a in g_main_loop_run ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007fffed561e16 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe95de700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fffe9e2f700 (LWP 6456)):
[#0](/bitcoin-bitcoin/0/)  0x00007ffff43cbfbd in poll () at ../sysdeps/unix/syscall-template.S:81
[#1](/bitcoin-bitcoin/1/)  0x00007ffff39c2fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#3](/bitcoin-bitcoin/3/)  0x00007fffe9e371ad in ?? ()
   from /usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so
[#4](/bitcoin-bitcoin/4/)  0x00007ffff39e7f15 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#5](/bitcoin-bitcoin/5/)  0x00007ffff58c9182 in start_thread (arg=0x7fffe9e2f700)
    at pthread_create.c:312
[#6](/bitcoin-bitcoin/6/)  0x00007ffff43d930d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fc77c0 (LWP 6454)):
[#0](/bitcoin-bitcoin/0/)  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
[#1](/bitcoin-bitcoin/1/)  0x00007ffff58cb672 in _L_lock_953 ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
[#2](/bitcoin-bitcoin/2/)  0x00007ffff58cb4da in __GI___pthread_mutex_lock (
    mutex=0x555555ebcb80 <cs_main>) at ../nptl/pthread_mutex_lock.c:114
[#3](/bitcoin-bitcoin/3/)  0x0000555555664d54 in lock (this=<optimized out>)
    at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
[#4](/bitcoin-bitcoin/4/)  lock (this=<optimized out>) at ../../src/sync.h:66
[#5](/bitcoin-bitcoin/5/)  lock (this=<synthetic pointer>)
    at /usr/include/boost/thread/lock_types.hpp:346
[#6](/bitcoin-bitcoin/6/)  Enter (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, this=<synthetic pointer>) at ../../src/sync.h:118
[#7](/bitcoin-bitcoin/7/)  CMutexLock (fTry=false, nLine=196, 
    pszFile=0x555555afb29d "transactiontablemodel.cpp", 
    pszName=<optimized out>, mutexIn=..., this=<synthetic pointer>)
    at ../../src/sync.h:139
[#8](/bitcoin-bitcoin/8/)  TransactionTablePriv::index (this=0x555556729f20, idx=idx@entry=40217)
    at transactiontablemodel.cpp:196
[#9](/bitcoin-bitcoin/9/)  0x0000555555663a70 in TransactionTableModel::index (this=0x555556852420, 
    row=40217, column=0, parent=...) at transactiontablemodel.cpp:615
[#10](/bitcoin-bitcoin/10/) 0x0000555555660d5d in TransactionFilterProxy::filterAcceptsRow (
    this=0x555557aa5960, sourceRow=40217, sourceParent=...)
    at transactionfilterproxy.cpp:33
[#11](/bitcoin-bitcoin/11/) 0x00007ffff6fccfd6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#12](/bitcoin-bitcoin/12/) 0x00007ffff6fcd1cf in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#13](/bitcoin-bitcoin/13/) 0x00007ffff61ed87a in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#14](/bitcoin-bitcoin/14/) 0x00007ffff6237664 in QAbstractItemModel::rowsInserted(QModelIndex const&, int, int) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#15](/bitcoin-bitcoin/15/) 0x00007ffff61d35ce in QAbstractItemModel::endInsertRows() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#16](/bitcoin-bitcoin/16/) 0x00005555556659fc in TransactionTablePriv::updateWallet (
    this=0x555556729f20, hash=..., status=status@entry=0)
    at transactiontablemodel.cpp:156
[#17](/bitcoin-bitcoin/17/) 0x000055555566405d in TransactionTableModel::updateTransaction (
    this=0x555556852420, hash=..., status=0) at transactiontablemodel.cpp:255
[#18](/bitcoin-bitcoin/18/) 0x0000555555614566 in WalletModel::updateTransaction (
    this=this@entry=0x555556226790, hash=..., status=<optimized out>)
    at walletmodel.cpp:127
[#19](/bitcoin-bitcoin/19/) 0x000055555562cfdc in WalletModel::qt_static_metacall (_o=0x555556226790, 
    _c=<optimized out>, _id=<optimized out>, _a=0x7fff9158d220)
    at moc_walletmodel.cpp:80
[#20](/bitcoin-bitcoin/20/) 0x00007ffff61f1c1e in QObject::event(QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#21](/bitcoin-bitcoin/21/) 0x00007ffff6a52e2c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#22](/bitcoin-bitcoin/22/) 0x00007ffff6a594a0 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#23](/bitcoin-bitcoin/23/) 0x00007ffff61d94dd in QCoreApplication::notifyInternal(QObject*, QEvent*)
    () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#24](/bitcoin-bitcoin/24/) 0x00007ffff61dcb3d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#25](/bitcoin-bitcoin/25/) 0x00007ffff6206f83 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#26](/bitcoin-bitcoin/26/) 0x00007ffff39c2e04 in g_main_context_dispatch ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#27](/bitcoin-bitcoin/27/) 0x00007ffff39c3048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#28](/bitcoin-bitcoin/28/) 0x00007ffff39c30ec in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
[#29](/bitcoin-bitcoin/29/) 0x00007ffff62067a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#30](/bitcoin-bitcoin/30/) 0x00007ffff6af4bb6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
[#31](/bitcoin-bitcoin/31/) 0x00007ffff61d80af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#32](/bitcoin-bitcoin/32/) 0x00007ffff61d83a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#33](/bitcoin-bitcoin/33/) 0x00007ffff61ddb79 in QCoreApplication::exec() ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
[#34](/bitcoin-bitcoin/34/) 0x00005555555b47e9 in main (argc=1, argv=<optimized out>)
    at bitcoin.cpp:580
(gdb)
  30. laanwj commented at 8:20 PM on April 23, 2014: member

    Looking at just thread 1 it again seems the same problem. Would be interesting to see if DEBUG_LOCKORDER turns something up.

  31. dooglus commented at 8:27 PM on April 23, 2014: contributor

    It does:

    2014-04-23 20:26:40 UpdateTip: new best=00000000000000002f4c09eee4cd30bb305bc28e048e8b0a4afd0ebb6ced16a7  height=297363  log2_work=78.174438  tx=37418866  date=2014-04-23 20:10:17 progress=0.999947
2014-04-23 20:26:40 AddToWallet 838a56f87e9bb3f8690b19268b3d83dd7311cc53858e5c10bb683416ee6cbfc8  
2014-04-23 20:26:40 ProcessBlock: ACCEPTED
2014-04-23 20:27:05 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:27:05 Previous lock order was:
2014-04-23 20:27:05  (1) wallet->cs_wallet  transactiontablemodel.cpp:99
2014-04-23 20:27:05  (2) cs_main  transactiontablemodel.cpp:196
2014-04-23 20:27:05 Current lock order is:
2014-04-23 20:27:05  (2) cs_main  wallet.cpp:837
2014-04-23 20:27:05  (1) cs_wallet  wallet.cpp:837
  32. dooglus commented at 8:29 PM on April 23, 2014: contributor

    For the record, that is commit b45e335db "Solve chainActive-related locking issues"

  33. dooglus commented at 8:34 PM on April 23, 2014: contributor

    It happened again on the very next block:

    2014-04-23 20:32:21 UpdateTip: new best=00000000000000005715cefa23772ecb945ac63695f48dc87bc7dcb529803e7d  height=297364  log2_work=78.174564  tx=37419550  date=2014-04-23 20:27:50 progress=0.999986
2014-04-23 20:32:21 AddToWallet 5718101a513ba18d902f7a543590edd82dc0e0bd61d7327c2c16bd3a5544c913  
2014-04-23 20:32:21 AddToWallet 1db71fe7dc58cef500d5c4ecef1e849618cf135ed05fe7f0158326643f38180a  
2014-04-23 20:32:21 AddToWallet 934c59fcfacd33bd9fecd85b849c1933fc03784762b12abb2f13476e27ff8a5d  
2014-04-23 20:32:21 ProcessBlock: ACCEPTED
2014-04-23 20:32:27 receive version message: /[...]
2014-04-23 20:32:27 Added time data, samples 4, offset -1 (+0 minutes)
2014-04-23 20:32:43 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:32:43 Previous lock order was:
2014-04-23 20:32:43  (1) wallet->cs_wallet  transactiontablemodel.cpp:99
2014-04-23 20:32:43  (2) cs_main  transactiontablemodel.cpp:196
2014-04-23 20:32:43 Current lock order is:
2014-04-23 20:32:43  (2) cs_main  wallet.cpp:837
2014-04-23 20:32:43  (1) cs_wallet  wallet.cpp:837
  34. laanwj commented at 8:37 PM on April 23, 2014: member

    Argh, seems that you have an earlier revision of that pull, not the one that was merged into master (which is 55a1db4). We're debugging a problem that was solved even before pulling it into master. Let's just update to master HEAD.

  35. dooglus commented at 9:04 PM on April 23, 2014: contributor

    I started it again. This time I see a bunch of warnings but it hasn't locked up yet:

    2014-04-23 20:46:19 Added time data, samples 5, offset -1 (+0 minutes)
2014-04-23 20:46:19 nTimeOffset = +0  (+0 minutes)
2014-04-23 20:46:53 receive version message: /[...]
2014-04-23 20:46:53 Added time data, samples 6, offset -2 (+0 minutes)
2014-04-23 20:46:53 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:46:53 Previous lock order was:
2014-04-23 20:46:53  pnode->cs_vRecvMsg  net.cpp:1541
2014-04-23 20:46:53  (1) cs_main  main.cpp:3245
2014-04-23 20:46:53  cs_mapRelay  main.cpp:3325
2014-04-23 20:46:53  (2) cs_vSend  net.h:449
2014-04-23 20:46:53 Current lock order is:
2014-04-23 20:46:53  (2) pnode->cs_vSend  net.cpp:1560
2014-04-23 20:46:53  (1) cs_main  main.cpp:4205
2014-04-23 20:46:53  (1) cs_main  main.cpp:1313
2014-04-23 20:46:59 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:46:59 Previous lock order was:
2014-04-23 20:46:59  pnode->cs_vRecvMsg  net.cpp:1541
2014-04-23 20:46:59  (1) cs_main  main.cpp:3245
2014-04-23 20:46:59  cs_mapRelay  main.cpp:3325
2014-04-23 20:46:59  (2) cs_vSend  net.h:449
2014-04-23 20:46:59 Current lock order is:
2014-04-23 20:46:59  (2) pnode->cs_vSend  net.cpp:1560
2014-04-23 20:46:59  (1) cs_main  main.cpp:4205
2014-04-23 20:46:59  (1) cs_main  main.cpp:1313
2014-04-23 20:47:06 receive version message: /[...]
2014-04-23 20:47:06 Added time data, samples 7, offset +0 (+0 minutes)
2014-04-23 20:47:06 nTimeOffset = +0  (+0 minutes)
2014-04-23 20:47:22 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:47:22 Previous lock order was:
2014-04-23 20:47:22  pnode->cs_vRecvMsg  net.cpp:1541
2014-04-23 20:47:22  (1) cs_main  main.cpp:3245
2014-04-23 20:47:22  cs_mapRelay  main.cpp:3325
2014-04-23 20:47:22  (2) cs_vSend  net.h:449
2014-04-23 20:47:22 Current lock order is:
2014-04-23 20:47:22  (2) pnode->cs_vSend  net.cpp:1560
2014-04-23 20:47:22  (1) cs_main  main.cpp:4205
2014-04-23 20:47:22  (1) cs_main  main.cpp:1313
2014-04-23 20:47:43 receive version message: /[...]
2014-04-23 20:47:43 Added time data, samples 8, offset -1 (+0 minutes)
2014-04-23 20:47:47 receive version message: /[...]
2014-04-23 20:47:47 Added time data, samples 9, offset +0 (+0 minutes)
2014-04-23 20:47:47 nTimeOffset = +0  (+0 minutes)
2014-04-23 20:48:26 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:48:26 Previous lock order was:
2014-04-23 20:48:26  pnode->cs_vRecvMsg  net.cpp:1541
2014-04-23 20:48:26  (1) cs_main  main.cpp:3245
2014-04-23 20:48:26  cs_mapRelay  main.cpp:3325
2014-04-23 20:48:26  (2) cs_vSend  net.h:449
2014-04-23 20:48:26 Current lock order is:
2014-04-23 20:48:26  (2) pnode->cs_vSend  net.cpp:1560
2014-04-23 20:48:26  (1) cs_main  main.cpp:4205
2014-04-23 20:48:26  (1) cs_main  main.cpp:1313
2014-04-23 20:52:01 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:52:01 Previous lock order was:
2014-04-23 20:52:01  pnode->cs_vRecvMsg  net.cpp:1541
2014-04-23 20:52:01  (1) cs_main  main.cpp:3245
2014-04-23 20:52:01  cs_mapRelay  main.cpp:3325
2014-04-23 20:52:01  (2) cs_vSend  net.h:449
2014-04-23 20:52:01 Current lock order is:
2014-04-23 20:52:01  (2) pnode->cs_vSend  net.cpp:1560
2014-04-23 20:52:01  (1) cs_main  main.cpp:4205
2014-04-23 20:52:01  (1) cs_main  main.cpp:1313
2014-04-23 20:55:26 POTENTIAL DEADLOCK DETECTED
2014-04-23 20:55:26 Previous lock order was:
2014-04-23 20:55:26  pnode->cs_vRecvMsg  net.cpp:1541
2014-04-23 20:55:26  (1) cs_main  main.cpp:3245
2014-04-23 20:55:26  cs_mapRelay  main.cpp:3325
2014-04-23 20:55:26  (2) cs_vSend  net.h:449
2014-04-23 20:55:26 Current lock order is:
2014-04-23 20:55:26  (2) pnode->cs_vSend  net.cpp:1560
2014-04-23 20:55:26  (1) cs_main  main.cpp:4205
2014-04-23 20:55:26  (1) cs_main  main.cpp:1313
2014-04-23 21:03:45 POTENTIAL DEADLOCK DETECTED
2014-04-23 21:03:45 Previous lock order was:
2014-04-23 21:03:45  pnode->cs_vRecvMsg  net.cpp:1541
2014-04-23 21:03:45  (1) cs_main  main.cpp:3245
2014-04-23 21:03:45  cs_mapRelay  main.cpp:3325
2014-04-23 21:03:45  (2) cs_vSend  net.h:449
2014-04-23 21:03:45 Current lock order is:
2014-04-23 21:03:45  (2) pnode->cs_vSend  net.cpp:1560
2014-04-23 21:03:45  (1) cs_main  main.cpp:4205
2014-04-23 21:03:45  (1) cs_main  main.cpp:1313
  36. dooglus commented at 9:58 PM on April 23, 2014: contributor

    OK, I rebuilt from HEAD, and haven't seen it hang since. I still get warnings like these:

    2014-04-23 21:52:39 Added time data, samples 8, offset +0 (+0 minutes)
2014-04-23 21:52:56 POTENTIAL DEADLOCK DETECTED
2014-04-23 21:52:56 Previous lock order was:
2014-04-23 21:52:56  pnode->cs_vRecvMsg  net.cpp:1537
2014-04-23 21:52:56  (1) cs_main  main.cpp:3246
2014-04-23 21:52:56  cs_mapRelay  main.cpp:3326
2014-04-23 21:52:56  (2) cs_vSend  net.h:449
2014-04-23 21:52:56 Current lock order is:
2014-04-23 21:52:56  (2) pnode->cs_vSend  net.cpp:1556
2014-04-23 21:52:56  (1) cs_main  main.cpp:4206
2014-04-23 21:52:56  (1) cs_main  main.cpp:1313
2014-04-23 21:53:47 receive version message: /Satoshi:0.8.5/:
2014-04-23 21:53:47 Added time data, samples 9, offset +14 (+0 minutes)
2014-04-23 21:53:47 nTimeOffset = -2  (+0 minutes)
2014-04-23 21:56:50 POTENTIAL DEADLOCK DETECTED
2014-04-23 21:56:50 Previous lock order was:
2014-04-23 21:56:50  pnode->cs_vRecvMsg  net.cpp:1537
2014-04-23 21:56:50  (1) cs_main  main.cpp:3246
2014-04-23 21:56:50  cs_mapRelay  main.cpp:3326
2014-04-23 21:56:50  (2) cs_vSend  net.h:449
2014-04-23 21:56:50 Current lock order is:
2014-04-23 21:56:50  (2) pnode->cs_vSend  net.cpp:1556
2014-04-23 21:56:50  (1) cs_main  main.cpp:4206
2014-04-23 21:56:50  (1) cs_main  main.cpp:1313
2014-04-23 21:56:51 POTENTIAL DEADLOCK DETECTED
2014-04-23 21:56:51 Previous lock order was:
2014-04-23 21:56:51  pnode->cs_vRecvMsg  net.cpp:1537
2014-04-23 21:56:51  (1) cs_main  main.cpp:3246
2014-04-23 21:56:51  cs_mapRelay  main.cpp:3326
2014-04-23 21:56:51  (2) cs_vSend  net.h:449
2014-04-23 21:56:51 Current lock order is:
2014-04-23 21:56:51  (2) pnode->cs_vSend  net.cpp:1556
2014-04-23 21:56:51  (1) cs_main  main.cpp:4206
2014-04-23 21:56:51  (1) cs_main  main.cpp:1313
2014-04-23 21:57:06 ERROR: CheckTransaction() : vin empty
2014-04-23 21:57:06 ERROR: AcceptToMemoryPool: : CheckTransaction failed
2014-04-23 21:57:06 POTENTIAL DEADLOCK DETECTED
2014-04-23 21:57:06 Previous lock order was:
2014-04-23 21:57:06  pnode->cs_vRecvMsg  net.cpp:1537
2014-04-23 21:57:06  (1) cs_main  main.cpp:3718
2014-04-23 21:57:06  (2) cs_vSend  net.h:449
2014-04-23 21:57:06 Current lock order is:
2014-04-23 21:57:06  (2) pnode->cs_vSend  net.cpp:1556
2014-04-23 21:57:06  (1) cs_main  main.cpp:4206
2014-04-23 21:57:06  (1) cs_main  main.cpp:1313
  37. dooglus commented at 12:54 AM on April 24, 2014: contributor

    I rebuilt against HEAD, but found a few transactions were missing / conflicted - presumably from all the times I quit it when it hung earlier. So I'm running with -zapwallettxes to get rid of the confusion, and that's going to take about 14 hours to run, if it continues at the current speed. Ugh.

  38. laanwj commented at 7:56 AM on April 24, 2014: member

    @dooglus so it appears that there are some locking inconsistencies in the network code. This is unrelated to the activechain-related locking issues (which only dealt with cs_main and cs_wallet) but worth creating a new issue for.

  39. dooglus commented at 8:23 AM on April 24, 2014: contributor

    The -zapwallettxes is still running: http://pastie.org/9107067

    When that's done, I'll see if I'm still getting the lock order warnings and make a new issue.

  40. MathyV referenced this in commit bf1b8ab64e on Nov 23, 2014
  41. MarcoFalke locked this on Sep 8, 2021
Contributors
doogluslaanwj
Labels
Bug
Linked (view graph)
#1 JSON-RPC support for mobile devices ("ultra-lightweight" clients)#2 Long-term, safe, store-of-value#3 Encrypt wallet#4 Export/Import wallet in a human readable, future-proof format#5 Make the version number the protocol version and not the client version#6 Treat wallet as a generic keystore#7 Block-header-only, faster startup client#4058 Solve chainActive-related locking issues #4091 locking inconsistencies in the network code
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-13 18:15 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me