LevelDB read failure: Corruption: block checksum mismatch #11355

1. 
   dooglus commented at 7:59 pm on September 16, 2017: contributor

   Build against commit f197a4c515757262eda0accbe762e311335be11b, a small change from master branch 09627b1dd41d1151a709d5ead82a924bf59e3d38 - bitcoin-qt had been running for 12 hours or so without a problem, was fully synced, then suddenly I saw an error frame pop up with no content. Here’s a screenshot:

   

   The debug.log shows block 485584 was received, then 36 seconds later db corruption was found:

   2017-09-16 15:21:03 UpdateTip: new best=000000000000000000f9dec71fb25bda22ba734ad133c24c8553fb66fee2f98e height=485584 version=0x20000000 log2_work=87.116783 tx=254942322 date='2017-09-16 15:20:14' progress=0.999999 cache=65.6MiB(504658txo) warning='5 of last 100 blocks have unexpected version'
   2017-09-16 15:21:39 LevelDB read failure: Corruption: block checksum mismatch
   2017-09-16 15:21:39 Corruption: block checksum mismatch
   2017-09-16 15:22:02 connect() to [2001:0:9d38:90d7:1097:91ba:f1ff:5782]:8333 failed: Network is unreachable (101)
   2017-09-16 15:41:38 socket sending timeout: 1201s
   2017-09-16 15:41:38 socket sending timeout: 1201s
   2017-09-16 15:41:38 socket sending timeout: 1201s
   2017-09-16 15:41:39 socket sending timeout: 1201s
   2017-09-16 15:41:39 socket sending timeout: 1201s
   2017-09-16 15:41:39 socket sending timeout: 1201s
   2017-09-16 15:41:39 socket sending timeout: 1201s
   2017-09-16 15:41:40 socket sending timeout: 1201s
   

   Nothing has been written to the log since then, and neither the main Bitcoin window nor the error popup has been painted since.

   A few weeks ago I saw a similar error, but at that point I was attempting to track down a race condition on startup and so wasn’t surprised when I saw some database corruption. But I fixed it by doing a -reindex-chainstate and hadn’t had a problem since, until now.

   I attached gdb to the frozen bitcoin-qt process to get the stack traces:

   12:51:18 chris@chris:~/Programs/bitcoin.git/src$ ps -ef | grep bitcoin
   chris    17648 31052  0 Sep15 pts/5    00:03:57 /home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt -datadir=/home/chris/.bitcoin
   chris    23785 31653  0 08:22 pts/7    00:00:00 tail -f /home/chris/.bitcoin/debug.log
   chris    28346 14411  0 12:51 pts/0    00:00:00 grep bitcoin
   12:51:53 chris@chris:~/Programs/bitcoin.git/src$ gdb /home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt 17648
   GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
   Copyright (C) 2016 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
   and "show warranty" for details.
   This GDB was configured as "x86_64-linux-gnu".
   Type "show configuration" for configuration details.
   For bug reporting instructions, please see:
   <http://www.gnu.org/software/gdb/bugs/>.
   Find the GDB manual and other documentation resources online at:
   <http://www.gnu.org/software/gdb/documentation/>.
   For help, type "help".
   Type "apropos word" to search for commands related to "word"...
   Reading symbols from /home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt...done.
   Attaching to program: /home/chris/Programs/bitcoin.git/src/qt/bitcoin-qt, process 17648
   [New LWP 17649]
   [New LWP 17650]
   [New LWP 17651]
   [New LWP 17652]
   [New LWP 17653]
   [New LWP 17654]
   [New LWP 17655]
   [New LWP 17656]
   [New LWP 17657]
   [New LWP 17658]
   [New LWP 17659]
   [New LWP 17660]
   [New LWP 17664]
   [New LWP 17671]
   [New LWP 17673]
   [New LWP 17674]
   [New LWP 17675]
   [New LWP 17676]
   [New LWP 17677]
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
   __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
   135	../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
   (gdb) thread apply all where
   
   Thread 20 (Thread 0x7ff63debb700 (LWP 17677)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d586ad in poll () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff69068f9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#2](/bitcoin-bitcoin/2/)  0x00007ff69068fb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#3](/bitcoin-bitcoin/3/)  0x00007ff69532004f in QEventDispatcherGlib::processEvents (this=0x7ff6240008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6952c99ca in QEventLoop::exec (this=this@entry=0x7ff63debacc0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6950f70f3 in QThread::exec (this=<optimized out>) at thread/qthread.cpp:507
   [#6](/bitcoin-bitcoin/6/)  0x00007ff6950fbda8 in QThreadPrivate::start (arg=0x56128c7f02a0) at thread/qthread_unix.cpp:368
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6947b9494 in start_thread (arg=0x7ff63debb700) at pthread_create.c:333
   [#8](/bitcoin-bitcoin/8/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 19 (Thread 0x7ff63d245700 (LWP 17676)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d586ad in poll () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff69068f9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#2](/bitcoin-bitcoin/2/)  0x00007ff69068fb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#3](/bitcoin-bitcoin/3/)  0x00007ff69532004f in QEventDispatcherGlib::processEvents (this=0x7ff6200008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6952c99ca in QEventLoop::exec (this=this@entry=0x7ff63d244cc0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6950f70f3 in QThread::exec (this=<optimized out>) at thread/qthread.cpp:507
   [#6](/bitcoin-bitcoin/6/)  0x00007ff6950fbda8 in QThreadPrivate::start (arg=0x56128b923d40) at thread/qthread_unix.cpp:368
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6947b9494 in start_thread (arg=0x7ff63d245700) at pthread_create.c:333
   [#8](/bitcoin-bitcoin/8/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 18 (Thread 0x7ff63e70c700 (LWP 17675)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6950fcc6b in QWaitConditionPrivate::wait (time=18446744073709551615, this=0x7ff633dd4480) at thread/qwaitcondition_unix.cpp:143
   [#2](/bitcoin-bitcoin/2/)  QWaitCondition::wait (this=this@entry=0x7ff6154c9468, mutex=mutex@entry=0x7ff6154c9460, time=time@entry=18446744073709551615)
       at thread/qwaitcondition_unix.cpp:215
   [#3](/bitcoin-bitcoin/3/)  0x00007ff6950f5b33 in QSemaphore::acquire (this=this@entry=0x7ff63e702890, n=n@entry=1) at thread/qsemaphore.cpp:143
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6952d68cd in QMetaMethod::invoke (this=this@entry=0x7ff63e702af8, object=object@entry=0x56128b9167c0, connectionType=Qt::BlockingQueuedConnection, 
       connectionType@entry=1047538560, returnValue=..., val0=..., val1=..., val2=..., val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...)
       at kernel/qmetaobject.cpp:2279
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6952dc58a in QMetaObject::invokeMethod (obj=0x56128b9167c0, member=0x56128a8e58de "message", type=1047538560, ret=..., val0=..., val1=..., val2=..., 
       val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...) at kernel/qmetaobject.cpp:1489
   [#6](/bitcoin-bitcoin/6/)  0x000056128a01880c in QMetaObject::invokeMethod (member=0x80 <error: Cannot access memory at address 0x80>, type=Qt::BlockingQueuedConnection, val0=..., 
       obj=<optimized out>, val1=..., val2=..., val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...)
       at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs.h:460
   [#7](/bitcoin-bitcoin/7/)  ThreadSafeMessageBox (gui=0x56128b9167c0, message="Error reading from database, shutting down.", caption=..., style=<optimized out>) at qt/bitcoingui.cpp:1167
   [#8](/bitcoin-bitcoin/8/)  0x000056128a01ccb7 in boost::_bi::list4<boost::_bi::value<BitcoinGUI*>, boost::arg<1>, boost::arg<2>, boost::arg<3> >::operator()<bool, bool (*)(BitcoinGUI*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::_bi::rrlist3<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int> > (f=<error reading variable>, this=<optimized out>, a=...)
       at /usr/include/boost/bind/bind.hpp:453
   [#9](/bitcoin-bitcoin/9/)  boost::_bi::bind_t<bool, bool (*)(BitcoinGUI*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::_bi::list4<boost::_bi::value<BitcoinGUI*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >::operator()<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&&) (a1=<error reading variable: Cannot access memory at address 0x88>, 
       a2=<error reading variable: Cannot access memory at address 0x9>, this=<optimized out>, a3=<optimized out>) at /usr/include/boost/bind/bind.hpp:1330
   [#10](/bitcoin-bitcoin/10/) boost::detail::function::function_obj_invoker3<boost::_bi::bind_t<bool, bool (*)(BitcoinGUI*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::_bi::list4<boost::_bi::value<BitcoinGUI*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>::invoke (function_obj_ptr=..., 
       a0=<error reading variable: Cannot access memory at address 0x88>, a1=<error reading variable: Cannot access memory at address 0x9>, a2=2491150687)
       at /usr/include/boost/function/function_template.hpp:138
   [#11](/bitcoin-bitcoin/11/) 0x000056128a17b81e in boost::function3<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>::operator() (a0=<error reading variable: Cannot access memory at address 0x88>, 
       a1=<error reading variable: Cannot access memory at address 0x9>, a2=2491150687, this=<optimized out>) at /usr/include/boost/function/function_template.hpp:770
   [#12](/bitcoin-bitcoin/12/) boost::signals2::detail::call_with_tuple_args<bool>::m_invoke<boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, 0u, 1u, 2u, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&>(boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>&, boost::signals2::detail::unsigned_meta_array<0u, 1u, 2u>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&> const&, boost::disable_if<boost::is_void<boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>::result_type>, void>::type*) const (this=<optimized out>, func=..., args=...)
       at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:98
   [#13](/bitcoin-bitcoin/13/) boost::signals2::detail::call_with_tuple_args<bool>::operator()<boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&, 3ul>(boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&> const&, mpl_::size_t<3ul>) const (this=<optimized out>, func=..., args=...)
       at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
   [#14](/bitcoin-bitcoin/14/) boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > const&) const (
       this=<optimized out>, connectionBody=...) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
   [#15](/bitcoin-bitcoin/15/) boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >::dereference() const (this=0x7ff63e703010) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:110
   [#16](/bitcoin-bitcoin/16/) 0x000056128a17ac98 in boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > const&) (f=...) at /usr/include/boost/iterator/iterator_facade.hpp:549
   [#17](/bitcoin-bitcoin/17/) boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >, bool, boost::iterators::single_pass_traversal_tag, bool&, long, false, false>::operator*() const (
       this=<optimized out>) at /usr/include/boost/iterator/iterator_facade.hpp:655
   [#18](/bitcoin-bitcoin/18/) boost::signals2::last_value<bool>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >) const (this=<optimized out>, first=..., last=...) at /usr/include/boost/signals2/last_value.hpp:48
   [#19](/bitcoin-bitcoin/19/) 0x000056128a17a930 in boost::signals2::detail::combiner_invoker<bool>::operator()<boost::signals2::last_value<bool>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::signals2::last_value<bool>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >) const (combiner=..., this=<optimized out>, first=..., last=...) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:53
   [#20](/bitcoin-bitcoin/20/) boost::signals2::detail::signal_impl<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::signals2::last_value<bool>, int, std::less<int>, boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::function<bool (boost::signals2::connection const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::signals2::mutex>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) (
       this=0x56128b705d20, args=<optimized out>, args=<optimized out>, args=<optimized out>) at /usr/include/boost/signals2/detail/signal_template.hpp:242
   [#21](/bitcoin-bitcoin/21/) 0x000056128a17a13d in boost::signals2::signal<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::signals2::last_value<bool>, int, std::less<int>, boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::function<bool (boost::signals2::connection const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::signals2::mutex>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) (args=..., args=..., this=<optimized out>, args=...) at /usr/include/boost/signals2/detail/signal_template.hpp:723
   [#22](/bitcoin-bitcoin/22/) CCoinsViewErrorCatcher::GetCoin (this=<optimized out>, outpoint=..., coin=...) at init.cpp:143
   [#23](/bitcoin-bitcoin/23/) 0x000056128a46e462 in CCoinsViewCache::FetchCoin (this=0x7ff648ab9ee0, outpoint=...) at coins.cpp:48
   [#24](/bitcoin-bitcoin/24/) 0x000056128a46e549 in CCoinsViewCache::GetCoin (this=0x7ff633dd44ac, outpoint=..., coin=...) at coins.cpp:61
   [#25](/bitcoin-bitcoin/25/) 0x000056128a2c8414 in CCoinsViewMemPool::GetCoin (this=0x7ff63e7038a8, outpoint=..., coin=...) at txmempool.cpp:902
   [#26](/bitcoin-bitcoin/26/) 0x000056128a46e462 in CCoinsViewCache::FetchCoin (this=0x7ff63e704530, outpoint=...) at coins.cpp:48
   [#27](/bitcoin-bitcoin/27/) 0x000056128a46eb03 in CCoinsViewCache::HaveCoin (this=0x7ff633dd44ac, outpoint=...) at coins.cpp:129
   [#28](/bitcoin-bitcoin/28/) 0x000056128a30c5cc in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=..., fLimitFree=<optimized out>, pfMissingInputs=<optimized out>, 
       nAcceptTime=140697029833055, plTxnReplaced=0x56128acc70f0 <pcoinsTip>, fOverrideMempoolLimit=false, nAbsurdFee=@0x7ff63e7044d0: -2962908112588391035, 
       coins_to_uncache=...) at validation.cpp:547
   [#29](/bitcoin-bitcoin/29/) 0x000056128a2de858 in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., 
       tx=<error reading variable: Cannot access memory at address 0x1be08247c8b50>, fLimitFree=<optimized out>, pfMissingInputs=0x0, nAcceptTime=1505575299, 
       plTxnReplaced=0x7ff63e7083a8, fOverrideMempoolLimit=<optimized out>, nAbsurdFee=0) at validation.cpp:886
   [#30](/bitcoin-bitcoin/30/) 0x000056128a2de122 in AcceptToMemoryPool (pool=..., state=..., tx=<error reading variable: Cannot access memory at address 0x1be08247c8b50>, fLimitFree=true, 
       pfMissingInputs=0x0, plTxnReplaced=0x7ff63e7083a8, fOverrideMempoolLimit=<optimized out>, nAbsurdFee=<optimized out>) at validation.cpp:902
   [#31](/bitcoin-bitcoin/31/) 0x000056128a1d7266 in ProcessMessage (pfrom=<optimized out>, strCommand=..., vRecv=..., nTimeReceived=<optimized out>, chainparams=..., 
       connman=<optimized out>, interruptMsgProc=...) at net_processing.cpp:1791
   [#32](/bitcoin-bitcoin/32/) 0x000056128a1cbf12 in PeerLogicValidation::ProcessMessages (this=0x7ff664046440, pfrom=0x7ff62822f0f0, interruptMsgProc=...) at net_processing.cpp:2718
   [#33](/bitcoin-bitcoin/33/) 0x000056128a1f8037 in non-virtual thunk to PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) () at net_processing.cpp:2643
   [#34](/bitcoin-bitcoin/34/) 0x000056128a1a767b in CConnman::ThreadMessageHandler (this=<optimized out>) at net.cpp:1995
   [#35](/bitcoin-bitcoin/35/) 0x000056128a1c24ba in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=<optimized out>, __t=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
   [#36](/bitcoin-bitcoin/36/) std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
   [#37](/bitcoin-bitcoin/37/) std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
   [#38](/bitcoin-bitcoin/38/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=<optimized out>, 
       __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
   [#39](/bitcoin-bitcoin/39/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
   [#40](/bitcoin-bitcoin/40/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
   [#41](/bitcoin-bitcoin/41/) 0x000056128a176e3d in std::function<void ()>::operator()() const (this=0x7ff63e70bd00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
   [#42](/bitcoin-bitcoin/42/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:309
   [#43](/bitcoin-bitcoin/43/) 0x000056128a1c267d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#44](/bitcoin-bitcoin/44/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#45](/bitcoin-bitcoin/45/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#46](/bitcoin-bitcoin/46/) 0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#47](/bitcoin-bitcoin/47/) 0x00007ff6947b9494 in start_thread (arg=0x7ff63e70c700) at pthread_create.c:333
   [#48](/bitcoin-bitcoin/48/) 0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 17 (Thread 0x7ff63ef5d700 (LWP 17674)):
   [#0](/bitcoin-bitcoin/0/)  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6947bbc06 in __GI___pthread_mutex_lock (mutex=0x56128acc6b88 <cs_main>) at ../nptl/pthread_mutex_lock.c:115
   [#2](/bitcoin-bitcoin/2/)  0x000056128a0259d5 in boost::recursive_mutex::lock (this=<optimized out>) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:113
   [#3](/bitcoin-bitcoin/3/)  AnnotatedMixin<boost::recursive_mutex>::lock (this=<optimized out>) at ./sync.h:59
   [#4](/bitcoin-bitcoin/4/)  boost::unique_lock<CCriticalSection>::lock (this=0x7ff63ef5c250) at /usr/include/boost/thread/lock_types.hpp:346
   [#5](/bitcoin-bitcoin/5/)  0x000056128a1c406a in CMutexLock<CCriticalSection>::Enter (this=0x56128acc6b88 <cs_main>, nLine=504, pszName=<optimized out>, pszFile=<optimized out>)
       at ./sync.h:123
   [#6](/bitcoin-bitcoin/6/)  CMutexLock<CCriticalSection>::CMutexLock (this=0x56128acc6b88 <cs_main>, nLine=504, fTry=false, mutexIn=..., pszName=<optimized out>, pszFile=<optimized out>)
       at ./sync.h:144
   [#7](/bitcoin-bitcoin/7/)  PeerLogicValidation::InitializeNode (this=0x7ff664046440, pnode=0x7ff6282bb0d0) at net_processing.cpp:504
   [#8](/bitcoin-bitcoin/8/)  0x000056128a1c5657 in non-virtual thunk to PeerLogicValidation::InitializeNode(CNode*) () at net_processing.cpp:499
   [#9](/bitcoin-bitcoin/9/)  0x000056128a1a4602 in CConnman::OpenNetworkConnection (this=0x7ff66456ad30, addrConnect=..., fCountFailure=<optimized out>, grantOutbound=<optimized out>, 
       pszDest=0x0, fOneShot=<optimized out>, fFeeler=<optimized out>, fAddnode=92) at net.cpp:1965
   [#10](/bitcoin-bitcoin/10/) 0x000056128a1a5508 in CConnman::ThreadOpenConnections (this=<optimized out>, connect=...) at net.cpp:1840
   [#11](/bitcoin-bitcoin/11/) 0x000056128a1c2887 in std::__invoke_impl<void, void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (
       __f=<optimized out>, __t=<optimized out>, __args=<error reading variable: Cannot access memory at address 0x8>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
   [#12](/bitcoin-bitcoin/12/) 0x000056128a1c270e in std::__invoke<void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__fn= [@0x56128acc6b88](/bitcoin-bitcoin/contributor/0x56128acc6b88/): (void (CConnman::*)(CConnman * const, std::vector<std::__cxx11::basic_string<char>, std::allocator<std::__cxx11::basic_string<char> > >)) 0x100000002, this adjustment 4294984971, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
   [#13](/bitcoin-bitcoin/13/) std::_Mem_fn_base<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), true>::operator()<CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__args=..., __args=..., 
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
   [#14](/bitcoin-bitcoin/14/) std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x56128acc6b88 <cs_main>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
   [#15](/bitcoin-bitcoin/15/) std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::operator()<, void>() (
       this=0x56128acc6b88 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
   [#16](/bitcoin-bitcoin/16/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
   [#17](/bitcoin-bitcoin/17/) 0x000056128a176e3d in std::function<void ()>::operator()() const (this=0x7ff63ef5cd00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
   [#18](/bitcoin-bitcoin/18/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:309
   [#19](/bitcoin-bitcoin/19/) 0x000056128a1c267d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#20](/bitcoin-bitcoin/20/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#21](/bitcoin-bitcoin/21/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#22](/bitcoin-bitcoin/22/) 0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#23](/bitcoin-bitcoin/23/) 0x00007ff6947b9494 in start_thread (arg=0x7ff63ef5d700) at pthread_create.c:333
   [#24](/bitcoin-bitcoin/24/) 0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 16 (Thread 0x7ff63f7ae700 (LWP 17673)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
   [#1](/bitcoin-bitcoin/1/)  0x000056128a49b56e in __gthread_cond_timedwait (__cond=<optimized out>, __mutex=<optimized out>, __abs_timeout=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
   [#2](/bitcoin-bitcoin/2/)  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=<optimized out>, __lock=..., __atime=...)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
   [#3](/bitcoin-bitcoin/3/)  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=<optimized out>, __lock=..., __atime=...)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
   [#4](/bitcoin-bitcoin/4/)  std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=<optimized out>, 
       __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
   [#5](/bitcoin-bitcoin/5/)  std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=<optimized out>, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
   [#6](/bitcoin-bitcoin/6/)  CThreadInterrupt::sleep_for (this=<optimized out>, rel_time=...) at threadinterrupt.cpp:30
   [#7](/bitcoin-bitcoin/7/)  0x000056128a49b5fa in CThreadInterrupt::sleep_for (this=0x7ff6645bb1a4, rel_time=...) at threadinterrupt.cpp:35
   [#8](/bitcoin-bitcoin/8/)  0x000056128a1a7316 in CConnman::ThreadOpenAddedConnections (this=<optimized out>) at net.cpp:1927
   [#9](/bitcoin-bitcoin/9/)  0x000056128a1c24ba in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=<optimized out>, __t=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
   [#10](/bitcoin-bitcoin/10/) std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
   [#11](/bitcoin-bitcoin/11/) std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
   [#12](/bitcoin-bitcoin/12/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=<optimized out>, 
       __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
   [#13](/bitcoin-bitcoin/13/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
   [#14](/bitcoin-bitcoin/14/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
   [#15](/bitcoin-bitcoin/15/) 0x000056128a176e3d in std::function<void ()>::operator()() const (this=0x7ff63f7add00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
   [#16](/bitcoin-bitcoin/16/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:309
   [#17](/bitcoin-bitcoin/17/) 0x000056128a1c267d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#18](/bitcoin-bitcoin/18/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#19](/bitcoin-bitcoin/19/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#20](/bitcoin-bitcoin/20/) 0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#21](/bitcoin-bitcoin/21/) 0x00007ff6947b9494 in start_thread (arg=0x7ff63f7ae700) at pthread_create.c:333
   [#22](/bitcoin-bitcoin/22/) 0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 15 (Thread 0x7ff645ec7700 (LWP 17671)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d5a3f3 in select () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x000056128a19da64 in CConnman::ThreadSocketHandler (this=<optimized out>) at net.cpp:1251
   [#2](/bitcoin-bitcoin/2/)  0x000056128a1c24ba in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=<optimized out>, __t=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
   [#3](/bitcoin-bitcoin/3/)  std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
   [#4](/bitcoin-bitcoin/4/)  std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
   [#5](/bitcoin-bitcoin/5/)  std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=<optimized out>, 
       __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
   [#6](/bitcoin-bitcoin/6/)  std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
   [#7](/bitcoin-bitcoin/7/)  std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
   [#8](/bitcoin-bitcoin/8/)  0x000056128a176e3d in std::function<void ()>::operator()() const (this=0x7ff645ec6d00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
   [#9](/bitcoin-bitcoin/9/)  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:309
   [#10](/bitcoin-bitcoin/10/) 0x000056128a1c267d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#11](/bitcoin-bitcoin/11/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#12](/bitcoin-bitcoin/12/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#13](/bitcoin-bitcoin/13/) 0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#14](/bitcoin-bitcoin/14/) 0x00007ff6947b9494 in start_thread (arg=0x7ff645ec7700) at pthread_create.c:333
   [#15](/bitcoin-bitcoin/15/) 0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 14 (Thread 0x7ff658cf7700 (LWP 17664)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [#1](/bitcoin-bitcoin/1/)  0x000056128a4f38ab in leveldb::(anonymous namespace)::PosixEnv::BGThread (this=<optimized out>) at leveldb/util/env_posix.cc:641
   [#2](/bitcoin-bitcoin/2/)  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=<optimized out>) at leveldb/util/env_posix.cc:556
   [#3](/bitcoin-bitcoin/3/)  0x00007ff6947b9494 in start_thread (arg=0x7ff658cf7700) at pthread_create.c:333
   [#4](/bitcoin-bitcoin/4/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 13 (Thread 0x7ff668e19700 (LWP 17660)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6935e750c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#2](/bitcoin-bitcoin/2/)  0x000056128a381c2a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:127
   [#3](/bitcoin-bitcoin/3/)  0x000056128a37baca in HTTPWorkQueueRun (queue=0x7ff664044c7c) at httpserver.cpp:351
   [#4](/bitcoin-bitcoin/4/)  0x000056128a383508 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#8](/bitcoin-bitcoin/8/)  0x00007ff6947b9494 in start_thread (arg=0x7ff668e19700) at pthread_create.c:333
   [#9](/bitcoin-bitcoin/9/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 12 (Thread 0x7ff66966a700 (LWP 17659)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6935e750c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#2](/bitcoin-bitcoin/2/)  0x000056128a381c2a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:127
   [#3](/bitcoin-bitcoin/3/)  0x000056128a37baca in HTTPWorkQueueRun (queue=0x7ff664044c7c) at httpserver.cpp:351
   [#4](/bitcoin-bitcoin/4/)  0x000056128a383508 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#8](/bitcoin-bitcoin/8/)  0x00007ff6947b9494 in start_thread (arg=0x7ff66966a700) at pthread_create.c:333
   [#9](/bitcoin-bitcoin/9/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 11 (Thread 0x7ff669ebb700 (LWP 17658)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6935e750c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#2](/bitcoin-bitcoin/2/)  0x000056128a381c2a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:127
   [#3](/bitcoin-bitcoin/3/)  0x000056128a37baca in HTTPWorkQueueRun (queue=0x7ff664044c7c) at httpserver.cpp:351
   [#4](/bitcoin-bitcoin/4/)  0x000056128a383508 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#8](/bitcoin-bitcoin/8/)  0x00007ff6947b9494 in start_thread (arg=0x7ff669ebb700) at pthread_create.c:333
   [#9](/bitcoin-bitcoin/9/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 10 (Thread 0x7ff66a70c700 (LWP 17657)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6935e750c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#2](/bitcoin-bitcoin/2/)  0x000056128a381c2a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:127
   [#3](/bitcoin-bitcoin/3/)  0x000056128a37baca in HTTPWorkQueueRun (queue=0x7ff664044c7c) at httpserver.cpp:351
   [#4](/bitcoin-bitcoin/4/)  0x000056128a383508 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#8](/bitcoin-bitcoin/8/)  0x00007ff6947b9494 in start_thread (arg=0x7ff66a70c700) at pthread_create.c:333
   [#9](/bitcoin-bitcoin/9/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 9 (Thread 0x7ff66af5d700 (LWP 17656)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d620f3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6938d9d98 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
   [#2](/bitcoin-bitcoin/2/)  0x00007ff6938c407a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
   [#3](/bitcoin-bitcoin/3/)  0x000056128a37b03f in ThreadHTTP (base=0x7ff6640443d0, http=<optimized out>) at httpserver.cpp:303
   [#4](/bitcoin-bitcoin/4/)  0x000056128a382c4d in std::__invoke_impl<bool, bool (*&)(event_base*, evhttp*), event_base*, evhttp*>(std::__invoke_other, bool (*&)(event_base*, evhttp*), event_base*&&, evhttp*&&) (__f=<optimized out>, __args=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:210
   [#5](/bitcoin-bitcoin/5/)  std::__invoke<bool (*&)(event_base*, evhttp*), event_base*, evhttp*>(bool (*&)(event_base*, evhttp*), event_base*&&, evhttp*&&) (__fn=<optimized out>, 
       __args=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
   [#6](/bitcoin-bitcoin/6/)  std::reference_wrapper<bool (*)(event_base*, evhttp*)>::operator()<event_base*, evhttp*>(event_base*&&, evhttp*&&) const (__args=<optimized out>, 
       this=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:465
   [#7](/bitcoin-bitcoin/7/)  std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (
       this=0x7ff692d620f3 <epoll_wait+51>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#8](/bitcoin-bitcoin/8/)  std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>::operator()() (this=0x7ff692d620f3 <epoll_wait+51>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#9](/bitcoin-bitcoin/9/)  std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>, bool>::operator()() const (this=0x7ff66af5ccd0)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:1325
   [#10](/bitcoin-bitcoin/10/) 0x000056128a382b8f in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1716
   [#11](/bitcoin-bitcoin/11/) 0x000056128a382aba in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (
       this=0x7ff664044660) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
   [#12](/bitcoin-bitcoin/12/) std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (this=0x7ff664044e80, __f=0x7ff664044660, __did_set=0x7ff66af5cc8f) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:533
   [#13](/bitcoin-bitcoin/13/) 0x000056128a382b56 in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=<optimized out>, __args=<optimized out>, __args=<optimized out>, __t=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
   [#14](/bitcoin-bitcoin/14/) std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=<optimized out>, __args=<optimized out>, __args=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
   [#15](/bitcoin-bitcoin/15/) void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:602
   [#16](/bitcoin-bitcoin/16/) void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:607
   [#17](/bitcoin-bitcoin/17/) void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::__invoke() () at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:607
   [#18](/bitcoin-bitcoin/18/) 0x00007ff6947c0739 in __pthread_once_slow (once_control=0x7ff664044e98, init_routine=0x7ff6935ec120 <__once_proxy>) at pthread_once.c:116
   [#19](/bitcoin-bitcoin/19/) 0x000056128a382a39 in __gthread_once (__func=0x7ff664044660, __once=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:699
   [#20](/bitcoin-bitcoin/20/) std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__args=<optimized out>, __args=<optimized out>, __args=<optimized out>, __once=..., __f=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:614
   [#21](/bitcoin-bitcoin/21/) std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x7ff664044e80, __res=..., __ignore_failure=false) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:392
   [#22](/bitcoin-bitcoin/22/) 0x000056128a382491 in std::__future_base::_Task_state<bool (*)(event_base*, evhttp*), std::allocator<int>, bool (event_base*, evhttp*)>::_M_run(event_base*&&, evhttp*&&) (this=0x17, __args=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:1409
   [#23](/bitcoin-bitcoin/23/) 0x000056128a38348b in std::packaged_task<bool (event_base*, evhttp*)>::operator()(event_base*, evhttp*) (
       __args=<error reading variable: Cannot access memory at address 0x20>, __args=<error reading variable: Cannot access memory at address 0x20>, 
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:1553
   [#24](/bitcoin-bitcoin/24/) std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#25](/bitcoin-bitcoin/25/) std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#26](/bitcoin-bitcoin/26/) std::thread::_State_impl<std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)> >::_M_run() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#27](/bitcoin-bitcoin/27/) 0x00007ff6935ece6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#28](/bitcoin-bitcoin/28/) 0x00007ff6947b9494 in start_thread (arg=0x7ff66af5d700) at pthread_create.c:333
   [#29](/bitcoin-bitcoin/29/) 0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 8 (Thread 0x7ff66b7ae700 (LWP 17655)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
   [#1](/bitcoin-bitcoin/1/)  0x000056128a257c82 in boost::condition_variable::do_wait_until (this=0x56128bfebea0, m=..., timeout=...)
       at /usr/include/boost/thread/pthread/condition_variable.hpp:109
   [#2](/bitcoin-bitcoin/2/)  0x000056128a489379 in boost::condition_variable::wait_until (this=<optimized out>, lk=..., tp=...)
       at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:252
   [#3](/bitcoin-bitcoin/3/)  boost::condition_variable::wait_until<boost::chrono::duration<long, boost::ratio<1l, 1000000000l> > > (this=<optimized out>, lock=..., t=...)
       at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:209
   [#4](/bitcoin-bitcoin/4/)  CScheduler::serviceQueue (this=<optimized out>) at scheduler.cpp:67
   [#5](/bitcoin-bitcoin/5/)  0x000056128a18553a in boost::_mfi::mf0<void, CScheduler>::operator() (p=<optimized out>, this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:49
   [#6](/bitcoin-bitcoin/6/)  boost::_bi::list1<boost::_bi::value<CScheduler*> >::operator()<boost::_mfi::mf0<void, CScheduler>, boost::_bi::list0> (this=<optimized out>, f=..., a=...)
       at /usr/include/boost/bind/bind.hpp:259
   [#7](/bitcoin-bitcoin/7/)  boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > >::operator() (this=<optimized out>)
       at /usr/include/boost/bind/bind.hpp:1294
   [#8](/bitcoin-bitcoin/8/)  std::_Function_handler<void (), boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > > >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
   [#9](/bitcoin-bitcoin/9/)  0x000056128a176e3d in std::function<void ()>::operator()() const (this=0x7ff66b7adcb8)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
   [#10](/bitcoin-bitcoin/10/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:309
   [#11](/bitcoin-bitcoin/11/) 0x000056128a185ab6 in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > >::operator()<void (*)(char const*, std::function<void ()>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(char const*, std::function<void ()>), boost::_bi::list0&, int) (this=0x7ff664043100, 
       f=<optimized out>, a=...) at /usr/include/boost/bind/bind.hpp:319
   [#12](/bitcoin-bitcoin/12/) 0x000056128a185a3f in boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > >::operator()() (this=<optimized out>) at /usr/include/boost/bind/bind.hpp:1294
   [#13](/bitcoin-bitcoin/13/) boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=<optimized out>) at /usr/include/boost/thread/detail/thread.hpp:116
   [#14](/bitcoin-bitcoin/14/) 0x00007ff69635d116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
   [#15](/bitcoin-bitcoin/15/) 0x00007ff6947b9494 in start_thread (arg=0x7ff66b7ae700) at pthread_create.c:333
   [#16](/bitcoin-bitcoin/16/) 0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 7 (Thread 0x7ff66bfff700 (LWP 17654)):
   [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
   [#1](/bitcoin-bitcoin/1/)  0x000056128a0017eb in boost::condition_variable::wait (this=<optimized out>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
   [#2](/bitcoin-bitcoin/2/)  0x000056128a3212f2 in CCheckQueue<CScriptCheck>::Loop (this=<optimized out>, fMaster=<optimized out>) at ./checkqueue.h:102
   [#3](/bitcoin-bitcoin/3/)  0x000056128a2e12c8 in CCheckQueue<CScriptCheck>::Thread (this=0x56128acc71ec <scriptcheckqueue+84>) at ./checkqueue.h:139
   [#4](/bitcoin-bitcoin/4/)  ThreadScriptCheck () at validation.cpp:1540
   [#5](/bitcoin-bitcoin/5/)  0x000056128a185114 in boost::detail::thread_data<void (*)()>::run (this=0x56128acc71ec <scriptcheckqueue+84>) at /usr/include/boost/thread/detail/thread.hpp:116
   [#6](/bitcoin-bitcoin/6/)  0x00007ff69635d116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6947b9494 in start_thread (arg=0x7ff66bfff700) at pthread_create.c:333
   [#8](/bitcoin-bitcoin/8/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 6 (Thread 0x7ff677398700 (LWP 17653)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d586ad in poll () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff69068f9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#2](/bitcoin-bitcoin/2/)  0x00007ff69068fb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#3](/bitcoin-bitcoin/3/)  0x00007ff69532004f in QEventDispatcherGlib::processEvents (this=0x7ff6640008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6952c99ca in QEventLoop::exec (this=this@entry=0x7ff677397cc0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6950f70f3 in QThread::exec (this=<optimized out>) at thread/qthread.cpp:507
   [#6](/bitcoin-bitcoin/6/)  0x00007ff6950fbda8 in QThreadPrivate::start (arg=0x56128bfe7de0) at thread/qthread_unix.cpp:368
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6947b9494 in start_thread (arg=0x7ff677398700) at pthread_create.c:333
   [#8](/bitcoin-bitcoin/8/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 5 (Thread 0x7ff677be9700 (LWP 17652)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d586ad in poll () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff69068f9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#2](/bitcoin-bitcoin/2/)  0x00007ff69068fb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#3](/bitcoin-bitcoin/3/)  0x00007ff69532004f in QEventDispatcherGlib::processEvents (this=0x7ff66c0008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6952c99ca in QEventLoop::exec (this=this@entry=0x7ff677be8c90, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6950f70f3 in QThread::exec (this=this@entry=0x7ff6955a6d60 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>)
       at thread/qthread.cpp:507
   [#6](/bitcoin-bitcoin/6/)  0x00007ff6955316d5 in QDBusConnectionManager::run (this=0x7ff6955a6d60 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>)
       at qdbusconnection.cpp:178
   [#7](/bitcoin-bitcoin/7/)  0x00007ff6950fbda8 in QThreadPrivate::start (arg=0x7ff6955a6d60 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>)
       at thread/qthread_unix.cpp:368
   [#8](/bitcoin-bitcoin/8/)  0x00007ff6947b9494 in start_thread (arg=0x7ff677be9700) at pthread_create.c:333
   [#9](/bitcoin-bitcoin/9/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 4 (Thread 0x7ff67df9e700 (LWP 17651)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d586ad in poll () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff69068f9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#2](/bitcoin-bitcoin/2/)  0x00007ff69068fd82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#3](/bitcoin-bitcoin/3/)  0x00007ff68473e656 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6906b73d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6947b9494 in start_thread (arg=0x7ff67df9e700) at pthread_create.c:333
   [#6](/bitcoin-bitcoin/6/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 3 (Thread 0x7ff67e7ef700 (LWP 17650)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d586ad in poll () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff69068f9f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#2](/bitcoin-bitcoin/2/)  0x00007ff69068fb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#3](/bitcoin-bitcoin/3/)  0x00007ff69068fb51 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6906b73d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6947b9494 in start_thread (arg=0x7ff67e7ef700) at pthread_create.c:333
   [#6](/bitcoin-bitcoin/6/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 2 (Thread 0x7ff688439700 (LWP 17649)):
   [#0](/bitcoin-bitcoin/0/)  0x00007ff692d586ad in poll () at ../sysdeps/unix/syscall-template.S:84
   [#1](/bitcoin-bitcoin/1/)  0x00007ff68e55c150 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
   [#2](/bitcoin-bitcoin/2/)  0x00007ff68e55dee9 in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
   [#3](/bitcoin-bitcoin/3/)  0x00007ff68ab9ab69 in QXcbEventReader::run (this=0x56128b83c900) at qxcbconnection.cpp:1343
   [#4](/bitcoin-bitcoin/4/)  0x00007ff6950fbda8 in QThreadPrivate::start (arg=0x56128b83c900) at thread/qthread_unix.cpp:368
   [#5](/bitcoin-bitcoin/5/)  0x00007ff6947b9494 in start_thread (arg=0x7ff688439700) at pthread_create.c:333
   [#6](/bitcoin-bitcoin/6/)  0x00007ff692d61aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   
   Thread 1 (Thread 0x7ff696f01900 (LWP 17648)):
   [#0](/bitcoin-bitcoin/0/)  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
   [#1](/bitcoin-bitcoin/1/)  0x00007ff6947bbc06 in __GI___pthread_mutex_lock (mutex=0x56128acc6df0 <mempool+56>) at ../nptl/pthread_mutex_lock.c:115
   [#2](/bitcoin-bitcoin/2/)  0x000056128a0259d5 in boost::recursive_mutex::lock (this=<optimized out>) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:113
   [#3](/bitcoin-bitcoin/3/)  AnnotatedMixin<boost::recursive_mutex>::lock (this=<optimized out>) at ./sync.h:59
   [#4](/bitcoin-bitcoin/4/)  boost::unique_lock<CCriticalSection>::lock (this=0x7fff8192c5e8) at /usr/include/boost/thread/lock_types.hpp:346
   [#5](/bitcoin-bitcoin/5/)  0x000056128a02541e in CMutexLock<CCriticalSection>::Enter (this=0x56128acc6df0 <mempool+56>, nLine=602, pszName=<optimized out>, pszFile=<optimized out>)
       at ./sync.h:123
   [#6](/bitcoin-bitcoin/6/)  CMutexLock<CCriticalSection>::CMutexLock (this=0x56128acc6df0 <mempool+56>, mutexIn=..., nLine=602, fTry=false, pszName=<optimized out>, 
       pszFile=<optimized out>) at ./sync.h:144
   [#7](/bitcoin-bitcoin/7/)  CTxMemPool::size (this=0x56128acc6db8 <mempool>) at ./txmempool.h:602
   [#8](/bitcoin-bitcoin/8/)  0x000056128a0240b9 in ClientModel::getMempoolSize (this=<optimized out>) at qt/clientmodel.cpp:130
   [#9](/bitcoin-bitcoin/9/)  ClientModel::updateTimer (this=0x56128b70d6e0) at qt/clientmodel.cpp:153
   [#10](/bitcoin-bitcoin/10/) 0x000056128a0ba739 in ClientModel::qt_static_metacall (_o=0x56128acc6df0 <mempool+56>, _c=128, _id=<optimized out>, _a=0x7ff6947c1f5c <__lll_lock_wait+28>)
       at qt/moc_clientmodel.cpp:140
   [#11](/bitcoin-bitcoin/11/) 0x00007ff6952f75e9 in QMetaObject::activate (sender=sender@entry=0x56128b71a310, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=0, 
       argv=argv@entry=0x0) at kernel/qobject.cpp:3740
   [#12](/bitcoin-bitcoin/12/) 0x00007ff6952f7f07 in QMetaObject::activate (sender=sender@entry=0x56128b71a310, m=m@entry=0x7ff695515760 <QTimer::staticMetaObject>, 
       local_signal_index=local_signal_index@entry=0, argv=argv@entry=0x0) at kernel/qobject.cpp:3602
   [#13](/bitcoin-bitcoin/13/) 0x00007ff695371780 in QTimer::timeout (this=this@entry=0x56128b71a310) at .moc/moc_qtimer.cpp:198
   [#14](/bitcoin-bitcoin/14/) 0x00007ff695303f78 in QTimer::timerEvent (this=0x56128b71a310, e=<optimized out>) at kernel/qtimer.cpp:254
   [#15](/bitcoin-bitcoin/15/) 0x00007ff6952f8453 in QObject::event (this=0x56128b71a310, e=<optimized out>) at kernel/qobject.cpp:1285
   [#16](/bitcoin-bitcoin/16/) 0x00007ff695c3ab8c in QApplicationPrivate::notify_helper (this=<optimized out>, receiver=0x56128b71a310, e=0x7fff8192ca80) at kernel/qapplication.cpp:3799
   [#17](/bitcoin-bitcoin/17/) 0x00007ff695c42341 in QApplication::notify (this=0x7fff8192d6b0, receiver=0x56128b71a310, e=0x7fff8192ca80) at kernel/qapplication.cpp:3556
   [#18](/bitcoin-bitcoin/18/) 0x00007ff6952cb9e0 in QCoreApplication::notifyInternal2 (receiver=0x56128b71a310, event=event@entry=0x7fff8192ca80) at kernel/qcoreapplication.cpp:988
   [#19](/bitcoin-bitcoin/19/) 0x00007ff69531efee in QCoreApplication::sendEvent (event=0x7fff8192ca80, receiver=<optimized out>)
       at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:231
   [#20](/bitcoin-bitcoin/20/) QTimerInfoList::activateTimers (this=0x56128b8c6940) at kernel/qtimerinfo_unix.cpp:644
   [#21](/bitcoin-bitcoin/21/) 0x00007ff69531f511 in timerSourceDispatch (source=<optimized out>) at kernel/qeventdispatcher_glib.cpp:182
   [#22](/bitcoin-bitcoin/22/) 0x00007ff69068f7f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#23](/bitcoin-bitcoin/23/) 0x00007ff69068fa60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#24](/bitcoin-bitcoin/24/) 0x00007ff69068fb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#25](/bitcoin-bitcoin/25/) 0x00007ff69532004f in QEventDispatcherGlib::processEvents (this=0x56128b8c6720, flags=...) at kernel/qeventdispatcher_glib.cpp:423
   [#26](/bitcoin-bitcoin/26/) 0x00007ff6952c99ca in QEventLoop::exec (this=this@entry=0x7fff8192ccd0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
   [#27](/bitcoin-bitcoin/27/) 0x00007ff695e290e7 in QDialog::exec (this=0x7fff8192cd58) at dialogs/qdialog.cpp:552
   [#28](/bitcoin-bitcoin/28/) 0x000056128a016d55 in BitcoinGUI::message (this=0x56128b9167c0, title=..., message=..., style=<optimized out>, ret=0x7ff63e702e8b) at qt/bitcoingui.cpp:925
   [#29](/bitcoin-bitcoin/29/) 0x000056128a0b9403 in BitcoinGUI::qt_static_metacall (_o=0x56128b9167c0, _c=<optimized out>, _id=<optimized out>, _a=0x7ff63e702920)
       at qt/moc_bitcoingui.cpp:216
   [#30](/bitcoin-bitcoin/30/) 0x00007ff6952f8499 in QObject::event (this=this@entry=0x56128b9167c0, e=e@entry=0x7ff614534610) at kernel/qobject.cpp:1263
   [#31](/bitcoin-bitcoin/31/) 0x00007ff695c8246b in QWidget::event (this=this@entry=0x56128b9167c0, event=event@entry=0x7ff614534610) at kernel/qwidget.cpp:9225
   [#32](/bitcoin-bitcoin/32/) 0x00007ff695d8192b in QMainWindow::event (this=0x56128b9167c0, event=0x7ff614534610) at widgets/qmainwindow.cpp:1543
   [#33](/bitcoin-bitcoin/33/) 0x00007ff695c3ab8c in QApplicationPrivate::notify_helper (this=<optimized out>, receiver=0x56128b9167c0, e=0x7ff614534610) at kernel/qapplication.cpp:3799
   [#34](/bitcoin-bitcoin/34/) 0x00007ff695c42341 in QApplication::notify (this=0x7fff8192d6b0, receiver=0x56128b9167c0, e=0x7ff614534610) at kernel/qapplication.cpp:3556
   [#35](/bitcoin-bitcoin/35/) 0x00007ff6952cb9e0 in QCoreApplication::notifyInternal2 (receiver=0x56128b9167c0, event=event@entry=0x7ff614534610) at kernel/qcoreapplication.cpp:988
   [#36](/bitcoin-bitcoin/36/) 0x00007ff6952ce16d in QCoreApplication::sendEvent (event=0x7ff614534610, receiver=<optimized out>)
       at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:231
   [#37](/bitcoin-bitcoin/37/) QCoreApplicationPrivate::sendPostedEvents (receiver=receiver@entry=0x0, event_type=event_type@entry=0, data=0x56128b821ce0) at kernel/qcoreapplication.cpp:1649
   [#38](/bitcoin-bitcoin/38/) 0x00007ff6952ce5d8 in QCoreApplication::sendPostedEvents (receiver=receiver@entry=0x0, event_type=event_type@entry=0) at kernel/qcoreapplication.cpp:1503
   [#39](/bitcoin-bitcoin/39/) 0x00007ff69531fc43 in postEventSourceDispatch (s=0x56128b8c67f0) at kernel/qeventdispatcher_glib.cpp:276
   [#40](/bitcoin-bitcoin/40/) 0x00007ff69068f7f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#41](/bitcoin-bitcoin/41/) 0x00007ff69068fa60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#42](/bitcoin-bitcoin/42/) 0x00007ff69068fb0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
   [#43](/bitcoin-bitcoin/43/) 0x00007ff69532004f in QEventDispatcherGlib::processEvents (this=0x56128b8c6720, flags=...) at kernel/qeventdispatcher_glib.cpp:423
   [#44](/bitcoin-bitcoin/44/) 0x00007ff6952c99ca in QEventLoop::exec (this=this@entry=0x7fff8192d4a0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
   [#45](/bitcoin-bitcoin/45/) 0x00007ff6952d213c in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1261
   [#46](/bitcoin-bitcoin/46/) 0x0000561289ffbb19 in main (argc=<optimized out>, argv=<optimized out>) at qt/bitcoin.cpp:714
   (gdb) 
   

   I suppose it’s possible that this is corruption in something other than the chainstate db, that I only fixed the chainstate db by reindexing and still have old corruption in another db.

   Either way, bitcoin-qt shouldn’t hang like this. It’s not responding to requests to “bitcoin-cli stop” or any other RPC.

2. 
   dooglus commented at 8:13 pm on September 16, 2017: contributor

   I killed the hung process, started it again, let it get to within 4 blocks of the point it crashes, shut it down cleanly, then restarted it again. It crashed again, but got a few blocks further this time, getting to height=485586 as opposed to 485584 last time. Also this time the error dialog showed the error message, and I was able to click it to exit bitcoin-qt cleanly.

   

3. 
   dooglus commented at 8:22 pm on September 16, 2017: contributor

   I set a breakpoint:

   (gdb) br leveldb::Status::Corruption
   Breakpoint 3 at 0x555555acd1a2: leveldb::Status::Corruption. (9 locations)
   

   It was hit:

   Thread 20 "bitcoin-msghand" hit Breakpoint 4, leveldb::ReadBlock (file=<optimized out>, options=..., handle=..., result=0x7fffa6702378)
       at leveldb/table/format.cc:95
   95	      s = Status::Corruption("block checksum mismatch");
   (gdb) where
   [#0](/bitcoin-bitcoin/0/)  leveldb::ReadBlock (file=<optimized out>, options=..., handle=..., result=0x7fffa6702378) at leveldb/table/format.cc:95
   [#1](/bitcoin-bitcoin/1/)  0x0000555555acd9f9 in leveldb::Table::BlockReader (arg=0x7fff8c176830, options=..., index_value=...) at leveldb/table/table.cc:189
   [#2](/bitcoin-bitcoin/2/)  0x0000555555acde21 in leveldb::Table::InternalGet (this=0x7fff8c176830, options=..., k=..., arg=<optimized out>, saver=<optimized out>)
       at leveldb/table/table.cc:241
   [#3](/bitcoin-bitcoin/3/)  0x0000555555abf37c in leveldb::TableCache::Get (this=0x7fffb7560680, options=..., file_number=<optimized out>, file_size=<optimized out>, k=..., arg=0x140, 
       saver=0x7) at leveldb/db/table_cache.cc:115
   [#4](/bitcoin-bitcoin/4/)  0x0000555555ac237a in leveldb::Version::Get (this=<optimized out>, options=..., k=..., value=<optimized out>, stats=<optimized out>)
       at leveldb/db/version_set.cc:408
   [#5](/bitcoin-bitcoin/5/)  0x0000555555ab6bf3 in leveldb::DBImpl::Get (this=0x7fffb7560940, options=..., key=..., value=0x7fffa67029e0) at leveldb/db/db_impl.cc:1141
   [#6](/bitcoin-bitcoin/6/)  0x000055555588f609 in CDBWrapper::Read<(anonymous namespace)::CoinEntry, Coin> (this=<optimized out>, key=..., value=...) at ./dbwrapper.h:233
   [#7](/bitcoin-bitcoin/7/)  CCoinsViewDB::GetCoin (this=0x7fffb755f2e0, outpoint=..., coin=...) at txdb.cpp:62
   [#8](/bitcoin-bitcoin/8/)  0x0000555555a50107 in CCoinsViewBacked::GetCoin (this=<optimized out>, outpoint=..., coin=...) at coins.cpp:26
   [#9](/bitcoin-bitcoin/9/)  0x000055555575c077 in CCoinsViewErrorCatcher::GetCoin (this=0x7fff8c000000, outpoint=..., coin=...) at init.cpp:141
   [#10](/bitcoin-bitcoin/10/) 0x0000555555a50462 in CCoinsViewCache::FetchCoin (this=0x7fffb75641b0, outpoint=...) at coins.cpp:48
   [#11](/bitcoin-bitcoin/11/) 0x0000555555a50549 in CCoinsViewCache::GetCoin (this=0x7fff8c000000, outpoint=..., coin=...) at coins.cpp:61
   [#12](/bitcoin-bitcoin/12/) 0x0000555555a50462 in CCoinsViewCache::FetchCoin (this=0x7fffa6704440, outpoint=...) at coins.cpp:48
   [#13](/bitcoin-bitcoin/13/) 0x0000555555a50b03 in CCoinsViewCache::HaveCoin (this=0x7fff8c000000, outpoint=...) at coins.cpp:129
   [#14](/bitcoin-bitcoin/14/) 0x0000555555a51254 in CCoinsViewCache::HaveInputs (this=<optimized out>, tx=...) at coins.cpp:240
   [#15](/bitcoin-bitcoin/15/) 0x00005555558d3210 in ConnectBlock (block=..., state=..., pindex=<optimized out>, view=..., chainparams=..., fJustCheck=<optimized out>) at validation.cpp:1763
   [#16](/bitcoin-bitcoin/16/) 0x00005555558c676e in ConnectTip (state=..., chainparams=..., pindexNew=<optimized out>, connectTrace=..., disconnectpool=..., pblock=...)
       at validation.cpp:2209
   [#17](/bitcoin-bitcoin/17/) ActivateBestChainStep (state=..., chainparams=..., pindexMostWork=<optimized out>, pblock=..., fInvalidFound=<optimized out>, connectTrace=...)
       at validation.cpp:2354
   [#18](/bitcoin-bitcoin/18/) 0x00005555558c5ad4 in ActivateBestChain (state=..., chainparams=..., pblock=...) at validation.cpp:2454
   [#19](/bitcoin-bitcoin/19/) 0x00005555558cff58 in ProcessNewBlock (chainparams=..., pblock=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CBlock, std::allocator<CBlock>, (__gnu_cxx::_Lock_policy)2>'
   warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CBlock, std::allocator<CBlock>, (__gnu_cxx::_Lock_policy)2>'
   std::shared_ptr (count 6, weak 0) 0x7fff8cd638e0, fForceProcessing=<optimized out>, 
       fNewBlock=0x7fffa6704aaf) at validation.cpp:3199
   [#20](/bitcoin-bitcoin/20/) 0x00005555557beabf in ProcessMessage (pfrom=<optimized out>, strCommand=..., vRecv=..., nTimeReceived=<optimized out>, chainparams=..., 
       connman=<optimized out>, interruptMsgProc=...) at net_processing.cpp:2391
   [#21](/bitcoin-bitcoin/21/) 0x00005555557adf12 in PeerLogicValidation::ProcessMessages (this=0x7fffc8410440, pfrom=0x7fff980008c0, interruptMsgProc=...) at net_processing.cpp:2718
   [#22](/bitcoin-bitcoin/22/) 0x00005555557da037 in non-virtual thunk to PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) () at net_processing.cpp:2643
   [#23](/bitcoin-bitcoin/23/) 0x000055555578967b in CConnman::ThreadMessageHandler (this=<optimized out>) at net.cpp:1995
   [#24](/bitcoin-bitcoin/24/) 0x00005555557a44ba in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=<optimized out>, __t=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
   [#25](/bitcoin-bitcoin/25/) std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
   [#26](/bitcoin-bitcoin/26/) std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=<optimized out>, __args=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
   [#27](/bitcoin-bitcoin/27/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=<optimized out>, 
       __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
   [#28](/bitcoin-bitcoin/28/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
   [#29](/bitcoin-bitcoin/29/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
   [#30](/bitcoin-bitcoin/30/) 0x0000555555758e3d in std::function<void ()>::operator()() const (this=0x7fffa670bd00)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
   [#31](/bitcoin-bitcoin/31/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:309
   [#32](/bitcoin-bitcoin/32/) 0x00005555557a467d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
   [#33](/bitcoin-bitcoin/33/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
       at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
   [#34](/bitcoin-bitcoin/34/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
       this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
   [#35](/bitcoin-bitcoin/35/) 0x00007ffff452ce6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   [#36](/bitcoin-bitcoin/36/) 0x00007ffff56f9494 in start_thread (arg=0x7fffa670c700) at pthread_create.c:333
   [#37](/bitcoin-bitcoin/37/) 0x00007ffff3ca1aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
   (gdb) 
   

   So it does look like my chainstate db is corrupted again doesn’t it? I wonder if there’s something wrong with the code that’s causing this corruption. Should the db be able to get corrupted from random crashes?

4. 
   dooglus commented at 8:31 pm on September 16, 2017: contributor

   I have two separate Bitcoin data directories. One is a full node (and that’s the one which is failing) and the other is pruned (prune=600).

   Is it safe to copy the chainstate folder from the pruned datadir into the full node’s datadir to fix this corruption issue? Or do I need to run the >24h chainstate reindex thing again?

   Edit: it appears to have worked. I did this:

   • made a bootstrap.dat file that I used to sync the pruned node to the point where it returns the same for ‘getblockcount’ as the corrupted full node gives
   • stopped both nodes
   • deleted the corrupted chainstate folder
   • copied the pruned node’s chainstate folder into the full node

   Then when I started the full node, it was able to continue syncing past the point it was previously failing.

   So that seems like a good cheap way of recovering from chainstate db corruption, if you don’t mind wasting the extra ~4GB of space for an extra pruned node. It can sync from the full node, so needn’t add to your bandwidth requirements.

5. 
   jlegoupil commented at 4:37 pm on September 18, 2017: none

   I experienced the same issue with my pruned node on a low end OSX computer on block #470915.

   I shut down Bitcoin QT and started it again »> problem remained. I shut down my computer, restarted it, started Bitcoin QT again »> problem has resolved itself.

   My debug log: 2017-09-18 04:46:34 UpdateTip: new best=000000000000000000051d8e3a33e4da2fdf008d35cf161382a5639e66b57c4a height=470914 version=0x20000000 log2_work=86.584992 tx=231183711 date=‘2017-06-12 06:50:28’ progress=0.903516 cache=69.4MiB(516833txo) 2017-09-18 04:46:34 UpdateTip: new best=00000000000000000004fcac1e558d2d6031d73b14db5cf67916f712081912e7 height=470915 version=0x20000002 log2_work=86.585028 tx=231184966 date=‘2017-06-12 06:50:36’ progress=0.903521 cache=69.9MiB(521248txo) 2017-09-18 04:46:35 LevelDB read failure: Corruption: block checksum mismatch 2017-09-18 04:46:35 Corruption: block checksum mismatch 2017-09-18 04:47:18 version handshake timeout from 5 2017-09-18 04:47:19 version handshake timeout from 6

6. 
   dooglus commented at 4:51 pm on September 18, 2017: contributor

   @jlegoupil what version of Bitcoin Core were you using?

   I don’t see why restarting your OS would help. That’s strange.

7. 
   jlegoupil commented at 5:02 pm on September 18, 2017: none

   I’m using Bitcoin Core 0.15.0.0
8. 
   dooglus commented at 8:57 pm on September 23, 2017: contributor

   It happened again to me, this time on shutdown:

   2017-09-23 20:54:21 connect() to 59.127.26.8:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:21 connect() to 220.253.70.199:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:22 connect() to 77.181.226.122:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:22 connect() to 35.185.131.115:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:23 connect() to [2001:0:9d38:6ab8:3c62:f87e:95d8:bcd2]:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:23 connect() to 139.59.149.67:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:24 connect() to 188.195.24.170:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:24 connect() to 24.90.192.28:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:25 connect() to [2001:0:9d38:78cf:20d3:3c09:a9d2:703a]:8333 failed: Network is unreachable (101)
   2017-09-23 20:54:25 addcon thread exit
   2017-09-23 20:54:25 scheduler thread interrupt
   2017-09-23 20:54:25 opencon thread exit
   2017-09-23 20:54:25 Shutdown: In progress...
   2017-09-23 20:54:25 net thread exit
   2017-09-23 20:54:25 msghand thread exit
   2017-09-23 20:54:25 Dumped mempool: 0.005588s to copy, 0.126193s to dump
   2017-09-23 20:54:25 Corruption: block checksum mismatch
   2017-09-23 20:54:25 *** System error while flushing: Database corrupted
   2017-09-23 20:54:33 Corruption: block checksum mismatch
   2017-09-23 20:54:33 *** System error while flushing: Database corrupted
   2017-09-23 20:54:34 GUI: QXcbConnection: XCB error: 3 (BadWindow), sequence: 7465, resource id: 11515671, major code: 40 (TranslateCoords), minor code: 0
   2017-09-23 20:54:34 Shutdown: done
   

   I was offline at the time, which explains all the “Network is unreachable” errors. Since the last time I fixed the chainstate db I never had Bitcoin crash, but I did have the laptop fail to resume from a suspend state once and had to do a cold reset. I guess that could have corrupted the db, but this really seems to be so much more common in 0.15 than it was before. This is the 3rd chainstate db corruption I’ve had with 0.15 compared to 6 years of corruption-free operation before 0.15.

9. 
   dooglus commented at 5:59 pm on October 22, 2017: contributor

   I’m still getting this chainstate database corruption fairly regularly.

   Since my last comment here it has happened three more times, on Oct 11th, 12th, and 22nd.

   This seems like a serious issue to me, because once the chainstate database is corrupted it requires a full blockchain rescan to repair it. For a pruned node that means starting the IBD from scratch, and will likely lead to lots of full nodes going offline due to inertia.

10. 
    dooglus commented at 6:15 pm on October 22, 2017: contributor

    One thing occurs to me. If I leave my laptop unattended, the screen locks after 5 minutes. It used to be that when I unlocked the screen any browser tabs I had open would have been updating behind the scenes. But for the last couple of months I’ve noticed that having the screen locked seems to freeze things. I unlock the screen, and the price charts haven’t updated since I left and such like.

    I don’t know what would cause that, but I’m wondering if it could also be causing the chainstate corruption I’m seeing if I leave bitcoin-qt running while the screen is locked.

    Just a thought.

11. 
    dooglus commented at 3:41 pm on October 26, 2017: contributor

    My chainstate database was corrupted again last night when I tried to close bitcoin-qt. From the debug.log:

    2017-10-25 23:38:25 opencon thread exit
    2017-10-25 23:38:25 addcon thread exit
    2017-10-25 23:38:25 net thread exit
    2017-10-25 23:38:25 msghand thread exit
    2017-10-25 23:38:25 scheduler thread interrupt
    2017-10-25 23:38:25 Shutdown: In progress...
    2017-10-25 23:38:26 Dumped mempool: 0.001688s to copy, 0.191935s to dump
    2017-10-25 23:38:27 Corruption: block checksum mismatch
    2017-10-25 23:38:27 *** System error while flushing: Database corrupted
    2017-10-25 23:38:31 Corruption: block checksum mismatch
    2017-10-25 23:38:31 *** System error while flushing: Database corrupted
    2017-10-25 23:38:32 Shutdown: done
    

    Each time it has happened recently, I have synced up my pruned node to the same block that the corrupted node is at, renamed the corrupted node’s chainstate directory, then copied the pruned node’s chainstate directory into the corrupted node’s .bitcoin directory. That has left me with a bunch of corrupted chainstate directories:

    ~/.bitcoin$ du -m chain*
    2812	chainstate
    2757	chainstate.bad
    2779	chainstate.bad2
    2800	chainstate.bad3
    2837	chainstate.bad4
    2842	chainstate.bad5
    ~/.bitcoin$ ls -ld chainstate*
    drwx------ 2 chris chris 61440 Oct 26 08:34 chainstate
    drwx------ 2 chris chris 69632 Sep 16 14:50 chainstate.bad
    drwx------ 2 chris chris 61440 Oct 11 07:29 chainstate.bad2
    drwx------ 2 chris chris 65536 Oct 12 22:10 chainstate.bad3
    drwx------ 2 chris chris 61440 Oct 22 09:39 chainstate.bad4
    drwx------ 2 chris chris 65536 Oct 26 00:19 chainstate.bad5
    

    I can make these available if that would be helpful (and if it is safe to do so).

12. 
    ryanofsky commented at 4:26 pm on October 26, 2017: member

    We should change the order of the log print & blocking message box calls so the corruption errors are printed in the log if the message box handler hangs:

     0--- a/src/init.cpp
     1+++ b/src/init.cpp
     2@@ -140,8 +140,8 @@ public:
     3         try {
     4             return CCoinsViewBacked::GetCoin(outpoint, coin);
     5         } catch(const std::runtime_error& e) {
     6-            uiInterface.ThreadSafeMessageBox(_("Error reading from database, shutting down."), "", CClientUIInterface::MSG_ERROR);
     7             LogPrintf("Error reading from database: %s\n", e.what());
     8+            uiInterface.ThreadSafeMessageBox(_("Error reading from database, shutting down."), "", CClientUIInterface::MSG_ERROR);
     9             // Starting the shutdown sequence and returning false to the caller would be
    10             // interpreted as 'entry not found' (as opposed to unable to read data), and
    11             // could lead to invalid interpretation. Just exit immediately, as we can't
    

    EDIT: Actually, it doesn’t matter very much, since the full leveldb error is logged right before the exception is thrown: https://github.com/bitcoin/bitcoin/blob/50fae68d416b4b8ec4ca192923dfd5ae9ea42773/src/dbwrapper.cpp#L203

13. 
    dooglus commented at 4:29 pm on October 26, 2017: contributor

    I only saw the message box hang once. It happened the first time the database was corrupted, and never since.
14. 
    gmaxwell commented at 4:37 pm on October 26, 2017: contributor

    seems to be so much more common in 0.15 than it was before

    So this was also happening before?

    A checksum error is a really really cut and dry condition: The system reads a record off the disk and the CRC32 didn’t match. We have never, so far, had a bug in the software which turned out to cause these errors: the cause has always been disk corruption due to a failing disk, faulty ram, or an overheating cpu. The latter of which is especially implicated in laptops (and other small systems) since most brands really just cannot handle being run at full intensity. If prior versions did it as well and 0.15 is more frequent about it, it could just be because 0.15 hits the machine more intensely.

    But it would be good to eliminate what other causes we can.. What exact cpu model and compiler version are you using? Have you run Memtest86 on this system?

15. 
    ryanofsky commented at 4:54 pm on October 26, 2017: member

    Might be worth trying to store your chainstate on a zfs or btrfs partition to see if any corruption errors can be detected at the filesystem level. COW snapshots could also help you recover more quickly if there is corruption above the filesystem level.
16. 
    dooglus commented at 5:23 pm on October 26, 2017: contributor

    So this was also happening before?

    I have had to run a full reindex maybe twice before 0.15, but both times were after I had had to do a cold reboot of my laptop while Core was running. I wasn’t surprised that doing a cold reboot would cause problems.

    Since running 0.15 the chainstate database has become corrupted 7 times in 6 weeks. Most if not all of these happened without any cold restart happening.

    the cause has always been disk corruption due to a failing disk, faulty ram, or an overheating cpu. The latter of which is especially implicated in laptops (and other small systems) since most brands really just cannot handle being run at full intensity. If prior versions did it as well and 0.15 is more frequent about it, it could just be because 0.15 hits the machine more intensely.

    All 7 times it has been the chainstate db, not the block index db that was corrupted. I am running a laptop, but it isn’t running at full intensity. The CPU isn’t often running flat out.

    But it would be good to eliminate what other causes we can.. What exact cpu model and compiler version are you using?

    According to /proc/cpuinfo, it’s a 4 core i7-5500U:

    $ grep 'model name' /proc/cpuinfo | uniq -c
          4 model name	: Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz
    

    The compiler version might be interesting. I switched from using gcc to clang a few months back when I was trying to debug a crash I get on startup about 1% of the time. I still see crashes when starting bitcoin-qt very occasionally. I saw one today in fact but I’ve given up trying to debug it.

    $ clang++ -v
    clang version 3.8.1-24 (tags/RELEASE_381/final)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /usr/bin
    Found candidate GCC installation: /usr/bin/../lib/gcc/i586-linux-gnu/4.9
    Found candidate GCC installation: /usr/bin/../lib/gcc/i586-linux-gnu/4.9.2
    Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/6.3.0
    Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
    Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8.4
    Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
    Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9.2
    Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0
    Found candidate GCC installation: /usr/lib/gcc/i586-linux-gnu/4.9
    Found candidate GCC installation: /usr/lib/gcc/i586-linux-gnu/4.9.2
    Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6.3.0
    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2
    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.3.0
    Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    

    Have you run Memtest86 on this system?

    I have, maybe 6 months ago. I let it run for a few hours and it didn’t find any problems. I could try letting it run to completion. I have replaced the RAM since then actually so I should run it again anyway. Will report back.

17. 
    dooglus commented at 5:24 pm on October 26, 2017: contributor

    Might be worth trying to store your chainstate on a zfs or btrfs partition to see if any corruption errors can be detected at the filesystem level. COW snapshots could also help you recover more quickly if there is corruption above the filesystem level.

    I have no experience of zfs or btrfs. Is one better than the other for this purpose? Do I need to set any particular configuration to detect filesystem corruption? I’m willing to experiment.

18. 
    TheBlueMatt commented at 5:35 pm on October 26, 2017: member

    Either should work fine, though btrfs is slightly better-supported in Linux-land, even if its newer/a bit less stable in strange edge-cases.
19. 
    dooglus commented at 5:39 pm on October 26, 2017: contributor

    Is it safe to assume that if bitcoin starts up successfully with the -forcecompactdb flag then the chainstate database currently doesn’t have any checksum errors?
20. 
    ryanofsky commented at 6:14 pm on October 26, 2017: member

    I have no experience of zfs or btrfs. Is one better than the other for this purpose? Do I need to set any particular configuration to detect filesystem corruption? I’m willing to experiment

    Personally I’ve only used btrfs, not zfs. But neither require configuration to detect corruption, you just need to create a partition, store data on it, and any errors should show up in your system log (and be returned to applications). In btrfs, if you want to create a COW snapshot of a file or directory, you can run cp -a --reflink SRC DEST.

    Is it safe to assume that if bitcoin starts up successfully with the -forcecompactdb flag then the chainstate database currently doesn’t have any checksum errors?

    I would guess not. From https://github.com/google/leveldb/blob/master/doc/impl.md, it doesn’t seem like a compaction will necessarily rewrite the whole database (for example if it is already compacted).

    Maybe one way to manually verify there are no checksum errors in the database would be to add error reporting to leveldb dump code in src/leveldb/db/dumpfile.cc, which can be run with:

    0make -C src/leveldb out-static/leveldbutil
    1src/leveldb/out-static/leveldbutil dump ~/.bitcoin/chainstate/*
    

21. 
    dooglus commented at 6:20 pm on October 26, 2017: contributor

    I don’t have any unpartitioned disk space, so I made a file within an existing partition, and made a btrfs filesystem in there. Hopefully that’s good enough:

    chris@chris:~/.bitcoin$ dd if=/dev/zero of=chainstate.btrfs bs=1M count=4096
    4096+0 records in
    4096+0 records out
    4294967296 bytes (4.3 GB, 4.0 GiB) copied, 40.403 s, 106 MB/s
    chris@chris:~/.bitcoin$ mkfs.btrfs chainstate.btrfs
    btrfs-progs v4.7.3
    See http://btrfs.wiki.kernel.org for more information.
    
    Label:              (null)
    UUID:               
    Node size:          16384
    Sector size:        4096
    Filesystem size:    4.00GiB
    Block group profiles:
      Data:             single            8.00MiB
      Metadata:         DUP             204.75MiB
      System:           DUP               8.00MiB
    SSD detected:       no
    Incompat features:  extref, skinny-metadata
    Number of devices:  1
    Devices:
       ID        SIZE  PATH
        1     4.00GiB  chainstate.btrfs
    
    chris@chris:~/.bitcoin$ bc stop
    Bitcoin server stopping
    chris@chris:~/.bitcoin$ mv chainstate chainstate.good6
    chris@chris:~/.bitcoin$ mkdir chainstate
    chris@chris:~/.bitcoin$ sudo mount chainstate.btrfs chainstate
    chris@chris:~/.bitcoin$ sudo chown chris:chris chainstate
    chris@chris:~/.bitcoin$ df chainstate
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop0      4.0G   17M  3.6G   1% /home/chris/.bitcoin/chainstate
    chris@chris:~/.bitcoin$ (cd ~/.bitcoin/chainstate.good6/ && tar cf - .) | (cd ~/.bitcoin/chainstate/ && tar xf -)
    chris@chris:~/.bitcoin$ df chainstate
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop0      4.0G  2.5G  1.3G  67% /home/chris/.bitcoin/chainstate
    

22. 
    dooglus commented at 0:04 am on October 27, 2017: contributor

    Have you run Memtest86 on this system?

    I have, maybe 6 months ago. I let it run for a few hours and it didn’t find any problems. I could try letting it run to completion. I have replaced the RAM since then actually so I should run it again anyway. Will report back.

    I ran it for 2 hours this morning. It got through about 1.5 test cycles without finding any error.

    I tried running it on all 4 cores at once (pressing F2 when memtest86 starts up, I think) and it locked up the computer after about 5 minutes. It remember it used to do that before I upgraded the RAM too. I don’t know if that’s cause for concern or not.

23. 
    dooglus commented at 6:17 pm on October 29, 2017: contributor

    @ryanofsky > Might be worth trying to store your chainstate on a zfs or btrfs partition to see if any corruption errors can be detected at the filesystem level.

    I just had the same crash again. This is the first crash since I moved chainstate onto a btrfs filesystem.

    debug.log:

    2017-10-29 13:19:16 Pre-allocating up to position 0x5000000 in blk01008.dat
    2017-10-29 13:19:16 Pre-allocating up to position 0xa00000 in rev01008.dat
    2017-10-29 13:19:16 UpdateTip: new best=00000000000000000092bb686c21f129852b3f24aff05113d02b708c9d6c11f6 height=492209 version=0x20000000 log2_work=87.373165 tx=266570069 date='2017-10-29 13:19:00' progress=1.000000 cache=92.8MiB(691920txo) warning='2 of last 100 blocks have unexpected version'
    2017-10-29 13:23:24 UpdateTip: new best=000000000000000000500829275e29849bb2a64a38d2f112d81c8fba43e620b0 height=492210 version=0x20000000 log2_work=87.37321 tx=266572022 date='2017-10-29 13:22:51' progress=1.000000 cache=93.4MiB(697320txo) warning='2 of last 100 blocks have unexpected version'
    2017-10-29 13:23:58 UpdateTip: new best=0000000000000000001d8f1d13239fb8544698ca2bb9bc1cdd2883fc2580556c height=492211 version=0x20000000 log2_work=87.373255 tx=266574358 date='2017-10-29 13:23:23' progress=1.000000 cache=93.9MiB(701053txo) warning='2 of last 100 blocks have unexpected version'
    2017-10-29 17:31:09 socket sending timeout: 14654s
    2017-10-29 17:31:09 socket sending timeout: 14654s
    2017-10-29 17:31:09 socket sending timeout: 14654s
    2017-10-29 17:31:09 socket sending timeout: 14654s
    2017-10-29 17:31:09 socket sending timeout: 14654s
    2017-10-29 17:31:09 socket sending timeout: 14654s
    2017-10-29 17:31:09 socket sending timeout: 14654s
    2017-10-29 17:31:09 socket sending timeout: 14656s
    2017-10-29 17:31:09 connect() to 178.49.137.89:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:10 connect() to 156.11.1.152:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:11 connect() to 18.111.110.151:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:11 connect() to [2002:d941:d96c:8000:0:5efe:a00:7d6]:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:12 connect() to 52.14.94.187:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:12 connect() to 46.59.78.181:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:13 connect() to 92.177.51.206:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:13 connect() to 5.189.165.102:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:14 connect() to [2a04:2180:0:2::77]:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:14 connect() to 71.178.245.103:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:15 connect() to 59.125.8.143:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:15 connect() to 176.159.164.10:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:16 connect() to 130.194.84.251:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:16 connect() to 190.95.220.138:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:17 connect() to 138.201.33.232:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:18 connect() to [2a01:4f8:10a:1d8f::2]:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:18 connect() to 97.87.213.112:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:19 connect() to 52.59.146.184:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:19 connect() to 83.135.85.203:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:20 connect() to 69.162.111.228:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:23 connect() to 212.164.233.103:8333 failed after select(): Connection refused (111)
    2017-10-29 17:31:30 connect() to [2001:0:9d38:6abd:245a:a009:fac1:d805]:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:42 receive version message: /Satoshi:0.14.2/: version 70015, blocks=492233, us=<me>:8284, peer=55
    2017-10-29 17:31:50 receive version message: /Satoshi:0.15.99/: version 70015, blocks=492233, us=<me>:13696, peer=56
    2017-10-29 17:31:50 receive version message: /Satoshi:0.13.2/: version 70015, blocks=492233, us=<me>:27749, peer=57
    2017-10-29 17:31:50 receive version message: /Satoshi:0.14.2/: version 70015, blocks=492233, us=<me>:16359, peer=58
    2017-10-29 17:31:50 connect() to [2001:0:9d38:6abd:1076:1102:aead:172]:8333 failed: Network is unreachable (101)
    2017-10-29 17:31:51 receive version message: /Satoshi:0.14.2/: version 70015, blocks=492233, us=<me>:22722, peer=60
    2017-10-29 17:31:55 socket recv error Connection reset by peer (104)
    2017-10-29 17:32:11 connect() to [2602:306:39de:ea90:c555:2374:2256:6c3d]:8333 failed: Network is unreachable (101)
    2017-10-29 17:32:36 UpdateTip: new best=00000000000000000065aedcb67e1ca2230c9c06a667f11f0a1528839f3c068b height=492212 version=0x20000000 log2_work=87.3733 tx=266576702 date='2017-10-29 13:42:56' progress=0.999840 cache=94.7MiB(707504txo) warning='2 of last 100 blocks have unexpected version'
    2017-10-29 17:32:36 receive version message: /Satoshi:0.15.0.1/: version 70015, blocks=492233, us=<me>:26163, peer=61
    2017-10-29 17:33:19 UpdateTip: new best=0000000000000000000bd68df3f9398932b1535aea34649358e24b5e2b41a9f6 height=492213 version=0x20000000 log2_work=87.373345 tx=266579252 date='2017-10-29 13:43:39' progress=0.999840 cache=95.1MiB(710427txo) warning='2 of last 100 blocks have unexpected version'
    2017-10-29 17:33:19 version handshake timeout from 61
    2017-10-29 17:33:19 version handshake timeout from 62
    2017-10-29 17:33:23 connect() to 51.174.51.66:8333 failed after select(): Connection refused (111)
    2017-10-29 17:33:47 UpdateTip: new best=0000000000000000002565c85bb351e222fd34eb68f0712261689ec7c5f05d33 height=492214 version=0x20000000 log2_work=87.37339 tx=266581963 date='2017-10-29 14:12:23' progress=0.999859 cache=95.6MiB(714560txo) warning='2 of last 100 blocks have unexpected version'
    2017-10-29 17:33:47 version handshake timeout from 63
    2017-10-29 17:34:12 UpdateTip: new best=0000000000000000001f0ffad4402279360668ba5b071ee3206971199bd732f7 height=492215 version=0x20000000 log2_work=87.373435 tx=266585112 date='2017-10-29 14:43:25' progress=0.999881 cache=96.2MiB(719487txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:34:12 receive version message: /Satoshi:0.15.0/: version 70015, blocks=492234, us=<me>:26749, peer=64
    2017-10-29 17:34:25 version handshake timeout from 64
    2017-10-29 17:34:37 UpdateTip: new best=0000000000000000006d7c1822119613361d0fdb4594ec68990a37d1c42472da height=492216 version=0x20000000 log2_work=87.37348 tx=266587484 date='2017-10-29 14:51:19' progress=0.999886 cache=96.6MiB(723194txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:34:37 receive version message: /Satoshi:0.13.1/: version 70014, blocks=492234, us=<me>:6057, peer=65
    2017-10-29 17:35:00 Pre-allocating up to position 0xb00000 in rev01008.dat
    2017-10-29 17:35:00 UpdateTip: new best=00000000000000000046d5cabb24f80c5f60e614cb9450b14f5f86548dbac7af height=492217 version=0x20000000 log2_work=87.373525 tx=266589807 date='2017-10-29 14:51:45' progress=0.999886 cache=97.2MiB(728063txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:35:00 version handshake timeout from 65
    2017-10-29 17:35:00 receive version message: /Satoshi:0.14.2/: version 70015, blocks=492234, us=<me>:6139, peer=66
    2017-10-29 17:35:01 connect() to 85.187.183.13:8333 failed after select(): Connection refused (111)
    2017-10-29 17:35:07 connect() to 157.193.121.83:8333 failed after select(): Connection refused (111)
    2017-10-29 17:35:08 connect() to [2003:f1:5bc3:500:7d68:661:1815:5090]:8333 failed: Network is unreachable (101)
    2017-10-29 17:35:28 UpdateTip: new best=0000000000000000001a5530ee669c4438fa706d53f8e5135f8256488df66ce2 height=492218 version=0x20000000 log2_work=87.373569 tx=266591778 date='2017-10-29 14:55:21' progress=0.999888 cache=97.8MiB(732858txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:35:28 version handshake timeout from 66
    2017-10-29 17:35:29 receive version message: /Satoshi:0.14.2/: version 70015, blocks=492234, us=<me>:20734, peer=68
    2017-10-29 17:35:29 Pre-allocating up to position 0x6000000 in blk01008.dat
    2017-10-29 17:35:36 UpdateTip: new best=000000000000000000882bdeda9d256d4c172bb4a8b369f1772af19fcb31451d height=492219 version=0x20000000 log2_work=87.373614 tx=266594510 date='2017-10-29 15:20:12' progress=0.999906 cache=98.2MiB(735883txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:35:40 connect() to 31.28.161.19:8333 failed after select(): Connection refused (111)
    2017-10-29 17:35:43 UpdateTip: new best=0000000000000000004e85fe4b36a585da072f329619001fb50cdcee7d58437c height=492220 version=0x20000000 log2_work=87.373659 tx=266597133 date='2017-10-29 15:36:37' progress=0.999917 cache=98.7MiB(740448txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:35:49 UpdateTip: new best=000000000000000000a7a14031d4eaac94e2794e0395a7cdef3be6a96c4c8ac1 height=492221 version=0x20000000 log2_work=87.373704 tx=266598920 date='2017-10-29 15:50:43' progress=0.999927 cache=98.7MiB(740177txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:35:49 receive version message: /Satoshi:0.14.2/: version 70015, blocks=492234, us=<me>:12875, peer=69
    2017-10-29 17:35:50 connect() to 192.42.116.16:8333 failed after select(): Connection refused (111)
    2017-10-29 17:35:50 connect() to 217.16.185.175:8333 failed after select(): Connection refused (111)
    2017-10-29 17:36:00 UpdateTip: new best=00000000000000000039cbb36466e124d82478a80a85d65ca36f0fd40064a993 height=492222 version=0x20000000 log2_work=87.373749 tx=266601379 date='2017-10-29 16:03:24' progress=0.999935 cache=99.2MiB(744167txo) warning='1 of last 100 blocks have unexpected version'
    2017-10-29 17:36:11 UpdateTip: new best=00000000000000000032f3507d2598dda323087d1a3ea128f0b8b4912851b096 height=492223 version=0x20000000 log2_work=87.373794 tx=266603427 date='2017-10-29 16:04:26' progress=0.999936 cache=100.1MiB(751940txo)
    2017-10-29 17:36:11 receive version message: /Satoshi:0.15.0.1/: version 70015, blocks=492234, us=<me>:27617, peer=70
    2017-10-29 17:36:31 Pre-allocating up to position 0xc00000 in rev01008.dat
    2017-10-29 17:36:31 UpdateTip: new best=0000000000000000003277b639e56dffe2b4e60d18aeedb1fe8b7e4256b2a526 height=492224 version=0x20000000 log2_work=87.373839 tx=266605826 date='2017-10-29 16:07:26' progress=0.999938 cache=100.6MiB(756143txo)
    2017-10-29 17:36:43 connect() to [2001:0:9d38:6ab8:1810:10a5:a051:8a63]:8333 failed: Network is unreachable (101)
    2017-10-29 17:36:45 UpdateTip: new best=00000000000000000083ebf6203ebab0082124b89e4be4eeaf1d05882758ce5e height=492225 version=0x20000000 log2_work=87.373884 tx=266607828 date='2017-10-29 16:10:00' progress=0.999939 cache=101.3MiB(761279txo)
    2017-10-29 17:36:50 UpdateTip: new best=000000000000000000bcc54ba6076238c9530fcc28c238d37ff8de13809c96fc height=492226 version=0x20000000 log2_work=87.373929 tx=266610888 date='2017-10-29 16:36:27' progress=0.999958 cache=101.7MiB(765107txo)
    2017-10-29 17:36:57 UpdateTip: new best=0000000000000000006f70ce5e670cb2eb81273a0084967ab3908dfa1f9d3b4b height=492227 version=0x20000000 log2_work=87.373974 tx=266613180 date='2017-10-29 16:44:11' progress=0.999963 cache=102.3MiB(769473txo)
    2017-10-29 17:37:05 UpdateTip: new best=000000000000000000654c6a0ac1fd849176bfe255fc6798f1b31647b0058e15 height=492228 version=0x20000000 log2_work=87.374018 tx=266615764 date='2017-10-29 16:48:42' progress=0.999966 cache=103.1MiB(776461txo)
    2017-10-29 17:37:11 UpdateTip: new best=000000000000000000212978b06e687f8b3a18d952bf049bbad4e08f31f6067f height=492229 version=0x20000000 log2_work=87.374063 tx=266618250 date='2017-10-29 16:51:42' progress=0.999968 cache=103.7MiB(781118txo)
    2017-10-29 17:37:18 UpdateTip: new best=000000000000000000394c6ed382de4b1ca6f1925c5c3aa1454356aec8f9bf99 height=492230 version=0x20000000 log2_work=87.374108 tx=266620126 date='2017-10-29 16:56:29' progress=0.999972 cache=104.1MiB(784164txo)
    2017-10-29 17:37:19 Pre-allocating up to position 0xd00000 in rev01008.dat
    2017-10-29 17:37:19 UpdateTip: new best=000000000000000000610ae3e70726f9dd2026f414a1327cb20f66b8b8ead02c height=492231 version=0x20000000 log2_work=87.374153 tx=266622135 date='2017-10-29 17:04:46' progress=0.999977 cache=104.3MiB(785998txo)
    2017-10-29 17:37:24 UpdateTip: new best=0000000000000000002d7e28f24480fbcecdad62d7f69e2822fcc162ca48e27f height=492232 version=0x20000000 log2_work=87.374198 tx=266624967 date='2017-10-29 17:06:27' progress=0.999978 cache=104.8MiB(789835txo)
    2017-10-29 17:37:25 UpdateTip: new best=000000000000000000a4c6aa4eac92080ce9e8ea94c41c116218477e0fece617 height=492233 version=0x20000000 log2_work=87.374243 tx=266629170 date='2017-10-29 17:07:45' progress=0.999979 cache=104.9MiB(790754txo)
    2017-10-29 17:37:27 UpdateTip: new best=0000000000000000004823ba5cd79517717ab8f9c190c03cb702e9e3432fd81d height=492234 version=0x20000000 log2_work=87.374288 tx=266632276 date='2017-10-29 17:31:59' progress=0.999996 cache=105.4MiB(795421txo)
    2017-10-29 17:37:51 UpdateTip: new best=0000000000000000005e6dd53b92fecd07db6f5668996b95f4d6e41598700ce4 height=492235 version=0x20000000 log2_work=87.374333 tx=266634727 date='2017-10-29 17:36:47' progress=0.999999 cache=106.0MiB(799924txo)
    2017-10-29 17:41:18 connect() to [2001:41d0:1:8135::1]:8333 failed: Network is unreachable (101)
    2017-10-29 17:42:54 Corruption: block checksum mismatch
    2017-10-29 17:42:54 *** System error while flushing: Database corrupted
    2017-10-29 17:43:24 connect() to [2001:0:9d38:6ab8:3018:8f0:26e4:7c97]:8333 failed: Network is unreachable (101)
    
    2017-10-29 17:45:26 GUI: QXcbConnection: XCB error: 3 (BadWindow), sequence: 17984, resource id: 11739945, major code: 40 (TranslateCoords), minor code: 0
    2017-10-29 17:45:26 scheduler thread interrupt
    2017-10-29 17:45:26 msghand thread exit
    2017-10-29 17:45:26 opencon thread exit
    2017-10-29 17:45:26 addcon thread exit
    2017-10-29 17:45:26 Shutdown: In progress...
    2017-10-29 17:45:26 net thread exit
    2017-10-29 17:45:30 Dumped mempool: 0.008012s to copy, 0.690163s to dump
    2017-10-29 17:45:30 Corruption: block checksum mismatch
    2017-10-29 17:45:30 *** System error while flushing: Database corrupted
    2017-10-29 17:46:21 Corruption: block checksum mismatch
    2017-10-29 17:46:21 *** System error while flushing: Database corrupted
    2017-10-29 17:46:27 GUI: QXcbConnection: XCB error: 3 (BadWindow), sequence: 18529, resource id: 11745836, major code: 40 (TranslateCoords), minor code: 0
    2017-10-29 17:46:27 Shutdown: done
    

    fsck on the btrfs system:

    10:48:29 chris@chris:~/.bitcoin$ mount | grep chainst
    /home/chris/.bitcoin/chainstate.btrfs on /home/chris/.bitcoin/chainstate type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
    10:48:49 chris@chris:~/.bitcoin$ sudo umount ~/.bitcoin/chainstate
    [sudo] password for chris: 
    10:52:53 chris@chris:~/.bitcoin$ btrfsck --readonly chainstate.btrfs
    Checking filesystem on chainstate.btrfs
    UUID: 24eb317a-d18d-4df4-8cac-a6e0b1e1105c
    checking extents
    checking free space cache
    checking fs roots
    checking csums
    checking root refs
    found 2982662144 bytes used err is 0
    total csum bytes: 2906980
    total tree bytes: 5324800
    total fs tree bytes: 999424
    total extent tree bytes: 196608
    btree space waste bytes: 1492325
    file data blocks allocated: 2977337344
     referenced 2977247232
    10:52:59 chris@chris:~/.bitcoin$ btrfsck --readonly --check-data-csum chainstate.btrfs
    Checking filesystem on chainstate.btrfs
    UUID: 24eb317a-d18d-4df4-8cac-a6e0b1e1105c
    checking extents
    checking free space cache
    checking fs roots
    checking csums
    checking root refs
    found 2982662144 bytes used err is 0
    total csum bytes: 2906980
    total tree bytes: 5324800
    total fs tree bytes: 999424
    total extent tree bytes: 196608
    btree space waste bytes: 1492325
    file data blocks allocated: 2977337344
     referenced 2977247232
    10:54:48 chris@chris:~/.bitcoin$ sudo mount ~/.bitcoin/chainstate.btrfs ~/.bitcoin/chainstate
    [sudo] password for chris: 
    11:05:10 chris@chris:~/.bitcoin$ df chainstate
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop0      4.0G  2.8G  831M  78% /home/chris/.bitcoin/chainstate
    11:05:15 chris@chris:~/.bitcoin$ 
    

    Which file(s) should I look in for any filesystem errors that may have been logged?

    Does this clean fsck rule out the possibility of the corruption being caused by hard drive or filesystem errors?

    Edit: I see that the laptop suspended which I was sleeping, probably when a kitten jumped on the laptop lid at 6:26am, and resumed around the time of the crash:

    ==> auth.log <==
    Oct 29 06:26:50 chris systemd-logind[649]: Lid closed.
    Oct 29 06:26:50 chris systemd-logind[649]: Suspending...
    Oct 29 10:31:09 chris systemd-logind[649]: Lid opened.
    

    I’m currently 7 hours behind UTC, so 10:31 in the auth.log is 17:31 in Bitcoin’s debug.log.

24. 
    dooglus commented at 1:33 am on October 30, 2017: contributor

    I restarted bitcoin-qt after the crash in my immediately previous comment and was able to sync the blockchain without error. I didn’t see any more “Corruption: block checksum mismatch” messages, and everything seemed to go fine. That’s without doing any kind of repair on the database which was earlier reported to be corrupted.

    Something strange is going on.

    I tried running with -forcecompactdb - it took 26 minutes to compact the block index, and another 7 minutes to compact the chainstate, but again managed to compact both without any warnings or errors being reported:

    2017-10-29 19:55:20 Starting database compaction of /home/chris/.bitcoin/blocks/index
    2017-10-29 20:21:00 Finished database compaction of /home/chris/.bitcoin/blocks/index
    2017-10-29 20:21:00 Using obfuscation key for /home/chris/.bitcoin/blocks/index: 0000000000000000
    2017-10-29 20:21:04 LoadBlockIndexDB: last block file = 1008
    2017-10-29 20:21:04 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=103, size=104965446, heights=492143...492245, time=2017-10-29...2017-10-29)
    2017-10-29 20:21:04 Checking all blk files are present...
    2017-10-29 20:21:04 LoadBlockIndexDB: transaction index enabled
    2017-10-29 20:21:04 Opening LevelDB in /home/chris/.bitcoin/chainstate
    2017-10-29 20:21:05 Opened LevelDB successfully
    2017-10-29 20:21:05 Starting database compaction of /home/chris/.bitcoin/chainstate
    2017-10-29 20:28:35 Finished database compaction of /home/chris/.bitcoin/chainstate
    2017-10-29 20:28:35 Using obfuscation key for /home/chris/.bitcoin/chainstate: 5d6a8c01975f3412
    

    I don’t understand why I’m seeing this particular error over and over again but almost nobody else is seeing it.

25. 
    sangaman commented at 4:23 am on October 30, 2017: contributor

    I just got this block checksum mismatch error during the late stages of an initial sync on a raspi 3 running raspian stretch configured for pruning. As far as I can tell it happened out of the blue while it was left to sync - I’d only been checking on the sync progress a couple of times a day. I have a nearly identical setup that’s working fine, but I used 0.14 for the initial sync on there (since upgraded to 0.15). Here’s the relevant portion from debug.log.

     02017-10-29 23:42:39 UpdateTip: new best=000000000000000002df85b73562d01bff8e13389ae99532093a6e0b019eb348 height=422372 version=0x20000000 log2_work=85.04258 tx=144960767 date='2016-07-26 18:11:29' progress=0.542795 cache=43.9MiB(255554txo)
     12017-10-29 23:42:39 socket send error Connection reset by peer (104)
     22017-10-29 23:42:40 receive version message: /Satoshi:0.12.1/: version 70012, blocks=492271, us=69.143.4.96:47562, peer=1217
     32017-10-29 23:42:40 Leaving block file 582: CBlockFileInfo(blocks=160, size=133623360, heights=422316...422733, time=2016-07-26...2016-07-29)
     42017-10-29 23:42:42 Pre-allocating up to position 0x1000000 in blk00583.dat
     52017-10-29 23:42:42 receive version message: /Satoshi:0.15.0.1/: version 70015, blocks=492271, us=[2601:141:201:79a0:2f44:dfe0:5692:275c]:38382, peer=1218
     62017-10-29 23:42:44 Pre-allocating up to position 0x2000000 in blk00583.dat
     72017-10-29 23:42:47 Prune: UnlinkPrunedFiles deleted blk/rev (00553)
     82017-10-29 23:42:48 Corruption: block checksum mismatch
     92017-10-29 23:42:48 *** System error while flushing: Database corrupted
    102017-10-29 23:42:48 Error: Error: A fatal internal error occurred, see debug.log for details
    112017-10-29 23:42:48 tor: Thread interrupt
    122017-10-29 23:42:48 torcontrol thread exit
    132017-10-29 23:42:48 addcon thread exit
    142017-10-29 23:42:48 opencon thread exit
    152017-10-29 23:42:48 net thread exit
    162017-10-29 23:42:48 msghand thread exit
    172017-10-29 23:42:48 scheduler thread interrupt
    182017-10-29 23:42:48 Shutdown: In progress...
    192017-10-29 23:42:50 UPNP_DeletePortMapping() returned: 0
    202017-10-29 23:42:50 upnp thread interrupt
    212017-10-29 23:42:52 Dumped mempool: 0.003313s to copy, 0.041511s to dump
    222017-10-29 23:42:52 Corruption: block checksum mismatch
    232017-10-29 23:42:52 *** System error while flushing: Database corrupted
    242017-10-29 23:42:52 Error: Error: A fatal internal error occurred, see debug.log for details
    252017-10-29 23:42:52 Corruption: block checksum mismatch
    262017-10-29 23:42:52 *** System error while flushing: Database corrupted
    272017-10-29 23:42:52 Error: Error: A fatal internal error occurred, see debug.log for details
    282017-10-29 23:42:53 Shutdown: done
    

26. 
    dooglus commented at 4:33 am on October 30, 2017: contributor

    Thanks for the report @sangaman. It’s reassuring to hear of others experiencing the same issue.

    Was the system which crashed running version 0.15 for the whole sync?

    I run a pruned 0.15 node and have never had it crash. I don’t leave it running though, I run it every few days until it has synced and then shut it down again.

27. 
    sangaman commented at 4:39 am on October 30, 2017: contributor

    The system that crashed has been running 0.15 the entire time. The other system completed the initial sync on 0.14, has been running 0.15 for the past few weeks ~24/7, and has never crashed. Both systems were created from the same image and I followed the same steps for setting them up, so the only real difference there is 0.14 vs 0.15.
28. 
    dooglus commented at 4:42 am on October 30, 2017: contributor

    Does your Pi ever suspend itself?

    Do you see a ‘gap’ in the debug.log timestamps shortly before the crash? See for example in this comment where there’s a 4 hour gap in the timestamps between the 5th and 6th lines of the debug.log I posted.

29. 
    sangaman commented at 4:49 am on October 30, 2017: contributor

    Hm I don’t believe it suspends itself, it has been syncing continuously pretty much since I set it up. I scrolled through the past ~48 hours of debug.log and didn’t see any gaps in the timestamps. My other pi is also running continuously as far as I can tell.
30. 
    sangaman commented at 9:34 pm on October 30, 2017: contributor

    Just an update - I have copied over the .bitcoin folder from my working raspi to the one that crashed yesterday (to avoid having to start the initial sync all over again). Everything running fine for the time being, but I’ll update this thread if I encounter the issue again. I also have preserved the old .bitcoin folder in the broken state in case it may be useful for debugging the issue.
31. 
    TheBlueMatt commented at 11:09 pm on October 30, 2017: member

    @dooglus your memtest86 issues scare me. Have you tried to run linpack recently? I’d be curious to see how your CPU does under such conditions and see if it generates any errors. @sangaman Does your RPi have a heatsink? Does it overheat during sync?
32. 
    dooglus commented at 5:07 am on October 31, 2017: contributor

    @TheBlueMatt I’ve never even heard of linpack.

    You’re talking about this?

    The LINPACK Benchmarks are a measure of a system’s floating point computing power. Introduced by Jack Dongarra, they measure how fast a computer solves a dense n by n system of linear equations Ax = b, which is a common task in engineering.

33. 
    TheBlueMatt commented at 12:05 pm on October 31, 2017: member

    Yes, linpack is one of the more aggressive benchmarking tools. I’m curious to see if your CPU instantly overheats if you run it.

    On October 31, 2017 1:07:30 AM EDT, Chris Moore notifications@github.com wrote:

    @TheBlueMatt I’ve never even heard of linpack.

    You’re talking about this?

    The LINPACK Benchmarks are a measure of a system’s floating point computing power. Introduced by Jack Dongarra, they measure how fast a computer solves a dense n by n system of linear equations Ax = b, which is a common task in engineering.

    – You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/11355#issuecomment-340661440

34. 
    sangaman commented at 1:42 pm on October 31, 2017: contributor

    @TheBlueMatt I do have heatsinks on them, although I didn’t ever check the temperatures on them during initial sync. They aren’t overclocked or modded in any way.
35. 
    dooglus commented at 8:01 pm on November 13, 2017: contributor

    Here’s a possibly interesting data point:

    My Bitcoin Cash node corrupted its chainstate yesterday.

    As I understand it, Bitcoin Cash is based on a pre 0.15 release of Bitcoin Core (they call it 0.14.6).

    I built it from commit a46e63c2bb3d189beb846a8d34a0d33fd563fa7b.

    This leads me to believe that the corruption I keep seeing in Core’s chainstate db isn’t new in 0.15 as I previously suspected.

36. 
    kollokollo commented at 7:11 am on November 15, 2017: none

    I had a checksum corruption error these days. The difference to this case was, that it happened after the syncing process was finished. See: #11627 The experts opinion was that it must have been a disk falior. That might be, but I have no other evidence that the disk used is wearing out. I do observe bugs in bitcoin core after having imported a private key from another wallet and after changing the wallet.dat file (causing a rescan). The data corruption problem begun after several rescans and ther is a chance that there is really a bug in bitcoin core which leads to data corruption. But so far no real proof.
37. 
    TheBlueMatt commented at 10:31 pm on November 15, 2017: member

    @kollokollo I dont think the “expert opinion” is that it is disk failure, much, much more likely CPU is corrupting data in-flight (due to thermal issues/overclock/other instability) or the memory is corrupting data (also likely for similar reasons).
38. 
    kollokollo commented at 7:42 am on November 16, 2017: none

    I am suprized to hear that a normal Laptop would miscalculate “on the fly”. But I am not the expert here. I just want to help debugging the application.
39. laanwj added the label Data corruption on Nov 16, 2017
40. 
    laanwj commented at 12:49 pm on November 16, 2017: member

    I am suprized to hear that a normal Laptop would miscalculate “on the fly”

    Especially those! Laptops are much more likely to have bad thermal management than desktops or servers (less room than desktops, not designed for heavy loads like servers, generally no ECC on memory). Also the fans tend to get cluttered with dust much quicker.

41. 
    laanwj commented at 12:52 pm on November 16, 2017: member

    BTW if anyone suffering leveldb corruption issues wants to help me inventarize the kind of corruptions that happen, please try #11674 to improve leveldb’s error logging and send me the specific file that it labels as corrupted.
42. 
    ryanofsky commented at 1:10 pm on November 16, 2017: member

    I am suprized to hear that a normal Laptop would miscalculate “on the fly”

    Especially those! Laptops are much more likely to have bad thermal management

    Maybe I am thinking about this wrong, but the thing I don’t understand is if the problem is cpu/memory hardware failures, how come we seem to be seeing more leveldb checksum errors than bitcoin validation errors (mismatched hashes, bad signatures)? Presumably leveldb checksums aren’t particularly expensive to compute and the hardware is spending much more time checking bitcoin signatures and hashes and computing leveldb checksums, so shouldn’t we expect more bitcoin errors and fewer leveldb errors?

    Is the answer that bitcoin signatures and hashes tend to be checked once and trusted forever, while leveldb checksums are read and checked more continuously? I would think overheating laptops and raspberry pis should never be able to get through an initial sync if they can’t even compute leveldb checksums reliably.

43. 
    laanwj commented at 4:01 pm on November 16, 2017: member

    Maybe I am thinking about this wrong, but the thing I don’t understand is if the problem is cpu/memory hardware failures, how come we seem to be seeing more leveldb checksum errors than bitcoin validation errors (mismatched hashes, bad signatures)?

    We do see validation errors and nodes getting stuck after failing to verify a valid block once in a while - but they don’t terminate the process with an immediate error, so might be less likely to get reported!

    Is the answer that bitcoin signatures and hashes tend to be checked once and trusted forever, while leveldb checksums are read and checked more continuously

    That’s a good point too. The database is continuously being read and written, so the chance of an error in the i/o there might be higher (I have certainly seen cases of corruption not caused by CPU overheating, but by faulty USB sticks and SD cards).

    But all in all we just don’t know for sure. The kind of glitches (e.g. bit toggling, zeroed blocks, duplicated sequences, file truncation) might reveal more about the underlying cause. Which is why I’m trying to collect corrupted leveldb files.

44. 
    dooglus commented at 6:21 pm on November 16, 2017: contributor

    I’m running your patch and will attach corrupted db files when I next get a corruption.

    I have a bunch of old corrupt chainstate/ directories here. Is there a safe way of finding out which file in each is corrupted without corrupting my block index? I don’t want to have to rescan everything again.

45. 
    laanwj commented at 8:26 pm on November 16, 2017: member

    I have a bunch of old corrupt chainstate/ directories here. Is there a safe way of finding out which file in each is corrupted without corrupting my block index? I don’t want to have to rescan everything again.

    Not easily. The only way I see to do this safely would be to use https://github.com/laanwj/blockdb-troubleshoot, which I’ve updated with the patch

    0git clone https://github.com/laanwj/blockdb-troubleshoot.git
    1cd blockdb-troubleshoot
    2./compile_leveldb.sh
    3...
    4./simplecheck.py /path/to/corrupted/datadir/
    5...
    6leveldb.LevelDBError: Corruption: block checksum mismatch: /.../blocks/index/000042.ldb
    

    This will do a linear scan as well as some probing over the databases (chainstate and block index), and probably trigger it, though I’ve seen issues that are only triggered by specific random access to records!

46. 
    dooglus commented at 2:39 am on November 17, 2017: contributor

    @laanwj Thanks, but that didn’t show any errors. I commented out the check of /blocks/index, and switched in each of the old corrupt chainstate databases in turn:

    $ ./simplecheck.py ~/.bitcoin # .bad
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 51256562 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad2
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 51932124 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad3
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 52049450 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad4
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 52591784 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad5
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 52497622 records checked
     * probe 10000 times (trigger bloom filters)
    $ 

47. 
    laanwj commented at 12:21 pm on November 17, 2017: member

    I pushed an update to make it more sensitive (need to re-invoke ./compile_leveldb.sh), apparently python-leveldb was ignoring errors during iteration.
48. 
    dooglus commented at 4:42 pm on November 17, 2017: contributor

    The update doesn’t appear to have changed anything:

    $ ./simplecheck.py ~/.bitcoin # .bad
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 51256562 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad2
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 51932124 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad3
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 52049450 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad4
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 52591784 records checked
     * probe 10000 times (trigger bloom filters)
    $ ./simplecheck.py ~/.bitcoin # .bad5
    * Checking chain state
     * opening /home/chris/.bitcoin/chainstate
     * 52497622 records checked
     * probe 10000 times (trigger bloom filters)
    $ 
    

    Here’s how things usually go down when my chainstate db gets corrupted:

    • the client crashes; I see in the log that it crashed at block N
    • I restart the client. It starts “processing blocks on disk” from block N-200 or so; it seems like it is re-importing all the blocks since the last time the chainstate db was flushed to disk or something
    • if I wait long enough, it gets back to block N and crashes again
    • if I stop the client before it gets back to block N, it exits successfully, and when I restart the client it continues processing from wherever it was when I stopped it; but eventually it gets back to block N and crashes.
    • sometimes it gets back to block N and is fine

    Could it be that my saved corrupted chainstate dbs don’t exhibit their checksum errors until they get synced to their respective ‘block N’s?

49. 
    laanwj commented at 12:54 pm on November 18, 2017: member

    Hmm in that case it’s really hard to trigger. Maybe like it is corrupted so that only a specific query, triggered by an input part of block N, will trigger the error, but that iteration jumps over the corrupted record. An external tool is not going to help then. If you really want to go down that rabbit hole you could log the last operation or last N leveldb operations in case of a crash.
50. 
    dooglus commented at 9:10 am on February 16, 2018: contributor

    It’s been a few months since I last saw this crash, but I just had it again. This time it happened on commit f0295becbf3ef1fb78095306408789253fe0c114 which is a few commits after v0.16.0rc1.

    2018-02-16 07:26:36 UpdateTip: new best=00000000000000000018c333f109f9a0197817cb656bc9e27cd204d4dfb4fd5c height=509422 version=0x20000000 log2_work=88.136891 tx=299747962 date='2018-02-16 07:26:19' progress=1.000000 cache=48.3MiB(363411txo)
    2018-02-16 07:31:41 connect() to [2607:fa18:3a01::50]:8333 failed: Network is unreachable (101)
    2018-02-16 07:35:51 connect() to [2001:0:4137:9e76:4a7:10c:b9b5:34c5]:8333 failed: Network is unreachable (101)
    2018-02-16 07:37:47 connect() to [2601:644:101:314b:2ccd:c6db:5fc7:4c4e]:8333 failed: Network is unreachable (101)
    2018-02-16 07:38:52 UpdateTip: new best=0000000000000000005ee50bf03d1cb2e1a61e1ffd4100a44161ff4738721cea height=509423 version=0x20000000 log2_work=88.136943 tx=299749241 date='2018-02-16 07:38:24' progress=1.000000 cache=48.9MiB(367635txo)
    2018-02-16 07:45:35 connect() to 77.20.12.245:8333 failed after select(): Connection refused (111)
    2018-02-16 07:53:20 connect() to 104.223.98.17:8333 failed after select(): Connection refused (111)
    2018-02-16 07:55:50 connect() to 185.108.128.7:8333 failed after select(): Connection refused (111)
    2018-02-16 07:57:13 connect() to [2001:0:5ef5:79fd:3c38:bad8:a2dc:6e65]:8333 failed: Network is unreachable (101)
    2018-02-16 07:58:22 connect() to 85.187.245.15:8333 failed after select(): Connection refused (111)
    2018-02-16 08:04:49 connect() to 95.88.10.246:8333 failed after select(): Connection refused (111)
    2018-02-16 08:06:07 connect() to 73.156.44.208:8333 failed after select(): No route to host (113)
    2018-02-16 08:18:33 Potential stale tip detected, will try using extra outbound peer (last tip update: 2381 seconds ago)
    2018-02-16 08:18:34 connect() to [2001:0:9d38:6abd:2427:1651:4d79:9ece]:8333 failed: Network is unreachable (101)
    2018-02-16 08:18:40 New outbound peer connected: version: 70015, blocks=509423, peer=21
    2018-02-16 08:19:19 connect() to [2001:0:5ef5:79fb:34e1:638:bf04:6a9e]:8333 failed: Network is unreachable (101)
    2018-02-16 08:20:19 UpdateTip: new best=00000000000000000022d76fe26eac6ec29c7073f03a69235b7e3f11c9558560 height=509424 version=0x20000000 log2_work=88.136995 tx=299751509 date='2018-02-16 08:20:09' progress=1.000000 cache=50.9MiB(384222txo)
    2018-02-16 08:32:00 connect() to 185.213.152.164:8333 failed after select(): Connection refused (111)
    2018-02-16 08:32:22 connect() to 82.78.215.208:8333 failed after select(): Connection refused (111)
    2018-02-16 08:33:34 connect() to [2a02:1205:505e:4850:6c64:6198:993d:9a78]:8333 failed: Network is unreachable (101)
    2018-02-16 08:35:21 LevelDB read failure: Corruption: block checksum mismatch
    2018-02-16 08:35:21 Corruption: block checksum mismatch
    2018-02-16 08:36:45 connect() to [2001:0:4137:9e76:248b:55b:e0ca:efa]:8333 failed: Network is unreachable (101)
    2018-02-16 08:38:13 connect() to 178.251.111.106:8333 failed after select(): Connection refused (111)
    2018-02-16 08:38:31 connect() to [2001:0:9d38:90d7:c38:fbff:aab4:fbbf]:8333 failed: Network is unreachable (101)
    2018-02-16 08:55:15 socket sending timeout: 1201s
    2018-02-16 08:55:18 socket sending timeout: 1201s
    2018-02-16 08:55:18 socket sending timeout: 1201s
    2018-02-16 08:55:18 socket sending timeout: 1201s
    2018-02-16 08:55:19 socket sending timeout: 1201s
    2018-02-16 08:55:20 socket sending timeout: 1201s
    2018-02-16 08:55:21 socket sending timeout: 1201s
    2018-02-16 08:55:21 socket sending timeout: 1201s
    
    (gdb) thread apply all where
    
    Thread 21 (Thread 0x7f73d3deb700 (LWP 15251)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447cfe6ad in poll () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f74456359f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#2](/bitcoin-bitcoin/2/)  0x00007f7445635b0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#3](/bitcoin-bitcoin/3/)  0x00007f744a2c604f in QEventDispatcherGlib::processEvents (this=0x7f73c80008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
    [#4](/bitcoin-bitcoin/4/)  0x00007f744a26f9ca in QEventLoop::exec (this=this@entry=0x7f73d3deacc0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
    [#5](/bitcoin-bitcoin/5/)  0x00007f744a09d0f3 in QThread::exec (this=<optimized out>) at thread/qthread.cpp:507
    [#6](/bitcoin-bitcoin/6/)  0x00007f744a0a1da8 in QThreadPrivate::start (arg=0x56427d04cc80) at thread/qthread_unix.cpp:368
    [#7](/bitcoin-bitcoin/7/)  0x00007f744975f494 in start_thread (arg=0x7f73d3deb700) at pthread_create.c:333
    [#8](/bitcoin-bitcoin/8/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 20 (Thread 0x7f73f0c42700 (LWP 15248)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447cfe6ad in poll () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f74456359f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#2](/bitcoin-bitcoin/2/)  0x00007f7445635b0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#3](/bitcoin-bitcoin/3/)  0x00007f744a2c604f in QEventDispatcherGlib::processEvents (this=0x7f73cc0008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
    [#4](/bitcoin-bitcoin/4/)  0x00007f744a26f9ca in QEventLoop::exec (this=this@entry=0x7f73f0c41cc0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
    [#5](/bitcoin-bitcoin/5/)  0x00007f744a09d0f3 in QThread::exec (this=<optimized out>) at thread/qthread.cpp:507
    [#6](/bitcoin-bitcoin/6/)  0x00007f744a0a1da8 in QThreadPrivate::start (arg=0x56427c44bdc0) at thread/qthread_unix.cpp:368
    [#7](/bitcoin-bitcoin/7/)  0x00007f744975f494 in start_thread (arg=0x7f73f0c42700) at pthread_create.c:333
    [#8](/bitcoin-bitcoin/8/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 19 (Thread 0x7f73f1493700 (LWP 15247)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
    [#1](/bitcoin-bitcoin/1/)  0x00007f744a0a2c6b in QWaitConditionPrivate::wait (time=18446744073709551615, this=0x7f73e3a09910) at thread/qwaitcondition_unix.cpp:143
    [#2](/bitcoin-bitcoin/2/)  QWaitCondition::wait (this=this@entry=0x7f73e3a09a98, mutex=mutex@entry=0x7f73e3a09a90, time=time@entry=18446744073709551615)
        at thread/qwaitcondition_unix.cpp:215
    [#3](/bitcoin-bitcoin/3/)  0x00007f744a09bb33 in QSemaphore::acquire (this=this@entry=0x7f73f148a3b0, n=n@entry=1) at thread/qsemaphore.cpp:143
    [#4](/bitcoin-bitcoin/4/)  0x00007f744a27c8cd in QMetaMethod::invoke (this=this@entry=0x7f73f148a618, object=object@entry=0x56427c348250, connectionType=Qt::BlockingQueuedConnection, 
        connectionType@entry=4048070304, returnValue=..., val0=..., val1=..., val2=..., val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...)
        at kernel/qmetaobject.cpp:2279
    [#5](/bitcoin-bitcoin/5/)  0x00007f744a28258a in QMetaObject::invokeMethod (obj=0x56427c348250, member=0x56427b1eee20 "message", type=4048070304, ret=..., val0=..., val1=..., val2=..., 
        val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...) at kernel/qmetaobject.cpp:1489
    [#6](/bitcoin-bitcoin/6/)  0x000056427a8ac33c in QMetaObject::invokeMethod (member=0x80 <error: Cannot access memory at address 0x80>, type=Qt::BlockingQueuedConnection, val0=..., 
        obj=<optimized out>, val1=..., val2=..., val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...)
        at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs.h:460
    [#7](/bitcoin-bitcoin/7/)  ThreadSafeMessageBox (gui=0x56427c348250, message="Error reading from database, shutting down.", caption=..., style=<optimized out>) at qt/bitcoingui.cpp:1168
    [#8](/bitcoin-bitcoin/8/)  0x000056427a8b07e7 in boost::_bi::list4<boost::_bi::value<BitcoinGUI*>, boost::arg<1>, boost::arg<2>, boost::arg<3> >::operator()<bool, bool (*)(BitcoinGUI*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::_bi::rrlist3<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int> > (f=<error reading variable>, this=<optimized out>, a=...)
        at /usr/include/boost/bind/bind.hpp:453
    [#9](/bitcoin-bitcoin/9/)  boost::_bi::bind_t<bool, bool (*)(BitcoinGUI*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::_bi::list4<boost::_bi::value<BitcoinGUI*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >::operator()<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&&) (a1=<error reading variable: Cannot access memory at address 0x88>, 
        a2=<error reading variable: Cannot access memory at address 0x9>, this=<optimized out>, a3=<optimized out>) at /usr/include/boost/bind/bind.hpp:1330
    [#10](/bitcoin-bitcoin/10/) boost::detail::function::function_obj_invoker3<boost::_bi::bind_t<bool, bool (*)(BitcoinGUI*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::_bi::list4<boost::_bi::value<BitcoinGUI*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>::invoke (function_obj_ptr=..., 
        a0=<error reading variable: Cannot access memory at address 0x88>, a1=<error reading variable: Cannot access memory at address 0x9>, a2=1232490847)
        at /usr/include/boost/function/function_template.hpp:138
    [#11](/bitcoin-bitcoin/11/) 0x000056427aa1388e in boost::function3<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>::operator() (a0=<error reading variable: Cannot access memory at address 0x88>, 
        a1=<error reading variable: Cannot access memory at address 0x9>, a2=1232490847, this=<optimized out>) at /usr/include/boost/function/function_template.hpp:770
    [#12](/bitcoin-bitcoin/12/) boost::signals2::detail::call_with_tuple_args<bool>::m_invoke<boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, 0u, 1u, 2u, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&>(boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>&, boost::signals2::detail::unsigned_meta_array<0u, 1u, 2u>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&> const&, boost::disable_if<boost::is_void<boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>::result_type>, void>::type*) const (this=<optimized out>, func=..., args=...)
        at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:98
    [#13](/bitcoin-bitcoin/13/) boost::signals2::detail::call_with_tuple_args<bool>::operator()<boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&, 3ul>(boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int&> const&, mpl_::size_t<3ul>) const (this=<optimized out>, func=..., args=...)
        at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
    [#14](/bitcoin-bitcoin/14/) boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > const&) const (
        this=<optimized out>, connectionBody=...) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
    [#15](/bitcoin-bitcoin/15/) boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >::dereference() const (this=0x7f73f148ab30) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:110
    [#16](/bitcoin-bitcoin/16/) 0x000056427aa12d08 in boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > const&) (f=...) at /usr/include/boost/iterator/iterator_facade.hpp:549
    [#17](/bitcoin-bitcoin/17/) boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >, bool, boost::iterators::single_pass_traversal_tag, bool&, long, false, false>::operator*() const (
        this=<optimized out>) at /usr/include/boost/iterator/iterator_facade.hpp:655
    [#18](/bitcoin-bitcoin/18/) boost::signals2::last_value<bool>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >) const (this=<optimized out>, first=..., last=...) at /usr/include/boost/signals2/last_value.hpp:48
    [#19](/bitcoin-bitcoin/19/) 0x000056427aa129a0 in boost::signals2::detail::combiner_invoker<bool>::operator()<boost::signals2::last_value<bool>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >(boost::signals2::last_value<bool>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)> >, boost::signals2::mutex> >) const (combiner=..., this=<optimized out>, first=..., last=...) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:53
    [#20](/bitcoin-bitcoin/20/) boost::signals2::detail::signal_impl<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::signals2::last_value<bool>, int, std::less<int>, boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::function<bool (boost::signals2::connection const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::signals2::mutex>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) (
        this=0x56427c062c50, args=<optimized out>, args=<optimized out>, args=<optimized out>) at /usr/include/boost/signals2/detail/signal_template.hpp:242
    [#21](/bitcoin-bitcoin/21/) 0x000056427aa121ad in boost::signals2::signal<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int), boost::signals2::last_value<bool>, int, std::less<int>, boost::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::function<bool (boost::signals2::connection const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)>, boost::signals2::mutex>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) (args=..., args=..., this=<optimized out>, args=...) at /usr/include/boost/signals2/detail/signal_template.hpp:723
    [#22](/bitcoin-bitcoin/22/) CCoinsViewErrorCatcher::GetCoin (this=<optimized out>, outpoint=..., coin=...) at init.cpp:143
    [#23](/bitcoin-bitcoin/23/) 0x000056427ad14a92 in CCoinsViewCache::FetchCoin (this=0x7f73fd0cfaf0, outpoint=...) at coins.cpp:45
    [#24](/bitcoin-bitcoin/24/) 0x000056427ad14b79 in CCoinsViewCache::GetCoin (this=0x7f73e3a0993c, outpoint=..., coin=...) at coins.cpp:58
    [#25](/bitcoin-bitcoin/25/) 0x000056427ab681b4 in CCoinsViewMemPool::GetCoin (this=0x7f73f148b3d0, outpoint=..., coin=...) at txmempool.cpp:904
    [#26](/bitcoin-bitcoin/26/) 0x000056427ad14a92 in CCoinsViewCache::FetchCoin (this=0x7f73f148c060, outpoint=...) at coins.cpp:45
    [#27](/bitcoin-bitcoin/27/) 0x000056427ad15133 in CCoinsViewCache::HaveCoin (this=0x7f73e3a0993c, outpoint=...) at coins.cpp:126
    [#28](/bitcoin-bitcoin/28/) 0x000056427abaf7c6 in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=..., pfMissingInputs=<optimized out>, nAcceptTime=<optimized out>, 
        plTxnReplaced=0x0, bypass_limits=95, nAbsurdFee=@0x56427b5c22e1: 257, coins_to_uncache=...) at validation.cpp:643
    [#29](/bitcoin-bitcoin/29/) 0x000056427ab7dd07 in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., 
        tx=<error reading variable: Cannot access memory at address 0x1be08247c8b50>, pfMissingInputs=0x7f73e3a09900, nAcceptTime=0, plTxnReplaced=0x7f73f148f208, 
        bypass_limits=<optimized out>, nAbsurdFee=0) at validation.cpp:985
    [#30](/bitcoin-bitcoin/30/) 0x000056427ab7d5eb in AcceptToMemoryPool (pool=..., state=..., tx=<error reading variable: Cannot access memory at address 0x1be08247c8b50>, 
        pfMissingInputs=0x7f73e3a09900, plTxnReplaced=0x7f73f148f208, bypass_limits=false, nAbsurdFee=<optimized out>) at validation.cpp:1001
    [#31](/bitcoin-bitcoin/31/) 0x000056427aa6cbc4 in ProcessMessage (pfrom=<optimized out>, strCommand="tx", vRecv=..., nTimeReceived=<optimized out>, chainparams=..., 
        connman=<optimized out>, interruptMsgProc=...) at net_processing.cpp:2141
    [#32](/bitcoin-bitcoin/32/) 0x000056427aa64044 in PeerLogicValidation::ProcessMessages (this=<optimized out>, pfrom=0x7f73dc1a3680, interruptMsgProc=...) at net_processing.cpp:2966
    [#33](/bitcoin-bitcoin/33/) 0x000056427aa8a537 in non-virtual thunk to PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) () at net_processing.cpp:2891
    [#34](/bitcoin-bitcoin/34/) 0x000056427aa4031e in CConnman::ThreadMessageHandler (this=<optimized out>) at net.cpp:2018
    [#35](/bitcoin-bitcoin/35/) 0x000056427aa59c6a in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=<optimized out>, __t=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
    [#36](/bitcoin-bitcoin/36/) std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
    [#37](/bitcoin-bitcoin/37/) std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
    [#38](/bitcoin-bitcoin/38/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=<optimized out>, 
        __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
    [#39](/bitcoin-bitcoin/39/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
    [#40](/bitcoin-bitcoin/40/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
    [#41](/bitcoin-bitcoin/41/) 0x000056427aa0bced in std::function<void ()>::operator()() const (this=0x7f73f1492d00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#42](/bitcoin-bitcoin/42/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:317
    [#43](/bitcoin-bitcoin/43/) 0x000056427aa59e2d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#44](/bitcoin-bitcoin/44/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#45](/bitcoin-bitcoin/45/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#46](/bitcoin-bitcoin/46/) 0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#47](/bitcoin-bitcoin/47/) 0x00007f744975f494 in start_thread (arg=0x7f73f1493700) at pthread_create.c:333
    [#48](/bitcoin-bitcoin/48/) 0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 18 (Thread 0x7f73f1ce4700 (LWP 15246)):
    [#0](/bitcoin-bitcoin/0/)  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
    [#1](/bitcoin-bitcoin/1/)  0x00007f7449761c06 in __GI___pthread_mutex_lock (mutex=0x56427b5c22b8 <cs_main>) at ../nptl/pthread_mutex_lock.c:115
    [#2](/bitcoin-bitcoin/2/)  0x000056427aa5b935 in __gthread_mutex_lock (__mutex=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
    [#3](/bitcoin-bitcoin/3/)  __gthread_recursive_mutex_lock (__mutex=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:810
    [#4](/bitcoin-bitcoin/4/)  std::recursive_mutex::lock (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:105
    [#5](/bitcoin-bitcoin/5/)  AnnotatedMixin<std::recursive_mutex>::lock (this=<optimized out>) at ./sync.h:59
    [#6](/bitcoin-bitcoin/6/)  std::unique_lock<CCriticalSection>::lock (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:267
    [#7](/bitcoin-bitcoin/7/)  CCriticalBlock::Enter (nLine=573, this=<optimized out>, pszName=<optimized out>, pszFile=<optimized out>) at ./sync.h:128
    [#8](/bitcoin-bitcoin/8/)  CCriticalBlock::CCriticalBlock (nLine=573, fTry=false, this=<optimized out>, mutexIn=..., pszName=<optimized out>, pszFile=<optimized out>) at ./sync.h:149
    [#9](/bitcoin-bitcoin/9/)  PeerLogicValidation::InitializeNode (this=0x7f741c47edb0, pnode=0x7f73dc346340) at net_processing.cpp:573
    [#10](/bitcoin-bitcoin/10/) 0x000056427aa5cf57 in non-virtual thunk to PeerLogicValidation::InitializeNode(CNode*) () at net_processing.cpp:568
    [#11](/bitcoin-bitcoin/11/) 0x000056427aa3ce96 in CConnman::OpenNetworkConnection (this=0x7f741c571a60, addrConnect=..., fCountFailure=<optimized out>, grantOutbound=<optimized out>, 
        pszDest=0x0, fOneShot=<optimized out>, fFeeler=<optimized out>, manual_connection=92) at net.cpp:1988
    [#12](/bitcoin-bitcoin/12/) 0x000056427aa3e431 in CConnman::ThreadOpenConnections (this=<optimized out>, connect=...) at net.cpp:1871
    [#13](/bitcoin-bitcoin/13/) 0x000056427aa5a037 in std::__invoke_impl<void, void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (
        __f=<optimized out>, __t=<optimized out>, __args=<error reading variable: Cannot access memory at address 0x8>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
    [#14](/bitcoin-bitcoin/14/) 0x000056427aa59ebe in std::__invoke<void (CConnman::* const&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__fn= [@0x56427b5c22b8](/bitcoin-bitcoin/contributor/0x56427b5c22b8/): (void (CConnman::*)(CConnman * const, std::vector<std::__cxx11::basic_string<char>, std::allocator<std::__cxx11::basic_string<char> > >)) 0x100000002, this adjustment 4294982543, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
    [#15](/bitcoin-bitcoin/15/) std::_Mem_fn_base<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), true>::operator()<CConnman*&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&> (__args=..., __args=..., 
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
    [#16](/bitcoin-bitcoin/16/) std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x56427b5c22b8 <cs_main>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
    [#17](/bitcoin-bitcoin/17/) std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)>::operator()<, void>() (
        this=0x56427b5c22b8 <cs_main>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
    [#18](/bitcoin-bitcoin/18/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> (CConnman*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
    [#19](/bitcoin-bitcoin/19/) 0x000056427aa0bced in std::function<void ()>::operator()() const (this=0x7f73f1ce3d00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#20](/bitcoin-bitcoin/20/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:317
    [#21](/bitcoin-bitcoin/21/) 0x000056427aa59e2d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#22](/bitcoin-bitcoin/22/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#23](/bitcoin-bitcoin/23/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#24](/bitcoin-bitcoin/24/) 0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#25](/bitcoin-bitcoin/25/) 0x00007f744975f494 in start_thread (arg=0x7f73f1ce4700) at pthread_create.c:333
    [#26](/bitcoin-bitcoin/26/) 0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 17 (Thread 0x7f73f2535700 (LWP 15245)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
    [#1](/bitcoin-bitcoin/1/)  0x000056427ad44b7e in __gthread_cond_timedwait (__cond=<optimized out>, __mutex=<optimized out>, __abs_timeout=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:871
    [#2](/bitcoin-bitcoin/2/)  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=<optimized out>, __lock=..., __atime=...)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:166
    [#3](/bitcoin-bitcoin/3/)  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=<optimized out>, __lock=..., __atime=...)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:106
    [#4](/bitcoin-bitcoin/4/)  std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=<optimized out>, 
        __lock=..., __atime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:129
    [#5](/bitcoin-bitcoin/5/)  std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, CThreadInterrupt::sleep_for(std::chrono::duration<long, std::ratio<1l, 1000l> >)::$_0) (this=<optimized out>, __lock=..., __rtime=..., __p=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/condition_variable:145
    [#6](/bitcoin-bitcoin/6/)  CThreadInterrupt::sleep_for (this=<optimized out>, rel_time=...) at threadinterrupt.cpp:30
    [#7](/bitcoin-bitcoin/7/)  0x000056427ad44c0a in CThreadInterrupt::sleep_for (this=0x7f741c5c1ecc, rel_time=...) at threadinterrupt.cpp:35
    [#8](/bitcoin-bitcoin/8/)  0x000056427aa4006c in CConnman::ThreadOpenAddedConnections (this=<optimized out>) at net.cpp:1950
    [#9](/bitcoin-bitcoin/9/)  0x000056427aa59c6a in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=<optimized out>, __t=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
    [#10](/bitcoin-bitcoin/10/) std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
    [#11](/bitcoin-bitcoin/11/) std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
    [#12](/bitcoin-bitcoin/12/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=<optimized out>, 
        __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
    [#13](/bitcoin-bitcoin/13/) std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
    [#14](/bitcoin-bitcoin/14/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
    [#15](/bitcoin-bitcoin/15/) 0x000056427aa0bced in std::function<void ()>::operator()() const (this=0x7f73f2534d00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#16](/bitcoin-bitcoin/16/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:317
    [#17](/bitcoin-bitcoin/17/) 0x000056427aa59e2d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#18](/bitcoin-bitcoin/18/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#19](/bitcoin-bitcoin/19/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#20](/bitcoin-bitcoin/20/) 0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#21](/bitcoin-bitcoin/21/) 0x00007f744975f494 in start_thread (arg=0x7f73f2535700) at pthread_create.c:333
    [#22](/bitcoin-bitcoin/22/) 0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 16 (Thread 0x7f73f467e700 (LWP 15243)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447d003f3 in select () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x000056427aa36611 in CConnman::ThreadSocketHandler (this=<optimized out>) at net.cpp:1279
    [#2](/bitcoin-bitcoin/2/)  0x000056427aa59c6a in std::__invoke_impl<void, void (CConnman::* const&)(), CConnman*&> (__f=<optimized out>, __t=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
    [#3](/bitcoin-bitcoin/3/)  std::__invoke<void (CConnman::* const&)(), CConnman*&> (__fn=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
    [#4](/bitcoin-bitcoin/4/)  std::_Mem_fn_base<void (CConnman::*)(), true>::operator()<CConnman*&> (this=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
    [#5](/bitcoin-bitcoin/5/)  std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=<optimized out>, 
        __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
    [#6](/bitcoin-bitcoin/6/)  std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)>::operator()<, void>() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
    [#7](/bitcoin-bitcoin/7/)  std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CConnman::*)()> (CConnman*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
    [#8](/bitcoin-bitcoin/8/)  0x000056427aa0bced in std::function<void ()>::operator()() const (this=0x7f73f467dd00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#9](/bitcoin-bitcoin/9/)  TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:317
    [#10](/bitcoin-bitcoin/10/) 0x000056427aa59e2d in std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#11](/bitcoin-bitcoin/11/) std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#12](/bitcoin-bitcoin/12/) std::thread::_State_impl<std::_Bind_simple<void (*(char const*, std::function<void ()>))(char const*, std::function<void ()>)> >::_M_run() (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#13](/bitcoin-bitcoin/13/) 0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#14](/bitcoin-bitcoin/14/) 0x00007f744975f494 in start_thread (arg=0x7f73f467e700) at pthread_create.c:333
    [#15](/bitcoin-bitcoin/15/) 0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 15 (Thread 0x7f73f35d7700 (LWP 15239)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447d080f3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f744887fd98 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
    [#2](/bitcoin-bitcoin/2/)  0x00007f744886a07a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
    [#3](/bitcoin-bitcoin/3/)  0x000056427ab44bd0 in TorControlThread () at torcontrol.cpp:740
    [#4](/bitcoin-bitcoin/4/)  0x000056427ab45a7b in TraceThread<void (*)()> (name=<optimized out>, func=0x56427ab44b20 <TorControlThread()>) at ./util.h:317
    [#5](/bitcoin-bitcoin/5/)  0x000056427ab4d0d5 in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<void (*)()> >::operator()<void (*)(char const*, void (*)()), boost::_bi::list0> (this=<optimized out>, f=<optimized out>, a=...) at /usr/include/boost/bind/bind.hpp:319
    [#6](/bitcoin-bitcoin/6/)  boost::_bi::bind_t<void, void (*)(char const*, void (*)()), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<void (*)()> > >::operator() (
        this=<optimized out>) at /usr/include/boost/bind/bind.hpp:1294
    [#7](/bitcoin-bitcoin/7/)  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, void (*)()), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<void (*)()> > > >::run (this=0xfffffffffffffffc) at /usr/include/boost/thread/detail/thread.hpp:116
    [#8](/bitcoin-bitcoin/8/)  0x00007f744b303116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
    [#9](/bitcoin-bitcoin/9/)  0x00007f744975f494 in start_thread (arg=0x7f73f35d7700) at pthread_create.c:333
    [#10](/bitcoin-bitcoin/10/) 0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 14 (Thread 0x7f740a169700 (LWP 15223)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
    [#1](/bitcoin-bitcoin/1/)  0x000056427ad9f98b in leveldb::(anonymous namespace)::PosixEnv::BGThread (this=<optimized out>) at leveldb/util/env_posix.cc:641
    [#2](/bitcoin-bitcoin/2/)  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=<optimized out>) at leveldb/util/env_posix.cc:556
    [#3](/bitcoin-bitcoin/3/)  0x00007f744975f494 in start_thread (arg=0x7f740a169700) at pthread_create.c:333
    [#4](/bitcoin-bitcoin/4/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 13 (Thread 0x7f7419ebb700 (LWP 15221)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
    [#1](/bitcoin-bitcoin/1/)  0x00007f744858d50c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#2](/bitcoin-bitcoin/2/)  0x000056427ac2449a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:128
    [#3](/bitcoin-bitcoin/3/)  0x000056427ac1e28a in HTTPWorkQueueRun (queue=0x7f741c047a3c) at httpserver.cpp:362
    [#4](/bitcoin-bitcoin/4/)  0x000056427ac257b8 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#7](/bitcoin-bitcoin/7/)  0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#8](/bitcoin-bitcoin/8/)  0x00007f744975f494 in start_thread (arg=0x7f7419ebb700) at pthread_create.c:333
    [#9](/bitcoin-bitcoin/9/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 12 (Thread 0x7f741a70c700 (LWP 15220)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
    [#1](/bitcoin-bitcoin/1/)  0x00007f744858d50c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#2](/bitcoin-bitcoin/2/)  0x000056427ac2449a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:128
    [#3](/bitcoin-bitcoin/3/)  0x000056427ac1e28a in HTTPWorkQueueRun (queue=0x7f741c047a3c) at httpserver.cpp:362
    [#4](/bitcoin-bitcoin/4/)  0x000056427ac257b8 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#7](/bitcoin-bitcoin/7/)  0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#8](/bitcoin-bitcoin/8/)  0x00007f744975f494 in start_thread (arg=0x7f741a70c700) at pthread_create.c:333
    [#9](/bitcoin-bitcoin/9/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 11 (Thread 0x7f741af5d700 (LWP 15219)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
    [#1](/bitcoin-bitcoin/1/)  0x00007f744858d50c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#2](/bitcoin-bitcoin/2/)  0x000056427ac2449a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:128
    [#3](/bitcoin-bitcoin/3/)  0x000056427ac1e28a in HTTPWorkQueueRun (queue=0x7f741c047a3c) at httpserver.cpp:362
    [#4](/bitcoin-bitcoin/4/)  0x000056427ac257b8 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#7](/bitcoin-bitcoin/7/)  0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#8](/bitcoin-bitcoin/8/)  0x00007f744975f494 in start_thread (arg=0x7f741af5d700) at pthread_create.c:333
    [#9](/bitcoin-bitcoin/9/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 10 (Thread 0x7f741b7ae700 (LWP 15218)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
    [#1](/bitcoin-bitcoin/1/)  0x00007f744858d50c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#2](/bitcoin-bitcoin/2/)  0x000056427ac2449a in WorkQueue<HTTPClosure>::Run (this=<optimized out>) at httpserver.cpp:128
    [#3](/bitcoin-bitcoin/3/)  0x000056427ac1e28a in HTTPWorkQueueRun (queue=0x7f741c047a3c) at httpserver.cpp:362
    [#4](/bitcoin-bitcoin/4/)  0x000056427ac257b8 in std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#5](/bitcoin-bitcoin/5/)  std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#6](/bitcoin-bitcoin/6/)  std::thread::_State_impl<std::_Bind_simple<void (*(WorkQueue<HTTPClosure>*))(WorkQueue<HTTPClosure>*)> >::_M_run() (this=0xfffffffffffffe00)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#7](/bitcoin-bitcoin/7/)  0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#8](/bitcoin-bitcoin/8/)  0x00007f744975f494 in start_thread (arg=0x7f741b7ae700) at pthread_create.c:333
    [#9](/bitcoin-bitcoin/9/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 9 (Thread 0x7f741bfff700 (LWP 15217)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447d080f3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f744887fd98 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
    [#2](/bitcoin-bitcoin/2/)  0x00007f744886a07a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
    [#3](/bitcoin-bitcoin/3/)  0x000056427ac1d7ff in ThreadHTTP (base=0x7f741c047190, http=<optimized out>) at httpserver.cpp:314
    [#4](/bitcoin-bitcoin/4/)  0x000056427ac251cd in std::__invoke_impl<bool, bool (*&)(event_base*, evhttp*), event_base*, evhttp*>(std::__invoke_other, bool (*&)(event_base*, evhttp*), event_base*&&, evhttp*&&) (__f=<optimized out>, __args=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:210
    [#5](/bitcoin-bitcoin/5/)  std::__invoke<bool (*&)(event_base*, evhttp*), event_base*, evhttp*>(bool (*&)(event_base*, evhttp*), event_base*&&, evhttp*&&) (__fn=<optimized out>, 
        __args=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
    [#6](/bitcoin-bitcoin/6/)  std::reference_wrapper<bool (*)(event_base*, evhttp*)>::operator()<event_base*, evhttp*>(event_base*&&, evhttp*&&) const (__args=<optimized out>, 
        this=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:465
    [#7](/bitcoin-bitcoin/7/)  std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (
        this=0x7f7447d080f3 <epoll_wait+51>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#8](/bitcoin-bitcoin/8/)  std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>::operator()() (this=0x7f7447d080f3 <epoll_wait+51>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#9](/bitcoin-bitcoin/9/)  std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>, bool>::operator()() const (this=0x7f741bffecd0)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:1325
    [#10](/bitcoin-bitcoin/10/) 0x000056427ac2510f in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<std::reference_wrapper<bool (*)(event_base*, evhttp*)> (event_base*, evhttp*)>, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1716
    [#11](/bitcoin-bitcoin/11/) 0x000056427ab192ba in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (
        this=0x7f741c047420) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#12](/bitcoin-bitcoin/12/) std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (this=0x7f741c047c80, __f=0x7f741c047420, __did_set=0x7f741bffec8f) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:533
    [#13](/bitcoin-bitcoin/13/) 0x000056427ab19356 in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=<optimized out>, __args=<optimized out>, __args=<optimized out>, __t=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
    [#14](/bitcoin-bitcoin/14/) std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=<optimized out>, __args=<optimized out>, __args=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
    [#15](/bitcoin-bitcoin/15/) void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:602
    [#16](/bitcoin-bitcoin/16/) void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:607
    [#17](/bitcoin-bitcoin/17/) void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::__invoke() () at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:607
    [#18](/bitcoin-bitcoin/18/) 0x00007f7449766739 in __pthread_once_slow (once_control=0x7f741c047c98, init_routine=0x7f7448592120 <__once_proxy>) at pthread_once.c:116
    [#19](/bitcoin-bitcoin/19/) 0x000056427ab19239 in __gthread_once (__func=0x7f741c047420, __once=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:699
    [#20](/bitcoin-bitcoin/20/) std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__args=<optimized out>, __args=<optimized out>, __args=<optimized out>, __once=..., __f=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:614
    [#21](/bitcoin-bitcoin/21/) std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x7f741c047c80, __res=..., __ignore_failure=false) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:392
    [#22](/bitcoin-bitcoin/22/) 0x000056427ac24cb1 in std::__future_base::_Task_state<bool (*)(event_base*, evhttp*), std::allocator<int>, bool (event_base*, evhttp*)>::_M_run(event_base*&&, evhttp*&&) (this=0x17, __args=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:1409
    [#23](/bitcoin-bitcoin/23/) 0x000056427ac2573b in std::packaged_task<bool (event_base*, evhttp*)>::operator()(event_base*, evhttp*) (
        __args=<error reading variable: Cannot access memory at address 0x20>, __args=<error reading variable: Cannot access memory at address 0x20>, 
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/future:1553
    [#24](/bitcoin-bitcoin/24/) std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390
    [#25](/bitcoin-bitcoin/25/) std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)>::operator()() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380
    [#26](/bitcoin-bitcoin/26/) std::thread::_State_impl<std::_Bind_simple<std::packaged_task<bool (event_base*, evhttp*)> (event_base*, evhttp*)> >::_M_run() (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197
    [#27](/bitcoin-bitcoin/27/) 0x00007f7448592e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    [#28](/bitcoin-bitcoin/28/) 0x00007f744975f494 in start_thread (arg=0x7f741bfff700) at pthread_create.c:333
    [#29](/bitcoin-bitcoin/29/) 0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 8 (Thread 0x7f7420925700 (LWP 15216)):
    [#0](/bitcoin-bitcoin/0/)  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
    [#1](/bitcoin-bitcoin/1/)  0x00007f7449761c06 in __GI___pthread_mutex_lock (mutex=0x56427b5c22b8 <cs_main>) at ../nptl/pthread_mutex_lock.c:115
    [#2](/bitcoin-bitcoin/2/)  0x000056427aa60e88 in __gthread_mutex_lock (__mutex=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
    [#3](/bitcoin-bitcoin/3/)  __gthread_recursive_mutex_lock (__mutex=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:810
    [#4](/bitcoin-bitcoin/4/)  std::recursive_mutex::lock (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:105
    [#5](/bitcoin-bitcoin/5/)  AnnotatedMixin<std::recursive_mutex>::lock (this=<optimized out>) at ./sync.h:59
    [#6](/bitcoin-bitcoin/6/)  std::unique_lock<CCriticalSection>::lock (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:267
    [#7](/bitcoin-bitcoin/7/)  CCriticalBlock::Enter (nLine=3128, this=<optimized out>, pszName=<optimized out>, pszFile=<optimized out>) at ./sync.h:128
    [#8](/bitcoin-bitcoin/8/)  CCriticalBlock::CCriticalBlock (nLine=3128, fTry=false, this=<optimized out>, mutexIn=..., pszName=<optimized out>, pszFile=<optimized out>) at ./sync.h:149
    [#9](/bitcoin-bitcoin/9/)  PeerLogicValidation::CheckForStaleTipAndEvictPeers (this=0x7f741c47edb0, consensusParams=...) at net_processing.cpp:3128
    [#10](/bitcoin-bitcoin/10/) 0x000056427aab4fb1 in std::__invoke_impl<void, void (PeerLogicValidation::* const&)(Consensus::Params const&), PeerLogicValidation*&, Consensus::Params&> (
        __args=..., __f=<optimized out>, __t=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227
    [#11](/bitcoin-bitcoin/11/) std::__invoke<void (PeerLogicValidation::* const&)(Consensus::Params const&), PeerLogicValidation*&, Consensus::Params&> (__args=<optimized out>, 
        __fn=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250
    [#12](/bitcoin-bitcoin/12/) std::_Mem_fn_base<void (PeerLogicValidation::*)(Consensus::Params const&), true>::operator()<PeerLogicValidation*&, Consensus::Params&> (
        __args=<optimized out>, this=<optimized out>, __args=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604
    [#13](/bitcoin-bitcoin/13/) std::_Bind<std::_Mem_fn<void (PeerLogicValidation::*)(Consensus::Params const&)> (PeerLogicValidation*, Consensus::Params)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=<optimized out>, __args=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:933
    [#14](/bitcoin-bitcoin/14/) std::_Bind<std::_Mem_fn<void (PeerLogicValidation::*)(Consensus::Params const&)> (PeerLogicValidation*, Consensus::Params)>::operator()<, void>() (
        this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:991
    [#15](/bitcoin-bitcoin/15/) std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (PeerLogicValidation::*)(Consensus::Params const&)> (PeerLogicValidation*, Consensus::Params)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
    [#16](/bitcoin-bitcoin/16/) 0x000056427ad31b17 in std::function<void ()>::operator()() const (this=0x7f7420924838)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#17](/bitcoin-bitcoin/17/) Repeat(CScheduler*, std::function<void ()>, long) (s=0x56427b5c9b30 <scheduler>, f=..., deltaMilliSeconds=45000) at scheduler.cpp:122
    [#18](/bitcoin-bitcoin/18/) 0x000056427ad32d5a in boost::_bi::list3<boost::_bi::value<CScheduler*>, boost::_bi::value<std::function<void ()> >, boost::_bi::value<long> >::operator()<void (*)(CScheduler*, std::function<void ()>, long), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(CScheduler*, std::function<void ()>, long), boost::_bi::list0&, int) (this=0x7f7414001688, f=<optimized out>, a=...) at /usr/include/boost/bind/bind.hpp:398
    [#19](/bitcoin-bitcoin/19/) 0x000056427ad32bb5 in boost::_bi::bind_t<void, void (*)(CScheduler*, std::function<void ()>, long), boost::_bi::list3<boost::_bi::value<CScheduler*>, boost::_bi::value<std::function<void ()> >, boost::_bi::value<long> > >::operator()() (this=0x80) at /usr/include/boost/bind/bind.hpp:1294
    [#20](/bitcoin-bitcoin/20/) std::_Function_handler<void (), boost::_bi::bind_t<void, void (*)(CScheduler*, std::function<void ()>, long), boost::_bi::list3<boost::_bi::value<CScheduler*>, boost::_bi::value<std::function<void ()> >, boost::_bi::value<long> > > >::_M_invoke(std::_Any_data const&) (__functor=...)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
    [#21](/bitcoin-bitcoin/21/) 0x000056427ad311a8 in std::function<void ()>::operator()() const (this=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#22](/bitcoin-bitcoin/22/) CScheduler::serviceQueue (this=<optimized out>) at scheduler.cpp:83
    [#23](/bitcoin-bitcoin/23/) 0x000056427aa1d3da in boost::_mfi::mf0<void, CScheduler>::operator() (p=<optimized out>, this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:49
    [#24](/bitcoin-bitcoin/24/) boost::_bi::list1<boost::_bi::value<CScheduler*> >::operator()<boost::_mfi::mf0<void, CScheduler>, boost::_bi::list0> (this=<optimized out>, f=..., a=...)
        at /usr/include/boost/bind/bind.hpp:259
    [#25](/bitcoin-bitcoin/25/) boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > >::operator() (this=<optimized out>)
        at /usr/include/boost/bind/bind.hpp:1294
    [#26](/bitcoin-bitcoin/26/) std::_Function_handler<void (), boost::_bi::bind_t<void, boost::_mfi::mf0<void, CScheduler>, boost::_bi::list1<boost::_bi::value<CScheduler*> > > >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1731
    [#27](/bitcoin-bitcoin/27/) 0x000056427aa0bced in std::function<void ()>::operator()() const (this=0x7f7420924cb8)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
    [#28](/bitcoin-bitcoin/28/) TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (name=<optimized out>, func=...) at ./util.h:317
    [#29](/bitcoin-bitcoin/29/) 0x000056427aa1d956 in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > >::operator()<void (*)(char const*, std::function<void ()>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(char const*, std::function<void ()>), boost::_bi::list0&, int) (this=0x7f741c0430e0, 
        f=<optimized out>, a=...) at /usr/include/boost/bind/bind.hpp:319
    [#30](/bitcoin-bitcoin/30/) 0x000056427aa1d8df in boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > >::operator()() (this=<optimized out>) at /usr/include/boost/bind/bind.hpp:1294
    [#31](/bitcoin-bitcoin/31/) boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=<optimized out>) at /usr/include/boost/thread/detail/thread.hpp:116
    [#32](/bitcoin-bitcoin/32/) 0x00007f744b303116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
    [#33](/bitcoin-bitcoin/33/) 0x00007f744975f494 in start_thread (arg=0x7f7420925700) at pthread_create.c:333
    [#34](/bitcoin-bitcoin/34/) 0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 7 (Thread 0x7f7421176700 (LWP 15215)):
    [#0](/bitcoin-bitcoin/0/)  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
    [#1](/bitcoin-bitcoin/1/)  0x000056427aa0ffab in boost::condition_variable::wait (this=<optimized out>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
    [#2](/bitcoin-bitcoin/2/)  0x000056427abc5b65 in CCheckQueue<CScriptCheck>::Loop (this=<optimized out>, fMaster=<optimized out>) at ./checkqueue.h:99
    [#3](/bitcoin-bitcoin/3/)  0x000056427ab825c8 in CCheckQueue<CScriptCheck>::Thread (this=0x56427b5ccadc <scriptcheckqueue+84>) at ./checkqueue.h:136
    [#4](/bitcoin-bitcoin/4/)  ThreadScriptCheck () at validation.cpp:1680
    [#5](/bitcoin-bitcoin/5/)  0x000056427aa1cfb4 in boost::detail::thread_data<void (*)()>::run (this=0x56427b5ccadc <scriptcheckqueue+84>) at /usr/include/boost/thread/detail/thread.hpp:116
    [#6](/bitcoin-bitcoin/6/)  0x00007f744b303116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
    [#7](/bitcoin-bitcoin/7/)  0x00007f744975f494 in start_thread (arg=0x7f7421176700) at pthread_create.c:333
    [#8](/bitcoin-bitcoin/8/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 6 (Thread 0x7f7423fff700 (LWP 15214)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447cfe6ad in poll () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f74456359f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#2](/bitcoin-bitcoin/2/)  0x00007f7445635b0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#3](/bitcoin-bitcoin/3/)  0x00007f744a2c604f in QEventDispatcherGlib::processEvents (this=0x7f741c0008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
    [#4](/bitcoin-bitcoin/4/)  0x00007f744a26f9ca in QEventLoop::exec (this=this@entry=0x7f7423ffecc0, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
    [#5](/bitcoin-bitcoin/5/)  0x00007f744a09d0f3 in QThread::exec (this=<optimized out>) at thread/qthread.cpp:507
    [#6](/bitcoin-bitcoin/6/)  0x00007f744a0a1da8 in QThreadPrivate::start (arg=0x56427c486540) at thread/qthread_unix.cpp:368
    [#7](/bitcoin-bitcoin/7/)  0x00007f744975f494 in start_thread (arg=0x7f7423fff700) at pthread_create.c:333
    [#8](/bitcoin-bitcoin/8/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 5 (Thread 0x7f7430bec700 (LWP 15213)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447cfe6ad in poll () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f74456359f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#2](/bitcoin-bitcoin/2/)  0x00007f7445635b0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#3](/bitcoin-bitcoin/3/)  0x00007f744a2c604f in QEventDispatcherGlib::processEvents (this=0x7f74280008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
    [#4](/bitcoin-bitcoin/4/)  0x00007f744a26f9ca in QEventLoop::exec (this=this@entry=0x7f7430bebc90, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
    [#5](/bitcoin-bitcoin/5/)  0x00007f744a09d0f3 in QThread::exec (this=this@entry=0x7f744a54cd60 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>)
        at thread/qthread.cpp:507
    [#6](/bitcoin-bitcoin/6/)  0x00007f744a4d76d5 in QDBusConnectionManager::run (this=0x7f744a54cd60 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>)
        at qdbusconnection.cpp:178
    [#7](/bitcoin-bitcoin/7/)  0x00007f744a0a1da8 in QThreadPrivate::start (arg=0x7f744a54cd60 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>)
        at thread/qthread_unix.cpp:368
    [#8](/bitcoin-bitcoin/8/)  0x00007f744975f494 in start_thread (arg=0x7f7430bec700) at pthread_create.c:333
    [#9](/bitcoin-bitcoin/9/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 4 (Thread 0x7f7432dc0700 (LWP 15212)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447cfe6ad in poll () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f74456359f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#2](/bitcoin-bitcoin/2/)  0x00007f7445635d82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#3](/bitcoin-bitcoin/3/)  0x00007f74354fd656 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
    [#4](/bitcoin-bitcoin/4/)  0x00007f744565d3d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#5](/bitcoin-bitcoin/5/)  0x00007f744975f494 in start_thread (arg=0x7f7432dc0700) at pthread_create.c:333
    [#6](/bitcoin-bitcoin/6/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 3 (Thread 0x7f7433611700 (LWP 15211)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447cfe6ad in poll () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f74456359f6 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#2](/bitcoin-bitcoin/2/)  0x00007f7445635b0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#3](/bitcoin-bitcoin/3/)  0x00007f7445635b51 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#4](/bitcoin-bitcoin/4/)  0x00007f744565d3d5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#5](/bitcoin-bitcoin/5/)  0x00007f744975f494 in start_thread (arg=0x7f7433611700) at pthread_create.c:333
    [#6](/bitcoin-bitcoin/6/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 2 (Thread 0x7f743d3de700 (LWP 15210)):
    [#0](/bitcoin-bitcoin/0/)  0x00007f7447cfe6ad in poll () at ../sysdeps/unix/syscall-template.S:84
    [#1](/bitcoin-bitcoin/1/)  0x00007f7443502150 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
    [#2](/bitcoin-bitcoin/2/)  0x00007f7443503ee9 in xcb_wait_for_event () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
    [#3](/bitcoin-bitcoin/3/)  0x00007f743fb3fb69 in QXcbEventReader::run (this=0x56427c1a0220) at qxcbconnection.cpp:1343
    [#4](/bitcoin-bitcoin/4/)  0x00007f744a0a1da8 in QThreadPrivate::start (arg=0x56427c1a0220) at thread/qthread_unix.cpp:368
    [#5](/bitcoin-bitcoin/5/)  0x00007f744975f494 in start_thread (arg=0x7f743d3de700) at pthread_create.c:333
    [#6](/bitcoin-bitcoin/6/)  0x00007f7447d07aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    
    Thread 1 (Thread 0x7f744bea6900 (LWP 15209)):
    [#0](/bitcoin-bitcoin/0/)  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
    [#1](/bitcoin-bitcoin/1/)  0x00007f7449761c06 in __GI___pthread_mutex_lock (mutex=0x56427b5cc7b8 <mempool+56>) at ../nptl/pthread_mutex_lock.c:115
    [#2](/bitcoin-bitcoin/2/)  0x000056427a8b7bf4 in __gthread_mutex_lock (__mutex=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:748
    [#3](/bitcoin-bitcoin/3/)  __gthread_recursive_mutex_lock (__mutex=<optimized out>)
        at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/x86_64-linux-gnu/c++/6.3.0/bits/gthr-default.h:810
    [#4](/bitcoin-bitcoin/4/)  std::recursive_mutex::lock (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/mutex:105
    [#5](/bitcoin-bitcoin/5/)  AnnotatedMixin<std::recursive_mutex>::lock (this=<optimized out>) at ./sync.h:59
    [#6](/bitcoin-bitcoin/6/)  std::unique_lock<CCriticalSection>::lock (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/std_mutex.h:267
    [#7](/bitcoin-bitcoin/7/)  CCriticalBlock::Enter (nLine=623, this=<optimized out>, pszName=<optimized out>, pszFile=<optimized out>) at ./sync.h:128
    [#8](/bitcoin-bitcoin/8/)  CCriticalBlock::CCriticalBlock (nLine=623, fTry=false, this=<optimized out>, mutexIn=..., pszName=<optimized out>, pszFile=<optimized out>) at ./sync.h:149
    [#9](/bitcoin-bitcoin/9/)  CTxMemPool::size (this=<optimized out>) at ./txmempool.h:623
    [#10](/bitcoin-bitcoin/10/) ClientModel::getMempoolSize (this=<optimized out>) at qt/clientmodel.cpp:130
    [#11](/bitcoin-bitcoin/11/) ClientModel::updateTimer (this=0x56427c071aa0) at qt/clientmodel.cpp:153
    [#12](/bitcoin-bitcoin/12/) 0x000056427a94e749 in ClientModel::qt_static_metacall (_o=0x56427b5cc7b8 <mempool+56>, _c=128, _id=<optimized out>, _a=0x7f7449767f5c <__lll_lock_wait+28>)
        at qt/moc_clientmodel.cpp:140
    [#13](/bitcoin-bitcoin/13/) 0x00007f744a29d5e9 in QMetaObject::activate (sender=sender@entry=0x56427c071de0, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=0, 
        argv=argv@entry=0x0) at kernel/qobject.cpp:3740
    [#14](/bitcoin-bitcoin/14/) 0x00007f744a29df07 in QMetaObject::activate (sender=sender@entry=0x56427c071de0, m=m@entry=0x7f744a4bb760 <QTimer::staticMetaObject>, 
        local_signal_index=local_signal_index@entry=0, argv=argv@entry=0x0) at kernel/qobject.cpp:3602
    [#15](/bitcoin-bitcoin/15/) 0x00007f744a317780 in QTimer::timeout (this=this@entry=0x56427c071de0) at .moc/moc_qtimer.cpp:198
    [#16](/bitcoin-bitcoin/16/) 0x00007f744a2a9f78 in QTimer::timerEvent (this=0x56427c071de0, e=<optimized out>) at kernel/qtimer.cpp:254
    [#17](/bitcoin-bitcoin/17/) 0x00007f744a29e453 in QObject::event (this=0x56427c071de0, e=<optimized out>) at kernel/qobject.cpp:1285
    [#18](/bitcoin-bitcoin/18/) 0x00007f744abe0b8c in QApplicationPrivate::notify_helper (this=<optimized out>, receiver=0x56427c071de0, e=0x7ffca7221b20) at kernel/qapplication.cpp:3799
    [#19](/bitcoin-bitcoin/19/) 0x00007f744abe8341 in QApplication::notify (this=0x7ffca7222750, receiver=0x56427c071de0, e=0x7ffca7221b20) at kernel/qapplication.cpp:3556
    [#20](/bitcoin-bitcoin/20/) 0x00007f744a2719e0 in QCoreApplication::notifyInternal2 (receiver=0x56427c071de0, event=event@entry=0x7ffca7221b20) at kernel/qcoreapplication.cpp:988
    [#21](/bitcoin-bitcoin/21/) 0x00007f744a2c4fee in QCoreApplication::sendEvent (event=0x7ffca7221b20, receiver=<optimized out>)
        at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:231
    [#22](/bitcoin-bitcoin/22/) QTimerInfoList::activateTimers (this=0x56427c22a290) at kernel/qtimerinfo_unix.cpp:644
    [#23](/bitcoin-bitcoin/23/) 0x00007f744a2c5511 in timerSourceDispatch (source=<optimized out>) at kernel/qeventdispatcher_glib.cpp:182
    [#24](/bitcoin-bitcoin/24/) 0x00007f74456357f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#25](/bitcoin-bitcoin/25/) 0x00007f7445635a60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#26](/bitcoin-bitcoin/26/) 0x00007f7445635b0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#27](/bitcoin-bitcoin/27/) 0x00007f744a2c604f in QEventDispatcherGlib::processEvents (this=0x56427c22a070, flags=...) at kernel/qeventdispatcher_glib.cpp:423
    [#28](/bitcoin-bitcoin/28/) 0x00007f744a26f9ca in QEventLoop::exec (this=this@entry=0x7ffca7221d70, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
    [#29](/bitcoin-bitcoin/29/) 0x00007f744adcf0e7 in QDialog::exec (this=0x7ffca7221df8) at dialogs/qdialog.cpp:552
    [#30](/bitcoin-bitcoin/30/) 0x000056427a8aa885 in BitcoinGUI::message (this=0x56427c348250, title=..., message=..., style=<optimized out>, ret=0x7f73f148a9ab) at qt/bitcoingui.cpp:926
    [#31](/bitcoin-bitcoin/31/) 0x000056427a94d413 in BitcoinGUI::qt_static_metacall (_o=0x56427c348250, _c=<optimized out>, _id=<optimized out>, _a=0x7f73f148a440)
        at qt/moc_bitcoingui.cpp:216
    [#32](/bitcoin-bitcoin/32/) 0x00007f744a29e499 in QObject::event (this=this@entry=0x56427c348250, e=e@entry=0x7f73e3814a50) at kernel/qobject.cpp:1263
    [#33](/bitcoin-bitcoin/33/) 0x00007f744ac2846b in QWidget::event (this=this@entry=0x56427c348250, event=event@entry=0x7f73e3814a50) at kernel/qwidget.cpp:9225
    [#34](/bitcoin-bitcoin/34/) 0x00007f744ad2792b in QMainWindow::event (this=0x56427c348250, event=0x7f73e3814a50) at widgets/qmainwindow.cpp:1543
    [#35](/bitcoin-bitcoin/35/) 0x00007f744abe0b8c in QApplicationPrivate::notify_helper (this=<optimized out>, receiver=0x56427c348250, e=0x7f73e3814a50) at kernel/qapplication.cpp:3799
    [#36](/bitcoin-bitcoin/36/) 0x00007f744abe8341 in QApplication::notify (this=0x7ffca7222750, receiver=0x56427c348250, e=0x7f73e3814a50) at kernel/qapplication.cpp:3556
    [#37](/bitcoin-bitcoin/37/) 0x00007f744a2719e0 in QCoreApplication::notifyInternal2 (receiver=0x56427c348250, event=event@entry=0x7f73e3814a50) at kernel/qcoreapplication.cpp:988
    [#38](/bitcoin-bitcoin/38/) 0x00007f744a27416d in QCoreApplication::sendEvent (event=0x7f73e3814a50, receiver=<optimized out>)
        at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:231
    [#39](/bitcoin-bitcoin/39/) QCoreApplicationPrivate::sendPostedEvents (receiver=receiver@entry=0x0, event_type=event_type@entry=0, data=0x56427c185580) at kernel/qcoreapplication.cpp:1649
    [#40](/bitcoin-bitcoin/40/) 0x00007f744a2745d8 in QCoreApplication::sendPostedEvents (receiver=receiver@entry=0x0, event_type=event_type@entry=0) at kernel/qcoreapplication.cpp:1503
    [#41](/bitcoin-bitcoin/41/) 0x00007f744a2c5c43 in postEventSourceDispatch (s=0x56427c22a140) at kernel/qeventdispatcher_glib.cpp:276
    [#42](/bitcoin-bitcoin/42/) 0x00007f74456357f7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#43](/bitcoin-bitcoin/43/) 0x00007f7445635a60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#44](/bitcoin-bitcoin/44/) 0x00007f7445635b0c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
    [#45](/bitcoin-bitcoin/45/) 0x00007f744a2c604f in QEventDispatcherGlib::processEvents (this=0x56427c22a070, flags=...) at kernel/qeventdispatcher_glib.cpp:423
    [#46](/bitcoin-bitcoin/46/) 0x00007f744a26f9ca in QEventLoop::exec (this=this@entry=0x7ffca7222540, flags=..., flags@entry=...) at kernel/qeventloop.cpp:212
    [#47](/bitcoin-bitcoin/47/) 0x00007f744a27813c in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1261
    [#48](/bitcoin-bitcoin/48/) 0x000056427a893329 in main (argc=<optimized out>, argv=<optimized out>) at qt/bitcoin.cpp:710
    (gdb)

51. 
    ryanofsky commented at 10:26 am on February 16, 2018: member

    Note: relevant part of stack trace is near CCoinsViewErrorCatcher::GetCoin call. But the stack traces don’t really shed much light on this problem, since the corruption bug/hardware failure is happening at an earlier point and the stack traces just show what bitcoin is doing at the time when it detects the corruption.
52. 
    laanwj commented at 10:37 am on February 16, 2018: member

    Are you sure this is 0.16.x? The leveldb errors don’t have a ldb filename in them. I added that in #11674 to allow for more specific analysis of corruptions.

    Edit: apparently that hasn’t made it into the branches yet, sorry.

53. 
    dooglus commented at 6:50 pm on February 16, 2018: contributor

    The debug.log shows:

    2018-02-16 09:18:16 Bitcoin Core version v0.16.99.0-67aa58299 (release build)

    commit 67aa58299927f4ce58b23a297de8cad66b67fd76 is a local change I made to add the ‘dumpbootstrap’ RPC call. I use it to regularly make partial bootstrap.dat files so I can easily recover from the database corruption by ‘replaying’ the bootstrap files in order. That’s the only change on top of commit f0295becbf3ef1fb78095306408789253fe0c114, which came after 0.16 rc1.

54. 
    dooglus commented at 9:42 pm on February 16, 2018: contributor

    So now I’m more confused. Last night after bitcoin-qt hung on that block corruption, I had to “kill -9” to stop it. After that I tried running bitcoin-qt twice more, but both times it died immediately due to database corruption, here:

    2018-02-16 09:17:40 Verifying last 1 blocks at level 3
    2018-02-16 09:17:40 [0%]...LevelDB read failure: Corruption: block checksum mismatch
    2018-02-16 09:17:40 Corruption: block checksum mismatch
    

    and here:

    2018-02-16 09:18:20 Verifying last 1 blocks at level 3
    2018-02-16 09:18:20 [0%]...LevelDB read failure: Corruption: block checksum mismatch
    2018-02-16 09:18:20 Corruption: block checksum mismatch
    

    I gave up for the night, thinking I’d have to dig out the external hdd to restore an old pre-corruption backup.

    This morning I rebuild bitcoin-qt, using the 0.16 rc4 version, and just on a whim tried running it, on my corrupted data. It ran just fine:

    2018-02-16 21:11:56 Verifying last 1 blocks at level 3
    2018-02-16 21:11:56 [0%]...[99%]...[DONE].
    2018-02-16 21:12:46 No coin database inconsistencies in last 2 blocks (4228 transactions)
    2018-02-16 21:12:46  block index           55654ms
    2018-02-16 21:12:46 init message: Loading wallet...
    

    So what could be happening there? How was the database corrupted the first 2 times we checked it, but fine the 3rd time?

55. 
    dooglus commented at 10:31 pm on February 16, 2018: contributor

    I tried again, checking the last 1000 blocks, and it also passed fine (I reported a new issue #12465 about how the first two log lines are merged into one):

    2018-02-16 21:45:24 Prune: pruned datadir may not have more than 288 blocks; only checking available blocksVerifying last 1000 blocks at level 3
    2018-02-16 21:45:24 [0%]...[10%]...[20%]...[30%]...[40%]...[50%]...[60%]...[70%]...[80%]...[90%]...[DONE].
    2018-02-16 21:58:34 No coin database inconsistencies in last 1001 blocks (1162920 transactions)
    2018-02-16 21:58:35  block index          795664ms
    2018-02-16 21:58:36 init message: Loading wallet...
    

56. 
    eklitzke commented at 11:11 pm on March 9, 2018: contributor

    I have been doing a lot of LevelDB work recently and am interested in helping you diagnose this. Can you tell me some more information:

    • What filesystem are you using?
    • What mount options do you have enabled?
57. 
    dooglus commented at 3:45 am on March 10, 2018: contributor

    19:46:14 chris@chris:~$ df ~/.bitcoin/
    Filesystem                  Size  Used Avail Use% Mounted on
    /dev/mapper/chris--vg-root  901G  757G   99G  89% /
    
    19:46:16 chris@chris:~$ mount | grep chris--vg
    /dev/mapper/chris--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
    

    I run bitcoin-qt almost all the time, and very rarely see this issue any more. I do post to this thread almost every time it happens though.

    I don’t think I’ve seen it since upgrading to the recent released version.

    I’m using an “incompatible” version of the Berkeley DB:

    19:47:10 chris@chris:~$ dpkg -l | grep -i berkeley
    ii  libdb5.3:amd64                      5.3.28-12+deb9u1       amd64                  Berkeley v5.3 Database Libraries [runtime]
    ii  libdb5.3++:amd64                    5.3.28-12+deb9u1       amd64                  Berkeley v5.3 Database Libraries for C++ [runtime]
    ii  libdb5.3++-dev                      5.3.28-12+deb9u1       amd64                  Berkeley v5.3 Database Libraries for C++ [development]
    ii  libdb5.3-dev                        5.3.28-12+deb9u1       amd64                  Berkeley v5.3 Database Libraries [development]
    

    and configure like this:

    $ ./configure CXX=clang++ --with-incompatible-bdb --with-miniupnpc=no
    

    If I omit the –with-incompatible-bdb then ./configure gives this error:

    checking for Berkeley DB C++ headers... default
    configure: error: Found Berkeley DB other than 4.8, required for portable wallets (--with-incompatible-bdb to ignore or --disable-wallet to disable wallet functionality)
    

    It seems that 5.3 is the only version available in the debian stable repos:

    $ apt-cache search libdb dev | grep -i dev.*berkeley
    libdb++-dev - Berkeley Database Libraries for C++ [development]
    libdb-dev - Berkeley Database Libraries [development]
    libdb-java-dev - Berkeley Database Libraries for Java [development]
    libdb-sql-dev - Berkeley Database Libraries [SQL development]
    libdb5.3++-dev - Berkeley v5.3 Database Libraries for C++ [development]
    libdb5.3-dev - Berkeley v5.3 Database Libraries [development]
    libdb5.3-java-dev - Berkeley v5.3 Database Libraries for Java [development]
    libdb5.3-sql-dev - Berkeley v5.3 Database Libraries [SQL development]
    libdb5.3-stl-dev - Berkeley v5.3 Database Libraries [STL development]

58. 
    eklitzke commented at 6:58 am on March 10, 2018: contributor

    Next time this happens (or maybe always) you should run with -debug=coindb -debug=leveldb which will produce more verbose logging information about what’s happening in the database and leveldb. Your filesystem options look fine. Maybe you have bad hardware like other people reported, it’s impossible to tell from the error messages here.

    I would be interested in seeing what the corrupted table files look like (e.g. is it truncated, zeroed, something else). I’ve submitted a PR that changes the log messages a bit so the corrupted table name will be more obvious.

59. 
    dooglus commented at 2:05 pm on March 10, 2018: contributor

    I’ve added those debug= lines to my bitcoin.conf.

    I’ll try using a different laptop to rule out hardware issues.

    Most times when the client crashes complaining about data corruption I can simply restart it and everything’s fine. I’ve only had to restore from backup 2 or 3 times I think.

    I’ll try running with your PR #12659 from now on.

60. MarcoFalke referenced this in commit 3d16f58153 on Mar 13, 2018
61. 
    meshcollider commented at 9:00 pm on March 28, 2018: contributor

    Has there been any further occurrences of this? If it has stopped happening, maybe we should close this, can always be reopened if it does pop up again
62. MarcoFalke closed this on Mar 28, 2018

    ---

63. 
    dooglus commented at 10:22 pm on March 28, 2018: contributor

    It doesn’t happen very often, but it never did.

    Closing the issue won’t help resolve the problem, but feel free I guess.

64. 
    simonk83 commented at 6:48 am on March 31, 2018: none

    I’m seeing it happen fairly regularly while trying to initial sync on a Pi. Going to try and reduce the CPU usage a little (or buy heatsinks) to try and solve it that way.
65. 
    dooglus commented at 2:14 am on April 13, 2018: contributor

    @eklitzke

    Next time this happens (or maybe always) you should run with -debug=coindb -debug=leveldb which will produce more verbose logging information about what’s happening in the database and leveldb.

    I have been running with debug=coindb and debug=leveldb in my bitcoin.conf since you suggested it, and have just seen the first crash since I did that.

    I’ve submitted a PR that changes the log messages a bit so the corrupted table name will be more obvious.

    I also applied your patch, so I got the improved log messages.

    I turned off the wireless interface on my laptop and then went out for the evening. When I came back I turned the wireless interface back on and noticed that bitcoin-qt had crashed. I don’t know which happened first.

    Here’s the end of the debug.log file:

    2018-04-12 21:26:47 UpdateTip: new best=0000000000000000001222a6e792d981ea1aaf23acf80c2575c56a2984afbc2e height=517916 version=0x20000000 log2_work=88.571705 tx=309842586 date='2018-04-12 21:26:06' progress=1.000000 cache=64.7MiB(496499txo)
    2018-04-12 21:29:57 UpdateTip: new best=00000000000000000004fda8f3226b265c8c1e391c16001fe8a7fb4b838b0cf0 height=517917 version=0x20000000 log2_work=88.571752 tx=309843005 date='2018-04-12 21:30:00' progress=1.000000 cache=64.8MiB(497061txo)
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/206"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/205"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/206"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/205"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/207"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/206"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/205"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/207"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/206"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/205"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/207"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/206"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/205"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/207"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/206"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/205"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/207"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/206"
    2018-04-12 21:55:36 GUI: "No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/ActiveConnection/205"
    2018-04-12 21:55:42 connect() to 34.216.21.223:8333 failed: Network is unreachable (101)
    2018-04-12 21:56:07 connect() to 108.173.208.45:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:29 Potential stale tip detected, will try using extra outbound peer (last tip update: 16712 seconds ago)
    2018-04-13 02:08:29 socket receive timeout: 15175s
    2018-04-13 02:08:29 socket receive timeout: 15188s
    2018-04-13 02:08:29 socket receive timeout: 15175s
    2018-04-13 02:08:29 socket receive timeout: 15188s
    2018-04-13 02:08:29 socket receive timeout: 15181s
    2018-04-13 02:08:29 socket receive timeout: 15180s
    2018-04-13 02:08:29 socket receive timeout: 15188s
    2018-04-13 02:08:29 socket sending timeout: 15175s
    2018-04-13 02:08:29 connect() to 212.52.168.118:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:29 connect() to 73.184.173.230:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:30 connect() to 190.10.8.211:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:30 connect() to 52.11.12.45:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:31 connect() to 110.142.250.119:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:31 connect() to 8.24.98.68:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:32 connect() to 82.95.67.118:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:32 connect() to 123.112.64.214:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:33 connect() to 114.218.100.170:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:33 connect() to 212.92.124.221:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:34 connect() to 104.199.0.235:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:34 connect() to 209.250.6.190:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:35 connect() to 94.23.88.166:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:35 connect() to 78.40.244.243:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:36 connect() to [2a01:4f8:150:64e9::2]:8333 failed: Cannot assign requested address (99)
    2018-04-13 02:08:36 connect() to 37.98.254.134:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:37 connect() to [2001:0:9d38:78cf:c83:16ed:3f57:4dc1]:8333 failed: Cannot assign requested address (99)
    2018-04-13 02:08:37 connect() to 84.92.55.182:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:38 connect() to 89.232.71.101:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:38 connect() to 45.63.18.100:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:39 connect() to 52.198.143.134:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:39 connect() to 138.197.177.202:8333 failed: Network is unreachable (101)
    2018-04-13 02:08:40 connect() to [2001:0:4137:9e76:14c5:3fa:5d3a:3542]:8333 failed: Cannot assign requested address (99)
    2018-04-13 02:08:52 connect() to 23.19.87.136:8333 failed after select(): Connection refused (111)
    2018-04-13 02:09:05 New outbound peer connected: version: 70015, blocks=517944, peer=82
    2018-04-13 02:09:11 LevelDB read failure: Corruption: block checksum mismatch: /home/chris/.bitcoin-prune/chainstate/743886.ldb
    2018-04-13 02:09:11 Fatal LevelDB error: Corruption: block checksum mismatch: /home/chris/.bitcoin-prune/chainstate/743886.ldb
    2018-04-13 02:09:11 You can use -debug=leveldb to get more complete diagnostic messages
    

    Hopefully that helps. I’ll make a backup of the whole folder as it currently stands in case you want to see anything from it.

    Edit: earlier in the day I had been seeing a bunch of database compacting going on, for example:

    2018-04-12 13:57:08 UpdateTip: new best=0000000000000000003d7729716df3a6ae287ce2c9144c6a52175a7c8828ec10 height=517864 version=0x20000000 log2_work=88.569243 tx=309751135 date='2018-04-12 13:57:20' progress=1.000000 cache=42.9MiB(318373txo)
    2018-04-12 13:57:30 connect() to 179.26.82.15:8333 failed after select(): Connection refused (111)
    2018-04-12 13:57:54 connect() to 60.2.78.18:8333 failed after select(): Connection refused (111)
    2018-04-12 14:01:27 leveldb: Compacting 1@3 + 2@4 files
    2018-04-12 14:01:27 leveldb: Generated table [#744786](/bitcoin-bitcoin/744786/)@3: 36550 keys, 2172606 bytes
    2018-04-12 14:01:27 leveldb: Generated table [#744787](/bitcoin-bitcoin/744787/)@3: 36057 keys, 2168714 bytes
    2018-04-12 14:01:27 leveldb: Compacted 1@3 + 2@4 files => 4341320 bytes
    2018-04-12 14:01:27 leveldb: compacted to: files[ 0 0 0 0 1429 0 0 ]
    2018-04-12 14:01:27 leveldb: Delete type=2 [#743335](/bitcoin-bitcoin/743335/)
    2018-04-12 14:01:27 leveldb: Delete type=2 [#743679](/bitcoin-bitcoin/743679/)
    2018-04-12 14:01:27 leveldb: Delete type=2 [#742984](/bitcoin-bitcoin/742984/)
    2018-04-12 14:04:16 connect() to 37.57.3.222:8333 failed after select(): Connection refused (111)
    2018-04-12 14:06:14 socket recv error Connection reset by peer (104)
    2018-04-12 14:06:43 New outbound peer connected: version: 70015, blocks=517864, peer=71
    

    so the leveldb debugging was definitely enabled.

66. 
    quannv108 commented at 2:49 pm on January 11, 2019: none

    I running my full node on my raspi 3, and it continue top stop on random index For examples: Fatal LevelDB error: Corruption: truncated block read: /home/pi/.bitcoin/indexes/txindex/066234.ldb Tried to run reIndex so much time, some time it stop after 7% check, sometime 61% check, take me too much time to stop and start bitcoin service. I don’t know that the issues is already resolved or not, I can’t not find any solution, just keep continue reindex is not a good choice.
67. jasonbcox referenced this in commit 19ba4062aa on Jul 25, 2019
68. jonspock referenced this in commit 4587107c79 on Sep 7, 2019
69. jonspock referenced this in commit df00ef61f3 on Sep 7, 2019
70. proteanx referenced this in commit 7e2973fce7 on Sep 9, 2019
71. 
    udovdh commented at 8:03 am on October 3, 2019: none

    Bitcoin core is corrupt as whatever. We can download 200G+ of a blockchain with no issue and within one month of data it finds corruption in the incoming data.
72. 
    udovdh commented at 8:06 am on October 3, 2019: none

    And when we reindex it simply discards all.
73. 
    udovdh commented at 8:38 am on October 3, 2019: none

    So why can’t it handle an error situation by repairing the problem?
74. 
    udovdh commented at 12:42 pm on October 3, 2019: none

    The whole idea is of a resilient decentralised system yet this does not reflect in the way the blockchain is handled…
75. 
    udovdh commented at 12:43 pm on October 3, 2019: none

    Storage is on raid10, cpu is well within operating limits and nothing crashes except bitcoin core which does not handle anything.
76. 
    dooglus commented at 2:10 pm on October 3, 2019: contributor

    For what it’s worth I was seeing this kind of error pretty regularly until I upgraded my hardware.

    I don’t know if it was being triggered by faulty RAM, faulty hard drive, or a race condition in the software which shows up more regularly in a slow system than a fast one, but I’ve not seen the problem at all since upgrading the hardware (faster CPU and RAM, SSD instead of HDD).

77. 
    udovdh commented at 5:17 am on October 4, 2019: none

    When things go wrong, the log looks like this:

    $ grep 003095.ldb debug.log 2019-10-04T05:03:28Z leveldb: Compaction error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:12Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:12Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:12Z Failed to connect best block (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb (code 0)) 2019-10-04T05:06:13Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z FlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb (code 0)) 2019-10-04T05:06:13Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z FlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb (code 0))

    So it says nothing about when or while the file was written nor what is in there or what is wrong except for a checksum. Please note there is no media error.

    2019-10-04T05:03:28Z leveldb: Compaction error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb (….)

    2019-10-04T05:06:12Z UpdateTip: new best=0000000000000000116dfb22fc23ea79f61734777a0660a3942fa6ecc1cb1a15 height=342530 version=0x00000002 log2_work=82.187887 tx=58998819 date=‘2015-02-08T08:28:04Z’ progress=0.129192 cache=723.8MiB(5399297txo) 2019-10-04T05:06:12Z UpdateTip: new best=0000000000000000068f1d2a8f90fcc2383bbdd18183e38fd1a29998462a2002 height=342531 version=0x00000002 log2_work=82.187933 tx=59000545 date=‘2015-02-08T08:58:24Z’ progress=0.129196 cache=723.9MiB(5400210txo) 2019-10-04T05:06:12Z UpdateTip: new best=0000000000000000182fd54b2b3c2e1df1c81ea3141d5e2b0b0adffbf48c4740 height=342532 version=0x00000002 log2_work=82.18798 tx=59000678 date=‘2015-02-08T09:00:40Z’ progress=0.129197 cache=723.9MiB(5400174txo) 2019-10-04T05:06:12Z UpdateTip: new best=000000000000000008d3e4e81af806f0144275cf338bcbbf537259c4d9d47515 height=342533 version=0x00000002 log2_work=82.188026 tx=59001160 date=‘2015-02-08T09:09:52Z’ progress=0.129198 cache=724.4MiB(5404082txo) 2019-10-04T05:06:12Z leveldb: Level-0 table #13: started 2019-10-04T05:06:12Z leveldb: Level-0 table #13: 2062883 bytes OK 2019-10-04T05:06:12Z WriteBatch memory usage: db=index, before=2.3MiB, after=4.1MiB 2019-10-04T05:06:12Z leveldb: Delete type=0 #10 2019-10-04T05:06:12Z leveldb: Compacting 4@0 + 0@1 files 2019-10-04T05:06:12Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:12Z You can use -debug=leveldb to get more complete diagnostic messages 2019-10-04T05:06:12Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:12Z Error: Error: A fatal internal error occurred, see debug.log for details 2019-10-04T05:06:12Z Failed to connect best block (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb (code 0)) 2019-10-04T05:06:12Z tor: Thread interrupt 2019-10-04T05:06:12Z addcon thread exit 2019-10-04T05:06:12Z Shutdown: In progress… 2019-10-04T05:06:12Z opencon thread exit 2019-10-04T05:06:12Z torcontrol thread exit 2019-10-04T05:06:12Z msghand thread exit 2019-10-04T05:06:12Z net thread exit 2019-10-04T05:06:12Z leveldb: Generated table #14@0: 14970 keys, 2141006 bytes 2019-10-04T05:06:13Z leveldb: Generated table #15@0: 14970 keys, 2140790 bytes 2019-10-04T05:06:13Z scheduler thread interrupt 2019-10-04T05:06:13Z leveldb: Generated table #16@0: 14970 keys, 2141034 bytes 2019-10-04T05:06:13Z leveldb: Level-0 table #19: started 2019-10-04T05:06:13Z WriteBatch memory usage: db=index, before=1.9MiB, after=1.9MiB 2019-10-04T05:06:13Z leveldb: Level-0 table #19: 1733707 bytes OK 2019-10-04T05:06:13Z leveldb: Delete type=0 #12 2019-10-04T05:06:13Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z You can use -debug=leveldb to get more complete diagnostic messages 2019-10-04T05:06:13Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z Error: Error: A fatal internal error occurred, see debug.log for details 2019-10-04T05:06:13Z FlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb (code 0)) 2019-10-04T05:06:13Z leveldb: Generated table #17@0: 14970 keys, 2141644 bytes 2019-10-04T05:06:13Z WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB 2019-10-04T05:06:13Z leveldb: Generated table #20@0: 14970 keys, 2142025 bytes 2019-10-04T05:06:13Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z You can use -debug=leveldb to get more complete diagnostic messages 2019-10-04T05:06:13Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb 2019-10-04T05:06:13Z Error: Error: A fatal internal error occurred, see debug.log for details 2019-10-04T05:06:13Z FlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/003095.ldb (code 0)) 2019-10-04T05:06:13Z leveldb: Generated table #21@0: 14970 keys, 2141588 bytes 2019-10-04T05:06:13Z leveldb: Generated table #22@0: 7326 keys, 1033193 bytes 2019-10-04T05:06:13Z leveldb: Compacted 4@0 + 0@1 files => 13881280 bytes 2019-10-04T05:06:13Z leveldb: compacted to: files[ 1 7 1 0 0 0 0 ] 2019-10-04T05:06:13Z leveldb: Delete type=2 #11 2019-10-04T05:06:13Z leveldb: Delete type=2 #7 2019-10-04T05:06:13Z leveldb: Delete type=2 #9 2019-10-04T05:06:13Z leveldb: Delete type=2 #13 2019-10-04T05:06:13Z leveldb: Compacting 1@1 + 1@2 files 2019-10-04T05:06:14Z leveldb: Generated table #23@1: 15180 keys, 2136400 bytes 2019-10-04T05:06:14Z leveldb: Generated table #24@1: 15180 keys, 2136247 bytes 2019-10-04T05:06:14Z leveldb: Generated table #25@1: 15180 keys, 2136455 bytes 2019-10-04T05:06:14Z leveldb: Generated table #26@1: 15180 keys, 2136422 bytes 2019-10-04T05:06:14Z leveldb: Generated table #27@1: 15180 keys, 2136125 bytes 2019-10-04T05:06:14Z leveldb: Generated table #28@1: 15180 keys, 2135914 bytes 2019-10-04T05:06:14Z leveldb: Generated table #29@1: 15270 keys, 2139288 bytes 2019-10-04T05:06:14Z leveldb: Generated table #30@1: 15270 keys, 2139297 bytes 2019-10-04T05:06:14Z leveldb: Generated table #31@1: 15270 keys, 2138600 bytes 2019-10-04T05:06:14Z leveldb: Generated table #32@1: 15270 keys, 2138684 bytes 2019-10-04T05:06:15Z leveldb: Generated table #33@1: 15270 keys, 2139227 bytes 2019-10-04T05:06:15Z leveldb: Generated table #34@1: 15270 keys, 2139012 bytes 2019-10-04T05:06:15Z leveldb: Generated table #35@1: 15270 keys, 2138954 bytes 2019-10-04T05:06:15Z leveldb: compacted to: files[ 1 7 1 0 0 0 0 ] 2019-10-04T05:06:15Z [default wallet] Releasing wallet 2019-10-04T05:06:15Z Shutdown: done

    What conclusions can we draw?

78. 
    udovdh commented at 9:55 am on November 24, 2019: none

    Just bought some BTC, saw the transaction arrive. closed bitcoin core. Restarted it after a wile and Fatal LevelDB error: Corruption: block checksum mismatch This is on fresh new hardwrae, on a raid10, no fs issues, etc. Bitcoin core has an issue. How to fix besides downloading the whole chain?
79. 
    udovdh commented at 10:35 am on November 24, 2019: none

    The main issue here is that bitcoin core has no recovery mechanism at all for these kinds of (common?) failures. My storage is redundant, the CPU does not get too hot, same for the rest of the board. Yet still issues. please provide a decent fix instead of putting the blame elsewhere.
80. 
    kollokollo commented at 1:32 pm on December 9, 2019: none

    Yes, udovdh is right. The hardware can be really good and expensive, but we all know, that eventually every storage media will sooner or later fail! Electronics an be perfect, and eventually it will fail (because of Single event upsets caused by comic radiation or watsoever). That why there are RAID-Arrays and redundant processors (on satellites p.ex.). It is not the users faul.
81. PastaPastaPasta referenced this in commit bbb3e74ce7 on Jun 13, 2020
82. PastaPastaPasta referenced this in commit 57a6035a90 on Jun 13, 2020
83. PastaPastaPasta referenced this in commit 185f2d6cfa on Jun 17, 2020
84. PastaPastaPasta referenced this in commit f2afaa324f on Jun 18, 2020
85. gades referenced this in commit 6a194112d9 on Jun 24, 2021
86. MarcoFalke locked this on Dec 16, 2021

Contributors
dooglus jlegoupil ryanofsky gmaxwell TheBlueMatt sangaman kollokollo laanwj eklitzke meshcollider simonk83 quannv108 udovdh

Labels
Data corruption