Bitcoind v0.11.0 segfaults on startup trying to read wallet. #6775

1. 
   OverlordQ commented at 4:33 pm on October 7, 2015: contributor

   GDB output:

    0(gdb) r -debug
    1Starting program: /usr/local/bin/bitcoind -debug
    2[Thread debugging using libthread_db enabled]
    3Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    4[New Thread 0x7ffff5238700 (LWP 17767)]
    5[New Thread 0x7ffff4a37700 (LWP 17768)]
    6[New Thread 0x7fffeffff700 (LWP 17769)]
    7[New Thread 0x7fffef7fe700 (LWP 17770)]
    8[New Thread 0x7fffeeffd700 (LWP 17771)]
    9[New Thread 0x7fffee7fc700 (LWP 17772)]
   10[New Thread 0x7fffedffb700 (LWP 17773)]
   11[New Thread 0x7fffed7fa700 (LWP 17774)]
   12[New Thread 0x7fffecff9700 (LWP 17775)]
   13[New Thread 0x7fffcffff700 (LWP 17776)]
   14[New Thread 0x7fffcf7fe700 (LWP 17777)]
   15[New Thread 0x7fffceffd700 (LWP 17778)]
   16
   17Program received signal SIGSEGV, Segmentation fault.
   18strlen () at ../sysdeps/x86_64/strlen.S:106
   19106     ../sysdeps/x86_64/strlen.S: No such file or directory.
   20(gdb) bt
   21[#0](/bitcoin-bitcoin/0/)  strlen () at ../sysdeps/x86_64/strlen.S:106
   22[#1](/bitcoin-bitcoin/1/)  0x00007ffff70bace2 in __os_strdup () from /usr/lib/libdb_cxx-4.8.so
   23[#2](/bitcoin-bitcoin/2/)  0x00007ffff708d6b4 in __env_add_data_dir () from /usr/lib/libdb_cxx-4.8.so
   24[#3](/bitcoin-bitcoin/3/)  0x00007ffff708d781 in __env_set_data_dir () from /usr/lib/libdb_cxx-4.8.so
   25[#4](/bitcoin-bitcoin/4/)  0x00007ffff6fac5d5 in DbEnv::set_data_dir(char const*) () from /usr/lib/libdb_cxx-4.8.so
   26[#5](/bitcoin-bitcoin/5/)  0x00005555557cf191 in CDBEnv::Open (this=0x555555c004a0 <bitdb>, pathIn=...) at wallet/db.cpp:92
   27[#6](/bitcoin-bitcoin/6/)  0x0000555555816a01 in CWallet::Verify (walletFile="wallet.dat", warningString="", errorString="") at wallet/wallet.cpp:353
   28[#7](/bitcoin-bitcoin/7/)  0x00005555555ab574 in AppInit2 (threadGroup=..., scheduler=...) at init.cpp:957
   29[#8](/bitcoin-bitcoin/8/)  0x000055555558ffe9 in AppInit (argc=<optimized out>, argv=<optimized out>) at bitcoind.cpp:147
   30[#9](/bitcoin-bitcoin/9/)  0x000055555558736f in main (argc=2, argv=0x7fffffffe6e8) at bitcoind.cpp:176
   

   Logfile Output:

    02015-10-07 16:30:21 Bitcoin version v0.11.0 (2015-07-10 19:23:55 +0200)
    12015-10-07 16:30:21 Using OpenSSL version OpenSSL 1.0.2d 9 Jul 2015
    22015-10-07 16:30:21 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
    32015-10-07 16:30:21 Default data directory /home/bitnode/.bitcoin
    42015-10-07 16:30:21 Using data directory /home/bitnode/.bitcoin
    52015-10-07 16:30:21 Using config file /home/bitnode/.bitcoin/bitcoin.conf
    62015-10-07 16:30:21 Using at most 700 connections (65536 file descriptors available)
    72015-10-07 16:30:21 Using 8 threads for script verification
    82015-10-07 16:30:21 scheduler thread start
    92015-10-07 16:30:21 Allowing RPC connections from: 127.0.0.0/255.0.0.0 ::1/ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
   102015-10-07 16:30:21 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0)
   112015-10-07 16:30:21 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0)
   122015-10-07 16:30:21 Using wallet wallet.dat
   132015-10-07 16:30:21 init message: Verifying wallet...
   142015-10-07 16:30:21 CDBEnv::Open: LogDir=/home/bitnode/.bitcoin/database ErrorFile=/home/bitnode/.bitcoin/db.log
   

2. 
   paveljanik commented at 4:37 pm on October 7, 2015: contributor

   /home/bitnode/.bitcoin/db.log is empty? It starts OK without wallet.dat?
3. 
   OverlordQ commented at 4:38 pm on October 7, 2015: contributor

   No db.log created, also, no wallet.dat in the folder.
4. 
   sipa commented at 4:39 pm on October 7, 2015: member

   What are you setting datadir or wallet to in bitcoin.conf?
5. 
   paveljanik commented at 4:41 pm on October 7, 2015: contributor

   Can you show us/remove bitcoin.conf please? Is your datadir writable etc.?
6. 
   OverlordQ commented at 4:42 pm on October 7, 2015: contributor

   Not being set in bitcoin.conf. Contents of bitcoin.conf:

   0rpcuser=yanot
   1rpcpassword=really
   2listen=1
   3txindex=1
   4server=1
   

7. 
   paveljanik commented at 4:42 pm on October 7, 2015: contributor

   Can you remove the conf file just for test?
8. 
   OverlordQ commented at 4:44 pm on October 7, 2015: contributor

   Same error moving bitcoin.conf out of the way.
9. 
   paveljanik commented at 4:46 pm on October 7, 2015: contributor

   can you please run ls -laR ~/.bitcoin?
10. 
    OverlordQ commented at 4:49 pm on October 7, 2015: contributor

    http://sprunge.us/CPhZ
11. 
    paveljanik commented at 4:51 pm on October 7, 2015: contributor

    Looks perfectly OK. So bitcoind is self-compiled? What OS? df -h OK?
12. 
    paveljanik commented at 4:53 pm on October 7, 2015: contributor

    Can you run bitcoind under strace -fff?
13. 
    OverlordQ commented at 4:56 pm on October 7, 2015: contributor

    Debian Stretch (yes I know its ’testing’), plenty of space free.

    Strace: http://sprunge.us/hOjT

    Moving blocks and chainstate folders out of the way had no effect either.

14. 
    sipa commented at 5:00 pm on October 7, 2015: member

    Are you able to create a ~/.bitcoin/database directory yourself? If not, what error do you get?
15. 
    OverlordQ commented at 5:01 pm on October 7, 2015: contributor

    Works fine from the shell.

    0bitnode@yig:~/.bitcoin$ ls
    1banlist.dat  bitcoin.conf.bak  bitcoind.pid  blocks  chainstate  database  debug.log  fee_estimates.dat  log  peers.dat
    2bitnode@yig:~/.bitcoin$ rm -rf database/
    3bitnode@yig:~/.bitcoin$ ls
    4banlist.dat  bitcoin.conf.bak  bitcoind.pid  blocks  chainstate  debug.log  fee_estimates.dat  log  peers.dat
    5bitnode@yig:~/.bitcoin$ mkdir database
    6bitnode@yig:~/.bitcoin$ ls
    7banlist.dat  bitcoin.conf.bak  bitcoind.pid  blocks  chainstate  database  debug.log  fee_estimates.dat  log  peers.dat
    

16. 
    paveljanik commented at 5:04 pm on October 7, 2015: contributor

    strace output with bitcoin.conf please :-)
17. 
    OverlordQ commented at 5:07 pm on October 7, 2015: contributor

    http://sprunge.us/cBhE
18. 
    paveljanik commented at 5:17 pm on October 7, 2015: contributor

    This is pretty strange 8) Can you try released version just to compare? There are plenty of

    0[pid  1512] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
    

    Debian Stretch bugs…

19. 
    OverlordQ commented at 5:29 pm on October 7, 2015: contributor

    Binary release of v0.11.0 appears to run w/o issue.

    FWIW:

    0$ ./test_bitcoin
    1Running 155 test cases...
    2unknown location(0): fatal error in "AlertApplies": memory access violation at address: 0x00000009: no mapping at fault address
    3test/arith_uint256_tests.cpp(564): last checkpoint
    4
    5*** 1 failure detected in test suite "Bitcoin Test Suite"
    

    and relavent BT:

     0(gdb) bt
     1[#0](/bitcoin-bitcoin/0/)  strlen () at ../sysdeps/x86_64/strlen.S:106
     2[#1](/bitcoin-bitcoin/1/)  0x00007ffff6e0cce2 in __os_strdup () from /usr/lib/libdb_cxx-4.8.so
     3[#2](/bitcoin-bitcoin/2/)  0x00007ffff6ddf6b4 in __env_add_data_dir () from /usr/lib/libdb_cxx-4.8.so
     4[#3](/bitcoin-bitcoin/3/)  0x00007ffff6ddf781 in __env_set_data_dir () from /usr/lib/libdb_cxx-4.8.so
     5[#4](/bitcoin-bitcoin/4/)  0x00007ffff6cfe5d5 in DbEnv::set_data_dir(char const*) () from /usr/lib/libdb_cxx-4.8.so
     6[#5](/bitcoin-bitcoin/5/)  0x0000555555b6b8c2 in CDBEnv::MakeMock (this=0x555556229c40 <bitdb>) at wallet/db.cpp:128
     7[#6](/bitcoin-bitcoin/6/)  0x0000555555754405 in TestingSetup::TestingSetup (this=0x7fffffffd8a0) at test/test_bitcoin.cpp:46
     8[#7](/bitcoin-bitcoin/7/)  0x00005555555bff31 in ReadAlerts::ReadAlerts (this=0x7fffffffd8a0) at test/alert_tests.cpp:89
     9[#8](/bitcoin-bitcoin/8/)  0x00005555555c02f9 in Alert_tests::AlertApplies::AlertApplies (this=0x7fffffffd8a0) at test/alert_tests.cpp:122
    10[#9](/bitcoin-bitcoin/9/)  0x00005555555ba74a in Alert_tests::AlertApplies_invoker () at test/alert_tests.cpp:122
    11[#10](/bitcoin-bitcoin/10/) 0x00005555555ba1cf in boost::unit_test::ut_detail::invoker<boost::unit_test::ut_detail::unused>::invoke<void (*)()> (this=0x7fffffffdab7,
    12    f=@0x55555623e018: 0x5555555ba720 <Alert_tests::AlertApplies_invoker()>) at /usr/include/boost/test/utils/callback.hpp:56
    13[#11](/bitcoin-bitcoin/11/) 0x00005555555b9c4b in boost::unit_test::ut_detail::callback0_impl_t<boost::unit_test::ut_detail::unused, void (*)()>::invoke (this=0x55555623e010)
    14    at /usr/include/boost/test/utils/callback.hpp:89
    15[#12](/bitcoin-bitcoin/12/) 0x00007ffff70d1c01 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    16[#13](/bitcoin-bitcoin/13/) 0x00007ffff70b1c56 in boost::execution_monitor::catch_signals(boost::unit_test::callback0<int> const&) ()
    17   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    18[#14](/bitcoin-bitcoin/14/) 0x00007ffff70b2473 in boost::execution_monitor::execute(boost::unit_test::callback0<int> const&) ()
    19   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    20[#15](/bitcoin-bitcoin/15/) 0x00007ffff70d1d32 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::unit_test::test_case const&) ()
    21   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    22[#16](/bitcoin-bitcoin/16/) 0x00007ffff70b934e in boost::unit_test::framework_impl::visit(boost::unit_test::test_case const&) ()
    23   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    24[#17](/bitcoin-bitcoin/17/) 0x00007ffff70ef3ab in boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) ()
    25   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    26[#18](/bitcoin-bitcoin/18/) 0x00007ffff70ef3ab in boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) ()
    27   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    28[#19](/bitcoin-bitcoin/19/) 0x00007ffff70b4ca6 in boost::unit_test::framework::run(unsigned long, bool) () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    29[#20](/bitcoin-bitcoin/20/) 0x00007ffff70d01d7 in boost::unit_test::unit_test_main(bool (*)(), int, char**) () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
    30[#21](/bitcoin-bitcoin/21/) 0x00005555557542fd in main (argc=1, argv=0x7fffffffe928) at /usr/include/boost/test/unit_test.hpp:59
    

    Actually looking at it now, I have no clue where it’s linking against libdb 4.8 from. Since it should be 5.3.1. So this may just be a strange build environment issue that lets it compile, but dies horribly when actually run.

20. 
    OverlordQ commented at 5:36 pm on October 7, 2015: contributor

    Yuuuup, I’m an idiot.

    Purging all traces of BDB 4.8 from the system with copious amounts of fire and recompiling seems to have resolved the issue also.

    I think that was from trying to compile a wallet-portable version and it just somehow magically worked through BDB 5.1, but BDB 5.3 finally caused it to break.

21. OverlordQ closed this on Oct 7, 2015

    ---

22. 
    paveljanik commented at 5:39 pm on October 7, 2015: contributor

    Great - thank you!
23. 
    ehoffman2 commented at 8:20 pm on January 25, 2019: none

    Look like duplicate of #12047 (see my comment). I know this is closed, but just as a reference…

    Eric

24. DrahtBot locked this on Dec 16, 2021

Contributors
OverlordQ paveljanik sipa ehoffman2