Bitcoind v0.11.0 segfaults on startup trying to read wallet. #6775

issue OverlordQ opened this issue on October 7, 2015

OverlordQ commented at 4:33 PM on October 7, 2015: contributor

GDB output:

(gdb) r -debug
Starting program: /usr/local/bin/bitcoind -debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff5238700 (LWP 17767)]
[New Thread 0x7ffff4a37700 (LWP 17768)]
[New Thread 0x7fffeffff700 (LWP 17769)]
[New Thread 0x7fffef7fe700 (LWP 17770)]
[New Thread 0x7fffeeffd700 (LWP 17771)]
[New Thread 0x7fffee7fc700 (LWP 17772)]
[New Thread 0x7fffedffb700 (LWP 17773)]
[New Thread 0x7fffed7fa700 (LWP 17774)]
[New Thread 0x7fffecff9700 (LWP 17775)]
[New Thread 0x7fffcffff700 (LWP 17776)]
[New Thread 0x7fffcf7fe700 (LWP 17777)]
[New Thread 0x7fffceffd700 (LWP 17778)]

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106     ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) bt
[#0](/bitcoin-bitcoin/0/)  strlen () at ../sysdeps/x86_64/strlen.S:106
[#1](/bitcoin-bitcoin/1/)  0x00007ffff70bace2 in __os_strdup () from /usr/lib/libdb_cxx-4.8.so
[#2](/bitcoin-bitcoin/2/)  0x00007ffff708d6b4 in __env_add_data_dir () from /usr/lib/libdb_cxx-4.8.so
[#3](/bitcoin-bitcoin/3/)  0x00007ffff708d781 in __env_set_data_dir () from /usr/lib/libdb_cxx-4.8.so
[#4](/bitcoin-bitcoin/4/)  0x00007ffff6fac5d5 in DbEnv::set_data_dir(char const*) () from /usr/lib/libdb_cxx-4.8.so
[#5](/bitcoin-bitcoin/5/)  0x00005555557cf191 in CDBEnv::Open (this=0x555555c004a0 <bitdb>, pathIn=...) at wallet/db.cpp:92
[#6](/bitcoin-bitcoin/6/)  0x0000555555816a01 in CWallet::Verify (walletFile="wallet.dat", warningString="", errorString="") at wallet/wallet.cpp:353
[#7](/bitcoin-bitcoin/7/)  0x00005555555ab574 in AppInit2 (threadGroup=..., scheduler=...) at init.cpp:957
[#8](/bitcoin-bitcoin/8/)  0x000055555558ffe9 in AppInit (argc=<optimized out>, argv=<optimized out>) at bitcoind.cpp:147
[#9](/bitcoin-bitcoin/9/)  0x000055555558736f in main (argc=2, argv=0x7fffffffe6e8) at bitcoind.cpp:176

Logfile Output:

2015-10-07 16:30:21 Bitcoin version v0.11.0 (2015-07-10 19:23:55 +0200)
2015-10-07 16:30:21 Using OpenSSL version OpenSSL 1.0.2d 9 Jul 2015
2015-10-07 16:30:21 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-10-07 16:30:21 Default data directory /home/bitnode/.bitcoin
2015-10-07 16:30:21 Using data directory /home/bitnode/.bitcoin
2015-10-07 16:30:21 Using config file /home/bitnode/.bitcoin/bitcoin.conf
2015-10-07 16:30:21 Using at most 700 connections (65536 file descriptors available)
2015-10-07 16:30:21 Using 8 threads for script verification
2015-10-07 16:30:21 scheduler thread start
2015-10-07 16:30:21 Allowing RPC connections from: 127.0.0.0/255.0.0.0 ::1/ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
2015-10-07 16:30:21 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0)
2015-10-07 16:30:21 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0)
2015-10-07 16:30:21 Using wallet wallet.dat
2015-10-07 16:30:21 init message: Verifying wallet...
2015-10-07 16:30:21 CDBEnv::Open: LogDir=/home/bitnode/.bitcoin/database ErrorFile=/home/bitnode/.bitcoin/db.log

paveljanik commented at 4:37 PM on October 7, 2015: contributor

/home/bitnode/.bitcoin/db.log is empty? It starts OK without wallet.dat?
OverlordQ commented at 4:38 PM on October 7, 2015: contributor

No db.log created, also, no wallet.dat in the folder.
sipa commented at 4:39 PM on October 7, 2015: member

What are you setting datadir or wallet to in bitcoin.conf?
paveljanik commented at 4:41 PM on October 7, 2015: contributor

Can you show us/remove bitcoin.conf please? Is your datadir writable etc.?
OverlordQ commented at 4:42 PM on October 7, 2015: contributor
Not being set in bitcoin.conf. Contents of bitcoin.conf:
```
rpcuser=yanot
rpcpassword=really
listen=1
txindex=1
server=1
```
paveljanik commented at 4:42 PM on October 7, 2015: contributor

Can you remove the conf file just for test?
OverlordQ commented at 4:44 PM on October 7, 2015: contributor

Same error moving bitcoin.conf out of the way.
paveljanik commented at 4:46 PM on October 7, 2015: contributor

can you please run ls -laR ~/.bitcoin?
OverlordQ commented at 4:49 PM on October 7, 2015: contributor

http://sprunge.us/CPhZ
paveljanik commented at 4:51 PM on October 7, 2015: contributor

Looks perfectly OK. So bitcoind is self-compiled? What OS? df -h OK?
paveljanik commented at 4:53 PM on October 7, 2015: contributor

Can you run bitcoind under strace -fff?
OverlordQ commented at 4:56 PM on October 7, 2015: contributor

Debian Stretch (yes I know its 'testing'), plenty of space free.

Strace: http://sprunge.us/hOjT

Moving blocks and chainstate folders out of the way had no effect either.
sipa commented at 5:00 PM on October 7, 2015: member

Are you able to create a ~/.bitcoin/database directory yourself? If not, what error do you get?

OverlordQ commented at 5:01 PM on October 7, 2015: contributor

Works fine from the shell.

bitnode@yig:~/.bitcoin$ ls
banlist.dat  bitcoin.conf.bak  bitcoind.pid  blocks  chainstate  database  debug.log  fee_estimates.dat  log  peers.dat
bitnode@yig:~/.bitcoin$ rm -rf database/
bitnode@yig:~/.bitcoin$ ls
banlist.dat  bitcoin.conf.bak  bitcoind.pid  blocks  chainstate  debug.log  fee_estimates.dat  log  peers.dat
bitnode@yig:~/.bitcoin$ mkdir database
bitnode@yig:~/.bitcoin$ ls
banlist.dat  bitcoin.conf.bak  bitcoind.pid  blocks  chainstate  database  debug.log  fee_estimates.dat  log  peers.dat

paveljanik commented at 5:04 PM on October 7, 2015: contributor

strace output with bitcoin.conf please :-)
OverlordQ commented at 5:07 PM on October 7, 2015: contributor

http://sprunge.us/cBhE
paveljanik commented at 5:17 PM on October 7, 2015: contributor
This is pretty strange 8) Can you try released version just to compare? There are plenty of
```
[pid  1512] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
```
Debian Stretch bugs...

OverlordQ commented at 5:29 PM on October 7, 2015: contributor

Binary release of v0.11.0 appears to run w/o issue.

FWIW:

$ ./test_bitcoin
Running 155 test cases...
unknown location(0): fatal error in "AlertApplies": memory access violation at address: 0x00000009: no mapping at fault address
test/arith_uint256_tests.cpp(564): last checkpoint

*** 1 failure detected in test suite "Bitcoin Test Suite"

and relavent BT:

(gdb) bt
[#0](/bitcoin-bitcoin/0/)  strlen () at ../sysdeps/x86_64/strlen.S:106
[#1](/bitcoin-bitcoin/1/)  0x00007ffff6e0cce2 in __os_strdup () from /usr/lib/libdb_cxx-4.8.so
[#2](/bitcoin-bitcoin/2/)  0x00007ffff6ddf6b4 in __env_add_data_dir () from /usr/lib/libdb_cxx-4.8.so
[#3](/bitcoin-bitcoin/3/)  0x00007ffff6ddf781 in __env_set_data_dir () from /usr/lib/libdb_cxx-4.8.so
[#4](/bitcoin-bitcoin/4/)  0x00007ffff6cfe5d5 in DbEnv::set_data_dir(char const*) () from /usr/lib/libdb_cxx-4.8.so
[#5](/bitcoin-bitcoin/5/)  0x0000555555b6b8c2 in CDBEnv::MakeMock (this=0x555556229c40 <bitdb>) at wallet/db.cpp:128
[#6](/bitcoin-bitcoin/6/)  0x0000555555754405 in TestingSetup::TestingSetup (this=0x7fffffffd8a0) at test/test_bitcoin.cpp:46
[#7](/bitcoin-bitcoin/7/)  0x00005555555bff31 in ReadAlerts::ReadAlerts (this=0x7fffffffd8a0) at test/alert_tests.cpp:89
[#8](/bitcoin-bitcoin/8/)  0x00005555555c02f9 in Alert_tests::AlertApplies::AlertApplies (this=0x7fffffffd8a0) at test/alert_tests.cpp:122
[#9](/bitcoin-bitcoin/9/)  0x00005555555ba74a in Alert_tests::AlertApplies_invoker () at test/alert_tests.cpp:122
[#10](/bitcoin-bitcoin/10/) 0x00005555555ba1cf in boost::unit_test::ut_detail::invoker<boost::unit_test::ut_detail::unused>::invoke<void (*)()> (this=0x7fffffffdab7,
    f=@0x55555623e018: 0x5555555ba720 <Alert_tests::AlertApplies_invoker()>) at /usr/include/boost/test/utils/callback.hpp:56
[#11](/bitcoin-bitcoin/11/) 0x00005555555b9c4b in boost::unit_test::ut_detail::callback0_impl_t<boost::unit_test::ut_detail::unused, void (*)()>::invoke (this=0x55555623e010)
    at /usr/include/boost/test/utils/callback.hpp:89
[#12](/bitcoin-bitcoin/12/) 0x00007ffff70d1c01 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#13](/bitcoin-bitcoin/13/) 0x00007ffff70b1c56 in boost::execution_monitor::catch_signals(boost::unit_test::callback0<int> const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#14](/bitcoin-bitcoin/14/) 0x00007ffff70b2473 in boost::execution_monitor::execute(boost::unit_test::callback0<int> const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#15](/bitcoin-bitcoin/15/) 0x00007ffff70d1d32 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::unit_test::test_case const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#16](/bitcoin-bitcoin/16/) 0x00007ffff70b934e in boost::unit_test::framework_impl::visit(boost::unit_test::test_case const&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#17](/bitcoin-bitcoin/17/) 0x00007ffff70ef3ab in boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#18](/bitcoin-bitcoin/18/) 0x00007ffff70ef3ab in boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) ()
   from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#19](/bitcoin-bitcoin/19/) 0x00007ffff70b4ca6 in boost::unit_test::framework::run(unsigned long, bool) () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#20](/bitcoin-bitcoin/20/) 0x00007ffff70d01d7 in boost::unit_test::unit_test_main(bool (*)(), int, char**) () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.58.0
[#21](/bitcoin-bitcoin/21/) 0x00005555557542fd in main (argc=1, argv=0x7fffffffe928) at /usr/include/boost/test/unit_test.hpp:59

Actually looking at it now, I have no clue where it's linking against libdb 4.8 from. Since it should be 5.3.1. So this may just be a strange build environment issue that lets it compile, but dies horribly when actually run.

OverlordQ commented at 5:36 PM on October 7, 2015: contributor

Yuuuup, I'm an idiot.

Purging all traces of BDB 4.8 from the system with copious amounts of fire and recompiling seems to have resolved the issue also.

I think that was from trying to compile a wallet-portable version and it just somehow magically worked through BDB 5.1, but BDB 5.3 finally caused it to break.
OverlordQ closed this on Oct 7, 2015
paveljanik commented at 5:39 PM on October 7, 2015: contributor

Great - thank you!
ehoffman2 commented at 8:20 PM on January 25, 2019: none

Look like duplicate of #12047 (see my comment). I know this is closed, but just as a reference...

Eric
DrahtBot locked this on Dec 16, 2021

Contributors

Linked (view graph)

#1 JSON-RPC support for mobile devices ("ultra-lightweight" clients)#2 Long-term, safe, store-of-value #3 Encrypt wallet #4 Export/Import wallet in a human readable, future-proof format #5 Make the version number the protocol version and not the client version #6 Treat wallet as a generic keystore #7 Block-header-only, faster startup client #8 RPC command to sign text with wallet private key #9 Fix for GUI on Macs and latest wxWidgets