Segmentation fault if configured with incompatible BDB

CryptAxe commented at 5:37 am on December 29, 2017: contributor

Hello, after updating my nodes with Master they will no longer start (segfault). Duplicating this issue requires building –with-incompatible-bdb so I don’t think it is super important. Regardless, Here is the segfault:

 0Segmentation fault (core dumped)
 1user@computer:~/bitcoin$ valgrind ./src/qt/bitcoin-qt 
 2...
 3==32418== 
 4==32418== 
 5==32418== Process terminating with default action of signal 11 (SIGSEGV)
 6==32418==  Access not within mapped region at address 0x0
 7==32418==    at 0x4C30D22: strlen (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 8==32418==    by 0x76C3331: __os_strdup (in /usr/lib/libdb_cxx-4.8.so)
 9==32418==    by 0x76959C3: __env_add_data_dir (in /usr/lib/libdb_cxx-4.8.so)
10==32418==    by 0x7695A90: __env_set_data_dir (in /usr/lib/libdb_cxx-4.8.so)
11==32418==    by 0x75B0B58: DbEnv::set_data_dir(char const*) (in /usr/lib/libdb_cxx-4.8.so)
12==32418==    by 0x51C1B7: CDBEnv::Open(boost::filesystem::path const&) (db.cpp:116)
13==32418==    by 0x51ED80: CDB::VerifyEnvironment(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::filesystem::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (db.cpp:272)
14==32418==    by 0x49FBB0: VerifyWallets() (init.cpp:230)
15==32418==    by 0x2BFA95: AppInitMain(boost::thread_group&, CScheduler&) (init.cpp:1266)
16==32418==    by 0x187096: BitcoinCore::initialize() (bitcoin.cpp:303)
17==32418==    by 0x6AD6698: QObject::event(QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.6.1)
18==32418==    by 0x5CE08AB: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.6.1)
19==32418==  If you believe this happened as a result of a stack
20==32418==  overflow in your program's main thread (unlikely but
21==32418==  possible), you can try to increase the size of the
22==32418==  main thread stack using the --main-stacksize= flag.
23==32418==  The main thread stack size used in this run was 8388608.
24==32418== 
25==32418== HEAP SUMMARY:
26==32418==     in use at exit: 47,372,704 bytes in 44,499 blocks
27==32418==   total heap usage: 137,047 allocs, 92,548 frees, 77,215,464 bytes allocated
28==32418== 
29==32418== LEAK SUMMARY:
30==32418==    definitely lost: 2,893 bytes in 6 blocks
31==32418==    indirectly lost: 14,328 bytes in 610 blocks
32==32418==      possibly lost: 16,868 bytes in 212 blocks
33==32418==    still reachable: 47,050,655 bytes in 42,427 blocks
34==32418==                       of which reachable via heuristic:
35==32418==                         length64           : 6,848 bytes in 77 blocks
36==32418==                         newarray           : 3,552 bytes in 40 blocks
37==32418==                         multipleinheritance: 1,864 bytes in 4 blocks
38==32418==         suppressed: 0 bytes in 0 blocks
39==32418== Rerun with --leak-check=full to see details of leaked memory
40==32418== 
41==32418== For counts of detected and suppressed errors, rerun with: -v
42==32418== Use --track-origins=yes to see where uninitialised values come from
43==32418== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)
44Killed

The segfault occurs during startup, the splash screen pops up for a second (if launching the GUI). To replicate, simply build from master (configured with incompatible bdb) and try to launch bitcoin-qt.

CryptAxe renamed this:
~~Segmentation fault if configured with incompatbile BDB~~
Segmentation fault if configured with incompatible BDB
on Dec 29, 2017

MarcoFalke added this to the milestone 0.16.0 on Dec 29, 2017

CryptAxe commented at 2:37 am on December 30, 2017: contributor

I tested on an updated Ubuntu (17.10) install and it doesn’t segfault at startup anymore. Looks like this segfault effects Ubuntu 16.10 and older using the libdb4.8 4.8.30-yakkety3 package. Not sure what other systems might be effected.

TheBlueMatt commented at 2:42 am on December 30, 2017: member

Hmm, error is in datadir parsing. Can you provide the path to -datadir and the full path implied by it?

On December 30, 2017 3:37:32 AM GMT+01:00, CryptAxe notifications@github.com wrote:

I tested on an updated Ubuntu (17.10) install and it doesn’t segfault at startup anymore. Looks like this segfault effects Ubuntu 16.10 and older using the libdb4.8 4.8.30-yakkety3 package. Not sure what other systems might be effected.

– You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/12047#issuecomment-354522955

fanquake added the label Linux/Unix on Dec 30, 2017

fanquake commented at 2:49 am on December 30, 2017: member

The segfault here looks very similar to the one in #6775.

Someone also posted this on the bitcoin stack-exchange a few weeks ago: https://bitcoin.stackexchange.com/questions/64541/building-bitcoin-segmentation-fault-on-building-wallet-ubuntu

CryptAxe commented at 2:53 am on December 30, 2017: contributor

@TheBlueMatt default data directory ~/.bitcoin

I also just tried ./src/qt/bitcoin-qt -datadir=/home/cryptaxe/newDataDir

and got

0Segmentation fault (core dumped)

Heres debug.log from that directory (it did manage to write some log messages before the segfault) if anyone can make sense of it:

 02017-12-30 02:40:01 Bitcoin version v0.15.99.0-5180a86
 12017-12-30 02:40:01 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
 22017-12-30 02:40:01 Assuming ancestors of block 0000000000000000003b9ce759c2a087d52abc4266f8f4ebd6d768b89defa50a have valid signatures.
 32017-12-30 02:40:01 Setting nMinimumChainWork=000000000000000000000000000000000000000000723d3581fe1bd55373540a
 42017-12-30 02:40:01 Using the 'sse4' SHA256 implementation
 52017-12-30 02:40:01 Default data directory /home/cryptaxe/.bitcoin
 62017-12-30 02:40:01 Using data directory /home/cryptaxe/newDataDir
 72017-12-30 02:40:01 Using config file /home/cryptaxe/newDataDir/bitcoin.conf
 82017-12-30 02:40:01 Using at most 125 automatic connections (1024 file descriptors available)
 92017-12-30 02:40:01 Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
102017-12-30 02:40:01 Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
112017-12-30 02:40:01 Using 4 threads for script verification
122017-12-30 02:40:01 scheduler thread start
132017-12-30 02:40:01 Using wallet directory /home/cryptaxe/newDataDir
142017-12-30 02:40:01 init message: Verifying wallet(s)...
152017-12-30 02:40:01 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
162017-12-30 02:40:01 Using wallet wallet.dat
172017-12-30 02:40:01 CDBEnv::Open: LogDir=/home/cryptaxe/newDataDir/database ErrorFile=/home/cryptaxe/newDataDir/db.log

CryptAxe commented at 2:54 am on December 30, 2017: contributor

@fanquake as with those you have linked to, I have no db.log created either. If that helps anyone.

laanwj commented at 10:56 am on December 30, 2017: member

It’s possible that you’re linking against a different version of BDB than the header files compiled against. This reeks like an ABI conflict.

jonasschnelli commented at 7:28 am on January 4, 2018: contributor

Can you pastebin your config.log and maybe look for possible multiple BerkleyDB-dev installation interference?

ghost commented at 4:20 pm on January 10, 2018: none

Had the same issue on 16.10. I fixed the issue by installing libdb4.8-dev, by default even on 16.10 you get libdb5.3-dev and when you configure with –with-incompatible-bdb you get a segmentation fault.

On 17.10 which you say you’re running you don’t have 4.8 https://packages.ubuntu.com/artful/libdb++-dev

You might want to try bitcoin/contrib/install_db4.sh

laanwj commented at 6:44 pm on January 11, 2018: member

This reeks like an ABI conflict.

Looking at the traceback I’m sure this is the case. The gdb error is in libdb_cxx-4.8.so while you had to provide --with-incompatible-bdb which means that it was detecting 5.x headers. Removing this from the 0.16.0 milestone as I don’t think this should block the release, this has been an issue forever.

If bdb provides a way to get the version number at runtime it might be possible to compare with the version at build time to give a sane error in this case instead of a crash.

laanwj removed this from the milestone 0.16.0 on Jan 11, 2018

laanwj added the label Build system on Jan 11, 2018

CryptAxe commented at 8:53 pm on January 11, 2018: contributor

@kekimusmaximus thanks for the advice. I had compatible BDB installed via the PPA (https://launchpad.net/~bitcoin/+archive/ubuntu/bitcoin) for a while but that uninstalled itself during an update which is when I started having this problem. Looks like this issue has been known for a while?

The gdb error is in libdb_cxx-4.8.so while you had to provide –with-incompatible-bdb which means that it was detecting 5.x headers. @jonasschnelli @laanwj Yes it is pretty confusing to me. 4.8 was removed from the system (by apt), 5.x was installed by apt, everything broke. I have since updated that installation, but I tried to recreate the issue in a virtual machine so that I could provide you with the config.log but I’m not getting the error now.

Maybe the issue only comes up if you upgrade from compatible (4.8) to incompatible (5.x) BDB and something weird gets messed up by autotools when you use the –with-incompatible-bdb option (leftover files it sees or something???)?

I need to test on the virtual machine whether doing the upgrade is what caused the issue, I’ll close this ASAP if I can’t replicate it that way.

CryptAxe closed this on Jan 25, 2018

ehoffman2 commented at 7:48 pm on January 25, 2019: none

I found the reason of this issue, and a solution (a good workaround).

The issue occurs (in my case) with Ubuntu 18.04, when I install both libdb-4.8++ and libdb-5.3++. I need those 2, because I do compile programs that require both versions (legacy and new programs).

I installed both versions from Ubuntu standard PPA, and you end up with both versions installed (libdb_cxx-4.8.so and libdb-5.3_cxx.so).

Also, and this is an important point, is that you also do get only one header set installed, which are db.h and db_cxx.h for version 5.3. So, when ‘configure’ goes through the detection phase, it does detect that the DB_VERSION_MAJOR/DB_VERSION_MINOR is 5.3 (this is from db.h for version 5.3).

So, you have no choice but to use libdb_cxx-5.3. But here, just specifying –with-incompatible-bdb is not enough when you have both version installed, you MUST ALSO specify BDB_LIBS="-ldb_cxx-5.3" on the configure line. That is, you must specify

./configure --with-incompatible-bdb BDB_LIBS="-ldb_cxx-5.3"

Without that, i.e. when you just specify the incompatible option, and not the BDB_LIBS option, then the ‘bitcoin-find-bdb48.m4’ autoconf file will just search for ’libdb_cxx-4.8.so’, and find that this file exist. So, it will create Makefiles linking with ‘-ldb_cxx4.8’, but uses include file db_cxx.h which does include the db.h for version 5.3. So, you end up with 3 things, that is: 5.3 header, 4.8 lib, and a segfault.

Again, this issue is there because I have both 4.8 and 5.3 installed, and the ‘bitcoin-find-bdb48.m4’ is not able to cope with this case, and use the version from the header file (i.e. detect that 5.3 header is present, and so, use -ldb_cxx-5.3).

By specifying

./configure --with-incompatible-bdb BDB_LIBS="-ldb_cxx-5.3"

you end up acknowledging that you use a ’legacy wallet incompatible’ BDB (ignore the fact that v5.3 is detected), and ACTUALLY link against 5.3

Eric

DrahtBot locked this on Dec 16, 2021

Segmentation fault if configured with incompatible BDB #12047