bitcoind immediately segfaults on ppc64 openbsd 7.4 #29517

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

v26.0 and v26.1rc1 both build successfully on big endian ppc64 openbsd 7.4 but when starting, bitcoind immediately segfaults. CPU is POWER9.

Unfortunately, gdb on this platform also crashes when attempting to debug so all I have is this:

$ egdb bitcoind bitcoind.core                                                            
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-openbsd7.4".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bitcoind...
[New process 613405]
Core was generated by `bitcoind'.
Program terminated with signal SIGSEGV, Segmentation fault.
[#0](/bitcoin-bitcoin/0/)  libunwind::LocalAddressSpace::get32 (this=<optimized out>, addr=<optimized out>) at /usr/src/gnu/lib/libcxxabi/../../../gnu/llvm/libunwind/src/AddressSpace.hpp:221
221     /usr/src/gnu/lib/libcxxabi/../../../gnu/llvm/libunwind/src/AddressSpace.hpp: No such file or directory.
(gdb) bt       
[#0](/bitcoin-bitcoin/0/)  libunwind::LocalAddressSpace::get32 (this=<optimized out>, addr=<optimized out>) at /usr/src/gnu/lib/libcxxabi/../../../gnu/llvm/libunwind/src/AddressSpace.hpp:221
Abort trap (core dumped) 

Expected behaviour

Bitcoind starts to sync.

Steps to reproduce

Build bitcoind on powerpc64 OpenBSD 7.4 on IBM POWER9. Start bitcoind.

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v26.0 and v26.1rc1

Operating system and version

OpenBSD 7.4

Compiled from source

What are the exact steps to reproduce? Does it also happen with another version of g++ or clang++? Does it happen also in valgrind?

Follow the build instructions in build-openbsd.md and then start bitcoind. I haven't tried other versions of clang or gcc. Just the clang in OBSD base. Also have not tried valgrind yet.

When building with gcc 11.2.0p9:

 CXX      bench/bench_bitcoin-data.o
  CXX      test/fuzz/libtest_fuzz_a-util.o
test/fuzz/util.cpp: In member function 'FILE* FuzzedFileProvider::open()':
test/fuzz/util.cpp:259:11: error: 'cookie_io_functions_t' does not name a type
  259 |     const cookie_io_functions_t io_hooks = {
      |           ^~~~~~~~~~~~~~~~~~~~~
test/fuzz/util.cpp:265:44: error: 'io_hooks' was not declared in this scope
  265 |     return fopencookie(this, mode.c_str(), io_hooks);
      |                                            ^~~~~~~~
test/fuzz/util.cpp:265:12: error: 'fopencookie' was not declared in this scope
  265 |     return fopencookie(this, mode.c_str(), io_hooks);
      |            ^~~~~~~~~~~
gmake[2]: *** [Makefile:12286: test/fuzz/libtest_fuzz_a-util.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[2]: Leaving directory '/home/user/builds/bitcoin/src'
gmake[1]: *** [Makefile:20136: all-recursive] Error 1
gmake[1]: Leaving directory '/home/user/builds/bitcoin/src'
gmake: *** [Makefile:812: all-recursive] Error 1

When building with gcc 11.2.0p9:

#29529 should resolve this issue.

#29529 should resolve this issue.

Thanks, that has allowed bitcoind to be built with gcc. Unfortunately bitcoind crashes immediately still.

If I load the bitcoind.core into gdb this is what I get for a backtrace:

Reading symbols from ./src/bitcoind...
[New process 594075]
Core was generated by `bitcoind'.
Program terminated with signal SIGABRT, Aborted.
[#0](/bitcoin-bitcoin/0/)  thrkill () at /tmp/-:3
3       /tmp/-: No such file or directory.
(gdb) bt
[#0](/bitcoin-bitcoin/0/)  thrkill () at /tmp/-:3
[#1](/bitcoin-bitcoin/1/)  0x0000000000000000 in ?? ()
(gdb) 

Unfortunately, gdb on this platform also crashes when attempting to debug

Maybe you could try lldb? Hard to debug this further without any more info. Could also configure with --enable-debug.

lldb also segfaults when loading the bitcoind.core dump. :-/

Does the bitcoind installable from ports run? It is version 26.0, and claims to run on powerpc64. If so, given I can't see any major patches in https://github.com/openbsd/ports/tree/master/net/bitcoin/patches, you could compare how that is being compiled, to your compilation of 26.0, as another avenue for debugging? Even though GCC compilation will be fixed shortly, my suggestion is to continue using the system compiler (Clang) and --enable-debug.

I've been unable to replicate this issue so far.

I haven't been able to get ppc64 + OpenBSD host running (ISO wouldn't boot) but I did manage to get a big endian ppc64 + FreeBSD running and was able to successfully build and run bitcoind.

I was able to get OpenBSD 7.4 running on x86 and was also able to build and run bitcoind using the default clang compiler and their default libc library.

That's good to hear. Keep trying to get OpenBSD booted. I had a little bit of trouble myself when using SATA SSD drives post-install. For some reason only a HDD would actually boot.

I'll give the ports bitcoind a go, @fanquake.

Any success with the ports one?

Closing for now due to inactivity. Please leave a comment if there are updates or more more information.

Update: I got access to a POWER9 machine again and tried the ports v27.1. It builds but still segfaults immediately.

Can you try if this happens with 29.0rc2 as well? https://github.com/bitcoin/bitcoin/releases/tag/v29.0rc2

I am asking, because it is using a new build system (cmake), so things may have changed.

Can you try with both g++ and clang++, again? Also, do all binaries in the build_dir/bin crash, or just bitcoind? Also, can you compile and run a simple stand-alone std::cout << "Hi" program?

Same issue with v29.0rc2 using both clang and gcc.

bitcoind, bitcoin-cli, bitcoin-util, and bitcoin-wallet crash. bitcoin-tx does not seem to crash but I haven't tested it much.

test_bitcoin prints a ton of these errors: terminate called recursively

The "Hi" program works though.

bsdppc$ ./a.out                                                                                            
Hi

FWIW I tried building on PPC64 Linux and bitcoind does not immediately crash. So it looks like something specific to OpenBSD powerpc64.

Tests pass on Linux too.

$ ctest --test-dir build

100% tests passed, 0 tests failed out of 137

bitcoind, bitcoin-cli, bitcoin-util, and bitcoin-wallet crash. bitcoin-tx does not seem to crash but I haven't tested it much.

If you want to debug this further and gdb doesn't help you with finding the place where the crash occurs you can try to place std::cout << __FILE__ << ":" << __LINE__ << std::endl; in the source code and then "bisect" from there until the exact line in the source code is known.