b-msghand[4988] general protection fault #30706

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

Hello dear devs, I caught segfault today, running latest Bitcoin Code in full node mode on AMD64 Linux server.

Expected behaviour

no segfault

Steps to reproduce

I had not seen bitcoind crash for a long time. It's a rare one I guess.

Relevant log output

dmesg: traps: b-msghand[4988] general protection fault ip:55783548b790 sp:7f57627f3978 error:0 in bitcoind[557834c96000+aee000]

gdb backtrace: debug.txt

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

v27.0.0

Operating system and version

Debian GNU/Linux 12 (bookworm)

Machine specifications

Debian Stable server with AMD Ryzen 7 1700 and ECC RAM, everything stable and reliable. Bitcoin Core is using Tor and I2P network only, running in full mode. Blocks on RAID10 array, hardware is reliable.

Thank you for reporting this.

While issues in b-msghand can be problematic and it's good to report them, I don't think this report contains any actionable details. The gbd backtrace only contains memory addresses and no further debug information.

Unless you can share more information or you don't start to see more frequent segmentation faults, I don't think it's worth keeping this issue open.

Was there anything unusual in the debug log before the crash?

Thank you for reporting this.

While issues in b-msghand can be problematic and it's good to report them, I don't think this report contains any actionable details. The gbd backtrace only contains memory addresses and no further debug information.

Unless you can share more information or you don't start to see more frequent segmentation faults, I don't think it's worth keeping this issue open.

Thanks for your reply. I am aware backtrace only contains ??? instead of anything useful, that's probably because I don't have debugging symbols for bitcoind, or maybe I would need to have it compiled locally with special flags and so on. This is not the case, so I only have what I shared. Logs contains nothing else worth mentioning, machine continues to run as intended. Bitcoind is in automatic restart loop, I've implemented this years ago when I was having problems with it. Not any more, this rare crash is first time I've seen in years. If you have any suggestions how to enable more debugging in bitcoind, so that any future backtrace can be more useful, please let me know.

The guix debug symbols are split from the bitcoind you are using. So someone would have download them before demangling. I tried to do that, but for some reason I had no success. I am not a build system expert, so I am not sure where the problem lies.

The guix debug symbols are split from the bitcoind you are using. So someone would have download them before demangling. I tried to do that, but for some reason I had no success. I am not a build system expert, so I am not sure where the problem lies.

My knowledge goes only as far as adding deb http://deb.debian.org/debian-debug/ bookworm-debug main to the sources.list and then installing *-dbgsym packages. But that will not include bitcoind, because that's an external executable I downloaded. If someone has any instructions how to get debugging symbols for bitcoind binary on Debian, please let me know :)

Was there anything unusual in the debug log before the crash?

Logs contains nothing else worth mentioning, machine continues to run as intended.

Do you mean the machine logs contain nothing worth mentioning or the Bitcoin Core debug.log?

Just in case you are not aware: Bitcoin Core writes to a debug.log file in your data directory (where the blockchain is stored too). It would be useful to know what Bitcoin Core logged at around 2024-08-23 23:47:55 BST.

The debug symbols are separately produced by guix and the downloads are on https://bitcoincore.org/bin/bitcoin-core-27.1/. In this case it would be x86_64-linux-gnu-debug.

Was there anything unusual in the debug log before the crash?

Logs contains nothing else worth mentioning, machine continues to run as intended.

Do you mean the machine logs contain nothing worth mentioning or the Bitcoin Core debug.log?

Just in case you are not aware: Bitcoin Core writes to a debug.log file in your data directory (where the blockchain is stored too). It would be useful to know what Bitcoin Core logged at around 2024-08-23 23:47:55 BST.

Of course, sorry for not including it earlier. I am attaching first and last 100 lines of the log. In between, there is nothing useful, I think. At the end of the log, you will see nothing logged around time of the crash, unfortunately.

bitcoind.log

The debug symbols are separately produced by guix and the downloads are on https://bitcoincore.org/bin/bitcoin-core-27.1/. In this case it would be x86_64-linux-gnu-debug.

That's great, thank you! Now I have upgraded to bitcoin-core-27.1 and debug symbols are downloaded. Is there anything else I should do before next crash (if it ever happens), regarding debugging symbols? How to "tell" gdb that I have them? Thanks.

Is there anything else I should do before next crash (if it ever happens), regarding debugging symbols? How to "tell" gdb that I have them? Thanks.

The idea was that you don't have to tell gdb before a crash, but that you can (even after the crash) convert the addr-stack to something readable.

However, I couldn't get that to work, so someone else will have to explain how it is supposed to work.

My next recommendation would be to re-compile with guix, but skip the split-debug.sh step, and then convert the addrs using the "full" binary.

The main issue here is no core dump (in future please try and include it in the issue), and I don't think there's anything we can do with the gdb backtrace. I tested that using the 27.0 binary, and the associated debug symbols, works for inspecting a core dump. i.e:

# assuming you have bitcoind and bitcoind.dbg from
# https://bitcoincore.org/bin/bitcoin-core-27.0/bitcoin-27.0-x86_64-linux-gnu.tar.gz
# https://bitcoincore.org/bin/bitcoin-core-27.0/bitcoin-27.0-x86_64-linux-gnu-debug.tar.gz
# mv bitcoind.dbg bitcoind.dbg.txt (for now)

# I used trip to invoke a malloc failure
# Note that how/where the core dump is generated will depend on your system.

trip malloc -- ./bitcoind
> terminate called without an active exception
> Aborted (core dumped)

# get the dump
cp /var/lib/apport/coredump/core._root_segfault_bitcoind.0.d7bf5f78-1c2a-4e45-b21a-1445b80537f5.21806.6531920 dump

Give GDB bitcoind and the dump:

gdb bitcoind dump
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bitcoind...
(No debugging symbols found in bitcoind)
[New LWP 21806]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by './bitcoind'.
Program terminated with signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44

warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) bt
[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
[#1](/bitcoin-bitcoin/1/)  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
[#2](/bitcoin-bitcoin/2/)  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
[#3](/bitcoin-bitcoin/3/)  0x00007b39a084526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
[#4](/bitcoin-bitcoin/4/)  0x00007b39a08288ff in __GI_abort () at ./stdlib/abort.c:79
[#5](/bitcoin-bitcoin/5/)  0x00005f813f8ae4f7 in ?? ()
[#6](/bitcoin-bitcoin/6/)  0x00005f814021fc76 in ?? ()
[#7](/bitcoin-bitcoin/7/)  0x00005f814021fce1 in ?? ()
[#8](/bitcoin-bitcoin/8/)  0x00005f814021eb7b in ?? ()
[#9](/bitcoin-bitcoin/9/)  0x00005f813f8ae3fc in ?? ()
[#10](/bitcoin-bitcoin/10/) 0x00005f813f8b463d in ?? ()
[#11](/bitcoin-bitcoin/11/) 0x00005f81402d8ec5 in ?? ()
[#12](/bitcoin-bitcoin/12/) 0x00007b39a082a264 in __libc_start_main_impl (main=0x5f813f8b2bf0, argc=1, argv=0x7fff15128688, init=0x5f81402d8e80, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff15128678) at ../csu/libc-start.c:343
[#13](/bitcoin-bitcoin/13/) 0x00005f813f8cb4ba in ?? ()

No symbols. Return bitcoind.dbg.txt to bitcoind.dgb and retry. Note that GDB will load the debug symbols automatically, but they can also be passed explicitly:

mv bitcoind.dbg.txt bitcoind.dbg
# gdb bitcoind dump
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bitcoind...
Reading symbols from /root/segfault/bitcoind.dbg...
[New LWP 21806]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/local/bin/trip
Downloading separate debug info for system-supplied DSO at 0x7fff151c3000                                                                                                                                                                                         
[Thread debugging using libthread_db enabled]                                                                                                                                                                                                                     
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by './bitcoind'.
Program terminated with signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44

warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) bt
[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
[#1](/bitcoin-bitcoin/1/)  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
[#2](/bitcoin-bitcoin/2/)  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
[#3](/bitcoin-bitcoin/3/)  0x00007b39a084526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
[#4](/bitcoin-bitcoin/4/)  0x00007b39a08288ff in __GI_abort () at ./stdlib/abort.c:79
[#5](/bitcoin-bitcoin/5/)  0x00005f813f8ae4f7 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] ()
[#6](/bitcoin-bitcoin/6/)  0x00005f814021fc76 in __cxxabiv1::__terminate(void (*)()) ()
[#7](/bitcoin-bitcoin/7/)  0x00005f814021fce1 in std::terminate() ()
[#8](/bitcoin-bitcoin/8/)  0x00005f814021eb7b in __cxa_allocate_exception ()
[#9](/bitcoin-bitcoin/9/)  0x00005f813f8ae3fc in operator new(unsigned long) [clone .cold] ()
[#10](/bitcoin-bitcoin/10/) 0x00005f813f8b463d in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> >::allocate (this=<optimized out>, __n=1)
    at /usr/include/c++/ext/new_allocator.h:103
[#11](/bitcoin-bitcoin/11/) std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> >::allocate (__n=1, this=<optimized out>) at /usr/include/c++/bits/allocator.h:181
[#12](/bitcoin-bitcoin/12/) std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> > >::allocate (__n=1, __a=...)
    at /usr/include/c++/bits/alloc_traits.h:460
[#13](/bitcoin-bitcoin/13/) std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> > >::_M_allocate_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable_policy.h:2033
[#14](/bitcoin-bitcoin/14/) std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Scoped_node::_Scoped_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (__h=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>, this=<synthetic pointer>) at /usr/include/c++/bits/hashtable.h:286
[#15](/bitcoin-bitcoin/15/) std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable.h:1728
[#16](/bitcoin-bitcoin/16/) std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable.h:836
[#17](/bitcoin-bitcoin/17/) std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned char, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/unordered_map.h:389
[#18](/bitcoin-bitcoin/18/) (anonymous namespace)::V2MessageMap::V2MessageMap (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at net.cpp:928
[#19](/bitcoin-bitcoin/19/) __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at net.cpp:940
[#20](/bitcoin-bitcoin/20/) _GLOBAL__sub_I__Z22NET_MESSAGE_TYPE_OTHERB5cxx11 () at net.cpp:3899
[#21](/bitcoin-bitcoin/21/) 0x00005f81402d8ec5 in __libc_csu_init (argc=argc@entry=1, argv=argv@entry=0x7fff15128688, envp=0x7fff15128698) at elf-init.c:88
[#22](/bitcoin-bitcoin/22/) 0x00007b39a082a264 in __libc_start_main_impl (main=0x5f813f8b2bf0 <main(int, char**)>, argc=1, argv=0x7fff15128688, init=0x5f81402d8e80 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff15128678)
    at ../csu/libc-start.c:343
[#23](/bitcoin-bitcoin/23/) 0x00005f813f8cb4ba in _start () at ../sysdeps/x86_64/start.S:120
(gdb) 

Thanks for checking @fanquake . I was under the impression that even without a core dump one could call addr2line or the equivalent in gdb (assuming bitcoind and bitcoind.dbg are available), but maybe I may have been mistaken.

Closing for now. If you do see any issues in future, please open new issues, including a core dump if possible (or inspect the dump locally and post any relevant info to the issue).

Thanks for all the replies! @fanquake where I can get trip program? Couldn't find it anywhere in Debian Stable using apt-file search.

where I can get trip program?

https://codeberg.org/pkal/trip

Thanks for fantastic instructions! I was able to trip bitcoind, and do full backtrace with proper symbols. Thanks! I will now await another crash, using upgraded bitcoin core 27.1. It may not happen anytime soon :)