b-msghand[4988] general protection fault

github12101 commented at 11:43 am on August 24, 2024: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

Hello dear devs, I caught segfault today, running latest Bitcoin Code in full node mode on AMD64 Linux server.

Expected behaviour

no segfault

Steps to reproduce

I had not seen bitcoind crash for a long time. It’s a rare one I guess.

Relevant log output

dmesg: traps: b-msghand[4988] general protection fault ip:55783548b790 sp:7f57627f3978 error:0 in bitcoind[557834c96000+aee000]

gdb backtrace: debug.txt

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

v27.0.0

Operating system and version

Debian GNU/Linux 12 (bookworm)

Machine specifications

Debian Stable server with AMD Ryzen 7 1700 and ECC RAM, everything stable and reliable. Bitcoin Core is using Tor and I2P network only, running in full mode. Blocks on RAID10 array, hardware is reliable.

0xB10C commented at 11:12 am on August 27, 2024: contributor

Thank you for reporting this.

While issues in b-msghand can be problematic and it’s good to report them, I don’t think this report contains any actionable details. The gbd backtrace only contains memory addresses and no further debug information.

Unless you can share more information or you don’t start to see more frequent segmentation faults, I don’t think it’s worth keeping this issue open.

maflcko added the label Linux/Unix on Aug 27, 2024

mzumsande commented at 11:32 am on August 27, 2024: contributor

Was there anything unusual in the debug log before the crash?

github12101 commented at 2:22 pm on August 27, 2024: none

Thank you for reporting this.

While issues in b-msghand can be problematic and it’s good to report them, I don’t think this report contains any actionable details. The gbd backtrace only contains memory addresses and no further debug information.

Unless you can share more information or you don’t start to see more frequent segmentation faults, I don’t think it’s worth keeping this issue open.

Thanks for your reply. I am aware backtrace only contains ??? instead of anything useful, that’s probably because I don’t have debugging symbols for bitcoind, or maybe I would need to have it compiled locally with special flags and so on. This is not the case, so I only have what I shared. Logs contains nothing else worth mentioning, machine continues to run as intended. Bitcoind is in automatic restart loop, I’ve implemented this years ago when I was having problems with it. Not any more, this rare crash is first time I’ve seen in years. If you have any suggestions how to enable more debugging in bitcoind, so that any future backtrace can be more useful, please let me know.

maflcko commented at 2:34 pm on August 27, 2024: member

The guix debug symbols are split from the bitcoind you are using. So someone would have download them before demangling. I tried to do that, but for some reason I had no success. I am not a build system expert, so I am not sure where the problem lies.

github12101 commented at 2:54 pm on August 27, 2024: none

The guix debug symbols are split from the bitcoind you are using. So someone would have download them before demangling. I tried to do that, but for some reason I had no success. I am not a build system expert, so I am not sure where the problem lies.

My knowledge goes only as far as adding deb http://deb.debian.org/debian-debug/ bookworm-debug main to the sources.list and then installing *-dbgsym packages. But that will not include bitcoind, because that’s an external executable I downloaded. If someone has any instructions how to get debugging symbols for bitcoind binary on Debian, please let me know :)

0xB10C commented at 3:04 pm on August 27, 2024: contributor

Was there anything unusual in the debug log before the crash?

Logs contains nothing else worth mentioning, machine continues to run as intended.

Do you mean the machine logs contain nothing worth mentioning or the Bitcoin Core debug.log?

Just in case you are not aware: Bitcoin Core writes to a debug.log file in your data directory (where the blockchain is stored too). It would be useful to know what Bitcoin Core logged at around 2024-08-23 23:47:55 BST.

maflcko commented at 3:06 pm on August 27, 2024: member

The debug symbols are separately produced by guix and the downloads are on https://bitcoincore.org/bin/bitcoin-core-27.1/. In this case it would be x86_64-linux-gnu-debug.

github12101 commented at 7:26 pm on August 27, 2024: none

Was there anything unusual in the debug log before the crash?

Logs contains nothing else worth mentioning, machine continues to run as intended.

Do you mean the machine logs contain nothing worth mentioning or the Bitcoin Core debug.log?

Just in case you are not aware: Bitcoin Core writes to a debug.log file in your data directory (where the blockchain is stored too). It would be useful to know what Bitcoin Core logged at around 2024-08-23 23:47:55 BST.

Of course, sorry for not including it earlier. I am attaching first and last 100 lines of the log. In between, there is nothing useful, I think. At the end of the log, you will see nothing logged around time of the crash, unfortunately.

bitcoind.log

github12101 commented at 7:38 pm on August 27, 2024: none

The debug symbols are separately produced by guix and the downloads are on https://bitcoincore.org/bin/bitcoin-core-27.1/. In this case it would be x86_64-linux-gnu-debug.

That’s great, thank you! Now I have upgraded to bitcoin-core-27.1 and debug symbols are downloaded. Is there anything else I should do before next crash (if it ever happens), regarding debugging symbols? How to “tell” gdb that I have them? Thanks.

maflcko added the label Build system on Aug 28, 2024

maflcko removed the label Build system on Aug 28, 2024

maflcko commented at 7:35 am on August 28, 2024: member

Is there anything else I should do before next crash (if it ever happens), regarding debugging symbols? How to “tell” gdb that I have them? Thanks.

The idea was that you don’t have to tell gdb before a crash, but that you can (even after the crash) convert the addr-stack to something readable.

However, I couldn’t get that to work, so someone else will have to explain how it is supposed to work.

My next recommendation would be to re-compile with guix, but skip the split-debug.sh step, and then convert the addrs using the “full” binary.

fanquake commented at 8:29 am on August 28, 2024: member

The main issue here is no core dump (in future please try and include it in the issue), and I don’t think there’s anything we can do with the gdb backtrace. I tested that using the 27.0 binary, and the associated debug symbols, works for inspecting a core dump. i.e:

 0# assuming you have bitcoind and bitcoind.dbg from
 1# https://bitcoincore.org/bin/bitcoin-core-27.0/bitcoin-27.0-x86_64-linux-gnu.tar.gz
 2# https://bitcoincore.org/bin/bitcoin-core-27.0/bitcoin-27.0-x86_64-linux-gnu-debug.tar.gz
 3# mv bitcoind.dbg bitcoind.dbg.txt (for now)
 4
 5# I used trip to invoke a malloc failure
 6# Note that how/where the core dump is generated will depend on your system.
 7
 8trip malloc -- ./bitcoind
 9> terminate called without an active exception
10> Aborted (core dumped)
11
12# get the dump
13cp /var/lib/apport/coredump/core._root_segfault_bitcoind.0.d7bf5f78-1c2a-4e45-b21a-1445b80537f5.21806.6531920 dump

Give GDB bitcoind and the dump:

 0gdb bitcoind dump
 1GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
 2Copyright (C) 2024 Free Software Foundation, Inc.
 3License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 4This is free software: you are free to change and redistribute it.
 5There is NO WARRANTY, to the extent permitted by law.
 6Type "show copying" and "show warranty" for details.
 7This GDB was configured as "x86_64-linux-gnu".
 8Type "show configuration" for configuration details.
 9For bug reporting instructions, please see:
10<https://www.gnu.org/software/gdb/bugs/>.
11Find the GDB manual and other documentation resources online at:
12    <http://www.gnu.org/software/gdb/documentation/>.
13
14For help, type "help".
15Type "apropos word" to search for commands related to "word"...
16Reading symbols from bitcoind...
17(No debugging symbols found in bitcoind)
18[New LWP 21806]
19
20This GDB supports auto-downloading debuginfo from the following URLs:
21  <https://debuginfod.ubuntu.com>
22Enable debuginfod for this session? (y or [n]) y
23Debuginfod has been enabled.
24To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
25[Thread debugging using libthread_db enabled]
26Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
27Core was generated by './bitcoind'.
28Program terminated with signal SIGABRT, Aborted.
29Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
30[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
31
32warning: 44	./nptl/pthread_kill.c: No such file or directory
33(gdb) bt
34[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
35[#1](/bitcoin-bitcoin/1/)  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
36[#2](/bitcoin-bitcoin/2/)  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
37[#3](/bitcoin-bitcoin/3/)  0x00007b39a084526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
38[#4](/bitcoin-bitcoin/4/)  0x00007b39a08288ff in __GI_abort () at ./stdlib/abort.c:79
39[#5](/bitcoin-bitcoin/5/)  0x00005f813f8ae4f7 in ?? ()
40[#6](/bitcoin-bitcoin/6/)  0x00005f814021fc76 in ?? ()
41[#7](/bitcoin-bitcoin/7/)  0x00005f814021fce1 in ?? ()
42[#8](/bitcoin-bitcoin/8/)  0x00005f814021eb7b in ?? ()
43[#9](/bitcoin-bitcoin/9/)  0x00005f813f8ae3fc in ?? ()
44[#10](/bitcoin-bitcoin/10/) 0x00005f813f8b463d in ?? ()
45[#11](/bitcoin-bitcoin/11/) 0x00005f81402d8ec5 in ?? ()
46[#12](/bitcoin-bitcoin/12/) 0x00007b39a082a264 in __libc_start_main_impl (main=0x5f813f8b2bf0, argc=1, argv=0x7fff15128688, init=0x5f81402d8e80, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff15128678) at ../csu/libc-start.c:343
47[#13](/bitcoin-bitcoin/13/) 0x00005f813f8cb4ba in ?? ()

No symbols. Return bitcoind.dbg.txt to bitcoind.dgb and retry. Note that GDB will load the debug symbols automatically, but they can also be passed explicitly:

 0mv bitcoind.dbg.txt bitcoind.dbg
 1# gdb bitcoind dump
 2GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
 3Copyright (C) 2024 Free Software Foundation, Inc.
 4License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 5This is free software: you are free to change and redistribute it.
 6There is NO WARRANTY, to the extent permitted by law.
 7Type "show copying" and "show warranty" for details.
 8This GDB was configured as "x86_64-linux-gnu".
 9Type "show configuration" for configuration details.
10For bug reporting instructions, please see:
11<https://www.gnu.org/software/gdb/bugs/>.
12Find the GDB manual and other documentation resources online at:
13    <http://www.gnu.org/software/gdb/documentation/>.
14
15For help, type "help".
16Type "apropos word" to search for commands related to "word"...
17Reading symbols from bitcoind...
18Reading symbols from /root/segfault/bitcoind.dbg...
19[New LWP 21806]
20
21This GDB supports auto-downloading debuginfo from the following URLs:
22  <https://debuginfod.ubuntu.com>
23Enable debuginfod for this session? (y or [n]) y
24Debuginfod has been enabled.
25To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
26Downloading separate debug info for /usr/local/bin/trip
27Downloading separate debug info for system-supplied DSO at 0x7fff151c3000                                                                                                                                                                                         
28[Thread debugging using libthread_db enabled]                                                                                                                                                                                                                     
29Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
30Core was generated by './bitcoind'.
31Program terminated with signal SIGABRT, Aborted.
32Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
33[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
34
35warning: 44	./nptl/pthread_kill.c: No such file or directory
36(gdb) bt
37[#0](/bitcoin-bitcoin/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
38[#1](/bitcoin-bitcoin/1/)  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
39[#2](/bitcoin-bitcoin/2/)  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
40[#3](/bitcoin-bitcoin/3/)  0x00007b39a084526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
41[#4](/bitcoin-bitcoin/4/)  0x00007b39a08288ff in __GI_abort () at ./stdlib/abort.c:79
42[#5](/bitcoin-bitcoin/5/)  0x00005f813f8ae4f7 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] ()
43[#6](/bitcoin-bitcoin/6/)  0x00005f814021fc76 in __cxxabiv1::__terminate(void (*)()) ()
44[#7](/bitcoin-bitcoin/7/)  0x00005f814021fce1 in std::terminate() ()
45[#8](/bitcoin-bitcoin/8/)  0x00005f814021eb7b in __cxa_allocate_exception ()
46[#9](/bitcoin-bitcoin/9/)  0x00005f813f8ae3fc in operator new(unsigned long) [clone .cold] ()
47[#10](/bitcoin-bitcoin/10/) 0x00005f813f8b463d in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> >::allocate (this=<optimized out>, __n=1)
48    at /usr/include/c++/ext/new_allocator.h:103
49[#11](/bitcoin-bitcoin/11/) std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> >::allocate (__n=1, this=<optimized out>) at /usr/include/c++/bits/allocator.h:181
50[#12](/bitcoin-bitcoin/12/) std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> > >::allocate (__n=1, __a=...)
51    at /usr/include/c++/bits/alloc_traits.h:460
52[#13](/bitcoin-bitcoin/13/) std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, true> > >::_M_allocate_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable_policy.h:2033
53[#14](/bitcoin-bitcoin/14/) std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Scoped_node::_Scoped_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (__h=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>, this=<synthetic pointer>) at /usr/include/c++/bits/hashtable.h:286
54[#15](/bitcoin-bitcoin/15/) std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable.h:1728
55[#16](/bitcoin-bitcoin/16/) std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/hashtable.h:836
56[#17](/bitcoin-bitcoin/17/) std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned char, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned char> > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long&> (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at /usr/include/c++/bits/unordered_map.h:389
57[#18](/bitcoin-bitcoin/18/) (anonymous namespace)::V2MessageMap::V2MessageMap (this=0x5f81406bbe80 <(anonymous namespace)::V2_MESSAGE_MAP>) at net.cpp:928
58[#19](/bitcoin-bitcoin/19/) __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at net.cpp:940
59[#20](/bitcoin-bitcoin/20/) _GLOBAL__sub_I__Z22NET_MESSAGE_TYPE_OTHERB5cxx11 () at net.cpp:3899
60[#21](/bitcoin-bitcoin/21/) 0x00005f81402d8ec5 in __libc_csu_init (argc=argc@entry=1, argv=argv@entry=0x7fff15128688, envp=0x7fff15128698) at elf-init.c:88
61[#22](/bitcoin-bitcoin/22/) 0x00007b39a082a264 in __libc_start_main_impl (main=0x5f813f8b2bf0 <main(int, char**)>, argc=1, argv=0x7fff15128688, init=0x5f81402d8e80 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff15128678)
62    at ../csu/libc-start.c:343
63[#23](/bitcoin-bitcoin/23/) 0x00005f813f8cb4ba in _start () at ../sysdeps/x86_64/start.S:120
64(gdb)

maflcko commented at 8:50 am on August 28, 2024: member

Thanks for checking @fanquake . I was under the impression that even without a core dump one could call addr2line or the equivalent in gdb (assuming bitcoind and bitcoind.dbg are available), but maybe I may have been mistaken.

fanquake commented at 3:37 pm on August 28, 2024: member

Closing for now. If you do see any issues in future, please open new issues, including a core dump if possible (or inspect the dump locally and post any relevant info to the issue).

fanquake closed this on Aug 28, 2024

github12101 commented at 3:49 pm on August 28, 2024: none

Thanks for all the replies! @fanquake where I can get trip program? Couldn’t find it anywhere in Debian Stable using apt-file search.

fanquake commented at 3:50 pm on August 28, 2024: member

where I can get trip program?

https://codeberg.org/pkal/trip

github12101 commented at 5:51 pm on August 28, 2024: none

Thanks for fantastic instructions! I was able to trip bitcoind, and do full backtrace with proper symbols. Thanks! I will now await another crash, using upgraded bitcoin core 27.1. It may not happen anytime soon :)

b-msghand[4988] general protection fault #30706