bitcoin-qt crashed after completing a full blockchain rescan #26339

issue dooglus opened this issue on October 19, 2022

dooglus commented at 3:55 PM on October 19, 2022: contributor
While investigating #26317 I tried running:
```
$ bitcoin-cli rescanblockchain
```
to rescan the full blockchain. It took about 9 hours to run the full rescan, and after the scan was complete bitcoin-qt crashed with this error in gdb:
```
Thread 35 "b-http" received signal SIGPIPE, Broken pipe.
```
The end of the debug.log file showed this at the time of the crash:
```
2022-10-19T09:15:07Z [chris] Still rescanning. At block 758431. Progress=0.997985
2022-10-19T09:16:07Z [chris] Still rescanning. At block 759032. Progress=0.999291
2022-10-19T09:16:39Z [chris] Scanning current mempool transactions.
2022-10-19T09:16:39Z [chris] Rescan completed in        34260150ms
```
I posted the full backtraces in #26317 (comment) before making a separate report for this crash.

This was bitcoin-qt built from git tag v24.0rc1 on Debian Linux, running on Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz with SSD hard drives.
dooglus added the label Bug on Oct 19, 2022

willcl-ark commented at 9:38 PM on October 19, 2022: contributor

With the same build on Ubuntu 22.04 I could not reproduce this error with a number of tests. I did however take a shortcut which could have influenced the outcome, namely only performing a ~10,000 block rescan, but using bitcoin-cli -rpctimeout=5 rescanblockchain 750000 so that my bitcoin-cli call would time out in the same way as yours before the rescan was complete.

The first timeout is expected as the function call will likely not return before a full chain rescan is complete. In my tests I could still check rescan progress and cancel the scan with abortrescan as expected. Additional bitcoin-cli calls after the rescan was complete such as listunspent also still worked.

SIGPIPE on b-http indicates that bitcoin-qt tried to write back to an RPC/REST socket, presumably the call from bitcoin-cli rescanblockchain in this case, and found it closed. However my tests indicate that Bitcoin Core (and libevent) is generally robust when trying to write back to closed FDs:

<details> <summary>Log file for early cli return on rescanblockchain</summary>

2022-10-19T21:25:10Z [libevent:debug] event_del: 0x7f7224004eb0 (fd -1), callback 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_del: 0x7f7224004f30 (fd -1), callback 0x7f7315b3b500
2022-10-19T21:25:10Z [libevent:debug] event_add: event: 0x7f7224004f30 (fd 21),  EV_WRITE   call 0x7f7315b3b500
2022-10-19T21:25:10Z [libevent:debug] Epoll ADD(4) on fd 21 okay. Old events were 0; read change was 0 (none); write change was 1 (add); close change was 0 (none)
2022-10-19T21:25:10Z [libevent:debug] event_add: event: 0x7f7224004eb0 (fd 21), EV_READ    call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] Epoll MOD(5) on fd 21 okay. Old events were 4; read change was 1 (add); write change was 0 (none); close change was 0 (none)
2022-10-19T21:25:10Z [libevent:debug] event_del: 0x7f7224004f30 (fd 21), callback 0x7f7315b3b500
2022-10-19T21:25:10Z [libevent:debug] Epoll MOD(1) on fd 21 okay. Old events were 6; read change was 0 (none); write change was 2 (del); close change was 0 (none)
2022-10-19T21:25:10Z [libevent:debug] event_add: event: 0x7f7224004eb0 (fd 21), EV_READ   EV_TIMEOUT call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_add: event 0x7f7224004eb0, timeout in 30 seconds 0 useconds, call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_del: 0x7f7224004f30 (fd 21), callback 0x7f7315b3b500
2022-10-19T21:25:10Z [libevent:debug] event_add: event: 0x7f7224004eb0 (fd 21), EV_READ   EV_TIMEOUT call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_add: event 0x7f7224004eb0, timeout in 30 seconds 0 useconds, call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] timeout_next: event: 0x7f7224004eb0, in 30 seconds, 0 useconds
2022-10-19T21:25:10Z [libevent:debug] epoll_dispatch: epoll_wait reports 1
2022-10-19T21:25:10Z [libevent:debug] event_active: 0x7f7224004eb0 (fd 21), res 2, callback 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_process_active: event: 0x7f7224004eb0, EV_READ   call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_add: event: 0x7f7224004eb0 (fd 21), EV_READ   EV_TIMEOUT call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_add: event 0x7f7224004eb0, timeout in 87440 seconds 75927 useconds, call 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] evhttp_add_header: key: Host val: 127.0.0.1

2022-10-19T21:25:10Z [libevent:debug] evhttp_add_header: key: Connection val: close

2022-10-19T21:25:10Z [libevent:debug] evhttp_add_header: key: Content-Type val: application/json

2022-10-19T21:25:10Z [libevent:debug] evhttp_add_header: key: Authorization val: Basic X19jb29raWVfXzo5MGUxNTRjOTA2NWM5ODlkMmM1MTg3NTE1NTM0NDljMzUwNzEzODFmNjQ5ZjA5MGU4ZjBmMTUzZGY3MWI1MzM0

2022-10-19T21:25:10Z [libevent:debug] evhttp_add_header: key: Content-Length val: 55

2022-10-19T21:25:10Z [libevent:debug] evhttp_read_header: checking for post data on 21

2022-10-19T21:25:10Z [libevent:debug] evhttp_get_body_length: bytes to read: 55 (in buffer 55)

2022-10-19T21:25:10Z [libevent:debug] event_del: 0x7f7224004eb0 (fd 21), callback 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] Epoll DEL(1) on fd 21 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2022-10-19T21:25:10Z [libevent:debug] event_del: 0x7f7224004eb0 (fd 21), callback 0x7f7315b3ad70
2022-10-19T21:25:10Z [libevent:debug] event_del: 0x7f7224004eb0 (fd 21), callback 0x7f7315b3ad70
2022-10-19T21:25:10Z [http] Received a POST request for / from 127.0.0.1:56524
2022-10-19T21:25:10Z [rpc] ThreadRPCServer method=rescanblockchain user=__cookie__
2022-10-19T21:25:10Z [timeouttestwallet] Rescan started from block 000000000000000000073cdaa0c1e5f91707f7b3217a4b3265d2e783f2fa83db...
2022-10-19T21:25:30Z [timeouttestwallet] Scanning current mempool transactions.
2022-10-19T21:25:30Z [timeouttestwallet] Rescan completed in           19911ms
2022-10-19T21:25:30Z [libevent:debug] evhttp_add_header: key: Content-Type val: application/json

2022-10-19T21:25:30Z [libevent:debug] event_active: 0x7f71e418b040 (fd -1), res 0, callback 0x56005b0b71c0
2022-10-19T21:25:30Z [libevent:debug] epoll_dispatch: epoll_wait reports 1
2022-10-19T21:25:30Z [libevent:debug] event_active: 0x7f72740688e0 (fd 44), res 2, callback 0x7f7315b45620
2022-10-19T21:25:30Z [libevent:debug] event_del: 0x7f71e418b040 (fd -1), callback 0x56005b0b71c0
2022-10-19T21:25:30Z [libevent:debug] event_process_active: event: 0x7f71e418b040,    call 0x56005b0b71c0
2022-10-19T21:25:30Z [libevent:debug] evhttp_add_header: key: Date val: Wed, 19 Oct 2022 21:25:30 GMT

2022-10-19T21:25:30Z [libevent:debug] evhttp_add_header: key: Content-Length val: 76

2022-10-19T21:25:30Z [libevent:debug] evhttp_add_header: key: Connection val: close

2022-10-19T21:25:30Z [libevent:debug] evhttp_write_buffer: preparing to write buffer

2022-10-19T21:25:30Z [libevent:debug] event_add: event: 0x7f7224004eb0 (fd 21), EV_READ   EV_TIMEOUT call 0x7f7315b3ad70
2022-10-19T21:25:30Z [libevent:debug] Epoll ADD(1) on fd 21 okay. Old events were 0; read change was 1 (add); write change was 0 (none); close change was 0 (none)
2022-10-19T21:25:30Z [libevent:debug] event_add: event 0x7f7224004eb0, timeout in 30 seconds 0 useconds, call 0x7f7315b3ad70
2022-10-19T21:25:30Z [libevent:debug] event_add: event: 0x7f7224004f30 (fd 21),  EV_WRITE  EV_TIMEOUT call 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] Epoll MOD(5) on fd 21 okay. Old events were 2; read change was 0 (none); write change was 1 (add); close change was 0 (none)
2022-10-19T21:25:30Z [libevent:debug] event_add: event 0x7f7224004f30, timeout in 30 seconds 0 useconds, call 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_add: event: 0x7f7224004eb0 (fd 21), EV_READ   EV_TIMEOUT call 0x7f7315b3ad70
2022-10-19T21:25:30Z [libevent:debug] event_add: event 0x7f7224004eb0, timeout in 30 seconds 0 useconds, call 0x7f7315b3ad70
2022-10-19T21:25:30Z [libevent:debug] event_add: event: 0x7f7224004f30 (fd 21),  EV_WRITE  EV_TIMEOUT call 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_add: event 0x7f7224004f30, timeout in 30 seconds 0 useconds, call 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_del: 0x7f71e418b040 (fd -1), callback 0x56005b0b71c0
2022-10-19T21:25:30Z [libevent:debug] event_process_active: event: 0x7f72740688e0, EV_READ   call 0x7f7315b45620
2022-10-19T21:25:30Z [libevent:debug] timeout_next: event: 0x7f7224004eb0, in 30 seconds, 0 useconds
2022-10-19T21:25:30Z [libevent:debug] epoll_dispatch: epoll_wait reports 1
2022-10-19T21:25:30Z [libevent:debug] event_active: 0x7f7224004f30 (fd 21), res 4, callback 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_active: 0x7f7224004eb0 (fd 21), res 2, callback 0x7f7315b3ad70
2022-10-19T21:25:30Z [libevent:debug] event_process_active: event: 0x7f7224004f30,  EV_WRITE  call 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_add: event: 0x7f7224004f30 (fd 21),  EV_WRITE  EV_TIMEOUT call 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_add: event 0x7f7224004f30, timeout in 87459 seconds 987941 useconds, call 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_del: 0x7f7224004f30 (fd 21), callback 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] Epoll MOD(1) on fd 21 okay. Old events were 6; read change was 0 (none); write change was 2 (del); close change was 0 (none)
2022-10-19T21:25:30Z [libevent:debug] event_callback_finalize_many_: 3 events finalizing
2022-10-19T21:25:30Z [libevent:debug] event_del: 0x7f7224004eb0 (fd 21), callback 0x7f7315b3ad70
2022-10-19T21:25:30Z [libevent:debug] Epoll DEL(1) on fd 21 gave Bad file descriptor: DEL was unnecessary.
2022-10-19T21:25:30Z [libevent:debug] event_del: 0x7f7224004f30 (fd 21), callback 0x7f7315b3b500
2022-10-19T21:25:30Z [libevent:debug] event_process_active: event: 0x7f7224004f30,  EV_WRITE  call 0x7f7315b38720

</details>

I am not able to decipher your particular libevent stacktrace, but perhaps someone else (@promag ?) might know more.

dooglus commented at 10:55 PM on October 19, 2022: contributor

@willcl-ark I tried reproducing the crash using the -rpctimeout option that you mentioned but am not finding it:

$ bitcoin-cli -rpctimeout=5 rescanblockchain 750000
Error parsing command line arguments: Invalid parameter -rpctimeout=5

Did you type it wrongly?

Edit: never mind - it's -rpcclienttimeout=5, and trying to shortcut it using that doesn't make it crash for me either. I'll try another very large rescan overnight tonight to see if it crashes again.

$ bitcoin-cli -rpcclienttimeout=1 rescanblockchain 700000 700100
error: timeout on transient error: Could not connect to the server 127.0.0.1:8332 (error code 0 - "timeout reached")

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.

$ bitcoin-cli -rpcclienttimeout=11 rescanblockchain 700000 700100
{
  "start_height": 700000,
  "stop_height": 700100
}

ie. after the timeout the bitcoin-qt is still responsive.

willcl-ark commented at 11:31 PM on October 19, 2022: contributor

Ah yes, it should say rpcclienttimeout. And it's not a crash per se, but just the rpc not returning before the timeout.
dooglus commented at 11:50 PM on October 19, 2022: contributor

The bitcoin-qt process exited immediately upon receiving a SIGPIPE signal. I'd call that a crash. No clean shutdown, it just stopped running.

I've seen lots of timeouts from bitcoin-cli in the past but never before yesterday have I seen it cause the bitcoin server to exit.

willcl-ark commented at 9:38 AM on October 20, 2022: contributor

I agree SIGPIPE should not crash the entire program. Unfortunately I am still unable to reproduce your crash. I have just tested sending SIGPIPE to the http thread whilst in the middle of a rescan (and watching it in gdb), and whilst the signal is detected, execution appears to continue as normal and bitcoin-qt remains operational.

Breakpoint 1 at 0x9c3d16: file ./src/httprpc.cpp, line 179.
Starting program: /home/will/src/bitcoin/src/qt/bitcoin-qt 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

// snip
// called `bitcoin-cli -rpcclienttimeout=1 rescanblockchain 790000`

[Switching to Thread 0x7fff24ff9640 (LWP 2248544)]

Thread 51 "b-httpworker.0" hit Breakpoint 1, HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x7fff00001b10) at ./src/httprpc.cpp:179
179	        UniValue valRequest;
Continuing.

// sent SIGPIPE to thread 2248544

Thread 51 "b-httpworker.0" received signal SIGPIPE, Broken pipe.
0x0000555555fcc66e in std::_Hash_impl::hash<unsigned long> (__val=@0x7fff24ff59b8: 140733814117952) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/functional_hash.h:203
203	      { return hash(&__val, sizeof(__val)); }
[#0](/bitcoin-bitcoin/0/)  0x0000555555fcc66e in std::_Hash_impl::hash<unsigned long> (__val=@0x7fff24ff59b8: 140733814117952)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/functional_hash.h:203
[#1](/bitcoin-bitcoin/1/)  0x0000555555fcc58a in std::hash<std::thread::id>::operator() (this=0x5555572646e0, __id=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:300
[#2](/bitcoin-bitcoin/2/)  0x0000555555fcc1e6 in std::__detail::_Hash_code_base<std::thread::id, std::pair<std::thread::id const, std::vector<std::pair<void*, CLockLocation>, std::allocator<std::pair<void*, CLockLocation> > > >, std::__detail::_Select1st, std::hash<std::thread::id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, false>::_M_hash_code (
    this=0x5555572646e0, __k=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable_policy.h:1219
[#3](/bitcoin-bitcoin/3/)  0x0000555555fcdb69 in std::__detail::_Map_base<std::thread::id, std::pair<std::thread::id const, std::vector<std::pair<void*, CLockLocation>, std::allocator<std::pair<void*, CLockLocation> > > >, std::allocator<std::pair<std::thread::id const, std::vector<std::pair<void*, CLockLocation>, std::allocator<std::pair<void*, CLockLocation> > > > >, std::__detail::_Select1st, std::equal_to<std::thread::id>, std::hash<std::thread::id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[] (this=0x5555572646e0, __k=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable_policy.h:729
[#4](/bitcoin-bitcoin/4/)  0x0000555555fca31e in std::unordered_map<std::thread::id, std::vector<std::pair<void*, CLockLocation>, std::allocator<std::pair<void*, CLockLocation> > >, std::hash<std::thread::id>, std::equal_to<std::thread::id>, std::allocator<std::pair<std::thread::id const, std::vector<std::pair<void*, CLockLocation>, std::allocator<std::pair<void*, CLockLocation> > > > > >::operator[] (this=0x5555572646e0, __k=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unordered_map.h:984
[#5](/bitcoin-bitcoin/5/)  0x0000555555fc7d80 in pop_lock () at ./src/sync.cpp:201
[#6](/bitcoin-bitcoin/6/)  0x0000555555fc7d0e in LeaveCritical () at ./src/sync.cpp:246
[#7](/bitcoin-bitcoin/7/)  0x00005555558d5e1c in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::~UniqueLock (this=0x7fff24ff5ac8) at ./sync.h:198
[#8](/bitcoin-bitcoin/8/)  0x00005555561e5609 in wallet::CWallet::GetDebit (this=0x7ffeb7161390, txin=..., filter=@0x7fff24ff5bf4: 3)
    at ./src/wallet/wallet.cpp:1404
[#9](/bitcoin-bitcoin/9/)  0x00005555561e59c7 in wallet::CWallet::GetDebit (this=0x7ffeb7161390, tx=..., filter=@0x7fff24ff5bf4: 3)
    at ./src/wallet/wallet.cpp:1462
[#10](/bitcoin-bitcoin/10/) 0x00005555561e3359 in wallet::CWallet::IsFromMe (this=0x7ffeb7161390, tx=...) at ./src/wallet/wallet.cpp:1454
[#11](/bitcoin-bitcoin/11/) 0x00005555561e2be6 in wallet::CWallet::AddToWalletIfInvolvingMe (this=0x7ffeb7161390, ptx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr<const CTransaction> (use count 1, weak count 0) = {...}, state=std::variant<wallet::TxStateConfirmed, wallet::TxStateInMempool, wallet::TxStateInactive> [index 0] = {...}, fUpdate=true, rescanning_old_block=true) at ./src/wallet/wallet.cpp:1121
[#12](/bitcoin-bitcoin/12/) 0x00005555561e478a in wallet::CWallet::SyncTransaction (this=0x7ffeb7161390, ptx=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr<const CTransaction> (use count 1, weak count 0) = {...}, state=std::variant<wallet::TxStateConfirmed, wallet::TxStateInMempool, wallet::TxStateInactive> [index 0] = {...}, update_tx=true, rescanning_old_block=true) at ./src/wallet/wallet.cpp:1290
[#13](/bitcoin-bitcoin/13/) 0x00005555561e8e90 in wallet::CWallet::ScanForWalletTransactions (this=0x7ffeb7161390, start_block=..., start_height=759000, max_height=std::optional<int> [no contained value], reserver=..., fUpdate=true, save_progress=false) at ./src/wallet/wallet.cpp:1806
[#14](/bitcoin-bitcoin/14/) 0x00005555563d961f in wallet::rescanblockchain()::$_7::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0x7fff24ff7180, self=..., request=...) at ./src/wallet/rpc/transactions.cpp:905
[#15](/bitcoin-bitcoin/15/) 0x00005555563d8ab7 in std::__invoke_impl<UniValue, wallet::rescanblockchain()::$_7&, RPCHelpMan const&, JSONRPCRequest const&>(std::__invoke_other, wallet::rescanblockchain()::$_7&, RPCHelpMan const&, JSONRPCRequest const&) (__f=..., __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
[#16](/bitcoin-bitcoin/16/) 0x00005555563d89c7 in std::__invoke_r<UniValue, wallet::rescanblockchain()::$_7&, RPCHelpMan const&, JSONRPCRequest const&>(wallet::rescanblockchain()::$_7&, RPCHelpMan const&, JSONRPCRequest const&) (__fn=..., __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
[#17](/bitcoin-bitcoin/17/) 0x00005555563d882b in std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), wallet::rescanblockchain()::$_7>::_M_invoke(std::_Any_data const&, RPCHelpMan const&, JSONRPCRequest const&) (__functor=..., __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
[#18](/bitcoin-bitcoin/18/) 0x000055555649aa4a in std::function<UniValue (RPCHelpMan const&, JSONRPCRequest const&)>::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0x7fff24ff7180, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
[#19](/bitcoin-bitcoin/19/) 0x0000555556494194 in RPCHelpMan::HandleRequest (this=0x7fff24ff7160, request=...) at ./src/rpc/util.cpp:585
[#20](/bitcoin-bitcoin/20/) 0x0000555555bfe00c in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x55555722d4c0 <wallet::GetWalletRPCCommands()::commands+6080>, request=..., result=...) at ./rpc/server.h:109
[#21](/bitcoin-bitcoin/21/) 0x0000555555bfdf20 in std::__invoke_impl<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(std::__invoke_other, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__f=..., __args=@0x7fff24ff73e7: true, __args=@0x7fff24ff73e7: true, __args=@0x7fff24ff73e7: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
[#22](/bitcoin-bitcoin/22/) 0x0000555555bfde1d in std::__invoke_r<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__fn=..., __args=@0x7fff24ff73e7: true, __args=@0x7fff24ff73e7: true, __args=@0x7fff24ff73e7: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
[#23](/bitcoin-bitcoin/23/) 0x0000555555bfdc4d in std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (__functor=..., __args=@0x7fff24ff73e7: true, __args=@0x7fff24ff73e7: true, __args=@0x7fff24ff73e7: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
[#24](/bitcoin-bitcoin/24/) 0x0000555555b54c9f in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x55555722d4c0 <wallet::GetWalletRPCCommands()::commands+6080>, __args=true, __args=true, __args=true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
[#25](/bitcoin-bitcoin/25/) 0x00005555560b2795 in wallet::(anonymous namespace)::WalletLoaderImpl::registerRpcs()::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x7fff5006a170, request=..., result=..., last_handler=true) at ./src/wallet/interfaces.cpp:541
[#26](/bitcoin-bitcoin/26/) 0x00005555560b265c in std::__invoke_impl<bool, wallet::(anonymous namespace)::WalletLoaderImpl::registerRpcs()::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(std::__invoke_other, wallet::(anonymous namespace)::WalletLoaderImpl::registerRpcs()::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__f=..., __args=@0x7fff24ff7717: true, __args=@0x7fff24ff7717: true, __args=@0x7fff24ff7717: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
[#27](/bitcoin-bitcoin/27/) 0x00005555560b2559 in std::__invoke_r<bool, wallet::(anonymous namespace)::WalletLoaderImpl::registerRpcs()::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(wallet::(anonymous namespace)::WalletLoaderImpl::registerRpcs()::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__fn=..., __args=@0x7fff24ff7717: true, __args=@0x7fff24ff7717: true, __args=@0x7fff24ff7717: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
[#28](/bitcoin-bitcoin/28/) 0x00005555560b238d in std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), wallet::(anonymous namespace)::WalletLoaderImpl::registerRpcs()::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (__functor=..., __args=@0x7fff24ff7717: true, __args=@0x7fff24ff7717: true, __args=@0x7fff24ff7717: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
[#29](/bitcoin-bitcoin/29/) 0x0000555555b54c9f in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x7fff5006a170, __args=true, __args=true, __args=true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
[#30](/bitcoin-bitcoin/30/) 0x0000555555b46840 in node::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x7fff5006a228, request=..., result=..., last_handler=true) at ./src/node/interfaces.cpp:469
[#31](/bitcoin-bitcoin/31/) 0x0000555555b4674c in std::__invoke_impl<bool, node::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(std::__invoke_other, node::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__f=..., __args=@0x7fff24ff7957: true, __args=@0x7fff24ff7957: true, __args=@0x7fff24ff7957: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
[#32](/bitcoin-bitcoin/32/) 0x0000555555b46649 in std::__invoke_r<bool, node::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(node::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__fn=..., __args=@0x7fff24ff7957: true, __args=@0x7fff24ff7957: true, __args=@0x7fff24ff7957: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
[#33](/bitcoin-bitcoin/33/) 0x0000555555b4647d in std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), node::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (__functor=..., __args=@0x7fff24ff7957: true, __args=@0x7fff24ff7957: true, __args=@0x7fff24ff7957: true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
[#34](/bitcoin-bitcoin/34/) 0x0000555555b54c9f in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x7fff5006a228, __args=true, __args=true, __args=true) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
[#35](/bitcoin-bitcoin/35/) 0x0000555555d2c3af in ExecuteCommand (command=..., request=..., result=..., last_handler=true) at ./src/rpc/server.cpp:476
[#36](/bitcoin-bitcoin/36/) 0x0000555555d27d9a in ExecuteCommands (commands=std::vector of length 1, capacity 1 = {...}, request=..., result=...) at ./src/rpc/server.cpp:440
[#37](/bitcoin-bitcoin/37/) 0x0000555555d27b2d in CRPCTable::execute (this=0x555557229398 <tableRPC>, request=...) at ./src/rpc/server.cpp:460
[#38](/bitcoin-bitcoin/38/) 0x0000555555f184a8 in HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x7fff00001b10) at ./src/httprpc.cpp:201
[#39](/bitcoin-bitcoin/39/) 0x0000555555f1741e in StartHTTPRPC(std::any const&)::$_0::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (this=0x7fff00001d20, req=0x7fff00001b10) at ./src/httprpc.cpp:300
[#40](/bitcoin-bitcoin/40/) 0x0000555555f173b7 in std::__invoke_impl<bool, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::__invoke_other, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__f=..., __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
, __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
[#41](/bitcoin-bitcoin/41/) 0x0000555555f172e4 in std::__invoke_r<bool, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__fn=..., __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
, __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
[#42](/bitcoin-bitcoin/42/) 0x0000555555f17048 in std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartHTTPRPC(std::any const&)::$_0>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__functor=..., __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
, __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
[#43](/bitcoin-bitcoin/43/) 0x0000555555f2f2e7 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (this=0x7fff00001cf0, __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
, __args=Python Exception <class 'gdb.error'>: There is no member named _M_p.
) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
[#44](/bitcoin-bitcoin/44/) 0x0000555555f2f11d in HTTPWorkItem::operator() (this=0x7fff00001cc0) at ./src/httpserver.cpp:56
[#45](/bitcoin-bitcoin/45/) 0x0000555555f35f01 in WorkQueue<HTTPClosure>::Run (this=0x7fff5004f490) at ./src/httpserver.cpp:111
[#46](/bitcoin-bitcoin/46/) 0x0000555555f25be2 in HTTPWorkQueueRun (queue=0x7fff5004f490, worker_num=0) at ./src/httpserver.cpp:343
[#47](/bitcoin-bitcoin/47/) 0x0000555555f39fbd in std::__invoke_impl<void, void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> (__f=@0x7fff5006c008: 0x555555f25b80 <HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int)>, __args=@0x7fff5006bff8: 0, __args=@0x7fff5006bff8: 0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
[#48](/bitcoin-bitcoin/48/) 0x0000555555f39dc8 in std::__invoke<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> (__fn=@0x7fff5006c008: 0x555555f25b80 <HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int)>, __args=@0x7fff5006bff8: 0, __args=@0x7fff5006bff8: 0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96
[#49](/bitcoin-bitcoin/49/) 0x0000555555f39d4c in std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7fff5006bff8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253
[#50](/bitcoin-bitcoin/50/) 0x0000555555f39cb6 in std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >::operator() (this=0x7fff5006bff8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260
[#51](/bitcoin-bitcoin/51/) 0x0000555555f399ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run (this=0x7fff5006bff0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211
[#52](/bitcoin-bitcoin/52/) 0x00007ffff66ac2b3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
[#53](/bitcoin-bitcoin/53/) 0x00007ffff6335b43 in start_thread (arg=<optimised out>) at ./nptl/pthread_create.c:442
[#54](/bitcoin-bitcoin/54/) 0x00007ffff63c7a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Continuing.

// snip
// sent `bitcoin-cli getwalletinfo` command

[Switching to Thread 0x7fff1ffff640 (LWP 2248545)]

Thread 52 "b-httpworker.1" hit Breakpoint 1, HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x7fff00001b60) at ./src/httprpc.cpp:179
179	        UniValue valRequest;
Continuing.

// sent `bitcoin-cli stop` command

[Switching to Thread 0x7fff1effd640 (LWP 2248546)]

Thread 53 "b-httpworker.2" hit Breakpoint 1, HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x7fff00001250) at ./src/httprpc.cpp:179
179	        UniValue valRequest;
Continuing.

// snip

[Inferior 1 (process 2247667) exited normally]

</details>

hebasto commented at 11:47 AM on October 20, 2022: member

bitcoin-qt crashed with this error in gdb

Could it be possible that a crash is caused by gdb?

See: https://ahlamnote.blogspot.com/2006/12/gdb-ignore-sigpipe.html.
hebasto added the label RPC/REST/ZMQ on Oct 20, 2022

willcl-ark commented at 12:00 PM on October 20, 2022: contributor

Interesting idea. I did just test that (once) and saw no difference in behaviour (i.e. it did not crash qt):

Breakpoint 1 at 0x9c3d16: file ./src/httprpc.cpp, line 179.
Signal        Stop	Print	Pass to program	Description
SIGPIPE       No	No	Yes		Broken pipe
Starting program: /home/will/src/bitcoin/src/qt/bitcoin-qt 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

// snip
// send bitcoin-cli rescanblockchain

[Switching to Thread 0x7fff0bfff640 (LWP 3898352)]

Thread 51 "b-httpworker.0" hit Breakpoint 1, HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x7fff04001b10) at ./src/httprpc.cpp:179
179	        UniValue valRequest;
Continuing.

// sent SIGPIPE which did not pause GDB
// rescan finished

// sent bitcoin-cli getwalletinfo

[Switching to Thread 0x7fff0b7fe640 (LWP 3898353)]

Thread 52 "b-httpworker.1" hit Breakpoint 1, HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x7fff04001b60) at ./src/httprpc.cpp:179
179	        UniValue valRequest;
Continuing.
[Switching to Thread 0x7fff0affd640 (LWP 3898354)]

// sent bitcoin-cli stop

Thread 53 "b-httpworker.2" hit Breakpoint 1, HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x7fff040023f0) at ./src/httprpc.cpp:179
179	        UniValue valRequest;
Continuing.

// snip

[Inferior 1 (process 3897242) exited normally]

</details>

dooglus commented at 7:40 PM on October 20, 2022: contributor

Last night I ran an almost-full rescan inside gdb and bitcoin-qt crashed on SIGPIPE again, the same as it did the previous night.

Now I'm trying it without gdb. I'll report what happens when the rescan completes.
dooglus commented at 1:30 PM on October 21, 2022: contributor

The full rescan ran without problems when I wasn't running bitcoin-qt inside gdb.

I'll close this issue since it's a gdb problem not a bitcoin problem.
dooglus closed this on Oct 21, 2022
willcl-ark commented at 5:32 PM on October 21, 2022: contributor

Thanks for the investigation and follow up @dooglus
dooglus commented at 6:57 PM on October 21, 2022: contributor
It turns out that when gdb sees the SIGPIPE it pauses, but I type continue and hit return and bitcoin-qt carries on running as normal.

You can stop gdb pausing bitcoin-qt execution by entering the following into gdb before running bitcoin-qt:
```
(gdb) handle SIGPIPE nostop print pass
```
Then when the SIGPIPE signal happens we see this in the gdb output, but execution of bitcoin-qt isn't interrupted:
```
Thread 35 "b-http" received signal SIGPIPE, Broken pipe.
```
bitcoin locked this on Oct 21, 2023

Contributors

Labels

Linked (view graph)

#26317 avoid_reuse=true is incorrectly skipping one of my UXTOs