validation: `CheckBlockIndex` crashes during block reconsideration #32173

issue dergoegge openend this issue on March 31, 2025

dergoegge commented at 11:50 am on March 31, 2025: member

Functional test to reproduce:

 0from test_framework.test_framework import BitcoinTestFramework
 1
 2class CheckBlockIndexBug(BitcoinTestFramework):
 3    def set_test_params(self):
 4        self.setup_clean_chain = True
 5        self.num_nodes = 1
 6
 7    def run_test(self):
 8        self.generatetoaddress(self.nodes[0], 1, "2N9hLwkSqr1cPQAPxbrGVUjxyjD11G2e1he");
 9        hashes = self.generatetoaddress(self.nodes[0], 1, "2N9hLwkSqr1cPQAPxbrGVUjxyjD11G2e1he");
10        self.generatetoaddress(self.nodes[0], 1, "2N2CmnxjBbPTHrawgG2FkTuBLcJtEzA86sF");
11
12        res = self.nodes[0].gettxoutsetinfo()
13        self.generatetoaddress(self.nodes[0], 3, "2N9hLwkSqr1cPQAPxbrGVUjxyjD11G2e1he");
14        self.log.info(self.nodes[0].invalidateblock(res["bestblock"]))
15        self.generatetoaddress(self.nodes[0], 3, "2N9hLwkSqr1cPQAPxbrGVUjxyjD11G2e1he");
16        self.nodes[0].reconsiderblock(hashes[0])
17        self.nodes[0].invalidateblock(hashes[0])
18        self.log.info(self.nodes[0].reconsiderblock(res["bestblock"]))
19
20if __name__ == '__main__':
21    CheckBlockIndexBug(__file__).main()

Stack trace:

 0bitcoind: validation.cpp:5392: void ChainstateManager::CheckBlockIndex(): Assertion `(pindex->nStatus & BLOCK_FAILED_MASK) == 0' failed.
 1AddressSanitizer:DEADLYSIGNAL
 2=================================================================
 3==3423379==ERROR: AddressSanitizer: ABRT on unknown address 0x000000343c93 (pc 0x7fb83abf0eec bp 0x7fb8348f16c0 sp 0x7fb8348efc30 T6)
 4    [#0](/bitcoin-bitcoin/0/) 0x7fb83abf0eec  (/lib/x86_64-linux-gnu/libc.so.6+0x8aeec) (BuildId: 79005c16293efa45b441fed45f4f29b138557e9e)
 5    [#1](/bitcoin-bitcoin/1/) 0x7fb83aba1fb1 in raise (/lib/x86_64-linux-gnu/libc.so.6+0x3bfb1) (BuildId: 79005c16293efa45b441fed45f4f29b138557e9e)
 6    [#2](/bitcoin-bitcoin/2/) 0x7fb83ab8c471 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x26471) (BuildId: 79005c16293efa45b441fed45f4f29b138557e9e)
 7    [#3](/bitcoin-bitcoin/3/) 0x7fb83ab8c394  (/lib/x86_64-linux-gnu/libc.so.6+0x26394) (BuildId: 79005c16293efa45b441fed45f4f29b138557e9e)
 8    [#4](/bitcoin-bitcoin/4/) 0x7fb83ab9aec1 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x34ec1) (BuildId: 79005c16293efa45b441fed45f4f29b138557e9e)
 9    [#5](/bitcoin-bitcoin/5/) 0x5578f6306f64 in ChainstateManager::CheckBlockIndex() /bitcoin/build_fuzz/src/./validation.cpp:5392:13
10    [#6](/bitcoin-bitcoin/6/) 0x5578f62fe81a in Chainstate::ActivateBestChain(BlockValidationState&, std::shared_ptr<CBlock const>) /bitcoin/build_fuzz/src/./validation.cpp:3627:16
11    [#7](/bitcoin-bitcoin/7/) 0x5578f5d03436 in ReconsiderBlock(ChainstateManager&, uint256) /bitcoin/build_fuzz/src/./rpc/blockchain.cpp:1669:33
12    [#8](/bitcoin-bitcoin/8/) 0x5578f5e3fbc7 in reconsiderblock()::$_0::operator()(RPCHelpMan const&, JSONRPCRequest const&) const /bitcoin/build_fuzz/src/./rpc/blockchain.cpp:1694:5
13    [#9](/bitcoin-bitcoin/9/) 0x5578f5e3fbc7 in UniValue std::__invoke_impl<UniValue, reconsiderblock()::$_0&, RPCHelpMan const&, JSONRPCRequest const&>(std::__invoke_other, reconsiderblock()::$_0&, RPCHelpMan const&, JSONRPCRequest const&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:61:14
14    [#10](/bitcoin-bitcoin/10/) 0x5578f5e3fbc7 in std::enable_if<is_invocable_r_v<UniValue, reconsiderblock()::$_0&, RPCHelpMan const&, JSONRPCRequest const&>, UniValue>::type std::__invoke_r<UniValue, reconsiderblock()::$_0&, RPCHelpMan const&, JSONRPCRequest const&>(reconsiderblock()::$_0&, RPCHelpMan const&, JSONRPCRequest const&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:114:9
15    [#11](/bitcoin-bitcoin/11/) 0x5578f5e3fbc7 in std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), reconsiderblock()::$_0>::_M_invoke(std::_Any_data const&, RPCHelpMan const&, JSONRPCRequest const&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:290:9
16    [#12](/bitcoin-bitcoin/12/) 0x5578f717fe3a in std::function<UniValue (RPCHelpMan const&, JSONRPCRequest const&)>::operator()(RPCHelpMan const&, JSONRPCRequest const&) const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
17    [#13](/bitcoin-bitcoin/13/) 0x5578f717fe3a in RPCHelpMan::HandleRequest(JSONRPCRequest const&) const /bitcoin/build_fuzz/src/./rpc/util.cpp:684:20
18    [#14](/bitcoin-bitcoin/14/) 0x5578f5dc8455 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, RPCHelpMan (*)())::'lambda'(JSONRPCRequest const&, UniValue&, bool)::operator()(JSONRPCRequest const&, UniValue&, bool) const /bitcoin/build_fuzz/src/./rpc/server.h:101:91
19    [#15](/bitcoin-bitcoin/15/) 0x5578f5dc8455 in bool std::__invoke_impl<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, RPCHelpMan (*)())::'lambda'(JSONRPCRequest const&, UniValue&, bool)&, JSONRPCRequest const&, UniValue&, bool>(std::__invoke_other, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, RPCHelpMan (*)())::'lambda'(JSONR
20PCRequest const&, UniValue&, bool)&, JSONRPCRequest const&, UniValue&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:61:14
21    [#16](/bitcoin-bitcoin/16/) 0x5578f5dc8455 in std::enable_if<is_invocable_r_v<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, RPCHelpMan (*)())::'lambda'(JSONRPCRequest const&, UniValue&, bool)&, JSONRPCRequest const&, UniValue&, bool>, bool>::type std::__invoke_r<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, RPCHelpMan
22(*)())::'lambda'(JSONRPCRequest const&, UniValue&, bool)&, JSONRPCRequest const&, UniValue&, bool>(CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, RPCHelpMan (*)())::'lambda'(JSONRPCRequest const&, UniValue&, bool)&, JSONRPCRequest const&, UniValue&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:114:9
23    [#17](/bitcoin-bitcoin/17/) 0x5578f5dc8455 in std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, RPCHelpMan (*)())::'lambda'(JSONRPCRequest const&, UniValue&, bool)>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:290:
249
25    [#18](/bitcoin-bitcoin/18/) 0x5578f6177d47 in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
26    [#19](/bitcoin-bitcoin/19/) 0x5578f6177d47 in ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) /bitcoin/build_fuzz/src/./rpc/server.cpp:512:20
27    [#20](/bitcoin-bitcoin/20/) 0x5578f6177d47 in ExecuteCommands(std::vector<CRPCCommand const*, std::allocator<CRPCCommand const*>> const&, JSONRPCRequest const&, UniValue&) /bitcoin/build_fuzz/src/./rpc/server.cpp:477:13
28    [#21](/bitcoin-bitcoin/21/) 0x5578f6176b16 in CRPCTable::execute(JSONRPCRequest const&) const /bitcoin/build_fuzz/src/./rpc/server.cpp:497:13
29    [#22](/bitcoin-bitcoin/22/) 0x5578f61759e6 in JSONRPCExec(JSONRPCRequest const&, bool) /bitcoin/build_fuzz/src/./rpc/server.cpp:353:31
30    [#23](/bitcoin-bitcoin/23/) 0x5578f6512301 in HTTPReq_JSONRPC(std::any const&, HTTPRequest*) /bitcoin/build_fuzz/src/./httprpc.cpp:217:21
31    [#24](/bitcoin-bitcoin/24/) 0x5578f653449f in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
32    [#25](/bitcoin-bitcoin/25/) 0x5578f653449f in HTTPWorkItem::operator()() /bitcoin/build_fuzz/src/./httpserver.cpp:60:9
33    [#26](/bitcoin-bitcoin/26/) 0x5578f653c305 in WorkQueue<HTTPClosure>::Run() /bitcoin/build_fuzz/src/./httpserver.cpp:115:13
34    [#27](/bitcoin-bitcoin/27/) 0x5578f6526470 in HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) /bitcoin/build_fuzz/src/./httpserver.cpp:417:12
35    [#28](/bitcoin-bitcoin/28/) 0x7fb83af2c4a2  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd44a2) (BuildId: 0c47cec75226c7736517d5acb61e373d541a5023)
36    [#29](/bitcoin-bitcoin/29/) 0x5578f56a1b06 in asan_thread_start(void*) asan_interceptors.cpp.o
37    [#30](/bitcoin-bitcoin/30/) 0x7fb83abef1f4  (/lib/x86_64-linux-gnu/libc.so.6+0x891f4) (BuildId: 79005c16293efa45b441fed45f4f29b138557e9e)
38    [#31](/bitcoin-bitcoin/31/) 0x7fb83ac6f89b  (/lib/x86_64-linux-gnu/libc.so.6+0x10989b) (BuildId: 79005c16293efa45b441fed45f4f29b138557e9e)

stratospher commented at 6:14 pm on March 31, 2025: contributor
I came across this error too when playing with @mzumsande’s fuzzer in #31533!

since reconsiderblock doesn’t traverse the whole tree of block headers and only ancestors/descendants, we could have failed blocks which don’t descend from invalid blocks. (there’s a picture below showing a similar situation.)

2 possible solutions:
1. modify the check in CheckBlockIndex() so that we check for this condition only on the active chain.
2. the crash doesn’t happen if we remove BLOCK_FAILED_VALID/BLOCK_FAILED_CHILD distinction - I have a branch in https://github.com/stratospher/bitcoin/commits/2025_02_remove_block_failed_child/ and some discussion about this idea in “alternate approach section” in #31835
picture of a similar situation:

mzumsande commented at 6:42 pm on March 31, 2025: contributor

A third possible fix: change pindexFirstInvalid so that it is set both on BLOCK_FAILED_VALID and BLOCK_FAILED_CHILD:

 0diff --git a/src/validation.cpp b/src/validation.cpp
 1index fedcb9ca57..4fea14b2a5 100644
 2--- a/src/validation.cpp
 3+++ b/src/validation.cpp
 4@@ -5323,7 +5323,7 @@ void ChainstateManager::CheckBlockIndex()
 5 
 6     while (pindex != nullptr) {
 7         nNodes++;
 8-        if (pindexFirstInvalid == nullptr && pindex->nStatus & BLOCK_FAILED_VALID) pindexFirstInvalid = pindex;
 9+        if (pindexFirstInvalid == nullptr && pindex->nStatus & BLOCK_FAILED_MASK) pindexFirstInvalid = pindex;
10         if (pindexFirstMissing == nullptr && !(pindex->nStatus & BLOCK_HAVE_DATA)) {
11             pindexFirstMissing = pindex;
12         }

Although removing BLOCK_FAILED_CHILD (Option 2) seems like the best solution long-term.

Contributors
dergoegge stratospher mzumsande

validation: CheckBlockIndex crashes during block reconsideration #32173

validation: `CheckBlockIndex` crashes during block reconsideration #32173