Block filter index corruption post reorg and unclean shutdown #34261

issue dergoegge opened this issue on January 12, 2026
  1. dergoegge commented at 11:36 AM on January 12, 2026: member

    The block filter index may be left in a corrupted state, preventing the node from restarting (without reindexing), after a series of specific events:

    1. Clean shutdown
    2. Restart combined with an immediate reorg of the previously stored best chain tip
    3. Unclean shutdown after reorg takes place (before chainstate flush)
    4. Restart, leading to the following
    [        62.579] [               node2] [inf] 2026-01-10T16:58:52Z [all:error] previous block header belongs to unexpected block 44ee2f358de32719802fe11d3985af7a0f9077b90ee5ca16c42f7278a8d89a50; expected 532c10b42179b4f3f5a41f9990108a63bcd7021dd02ba6997f5ff3375d297676
    [        62.579] [               node2] [inf] 2026-01-10T16:58:52Z [all:error] Cannot read last block filter header; index may be corrupted
    

    See attached debug log for an example of this happening.

    blockfilter_corruption.log

    This was found with a test running on Antithesis.

  2. maflcko added the label UTXO Db and Indexes on Jan 12, 2026
  3. maflcko added the label Data corruption on Jan 12, 2026
  4. mzumsande commented at 1:29 PM on January 12, 2026: contributor

    Looks very similar to #33208 / #33212#pullrequestreview-3140857089 . I did some work on a fix for that in fall (making sure that we never commit an index ahead of the flushed chainstate) but there were some complications with my approach and I never PR'ed it.

  5. mzumsande commented at 5:56 AM on March 23, 2026: contributor

    I now opened #34897 to fix it.

  6. achow101 closed this on Jul 9, 2026

  7. Fabcien referenced this in commit 7d8137c141 on Jul 9, 2026
  8. Crypt-iQ commented at 1:35 AM on October 11, 2026: contributor

    I don't think #34897 fully fixed this issue. I am able to reproduce this with the following LLM-generated functional test:

    <details> <summary>test</summary>

    #!/usr/bin/env python3
    # Copyright (c) 2026-present The Bitcoin Core developers
    # Distributed under the MIT software license, see the accompanying
    # file COPYING or http://www.opensource.org/licenses/mit-license.php.
    """Test that the block filter index survives a reorg followed by an unclean shutdown.
    
    The block filter index writes its height-keyed records eagerly, in
    BlockFilterIndex::Write, rather than through the deferred BaseIndex::Commit
    batch. So after a reorg the record at a given height holds the new branch's
    block even if the chainstate never flushed. An unclean shutdown then rolls the
    chainstate back to the last flushed tip, and on the next start
    BlockFilterIndex::CustomInit reads the height record for that restored tip,
    finds the other branch's block there, and fails fatally:
    
        [error] previous block header belongs to unexpected block <new>; expected <restored>
        [error] Cannot read last block filter header; index may be corrupted
    
    The node then cannot start again without a reindex.
    
    The filter for the restored tip is not lost -- BlockFilterIndex::CustomRemove
    copies it to the hash index before the height record is overwritten -- so the
    init path can recover it via index_util::LookUpOne, which already implements
    exactly that height-then-hash fallback.
    
    Related: bitcoin#34261, which PR bitcoin#34897 only fixed for indexes whose
    state goes through Commit() (coinstatsindex).
    """
    
    from test_framework.test_framework import BitcoinTestFramework
    from test_framework.util import assert_equal
    
    # Blocks on the branch that gets flushed, and on the branch that replaces it.
    # The second branch only has to be longer than the first.
    FLUSHED_BRANCH_LEN = 10
    REORG_BRANCH_LEN = 12
    
    
    class BlockFilterIndexReorgUncleanShutdownTest(BitcoinTestFramework):
        def set_test_params(self):
            self.num_nodes = 2
            self.setup_clean_chain = True
            # node0 carries the index and is the one that gets killed. node1 only
            # exists to hand it a competing branch.
            self.extra_args = [["-blockfilterindex=1"], []]
    
        def setup_network(self):
            # Connected on demand: the two branches have to be mined in isolation.
            self.setup_nodes()
    
        def wait_for_index(self, node):
            """Wait until the block filter index has caught up with the node's tip."""
            node.syncwithvalidationinterfacequeue()
            self.wait_until(lambda: node.getindexinfo("basic block filter index")
                            ["basic block filter index"]["best_block_height"] == node.getblockcount())
    
        def run_test(self):
            node, helper = self.nodes
    
            self.log.info("Mine the branch that will be flushed to disk")
            self.generate(node, FLUSHED_BRANCH_LEN, sync_fun=self.no_op)
            self.wait_for_index(node)
            flushed_tip = node.getbestblockhash()
    
            self.log.info("Mine a longer competing branch on the helper node")
            self.generate(helper, REORG_BRANCH_LEN, sync_fun=self.no_op)
            reorg_tip = helper.getbestblockhash()
            assert flushed_tip != reorg_tip
    
            # A clean shutdown is what makes the chainstate and the index durable at
            # the same point. Everything after this restart is unflushed.
            self.log.info("Restart cleanly, so chainstate and index are durable at this tip")
            self.restart_node(0)
            assert_equal(node.getbestblockhash(), flushed_tip)
    
            self.log.info("Reorg onto the competing branch, without a chainstate flush")
            self.connect_nodes(0, 1)
            self.sync_blocks()
            assert_equal(node.getbestblockhash(), reorg_tip)
            # The index follows the reorg and overwrites its height records with the
            # new branch's blocks, copying the displaced ones to the hash index.
            self.wait_for_index(node)
    
            # Nothing below may reconnect or re-announce the reorged branch.
            self.disconnect_nodes(0, 1)
            self.stop_node(1)
    
            self.log.info("Kill the node before it can flush the reorg")
            node.kill_process()
    
            self.log.info("Restart: the node must come back on the last flushed tip")
            # On the buggy build this start fails with "Cannot read last block
            # filter header; index may be corrupted" and the node exits 1.
            self.start_node(0)
            assert_equal(node.getbestblockhash(), flushed_tip)
    
            self.log.info("The index must still serve the restored tip's filter")
            node.getblockfilter(flushed_tip)
            self.wait_for_index(node)
            assert_equal(node.getindexinfo("basic block filter index")
                         ["basic block filter index"]["best_block_height"], FLUSHED_BRANCH_LEN)
    
    
    if __name__ == '__main__':
        BlockFilterIndexReorgUncleanShutdownTest(__file__).main()
    

    </details>

    The logs of node0 show the exact logs in the OP on 228877d7836a55c1d6fde90b0b7ba2d4c90248ab.

    Ran into this with fuzzamoto + bedrock.

  9. fanquake reopened this on Oct 11, 2026


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-10-19 06:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me