Background sync requests blocks up to and including the snapshot block from several peers at once, so the snapshot block can be stored before one of its ancestors. ReceivedBlockTransactions() then keeps it in m_blocks_unlinked, but that isn't persisted. On restart LoadBlockIndex() gives it the hardcoded m_chain_tx_count, PopulateBlockIndexCandidates() makes it a background candidate, and FindMostWorkChain() walks back to its parent, which has no chain tx count:
Assertion failed: (pindexTest->HaveNumChainTxs() || pindexTest->nHeight == 0), function FindMostWorkChain, file validation.cpp, line 3217.
It needs a restart in that window, so it should be rare, but after that the node aborts on every start. The only ways out I found are -reindex-chainstate or -reindex, and both delete the snapshot chainstate.
Fix this by adding the snapshot block to m_blocks_unlinked on startup if an ancestor is missing, and not making it a background candidate until then. Both are needed, without the first one background validation stalls. A node that is already stuck starts with the fix and finishes background validation without a reindex.
The new step in feature_assumeutxo.py restarts the node after the snapshot block is submitted out of order, and fails on master. I also reproduced it over p2p with a test peer that withholds the snapshot block's parent.
These are the two changes from #35307 (closed, the motivation there was submitblock before loadtxoutset) without its blockfile and VerifyDB changes. cyb3ralbert hit the same assertion while testing #36452 (https://github.com/bitcoin/bitcoin/pull/36452#issuecomment-6464524112), which is still needed when the snapshot tip is at the snapshot block. #36149 drops the BLOCK_HAVE_DATA check in the branch right below, so whichever is merged second should match it.
