Make -reindex cope with out-of-order blocks #5057

1. 
   laanwj commented at 4:11 pm on October 7, 2014: member

   Remember out-of-order block headers along with their disk positions. This is likely the simplest and least-impact way to make -reindex work with headers first.

   Based on top of #4468, only the last commit is mine.

   Edit: Known issue:

   • when restarting after -reindex it doesn’t start from the right place; it should start at the position of the earliest block with unknown parent, not that of the latest valid block
2. Headers-first synchronization

   Many changes:
   * Do not use 'getblocks', but 'getheaders', and use it to build a headers tree.
   * Blocks are fetched in parallel from all available outbound peers, using a
     limited moving window. When one peer stalls the movement of the window, it is
     disconnected.
   * No more orphan blocks. At all. We only ever request a block for which we have
     verified the headers, and store it to disk immediately. This means that a
     disk-fill attack would require PoW.
   * Require protocol version 31800 for every peer (released in december 2010).
   * No more syncnode (we sync from everyone we can, though limited to 1 during
     initial *headers* sync).
   * Introduce some extra named constants, comments and asserts.

   d3fd120f32
3. RPC additions after headers-first 39dc93c3f2
4. Remove CheckMinWork, as we always know all parent headers e30045d5b4
5. Improve getheaders (sending) logging 84d1ee56b8
6. Better logging of stalling 2ea9ff53c8
7. Add height to "Requesting block" debug 9a3ab2c814
8. Rename setBlockIndexValid to setBlockIndexCandidateTips f062199453
9. Make -reindex cope with out-of-order blocks

   Remember out-of-order block headers along with disk positions. This is
   likely the simplest and least-impact way to make -reindex work with
   headers first.
   
   Based on top of #4468.

   8377e2807b
10. laanwj force-pushed on Oct 7, 2014
11. squashme: don't ignore ReadBlockFromDisk return value 6213af13e4
12. squashme: allow interruption while processing out-of-order blocks f08c86342e
13. squashme: reduce debug logging da1fa38485
14. 
    laanwj commented at 6:22 am on October 8, 2014: member

    @sipa I added two minor extra commits here, f08c863 and da1fa38, can you squash them in to #4468 as well?
15. 
    laanwj commented at 6:40 am on October 8, 2014: member

    Hmm thinking about it I’m not sure early termination (f08c863) is safe given restarts:

    • Bitcoind given the stop command while reindexing, it has just processed block A and is now processing earlier child blocks of that B, C and D. It stops after B.
    • On the re-run, it will encounter B again and skip it (already known), and C and D again and put them in mapBlocksUnknownParent. So far so good.
    • However, when it encounters A it will skip it too. It never gets to the “Recursively process earlier encountered successors of this block” loop for A. So the unprocessed children, C and D will never be added.

    I think this could be solved by executing the “Recursively process earlier encountered successors of this block” loop even for known blocks.

16. 
    laanwj commented at 6:42 am on October 8, 2014: member

    Closing in favor of #4468 which has this merged already.
17. laanwj closed this on Oct 8, 2014

    ---

18. MarcoFalke locked this on Sep 8, 2021

Contributors
laanwj