contrib: make linearize-data.py cope with out-of-order blocks #5051

1. 
   laanwj commented at 5:27 pm on October 6, 2014: member

   Makes it possible to read blocks in any order. This will be required after headers-first (#4468), so should be merged before that.

   • For in-order blocks: copy block contents directly. Write prior out-of-order blocks if this connects a consecutive span.
   • For out-of-order blocks, store extents of block data for later retrieval. Also cache out-of-order block data in memory up to 100MB (configurable).

   Tested: resulting bootstrap.dat (from a node where the chain is out-of-order) can be imported; sha256sum of default range (0..313000):

   008ec49237decea485b246ff7f29e8ec900ffcfef4388a53be92428e800317df6  bootstrap.dat
   121575802677 bytes
   

   Import success:

   02014-10-06 19:05:22 UpdateTip: new best=000000000000000018f6d53196c6f51b402a82e3e5c04164b46efe1f33a1fca8  height=313000  log2_work=79.944961  tx=43442600  date=2014-07-29 11:05:31 progress=0.773505
   12014-10-06 19:05:22 Loaded 313000 blocks from external file in 11534120ms
   

2. contrib: make linearize-data.py cope with out-of-order blocks

   Make it possible to read blocks in any order. This will be required
   after headers-first (#4468), so should be merged before that.
   
   - Read block header. For expected blocks, continue, else skip.
   - For in-order blocks: copy block contents directly. Write prior
     out-of-order blocks if this connects a consecutive span.
   - For out-of-order blocks, store extents of block data for later
     retrieval. Cache out-of-order blocks in memory up to 100MB
     (configurable).

   aedc74dfa6
3. 
   jgarzik commented at 5:36 pm on October 6, 2014: contributor

   ut ACK
4. 
   sipa commented at 10:04 pm on October 6, 2014: member

   The headersfirst patch as-is allows a 1024-block download window, which means up to 1024 MiB, though in practice it will be a lot less. I think we’ll want to change the download window to a bytes limits rather than a block count, but that’s nontrivial, and shouldn’t be a blocker.
5. 
   laanwj commented at 6:33 am on October 7, 2014: member

   @sipa Functionality is not affected by that here. The 100MB is just an optimization to be able to resolve small order inconsistencies without seeking. If it fills that up, it will still be able to function as it remembers where the block is.
6. laanwj merged this on Oct 8, 2014
7. laanwj closed this on Oct 8, 2014

   ---

8. laanwj referenced this in commit 97a34c28d5 on Oct 8, 2014
9. MarcoFalke locked this on Sep 8, 2021

Contributors
laanwj jgarzik sipa