Allow proper linearize-data execution on pruned node #16431

pull cryptogopher wants to merge 2 commits into bitcoin:master from cryptogopher:master changing 1 files +10 −0

cryptogopher commented at 7:32 PM on July 21, 2019: none
linearize-hashes.py properly creates hashlist within given min_height and max_height on a pruned node.

linearize-data.py expects to find blk00000.dat (which may be missing on pruned node along with additional subsequent blk files) and throws error:
```
   $ ./linearize-data.py linearize.cfg 
   Read 15000 hashes
   Input file <some-dir>/blocks/blk00000.dat
   Premature end of block data
```
even though blocks in the range min_height-max_height are not pruned.

To avoid this error and allow for proper linearize-data run on pruned node it is enough to start with first non-pruned blk file. Attached code does exactly this.
Added searching for first *.blk file on pruned node 17f9f745f0
DrahtBot added the label Scripts and tools on Jul 21, 2019
in contrib/linearize/linearize-data.py:106 in 17f9f745f0 outdated
```
 102 | @@ -102,6 +103,13 @@ def __init__(self, settings, blkindex, blkmap):
 103 |          self.blkmap = blkmap
 104 |  
 105 |          self.inFn = 0
 106 | +        fnPattern = os.path.join(self.settings['input'], "blk[0-9][0-9][0-9][0-9][0-9].dat")
```
promag commented at 8:51 AM on July 23, 2019:

Could add a comment here explaining the following lines.

cryptogopher commented at 7:01 PM on July 26, 2019:

Added comment. Please feel free to modify at the discretion.

laanwj commented at 4:16 PM on July 30, 2019:

would make sense to factor this out to a function, getFirstBlockFileId(path) or such
promag commented at 8:51 AM on July 23, 2019: member

utACK 17f9f745f02c685f03689e412189dd880db376bf.
Adding change comment as per @promag suggestion 230a7f17a5
jgarzik approved
jgarzik commented at 9:07 PM on July 26, 2019: contributor

utACK
laanwj commented at 4:17 PM on July 30, 2019: member

Concept ACK
laanwj commented at 11:30 AM on September 30, 2019: member

Please squash the commits, and remove any @ username from the commit message (because that causes trouble).
laanwj added the label Waiting for author on Sep 30, 2019
laanwj removed the label Waiting for author on Oct 8, 2019
laanwj added the label Up for grabs on Oct 8, 2019
laanwj commented at 9:52 AM on October 8, 2019: member

The author is taking too long to respond to comments. Closing as "Up for grabs". Let me know if you want to start work on this again and I'll re-open.
laanwj closed this on Oct 8, 2019
Rjected commented at 3:12 AM on October 28, 2019: contributor

Do you mind if I submit a PR for this? I have a refactored & commented version in my fork: 317fb96
fanquake removed the label Up for grabs on Jan 5, 2020
fanquake referenced this in commit 8625446b4d on Feb 5, 2020
sidhujag referenced this in commit 87d9d6b05c on Feb 9, 2020
sidhujag referenced this in commit 6f338c22b4 on Nov 10, 2020
PastaPastaPasta referenced this in commit 128de3ef18 on Jun 27, 2021
PastaPastaPasta referenced this in commit c66c35ab24 on Jun 28, 2021
PastaPastaPasta referenced this in commit cb72156a40 on Jun 29, 2021
PastaPastaPasta referenced this in commit 5d33f9c5d2 on Jul 1, 2021
PastaPastaPasta referenced this in commit 0fdef0d5a9 on Jul 1, 2021
PastaPastaPasta referenced this in commit 9465b216f0 on Jul 14, 2021
PastaPastaPasta referenced this in commit 573d3a0e02 on Jul 14, 2021
DrahtBot locked this on Feb 15, 2022

Contributors

Labels

Linked (view graph)

#17336 scripts: search for first block file for linearize-data with some block files pruned