Linearize script update (hash byte reversal and Python 3 support) #9373

pull droark wants to merge 2 commits into bitcoin:master from droark:linearize-update changing 4 files +97 −37

droark commented at 9:43 PM on December 17, 2016: contributor
- Add support for reversed hash bytes.
- Add support for Python 3. The linearization scripts can now run on Python 2 or 3.
Continuing #9304.
jonasschnelli added the label Scripts and tools on Dec 18, 2016
droark commented at 10:07 PM on December 18, 2016: contributor

Found a small bug affecting Python 3. Will update in a bit.
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 20, 2016
droark commented at 1:57 AM on December 20, 2016: contributor

Nevermind. It wasn't a bug after all. Once I clarified things (command line foolishness on my behalf), I ran Python 2 and 3 with and without the bytes reversed, then ran shasum -a 256 bootstrap.dat each time the bootstrap file was written. The results were the same.

While here, I did add a few comments and then did a fixup and rebase.
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 20, 2016
droark commented at 11:32 PM on December 20, 2016: contributor

Added some more documentation updates but seem to have borked my fixup. Will fix it momentarily.
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 20, 2016
droark force-pushed on Dec 21, 2016
droark closed this on Dec 21, 2016
droark force-pushed on Dec 21, 2016
droark reopened this on Dec 21, 2016
droark commented at 3:31 AM on December 21, 2016: contributor

Back to normal. Did a quick sanity check. Everything looks good.
droark force-pushed on Dec 30, 2016

in contrib/linearize/README.md:None in 4940f9c92b outdated

  12 | +* RPC: `rpcuser`, `rpcpassword`
  13 |  
  14 |  Optional config file setting for linearize-hashes:
  15 | -* RPC: host, port
  16 | -* Block chain: min_height, max_height
  17 | +* RPC: `host`, `port`  (Default: `127.0.0.1:8332`)

cdecker commented at 2:00 PM on December 30, 2016:

Minor nit: the default suggests that the ip:port format should be used, while it really is two different options.

in contrib/linearize/README.md:None in 4940f9c92b outdated

  47 | -* "file_timestamp": Set each file's last-modified time to that of the
  48 | -most recent block in that file.
  49 | +* `file_timestamp`: Set each file's last-modified time to that of the most
  50 | +recent block in that file.
  51 | +* `genesis`: The hash of the genesis block in the blockchain.
  52 | +* `input: bitcoind blocks/ directory containing blkNNNNN.dat

cdecker commented at 2:03 PM on December 30, 2016:

Missing closing quote on for input.

in contrib/linearize/linearize-data.py:None in 4940f9c92b outdated
```
  30 |  
  31 | +##### Switch endian-ness #####
  32 | +def hex_switchEndian(s):
  33 | +	""" Switches the endianness of a hex string (in pairs of hex chars) """
  34 | +	pairList = [s[i:i+2].encode() for i in range(0, len(s), 2)]
  35 | +	return b''.join(pairList[::-1]).decode()
```
cdecker commented at 2:17 PM on December 30, 2016:

The encode and decode call seem to be unnecessary. Also my highly unscientific test shows that

s.decode('hex')[::-1].encode('hex')

is about 5 times faster than the above loop, but may be a bit more confusing.
cdecker commented at 2:24 PM on December 30, 2016:

hexlify(unhexlify(s)[::-1]) is even quicker :wink:

droark commented at 7:30 PM on December 31, 2016:

Thanks! Will fix everything up.

droark commented at 10:09 PM on December 31, 2016:

Hello. After playing around with timeit(), it looks like the original code and hexlify(unhexlify(s)[::-1]).decode() are about the same speed. (s.decode('hex')[::-1].encode('hex') doesn't work, and I think it'd return incorrect results anyway.) Can you let me know how you benchmarked everything? As is, I think I'm going to leave this alone.

As for encode/decode, that has more to do with writing code that can run on Python 2 & 3; I'm also trying to maintain backwards compatibility. hexlify(unhexlify(s)[::-1]) works by itself but the result is b'0e0ac90a'. That won't work without decoding it.

cdecker commented at 10:06 AM on January 2, 2017:

Yeah, I noticed too late that the encode/decode hex variant will not work on Python3. We're probably micro-optimizing here anyway, given that the rest of the code is doing RPC calls :-)
droark force-pushed on Dec 31, 2016
droark commented at 11:51 PM on December 31, 2016: contributor

Fixup based on @cdecker feedback.
droark force-pushed on Jan 1, 2017
droark force-pushed on Jan 4, 2017

Allow linearization scripts to support hash byte reversal

Currently, the linearization scripts require input hashes to be in one endian form. Add support for byte reversal.

d5aa19813c

Make linearize scripts Python 3-compatible. 3c8f63ba7c
droark force-pushed on Jan 5, 2017
laanwj merged this on Jan 5, 2017
laanwj closed this on Jan 5, 2017
laanwj referenced this in commit 406f35d99d on Jan 5, 2017
laanwj commented at 10:20 AM on January 5, 2017: member

utACK 3c8f63b
droark deleted the branch on Jan 5, 2017
codablock referenced this in commit 882e2cdb9f on Jan 18, 2018
andvgal referenced this in commit 405ba5374c on Jan 6, 2019
CryptoCentric referenced this in commit 2c9fb1f8f6 on Feb 26, 2019
MarcoFalke locked this on Sep 8, 2021

Contributors

Labels

Linked (view graph)

#9304 Allow linearization scripts to support hashes with reversed bytes