Allow linearization scripts to support hashes with reversed bytes #9304

pull droark wants to merge 1 commits into bitcoin:master from droark:master changing 4 files +34 −3

droark commented at 3:39 AM on December 9, 2016: contributor

Currently, the linearization scripts require input hashes to be in ~~big endian form~~ one particular endian format. Add support for ~~little endian hashes~~ hashes with reversed bytes.
gmaxwell commented at 3:41 AM on December 9, 2016: contributor

But why?
droark commented at 3:43 AM on December 9, 2016: contributor

Note that, while the code is written to support the entire linearization process, the primary purpose is to help support other libraries. For example, python-bitcoinlib's getblock call requires a little endian hash; the library converts the endianness internally before making the actual RPC call. Being able to write a hash list with linearize-hashes.py that's compatible with such libraries is, IMO, a nice feature to have. I know one could just convert the hashes separate from linearize-hashes. That would slow down other programs/scripts, though, even if only just a little bit.
jonasschnelli added the label Scripts and tools on Dec 9, 2016
laanwj commented at 12:53 PM on December 9, 2016: member

Nit: Hash digests are just binary blobs. There is no little/big endian for them, please just call it "reversed" (or to be pedantic, byte-reversed).
droark commented at 12:51 AM on December 10, 2016: contributor

Nit: Hash digests are just binary blobs. There is no little/big endian for them, please just call it "reversed" (or to be pedantic, byte-reversed).

Can do. Thanks for bringing this up. It actually helped me straighten out some partially misunderstood ideas I had regarding the code.
droark force-pushed on Dec 10, 2016
droark force-pushed on Dec 10, 2016
droark force-pushed on Dec 10, 2016
droark commented at 7:00 AM on December 10, 2016: contributor

Updated and rebased.
droark force-pushed on Dec 10, 2016
droark force-pushed on Dec 13, 2016
droark force-pushed on Dec 13, 2016
droark force-pushed on Dec 13, 2016
droark force-pushed on Dec 13, 2016
petertodd commented at 10:53 AM on December 15, 2016: contributor

@droark Wait, why do you think python-bitcoinlib needs little-endian? There's both x() and lx() to convert a hex-encoded has to the binary representation in big and little-endian form.
droark commented at 8:22 PM on December 15, 2016: contributor

@petertodd - This line is the one I had in mind. If I take the linearization-hashes output as-is, convert it into bytes, and try to feed it into a getblock() call, python-bitcoinlib chokes. If I reverse the bytes, as seen in this PR, python-bitcoinlib is happy. Even if this is a bug in python-bitcoinlib, I still like the idea of giving users the option of reversing the bytes for their own purposes.
droark commented at 10:53 PM on December 16, 2016: contributor

Okay, I might close this and do a separate PR with this code and a small rewrite to the linearize scripts that I put together last night. It lets them run under Python 2 & 3. (Currently, they're Py2 only.) Is this recommended, or should I keep them separate?

Thanks.
droark renamed this:
~~Allow linearization scripts to support little endian hashes~~
Allow linearization scripts to support hashes with reversed bytes
on Dec 17, 2016

Allow linearization scripts to support little endian hashes

Currently, the linearization scripts require input hashes to be in big endian form. Add support for little endian hashes.

7cd990b68a

droark force-pushed on Dec 17, 2016
droark commented at 9:43 PM on December 17, 2016: contributor

Closing in favor of #9373. Can re-open if needed.
droark closed this on Dec 17, 2016
DrahtBot locked this on Sep 8, 2021

Contributors

Labels

Linked (view graph)

#9373 Linearize script update (hash byte reversal and Python 3 support)