test: Openssl removed ripemd160 #23710

MarcoFalke commented at 4:28 PM on December 8, 2021: member

Openssl removed ripemd160, see https://github.com/openssl/openssl/issues/16994

Thus, our tests fail. See below for excerpt.

Not sure how to fix this, so leaving an issue here.

Running Unit Tests for Test Framework Modules
..........
----------------------------------------------------------------------
Ran 10 tests in 0.812s

OK
4/225 - feature_block.py failed, Duration: 0 s                                                                                                                                          

stdout:


stderr:
Traceback (most recent call last):
  File "/usr/lib/python3.9/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] initialization error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/feature_block.py", line 54, in <module>
    from data import invalid_txs
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/data/invalid_txs.py", line 57, in <module>
    basic_p2sh = script_to_p2sh_script(CScript([OP_0]))
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/script_util.py", line 74, in script_to_p2sh_script
    return scripthash_to_p2sh_script(hash160(script))
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/script.py", line 35, in hash160
    return hashlib.new('ripemd160', sha256(s)).digest()
  File "/usr/lib/python3.9/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/usr/lib/python3.9/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160



TEST                                               | STATUS    | DURATION

wallet_backup.py --descriptors                     | ✓ Passed  | 109 s
wallet_hd.py --descriptors                         | ✓ Passed  | 92 s
wallet_hd.py --legacy-wallet                       | ✓ Passed  | 125 s
feature_block.py                                   | ✖ Failed  | 0 s

ALL                                                | ✖ Failed  | 326 s (accumulated) 
Runtime: 125 s

MarcoFalke added the label Bug on Dec 8, 2021

MarcoFalke added the label Tests on Dec 8, 2021

MarcoFalke added this to the milestone 23.0 on Dec 8, 2021

MarcoFalke commented at 4:41 PM on December 8, 2021: member

achow101 commented at 6:08 PM on December 8, 2021: member

One potential solution that I am investigating for HWI is to overwrite hashlib.new with our own implementation of it. This can then identify if the hash function is ripemd160 and return a constructor to our own implementation of ripemd160 rather than delegating to openssl. This function can fallback to the actual hashlib.new too. Since imports are global per interpreter session, modifying the imported library in one module will affect all other subsequent uses of that library.

Here is an example of how this overwrite operates:

First a new module, I calling it bitcoinhashes.py:

import hashlib

hashlib_new = hashlib.new

def our_hashlib_new(name, data=b"", **kwargs):
    print("OUR hashlib.new")
    return hashlib_new(name, data, **kwargs)

hashlib.new = our_hashlib_new

Then observe what happens when this overwritten hashlib.new is used:

Python 3.9.7 (default, Oct 25 2021, 18:20:41) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> import bitcoinhashes
>>> hashlib.new("sha256")
OUR hashlib.new
<sha256 _hashlib.HASH object @ 0x7f0367fafc70>
>>>

The ripemd160 implementation would have to be in python though. I don't think this particularly matters, but it will be way slower than openssl's, or even our own c++ implementation.

This approach is probably more suitable for the test framework rather than a library like HWI as there are no downstream consumers where changing the behavior of hashlib might be surprising.

MarcoFalke commented at 6:29 PM on December 8, 2021: member

ripemd160 is only used in one place, so I think we can just inline ripemd160 python code for that in that place.

The issue is that no python code seems to exists, since everyone was falling back to openssl?

I don't really understand how to un-deprecate ripemd160 in openssl, but if there is an easy hack (for example via a config file), then that might be a good enough solution for the short term, also making our CI green again.

sipa commented at 6:31 PM on December 8, 2021: member

I can write a pure-python ripemd160 if that's useful.

achow101 commented at 6:33 PM on December 8, 2021: member

I can write a pure-python ripemd160 if that's useful.

It would (probably) be useful for HWI.

SomberNight commented at 12:03 AM on December 9, 2021: contributor

The issue is that no python code seems to exists, since everyone was falling back to openssl?

Electrum has a fallback pure-python implementation.

See https://github.com/spesmilo/electrum/blob/0df05dd914c823acae1828cad3b20bdeb13150e9/electrum/crypto.py#L312-L324 https://github.com/spesmilo/electrum/blob/0df05dd914c823acae1828cad3b20bdeb13150e9/electrum/ripemd.py

Besides the license mentioned there, I can't find where it is from exactly. It was added in https://github.com/spesmilo/electrum/commit/ad6c0ab2736edd2d7d2f0871a6b6a4d716441ace.

MarcoFalke closed this on Dec 9, 2021

sidhujag referenced this in commit 8fe77bcc20 on Dec 10, 2021

knst referenced this in commit 2f38f567b9 on May 20, 2022

PastaPastaPasta referenced this in commit e4dbd22532 on May 29, 2022

UdjinM6 referenced this in commit 1233977a0f on May 30, 2022

PastaPastaPasta referenced this in commit 1babc5abff on May 30, 2022

petertodd referenced this in commit de5c6a058c on Jul 29, 2022

niftynei referenced this in commit 2101ecf343 on Sep 8, 2022

niftynei referenced this in commit e87a43d112 on Sep 8, 2022

niftynei referenced this in commit a1093f5b11 on Sep 9, 2022

gimre-xymcity commented at 3:22 PM on September 22, 2022: none

I don't think it's worth opening new issue.

We've had similar issue, we've stripped pycryptodome from everything, leaving just ripemd: https://github.com/symbol/ripemd https://pypi.org/project/ripemd-hash/

Fuzzbawls referenced this in commit 0d779aa53d on Apr 4, 2023

mo-bay referenced this in commit 5218ea5774 on Apr 10, 2023

bitcoin locked this on Sep 22, 2023