libbitcoinconsensus.dylib in MacOS releases is broken #21794

The consensus library that is released for MacOS (libbitcoinconsensus.dylib) does not seem to contain the necessary functions.

Just the file size (~400KB) of the lib looks to be far too small to fit in the required functions. E.g. the consensus lib for linux is over 4MB.

Moreover, running nm -gU returns only:

00000000000633c0 S __ZTINSt3__115basic_stringbufIcNS_11char_traitsIcEENS_9allocatorIcEEEE
0000000000063328 S __ZTINSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE
0000000000063570 S __ZTINSt3__119basic_istringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE
0000000000063498 S __ZTINSt3__119basic_ostringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE
0000000000052576 S __ZTSNSt3__115basic_stringbufIcNS_11char_traitsIcEENS_9allocatorIcEEEE
00000000000525b8 S __ZTSNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE
0000000000052530 S __ZTSNSt3__119basic_istringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE
00000000000524d0 S __ZTSNSt3__119basic_ostringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE
0000000000008780 T _secp256k1_context_clone
0000000000008660 T _secp256k1_context_create
0000000000008880 T _secp256k1_context_destroy
00000000000636a8 D _secp256k1_context_no_precomp
0000000000008700 T _secp256k1_context_preallocated_clone
00000000000082e0 T _secp256k1_context_preallocated_clone_size
0000000000008300 T _secp256k1_context_preallocated_create
0000000000008830 T _secp256k1_context_preallocated_destroy
00000000000082a0 T _secp256k1_context_preallocated_size
0000000000011600 T _secp256k1_context_randomize
0000000000008950 T _secp256k1_context_set_error_callback
00000000000088c0 T _secp256k1_context_set_illegal_callback
000000000000ec30 T _secp256k1_ec_privkey_negate
000000000000f190 T _secp256k1_ec_privkey_tweak_add
00000000000105b0 T _secp256k1_ec_privkey_tweak_mul
0000000000012bb0 T _secp256k1_ec_pubkey_combine
000000000000dbb0 T _secp256k1_ec_pubkey_create
000000000000ec40 T _secp256k1_ec_pubkey_negate
0000000000008ac0 T _secp256k1_ec_pubkey_parse
0000000000009670 T _secp256k1_ec_pubkey_serialize
000000000000f1a0 T _secp256k1_ec_pubkey_tweak_add
00000000000105c0 T _secp256k1_ec_pubkey_tweak_mul
000000000000ea10 T _secp256k1_ec_seckey_negate
000000000000eed0 T _secp256k1_ec_seckey_tweak_add
00000000000101a0 T _secp256k1_ec_seckey_tweak_mul
000000000000db30 T _secp256k1_ec_seckey_verify
0000000000016b40 T _secp256k1_ecdsa_recover
0000000000016980 T _secp256k1_ecdsa_recoverable_signature_convert
0000000000016540 T _secp256k1_ecdsa_recoverable_signature_parse_compact
00000000000166a0 T _secp256k1_ecdsa_recoverable_signature_serialize_compact
000000000000c1c0 T _secp256k1_ecdsa_sign
0000000000016a40 T _secp256k1_ecdsa_sign_recoverable
000000000000a890 T _secp256k1_ecdsa_signature_normalize
0000000000009f10 T _secp256k1_ecdsa_signature_parse_compact
0000000000009c90 T _secp256k1_ecdsa_signature_parse_der
000000000000a5d0 T _secp256k1_ecdsa_signature_serialize_compact
000000000000a110 T _secp256k1_ecdsa_signature_serialize_der
000000000000aa30 T _secp256k1_ecdsa_verify
00000000000209d0 T _secp256k1_keypair_create
00000000000219c0 T _secp256k1_keypair_pub
0000000000021a90 T _secp256k1_keypair_xonly_pub
0000000000021f80 T _secp256k1_keypair_xonly_tweak_add
0000000000063698 S _secp256k1_nonce_function_bip340
0000000000063690 S _secp256k1_nonce_function_default
0000000000063688 S _secp256k1_nonce_function_rfc6979
0000000000022aa0 T _secp256k1_schnorrsig_sign
0000000000024900 T _secp256k1_schnorrsig_verify
00000000000089e0 T _secp256k1_scratch_space_create
0000000000008a50 T _secp256k1_scratch_space_destroy
000000000001fe20 T _secp256k1_xonly_pubkey_from_pubkey
00000000000173f0 T _secp256k1_xonly_pubkey_parse
000000000001fbf0 T _secp256k1_xonly_pubkey_serialize
0000000000020080 T _secp256k1_xonly_pubkey_tweak_add
0000000000020310 T _secp256k1_xonly_pubkey_tweak_add_check

There is no bitcoinconsensus_verify_script, bitcoinconsensus_verify_script_with_amount, or bitcoinconsensus_version.

It looks more like secp256k1 alone, rather the the consensus lib.

I have checked only the recent three releases: 0.19.1, 0.20.1 and 0.21.0 All downloaded as a .tar.gz file from bitcoincore.org - eg: https://bitcoincore.org/bin/bitcoin-core-0.21.0/bitcoin-0.21.0-osx64.tar.gz

Do you see the issue when building master? This should have been fixed by #19522, which missed making it into 0.21.0.

I don't have the tools to build ATM. Will gladly test in the next release.

Nope.

The lib from the 0.21.1 release is still only ~400KB and still does not seem to contain the necessary functions.

The lib from the 0.21.1 release is still only ~400KB and still does not seem to contain the necessary functions.

That is expected. #19522 has been merged into the master branch, which will become 22.0. That change is simple enough that it could potentially be backported for a 0.21.2 release.

OK. I built the library from master and it was good, although I am not convinced that my building environment is identical to the one you use for releases.

But I noticed that the lib's glue code has not been updated with the Taproot flag. Calling bitcoinconsensus_verify_script_with_amount with SCRIPT_VERIFY_TAPROOT flag returns bitcoinconsensus_ERR_INVALID_FLAGS. I'm guessing it was overlooked when taproot was added to the rest of the code.

Moreover, it used to be possible to call the verify_script function with the soft flags (e.g. SCRIPT_VERIFY_LOW_S) as well. It was useful if one wanted to check a transaction before placing it in a mempool, to use the same rules as bitcoin core. But now I see that the consensus lib only allows the actually enforced (hard) consensus flags and trying any of the other ones throws bitcoinconsensus_ERR_INVALID_FLAGS . I'm guessing someone thought it would be an improvement, but it is not backward compatible and IMHO it limits the functionality of the library while not giving any real benefits.

@piotrnar Fyi, #21158 implements the necessary changes for taproot

I'm going to close this, as the macOS libbitcoinconsensus.dylib is fixed as of 22.0, and I don't think we are going to backport the fix (#19522) into the 0.21.x branch. If you've got further issues with the library (flags), or questions about taproot support, I'd suggest opening a new issue, or following up in #21133 or #21158.