The upstream function secp256k1_ge_set_xo_var() in src/group_impl.h returns an
int signalling an error condition, but it is not marked with
SECP256K1_WARN_UNUSED_RESULT.
Thus, when compiling the Frost module, no warning was generated when
deserialize_frost_signature() had no check covering the case that
secp256k1_ge_set_xo_var() could fail.
The GCC static analyzer found two potential execution paths in
secp256k1_frost_verify() that would lead to undefined behaviour. For brevity,
the issues are not reported here, but only in the Github PR associated with
this commit (https://github.com/bancaditalia/secp256k1-frost/pull/5).
As of today (2023-05-29), in bitcoin-core secp256k1, secp256k1_ge_set_xo_var()
is still not marked SECP256K1_WARN_UNUSED_RESULT, see:
https://github.com/bitcoin-core/secp256k1/blob/908e02d596b66203788e8945b1f9c93ff28a4536/src/group_impl.h#L280
It may make sense to propose upstream to add that annotation.
In order to replicate the issues fixed by this commit, compile with:
./configure SECP_CFLAGS="-fanalyzer -fanalyzer-transitivity" --disable-tests --disable-exhaustive-tests --disable-benchmark --enable-experimental --enable-module-frost