libbitcoinconsensus flag issues #5779

I've been thinking lately about how a dev may use libbitcoinconsensus to add verification to his program. Unfortunately, there are several current issues that (I believe) seriously hamper real-world usage.

For example, bitcoinconsensus_SCRIPT_FLAGS_VERIFY_DERSIG was just recently added as a flag for the lib. For the sake of discussion, let's imagine that this flag was not present in the 0.10 release at all, but it is present in 0.11.

A dev is working against the 0.11 version. He releases a program that loads libbitcoinconsensus.so and uses it to verify. It specifies the bitcoinconsensus_SCRIPT_FLAGS_VERIFY_DERSIG flag and compiles fine, and he distributes it to users. Someone with the 0.10 version of the lib runs the program, which silently verifies without taking the new flag into account. Ouch.

The obvious guard against that would be for the lib to first verify that it understands the flags passed in. If it doesn't, then it bails out with an error.

But there's a problem with that as well... the lib currently understands more flags than it exposes externally. By adding a check to ensure that the user has only used exposed flags, we'd be artificially limiting it. In this case, adding that check would break verification for the user that would've otherwise worked.

I've run in circles trying to figure out how to deal with these issues, so any suggestions would be appreciated.

1. If libbitcoinconsensus is passed a flag that is not exposed externally, should it immediately fail?
2. If 'no' to 1., if it is passed a flag that is also not understood internally should it immediately fail?
3. Despite objections about purity, I believe that ultimately devs will eventually want a bitcoinconsensus_script_isstandard() function. This would likely require having the other flags available as well, so it may change answers to 1. and 2.

I believe we should have per-layer flags.

That means making the flags exported by libconsensus separate from the ones in the internal implementations, and libconsensus translates them as necessary. That both means implementation detals don't leak out (visibly or invisibly).

I also don't think libconsensus should be implementing any standardness checks. They're policy, and not part of libconsensus' goal (which is to be policy neutral, and just provide the one thing that is too risky to reimplement).

Agree with @sipa

@sipa let's take the scenario I laid out as an example. Say there are 2 versions of libbitcoinconsensus in the wild, one that includes VERIFY_DERSIG externally and one that doesn't (regardless of whether it's a 1:1 flag mapping or not). Dev builds a program against the new version with the flag and releases it. user runs against the older version of the lib without the flag.

If I'm understanding correctly, it sounds as though you want an actual mapping function that's hit before feeding the flags into VerifyScript(), so that the passed-in flags are never used directly?

In this case, what would happen for the user? The map function sees a VERIFY_DERSIG which is not exposed externally. Does it simply immediately fail?

That's why there is a version number, no, to see whether the library is what they expect to be?

A possibility is providing a function which takes a bunch of flags, and tells you whether they're all supported. Using an unknown flag is undefined.

Yes, the version number could be used I suppose.

Ultimately what scares me is the possibility of a flag being silently eaten and ignored. A flag checker as you suggest would certainly help, though.

If any unrecongised flags are passed, maybe we should abort()

The library validates that flags should be exposed after this commit: https://github.com/bitcoin/bitcoin/commit/5ca8ef299a08aae91d5061750533694b58d810b2