More test cases for transaction scripts #4560

These are some edge cases that I have detected using QuickCheck in the Haskoin project which are not covered by the existing test suite.

In script_valid.json:

1. All bytes of a boolean constant are significant, not only the last one. I had a dumb bug where I only checked the last byte, but it passed the test suite.
2. Also casting the stack value to a 64 bit int works most of the time, unless the value overflows to 0. Added a test for that.
3. SIZE does not consume the argument. Many commands consume the input argument, some don't, I think generally there could be more test coverage of that. Maybe build a test that puts a bunch of constants on the stack, execute a bunch of operations (for which the test makes sense) and test the value and stack depth in the end.

In script_invalid.json:

1. BOOLAND + BOOLOR have a slightly different conception of boolean values: these functions cast inputs to int64 and compare against zero, so they must fail for five-byte stack values.
2. Limits for CHECKMULTISIG were not tested: (nKeys < 20) and (nSigs <= nKeys)

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/p4560_d2d9dc063f8da87e02e588064f8ea9253e201907/ for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

Stuff like the OP_SIZE makes me wonder if we should just add something that checks the exact stack returned by script evaluation. As this is consensus critical stuff it may even be worthwhile to just have a step-by-step stack trace that is compared exactly.

But yeah, good tests! ACK

#3901 sounds really useful for implementing such tracing.

@sipa Yes, but we could also do it with a lot less risk with a simple callback that if set took the state of execution as arguments. That'd keep the consensus-critical changes to an absolute minimum. #3901 would be safer to implement after we had testcases using the simpler callback mechanism too.

@petertodd Good idea

Checking the the stack is also be possible by using EvalScript instead VerifyScript. In fact I found some of the bugs by comparing the stack after running EvalScript.