Verify Tree-SHA512s in merge commits, enforce sigs are not SHA1 #9880

This will validate Tree-SHA512s in merge commits if verify-commits.sh is called with --tree-checks as a second arg (eg verify-commits.sh upstream/master --tree-checks), as added by #9871. Note that I think we need to fix handling of symlinks in #9871 first.

Additionally, it disables sha1 for signature checking.

Due to some SHA1s in folks' selfsigs, the second part currently fails (@MarcoFalke, @sipa, @jonasschnelli.....)

While concept ACK, note that collision attacks on sha1 signatures - particularly in selfsigs - aren't as insecure as you might think. The reason is they're basically an attack on yourself. For instance, in the case of a selfsig, you already can add multiple subkeys to your PGP identity anyway, and you can do so without revealing that fact to the public, so there isn't a direct attack vector.

Similarly, given that a git commit signature is an expression of trust anyway, creating two commit objects so you can sign both with the same OpenPGP signature isn't a clear attack on anything; unlike the case where someone inserts a malicious file into the Git tree, that can only be done by the signer

Meanwhile, with second-preimage attacks that third-parties can do, almost every cryptographic hash function in history has proven immune. In fact, in my OpenTimestamps Python library I note that even MD5 is secure enough for timestamping (though I'm still not inclined to add support for it out of a healthy sense of caution!).

Now there are more complex cases where this stuff matters - timestamping is one of them depending on how it's implemented - so we probably should still merge this for the long-term. But we can certainly wait until people fix their keys.

Oh, and another reason to Concept ACK this: it wouldn't be surprising if a colliding self-sig or similar could be used to trigger a non-cryptographic bug in validation somewhere, given that you're violating the common assumption that hash functions don't collide.

@petertodd indeed, selfsig attacks aren't so interesting, but I'm not aware of another way to prevent sha1-based signatures. While probably also not particularly interesting attack vectors, its good belt-and-suspenders when signing "untrusted input" (in the form of the git tree root/sha512 tree root/commit messages/etc) to avoid sha1 inputs.

@TheBlueMatt Oh, yeah you make a good point in saying this is a case of "untrusted input", as the commit message to be made is semi-deterministic and under the control of the pull-requester.

One sec, I'll write up a comment for you. :)

• Rebased.
• Now enforces lack of symlinks in github-merge.py.
• Now enforces lack of symlinks in verify-commits.sh
• Always enforce non-SHA1-sigs even when --tree-checks is off
• Update trusted-sha256-root-commit until Marco updates his subkeys.

Concept ACK

Note that verify-commits on master is currently failing. I've tried it with this version and it doesn't pass either:

Using verify-commits data from /.../bitcoin/contrib/verify-commits.new
No parent of 58861ad91b499f6902c0f1c51bb745b91a275826 was signed with a trusted key!
Parents are:
commit f7ec7cfd38b543ba81ac7bed5b77f9a19739460b
Merge: 65d90f5 7ed143c
Author: MarcoFalke <falke.marco@gmail.com>
Date:   Thu Mar 2 22:30:12 2017 +0100

    Merge [#9359](/bitcoin-bitcoin/9359/): Add test for CWalletTx::GetImmatureCredit() returning stale values.
    
    7ed143c Add test for CWalletTx::GetImmatureCredit() returning stale values. (Russell Yanofsky)
    
    Tree-SHA512: c95088ed6dfc5a0774ddaa2fe14ac0a9ebd830922a4d77100ec3d51fdeb6df40ad97de4f2ea970ed0f4122dcc0022ee1d43ab3c7188becd7f90c1c6af0ed39b7
commit 64854666f57d9f94269000f10897858e8c60d99f
Author: Wladimir J. van der Laan <laanwj@gmail.com>
Date:   Thu Mar 2 14:49:11 2017 +0100

    test: Report InitBlockIndex result
    
    If InitBlockIndex fails, then it will segfault later. Same for the later
    ActivateBestChain. BOOST_REQUIRE the result, so that an error will be
    reported and the test case aborted.

• Updated the trusted-sha256-root-commit for Jonas' sha2 commit.
• Allow any subkey that is properly signed, instead of only listing specific subkeys (this should fix verify-commits on master).

utACK 6b2a4a3

We may want to backport at least the "Allow any subkey in verify-commits" commit for 0.14 so that verify-commits passes on release

Pushed a new commit which will fix the checks on the 0.14 branch.

Verified the new/changed keys (same as in #9920). Tested ACK bbd757940bcb0628df6f7a5bd1fb348cf2290502

Going to merge this to fix travis. The shasum nit can be fixed later.