util, ci: Hard code previous release tarball checksums #19813

1. 
   hebasto commented at 12:09 pm on August 26, 2020: member

   #19205 introduced signature verifying for the downloaded SHA256SUMS.asc. This approach is brittle and does not work in CI environment for many reasons:

   • #19812 (comment)
   • #19013 (review)

   This PR:

   • implements Sjors’ idea:

   Alternatively we might as well hard code the checksum for each tar.gz release in the source code, here.

   • is an alternative to 5a2c31e528e6bd60635096f233252f3c717f366d (#19013)

   • fixes #19812

   • updates v0.17.1 to v0.17.2

2. 
   hebasto commented at 12:10 pm on August 26, 2020: member

   cc @bliotti @Sjors @MarcoFalke
3. DrahtBot added the label Scripts and tools on Aug 26, 2020
4. 
   DrahtBot commented at 9:27 pm on August 26, 2020: member

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #19245 ([WIP DONOTMERGE] Replace boost::filesystem with std::filesystem (in c++17) by kiminuo)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

5. 
   practicalswift commented at 6:19 am on August 27, 2020: contributor

   Concept ACK: this is better than than fetching the checksums from the same host as the binaries :)
6. 
   laanwj commented at 11:56 am on August 27, 2020: member

   Does this need a mention in the release process?
7. 
   hebasto commented at 11:59 am on August 27, 2020: member

   @laanwj

   Does this need a mention in the release process?

   Here https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#after-3-or-more-people-have-gitian-built-and-their-results-match ?

8. 
   MarcoFalke commented at 1:08 pm on August 27, 2020: member

   This is only used by tests to download binaries, so no docs need to change. Maybe this should be moved to ./test/download_previous_releases.py or so?
9. 
   hebasto commented at 1:29 pm on August 27, 2020: member

   Maybe this should be moved to ./test/download_previous_releases.py or so?

   Out from contrib/devtools/?

10. 
    theStack commented at 11:03 am on August 28, 2020: member

    Concept ACK
11. 
    Sjors commented at 1:20 pm on August 28, 2020: member

    Concept ACK. Maybe put the shasums in a text file so they’re easier for other tools to use?
12. 
    laanwj commented at 3:47 pm on August 28, 2020: member

    Here https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#after-3-or-more-people-have-gitian-built-and-their-results-match ?

    I’m aware of that. I meant to add the hashes to the Python script.

    Concept ACK. Maybe put the shasums in a text file so they’re easier for other tools to use?

    -0 on this. I mean, the preferred way to validate downloaded binaries is still using GPG, and while I think this is a good shortcut to optimize CI, I’m not sure we should encourage its use in other tools.

13. 
    hebasto commented at 3:52 pm on August 28, 2020: member

    @laanwj

    Here https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#after-3-or-more-people-have-gitian-built-and-their-results-match ?

    I’m aware of that. I meant to add the hashes to the Python script.

    Sorry. It seems I did not understand correctly your initial suggestion:

    Does this need a mention in the release process?

    Mind rewording?

14. 
    MarcoFalke commented at 4:02 pm on August 28, 2020: member

    The tests will need to be adjusted anyway to use the new binaries, so the hashes can be added then (if needed)
15. 
    laanwj commented at 4:02 pm on August 28, 2020: member

    No, it’s okay, I agree with @MarcoFalke that that is not needed.

    This is only used by tests to download binaries, so no docs need to change.

16. scripted-diff: Move previous_release.py to test/get_previous_releases.py

    -BEGIN VERIFY SCRIPT-
    OLD=contrib/devtools/previous_release.py
    NEW=test/get_previous_releases.py
    sed -i "s|$OLD|$NEW|g" $(git grep -l $OLD)
    git mv $OLD $NEW
    -END VERIFY SCRIPT-

    bd897ce79f
17. util: Hard code previous release tarball checksums 0374e821bd
18. hebasto force-pushed on Aug 29, 2020
19. 
    hebasto commented at 8:34 am on August 29, 2020: member

    Updated c5affe70902ccf6ced959c8424cf7f35a3e5e6e1 -> 0374e821bd9e9498ce9c03aa8e5435870019978b (pr19813.01 -> pr19813.02, diff).

    Addressed @MarcoFalke’ comment:

    This is only used by tests to download binaries, so no docs need to change. Maybe this should be moved to ./test/download_previous_releases.py or so?

20. 
    MarcoFalke commented at 8:39 am on August 29, 2020: member

    cr ACK 0374e821bd9e9498ce9c03aa8e5435870019978b

    Hardcoding the hashes is equivalent from a trust perspective to hardcoding a gpg fingerprint and adding it to a keyring with ultimate trust.

    The hashes will need to be updated whenever a new previous release is added to the tests, but that seems acceptable.

21. 
    luke-jr commented at 4:58 pm on August 29, 2020: member

    I noticed gitian building old releases doesn’t match anymore, but since this only affects the downloader script (not the tests themselves), concept ACK.
22. 
    MarcoFalke commented at 5:10 pm on August 29, 2020: member

    I noticed gitian building old releases doesn’t match anymore

    guix fixes this (maybe)

23. 
    Sjors commented at 12:16 pm on August 30, 2020: member

    tACK 0374e821bd9e9498ce9c03aa8e5435870019978b
24. MarcoFalke merged this on Aug 31, 2020
25. MarcoFalke closed this on Aug 31, 2020

    ---

26. hebasto deleted the branch on Aug 31, 2020
27. sidhujag referenced this in commit dd6081216a on Aug 31, 2020
28. laanwj referenced this in commit 4053de04e2 on Sep 3, 2020
29. DrahtBot locked this on Feb 15, 2022

Contributors
hebasto DrahtBot practicalswift laanwj MarcoFalke theStack Sjors luke-jr

Labels
Scripts and tools