guix: Overhaul how guix-{attest,verify} works and hierarchy

dongcarl commented at 7:28 pm on June 7, 2021: member

Based on: #22075 Code reviewers: I recommend reading the new guix-{attest,verify} files instead of trying to read the diff

The following changes resolve many usability improvements which were pointed out to me:

Some maintainers like to extract their “uncodesigned tarball” inside the output/ directory, resulting in the older guix-attest mistakenly attesting to the extracted contents
Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of output/, namely output/*/SHA256SUMS.part, in order to perform a signature (you may need to specify an $OUTDIR_BASE env var)
An all.SHA256SUMS file should be usable as the base SHA256SUMS in bitcoin core torrents and on the release server.

For those who sign on an separate machine than the one you do builds on, the following steps will work:

env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest
Copy /home/achow101/guix.sigs/<tag>/achow101 (which does not yet have signatures) to signing machine

Sign the SHA256SUMS files:

0for i in "<path-to-achow101>/*.SHA256SUMS"; do
1    gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
2done

Upload <path-to-achow101> (now with signatures) to guix.sigs

After this change, output directories will now include a SHA256SUMS.part fragment, created immediately after a successful build:

0output
1└── x86_64-w64-mingw32
2    ├── bitcoin-4e069f7589da-win64-debug.zip
3    ├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
4    ├── bitcoin-4e069f7589da-win64.zip
5    ├── bitcoin-4e069f7589da-win-unsigned.tar.gz
6    └── SHA256SUMS.part

These SHA256SUMS.part fragments look something like:

03ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825  dist-archive/bitcoin-4e069f7589da.tar.gz
1def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b  x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
2643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e  x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
3a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288  x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
4fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c  x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip

Meaning that they are valid SHA256SUMS files when sha256sum --check’d at the guix-build-*/output directory level

When guix-attest is invoked, these SHA256SUMS.part files are combined and sorted (by -k2, LC_ALL=C) to create:

noncodesigned.SHA256SUMS for a manifest of all non-codesigned outputs, and
all.SHA256SUMS for a manifest of all outputs including non-codesigned outputs

Then both files are signed, resulting in the following guix.sigs hierarchy:

04e069f7589da/
1└── dongcarl
2    ├── all.SHA256SUMS
3    ├── all.SHA256SUMS.asc
4    ├── noncodesigned.SHA256SUMS
5    └── noncodesigned.SHA256SUMS.asc

DrahtBot added the label Scripts and tools on Jun 7, 2021

fanquake commented at 3:17 am on June 9, 2021: member

@dongcarl can you rebase now that #22075 is merged.

Make SHA256SUMS fragment right after build 28a9c9b839

Rewrite guix-{attest,verify} for new hier 4cc35daed5

dongcarl force-pushed on Jun 9, 2021

dongcarl commented at 4:25 pm on June 9, 2021: member

Pushed d90ca1e8ffc0ede1da7f47cb2da864e2d29fddb0…4cc35daed557f38b080360a89036b2e97a6f78c2

Rebased after merge of #22075
Added section in OP about signing on another computer

fanquake added this to the "Next (Not based on any other PRs)" column in a project

fanquake commented at 4:23 am on June 10, 2021: member

Concept ACK - Haven’t really looked at the changes, but I ran though this and tried signing on a separate machine, which is my usual workflow. Guix sigs for 4cc35daed557f38b080360a89036b2e97a6f78c2 here: https://github.com/bitcoin-core/guix.sigs/pull/23.

laanwj added this to the milestone 22.0 on Jun 10, 2021

achow101 commented at 7:08 pm on June 11, 2021: member

Is this still a draft?

If you do a build and attest with some hosts, and then another build and attest with more hosts, the second attest will not update either of the .SHA256SUMS file. I think it should because there are some cases where a builder may build some of the hosts at a different time, e.g. if they acquire the macOS SDK after building and attesting.

Otherwise I like the changes done here.

dongcarl marked this as ready for review on Jun 14, 2021

laanwj commented at 5:57 pm on June 14, 2021: member

Concept ACK, changes sound great to me. Will test.

achow101 commented at 7:46 pm on June 14, 2021: member

ACK 4cc35daed557f38b080360a89036b2e97a6f78c2

Reviewed code and did a build, attest, and verify to test.

guix-attest: Error out if SHA256SUMS is unexpected e2c40a4ed5

dongcarl commented at 9:11 pm on June 14, 2021: member

Pushed 4cc35daed5..e2c40a4ed5

Updated to address the possible user flow @achow101 mentioned here: #22182 (comment)

Specifically: before using an existing noncodesigned.SHA256SUMS file, we will check if it is up-to-date (if it exactly the same as merging all the noncodesigned SHA256SUMS.parts). If it is, it will be used, otherwise, we will error out with a helpful error message prompting the user to deal with the (likely faulty) noncodesigned.SHA256SUMS file.

achow101 commented at 5:55 pm on June 15, 2021: member

ACK e2c40a4ed5272d72fea997bd936fba28bb753226

fanquake commented at 5:10 am on June 16, 2021: member

ping @Emzy @hebasto as you have both previously tested Guix building.

hebasto commented at 1:46 am on June 17, 2021: member

Concept ACK. Testing…

hebasto commented at 2:34 am on June 17, 2021: member

Not related to this PR, so for future follow ups: in https://github.com/bitcoin/bitcoin/blob/6bc1eca01b2f88e081e71b783b3d45287700f8a5/contrib/guix/guix-attest#L21

xargs and find are not used in this script. But diff and sort are missed.

hebasto approved

hebasto commented at 4:07 am on June 17, 2021: member

ACK e2c40a4ed5272d72fea997bd936fba28bb753226, tested on Linux Mint 20.1 (x86_64) with and w/o NO_SIGN=1. Changes in contrib/guix/libexec/codesign.sh and contrib/guix/guix-verify are reviewed only.

See https://github.com/bitcoin-core/guix.sigs/pull/24

fanquake commented at 5:08 am on June 17, 2021: member

Ran through building and using NO_SIGN again and produced new signatures for e2c40a4ed5272d72fea997bd936fba28bb753226 which match @hebasto.

fanquake merged this on Jun 17, 2021

fanquake closed this on Jun 17, 2021

fanquake moved this from the "Next (Not based on any other PRs)" to the "Done" column in a project

Emzy commented at 10:00 am on June 18, 2021: contributor

tested ACK Run through guix building. Signing device was plugged intro the build machine, so only one signature improved the usability.

sidhujag referenced this in commit 1837605b96 on Jun 18, 2021

fanquake referenced this in commit 2b5563bb1e on Jul 23, 2021

sidhujag referenced this in commit 2c6f446670 on Jul 23, 2021

gwillen referenced this in commit 2fa63315e5 on Jun 1, 2022

DrahtBot locked this on Aug 18, 2022

guix: Overhaul how guix-{attest,verify} works and hierarchy #22182