gitian: Use custom MacOS code signing tool #20880

achow101 commented at 8:58 PM on January 7, 2021: member

The MacOS code signing issues that were encountered during the 0.21.0 release cycle have shown that it is necessary for us to use a code signing tool for which the source code is available and modifiable by us. Given that there appears to not be such a tool available, I have written such a tool, signapple, that we can use. This tool is able to create a valid MacOS code signature, detach it in a way that we were doing previously, and attach it to the unsigned binary. This tool can also verify that the signature is correct.

This PR implements the usage of that tool in the gitian build for the code signed MacOS binary. The code signer will use this tool to create the detached signature. Gitian builders will use this tool to apply the detached signature. The gitian-osx-signer.yml descriptor has been modified to install this tool so that the detached-sig-apply.sh script can use it. Additionally, the codesign_allocate and pagestuff tools are no longer necessary so they are no longer added to the tarball used in code signing. Lastly, both the detached-sig-create.sh and detached-sig-apply.sh scripts are made to be significantly less complex and to not do unexpected things such as unpacking an already unpacked tarball.

The detached code signature that signapple creates is almost identical to that which we were previously creating. The only difference is that the cpu architecture name is included in the extension (e.g. we have bitcoin-qt.x86_64sign instead of bitcoin-qt.sign). This was done in order to support signing universal binaries which we may want to do in the future. However signapple can still apply existing code signatures as it will accept the .sign extension. If it is desired, it can be modified to produce signatures with just the .sign extension. However I do not think it is necessary to maintain compatibility with the old process.

gitian: install signapple in gitian-osx-signer.yml 42bb1ea363

gitian: use signapple to apply the MacOS code signature 95b06d2185

gitian: use signapple to create the MacOS code signature f55eed2514

gitian: Remove codesign_allocate and pagestuff from MacOS build 2c403279e2

benthecarman commented at 9:14 PM on January 7, 2021: contributor

Concept ACK

DrahtBot added the label Build system on Jan 7, 2021

DrahtBot added the label Scripts and tools on Jan 7, 2021

laanwj commented at 9:52 PM on January 7, 2021: member

Concept ACK, nice!

DrahtBot commented at 12:11 AM on January 8, 2021: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#20630 (Allow providing local signatures in gitian osx signer by jonasschnelli)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

jonasschnelli commented at 8:57 AM on January 8, 2021: contributor

Impressive work! Well done @achow101. I used this tool to sign 0.21.0rc5 (with external codesign_allocate). Concept ACK (will test soon).

jonatack commented at 9:23 AM on January 8, 2021: member

Concept ACK -- nice work!

MarcoFalke added the label Needs gitian build on Jan 12, 2021

MarcoFalke added the label Needs Guix build on Jan 12, 2021

DrahtBot commented at 7:49 AM on January 13, 2021: member

Guix builds

File	commit 7838db141b76fa72fcb9c3a3081859f9db102039<br>(master)	commit 3343f320287d6b520b46ed062e26dde51cb329d0<br>(master and this pull)
*-aarch64-linux-gnu-debug.tar.gz	`8093dc2cc77f5e62...`	`8816ed48879ae85c...`
*-aarch64-linux-gnu.tar.gz	`96c6dc48f1df87f0...`	`1da08e1d691ad3ae...`
*-arm-linux-gnueabihf-debug.tar.gz	`820015874934cc21...`	`43e8f58df97c4b0a...`
*-arm-linux-gnueabihf.tar.gz	`d4bd4a8809b37004...`	`0e0be7f0e826b3e6...`
*-riscv64-linux-gnu-debug.tar.gz	`c6fa94dc565668e1...`	`bb6a6f94f75d70a8...`
*-riscv64-linux-gnu.tar.gz	`96f4ac5d990f244d...`	`d54c423638111834...`
*-win-unsigned.tar.gz	`760ddb798cec381d...`	`1f102058f2d1c17e...`
*-win64-debug.zip	`67a033edbdc22d99...`	`51a89a6b8df0baed...`
*-win64-setup-unsigned.exe	`0dea7c57960acdc4...`	`304123eebfb4fc9a...`
*-win64.zip	`bbd519ae55978259...`	`e8034bf11676bb15...`
*-x86_64-linux-gnu-debug.tar.gz	`6ecaa1e63630cc45...`	`5a076f091e0997b1...`
*-x86_64-linux-gnu.tar.gz	`87c8fcc8cb888efd...`	`1626bb21ddc9cd4c...`
*.tar.gz	`69d589c9e9b4152c...`	`c11f84cdf66d2f6f...`
guix_build.log	`13410f62f1e3176b...`	`82caa52f891abeb3...`
guix_build.log.diff		`45becbccac40d0e0...`

DrahtBot removed the label Needs Guix build on Jan 13, 2021

DrahtBot commented at 11:45 AM on January 13, 2021: member

🕵️ @sipa @practicalswift have been requested to review this pull request as specified in the REVIEWERS file.

MarcoFalke removed the label Needs gitian build on Jan 14, 2021

MarcoFalke added the label Needs gitian build on Jan 14, 2021

laanwj commented at 10:14 AM on January 14, 2021: member

Code review ACK 2c403279e2f0f7c8c27c56d4e7b0573c59571f0a

Also cursory code review ACK on the signer tool achow101/signapple@c7e73aa27a7615ac9506559173f787e2906b25eb—I can't vouch it does exactly the same as Apple's tool, but it definitely follows a similar structure as their code, and modifies the same parts of the binary. It most notably doesn't change any code so it won't introduce a back door itself.

Additionally, the codesign_allocate and pagestuff tools are no longer necessary so they are no longer added to the tarball used in code signing.

+24 −67 Good riddance. Nice cleanups!

DrahtBot commented at 10:22 AM on January 14, 2021: member

Gitian builds

File	commit 8ffaf5c2f5aaa31c6ee6075689c5dbb89a7d16c6<br>(master)	commit 4044d10f8ed7baff70542ee8b2b3373dcdac3a99<br>(master and this pull)
*-osx-unsigned.dmg	`d6287425560ba4c8...`	`8b5537852524c54d...`
*-osx64.tar.gz	`d2636c99f4f6ae44...`	`32eb08228f92c5d6...`
*.tar.gz	`ce4d207f0203c991...`	`12def18cbd5b578f...`
bitcoin-core-osx-22-res.yml	`f565610bc956d740...`	`0cc5413fe2502acf...`
linux-build.log	`5ae7d55d43bde935...`	`141ee861fff8c272...`
osx-build.log	`14b6de642b3613a1...`	`e5dd5bebc9d57036...`
win-build.log	`7c2f349972850c1b...`	`748a218c0ca36b13...`
bitcoin-core-osx-22-res.yml.diff		`d54e47a630f5db9a...`
linux-build.log.diff		`e8972241a1433c14...`
osx-build.log.diff		`7939aa708a2ee391...`
win-build.log.diff		`f188f742572cd1a7...`

DrahtBot removed the label Needs gitian build on Jan 14, 2021

laanwj merged this on Jan 18, 2021

laanwj closed this on Jan 18, 2021

MarcoFalke commented at 1:22 PM on January 20, 2021: member

Does this need backport?

sidhujag referenced this in commit 2065b58336 on Jan 20, 2021

achow101 referenced this in commit 65ce833042 on Jun 8, 2021

achow101 referenced this in commit 2f33e339a8 on Jun 8, 2021

achow101 referenced this in commit 27d691b6b5 on Jun 8, 2021

achow101 referenced this in commit 5313d6aed2 on Jun 8, 2021

achow101 referenced this in commit a2650f6495 on Jun 8, 2021

achow101 referenced this in commit f8344856d6 on Jun 8, 2021

achow101 referenced this in commit 9c7c0e6401 on Jun 8, 2021

achow101 referenced this in commit a17041e155 on Jun 8, 2021

MarcoFalke referenced this in commit 926f76cb20 on Jun 19, 2021

MarcoFalke referenced this in commit 466acb3fbc on Jun 19, 2021

in contrib/gitian-descriptors/gitian-osx-signer.yml:40 in 2c403279e2

  34 | @@ -31,11 +35,19 @@ script: |
  35 |      chmod +x ${WRAP_DIR}/${prog}
  36 |    done
  37 |  
  38 | -  UNSIGNED=bitcoin-osx-unsigned.tar.gz
  39 | +  # Install signapple
  40 | +  cd signapple
  41 | +  python3 -m pip install -U pip setuptools

hebasto commented at 8:08 PM on August 16, 2021:

When using LXC (LXC 4.0.6 on Ubuntu Focal) this line fails with the following error:

...
+ python3 -m pip install -U pip setuptools
Collecting pip
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f82adc3a860>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/pip/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f82adc3a978>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/pip/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f82adc3aa58>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/pip/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f82adc3ab38>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/pip/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f82adc3abe0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/pip/
  Could not find a version that satisfies the requirement pip (from versions: )
No matching distribution found for pip
...

UdjinM6 referenced this in commit c57de2596a on Aug 17, 2022

DrahtBot locked this on Aug 18, 2022