[22.x] guix: ignore additional failing certvalidator test

fanquake commented at 1:10 pm on January 31, 2022: member

Backports https://github.com/bitcoin/bitcoin/commit/85885919656a3f606f3d7f208378aabe95f3f62d from #24057 so that from-scratch Guix builds for the Darwin host aren’t broken due to a (very recently) expired certificate causing one of the python-certvalidator tests to fail. Kept separate from #23276 because that hasn’t gotten review attention, and I don’t think we should leave 22.x Darwin Guix builds broken for any longer than we have to.

Fixes #24110.

 0======================================================================
 1ERROR: test_revocation_mode_soft (tests.test_validate.ValidateTests)
 2----------------------------------------------------------------------
 3Traceback (most recent call last):
 4  File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/test_validate.py", line 85, in test_revocation_mode_soft
 5    validate_path(context, path)
 6  File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/../certvalidator/validate.py", line 50, in validate_path
 7    return _validate_path(validation_context, path)
 8  File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/../certvalidator/validate.py", line 358, in _validate_path
 9    raise PathValidationError(pretty_message(
10certvalidator.errors.PathValidationError: The path could not be validated because the end-entity certificate expired 2022-01-14 12:00:00Z

Guix Build:

0bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
1359755bffecc64b4c005c5cdee3824190f6b1759dbc6c20034476dcc06413959  guix-build-b7ecef1ddf0c/output/dist-archive/bitcoin-b7ecef1ddf0c.tar.gz
20c6700270ec75991d70a97cad77e22cc00553f812edb56c1bac5ef6421f963e1  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/SHA256SUMS.part
387d4637a87959a304422550edf87feda3953d7305894154a6a2d413cc0dd2034  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/bitcoin-b7ecef1ddf0c-osx-unsigned.dmg
49cabae32689bd5f93e7faaaf341827f1c4069a63ab6f74276564e47819343b6c  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/bitcoin-b7ecef1ddf0c-osx-unsigned.tar.gz
5bb5fb113bc022a305e49783d0ba48be90aca61e4a942beeb45206dbc5b91ca6e  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/bitcoin-b7ecef1ddf0c-x86_64-apple-darwin.tar.gz

guix: ignore additioanl failing certvalidator test

======================================================================
ERROR: test_revocation_mode_soft (tests.test_validate.ValidateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/test_validate.py", line 85, in test_revocation_mode_soft
    validate_path(context, path)
  File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/../certvalidator/validate.py", line 50, in validate_path
    return _validate_path(validation_context, path)
  File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/../certvalidator/validate.py", line 358, in _validate_path
    raise PathValidationError(pretty_message(
certvalidator.errors.PathValidationError: The path could not be validated because the end-entity certificate expired 2022-01-14 12:00:00Z

Github-Pull: #24057
Rebased-From: 85885919656a3f606f3d7f208378aabe95f3f62d

b7ecef1ddf

fanquake added the label Backport on Jan 31, 2022

Sjors commented at 6:26 pm on January 31, 2022: member

I did a a Guix clean followed by:

0env HOSTS='x86_64-apple-darwin' ./contrib/guix/guix-build

This succeeds, but code-sign doesn’t:

 0env HOSTS='x86_64-apple-darwin' ./contrib/guix/guix-codesign 
 1Checking that we can connect to the guix-daemon...
 2
 3Hint: If this hangs, you may want to try turning your guix-daemon off and on
 4      again.
 5
 6INFO: Codesigning b7ecef1ddf0c for platform triple x86_64-apple-darwin:
 7      ...using reference timestamp: 1642652187
 8      ...from worktree directory: '/home/guix/bitcoin'
 9          ...bind-mounted in container to: '/bitcoin'
10      ...in build directory: '/home/guix/bitcoin/guix-build-b7ecef1ddf0c/distsrc-b7ecef1ddf0c-x86_64-apple-darwin-codesigned'
11          ...bind-mounted in container to: '/distsrc-base/distsrc-b7ecef1ddf0c-x86_64-apple-darwin-codesigned'
12      ...outputting in: '/home/guix/bitcoin/guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin-codesigned'
13          ...bind-mounted in container to: '/outdir-base/x86_64-apple-darwin-codesigned'
14      ...using detached signatures in: '/home/guix/bitcoin-detached-sigs'
15          ...bind-mounted in container to: '/detached-sigs'
16Required environment variables as seen inside the container:
17    UNSIGNED_TARBALL: /outdir-base/x86_64-apple-darwin/bitcoin-b7ecef1ddf0c-osx-unsigned.tar.gz
18    DETACHED_SIGS_REPO: /detached-sigs
19    DIST_ARCHIVE_BASE: /outdir-base/dist-archive
20    DISTNAME: bitcoin-b7ecef1ddf0c
21    HOST: x86_64-apple-darwin
22    SOURCE_DATE_EPOCH: 1642652187
23    DISTSRC: /distsrc-base/distsrc-b7ecef1ddf0c-x86_64-apple-darwin-codesigned
24    OUTDIR: /outdir-base/x86_64-apple-darwin-codesigned
25/gnu/store/q3y2bpd61bvb7d0g9ils1zi6pax5yvb1-python-elfesteem-0.1-1.87bbd79/lib/python3.8/site-packages/elfesteem/cstruct.py:412: SyntaxWarning: "is not" with a literal. Did you mean "!="?
26  if name is not '' and not name in table: table[name] = {}
27/gnu/store/q3y2bpd61bvb7d0g9ils1zi6pax5yvb1-python-elfesteem-0.1-1.87bbd79/lib/python3.8/site-packages/elfesteem/cstruct.py:415: SyntaxWarning: "is not" with a literal. Did you mean "!="?
28  if name is not '':
29Code signature applied
30Traceback (most recent call last):
31  File "/gnu/store/vsi3743mm41cfdv0zmap6p2qfsqylcmx-python-signapple-0.1-1.b084cbb/bin/.signapple-real", line 11, in <module>
32    load_entry_point('signapple==0.1.0', 'console_scripts', 'signapple')()
33  File "/gnu/store/vsi3743mm41cfdv0zmap6p2qfsqylcmx-python-signapple-0.1-1.b084cbb/lib/python3.8/site-packages/signapple/__init__.py", line 112, in main
34    args.func(args)
35  File "/gnu/store/vsi3743mm41cfdv0zmap6p2qfsqylcmx-python-signapple-0.1-1.b084cbb/lib/python3.8/site-packages/signapple/__init__.py", line 36, in apply
36    verify(args)
37  File "/gnu/store/vsi3743mm41cfdv0zmap6p2qfsqylcmx-python-signapple-0.1-1.b084cbb/lib/python3.8/site-packages/signapple/__init__.py", line 10, in verify
38    verify_mach_o_signature(args.filename)
39  File "/gnu/store/vsi3743mm41cfdv0zmap6p2qfsqylcmx-python-signapple-0.1-1.b084cbb/lib/python3.8/site-packages/signapple/verify.py", line 227, in verify_mach_o_signature
40    _verify_single(filepath, header)
41  File "/gnu/store/vsi3743mm41cfdv0zmap6p2qfsqylcmx-python-signapple-0.1-1.b084cbb/lib/python3.8/site-packages/signapple/verify.py", line 161, in _verify_single
42    _validate_code_hashes(f, sig_superblob.code_dir_blob)
43  File "/gnu/store/vsi3743mm41cfdv0zmap6p2qfsqylcmx-python-signapple-0.1-1.b084cbb/lib/python3.8/site-packages/signapple/verify.py", line 54, in _validate_code_hashes
44    raise Exception(
45Exception: Code slot hash mismatch. Expected 8cdb98ee7dbd9c1a5b021603c4b0ef933a31717f57884bc3a72536257e4d53dd, Calculated 95bcde7c12f864fcf3de59c87142b9fde1b728328a48cd923023381b730b4eab

Some hashes:

08f70852feb39078e02182563517d17bdfc4a12904cf1bdabbae95594d9a1e473  guix-build-b7ecef1ddf0c/output/dist-archive/bitcoin-b7ecef1ddf0c-codesignatures-22.0.tar.gz
1359755bffecc64b4c005c5cdee3824190f6b1759dbc6c20034476dcc06413959  guix-build-b7ecef1ddf0c/output/dist-archive/bitcoin-b7ecef1ddf0c.tar.gz
20c6700270ec75991d70a97cad77e22cc00553f812edb56c1bac5ef6421f963e1  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/SHA256SUMS.part
387d4637a87959a304422550edf87feda3953d7305894154a6a2d413cc0dd2034  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/bitcoin-b7ecef1ddf0c-osx-unsigned.dmg
49cabae32689bd5f93e7faaaf341827f1c4069a63ab6f74276564e47819343b6c  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/bitcoin-b7ecef1ddf0c-osx-unsigned.tar.gz
5bb5fb113bc022a305e49783d0ba48be90aca61e4a942beeb45206dbc5b91ca6e  guix-build-b7ecef1ddf0c/output/x86_64-apple-darwin/bitcoin-b7ecef1ddf0c-x86_64-apple-darwin.tar.gz

Sjors commented at 6:27 pm on January 31, 2022: member

cc @dongcarl

achow101 commented at 6:44 pm on January 31, 2022: member

I’ve also updated the certvalidator fork we use to have passing tests.

hebasto commented at 6:48 pm on January 31, 2022: member

@Sjors

This succeeds, but code-sign doesn’t:

Maybe #21851 (comment) and #22546 ?

Sjors commented at 8:22 pm on January 31, 2022: member

@hebasto I’m running Guix on an Ubuntu machine. Are you saying I should copy it over to my Mac and code-sign it? That seems odd. Or is that a workaround because we don’t have detached signatures for this commit? @achow101 certvalidator?

dongcarl commented at 10:54 pm on January 31, 2022: member

Wondering if we should just switch to achow’s https://github.com/achow101/certvalidator/commit/e5bdb4bfcaa09fa0af355eb8867d00dfeecba08c instead?

fanquake commented at 11:00 pm on January 31, 2022: member

Wondering if we should just switch to achow’s achow101/certvalidator@e5bdb4b instead?

Isn’t that the version we are already using?

achow101 commented at 11:07 pm on January 31, 2022: member

Wondering if we should just switch to achow’s achow101/certvalidator@e5bdb4b instead?

I think you mean https://github.com/achow101/certvalidator/commit/a145bf25eb75a9f014b3e7678826132efbba6213

@hebasto I’m running Guix on an Ubuntu machine. Are you saying I should copy it over to my Mac and code-sign it? That seems odd. Or is that a workaround because we don’t have detached signatures for this commit?

The error you get there is because the detached sig it is using is for a different release. signapple does not know that the detached sig is not for the binary you have built.

@achow101 certvalidator?

The tool that this patch is fixing.

fanquake commented at 6:44 am on February 3, 2022: member

I think we could switch to the newer certvalidator branch in master, and then re-enable tests. However for 22.x I’d rather fix the build by backporting this change. I’ll add some Guix hashes to the PR description shortly.

Sjors commented at 10:15 am on February 3, 2022: member

My hashes match those in the PR description, but those don’t include the signed DMG. I guess I’m confused about what certificate related problem this PR is solving.

fanquake commented at 1:58 am on February 4, 2022: member

I guess I’m confused about what certificate related problem this PR is solving.

If you Guix build the current 22.x branch, from scratch (so that python-certvalidator is built instead of a cached version being used), it will fail. This PR is making it not fail. The python-certvalidator package has a test that has recently started failing, due to a certificate used in it’s test suite expiring. All this change does is ignore the failing test, similar to how we already ignore some of its tests, so that the Guix build will work.

but those don’t include the signed DMG.

They don’t need to. You don’t need to run the code-sign step at all to verify this fixing what it’s supposed to be fixing.

Sjors commented at 9:09 am on February 4, 2022: member

What is python-certvalidator used for roughly?

fanquake commented at 9:16 am on February 4, 2022: member

What is python-certvalidator used for roughly?

https://github.com/wbond/certvalidator: “A Python library for validating X.509 certificates or paths. “. We use a fork maintained by achow. It’s a dependency of sign-apple.

Sjors commented at 10:36 am on February 4, 2022: member

from scratch

I did a guix clean and then built the commit before PR. Which works fine. I’ll try a more thorough nuke of Guix files…

MarcoFalke commented at 9:25 am on February 9, 2022: member

Concept ACK b7ecef1ddf0c9f1f53ab220bee2e19a6b8978e34

Seems good to make it possible to compile the branch again

fanquake commented at 9:38 am on February 9, 2022: member

I did a guix clean and then built the commit before PR. Which works fine.

You need to Guix build such that the python-certvalidator package is actually built, and the tests run.

I am going to merge this now to un-break from-scratch builds.

fanquake merged this on Feb 9, 2022

fanquake closed this on Feb 9, 2022

fanquake deleted the branch on Feb 9, 2022

DrahtBot locked this on Feb 9, 2023

[22.x] guix: ignore additional failing certvalidator test #24215