contrib: macdeploy: make gen-sdk deterministic #24534

pull prusnak wants to merge 2 commits into bitcoin:master from prusnak:deterministic-macos-sdk changing 2 files +32 −4
  1. prusnak commented at 3:23 pm on March 11, 2022: contributor

    This PR attempts to make contrib/macdeploy/gen-sdk deterministic

    Can anyone with the Xcode_12.2.xip confirm that gen-sdk produces the same hash? => e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae

  2. fanquake added the label macOS on Mar 11, 2022
  3. fanquake added the label Build system on Mar 11, 2022
  4. fanquake removed the label Build system on Mar 11, 2022
  5. fanquake added the label Scripts and tools on Mar 11, 2022
  6. prusnak force-pushed on Mar 11, 2022
  7. prusnak force-pushed on Mar 11, 2022
  8. fanquake commented at 3:55 pm on March 11, 2022: member

    Testing your branch @ 13a4a092dbeb3ec9a99398450941f4066dc92fcf:

     0shasum -a 256 Xcode_12.2.xip
 128d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0  Xcode_12.2.xip
 2....
 3contrib/macdeploy/gen-sdk Xcode.app
 4Found Xcode (version: 12.2, build id: 12B45b)
 5Found MacOSX SDK (version: 11.0, build id: 20A2408)
 6Creating output .tar.gz file...
 7Adding MacOSX SDK 11.0 files...
 8Adding libc++ headers...
 9Done! Find the resulting gzipped tarball at:
10Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
11...
12shasum -a 256 Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz 
13a396dd24f61fb55a6d3ec98b8b58fc0b04cdb6b2695039869d04105f885e0867  Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
  9. prusnak commented at 4:23 pm on March 11, 2022: contributor

    Testing your branch @ 13a4a09:

    How did you extract the xip file? Via xip -x or apple-sdk-tools/extract_xcode.py ?

    Edit: I tried to unpack the xip file using both methods and despite the extracted folders have different timestamps (xip -x keeps the timestamps from the archive, the cpio method ruins them), the updated gen-sdk script produced the same hash for me.

    Will investigate further …

  10. prusnak force-pushed on Mar 11, 2022
  11. prusnak commented at 5:12 pm on March 11, 2022: contributor

    It seems the default TAR format has changed from GNU to PAX in Python 3.8.

    This is addressed by change in 140f8e7831fc191f4981c4e681f501ef744a6129 which sets the GNU format explicitly.

  12. prusnak commented at 6:46 pm on March 11, 2022: contributor

    Argh, found another culprit - https://bugs.python.org/issue18819 (fixed via https://github.com/python/cpython/pull/18080)

    Python 3.8 and older use ASCII string 0000000\x00 for devmajor/devminor to indicate the entry is not a device. Python 3.9 and newer use the binary string \x00\x00\x00\x00\x00\x00\x00\x00 (to match the original TAR behaviour) 😭

    This might be fixed by monkey-patching the tarfile.TarInfo._create_header function.

  13. contrib: macdeploy: make gen-sdk deterministic 1868a17e5a
  14. contrib: macdeploy: monkey-patch gen-sdk to be deterministic 
    on different Python versions (there was a change in TAR handling
between Python 3.8 and Python 3.9)
    ba30a5407e
  15. prusnak force-pushed on Mar 12, 2022
  16. prusnak commented at 11:38 am on March 12, 2022: contributor

    This might be fixed by monkey-patching the tarfile.TarInfo._create_header function.

    I monkey-patched the Python bug in ba30a5407e065e9d6dd037351e83f56a43f38f19.

    The new deterministic hash should be e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae,

    I was able to reproduce this hash on:

    • Linux with Python 3.9 where the SDK was extracted via apple-sdk-tools/extract_xcode.py
    • macOS with Python 3.8 where the SDK was extracted via xip -x
  17. hebasto commented at 1:32 pm on March 12, 2022: member
    Concept ACK.
  18. fanquake commented at 8:33 pm on March 12, 2022: member

    The new deterministic hash should be e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae,

    Nice. I’m now getting a matching hash.

    0➜  SDK ./contrib/macdeploy/gen-sdk Xcode.app                                     
1Found Xcode (version: 12.2, build id: 12B45b)
2Found MacOSX SDK (version: 11.0, build id: 20A2408)
3Creating output .tar.gz file...
4Adding MacOSX SDK 11.0 files...
5Adding libc++ headers...
6Done! Find the resulting gzipped tarball at:
7Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
8➜  SDK shasum -a 256 Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
9e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae  Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
  19. prusnak marked this as ready for review on Mar 12, 2022
  20. prusnak commented at 9:45 pm on March 12, 2022: contributor

    Nice. I’m now getting a matching hash.

    Thanks. Switching from Draft to Ready for review.

  21. DrahtBot commented at 9:58 pm on March 12, 2022: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #21778 (build: LLVM 14 & LLD based macOS toolchain by fanquake)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  22. laanwj commented at 12:13 pm on April 6, 2022: member
    Concept ACK. I think having a deterministic MacOS SDK input is very useful, so that we can be sure to start from the same point based on hashes. I think we should at least get this in before the next SDK bump.
  23. jarolrod commented at 5:25 am on April 13, 2022: member

    Concept ACK, would love to see this be deterministic but I have a hash mismatch with the resulting archive file.

    I get the same hash for the downloaded xip file:

    0$ sha256sum ./Xcode_12.2.xip 
1
228d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0

    But, I get a different hash for the resulting archive file:

    0$ ./contrib/macdeploy/gen-sdk ./Xcode.app
1
2Found Xcode (version: 12.2, build id: 12B45b)
3Found MacOSX SDK (version: 11.0, build id: 20A2408)
4Creating output .tar.gz file...
5Adding MacOSX SDK 11.0 files...
6Adding libc++ headers...
7Done! Find the resulting gzipped tarball at:
8/home/xyz/Code/Bitcoin/review/bitcoin-24534/bitcoin/Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz

    0$ sha256sum ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz 
1
2501625bd401d7f228b3bae18f264fd3739da5788464f0897e5c455451d2128c2  ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
  24. prusnak commented at 7:30 pm on April 15, 2022: contributor

    Concept ACK, would love to see this be deterministic but I have a hash mismatch with the resulting archive file.

    OS version and Python version? Also what filesystem are you using?

  25. laanwj commented at 11:18 am on April 19, 2022: member

    Tested ACK ba30a5407e065e9d6dd037351e83f56a43f38f19

    Output matches the hash in the OP (tried on Ubuntu 22.04, x86_64, Python 3.10.4):

     0$ sha256sum ~/Downloads/Xcode_12.2.xip
 128d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0  /home/orion/Downloads/Xcode_12.2.xip
 2$ …/apple-sdk-tools/extract_xcode.py -f ~/Downloads/Xcode_12.2.xip | cpio -d -i
 3$ …/bitcoin/contrib/macdeploy/gen-sdk $PWD/Xcode.app/
 4Found Xcode (version: 12.2, build id: 12B45b)
 5Found MacOSX SDK (version: 11.0, build id: 20A2408)
 6Creating output .tar.gz file...
 7Adding MacOSX SDK 11.0 files...
 8Adding libc++ headers...
 9Done! Find the resulting gzipped tarball at:
10…/apple-sdk-tools/Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
11$ sha256sum Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
12e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae  Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
  26. jarolrod commented at 6:21 pm on April 19, 2022: member

    @prusnak That run was Ubuntu 22.04, python 3.10, on arm64 cpu. Will rerun on x86 ubuntu

    reserved for run on x86 ubuntu

  27. prusnak commented at 9:33 pm on April 19, 2022: contributor
    @jarolrod Hm, weird. can you please share the tarball with the 501625bd401d7f228b3bae18f264fd3739da5788464f0897e5c455451d2128c2 hash? I could compare locally and try to figure out what went wrong.
  28. jarolrod approved
  29. jarolrod commented at 4:44 am on April 21, 2022: member

    Tested ACK https://github.com/bitcoin/bitcoin/commit/ba30a5407e065e9d6dd037351e83f56a43f38f19

    I don’t know what had happened in my original run, its possible that on accident I hadn’t successfully changed to the PR branch. I re-ran everything on my setup and now I get a matching hash :)

    0sha256sum ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz 
1e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae  ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
  30. laanwj merged this on Apr 21, 2022
  31. laanwj closed this on Apr 21, 2022
  32. prusnak deleted the branch on Apr 21, 2022
  33. sidhujag referenced this in commit 26f45057fb on Apr 22, 2022
  34. laanwj commented at 7:08 pm on April 22, 2022: member
    I think this causes an error while building from the SDK: #24947
  35. DrahtBot locked this on Apr 22, 2023


prusnak fanquake hebasto DrahtBot laanwj jarolrod

Labels
macOS Scripts and tools

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me