contrib: turn off compression of macOS SDK to fix determinism (across distros) #32009

pull fanquake wants to merge 3 commits into bitcoin:master from fanquake:macos_sdk_select changing 4 files +23 −26
  1. fanquake commented at 4:23 PM on March 6, 2025: member

    This includes three changes. The first is to more selectively pick files for inclusion into our macOS SDK tarball (skip manpages, binaries etc), which is nice because it redues the size of the tarball (from ~80mb to 20mb), and makes the size increase that happens with the next commit, less-bad.

    The second change removes compression of the tarball. Starting with Python 3.11, Pythons gzip might delegate to zlib. Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib implementation might differ, resulting in different output.

    For now, or until a better solution exists, remove compression. This results in the SDK increasing in size to ~157mb. Which is not unreasonable, to regain determinism (and would be significantly worse without the previous commit).

    See: https://docs.python.org/3/library/gzip.html#gzip.compress

    The third renames gen-sdk to gen-sdk.py, so that it will be linted, along with the rest of our Python files.

    Fixes #31873. We could probably also put this into 30.x.

  2. DrahtBot commented at 4:23 PM on March 6, 2025: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32009.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK stickies-v, davidgumberg
    Concept ACK hebasto, willcl-ark, laanwj

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. achow101 commented at 4:29 PM on March 6, 2025: member

    Is determinism of that tarball necessary? IIRC it doesn't actually affect the build.

  4. fanquake commented at 4:31 PM on March 6, 2025: member

    Is determinism of that tarball necessary?

    I don't see why we wouldn't want it. If we don't care about it, we should remove: "The sha256sum should be c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d." So that issues like #31873 aren't opened.

  5. davidgumberg commented at 9:37 PM on March 6, 2025: contributor

    Concept ACK, while it might not be strictly necessary for this tarball to be deterministic, I think matching hashes of tarballs is the simplest way to verify that you've followed the same procedure as others to build the macOS sdk and have gotten the same result.

    150MB seems to me a small price to pay to fix #31873 and reduce surface area for nondeterminism in generating the macOS sdk.

  6. hebasto commented at 6:58 AM on March 7, 2025: member

    Concept ACK for the same reasoning as in #32009 (comment).

  7. willcl-ark commented at 10:06 AM on March 7, 2025: member

    Concept ACK.

    I don't think a few hundred MB more disk space required for (power-)users doing guix builds will be a great concern to anyone, and I agree with @davidgumberg that having as many steps in the guix workflow as possible be reproducible, is probably best.

  8. fanquake renamed this:
    RFC contrib: turn off compression of macOS SDK to fix determinism (across distros)
    contrib: turn off compression of macOS SDK to fix determinism (across distros)
    on Mar 7, 2025
  9. DrahtBot added the label Scripts and tools on Mar 7, 2025
  10. laanwj commented at 5:29 AM on March 8, 2025: member

    Concept ACK, imo determinism is good because it makes it possible to check if mistakes have been made in the process at earlier steps.

    One comment i was about to give is "why use a .tar.gz instead of .tar, that's the only way to be sure to remove dependency on zlib", but that's a much more spread out code change. This is most likely fine.

    edit: Could do that when we have to upgrade the MacOS SDK anyway, but definitely not for 29.0

  11. Sjors commented at 3:33 PM on March 10, 2025: member

    I'm not too worried about the size increase. A typical guix build eats 30GB on my Ubuntu VM (3 GB for the outputs).

    it makes it possible to check if mistakes have been made in the process at earlier steps.

    Agreed.

    It seems simple enough to backport, especially because we're not bumping the version.

    I'll see if I can reproduce the new hashes.

  12. Sjors commented at 3:43 PM on March 10, 2025: member

    One slight problem is that Apple no longer makes XCode 15.0 available for download. They do have Xcode 15.0.1: https://download.developer.apple.com/Developer_Tools/Xcode_15.0.1/Xcode_15.0.1.xip

    I still have the original xip on one of my machines, I'll upload it somewhere for testing. But we should probably bump it.

    Here you go: https://download.sprovoost.nl/download.php?id=21&token=ee9af1cfea8fa495cc842a407249126d

    (sha256 checksum is in contrib/macdeploy/README.md)

    Update: XCode 15 is still there, see below.


    I was able to reproduce the hash for the first commit 6998e933f935a379c3ad55c2fb16eca9b854f40b, but not for the second commit 20778eb0235df70397fc285f9e3b72270bd4aaf4. I get 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82 instead on macOS 15.3.1, Python 3.10.14. Ditto on my Ubuntu 24.04 VM inside the same machine.


    It looks like Apple Silicon has hardware acceleration for zip, maybe that's a factor? ~(and would also explain why extract_xcode.py is super fast on my VM running on the M4 mac, while super slow on the AMD Ryzen 7950x)~. (probably not, because the first commit did produce identical results)


    I also get 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82 for Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz on the AMD Ubuntu machine.

  13. davidgumberg commented at 12:13 AM on March 11, 2025: contributor

    One slight problem is that Apple no longer makes XCode 15.0 available for download. They do have Xcode 15.0.1: https://download.developer.apple.com/Developer_Tools/Xcode_15.0.1/Xcode_15.0.1.xip

    I'm still seeing it here: https://developer.apple.com/download/all/?q=Xcode%2015 , and with this download link: https://download.developer.apple.com/Developer_Tools/Xcode_15/Xcode_15.xip, it's labeled Xcode 15 rather than Xcode 15.0.

  14. Sjors commented at 8:14 AM on March 11, 2025: member

    @davidgumberg ah indeed, I see it as well. You have to click on the download link in order to for the download URL to work at all, so that's probably why it didn't work for me.

    So that just leaves the hash mismatch to figure out.

  15. willcl-ark commented at 1:11 PM on March 19, 2025: member

    At commit 20778eb0235df70397fc285f9e3b72270bd4aaf4 I get the following guix output:

    env HOSTS="x86_64-apple-darwin arm64-apple-darwin" ./contrib/guix/guix-build
    
    <snip>
    
    $ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
    30aa32906f879c5347be27bd9cb1a65b3d839acd72805fe3185276f788971fdb  guix-build-20778eb0235d/output/arm64-apple-darwin/SHA256SUMS.part
    0763a089bcc5d1d22191414f556bf461cefa8ea8a18971b171cf6fc5570b8790  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
    18cd712ac4cbd63122483a083a9f29d7c184ffaead45078fb27b3d6381b691a6  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
    c3fdc3b50077f2a4b772d4c1cf50e230f358542844e800e77194bf85dee9eb28  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.zip
    b97da95ff464b8bdcb9ce31dc783f947dd24d93eb6bfdfea71c3070d51991058  guix-build-20778eb0235d/output/dist-archive/bitcoin-20778eb0235d.tar.gz
    e180a9ef657ea3f0a6e42c4babe7b38a8e585902fbac301a1f16bccb0c53a18b  guix-build-20778eb0235d/output/x86_64-apple-darwin/SHA256SUMS.part
    5b032a2b6dd2dad11f3884194bcb85aea37237246162dfad27099e6c5175071d  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
    39d23ec95f34cd7d8bbdb6228e55f7f8a0f48a3997c3dcb146a2ff3744f37629  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz
    76e36fa1af2f788dea631a7fcc93451fdf794a40dded0e3c7c12f74ab136aefb  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.zip
    
    $ eza -al guix-build-20778eb0235d/output/*/*apple*.tar.gz
    .rw-r--r-- 52M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
    .rw-r--r-- 35M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
    .rw-r--r-- 56M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
    .rw-r--r-- 38M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz
    
    
    
    
  16. davidgumberg commented at 5:37 AM on March 22, 2025: contributor

    I don't have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

    pkgdiff again reports the mismatching archives as identical. But, now that the archives are uncompressed, the binary diff output is a little more revealing, here's the start of the diff:

    <details> <summary>

    git diff <(xxd rocky9.3/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(xxd fedora41/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)

    </summary>

    @@ -2050,7 +2050,7 @@
     00008010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     00008020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     00008030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    -00008040: 0000 0000 0000 0000 00a0 815f 7e58 636f  ..........._~Xco
    +00008040: 0000 0000 0000 0000 0000 d0ff 2f58 636f  ............/Xco
     00008050: 6465 2d31 352e 302d 3135 4132 3430 642d  de-15.0-15A240d-
     00008060: 6578 7472 6163 7465 642d 5344 4b2d 7769  extracted-SDK-wi
     00008070: 7468 2d6c 6962 6378 782d 6865 6164 6572  th-libcxx-header
    @@ -4124,30 +4124,30 @@
     000101b0: 6129 2041 5050 4c45 5f41 5243 4849 5645  a) APPLE_ARCHIVE
     000101c0: 5f53 5749 4654 5f50 5249 5641 5445 3b0a  _SWIFT_PRIVATE;.
     000101d0: 0a23 6966 6465 6620 5f5f 6370 6c75 7370  .#ifdef __cplusp
    -000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 608e  lus.}.#endif..`.
    -000101f0: 9f71 0000 0000 0000 0000 0000 0000 0000  .q..............
    +000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 0000  lus.}.#endif....
    +000101f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     00010200: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     00010210: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     00010220: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     00010230: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    -00010240: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    -00010250: 0000 5863 6f64 652d 3135 2e30 2d31 3541  ..Xcode-15.0-15A
    -00010260: 3234 3064 2d65 7874 7261 6374 6564 2d53  240d-extracted-S
    -00010270: 444b 2d77 6974 682d 6c69 6263 7878 2d68  DK-with-libcxx-h
    -00010280: 6561 6465 7273 2f75 7372 2f69 6e63 6c75  eaders/usr/inclu
    -00010290: 6465 2f41 7070 6c65 4172 6368 6976 652f  de/AppleArchive/
    -000102a0: 4141 456e 7472 7958 4154 426c 6f62 2e68  AAEntryXATBlob.h
    -000102b0: 0000 0000 0000 3030 3030 3634 3400 3030  ......0000644.00
    -000102c0: 3030 3030 3000 3030 3030 3030 3000 3030  00000.0000000.00
    -000102d0: 3030 3030 3135 3437 3400 3030 3030 3030  000015474.000000
    -000102e0: 3030 3030 3000 3032 3631 3631 0020 3000  00000.026161. 0.
    +00010240: 0000 0000 0000 0000 0000 0000 0058 636f  .............Xco
    +00010250: 6465 2d31 352e 302d 3135 4132 3430 642d  de-15.0-15A240d-
    +00010260: 6578 7472 6163 7465 642d 5344 4b2d 7769  extracted-SDK-wi
    +00010270: 7468 2d6c 6962 6378 782d 6865 6164 6572  th-libcxx-header
    +00010280: 732f 7573 722f 696e 636c 7564 652f 4170  s/usr/include/Ap
    +00010290: 706c 6541 7263 6869 7665 2f41 4145 6e74  pleArchive/AAEnt
    +000102a0: 7279 5841 5442 6c6f 622e 6800 0000 0000  ryXATBlob.h.....
    +000102b0: 0030 3030 3036 3434 0030 3030 3030 3030  .0000644.0000000
    +000102c0: 0030 3030 3030 3030 0030 3030 3030 3031  .0000000.0000001
    +000102d0: 3534 3734 0030 3030 3030 3030 3030 3030  5474.00000000000
    +000102e0: 0030 3236 3136 3100 2030 0000 0000 0000  .026161. 0......
    

    </details>

    <details>

    <summary>

    Testing methodology

    </summary>

    Modified from the repro instructions in #31873 to be faster, most importantly reusing the result of the lengthy extraction step, and just downloading the relevant branches once.

    Setup phase:

    # assumes you have acquired `Xcode_15.xip` based on the instructions in
    # contrib/macdeploy/README.md and placed it at ~/xcode/Xcode_15.xip
    XCODE=~/xcode # feel free to change this
    
    cd $XCODE
    sha256sum Xcode_15.xip
    # 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
    git clone --depth 1 https://github.com/bitcoin/bitcoin.git bitcoin-master
    cd bitcoin-master && git fetch --depth 1 origin pull/32009/head:32009 && git worktree add ../bitcoin-32009 32009 && cd ../ # fetch the pull branch
    
    git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
    python3 apple-sdk-tools/extract_xcode.py -f Xcode_15.xip | cpio -d -i # single threaded, slow, we want to reuse this.
    

    To verify that the source material for gen-sdk is good, we can generate the sdk using a known good setup, ubuntu 24.04:

    <details>

    <summary>Verifying the setup</summary>

    docker pull ubuntu:24.04
    docker run -it \
      -v $XCODE:/xcode \
      ubuntu:24.04 \
      /bin/bash
    

    Inside the container:

    export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
    sha256sum /xcode/Xcode_15.xip
    # 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e
    apt update
    apt install python3 -y
    
    xcode/bitcoin-master/contrib/macdeploy/gen-sdk xcode/Xcode.app/  # we are reusing the extracted result from above
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    # c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    We should get the same hash here as what contrib/macdeploy/README.md in master promises.

    </details>


    To test this branch's version of gen-sdk on a variety of platforms:

    1. Container setup

    XCODE=~/xcode # where the files from the setup above went.
    # specify the container platform
    PLATFORM=fedora:40 
    
    docker pull $PLATFORM && \
      docker run -it \
        -v $XCODE:/xcode \
        $PLATFORM \
        /bin/bash
    

    2. In the container

    Compare the output of the final sha256sum to what contrib/macdeploy/README.md in this branch promises.

    Debian/Ubuntu
    export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
    apt update > /dev/null
    apt install python3 -y > /dev/null
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    Fedora/CentOS

    dnf install python -y --quiet # python3 on rocky8.9
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    Arch

    pacman -Sy
    pacman --noconfirm -S python
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    </details>

    <details> <summary>

    Results Tables

    </summary>

    Success

    image python --version python -c "import zlib; print(zlib.ZLIB_VERSION)"
    fedora:39 3.12.7 1.2.13
    fedora:40 3.12.9 1.3.1-zlib-ng
    fedora:41 3.13.2 1.3.1-zlib-ng

    |ubuntu:24.04 |3.12.3 |1.3 |ubuntu:24.10 |3.12.7 |1.3.1 |archlinux:latest| 3.13.2 |1.3.1 |

    Failed to reproduce:

    image python --version python -c "import zlib; print(zlib.ZLIB_VERSION)" hash
    debian:bookworm 3.11.2 1.2.13 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82
    rockylinux:8.9 3.6.8 1.2.11 e779914636e6a3a417bf2a19dbce6f0bf8fab10b16717df769d107a5aad6aa2e
    rockylinux:9.3 3.9.18 1.2.11 07b12c2a489c241bbc8c853fe78f2e92faf8ff51631311d142aeb8c7e20e7268

    </details>

    I downgraded to python 3.11.3 on fedora 41 with zlib 1.3, and got a bad hash, upgraded to python 3.12.0 with zlib-ng 1.3 on debian:bookworm and got a good hash.

    <details>

    <summary>custom zlib pyenv setup</summary>

    Also see #31873 (comment)

    dnf install -y gcc git make openssl-devel xz-devel git
    curl -L https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.gz | tar xzvf -
    cd zlib-1.3 && ./configure && make -j $(nproc) && make install && cd ..
    
    curl https://pyenv.run | bash
    export PYENV_ROOT="$HOME/.pyenv"
    [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
    eval "$(pyenv init - bash)"
    
    pyenv install 3.11.3
    pyenv global 3.11.3
    
    # verify zlib version
    python -c "import zlib; print(zlib.ZLIB_VERSION)"
    # 1.3
    
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    # get something bad
    
    pyenv install 3.12.0
    pyenv global 3.12.0
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    </details>

    Don't have super strong evidence for this, but I suspect that this might be a second source of nondeterministic behavior that was already present before, as in my previous testing of master when I reported this issue, when I was trying different combinations of python and zlib, I also saw a similar pattern where when changing to random python, some of the hashes were randomly mismatched, and the rest were either of two hashes, one right and one wrong. I avoided sharing the random hashes in my table in #31873, and all of the values I share are for python 3.12.x

  17. maflcko commented at 6:05 AM on March 22, 2025: member

    I don't have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

    A wild guess: Could this be related to https://github.com/python/cpython/issues/95385?

  18. laanwj commented at 4:07 PM on March 26, 2025: member

    git diff <(xxd rocky9.3/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(xxd fedora41/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)

    This seems to suggest the divergence is in the tarring, not the gzipping. Do you still see a difference if you gunzip the .gzs and compare the tars only?

  19. luke-jr commented at 11:29 PM on April 9, 2025: member

    Maybe keep the filename the same, so people can optionally compress it on their own (or even use the current file) without breaking the build?

  20. fanquake force-pushed on Jun 2, 2025
  21. fanquake commented at 3:19 PM on June 2, 2025: member

    Rebased and added a change to skip more files to the first commit, which reduces the final uncompressed size from ~230mb to ~157mb.

  22. Sjors commented at 5:47 AM on June 3, 2025: member

    nit: can drop "RFC:" from the second commit?

    I was able to reproduce the hash for the first commit 6998e93, but not for the second commit 20778eb.

    Tested this again against a2bf3b03b19a0e668bf86eea7de3e3a592b4b23f and fc81a759413230daf3e05b448b357e62915bb3ba on my M4 machine running macOS 15.5 with Python 3.10.14 (via PyEnv). Again the hash matches in the first commit, but with second commit I get:

    fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    If I use Python 3.11.11 instead I get the same hash as with 3.10.14. @davidgumberg wrote:

    I don't have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

    With Python 3.12.9 I indeed get ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6.

    We could bump the version in .python-version and/or just mention a minimum Python version for this tool? The 3.10 security fix maintenance ends before our v30 maintenance window, and they're already not fixing (non-security) bugs: https://devguide.python.org/versions/

  23. fanquake force-pushed on Jun 3, 2025
  24. fanquake commented at 10:09 AM on June 3, 2025: member

    nit: can drop "RFC:" from the second commit?

    Dropped.

  25. maflcko commented at 10:45 AM on June 3, 2025: member

    We could bump the version in .python-version and/or just mention a minimum Python version for this tool?

    Wholesale bumping to 3.12 seems a bit early. Maybe just add an option to this tool --force-non-determinstic-old-python to enable 3.10 and 3.11 and otherwise use 3.12+ by default?

  26. achow101 requested review from achow101 on Oct 22, 2025
  27. achow101 requested review from stickies-v on Oct 22, 2025
  28. achow101 requested review from davidgumberg on Oct 22, 2025
  29. stickies-v commented at 12:23 PM on October 29, 2025: contributor

    utACK 71b0f474eee3aceb8c17b9ffa7fa866ecb3a93af, Concept ACK on disabling compression to increase determinism.

    71b0f474eee3aceb8c17b9ffa7fa866ecb3a93af seems like a strict win, no need to include what we don't use. I think conceptually 0a9b63b1f447bc31e6b56410f7c9c25abc7f5047 also makes sense: the increased size of the tarball should not be a blocked for increased determinism. However, on my machines (arm64 macOS 15.6 and x86_64 Debian Bookworm ) determinism seems to be reduced without compression across Python versions (as also reported by other reviewers). I'm not sure whether increased determinism across distros is more important than determinism across (supported) Python versions?

    (note: below results are from macOS, the hashes are the same for Debian but the filesizes vary a bit)

    With selective picking and no compression (0a9b63b1f447bc31e6b56410f7c9c25abc7f5047) different sha256sum hashes are produced by python 3.{10,11}, 3.{12, 13} and 3.{14}:

    py10: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
    py11: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
    py12: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
    py13: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
    py14: ea4684c568bbf63ca9fa4d1a9116751ba5b870728c4e9f72862f1b0d453ef05d (161mb)
    

    With selective picking and compression (71b0f474eee3aceb8c17b9ffa7fa866ecb3a93af) the results are consistent:

    py10: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    py11: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    py12: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    py13: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    py14: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    

    On master (e872a566f251c73908de8b6d243c94a6679c2eac) the results are consistent:

    py10: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    py11: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    py12: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    py13: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    py14: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    
  30. achow101 commented at 11:09 PM on November 17, 2025: member

    One comment i was about to give is "why use a .tar.gz instead of .tar, that's the only way to be sure to remove dependency on zlib", but that's a much more spread out code change.

    I don't think that's true. Our guix build documentation states that the tarball needs to be extracted into depends/SDK (or some directory that SDK_PATH points to), and depends/hosts/darwin.mk uses the path to the extracted directory, not the tarball itself. Other than in the docs, I don't think the tarball itself is ever handled by our build scripts. So it seems to me that dropping the gzip altogether, and just going with a .tar would resolve this problem with minimal issues.

  31. fanquake force-pushed on Nov 18, 2025
  32. fanquake commented at 12:06 PM on November 18, 2025: member

    I think this could actually be a fine solution, and the additional changes needed are minimal, so I've pushed that up. The hash of the tarball using this branch is 95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498. The size is ~157mb.

  33. fanquake commented at 12:32 PM on November 18, 2025: member

    Guix Build (aarch64):

    3bd9c90945da34a1398beceb2474401a486a6f3436bb4a12a573bdfa38df6d62  guix-build-5513cd0941a2/output/arm64-apple-darwin/SHA256SUMS.part
    e78e73b65107c95fd4199f5a354471e8f61165c51ee7602b775a59093cf345f5  guix-build-5513cd0941a2/output/arm64-apple-darwin/bitcoin-5513cd0941a2-arm64-apple-darwin-codesigning.tar.gz
    3fee841b5625963dda994f68035e2d87decbbce4df666485e76f42fbe2b7f61b  guix-build-5513cd0941a2/output/arm64-apple-darwin/bitcoin-5513cd0941a2-arm64-apple-darwin-unsigned.tar.gz
    6ec06c295b6881c0b06df9e3ec28f873b273de1d4a903a3d8e5872131895c3fd  guix-build-5513cd0941a2/output/arm64-apple-darwin/bitcoin-5513cd0941a2-arm64-apple-darwin-unsigned.zip
    346be922613989dbcbbc8ca5ba3ae476309388d91432788e8cec20aecf4fb7dd  guix-build-5513cd0941a2/output/dist-archive/bitcoin-5513cd0941a2.tar.gz
    f69af9fb3949cb815e1868bc0a20147e918611c3221fa36bed48060aa16d7372  guix-build-5513cd0941a2/output/x86_64-apple-darwin/SHA256SUMS.part
    6cb7f0f296bdeaef38f125d2d2ad0bbf2620af67e433d993e793a6be152ce3e3  guix-build-5513cd0941a2/output/x86_64-apple-darwin/bitcoin-5513cd0941a2-x86_64-apple-darwin-codesigning.tar.gz
    f5f6fef78e138adf3d211d9ce1d1ccce461970843ff63bea443cc6757c4ebdca  guix-build-5513cd0941a2/output/x86_64-apple-darwin/bitcoin-5513cd0941a2-x86_64-apple-darwin-unsigned.tar.gz
    cf936dbd8fa32cfdc0ef97a6f05893c3f2c98ffa74e7809d5136282bffa141a5  guix-build-5513cd0941a2/output/x86_64-apple-darwin/bitcoin-5513cd0941a2-x86_64-apple-darwin-unsigned.zip
    
  34. in contrib/macdeploy/gen-sdk.py:1 in 5513cd0941 outdated


    stickies-v commented at 1:51 PM on November 18, 2025:

    nit: could be cleaned up a bit more

    <details> <summary>git diff on 5513cd0941</summary>

    diff --git a/contrib/macdeploy/gen-sdk b/contrib/macdeploy/gen-sdk
    index 0cfd2b1379..426d82e46c 100755
    --- a/contrib/macdeploy/gen-sdk
    +++ b/contrib/macdeploy/gen-sdk
    @@ -2,9 +2,7 @@
     import argparse
     import plistlib
     import pathlib
    -import sys
     import tarfile
    -import gzip
     import os
     import contextlib
     
    @@ -23,7 +21,7 @@ def run():
             description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
     
         parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
    -    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
    +    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
     
         args = parser.parse_args()
     
    @@ -47,11 +45,11 @@ def run():
     
         out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
     
    -    if args.out_sdktgz:
    -        out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
    +    if args.out_sdkt:
    +        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
         else:
    -        # Construct our own out_sdktgz if not specified on the command line
    -        out_sdktgz_path = pathlib.Path("./{}.tar".format(out_name))
    +        # Construct our own out_sdkt if not specified on the command line
    +        out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
     
         def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
             """Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'
    @@ -88,12 +86,12 @@ def run():
                 tarfp.add("./System/Library/Frameworks", recursive=True, filter=change_tarinfo_base)
     
         print("Creating output .tar file...")
    -    with out_sdktgz_path.open("wb") as fp:
    +    with out_sdkt_path.open("wb") as fp:
             with tarfile.open(mode="w", fileobj=fp, format=tarfile.PAX_FORMAT) as tarfp:
                 print("Adding MacOSX SDK {} files...".format(sdk_version))
                 tarfp_add_with_base_change(tarfp, sdk_dir, out_name)
         print("Done! Find the resulting tarball at:")
    -    print(out_sdktgz_path.resolve())
    +    print(out_sdkt_path.resolve())
     
     if __name__ == '__main__':
         run()
    
    

    </details>


    stickies-v commented at 2:09 PM on November 18, 2025:

    The CI script is still using the .tar.gz name (which is fine as long as that file is available on SDK_URL): https://github.com/bitcoin/bitcoin/blob/2444488f6ad32dcbbed51a73cd4f59ff3a239e32/ci/test/01_base_install.sh#L93

    Perhaps we can temporarily add both the .tar.gz and .tar files to the SDK_URL and update all of this in one go?


    maflcko commented at 2:18 PM on November 18, 2025:

    Not sure it this is possible, but if the file was renamed, the existing linters will pick up and lint this file:

    git mv ./contrib/macdeploy/gen-sdk ./contrib/macdeploy/gen-sdk.py
    

    fanquake commented at 2:22 PM on November 18, 2025:

    Perhaps we can temporarily add both the

    We will upload the new SDK before this PR is merged (leaving the existing one in place), I don't think we need interim code changes here.


    fanquake commented at 2:36 PM on November 18, 2025:

    Have renamed to gen-sdk.py. Tested that the linter works locally by reintroduing rogue import time.


    fanquake commented at 2:36 PM on November 18, 2025:

    Took the rest of the suggestions.


    stickies-v commented at 2:46 PM on November 18, 2025:

    We will upload the new SDK before this PR is merged (leaving the existing one in place), I don't think we need interim code changes here.

    Yeah that's what I meant, but I now see how my phrasing was ambiguous. The "temporary" referred to having both versions on the server, until the gz is removed after this PR is merged. Thanks.


    maflcko commented at 2:50 PM on November 18, 2025:

    i think the gz will be needed for old release branches?


    fanquake commented at 2:51 PM on November 18, 2025:

    Yea, the .tar.gz will be left as-is.

  35. stickies-v approved
  36. stickies-v commented at 2:11 PM on November 18, 2025: contributor

    tACK 5513cd0941a2300c0b78758d980ef5eee5079b4c

    Determinism across Python versions (3.10-3.14) is now restored on my machine. New approach of just avoiding gzip entirely seems straightforward and preferable.

  37. DrahtBot requested review from laanwj on Nov 18, 2025
  38. DrahtBot requested review from hebasto on Nov 18, 2025
  39. DrahtBot requested review from willcl-ark on Nov 18, 2025
  40. fanquake force-pushed on Nov 18, 2025
  41. stickies-v commented at 2:59 PM on November 18, 2025: contributor

    ACK e7c22c8eb446727df00d1cb2ce554be9324c7fbd once the new .tar SDK is uploaded

  42. DrahtBot added the label CI failed on Nov 18, 2025
  43. in contrib/macdeploy/gen-sdk.py:24 in e7c22c8eb4
      20 | @@ -23,7 +21,7 @@ def run():
      21 |          description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
      22 |  
      23 |      parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
      24 | -    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
      25 | +    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
    


    l0rinc commented at 8:28 AM on November 25, 2025:

    should we adjust the help description as well?

        parser.add_argument("-o", metavar='OUTSDKTAR', nargs=1, dest='out_sdkt', required=False)
    

    usage: gen-sdk.py [-h] [-o OUTSDKTAR] XCODEAPP


    fanquake commented at 11:38 AM on November 25, 2025:

    Done.

  44. in contrib/macdeploy/gen-sdk.py:69 in e7c22c8eb4
      65 | @@ -68,6 +66,8 @@ def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
      66 |  
      67 |          """
      68 |          def change_tarinfo_base(tarinfo):
      69 | +            if tarinfo.name and tarinfo.name.endswith(("swiftmodule", "modulemap")):
    


    l0rinc commented at 8:49 AM on November 25, 2025:

    are these file extensions? I wasn't familiar with them, but e.g. https://medium.com/@mail2ashislaha/swift-objective-c-interoperability-static-libraries-modulemap-etc-39caa77ce1fc indicates they're extensions, in which case:

                if tarinfo.name and tarinfo.name.endswith((".swiftmodule", ".modulemap")):
    

    fanquake commented at 11:38 AM on November 25, 2025:

    Some of these are folders, but this will also work.

  45. in contrib/macdeploy/gen-sdk.py:49 in e7c22c8eb4 outdated
      44 | @@ -47,11 +45,11 @@ def run():
      45 |  
      46 |      out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
      47 |  
      48 | -    if args.out_sdktgz:
      49 | -        out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
      50 | +    if args.out_sdkt:
      51 | +        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    


    l0rinc commented at 8:57 AM on November 25, 2025:

    are we reassigning out_sdkt_path here? Based on the condition checking args.out_sdkt, shouldn't this rather be:

        if args.out_sdkt:
            out_sdkt_path = pathlib.Path(args.out_sdkt[0])
    

    or maybe if we remove nargs=1 from above we could do:

        if args.out_sdkt:
            out_sdkt_path = pathlib.Path(args.out_sdkt)
    

    Was this working before? I don't really have experience with this, so I can't tell...


    stickies-v commented at 11:04 AM on November 26, 2025:

    Good catch @l0rinc, I didn't test this in my review, and you're right, it doesn't work (AttributeError: 'Namespace' object has no attribute 'out_sdkt_path') on master nor in the current HEAD. I also agree with removing nargs=1:

    <details> <summary>git diff on 266a2bf40e</summary>

    diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
    index e9221cf60d..bfac1542a2 100755
    --- a/contrib/macdeploy/gen-sdk.py
    +++ b/contrib/macdeploy/gen-sdk.py
    @@ -21,7 +21,7 @@ def run():
             description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
     
         parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
    -    parser.add_argument("-o", metavar='OUTSDKTAR', nargs=1, dest='out_sdkt', required=False)
    +    parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', required=False)
     
         args = parser.parse_args()
     
    @@ -46,7 +46,7 @@ def run():
         out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
     
         if args.out_sdkt:
    -        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    +        out_sdkt_path = pathlib.Path(args.out_sdkt)
         else:
             # Construct our own out_sdkt if not specified on the command line
             out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
    
    

    </details>

    Or, preferably, the relevant logic can be simplified even more:

    <details> <summary>git diff on 266a2bf40e</summary>

    diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
    index e9221cf60d..cf37929287 100755
    --- a/contrib/macdeploy/gen-sdk.py
    +++ b/contrib/macdeploy/gen-sdk.py
    @@ -20,12 +20,12 @@ def run():
         parser = argparse.ArgumentParser(
             description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
     
    -    parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
    -    parser.add_argument("-o", metavar='OUTSDKTAR', nargs=1, dest='out_sdkt', required=False)
    +    parser.add_argument('xcode_app', metavar='XCODEAPP', type=pathlib.Path)
    +    parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', type=pathlib.Path, required=False)
     
         args = parser.parse_args()
     
    -    xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
    +    xcode_app = args.xcode_app.resolve()
         assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
     
         xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
    @@ -45,11 +45,7 @@ def run():
     
         out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
     
    -    if args.out_sdkt:
    -        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    -    else:
    -        # Construct our own out_sdkt if not specified on the command line
    -        out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
    +    out_sdkt_path = args.out_sdkt or pathlib.Path("./{}.tar".format(out_name))
     
         def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
             """Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'
    
    

    </details>

  46. l0rinc commented at 9:17 AM on November 25, 2025: contributor

    I don't yet have experience with this part of the code, left a few nits. I arrived here while investigating an upgrade to Xcode 16.3 in #29415 (review) and noticed a couple of potential issues around the argparse usage and the -o handling.

    I have tested the following changes locally:

    diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
    index 426d82e46c..3bf9154887 100755
    --- a/contrib/macdeploy/gen-sdk.py
    +++ b/contrib/macdeploy/gen-sdk.py
    @@ -20,12 +20,12 @@ def run():
         parser = argparse.ArgumentParser(
             description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
     
    -    parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
    -    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
    +    parser.add_argument('xcode_app', metavar='XCODEAPP')
    +    parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', required=False)
     
         args = parser.parse_args()
     
    -    xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
    +    xcode_app = pathlib.Path(args.xcode_app).resolve()
         assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
     
         xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
    @@ -46,10 +46,10 @@ def run():
         out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
     
         if args.out_sdkt:
    -        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    +        out_sdkt_path = pathlib.Path(args.out_sdkt)
         else:
             # Construct our own out_sdkt if not specified on the command line
    -        out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
    +        out_sdkt_path = pathlib.Path(f"./{out_name}.tar")
     
         def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
             """Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'
    

    This removes the unnecessary nargs=1 (so the positional is a scalar string instead of a 1-element list), fixes the -o code path to actually use args.out_sdkt, and updates the metavar to reflect that we now produce a .tar instead of .tar.gz.


    Ran

    python3 contrib/macdeploy/gen-sdk.py /Applications/Xcode.app -o /tmp/Xcode-sdk-test.tar
    

    it shows:

    Found Xcode (version: 26.1.1, build id: 17B100) Found MacOSX SDK (version: 26.1, build id: 25B74) Creating output .tar file... Adding MacOSX SDK 26.1 files... Done! Find the resulting tarball at: /private/tmp/Xcode-sdk-test.tar

  47. fanquake force-pushed on Nov 25, 2025
  48. contrib: more selectively pick files for macOS SDK
    Only include what we really need. Skip 100s of mb of manpages,
    swiftmodules, modulemaps.
    Note that System/Library is only needed for the Qt build.
    a33d034545
  49. macdeploy: disable compression in macOS gen-sdk script
    Starting with Python 3.11, Pythons gzip might delegate to zlib.
    Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib
    implementation might differ, resulting in different output.
    
    For now, or until a better solution exists, disable compression. This
    results in the SDK increasing in size to ~157mb. Which is not
    unreasonable, to regain determinism (and would be significantly worse
    without the previous commit).
    
    See: https://docs.python.org/3/library/gzip.html#gzip.compress
    
    Co-authored-by: stickies-v <stickies-v@protonmail.com>
    c1213a35ab
  50. contrib: rename gen-sdk to gen-sdk.py
    This puts it in scope for the Python linters.
    3e01b5d0e7
  51. fanquake force-pushed on Nov 26, 2025
  52. stickies-v commented at 12:14 PM on November 26, 2025: contributor

    ACK 3e01b5d0e7 modulo the new .tar SDK being uploaded

  53. fanquake closed this on Nov 26, 2025

  54. fanquake reopened this on Nov 26, 2025

  55. fanquake commented at 7:15 PM on November 26, 2025: member

    New tarball is now available. Open/close to kick the CI.

  56. DrahtBot removed the label CI failed on Nov 26, 2025
  57. davidgumberg commented at 2:45 AM on November 27, 2025: contributor

    Tested ACK 3e01b5d0e7be3d

    This fixes #31873.

    sha256 os:ver python zlib
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 archlinux:latest 3.13.7 1.3.1
    19e29377936086e7c8079c1047f6536e9bd53a46e180a3f49441ef38aaed900d rockylinux:8 3.6.8 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 rockylinux:9 3.9.21 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:38 3.11.8 1.2.13
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:39 3.12.7 1.2.13
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:40 3.12.8 1.3.1.zlib-ng
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:41 3.13.9 1.3.1.zlib-ng
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:42 3.13.9 1.3.1.zlib-ng
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:43 3.14.0 1.3.1.zlib-ng
    8d0d3dce5da18e8eb27d02e1921c5feef971c0a7f94b8229dfe4da27f3a5b401 ubuntu:20.04 3.8.10 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:22.04 3.10.12 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:24.04 3.12.3 1.3
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:25.04 3.13.3 1.3.1
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:25.10 3.13.7 1.3.1
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 debian:11 3.9.2 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 debian:12 3.11.2 1.2.13

    Versions of python <3.9.0 are still nondeterministic but this is fine, since we have a minimum python >= 3.10.

    I bisected to this commit that fixed the non-determinism: https://github.com/python/cpython/commit/674935b8caf33e47c78f1b8e197b1b77a04992d2, prior to which python was incorrectly writing the device id of the target file to the tar header, which obviously varies between systems.

    Using the same methodology of testing this branch, I tested master and it exhibited inconsistent hashes depending on which version of zlib python was built with, as discussed here and in #31873.

    <details>

    <summary>

    master hashes

    </summary>

    sha256 os:ver python zlib
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d archlinux:latest 3.13.7 1.3.1
    be232dd9c11726b56bebd4bdc85620a6eaf77dda46f1d45de1ebfdd36ecc5d57 rockylinux:8 3.6.8 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d rockylinux:9 3.9.21 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d fedora:38 3.11.8 1.2.13
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d fedora:39 3.12.7 1.2.13
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:40 3.12.8 1.3.1.zlib-ng
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:41 3.13.9 1.3.1.zlib-ng
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:42 3.13.9 1.3.1.zlib-ng
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:43 3.14.0 1.3.1.zlib-ng
    96152d637d375e94c061ab0d8441805788755eea07600ed654eeea602a33f763 ubuntu:20.04 3.8.10 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:22.04 3.10.12 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:24.04 3.12.3 1.3
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:25.04 3.13.3 1.3.1
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:25.10 3.13.7 1.3.1
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d debian:11 3.9.2 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d debian:12 3.11.2 1.2.13

    </details>

    <details>

    <summary>

    Testing methodology

    </summary>

    Modified from the repro instructions I posted above.

    Setup

    # assumes you have acquired `Xcode_15.xip` based on the instructions in
    # contrib/macdeploy/README.md and placed it at ~/xcode/Xcode_15.xip
    XCODE=~/xcode # feel free to change this
    
    cd $XCODE
    sha256sum Xcode_15.xip
    # 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
    git clone --depth 1 https://github.com/bitcoin/bitcoin.git bitcoin-master
    cd bitcoin-master && git fetch --depth 1 origin pull/32009/head:32009 && git worktree add ../bitcoin-32009 32009 && cd ../ # fetch the pull branch
    
    git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
    python3 apple-sdk-tools/extract_xcode.py -f Xcode_15.xip | cpio -d -i # single threaded, slow, we want to reuse this.
    

    To verify that the source material for gen-sdk is good, we can generate the sdk using a known good setup on master, ubuntu 24.04:

    <details>

    <summary>Verifying the setup</summary>

    docker pull ubuntu:24.04
    docker run -it \
      -v $XCODE:/xcode \
      ubuntu:24.04 \
      /bin/bash
    

    Inside the container:

    export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
    sha256sum /xcode/Xcode_15.xip
    # 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e
    apt update
    apt install python3 -y
    
    xcode/bitcoin-master/contrib/macdeploy/gen-sdk xcode/Xcode.app/  # we are reusing the extracted result from above
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    # c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    We should get the same hash here as what contrib/macdeploy/README.md in master promises.

    </details>


    To test this branch's version of gen-sdk on a variety of platforms:

    1. Container setup

    XCODE=~/xcode # where the files from the setup above went.
    
    # Where stuff is located in the container.
    GEN_SDK="/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py"
    EXTRACTED="/xcode/Xcode.app"
    OUT="Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar"
    # If you want to test master 
    #GEN_SDK="/xcode/bitcoin-master/contrib/macdeploy/gen-sdk"
    #OUT="Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz"
    
    # The only difference between platforms is how python is installed.
    declare -A INSTALL_PYTHON=(
        [rhel]="dnf install -y python3 >/dev/null 2>&1"
        [deb]="export DEBIAN_FRONTEND=noninteractive && \
               apt-get update >/dev/null 2>&1 && \
               apt-get install -y python3 >/dev/null 2>&1"
        [arch]="pacman -Sy --noconfirm >/dev/null && \
                pacman -S --noconfirm python >/dev/null"
    )
    
    tartest() {
        docker pull "$1" -q
        docker run --rm -i \
            -v "$XCODE:/xcode" \
            "$1" /bin/bash -c "
                ${INSTALL_PYTHON[$2]}
                $GEN_SDK $EXTRACTED
                PY_VER=\$(python3 --version | awk '{print \$2}')
                ZLIB_VER=\$(python3 -c 'import zlib; print(zlib.ZLIB_VERSION)')
                HASH=\$(sha256sum $OUT | awk '{print \$1}')
                echo \"|\$HASH|$1|\$PY_VER|\$ZLIB_VER|\"
            "
    }
    
    
    # e.g. Fedora 42
    tartest fedora:42 rhel
    
    # e.g. Ubuntu 25.04
    tartest ubuntu:25.04 deb
    
    # e.g. Arch
    tartest archlinux:latest arch
    

    2. Loops to check multiple versions

    declare -A versions=(
        [fedora]="$(echo {38..43})"
        [rockylinux]="8 9"
        [ubuntu]="20.04 22.04 24.04 25.04 25.10"
        [debian]="11 12"
        [archlinux]="latest"
    )
    
    declare -A family=(
        [fedora]="rhel"
        [rockylinux]="rhel"
        [ubuntu]="deb"
        [debian]="deb"
        [archlinux]="arch"
    )
    
    echo "|sha256                                                          | os:ver | python | zlib |"
    echo "|----------------------------------------------------------------|--------|--------|------|"
    for os in "${!versions[@]}"; do
        for ver in ${versions[$os]}; do
            tartest "${os}:${ver}" ${family[$os]} | tail -n1
        done
    done
    

    3. Checking specific python versions

    XCODE=~/xcode # where the files from the setup above went.
    # specify the container platform
    PLATFORM=fedora:40 
    
    docker pull $PLATFORM && \
      docker run -it \
        -v $XCODE:/xcode \
        $PLATFORM \
        /bin/bash
    

    In the container:

    dnf install -y gcc git make openssl-devel patch xz-devel zlib-devel
    
    curl https://pyenv.run | bash
    export PYENV_ROOT="$HOME/.pyenv"
    [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
    eval "$(pyenv init - bash)"
    

    Now you can install and test with any python release e.g.

    pyenv install 3.9.0 && pyenv global 3.9.0
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
    

    We can use a pyenv build file to build a specific commit hash, e.g. to verify the commit that fixes the issue:

    echo 'install_package "badcommit" "https://github.com/python/cpython/archive/9017e0bd5e124ae6d2ed94b9e9cacb2e86270980.tar.gz" standard' > badcommit
    pyenv install badcommit && pyenv global badcommit
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
    # Nondeterministic hash!
    
    echo 'install_package "goodcommit" "https://github.com/python/cpython/archive/674935b8caf33e47c78f1b8e197b1b77a04992d2.tar.gz" standard' > goodcommit
    pyenv install goodcommit && pyenv global goodcommit
    /xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
    sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
    

    </details>

  58. fanquake commented at 2:54 PM on November 27, 2025: member

    Guix Build (x86_64):

    cc72adc0e567f33605050a98cd7cd08db2b0c7da1aa11dd5f73b9077653b5e00  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/SHA256SUMS.part
    15d23cda22ba4afceddb733820893e4ef3a5307717002403cdc5e66d3561bafb  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu-debug.tar.gz
    2d0e852259c802ea3c2411a38b93148dcaa2415b5d5d34d1f6362df39108e718  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu.tar.gz
    b9c4572af1a6312575f01414b0bb1530250c3c52da5dc53638134c399ca1048b  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/SHA256SUMS.part
    ee2d60c4b260107b36933e66d7eb8c2a1b69f323661348ac461a66056d2522d4  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf-debug.tar.gz
    900d4bc2e57448087e3bdda96873d98eeceb9ac05d48ca33843eee3c2a60e2bc  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf.tar.gz
    324e3036941d34dae95579ba7f4ca006514d2a7a99431deb8333c61d61e2a527  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/SHA256SUMS.part
    fe24816c78e64f0ac463eebd8dda589d6e64630527f67c10f97f4118380097dd  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-codesigning.tar.gz
    ddd416028f6858282d485f0017d8d777d3cee0bcd207f0d8e2542191425602cb  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.tar.gz
    8088cd6ef218acc83928bd415b88bbeb082447890e7e497ee0b28e149b0bfc59  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.zip
    1b254bdac0893af4c012b6f093ee7be5b6f5662ceb114841b16d625c1586d415  guix-build-3e01b5d0e7be/output/dist-archive/bitcoin-3e01b5d0e7be.tar.gz
    e5758de2f2acc35d2ef953b80b1e8bf618e1e157f325b5d9bfa94f74e7be9c86  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/SHA256SUMS.part
    5800c2135a6148bba763cb68f23708ef7fc62c850dac507a544381cccba49fa9  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu-debug.tar.gz
    eae287e264b5e0ae88e0aee92a520e13aa00341266239a5ec257577f4c1a2d5c  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu.tar.gz
    eafdaacbc9540f9e9706db281ec9f6db859e80dcb8e0e79efdc7e536799029d7  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/SHA256SUMS.part
    fba989ba1b4890e1161e98b05d6ebf53cd19aab5f5be44a0081cd9eed1499138  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu-debug.tar.gz
    c0e4f20087abb4ad0a58b06a7718913dd851ff46e092f95384618ad74cebff58  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu.tar.gz
    78a684a96bc9c0d901cc77601263277cf142cb20659b99451247f671b11afafc  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/SHA256SUMS.part
    04713a87ee3c418a35103f843c2d725f237a44aebc2fe28b6fbc3f5b73424f18  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-codesigning.tar.gz
    3ce4ea97fe2da7f41f0886f933ac0712528e90279386783af64b2ef0394fa2f3  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.tar.gz
    ddfdbfa347386326fd274069009ad9ed6c5e8837d7595027d6e095193e251176  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.zip
    d95954c5818d7d8036c5f8353063739e2c1fe65dc40eb1224e00a3dc3f4b56b4  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/SHA256SUMS.part
    d8cf149a5d762da733f0efdcd67d09fbde8db601d1cd4d22ee57e87a2dcf6ce2  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu-debug.tar.gz
    ffc1fb0b72f22ae574628e15dbfcd52bc37d5c37bdfd1665ecc320cca4830eae  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu.tar.gz
    ba812f2cd3b757c0096c6070e5bf072442a74a5614b1dfc92a8af39c6afbffdf  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/SHA256SUMS.part
    4dac4c0f9bc0109e2beca6ec3c14d650538d1a3271bae92bb84a4d05612665ff  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-codesigning.tar.gz
    06f08c02fc90e461ce837f4b7cc4d0a32ada0bc3a1ebec3eae4507b00e4ca8af  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-debug.zip
    ef1ce4596f1d98f684604c58c8e207e2cb30858d90232ea7e3978e03c7545a85  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-setup-unsigned.exe
    301f9ee12bb95e109b0f278e3ffaace61c5c218effdf6dc475dbb358bf4116fc  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-unsigned.zip
    
  59. fanquake merged this on Nov 27, 2025
  60. fanquake closed this on Nov 27, 2025

  61. fanquake deleted the branch on Nov 27, 2025
  62. pinheadmz commented at 1:19 PM on November 28, 2025: member

    post-merge ACK 3e01b5d0e7 instructions are clear, built mac outputs on x86 debian, my usual guix builder:

    fe24816c78e64f0ac463eebd8dda589d6e64630527f67c10f97f4118380097dd  bitcoin-3e01b5d0e7be-arm64-apple-darwin-codesigning.tar.gz
    ddd416028f6858282d485f0017d8d777d3cee0bcd207f0d8e2542191425602cb  bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.tar.gz
    8088cd6ef218acc83928bd415b88bbeb082447890e7e497ee0b28e149b0bfc59  bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.zip
    1b254bdac0893af4c012b6f093ee7be5b6f5662ceb114841b16d625c1586d415  bitcoin-3e01b5d0e7be.tar.gz
    04713a87ee3c418a35103f843c2d725f237a44aebc2fe28b6fbc3f5b73424f18  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-codesigning.tar.gz
    3ce4ea97fe2da7f41f0886f933ac0712528e90279386783af64b2ef0394fa2f3  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.tar.gz
    ddfdbfa347386326fd274069009ad9ed6c5e8837d7595027d6e095193e251176  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.zip
    
  63. alexanderwiederin referenced this in commit 7334556a3f on Dec 1, 2025
  64. fanquake referenced this in commit 338570de5c on Dec 3, 2025
  65. fanquake referenced this in commit 5408e85145 on Dec 3, 2025
  66. fanquake referenced this in commit 2256f8965e on Dec 3, 2025
  67. fanquake commented at 11:34 AM on December 3, 2025: member

    Backported to 30.x in #33997.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-02 12:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me