contrib: turn off compression of macOS SDK to fix determinism (across distros) #32009

pull fanquake wants to merge 3 commits into bitcoin:master from fanquake:macos_sdk_select changing 4 files +23 −26
  1. fanquake commented at 4:23 pm on March 6, 2025: member

    This includes three changes. The first is to more selectively pick files for inclusion into our macOS SDK tarball (skip manpages, binaries etc), which is nice because it redues the size of the tarball (from ~80mb to 20mb), and makes the size increase that happens with the next commit, less-bad.

    The second change removes compression of the tarball. Starting with Python 3.11, Pythons gzip might delegate to zlib. Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib implementation might differ, resulting in different output.

    For now, or until a better solution exists, remove compression. This results in the SDK increasing in size to ~157mb. Which is not unreasonable, to regain determinism (and would be significantly worse without the previous commit).

    See: https://docs.python.org/3/library/gzip.html#gzip.compress

    The third renames gen-sdk to gen-sdk.py, so that it will be linted, along with the rest of our Python files.

    Fixes #31873. We could probably also put this into 30.x.

  2. DrahtBot commented at 4:23 pm on March 6, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32009.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK stickies-v, davidgumberg
    Concept ACK hebasto, willcl-ark, laanwj

    If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.

  3. achow101 commented at 4:29 pm on March 6, 2025: member
    Is determinism of that tarball necessary? IIRC it doesn’t actually affect the build.
  4. fanquake commented at 4:31 pm on March 6, 2025: member

    Is determinism of that tarball necessary?

    I don’t see why we wouldn’t want it. If we don’t care about it, we should remove: “The sha256sum should be c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d.” So that issues like #31873 aren’t opened.

  5. davidgumberg commented at 9:37 pm on March 6, 2025: contributor

    Concept ACK, while it might not be strictly necessary for this tarball to be deterministic, I think matching hashes of tarballs is the simplest way to verify that you’ve followed the same procedure as others to build the macOS sdk and have gotten the same result.

    150MB seems to me a small price to pay to fix #31873 and reduce surface area for nondeterminism in generating the macOS sdk.

  6. hebasto commented at 6:58 am on March 7, 2025: member
    Concept ACK for the same reasoning as in #32009 (comment).
  7. willcl-ark commented at 10:06 am on March 7, 2025: member

    Concept ACK.

    I don’t think a few hundred MB more disk space required for (power-)users doing guix builds will be a great concern to anyone, and I agree with @davidgumberg that having as many steps in the guix workflow as possible be reproducible, is probably best.

  8. fanquake renamed this:
    RFC contrib: turn off compression of macOS SDK to fix determinism (across distros)
    contrib: turn off compression of macOS SDK to fix determinism (across distros)
    on Mar 7, 2025
  9. DrahtBot added the label Scripts and tools on Mar 7, 2025
  10. laanwj commented at 5:29 am on March 8, 2025: member

    Concept ACK, imo determinism is good because it makes it possible to check if mistakes have been made in the process at earlier steps.

    One comment i was about to give is “why use a .tar.gz instead of .tar, that’s the only way to be sure to remove dependency on zlib”, but that’s a much more spread out code change. This is most likely fine.

    edit: Could do that when we have to upgrade the MacOS SDK anyway, but definitely not for 29.0

  11. Sjors commented at 3:33 pm on March 10, 2025: member

    I’m not too worried about the size increase. A typical guix build eats 30GB on my Ubuntu VM (3 GB for the outputs).

    it makes it possible to check if mistakes have been made in the process at earlier steps.

    Agreed.

    It seems simple enough to backport, especially because we’re not bumping the version.

    I’ll see if I can reproduce the new hashes.

  12. Sjors commented at 3:43 pm on March 10, 2025: member

    One slight problem is that Apple no longer makes XCode 15.0 available for download. They do have Xcode 15.0.1: https://download.developer.apple.com/Developer_Tools/Xcode_15.0.1/Xcode_15.0.1.xip

    I still have the original xip on one of my machines, I’ll upload it somewhere for testing. But we should probably bump it.

    Here you go: https://download.sprovoost.nl/download.php?id=21&token=ee9af1cfea8fa495cc842a407249126d

    (sha256 checksum is in contrib/macdeploy/README.md)

    Update: XCode 15 is still there, see below.


    I was able to reproduce the hash for the first commit 6998e933f935a379c3ad55c2fb16eca9b854f40b, but not for the second commit 20778eb0235df70397fc285f9e3b72270bd4aaf4. I get 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82 instead on macOS 15.3.1, Python 3.10.14. Ditto on my Ubuntu 24.04 VM inside the same machine.


    It looks like Apple Silicon has hardware acceleration for zip, maybe that’s a factor? (and would also explain why extract_xcode.py is super fast on my VM running on the M4 mac, while super slow on the AMD Ryzen 7950x). (probably not, because the first commit did produce identical results)


    I also get 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82 for Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz on the AMD Ubuntu machine.

  13. davidgumberg commented at 0:13 am on March 11, 2025: contributor

    One slight problem is that Apple no longer makes XCode 15.0 available for download. They do have Xcode 15.0.1: https://download.developer.apple.com/Developer_Tools/Xcode_15.0.1/Xcode_15.0.1.xip

    I’m still seeing it here: https://developer.apple.com/download/all/?q=Xcode%2015 , and with this download link: https://download.developer.apple.com/Developer_Tools/Xcode_15/Xcode_15.xip, it’s labeled Xcode 15 rather than Xcode 15.0.

  14. Sjors commented at 8:14 am on March 11, 2025: member

    @davidgumberg ah indeed, I see it as well. You have to click on the download link in order to for the download URL to work at all, so that’s probably why it didn’t work for me.

    So that just leaves the hash mismatch to figure out.

  15. willcl-ark commented at 1:11 pm on March 19, 2025: member

    At commit 20778eb0235df70397fc285f9e3b72270bd4aaf4 I get the following guix output:

     0env HOSTS="x86_64-apple-darwin arm64-apple-darwin" ./contrib/guix/guix-build
     1
     2<snip>
     3
     4$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
     530aa32906f879c5347be27bd9cb1a65b3d839acd72805fe3185276f788971fdb  guix-build-20778eb0235d/output/arm64-apple-darwin/SHA256SUMS.part
     60763a089bcc5d1d22191414f556bf461cefa8ea8a18971b171cf6fc5570b8790  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
     718cd712ac4cbd63122483a083a9f29d7c184ffaead45078fb27b3d6381b691a6  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
     8c3fdc3b50077f2a4b772d4c1cf50e230f358542844e800e77194bf85dee9eb28  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.zip
     9b97da95ff464b8bdcb9ce31dc783f947dd24d93eb6bfdfea71c3070d51991058  guix-build-20778eb0235d/output/dist-archive/bitcoin-20778eb0235d.tar.gz
    10e180a9ef657ea3f0a6e42c4babe7b38a8e585902fbac301a1f16bccb0c53a18b  guix-build-20778eb0235d/output/x86_64-apple-darwin/SHA256SUMS.part
    115b032a2b6dd2dad11f3884194bcb85aea37237246162dfad27099e6c5175071d  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
    1239d23ec95f34cd7d8bbdb6228e55f7f8a0f48a3997c3dcb146a2ff3744f37629  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz
    1376e36fa1af2f788dea631a7fcc93451fdf794a40dded0e3c7c12f74ab136aefb  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.zip
    14
    15$ eza -al guix-build-20778eb0235d/output/*/*apple*.tar.gz
    16.rw-r--r-- 52M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
    17.rw-r--r-- 35M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
    18.rw-r--r-- 56M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
    19.rw-r--r-- 38M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz
    
  16. davidgumberg commented at 5:37 am on March 22, 2025: contributor

    I don’t have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

    pkgdiff again reports the mismatching archives as identical. But, now that the archives are uncompressed, the binary diff output is a little more revealing, here’s the start of the diff:

    git diff <(xxd rocky9.3/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(xxd fedora41/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)

     0@@ -2050,7 +2050,7 @@
     1 00008010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     2 00008020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     3 00008030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
     4-00008040: 0000 0000 0000 0000 00a0 815f 7e58 636f  ..........._~Xco
     5+00008040: 0000 0000 0000 0000 0000 d0ff 2f58 636f  ............/Xco
     6 00008050: 6465 2d31 352e 302d 3135 4132 3430 642d  de-15.0-15A240d-
     7 00008060: 6578 7472 6163 7465 642d 5344 4b2d 7769  extracted-SDK-wi
     8 00008070: 7468 2d6c 6962 6378 782d 6865 6164 6572  th-libcxx-header
     9@@ -4124,30 +4124,30 @@
    10 000101b0: 6129 2041 5050 4c45 5f41 5243 4849 5645  a) APPLE_ARCHIVE
    11 000101c0: 5f53 5749 4654 5f50 5249 5641 5445 3b0a  _SWIFT_PRIVATE;.
    12 000101d0: 0a23 6966 6465 6620 5f5f 6370 6c75 7370  .#ifdef __cplusp
    13-000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 608e  lus.}.#endif..`.
    14-000101f0: 9f71 0000 0000 0000 0000 0000 0000 0000  .q..............
    15+000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 0000  lus.}.#endif....
    16+000101f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    17 00010200: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    18 00010210: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    19 00010220: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    20 00010230: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    21-00010240: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    22-00010250: 0000 5863 6f64 652d 3135 2e30 2d31 3541  ..Xcode-15.0-15A
    23-00010260: 3234 3064 2d65 7874 7261 6374 6564 2d53  240d-extracted-S
    24-00010270: 444b 2d77 6974 682d 6c69 6263 7878 2d68  DK-with-libcxx-h
    25-00010280: 6561 6465 7273 2f75 7372 2f69 6e63 6c75  eaders/usr/inclu
    26-00010290: 6465 2f41 7070 6c65 4172 6368 6976 652f  de/AppleArchive/
    27-000102a0: 4141 456e 7472 7958 4154 426c 6f62 2e68  AAEntryXATBlob.h
    28-000102b0: 0000 0000 0000 3030 3030 3634 3400 3030  ......0000644.00
    29-000102c0: 3030 3030 3000 3030 3030 3030 3000 3030  00000.0000000.00
    30-000102d0: 3030 3030 3135 3437 3400 3030 3030 3030  000015474.000000
    31-000102e0: 3030 3030 3000 3032 3631 3631 0020 3000  00000.026161. 0.
    32+00010240: 0000 0000 0000 0000 0000 0000 0058 636f  .............Xco
    33+00010250: 6465 2d31 352e 302d 3135 4132 3430 642d  de-15.0-15A240d-
    34+00010260: 6578 7472 6163 7465 642d 5344 4b2d 7769  extracted-SDK-wi
    35+00010270: 7468 2d6c 6962 6378 782d 6865 6164 6572  th-libcxx-header
    36+00010280: 732f 7573 722f 696e 636c 7564 652f 4170  s/usr/include/Ap
    37+00010290: 706c 6541 7263 6869 7665 2f41 4145 6e74  pleArchive/AAEnt
    38+000102a0: 7279 5841 5442 6c6f 622e 6800 0000 0000  ryXATBlob.h.....
    39+000102b0: 0030 3030 3036 3434 0030 3030 3030 3030  .0000644.0000000
    40+000102c0: 0030 3030 3030 3030 0030 3030 3030 3031  .0000000.0000001
    41+000102d0: 3534 3734 0030 3030 3030 3030 3030 3030  5474.00000000000
    42+000102e0: 0030 3236 3136 3100 2030 0000 0000 0000  .026161. 0......
    

    Testing methodology

    Modified from the repro instructions in #31873 to be faster, most importantly reusing the result of the lengthy extraction step, and just downloading the relevant branches once.

    Setup phase:

     0# assumes you have acquired `Xcode_15.xip` based on the instructions in
     1# contrib/macdeploy/README.md and placed it at ~/xcode/Xcode_15.xip
     2XCODE=~/xcode # feel free to change this
     3
     4cd $XCODE
     5sha256sum Xcode_15.xip
     6# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
     7git clone --depth 1 https://github.com/bitcoin/bitcoin.git bitcoin-master
     8cd bitcoin-master && git fetch --depth 1 origin pull/32009/head:32009 && git worktree add ../bitcoin-32009 32009 && cd ../ # fetch the pull branch
     9
    10git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
    11python3 apple-sdk-tools/extract_xcode.py -f Xcode_15.xip | cpio -d -i # single threaded, slow, we want to reuse this.
    

    To verify that the source material for gen-sdk is good, we can generate the sdk using a known good setup, ubuntu 24.04:

    0docker pull ubuntu:24.04
    1docker run -it \
    2  -v $XCODE:/xcode \
    3  ubuntu:24.04 \
    4  /bin/bash
    

    Inside the container:

    0export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
    1sha256sum /xcode/Xcode_15.xip
    2# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e
    3apt update
    4apt install python3 -y
    5
    6xcode/bitcoin-master/contrib/macdeploy/gen-sdk xcode/Xcode.app/  # we are reusing the extracted result from above
    7sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    8# c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    We should get the same hash here as what contrib/macdeploy/README.md in master promises.


    To test this branch’s version of gen-sdk on a variety of platforms:

    1. Container setup

    0XCODE=~/xcode # where the files from the setup above went.
    1# specify the container platform
    2PLATFORM=fedora:40 
    3
    4docker pull $PLATFORM && \
    5  docker run -it \
    6    -v $XCODE:/xcode \
    7    $PLATFORM \
    8    /bin/bash
    

    2. In the container

    Compare the output of the final sha256sum to what contrib/macdeploy/README.md in this branch promises.

    Debian/Ubuntu
    0export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
    1apt update > /dev/null
    2apt install python3 -y > /dev/null
    3/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    4sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    Fedora/CentOS

    0dnf install python -y --quiet # python3 on rocky8.9
    1/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    2sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    Arch

    0pacman -Sy
    1pacman --noconfirm -S python
    2/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    3sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    Results Tables

    Success

    image python --version python -c "import zlib; print(zlib.ZLIB_VERSION)"
    fedora:39 3.12.7 1.2.13
    fedora:40 3.12.9 1.3.1-zlib-ng
    fedora:41 3.13.2 1.3.1-zlib-ng
    ubuntu:24.04 3.12.3 1.3
    ubuntu:24.10 3.12.7 1.3.1
    archlinux:latest 3.13.2 1.3.1

    Failed to reproduce:

    image python --version python -c "import zlib; print(zlib.ZLIB_VERSION)" hash
    debian:bookworm 3.11.2 1.2.13 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82
    rockylinux:8.9 3.6.8 1.2.11 e779914636e6a3a417bf2a19dbce6f0bf8fab10b16717df769d107a5aad6aa2e
    rockylinux:9.3 3.9.18 1.2.11 07b12c2a489c241bbc8c853fe78f2e92faf8ff51631311d142aeb8c7e20e7268

    I downgraded to python 3.11.3 on fedora 41 with zlib 1.3, and got a bad hash, upgraded to python 3.12.0 with zlib-ng 1.3 on debian:bookworm and got a good hash.

    Also see #31873 (comment)

     0dnf install -y gcc git make openssl-devel xz-devel git
     1curl -L https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.gz | tar xzvf -
     2cd zlib-1.3 && ./configure && make -j $(nproc) && make install && cd ..
     3
     4curl https://pyenv.run | bash
     5export PYENV_ROOT="$HOME/.pyenv"
     6[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
     7eval "$(pyenv init - bash)"
     8
     9pyenv install 3.11.3
    10pyenv global 3.11.3
    11
    12# verify zlib version
    13python -c "import zlib; print(zlib.ZLIB_VERSION)"
    14# 1.3
    15
    16/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    17sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    18# get something bad
    19
    20pyenv install 3.12.0
    21pyenv global 3.12.0
    22/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
    23sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    Don’t have super strong evidence for this, but I suspect that this might be a second source of nondeterministic behavior that was already present before, as in my previous testing of master when I reported this issue, when I was trying different combinations of python and zlib, I also saw a similar pattern where when changing to random python, some of the hashes were randomly mismatched, and the rest were either of two hashes, one right and one wrong. I avoided sharing the random hashes in my table in #31873, and all of the values I share are for python 3.12.x

  17. maflcko commented at 6:05 am on March 22, 2025: member

    I don’t have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

    A wild guess: Could this be related to https://github.com/python/cpython/issues/95385?

  18. laanwj commented at 4:07 pm on March 26, 2025: member

    git diff <(xxd rocky9.3/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(xxd fedora41/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)

    This seems to suggest the divergence is in the tarring, not the gzipping. Do you still see a difference if you gunzip the .gzs and compare the tars only?

  19. luke-jr commented at 11:29 pm on April 9, 2025: member
    Maybe keep the filename the same, so people can optionally compress it on their own (or even use the current file) without breaking the build?
  20. fanquake force-pushed on Jun 2, 2025
  21. fanquake commented at 3:19 pm on June 2, 2025: member
    Rebased and added a change to skip more files to the first commit, which reduces the final uncompressed size from ~230mb to ~157mb.
  22. Sjors commented at 5:47 am on June 3, 2025: member

    nit: can drop “RFC:” from the second commit?

    I was able to reproduce the hash for the first commit 6998e93, but not for the second commit 20778eb.

    Tested this again against a2bf3b03b19a0e668bf86eea7de3e3a592b4b23f and fc81a759413230daf3e05b448b357e62915bb3ba on my M4 machine running macOS 15.5 with Python 3.10.14 (via PyEnv). Again the hash matches in the first commit, but with second commit I get:

    0fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    If I use Python 3.11.11 instead I get the same hash as with 3.10.14. @davidgumberg wrote:

    I don’t have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

    With Python 3.12.9 I indeed get ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6.

    We could bump the version in .python-version and/or just mention a minimum Python version for this tool? The 3.10 security fix maintenance ends before our v30 maintenance window, and they’re already not fixing (non-security) bugs: https://devguide.python.org/versions/

  23. fanquake force-pushed on Jun 3, 2025
  24. fanquake commented at 10:09 am on June 3, 2025: member

    nit: can drop “RFC:” from the second commit?

    Dropped.

  25. maflcko commented at 10:45 am on June 3, 2025: member

    We could bump the version in .python-version and/or just mention a minimum Python version for this tool?

    Wholesale bumping to 3.12 seems a bit early. Maybe just add an option to this tool --force-non-determinstic-old-python to enable 3.10 and 3.11 and otherwise use 3.12+ by default?

  26. achow101 requested review from achow101 on Oct 22, 2025
  27. achow101 requested review from stickies-v on Oct 22, 2025
  28. achow101 requested review from davidgumberg on Oct 22, 2025
  29. stickies-v commented at 12:23 pm on October 29, 2025: contributor

    utACK 71b0f474eee3aceb8c17b9ffa7fa866ecb3a93af, Concept ACK on disabling compression to increase determinism.

    71b0f474eee3aceb8c17b9ffa7fa866ecb3a93af seems like a strict win, no need to include what we don’t use. I think conceptually 0a9b63b1f447bc31e6b56410f7c9c25abc7f5047 also makes sense: the increased size of the tarball should not be a blocked for increased determinism. However, on my machines (arm64 macOS 15.6 and x86_64 Debian Bookworm ) determinism seems to be reduced without compression across Python versions (as also reported by other reviewers). I’m not sure whether increased determinism across distros is more important than determinism across (supported) Python versions?

    (note: below results are from macOS, the hashes are the same for Debian but the filesizes vary a bit)

    With selective picking and no compression (0a9b63b1f447bc31e6b56410f7c9c25abc7f5047) different sha256sum hashes are produced by python 3.{10,11}, 3.{12, 13} and 3.{14}:

    0py10: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
    1py11: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
    2py12: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
    3py13: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
    4py14: ea4684c568bbf63ca9fa4d1a9116751ba5b870728c4e9f72862f1b0d453ef05d (161mb)
    

    With selective picking and compression (71b0f474eee3aceb8c17b9ffa7fa866ecb3a93af) the results are consistent:

    0py10: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    1py11: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    2py12: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    3py13: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    4py14: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
    

    On master (e872a566f251c73908de8b6d243c94a6679c2eac) the results are consistent:

    0py10: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    1py11: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    2py12: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    3py13: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    4py14: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
    
  30. achow101 commented at 11:09 pm on November 17, 2025: member

    One comment i was about to give is “why use a .tar.gz instead of .tar, that’s the only way to be sure to remove dependency on zlib”, but that’s a much more spread out code change.

    I don’t think that’s true. Our guix build documentation states that the tarball needs to be extracted into depends/SDK (or some directory that SDK_PATH points to), and depends/hosts/darwin.mk uses the path to the extracted directory, not the tarball itself. Other than in the docs, I don’t think the tarball itself is ever handled by our build scripts. So it seems to me that dropping the gzip altogether, and just going with a .tar would resolve this problem with minimal issues.

  31. fanquake force-pushed on Nov 18, 2025
  32. fanquake commented at 12:06 pm on November 18, 2025: member
    I think this could actually be a fine solution, and the additional changes needed are minimal, so I’ve pushed that up. The hash of the tarball using this branch is 95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498. The size is ~157mb.
  33. fanquake commented at 12:32 pm on November 18, 2025: member

    Guix Build (aarch64):

    03bd9c90945da34a1398beceb2474401a486a6f3436bb4a12a573bdfa38df6d62  guix-build-5513cd0941a2/output/arm64-apple-darwin/SHA256SUMS.part
    1e78e73b65107c95fd4199f5a354471e8f61165c51ee7602b775a59093cf345f5  guix-build-5513cd0941a2/output/arm64-apple-darwin/bitcoin-5513cd0941a2-arm64-apple-darwin-codesigning.tar.gz
    23fee841b5625963dda994f68035e2d87decbbce4df666485e76f42fbe2b7f61b  guix-build-5513cd0941a2/output/arm64-apple-darwin/bitcoin-5513cd0941a2-arm64-apple-darwin-unsigned.tar.gz
    36ec06c295b6881c0b06df9e3ec28f873b273de1d4a903a3d8e5872131895c3fd  guix-build-5513cd0941a2/output/arm64-apple-darwin/bitcoin-5513cd0941a2-arm64-apple-darwin-unsigned.zip
    4346be922613989dbcbbc8ca5ba3ae476309388d91432788e8cec20aecf4fb7dd  guix-build-5513cd0941a2/output/dist-archive/bitcoin-5513cd0941a2.tar.gz
    5f69af9fb3949cb815e1868bc0a20147e918611c3221fa36bed48060aa16d7372  guix-build-5513cd0941a2/output/x86_64-apple-darwin/SHA256SUMS.part
    66cb7f0f296bdeaef38f125d2d2ad0bbf2620af67e433d993e793a6be152ce3e3  guix-build-5513cd0941a2/output/x86_64-apple-darwin/bitcoin-5513cd0941a2-x86_64-apple-darwin-codesigning.tar.gz
    7f5f6fef78e138adf3d211d9ce1d1ccce461970843ff63bea443cc6757c4ebdca  guix-build-5513cd0941a2/output/x86_64-apple-darwin/bitcoin-5513cd0941a2-x86_64-apple-darwin-unsigned.tar.gz
    8cf936dbd8fa32cfdc0ef97a6f05893c3f2c98ffa74e7809d5136282bffa141a5  guix-build-5513cd0941a2/output/x86_64-apple-darwin/bitcoin-5513cd0941a2-x86_64-apple-darwin-unsigned.zip
    
  34. in contrib/macdeploy/gen-sdk.py:1 in 5513cd0941 outdated


    stickies-v commented at 1:51 pm on November 18, 2025:

    nit: could be cleaned up a bit more

     0diff --git a/contrib/macdeploy/gen-sdk b/contrib/macdeploy/gen-sdk
     1index 0cfd2b1379..426d82e46c 100755
     2--- a/contrib/macdeploy/gen-sdk
     3+++ b/contrib/macdeploy/gen-sdk
     4@@ -2,9 +2,7 @@
     5 import argparse
     6 import plistlib
     7 import pathlib
     8-import sys
     9 import tarfile
    10-import gzip
    11 import os
    12 import contextlib
    13 
    14@@ -23,7 +21,7 @@ def run():
    15         description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
    16 
    17     parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
    18-    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
    19+    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
    20 
    21     args = parser.parse_args()
    22 
    23@@ -47,11 +45,11 @@ def run():
    24 
    25     out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
    26 
    27-    if args.out_sdktgz:
    28-        out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
    29+    if args.out_sdkt:
    30+        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    31     else:
    32-        # Construct our own out_sdktgz if not specified on the command line
    33-        out_sdktgz_path = pathlib.Path("./{}.tar".format(out_name))
    34+        # Construct our own out_sdkt if not specified on the command line
    35+        out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
    36 
    37     def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
    38         """Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'
    39@@ -88,12 +86,12 @@ def run():
    40             tarfp.add("./System/Library/Frameworks", recursive=True, filter=change_tarinfo_base)
    41 
    42     print("Creating output .tar file...")
    43-    with out_sdktgz_path.open("wb") as fp:
    44+    with out_sdkt_path.open("wb") as fp:
    45         with tarfile.open(mode="w", fileobj=fp, format=tarfile.PAX_FORMAT) as tarfp:
    46             print("Adding MacOSX SDK {} files...".format(sdk_version))
    47             tarfp_add_with_base_change(tarfp, sdk_dir, out_name)
    48     print("Done! Find the resulting tarball at:")
    49-    print(out_sdktgz_path.resolve())
    50+    print(out_sdkt_path.resolve())
    51 
    52 if __name__ == '__main__':
    53     run()
    

    stickies-v commented at 2:09 pm on November 18, 2025:

    The CI script is still using the .tar.gz name (which is fine as long as that file is available on SDK_URL): https://github.com/bitcoin/bitcoin/blob/2444488f6ad32dcbbed51a73cd4f59ff3a239e32/ci/test/01_base_install.sh#L93

    Perhaps we can temporarily add both the .tar.gz and .tar files to the SDK_URL and update all of this in one go?


    maflcko commented at 2:18 pm on November 18, 2025:

    Not sure it this is possible, but if the file was renamed, the existing linters will pick up and lint this file:

    0git mv ./contrib/macdeploy/gen-sdk ./contrib/macdeploy/gen-sdk.py
    

    fanquake commented at 2:22 pm on November 18, 2025:

    Perhaps we can temporarily add both the

    We will upload the new SDK before this PR is merged (leaving the existing one in place), I don’t think we need interim code changes here.


    fanquake commented at 2:36 pm on November 18, 2025:
    Have renamed to gen-sdk.py. Tested that the linter works locally by reintroduing rogue import time.

    fanquake commented at 2:36 pm on November 18, 2025:
    Took the rest of the suggestions.

    stickies-v commented at 2:46 pm on November 18, 2025:

    We will upload the new SDK before this PR is merged (leaving the existing one in place), I don’t think we need interim code changes here.

    Yeah that’s what I meant, but I now see how my phrasing was ambiguous. The “temporary” referred to having both versions on the server, until the gz is removed after this PR is merged. Thanks.


    maflcko commented at 2:50 pm on November 18, 2025:
    i think the gz will be needed for old release branches?

    fanquake commented at 2:51 pm on November 18, 2025:
    Yea, the .tar.gz will be left as-is.
  35. stickies-v approved
  36. stickies-v commented at 2:11 pm on November 18, 2025: contributor

    tACK 5513cd0941a2300c0b78758d980ef5eee5079b4c

    Determinism across Python versions (3.10-3.14) is now restored on my machine. New approach of just avoiding gzip entirely seems straightforward and preferable.

  37. DrahtBot requested review from laanwj on Nov 18, 2025
  38. DrahtBot requested review from hebasto on Nov 18, 2025
  39. DrahtBot requested review from willcl-ark on Nov 18, 2025
  40. fanquake force-pushed on Nov 18, 2025
  41. stickies-v commented at 2:59 pm on November 18, 2025: contributor
    ACK e7c22c8eb446727df00d1cb2ce554be9324c7fbd once the new .tar SDK is uploaded
  42. DrahtBot added the label CI failed on Nov 18, 2025
  43. in contrib/macdeploy/gen-sdk.py:24 in e7c22c8eb4
    20@@ -23,7 +21,7 @@ def run():
    21         description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
    22 
    23     parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
    24-    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
    25+    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
    


    l0rinc commented at 8:28 am on November 25, 2025:

    should we adjust the help description as well?

    0    parser.add_argument("-o", metavar='OUTSDKTAR', nargs=1, dest='out_sdkt', required=False)
    

    usage: gen-sdk.py [-h] [-o OUTSDKTAR] XCODEAPP


    fanquake commented at 11:38 am on November 25, 2025:
    Done.
  44. in contrib/macdeploy/gen-sdk.py:69 in e7c22c8eb4
    65@@ -68,6 +66,8 @@ def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
    66 
    67         """
    68         def change_tarinfo_base(tarinfo):
    69+            if tarinfo.name and tarinfo.name.endswith(("swiftmodule", "modulemap")):
    


    l0rinc commented at 8:49 am on November 25, 2025:

    are these file extensions? I wasn’t familiar with them, but e.g. https://medium.com/@mail2ashislaha/swift-objective-c-interoperability-static-libraries-modulemap-etc-39caa77ce1fc indicates they’re extensions, in which case:

    0            if tarinfo.name and tarinfo.name.endswith((".swiftmodule", ".modulemap")):
    

    fanquake commented at 11:38 am on November 25, 2025:
    Some of these are folders, but this will also work.
  45. in contrib/macdeploy/gen-sdk.py:49 in e7c22c8eb4 outdated
    44@@ -47,11 +45,11 @@ def run():
    45 
    46     out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
    47 
    48-    if args.out_sdktgz:
    49-        out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
    50+    if args.out_sdkt:
    51+        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    


    l0rinc commented at 8:57 am on November 25, 2025:

    are we reassigning out_sdkt_path here? Based on the condition checking args.out_sdkt, shouldn’t this rather be:

    0    if args.out_sdkt:
    1        out_sdkt_path = pathlib.Path(args.out_sdkt[0])
    

    or maybe if we remove nargs=1 from above we could do:

    0    if args.out_sdkt:
    1        out_sdkt_path = pathlib.Path(args.out_sdkt)
    

    Was this working before? I don’t really have experience with this, so I can’t tell…


    stickies-v commented at 11:04 am on November 26, 2025:

    Good catch @l0rinc, I didn’t test this in my review, and you’re right, it doesn’t work (AttributeError: 'Namespace' object has no attribute 'out_sdkt_path') on master nor in the current HEAD. I also agree with removing nargs=1:

     0diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
     1index e9221cf60d..bfac1542a2 100755
     2--- a/contrib/macdeploy/gen-sdk.py
     3+++ b/contrib/macdeploy/gen-sdk.py
     4@@ -21,7 +21,7 @@ def run():
     5         description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
     6 
     7     parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
     8-    parser.add_argument("-o", metavar='OUTSDKTAR', nargs=1, dest='out_sdkt', required=False)
     9+    parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', required=False)
    10 
    11     args = parser.parse_args()
    12 
    13@@ -46,7 +46,7 @@ def run():
    14     out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
    15 
    16     if args.out_sdkt:
    17-        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    18+        out_sdkt_path = pathlib.Path(args.out_sdkt)
    19     else:
    20         # Construct our own out_sdkt if not specified on the command line
    21         out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
    

    Or, preferably, the relevant logic can be simplified even more:

     0diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
     1index e9221cf60d..cf37929287 100755
     2--- a/contrib/macdeploy/gen-sdk.py
     3+++ b/contrib/macdeploy/gen-sdk.py
     4@@ -20,12 +20,12 @@ def run():
     5     parser = argparse.ArgumentParser(
     6         description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
     7 
     8-    parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
     9-    parser.add_argument("-o", metavar='OUTSDKTAR', nargs=1, dest='out_sdkt', required=False)
    10+    parser.add_argument('xcode_app', metavar='XCODEAPP', type=pathlib.Path)
    11+    parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', type=pathlib.Path, required=False)
    12 
    13     args = parser.parse_args()
    14 
    15-    xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
    16+    xcode_app = args.xcode_app.resolve()
    17     assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
    18 
    19     xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
    20@@ -45,11 +45,7 @@ def run():
    21 
    22     out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
    23 
    24-    if args.out_sdkt:
    25-        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    26-    else:
    27-        # Construct our own out_sdkt if not specified on the command line
    28-        out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
    29+    out_sdkt_path = args.out_sdkt or pathlib.Path("./{}.tar".format(out_name))
    30 
    31     def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
    32         """Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'
    
  46. l0rinc commented at 9:17 am on November 25, 2025: contributor

    I don’t yet have experience with this part of the code, left a few nits. I arrived here while investigating an upgrade to Xcode 16.3 in #29415 (review) and noticed a couple of potential issues around the argparse usage and the -o handling.

    I have tested the following changes locally:

     0diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
     1index 426d82e46c..3bf9154887 100755
     2--- a/contrib/macdeploy/gen-sdk.py
     3+++ b/contrib/macdeploy/gen-sdk.py
     4@@ -20,12 +20,12 @@ def run():
     5     parser = argparse.ArgumentParser(
     6         description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
     7 
     8-    parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
     9-    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
    10+    parser.add_argument('xcode_app', metavar='XCODEAPP')
    11+    parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', required=False)
    12 
    13     args = parser.parse_args()
    14 
    15-    xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
    16+    xcode_app = pathlib.Path(args.xcode_app).resolve()
    17     assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
    18 
    19     xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
    20@@ -46,10 +46,10 @@ def run():
    21     out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
    22 
    23     if args.out_sdkt:
    24-        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
    25+        out_sdkt_path = pathlib.Path(args.out_sdkt)
    26     else:
    27         # Construct our own out_sdkt if not specified on the command line
    28-        out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
    29+        out_sdkt_path = pathlib.Path(f"./{out_name}.tar")
    30 
    31     def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
    32         """Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'
    

    This removes the unnecessary nargs=1 (so the positional is a scalar string instead of a 1-element list), fixes the -o code path to actually use args.out_sdkt, and updates the metavar to reflect that we now produce a .tar instead of .tar.gz.


    Ran

    0python3 contrib/macdeploy/gen-sdk.py /Applications/Xcode.app -o /tmp/Xcode-sdk-test.tar
    

    it shows:

    Found Xcode (version: 26.1.1, build id: 17B100) Found MacOSX SDK (version: 26.1, build id: 25B74) Creating output .tar file… Adding MacOSX SDK 26.1 files… Done! Find the resulting tarball at: /private/tmp/Xcode-sdk-test.tar

  47. fanquake force-pushed on Nov 25, 2025
  48. contrib: more selectively pick files for macOS SDK
    Only include what we really need. Skip 100s of mb of manpages,
    swiftmodules, modulemaps.
    Note that System/Library is only needed for the Qt build.
    a33d034545
  49. macdeploy: disable compression in macOS gen-sdk script
    Starting with Python 3.11, Pythons gzip might delegate to zlib.
    Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib
    implementation might differ, resulting in different output.
    
    For now, or until a better solution exists, disable compression. This
    results in the SDK increasing in size to ~157mb. Which is not
    unreasonable, to regain determinism (and would be significantly worse
    without the previous commit).
    
    See: https://docs.python.org/3/library/gzip.html#gzip.compress
    
    Co-authored-by: stickies-v <stickies-v@protonmail.com>
    c1213a35ab
  50. contrib: rename gen-sdk to gen-sdk.py
    This puts it in scope for the Python linters.
    3e01b5d0e7
  51. fanquake force-pushed on Nov 26, 2025
  52. stickies-v commented at 12:14 pm on November 26, 2025: contributor
    ACK 3e01b5d0e7 modulo the new .tar SDK being uploaded
  53. fanquake closed this on Nov 26, 2025

  54. fanquake reopened this on Nov 26, 2025

  55. fanquake commented at 7:15 pm on November 26, 2025: member
    New tarball is now available. Open/close to kick the CI.
  56. DrahtBot removed the label CI failed on Nov 26, 2025
  57. davidgumberg commented at 2:45 am on November 27, 2025: contributor

    Tested ACK 3e01b5d0e7be3d

    This fixes #31873.

    sha256 os:ver python zlib
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 archlinux:latest 3.13.7 1.3.1
    19e29377936086e7c8079c1047f6536e9bd53a46e180a3f49441ef38aaed900d rockylinux:8 3.6.8 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 rockylinux:9 3.9.21 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:38 3.11.8 1.2.13
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:39 3.12.7 1.2.13
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:40 3.12.8 1.3.1.zlib-ng
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:41 3.13.9 1.3.1.zlib-ng
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:42 3.13.9 1.3.1.zlib-ng
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:43 3.14.0 1.3.1.zlib-ng
    8d0d3dce5da18e8eb27d02e1921c5feef971c0a7f94b8229dfe4da27f3a5b401 ubuntu:20.04 3.8.10 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:22.04 3.10.12 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:24.04 3.12.3 1.3
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:25.04 3.13.3 1.3.1
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:25.10 3.13.7 1.3.1
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 debian:11 3.9.2 1.2.11
    95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 debian:12 3.11.2 1.2.13

    Versions of python <3.9.0 are still nondeterministic but this is fine, since we have a minimum python >= 3.10.

    I bisected to this commit that fixed the non-determinism: https://github.com/python/cpython/commit/674935b8caf33e47c78f1b8e197b1b77a04992d2, prior to which python was incorrectly writing the device id of the target file to the tar header, which obviously varies between systems.

    Using the same methodology of testing this branch, I tested master and it exhibited inconsistent hashes depending on which version of zlib python was built with, as discussed here and in #31873.

    master hashes

    sha256 os:ver python zlib
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d archlinux:latest 3.13.7 1.3.1
    be232dd9c11726b56bebd4bdc85620a6eaf77dda46f1d45de1ebfdd36ecc5d57 rockylinux:8 3.6.8 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d rockylinux:9 3.9.21 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d fedora:38 3.11.8 1.2.13
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d fedora:39 3.12.7 1.2.13
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:40 3.12.8 1.3.1.zlib-ng
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:41 3.13.9 1.3.1.zlib-ng
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:42 3.13.9 1.3.1.zlib-ng
    5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:43 3.14.0 1.3.1.zlib-ng
    96152d637d375e94c061ab0d8441805788755eea07600ed654eeea602a33f763 ubuntu:20.04 3.8.10 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:22.04 3.10.12 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:24.04 3.12.3 1.3
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:25.04 3.13.3 1.3.1
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:25.10 3.13.7 1.3.1
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d debian:11 3.9.2 1.2.11
    c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d debian:12 3.11.2 1.2.13

    Testing methodology

    Modified from the repro instructions I posted above.

    Setup

     0# assumes you have acquired `Xcode_15.xip` based on the instructions in
     1# contrib/macdeploy/README.md and placed it at ~/xcode/Xcode_15.xip
     2XCODE=~/xcode # feel free to change this
     3
     4cd $XCODE
     5sha256sum Xcode_15.xip
     6# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
     7git clone --depth 1 https://github.com/bitcoin/bitcoin.git bitcoin-master
     8cd bitcoin-master && git fetch --depth 1 origin pull/32009/head:32009 && git worktree add ../bitcoin-32009 32009 && cd ../ # fetch the pull branch
     9
    10git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
    11python3 apple-sdk-tools/extract_xcode.py -f Xcode_15.xip | cpio -d -i # single threaded, slow, we want to reuse this.
    

    To verify that the source material for gen-sdk is good, we can generate the sdk using a known good setup on master, ubuntu 24.04:

    0docker pull ubuntu:24.04
    1docker run -it \
    2  -v $XCODE:/xcode \
    3  ubuntu:24.04 \
    4  /bin/bash
    

    Inside the container:

    0export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
    1sha256sum /xcode/Xcode_15.xip
    2# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e
    3apt update
    4apt install python3 -y
    5
    6xcode/bitcoin-master/contrib/macdeploy/gen-sdk xcode/Xcode.app/  # we are reusing the extracted result from above
    7sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    8# c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
    

    We should get the same hash here as what contrib/macdeploy/README.md in master promises.


    To test this branch’s version of gen-sdk on a variety of platforms:

    1. Container setup

     0XCODE=~/xcode # where the files from the setup above went.
     1
     2# Where stuff is located in the container.
     3GEN_SDK="/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py"
     4EXTRACTED="/xcode/Xcode.app"
     5OUT="Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar"
     6# If you want to test master 
     7#GEN_SDK="/xcode/bitcoin-master/contrib/macdeploy/gen-sdk"
     8#OUT="Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz"
     9
    10# The only difference between platforms is how python is installed.
    11declare -A INSTALL_PYTHON=(
    12    [rhel]="dnf install -y python3 >/dev/null 2>&1"
    13    [deb]="export DEBIAN_FRONTEND=noninteractive && \
    14           apt-get update >/dev/null 2>&1 && \
    15           apt-get install -y python3 >/dev/null 2>&1"
    16    [arch]="pacman -Sy --noconfirm >/dev/null && \
    17            pacman -S --noconfirm python >/dev/null"
    18)
    19
    20tartest() {
    21    docker pull "$1" -q
    22    docker run --rm -i \
    23        -v "$XCODE:/xcode" \
    24        "$1" /bin/bash -c "
    25            ${INSTALL_PYTHON[$2]}
    26            $GEN_SDK $EXTRACTED
    27            PY_VER=\$(python3 --version | awk '{print \$2}')
    28            ZLIB_VER=\$(python3 -c 'import zlib; print(zlib.ZLIB_VERSION)')
    29            HASH=\$(sha256sum $OUT | awk '{print \$1}')
    30            echo \"|\$HASH|$1|\$PY_VER|\$ZLIB_VER|\"
    31        "
    32}
    33
    34
    35# e.g. Fedora 42
    36tartest fedora:42 rhel
    37
    38# e.g. Ubuntu 25.04
    39tartest ubuntu:25.04 deb
    40
    41# e.g. Arch
    42tartest archlinux:latest arch
    

    2. Loops to check multiple versions

     0declare -A versions=(
     1    [fedora]="$(echo {38..43})"
     2    [rockylinux]="8 9"
     3    [ubuntu]="20.04 22.04 24.04 25.04 25.10"
     4    [debian]="11 12"
     5    [archlinux]="latest"
     6)
     7
     8declare -A family=(
     9    [fedora]="rhel"
    10    [rockylinux]="rhel"
    11    [ubuntu]="deb"
    12    [debian]="deb"
    13    [archlinux]="arch"
    14)
    15
    16echo "|sha256                                                          | os:ver | python | zlib |"
    17echo "|----------------------------------------------------------------|--------|--------|------|"
    18for os in "${!versions[@]}"; do
    19    for ver in ${versions[$os]}; do
    20        tartest "${os}:${ver}" ${family[$os]} | tail -n1
    21    done
    22done
    

    3. Checking specific python versions

    0XCODE=~/xcode # where the files from the setup above went.
    1# specify the container platform
    2PLATFORM=fedora:40 
    3
    4docker pull $PLATFORM && \
    5  docker run -it \
    6    -v $XCODE:/xcode \
    7    $PLATFORM \
    8    /bin/bash
    

    In the container:

    0dnf install -y gcc git make openssl-devel patch xz-devel zlib-devel
    1
    2curl https://pyenv.run | bash
    3export PYENV_ROOT="$HOME/.pyenv"
    4[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
    5eval "$(pyenv init - bash)"
    

    Now you can install and test with any python release e.g.

    0pyenv install 3.9.0 && pyenv global 3.9.0
    1/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
    2sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
    

    We can use a pyenv build file to build a specific commit hash, e.g. to verify the commit that fixes the issue:

    0echo 'install_package "badcommit" "https://github.com/python/cpython/archive/9017e0bd5e124ae6d2ed94b9e9cacb2e86270980.tar.gz" standard' > badcommit
    1pyenv install badcommit && pyenv global badcommit
    2/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
    3sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
    4# Nondeterministic hash!
    5
    6echo 'install_package "goodcommit" "https://github.com/python/cpython/archive/674935b8caf33e47c78f1b8e197b1b77a04992d2.tar.gz" standard' > goodcommit
    7pyenv install goodcommit && pyenv global goodcommit
    8/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
    9sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
    
  58. fanquake commented at 2:54 pm on November 27, 2025: member

    Guix Build (x86_64):

     0cc72adc0e567f33605050a98cd7cd08db2b0c7da1aa11dd5f73b9077653b5e00  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/SHA256SUMS.part
     115d23cda22ba4afceddb733820893e4ef3a5307717002403cdc5e66d3561bafb  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu-debug.tar.gz
     22d0e852259c802ea3c2411a38b93148dcaa2415b5d5d34d1f6362df39108e718  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu.tar.gz
     3b9c4572af1a6312575f01414b0bb1530250c3c52da5dc53638134c399ca1048b  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/SHA256SUMS.part
     4ee2d60c4b260107b36933e66d7eb8c2a1b69f323661348ac461a66056d2522d4  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf-debug.tar.gz
     5900d4bc2e57448087e3bdda96873d98eeceb9ac05d48ca33843eee3c2a60e2bc  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf.tar.gz
     6324e3036941d34dae95579ba7f4ca006514d2a7a99431deb8333c61d61e2a527  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/SHA256SUMS.part
     7fe24816c78e64f0ac463eebd8dda589d6e64630527f67c10f97f4118380097dd  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-codesigning.tar.gz
     8ddd416028f6858282d485f0017d8d777d3cee0bcd207f0d8e2542191425602cb  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.tar.gz
     98088cd6ef218acc83928bd415b88bbeb082447890e7e497ee0b28e149b0bfc59  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.zip
    101b254bdac0893af4c012b6f093ee7be5b6f5662ceb114841b16d625c1586d415  guix-build-3e01b5d0e7be/output/dist-archive/bitcoin-3e01b5d0e7be.tar.gz
    11e5758de2f2acc35d2ef953b80b1e8bf618e1e157f325b5d9bfa94f74e7be9c86  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/SHA256SUMS.part
    125800c2135a6148bba763cb68f23708ef7fc62c850dac507a544381cccba49fa9  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu-debug.tar.gz
    13eae287e264b5e0ae88e0aee92a520e13aa00341266239a5ec257577f4c1a2d5c  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu.tar.gz
    14eafdaacbc9540f9e9706db281ec9f6db859e80dcb8e0e79efdc7e536799029d7  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/SHA256SUMS.part
    15fba989ba1b4890e1161e98b05d6ebf53cd19aab5f5be44a0081cd9eed1499138  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu-debug.tar.gz
    16c0e4f20087abb4ad0a58b06a7718913dd851ff46e092f95384618ad74cebff58  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu.tar.gz
    1778a684a96bc9c0d901cc77601263277cf142cb20659b99451247f671b11afafc  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/SHA256SUMS.part
    1804713a87ee3c418a35103f843c2d725f237a44aebc2fe28b6fbc3f5b73424f18  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-codesigning.tar.gz
    193ce4ea97fe2da7f41f0886f933ac0712528e90279386783af64b2ef0394fa2f3  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.tar.gz
    20ddfdbfa347386326fd274069009ad9ed6c5e8837d7595027d6e095193e251176  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.zip
    21d95954c5818d7d8036c5f8353063739e2c1fe65dc40eb1224e00a3dc3f4b56b4  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/SHA256SUMS.part
    22d8cf149a5d762da733f0efdcd67d09fbde8db601d1cd4d22ee57e87a2dcf6ce2  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu-debug.tar.gz
    23ffc1fb0b72f22ae574628e15dbfcd52bc37d5c37bdfd1665ecc320cca4830eae  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu.tar.gz
    24ba812f2cd3b757c0096c6070e5bf072442a74a5614b1dfc92a8af39c6afbffdf  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/SHA256SUMS.part
    254dac4c0f9bc0109e2beca6ec3c14d650538d1a3271bae92bb84a4d05612665ff  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-codesigning.tar.gz
    2606f08c02fc90e461ce837f4b7cc4d0a32ada0bc3a1ebec3eae4507b00e4ca8af  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-debug.zip
    27ef1ce4596f1d98f684604c58c8e207e2cb30858d90232ea7e3978e03c7545a85  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-setup-unsigned.exe
    28301f9ee12bb95e109b0f278e3ffaace61c5c218effdf6dc475dbb358bf4116fc  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-unsigned.zip
    
  59. fanquake merged this on Nov 27, 2025
  60. fanquake closed this on Nov 27, 2025

  61. fanquake deleted the branch on Nov 27, 2025
  62. pinheadmz commented at 1:19 pm on November 28, 2025: member

    post-merge ACK 3e01b5d0e7 instructions are clear, built mac outputs on x86 debian, my usual guix builder:

    0fe24816c78e64f0ac463eebd8dda589d6e64630527f67c10f97f4118380097dd  bitcoin-3e01b5d0e7be-arm64-apple-darwin-codesigning.tar.gz
    1ddd416028f6858282d485f0017d8d777d3cee0bcd207f0d8e2542191425602cb  bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.tar.gz
    28088cd6ef218acc83928bd415b88bbeb082447890e7e497ee0b28e149b0bfc59  bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.zip
    31b254bdac0893af4c012b6f093ee7be5b6f5662ceb114841b16d625c1586d415  bitcoin-3e01b5d0e7be.tar.gz
    404713a87ee3c418a35103f843c2d725f237a44aebc2fe28b6fbc3f5b73424f18  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-codesigning.tar.gz
    53ce4ea97fe2da7f41f0886f933ac0712528e90279386783af64b2ef0394fa2f3  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.tar.gz
    6ddfdbfa347386326fd274069009ad9ed6c5e8837d7595027d6e095193e251176  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.zip
    
  63. alexanderwiederin referenced this in commit 7334556a3f on Dec 1, 2025
  64. fanquake referenced this in commit 338570de5c on Dec 3, 2025
  65. fanquake referenced this in commit 5408e85145 on Dec 3, 2025
  66. fanquake referenced this in commit 2256f8965e on Dec 3, 2025
  67. fanquake commented at 11:34 am on December 3, 2025: member
    Backported to 30.x in #33997.

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-12-12 00:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me