On Fedora 40, following the instructions in contrib/macdeploy/README.md to generate the macOS SDK tarball that is used during GUIX builds, I am not able to reproduce the hash in the readme for the generated tarball. I have reproduced this on two different Fedora 40 machines, and with a Fedora 40 docker image, but the issue does not appear when generating the tarball in an Ubuntu 24.04 docker image, or on a macOS Sequoia arm device, I've included reproduction instructions below.

Steps to reproduce

You first have to acquire the Xcode_15.xip archive according to the instructions in contrib/macdeploy/README.md. The steps below assume you have placed this archive at ~/xcode/Xcode_15.xip.

Making the bad tarball using Fedora 40

Container setup

sha256sum ~/xcode/Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
docker pull fedora:40
docker run -it \
  -v ~/xcode:/xcode \
  fedora:40 \
  /bin/bash

In the container

sha256sum /xcode/Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /xcode/Xcode_15.xip
dnf install cpio git python -y
git clone --depth 1 https://github.com/bitcoin/bitcoin.git
git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
python3 apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
# 23498380 blocks
/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
# Found Xcode (version: 15.0, build id: 15A240d)
# Found MacOSX SDK (version: 14.0, build id: 23A334)
# Creating output .tar.gz file...
# Adding MacOSX SDK 14.0 files...
# Done! Find the resulting gzipped tarball at:
/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# Wrong hash!

Making a good tarball using Ubuntu 24.04

Container setup

sha256sum ~/xcode/Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
docker pull ubuntu:24.04
docker run -it \
  -v ~/xcode:/xcode \
  ubuntu:24.04 \
  /bin/bash

In the container

export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
sha256sum /xcode/Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /xcode/Xcode_15.xip
apt update
apt install cpio git python3 -y
git clone --depth 1 https://github.com/bitcoin/bitcoin.git
git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
python3 apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
# 23498380 blocks
/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
# Found Xcode (version: 15.0, build id: 15A240d)
# Found MacOSX SDK (version: 14.0, build id: 23A334)
# Creating output .tar.gz file...
# Adding MacOSX SDK 14.0 files...
# Done! Find the resulting gzipped tarball at:
# /Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

Further investigation

pkgdiff

pkgdiff reports the contents of the two tarballs are identical:

$ sha256sum badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
$ sha256sum goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
$ pkgdiff badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
Reading packages ...
Comparing packages ...
creating report ...
result: UNCHANGED

<details>

<summary> pkgdiff report screenshot </summary>

</details>

Python and zlib versions

| ubuntu | 3.12.8 | 1.3.1.zlib-ng | pyenv | 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 | ubuntu | 3.12.3 | 1.3.1.zlib-ng | pyenv | 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765

pkgdiff reports the contents of the two tarballs are identical:

Try diffoscope (be prepared for very long output, and probably very long runtime).

Try diffoscope (be prepared for very long output, and probably very long runtime).

diffoscope says:

Format-specific differences are supported for Gzipped files but no file-specific differences were detected; falling back to a binary diff.`

diffoscope --no-default-limits --max-page-diff-block-line 16384 --html diffoscope.html badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

and the binary seems totally different to my untrained eyes:

First 16,000 lines of diff: https://davidgumberg.github.io/fg/other/sdkdiffoscope.html

<details>

<summary> screenshot of diffoscope html output </summary>

</details>

Can you gunzip the files and compare the .tars?

Can you gunzip the files and compare the .tars?

Nice, the gunzip'ed tars are identical:

$ gunzip badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
$ gunzip goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
$ sha256sum badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
d91d39ca0dbff805677afc46abe0064056e7406728aa58e67ad98e147ef0e04c  badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
$ sha256sum goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
d91d39ca0dbff805677afc46abe0064056e7406728aa58e67ad98e147ef0e04c  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar

Fedora 40 has gzip 1.13 and Ubuntu 24.04 has gzip 1.12, I'll see if this issue occurs with gzip 1.13 on ubuntu.

Edit: No change after swapping out gzip version on both Fedora and Ubuntu, and I realized that python's gzip module doesn't rely on the system's gzip binary

import zlib
print(zlib.ZLIB_VERSION)

1.3.1.zlib-ng on Fedora 40 and 1.3 on Ubuntu 24.04

Tried Fedora with the same zlib and python version as Ubuntu 24.04 and I still get the same bad hash.

<details>

<summary> python 3.12.3 w/ zlib-1.3 on fedora </summary>

sudo dnf install -y gcc openssl-devel xz-devel
curl -L https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.gz | tar xzvf -
cd zlib-1.3 && ./configure && make -j $(nproc) && make install && cd ..

curl https://pyenv.run | bash
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"

pyenv install 3.12.3
pyenv global 3.12.3

# verify zlib version
python -c "import zlib; print(zlib.ZLIB_VERSION)"
# 1.3

# regenerate Xcode.app
rm -rf Xcode.app/ && python3 apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

</details>

I also tried the opposite on ubuntu, python 3.12.8 with the zlib-ng "1.3" and I got the same bad hash as fedora 40:

<details>

<summary> python 3.12.8 w/ zlib-ng 1.3.1 on ubuntu </summary>

apt install -y build-essential curl liblzma-dev libssl-dev pkg-config
curl -L https://github.com/zlib-ng/zlib-ng/archive/refs/tags/2.2.3.tar.gz | tar xzvf -
cd zlib-ng-2.2.3/ && ./configure --zlib-compat && make -j $(nproc) && make install && cd ../

curl https://pyenv.run | bash
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"

pyenv install 3.12.8
pyenv global 3.12.8

# verify zlib version
python -c "import zlib; print(zlib.ZLIB_VERSION)"
# 1.3.1.zlib-ng


# try gen-sdk again
rm -rf Xcode.app/ && python apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

</details>

A bit mysterious, I'll keep experimenting.

Is the output of tar tvvf the same for both archives?

Is the output of tar tvvf the same for both archives?

Diffing tar tvvf of the two archives results in no output.

$ sha256sum badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
$ sha256sum goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
$ diff <(tar tvvf badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(tar tvvf goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)
# no output

@davidgumberg did you follow up any further here?

This is likely because python >= 3.11 delegates gzip.compress to zlib.compress, which then delegates to the zlib specified at configure-time for python.

On Ubuntu 24.04, this is zlib/zlib1g: https://packages.ubuntu.com/noble/python3.12-minimal On Fedora 40, this is zlib-ng-compat: https://packages.fedoraproject.org/pkgs/python3.11/python3.11-libs/fedora-40.html

@dongcarl Nice find. I wonder if we can just set mtime to 1, and avoid the delegation.

Looks like even if that worked now, it wouldn't work in future (Python 3.14+), as the "only delegate to zlib if mtime is 0" condition has been removed, in favour of always delegating: https://github.com/python/cpython/commit/08d09cf5ba041c9c5c3860200b56bab66fd44a23.

Opened #32009 to discuss.