guix: Unable to reproduce macOS SDK tarball on Fedora 40 #31873

issue davidgumberg openend this issue on February 14, 2025
  1. davidgumberg commented at 10:10 pm on February 14, 2025: contributor

    On Fedora 40, following the instructions in contrib/macdeploy/README.md to generate the macOS SDK tarball that is used during GUIX builds, I am not able to reproduce the hash in the readme for the generated tarball. I have reproduced this on two different Fedora 40 machines, and with a Fedora 40 docker image, but the issue does not appear when generating the tarball in an Ubuntu 24.04 docker image, or on a macOS Sequoia arm device, I’ve included reproduction instructions below.

    Steps to reproduce

    You first have to acquire the Xcode_15.xip archive according to the instructions in contrib/macdeploy/README.md. The steps below assume you have placed this archive at ~/xcode/Xcode_15.xip.

    Making the bad tarball using Fedora 40

    Container setup

    0sha256sum ~/xcode/Xcode_15.xip
1# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
2docker pull fedora:40
3docker run -it \
4  -v ~/xcode:/xcode \
5  fedora:40 \
6  /bin/bash

    In the container

     0sha256sum /xcode/Xcode_15.xip
 1# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /xcode/Xcode_15.xip
 2dnf install cpio git python -y
 3git clone --depth 1 https://github.com/bitcoin/bitcoin.git
 4git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
 5python3 apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
 6# 23498380 blocks
 7/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
 8# Found Xcode (version: 15.0, build id: 15A240d)
 9# Found MacOSX SDK (version: 14.0, build id: 23A334)
10# Creating output .tar.gz file...
11# Adding MacOSX SDK 14.0 files...
12# Done! Find the resulting gzipped tarball at:
13/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
14sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
15# 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
16# Wrong hash!

    Making a good tarball using Ubuntu 24.04

    Container setup

    0sha256sum ~/xcode/Xcode_15.xip
1# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
2docker pull ubuntu:24.04
3docker run -it \
4  -v ~/xcode:/xcode \
5  ubuntu:24.04 \
6  /bin/bash

    In the container

     0export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
 1sha256sum /xcode/Xcode_15.xip
 2# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /xcode/Xcode_15.xip
 3apt update
 4apt install cpio git python3 -y
 5git clone --depth 1 https://github.com/bitcoin/bitcoin.git
 6git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
 7python3 apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
 8# 23498380 blocks
 9/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
10# Found Xcode (version: 15.0, build id: 15A240d)
11# Found MacOSX SDK (version: 14.0, build id: 23A334)
12# Creating output .tar.gz file...
13# Adding MacOSX SDK 14.0 files...
14# Done! Find the resulting gzipped tarball at:
15# /Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
16sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
17# c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

    Further investigation

    pkgdiff

    pkgdiff reports the contents of the two tarballs are identical:

    0$ sha256sum badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
15b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
2$ sha256sum goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
3c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
4$ pkgdiff badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
5Reading packages ...
6Comparing packages ...
7creating report ...
8result: UNCHANGED

    Image

    Python and zlib versions

    os python zlib pyenv / repo hash
    ubuntu 3.12.3 1.3 repo c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d
    fedora 3.12.8 1.3.1.zlib-ng repo 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765
    ubuntu 3.12.3 1.3 pyenv c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d
    fedora 3.12.3 1.3 pyenv 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765
    ubuntu 3.12.8 1.3.1.zlib-ng pyenv 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765
    ubuntu 3.12.3 1.3.1.zlib-ng pyenv 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765
  2. achow101 commented at 10:50 pm on February 14, 2025: member

    pkgdiff reports the contents of the two tarballs are identical:

    Try diffoscope (be prepared for very long output, and probably very long runtime).

  3. davidgumberg commented at 11:50 pm on February 14, 2025: contributor

    Try diffoscope (be prepared for very long output, and probably very long runtime).

    diffoscope says:

    Format-specific differences are supported for Gzipped files but no file-specific differences were detected; falling back to a binary diff.`

    0diffoscope --no-default-limits --max-page-diff-block-line 16384 --html diffoscope.html badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

    and the binary seems totally different to my untrained eyes:

    First 16,000 lines of diff: https://davidgumberg.github.io/fg/other/sdkdiffoscope.html

    Image

  4. achow101 commented at 0:02 am on February 15, 2025: member
    Can you gunzip the files and compare the .tars?
  5. davidgumberg commented at 0:21 am on February 15, 2025: contributor

    Can you gunzip the files and compare the .tars?

    Nice, the gunzip‘ed tars are identical:

    0$ gunzip badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
1$ gunzip goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
2$ sha256sum badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
3d91d39ca0dbff805677afc46abe0064056e7406728aa58e67ad98e147ef0e04c  badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
4$ sha256sum goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
5d91d39ca0dbff805677afc46abe0064056e7406728aa58e67ad98e147ef0e04c  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar

    Fedora 40 has gzip 1.13 and Ubuntu 24.04 has gzip 1.12, I’ll see if this issue occurs with gzip 1.13 on ubuntu.

    Edit: No change after swapping out gzip version on both Fedora and Ubuntu, and I realized that python’s gzip module doesn’t rely on the system’s gzip binary

    0import zlib
1print(zlib.ZLIB_VERSION)

    1.3.1.zlib-ng on Fedora 40 and 1.3 on Ubuntu 24.04

  6. davidgumberg commented at 3:26 am on February 15, 2025: contributor

    Tried Fedora with the same zlib and python version as Ubuntu 24.04 and I still get the same bad hash.

     0sudo dnf install -y gcc openssl-devel xz-devel
 1curl -L https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.gz | tar xzvf -
 2cd zlib-1.3 && ./configure && make -j $(nproc) && make install && cd ..
 3
 4curl https://pyenv.run | bash
 5export PYENV_ROOT="$HOME/.pyenv"
 6[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
 7eval "$(pyenv init - bash)"
 8
 9pyenv install 3.12.3
10pyenv global 3.12.3
11
12# verify zlib version
13python -c "import zlib; print(zlib.ZLIB_VERSION)"
14# 1.3
15
16# regenerate Xcode.app
17rm -rf Xcode.app/ && python3 apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
18/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
19sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
20# 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

    I also tried the opposite on ubuntu, python 3.12.8 with the zlib-ng “1.3” and I got the same bad hash as fedora 40:

     0apt install -y build-essential curl liblzma-dev libssl-dev pkg-config
 1curl -L https://github.com/zlib-ng/zlib-ng/archive/refs/tags/2.2.3.tar.gz | tar xzvf -
 2cd zlib-ng-2.2.3/ && ./configure --zlib-compat && make -j $(nproc) && make install && cd ../
 3
 4curl https://pyenv.run | bash
 5export PYENV_ROOT="$HOME/.pyenv"
 6[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
 7eval "$(pyenv init - bash)"
 8
 9pyenv install 3.12.8
10pyenv global 3.12.8
11
12# verify zlib version
13python -c "import zlib; print(zlib.ZLIB_VERSION)"
14# 1.3.1.zlib-ng
15
16
17# try gen-sdk again
18rm -rf Xcode.app/ && python apple-sdk-tools/extract_xcode.py -f /xcode/Xcode_15.xip | cpio -d -i
19/bitcoin/contrib/macdeploy/gen-sdk Xcode.app/
20sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
21# 5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

    A bit mysterious, I’ll keep experimenting.

  7. TheCharlatan commented at 7:01 am on February 15, 2025: contributor
    Is the output of tar tvvf the same for both archives?
  8. davidgumberg commented at 4:22 pm on February 15, 2025: contributor

    Is the output of tar tvvf the same for both archives?

    Diffing tar tvvf of the two archives results in no output.

    0$ sha256sum badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
15b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765  badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
2$ sha256sum goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
3c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
4$ diff <(tar tvvf badsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(tar tvvf goodsdk/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)
5# no output


davidgumberg achow101 TheCharlatan

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-02-22 06:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me