build: reproducibility issue with macOS Guix builds #30815

There seems to be a recurring (permissions related?) issue in the macOS guix builds, that is causing non-determinsm. It seems to have started since the CMake switchover. Copied in the related discussion / diff of comparing two outputs from #30743:

EDIT: Probably something with the zipping again?

Yea, looks like a permissions issue:

Comparing bitcoin-556775408797-arm64-apple-darwin-unsigned.zip.fanquake & bitcoin-556775408797-arm64-apple-darwin-unsigned.zip.thecharlatan

zipinfo {}

@@ -1,18 +1,18 @@
 Zip file size: 16309329 bytes, number of entries: 46
 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/
 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/
--rw-rw-r--  3.0 unx     1613 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Info.plist
+-rw-r--r--  3.0 unx     1613 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Info.plist
 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/MacOS/
 -rwxr-xr-x  3.0 unx 30234116 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
 -rw-r--r--  3.0 unx        8 t- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/PkgInfo
 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/
 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/Base.lproj/
 -rw-r--r--  3.0 unx       72 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/Base.lproj/InfoPlist.strings
--rw-rw-r--  3.0 unx   919273 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/bitcoin.icns
+-rw-r--r--  3.0 unx   919273 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/bitcoin.icns
 -rw-r--r--  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/empty.lproj
 -rw-r--r--  3.0 unx       47 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt.conf
 -rw-r--r--  3.0 unx      130 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_ar.qm
 -rw-r--r--  3.0 unx      153 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_bg.qm
 -rw-r--r--  3.0 unx      153 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_ca.qm
 -rw-r--r--  3.0 unx      157 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_cs.qm
 -rw-r--r--  3.0 unx      153 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_da.qm

zipdetails --redact --scan --utc {}

@@ -832,15 +832,15 @@
 F8CD33 Uncompressed Length   0000064D
 F8CD37 Filename Length       0022
 F8CD39 Extra Length          0000
 F8CD3B Comment Length        0000
 F8CD3D Disk Start            0000
 F8CD3F Int File Attributes   0001
        [Bit 0]               1 Text Data
-F8CD41 Ext File Attributes   81B40000
+F8CD41 Ext File Attributes   81A40000
 F8CD45 Local Header Offset   00000063
 F8CD49 Filename              'XXXXXXXXXXXXXX/XXXXXXXX/XXXXXXXXXX'
 
 F8CD6B CENTRAL HEADER [#4](/bitcoin-bitcoin/4/)     02014B50
 F8CD6F Created Zip Spec      1E '3.0'
 F8CD70 Created OS            03 'Unix'
 F8CD71 Extract Zip Spec      0A '1.0'
@@ -988,15 +988,15 @@
 F8CF89 Uncompressed Length   000E06E9
 F8CF8D Filename Length       002E
 F8CF8F Extra Length          0000
 F8CF91 Comment Length        0000
 F8CF93 Disk Start            0000
 F8CF95 Int File Attributes   0000
        [Bit 0]               0 'Binary Data'
-F8CF97 Ext File Attributes   81B40000
+F8CF97 Ext File Attributes   81A40000
 F8CF9B Local Header Offset   00E74C26
 F8CF9F Filename              'XXXXXXXXXXXXXX/XXXXXXXX/XXXXXXXXX/XXXXX
                              XXXXXXX'
 
 F8CFCD CENTRAL HEADER #B     02014B50
 F8CFD1 Created Zip Spec      1E '3.0'
 F8CFD2 Created OS            03 'Unix'

This issue boils down to two following calls:

• https://github.com/bitcoin/bitcoin/blob/93e48240bfdc25c2760d33da69e739ba1f92da9b/cmake/module/Maintenance.cmake#L95
• https://github.com/bitcoin/bitcoin/blob/93e48240bfdc25c2760d33da69e739ba1f92da9b/cmake/module/Maintenance.cmake#L99

@fanquake @TheCharlatan

I wonder if binaries are reproducible on the same machine during repeated builds?

They seem to be, yes.

This issue boils down to two following calls:

At a minimum I think we should start explicitly specifying NO_SOURCE_PERMISSIONS or USE_SOURCE_PERMISSIONS (the default behaviour) everywhere that we call configure_file(), to make our intent explicit in all cases.

Opened #30823 to add the explicit permissions usage. We can also use that PR to try and solve this issue, if wanted.

Any idea why this happens?

Is it something else from the context outside GUIX that leaks in? Umasks? File system differences?

We can likely work around this at the cmake level but ideally the guix build shell would prevent this from happening in the first place.

Edit: oh i think i see, guix mounts the git directory into its build environment, in which files might have different permissions based on user's umask / git configuration. When they're packed as-is into the zip file this can cause divergence. This is not a problem for tar due to normalization with --mode='u+rw,go+r-w,a+X'.

Is it something else from the context outside GUIX that leaks in? Umasks? File system differences? @fanquake @TheCharlatan

Can you confirm filesystem permissions for a source directory within a Guix container on your systems?

Any idea why this happens?

Is it something else from the context outside GUIX that leaks in? Umasks? File system differences?

I can confirm that the problem is caused by different umask values for the Linux users running the ./contrib/guix/guix-build script`.

We can likely work around this at the cmake level but ideally the guix build shell would prevent this from happening in the first place.

Edit: oh i think i see, guix mounts the git directory into its build environment, in which files might have different permissions based on user's umask / git configuration. When they're packed as-is into the zip file this can cause divergence. This is not a problem for tar due to normalization with --mode='u+rw,go+r-w,a+X'.

We are already using a tool-level mitigation (for tar). Therefore, using another tool-level mitigation (for CMake) seems consistent.

We are already using a tool-level mitigation (for tar). Therefore, using another tool-level mitigation (for CMake) seems consistent.

Sure. No big deal to add another one. Just important to realize GUIX isolation is 'leaky' in some senses.

(and besides, aside from this there's a good argument for normalizing permissions and other metadata at archival time, just to be sure the distribution is consistent and known)

I can confirm that the problem is caused by different umask values for the Linux users running the ./contrib/guix/guix-build script`.

We are already using a tool-level mitigation (for tar). Therefore, using another tool-level mitigation (for CMake) seems consistent.

Why is it necessary though? Aren't we already setting umask: https://github.com/bitcoin/bitcoin/blob/0725a374941355349bb4bc8a79dad1affb27d3b9/contrib/guix/libexec/build.sh#L9-L17

Why is it necessary though? Aren't we already setting umask:

Does it work for directories shared via --share=... options?

Why is it necessary though? Aren't we already setting umask:

Yes, but too late. The git tree is already (by definition) checked out at that point. It won't work retroactively on files that already exist.