build: reproducibility issue with macOS Guix builds #30815

issue fanquake openend this issue on September 4, 2024
  1. fanquake commented at 4:17 pm on September 4, 2024: member

    There seems to be a recurring (permissions related?) issue in the macOS guix builds, that is causing non-determinsm. It seems to have started since the CMake switchover. Copied in the related discussion / diff of comparing two outputs from #30743:

    EDIT: Probably something with the zipping again?

    Yea, looks like a permissions issue:

    Comparing bitcoin-556775408797-arm64-apple-darwin-unsigned.zip.fanquake & bitcoin-556775408797-arm64-apple-darwin-unsigned.zip.thecharlatan

    zipinfo

     0@@ -1,18 +1,18 @@
 1 Zip file size: 16309329 bytes, number of entries: 46
 2 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/
 3 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/
 4--rw-rw-r--  3.0 unx     1613 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Info.plist
 5+-rw-r--r--  3.0 unx     1613 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Info.plist
 6 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/MacOS/
 7 -rwxr-xr-x  3.0 unx 30234116 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
 8 -rw-r--r--  3.0 unx        8 t- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/PkgInfo
 9 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/
10 drwxr-xr-x  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/Base.lproj/
11 -rw-r--r--  3.0 unx       72 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/Base.lproj/InfoPlist.strings
12--rw-rw-r--  3.0 unx   919273 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/bitcoin.icns
13+-rw-r--r--  3.0 unx   919273 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/bitcoin.icns
14 -rw-r--r--  3.0 unx        0 b- stor 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/empty.lproj
15 -rw-r--r--  3.0 unx       47 t- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt.conf
16 -rw-r--r--  3.0 unx      130 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_ar.qm
17 -rw-r--r--  3.0 unx      153 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_bg.qm
18 -rw-r--r--  3.0 unx      153 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_ca.qm
19 -rw-r--r--  3.0 unx      157 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_cs.qm
20 -rw-r--r--  3.0 unx      153 b- defN 24-Aug-28 16:35 Bitcoin-Qt.app/Contents/Resources/qt_da.qm

    zipdetails –redact –scan –utc

     0@@ -832,15 +832,15 @@
 1 F8CD33 Uncompressed Length   0000064D
 2 F8CD37 Filename Length       0022
 3 F8CD39 Extra Length          0000
 4 F8CD3B Comment Length        0000
 5 F8CD3D Disk Start            0000
 6 F8CD3F Int File Attributes   0001
 7        [Bit 0]               1 Text Data
 8-F8CD41 Ext File Attributes   81B40000
 9+F8CD41 Ext File Attributes   81A40000
10 F8CD45 Local Header Offset   00000063
11 F8CD49 Filename              'XXXXXXXXXXXXXX/XXXXXXXX/XXXXXXXXXX'
12 
13 F8CD6B CENTRAL HEADER [#4](/bitcoin-bitcoin/4/)     02014B50
14 F8CD6F Created Zip Spec      1E '3.0'
15 F8CD70 Created OS            03 'Unix'
16 F8CD71 Extract Zip Spec      0A '1.0'
17@@ -988,15 +988,15 @@
18 F8CF89 Uncompressed Length   000E06E9
19 F8CF8D Filename Length       002E
20 F8CF8F Extra Length          0000
21 F8CF91 Comment Length        0000
22 F8CF93 Disk Start            0000
23 F8CF95 Int File Attributes   0000
24        [Bit 0]               0 'Binary Data'
25-F8CF97 Ext File Attributes   81B40000
26+F8CF97 Ext File Attributes   81A40000
27 F8CF9B Local Header Offset   00E74C26
28 F8CF9F Filename              'XXXXXXXXXXXXXX/XXXXXXXX/XXXXXXXXX/XXXXX
29                              XXXXXXX'
30 
31 F8CFCD CENTRAL HEADER #B     02014B50
32 F8CFD1 Created Zip Spec      1E '3.0'
33 F8CFD2 Created OS            03 'Unix'
  2. fanquake added the label macOS on Sep 4, 2024
  3. fanquake added the label Build system on Sep 4, 2024
  5. hebasto commented at 10:13 pm on September 4, 2024: member

    @fanquake @TheCharlatan

    I wonder if binaries are reproducible on the same machine during repeated builds?

  6. TheCharlatan commented at 5:35 am on September 5, 2024: contributor
    They seem to be, yes.
  7. fanquake commented at 9:58 am on September 5, 2024: member

    This issue boils down to two following calls:

    At a minimum I think we should start explicitly specifying NO_SOURCE_PERMISSIONS or USE_SOURCE_PERMISSIONS (the default behaviour) everywhere that we call configure_file(), to make our intent explicit in all cases.

  8. fanquake referenced this in commit 26824e90bc on Sep 5, 2024
  9. fanquake commented at 2:51 pm on September 5, 2024: member
    Opened #30823 to add the explicit permissions usage. We can also use that PR to try and solve this issue, if wanted.
  10. fanquake referenced this in commit 1f054eca4e on Sep 6, 2024
  11. laanwj commented at 2:05 pm on September 8, 2024: member

    Any idea why this happens?

    Is it something else from the context outside GUIX that leaks in? Umasks? File system differences?

    We can likely work around this at the cmake level but ideally the guix build shell would prevent this from happening in the first place.

    Edit: oh i think i see, guix mounts the git directory into its build environment, in which files might have different permissions based on user’s umask / git configuration. When they’re packed as-is into the zip file this can cause divergence. This is not a problem for tar due to normalization with --mode='u+rw,go+r-w,a+X'.

  12. fanquake referenced this in commit ba84c2774d on Sep 9, 2024
  13. hebasto commented at 10:05 am on September 9, 2024: member

    Is it something else from the context outside GUIX that leaks in? Umasks? File system differences? @fanquake @TheCharlatan

    Can you confirm filesystem permissions for a source directory within a Guix container on your systems?

  14. hebasto commented at 2:41 pm on September 10, 2024: member

    Any idea why this happens?

    Is it something else from the context outside GUIX that leaks in? Umasks? File system differences?

    I can confirm that the problem is caused by different umask values for the Linux users running the ./contrib/guix/guix-build script`.

    We can likely work around this at the cmake level but ideally the guix build shell would prevent this from happening in the first place.

    Edit: oh i think i see, guix mounts the git directory into its build environment, in which files might have different permissions based on user’s umask / git configuration. When they’re packed as-is into the zip file this can cause divergence. This is not a problem for tar due to normalization with --mode='u+rw,go+r-w,a+X'.

    We are already using a tool-level mitigation (for tar). Therefore, using another tool-level mitigation (for CMake) seems consistent.

  15. laanwj commented at 9:56 am on September 11, 2024: member

    We are already using a tool-level mitigation (for tar). Therefore, using another tool-level mitigation (for CMake) seems consistent.

    Sure. No big deal to add another one. Just important to realize GUIX isolation is ’leaky’ in some senses.

    (and besides, aside from this there’s a good argument for normalizing permissions and other metadata at archival time, just to be sure the distribution is consistent and known)

  16. fanquake commented at 11:39 am on September 11, 2024: member

    I can confirm that the problem is caused by different umask values for the Linux users running the ./contrib/guix/guix-build script`.

    We are already using a tool-level mitigation (for tar). Therefore, using another tool-level mitigation (for CMake) seems consistent.

    Why is it necessary though? Aren’t we already setting umask: https://github.com/bitcoin/bitcoin/blob/0725a374941355349bb4bc8a79dad1affb27d3b9/contrib/guix/libexec/build.sh#L9-L17

  17. hebasto commented at 11:43 am on September 11, 2024: member

    Why is it necessary though? Aren’t we already setting umask:

    Does it work for directories shared via --share=... options?

  18. laanwj commented at 1:53 pm on September 11, 2024: member

    Why is it necessary though? Aren’t we already setting umask:

    Yes, but too late. The git tree is already (by definition) checked out at that point. It won’t work retroactively on files that already exist.

  19. fanquake closed this on Sep 12, 2024
  20. fanquake referenced this in commit a86e7a476d on Sep 12, 2024


fanquake hebasto TheCharlatan laanwj

Labels
macOS Build system

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me