guix: Build support for macOS #17920

pull dongcarl wants to merge 7 commits into bitcoin:master from dongcarl:2020-01-guix-macos changing 6 files +173 −35
  1. dongcarl commented at 8:59 pm on January 13, 2020: member

    This PR brings our Guix builds on par with Gitian in terms of supported architectures.

    Reviewers: if you run a build, please submit:

    0find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum

    So that we can compare hashes and ensure reproducibility!

  2. dongcarl added the label Build system on Jan 13, 2020
  3. dongcarl added this to the "PRs" column in a project
  4. DrahtBot commented at 11:47 pm on January 13, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #20629 (depends: Improve id string robustness by dongcarl)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  5. in contrib/guix/libexec/build-osx.sh:120 in e61380a065 outdated 
    115+#         riscv64-linux-gnu)   echo /lib/ld-linux-riscv64-lp64d.so.1 ;;
116+#         *)                   exit 1 ;;
117+#     esac
118+# )
119+
120+# Environment variables for determinism
    fanquake commented at 4:02 am on January 14, 2020:

    I assume we no longer have to set ZERO_AR_DATE here because we’re not using a libtool from cctools?

    From libtool.c:

    all of these values are initialized to reasonable defaults for deterministic archives: archives that are consistent regardless of user, time, or umask differences. Ordinarily, these values and will be set to more specific ones when building new archives, but that can be suppressed using the ‘-D’ option or the ZERO_AR_DATE environment variable.

  6. in contrib/guix/libexec/build.sh:104 in e61380a065 outdated 
    100@@ -63,6 +101,8 @@ for p in "${PATHS[@]}"; do
101     fi
102 done
103 
104+echo hee
    fanquake commented at 4:08 am on January 14, 2020:
    🗣
  7. fanquake commented at 4:08 am on January 14, 2020: member
    Obviously concept ACK on macOS GUIX builds. Not going to comment on the interim system Clang commits here.
  8. dongcarl force-pushed on Jan 23, 2020
  9. dongcarl force-pushed on Jan 31, 2020
  10. dongcarl force-pushed on Jan 31, 2020
  11. fanquake added this to the milestone 0.21.0 on Mar 28, 2020
  12. dongcarl commented at 6:28 pm on July 30, 2020: member
    Update: this is still being actively worked on. Current hurdle: some weird clang internal search path logic.
  13. laanwj removed this from the milestone 0.21.0 on Oct 8, 2020
  14. laanwj added this to the milestone 0.22.0 on Oct 8, 2020
  15. dongcarl force-pushed on Nov 24, 2020
  16. dongcarl force-pushed on Dec 7, 2020
  17. dongcarl force-pushed on Dec 11, 2020
  18. Sjors commented at 6:34 pm on December 11, 2020: member

    Tested 40ad76806b9f750d9c94675b3a136132bf1e6449. I (re)installed guix on an Ubuntu 20 machine and used --no-substitutes and --bootstrap to exercise my low time preference.

    This did not go without hickups, e.g.:

    • guix install glibc-locales (as root, this would fail without substitutes) and configuring GUIX_LOCPATH (as guix user) is not particularly well documented upstream
    • during guix-build.sh the step /gnu/store/36fgj9n3c8bmix2pd12kfaszi7bd5y7a-ghostscript-9.27.drv' failed without too much actionable info. I tried dropping --bootstrap, but got the same error, so I ended up dropping --no-substitutes as well
    • noticed a warning: Computing Guix derivation for 'x86_64-linux'... WARNING: (guix build emacs-build-system): imported module (guix build utils) overrides core binding delete'`

    Not sure what file is most useful to compare, but here’s a few:

    00aa66de467110bf45e7606d2696ea4417d4a94fdc59495646428c1ca05e2e744  distsrc-x86_64-linux-gnu/src/bitcoind
12b75f3955337b0345c7b6e33ad83185bba06ea96f1aedc30b7894c20af4489ee  distsrc-x86_64-apple-darwin16/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt

    I can also still build and use depends on native macOS Big Sur. CI seems to struggle, maybe a spurious failure.

    You may want to link to the macOS SDK extraction instructions (and that it needs to be untarred into depends/SDK’s).

    You may want to rebase (#20470) now that #20422 landed, since it changed make deploy (I just tried a rebase and it seems to behave at least on macOS itself).

    Does this PR mean you can switch to the upstream GUIX repo? Or are there still patches? See https://github.com/bitcoin/bitcoin/pull/17595/commits/93439a71eda49fb69f1e82966a23a946733aa6fa

  19. dongcarl commented at 7:48 pm on December 11, 2020: member

    Many thanks for testing @Sjors! Your review is super helpful as I want to make this process as user-friendly as possible. Even for the folks building without substitutes! Cheers.

    To start off, it seems like reproducibility is working!

    0$ sha256sum distsrc-x86_64-linux-gnu/src/bitcoind distsrc-x86_64-apple-darwin16/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt
10aa66de467110bf45e7606d2696ea4417d4a94fdc59495646428c1ca05e2e744  distsrc-x86_64-linux-gnu/src/bitcoind
22b75f3955337b0345c7b6e33ad83185bba06ea96f1aedc30b7894c20af4489ee  distsrc-x86_64-apple-darwin16/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt

    Now onto your concerns:

    • guix install glibc-locales (as root, this would fail without substitutes) and configuring GUIX_LOCPATH (as guix user) is not particularly well documented upstream

    Very true, I’ve had experience with this myself, I will write up some documentation. As for the glibc-locales failure, could you send me your guix describe?

    • during guix-build.sh the step /gnu/store/36fgj9n3c8bmix2pd12kfaszi7bd5y7a-ghostscript-9.27.drv' failed without too much actionable info. I tried dropping --bootstrap, but got the same error, so I ended up dropping --no-substitutes as well

    I’ve encountered this a while ago, so this must be an outstanding bug. I will look into it! I also think it may be good for me to write down in the README how to report Guix build failures (Guix always saves a bzip’d full log file in /var/log/guix for failed builds).

    • noticed a warning: Computing Guix derivation for 'x86_64-linux'... WARNING: (guix build emacs-build-system): imported module (guix build utils) overrides core binding delete'`

    That warning is an old bug that has since been fixed here: https://issues.guix.gnu.org/41595. I will see how we can mitigate against this as well, may just have to bump my time-machine commit!

    You may want to link to the macOS SDK extraction instructions (and that it needs to be untarred into depends/SDK’s).

    Sure! I added a check for it in the script but I agree that it should be in the README as well!

    You may want to rebase (#20470) now that #20422 landed, since it changed make deploy (I just tried a rebase and it seems to behave at least on macOS itself).

    Ah I did rebase that one, but not the cherry-picked commits in here. Will do!

    Does this PR mean you can switch to the upstream GUIX repo? Or are there still patches? See 93439a7

    I want to switch over to upstream GUIX once everything is finalized. There could still be things that crop up I need to patch, but it’s looking increasingly unlikely.

  20. Sjors commented at 2:04 pm on December 13, 2020: member

    guix describe (as root)

    0/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
1guix describe: error: failed to determine origin
2hint: Perhaps this `guix' command was not obtained with `guix pull'? Its version string is 1.2.0.

    (my Ubuntu install in general is a cluster f*** that I very much feel like reinstalling from scratch, FWIW)

    may just have to bump my time-machine commit

    Happy to try again after that.

  21. dongcarl commented at 11:51 pm on December 14, 2020: member

    @Sjors Looking into a few things… Could you send me the file that’s returned by:

    0$ guix build --log-file /gnu/store/36fgj9n3c8bmix2pd12kfaszi7bd5y7a-ghostscript-9.27.drv

    Much appreciated.

  22. Sjors commented at 12:48 pm on December 15, 2020: member

    As user guix as well as root:

    0/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
1guix build: error: no build log for '/gnu/store/3i5qm6z0qihhjbdgs3pwj42gy63sf74h-ghostscript-9.27'

    I ran the garbage collector guix gc and git clean -xdff --exclude='/depends/SDKs/*' and did another build (same hashes).

    By the way, before I ran that clean command I got:

    0DISTSRC directory '/bitcoin/distsrc-x86_64-linux-gnu' exists, probably because of previous builds... Aborting...

    It would be nice if e.g. guix-build.sh checked this at the start, and maybe offers to nuke it.

  23. dongcarl commented at 11:05 pm on December 15, 2020: member

    @Sjors Would you mind giving me more info on your setup? I’m particularly interested in what distro and kernel version you were using when you encountered:

    guix install glibc-locales (as root, this would fail without substitutes)

  24. Sjors commented at 12:37 pm on December 17, 2020: member

    Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-33-generic x86_64)

    I installed Guix as root using their guix-install.sh script. I have a user guix where I cloned this branch and from which I run the builds.

  25. dongcarl force-pushed on Dec 23, 2020
  26. dongcarl force-pushed on Jan 7, 2021
  27. dongcarl force-pushed on Jan 7, 2021
  28. dongcarl force-pushed on Jan 12, 2021
  29. dongcarl moved this from the "PRs" to the "Next (Not based on any other PRs)" column in a project
  30. in contrib/guix/README.md:109 in 0e5c7bf599 outdated 
    104@@ -105,7 +105,8 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
105 
106   Override the space-separated list of platform triples for which to perform a
107   bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf
108-  aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32")_
109+  aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32
110+  x86_64-apple-darwin16")_
    fanquake commented at 0:30 am on January 13, 2021: 
    0  x86_64-apple-darwin18")_

    Throughout this PR, to match depends, gitian etc.

    dongcarl commented at 7:52 pm on January 13, 2021:
    Thanks! Fixed as of 05a31a95832b592ee565d3a429afcf6cbf5ae672
  31. in contrib/guix/libexec/build.sh:258 in 0e5c7bf599 outdated 
    257     make --jobs="$MAX_JOBS" ${V:+V=1}
258 
259-    # Perform basic ELF security checks on a series of executables.
260-    make -C src --jobs=1 check-security ${V:+V=1}
261+    case "$HOST" in
262+        *darwin*) ;;
    fanquake commented at 0:31 am on January 13, 2021:

    Why are we skipping the security checks for macOS builds? Same Q re symbol checks below. If there’s a specific reason why these can’t be run right now, we should add a comment.

    You could also drop ELF, as these are already being done on Linux & Win binaries.

    dongcarl commented at 7:53 pm on January 13, 2021:
    Originally I skipped them because we don’t perform them in gitian osx builds, which I thought had some significance. As of 05a31a95832b592ee565d3a429afcf6cbf5ae672 they are enabled for all builds in Guix. Sweet!
  32. fanquake commented at 2:43 am on January 13, 2021: member

    Built at 0e5c7bf5991a3e1c5e3df779a445e1690f067f5b. Tested the .dmg and Bitcoin-Qt. Is there a reason this is still draft?

    0d01c95e0040f55e610f2c99e81c307b4cd97bd2c9a9d14415709f8dcaaa5c380  output/bitcoin-0e5c7bf5991a-osx-unsigned.dmg
12f8b5077339a0eac72f78cae37437027608e0c88b3e250a4c82e73dc099dc224  output/bitcoin-0e5c7bf5991a-osx-unsigned.tar.gz
282b380107dbcde96ff66791c7b512498ecefc5bc2f2428390cccd83b8fce2bb4  output/bitcoin-0e5c7bf5991a-x86_64-apple-darwin18.tar.gz
3107a81d69796b148360594c5c07a646155a1f3b14051a7b804fe27831f0f833c  output/src/bitcoin-0e5c7bf5991a.tar.gz
  33. dongcarl force-pushed on Jan 13, 2021
  34. dongcarl commented at 7:54 pm on January 13, 2021: member

    Pushed 0e5c7bf5991a3e1c5e3df779a445e1690f067f5b -> 05a31a95832b592ee565d3a429afcf6cbf5ae672

    Is there a reason this is still draft?

    Nope!

  35. dongcarl marked this as ready for review on Jan 13, 2021
  36. fanquake commented at 0:57 am on January 14, 2021: member

    05a31a95832b592ee565d3a429afcf6cbf5ae672

    0find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
13f2762da34e6681471a7ede52ff7739e0e5ad64dd4584ae1e6d4af47456c2a98  output/bitcoin-05a31a95832b-osx-unsigned.dmg
2ffc15b01f5028bf15fda61c1b3d785c2ba6e72fdfdd3292ea7f7d7f0b5230f9f  output/bitcoin-05a31a95832b-osx-unsigned.tar.gz
3538ba85976b7dec9353d9e246346cac792916ed2a59360c5ded8c7e4ddebe8b7  output/bitcoin-05a31a95832b-osx64.tar.gz
421118fd0183fc430fe6de82bbcd5ab353643fc51a280a11cb188e9e4ae7c7cd5  output/src/bitcoin-05a31a95832b.tar.gz
  37. dongcarl commented at 3:36 am on January 14, 2021: member

    I’m getting matching hashes!

    0find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
13f2762da34e6681471a7ede52ff7739e0e5ad64dd4584ae1e6d4af47456c2a98  output/bitcoin-05a31a95832b-osx-unsigned.dmg
2ffc15b01f5028bf15fda61c1b3d785c2ba6e72fdfdd3292ea7f7d7f0b5230f9f  output/bitcoin-05a31a95832b-osx-unsigned.tar.gz
3538ba85976b7dec9353d9e246346cac792916ed2a59360c5ded8c7e4ddebe8b7  output/bitcoin-05a31a95832b-osx64.tar.gz
421118fd0183fc430fe6de82bbcd5ab353643fc51a280a11cb188e9e4ae7c7cd5  output/src/bitcoin-05a31a95832b.tar.gz
  38. MarcoFalke added the label Needs Guix build on Jan 14, 2021
  39. MarcoFalke added the label Needs gitian build on Jan 14, 2021
  40. MarcoFalke removed the label Needs gitian build on Jan 14, 2021
  41. MarcoFalke added the label Needs gitian build on Jan 15, 2021
  42. MarcoFalke removed the label Needs gitian build on Jan 15, 2021
  43. dongcarl removed the label Needs Guix build on Jan 17, 2021
  44. dongcarl added the label Needs Guix build on Jan 17, 2021
  45. DrahtBot added the label Needs rebase on Jan 20, 2021
  46. build: Add var printing target to src/Makefile.am 
    See 181989f6c9427fc266dbdcc84cb60ac03e67cdb2 for more info. I missed
this one last time.
    37fe73a092
  47. guix: Add support for darwin builds c9eb4cf3a0
  48. build: Make xorrisofs reproducible with -volume_date 
    We need this to be after a '--' as '-volume_date' is a xorriso
flag, not a xorrisofs flag. See the respective man pages.

For more details: https://issues.guix.info/issue/35283#2
    f3835dc6a3
  49. guix: Set ZERO_AR_DATE for darwin build determinism 
    See comments inserted in this commit.
    34b23f597e
  50. guix: Check for macOS SDK before building anything 8dbf18cb1d
  51. guix: README: Add darwin HOSTS entry 771c4b98a8
  52. guix: Fix typo f1694757dd
  53. dongcarl force-pushed on Jan 21, 2021
  54. dongcarl commented at 4:19 pm on January 21, 2021: member

    Pushed 05a31a95832b592ee565d3a429afcf6cbf5ae672 -> f1694757ddbcb3635213b085e864851e285c8c12

    • Rebased over master
    • Fixed typo

    Hashes:

    02d17b6c787f91bf32e5d3732a6a054df1940d26030fc19d845ca0b472283a1bb  output/bitcoin-f1694757ddbc-osx-unsigned.dmg
1c5a8207f4d76928b88ac9dfed61197be9bfd62ba93821898a9c90e1eb8a5a7d5  output/bitcoin-f1694757ddbc-osx-unsigned.tar.gz
2bb972ac7f0f03c31d30c15cc9667530cd25f2237f4eb9d32cd3a1ea17ce1ff6a  output/bitcoin-f1694757ddbc-osx64.tar.gz
3262cb14aabffdb98260ca53e5d420b14f4242c83e414963abf9c4798eba899b6  output/src/bitcoin-f1694757ddbc.tar.gz
  55. DrahtBot removed the label Needs rebase on Jan 21, 2021
  56. fanquake commented at 4:16 am on January 22, 2021: member

    ACK f1694757ddbcb3635213b085e864851e285c8c12 - I think we can make some small usability improvements, but this is ok to merge now.

    0find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
12d17b6c787f91bf32e5d3732a6a054df1940d26030fc19d845ca0b472283a1bb  output/bitcoin-f1694757ddbc-osx-unsigned.dmg
2c5a8207f4d76928b88ac9dfed61197be9bfd62ba93821898a9c90e1eb8a5a7d5  output/bitcoin-f1694757ddbc-osx-unsigned.tar.gz
3bb972ac7f0f03c31d30c15cc9667530cd25f2237f4eb9d32cd3a1ea17ce1ff6a  output/bitcoin-f1694757ddbc-osx64.tar.gz
4262cb14aabffdb98260ca53e5d420b14f4242c83e414963abf9c4798eba899b6  output/src/bitcoin-f1694757ddbc.tar.gz
  57. fanquake merged this on Jan 22, 2021
  58. fanquake closed this on Jan 22, 2021
  59. fanquake moved this from the "Next (Not based on any other PRs)" to the "Done" column in a project
  60. in contrib/guix/manifest.scm:226 in f1694757dd 
    221@@ -195,8 +222,8 @@ chain for " target " development."))
222         python-3.7
223         ;; Git
224         git
225-        ;; Native gcc 9 toolchain targeting glibc 2.27
226-        (make-gcc-toolchain gcc-9 glibc-2.27))
227+        ;; Native gcc 7 toolchain
228+        gcc-toolchain-7)
    MarcoFalke commented at 6:38 am on January 22, 2021:
    Why is this changed, seems unrelated to macos, no? Does this break gcc cross builds for our linux archs that need gcc 8 at least?
    MarcoFalke commented at 7:40 am on January 23, 2021:

    Btw, this is the comment I was referring to: #13665 (comment)

    I presume that guix is using a gcc-7 that has a fix backported?

  61. sidhujag referenced this in commit b47870bd6a on Jan 22, 2021
  62. Sjors commented at 7:20 pm on January 22, 2021: member
    Post merge test: I get the same checksums!
  63. MarcoFalke removed the label Needs Guix build on Jan 28, 2021
  64. DrahtBot locked this on Aug 18, 2022


dongcarl DrahtBot fanquake Sjors MarcoFalke

Labels
Build system

Milestone
22.0

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-03 10:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me