guix: only download sources for hosts being built #21078

pull fanquake wants to merge 1 commits into bitcoin:master from fanquake:guix_selective_download changing 1 files +16 −1
  1. fanquake commented at 2:46 AM on February 4, 2021: member

    For example, if a user is only interested in building for Linux, this saves downloading the macOS compiler and additional dependencies, which is meaningful on a slow/poor connection. This will result in a few additional make invocations, for the Linux hosts, however this is low overhead, and time-wise irrelevant in terms of the overall build.

  2. fanquake added the label Build system on Feb 4, 2021
  3. fanquake requested review from dongcarl on Feb 4, 2021
  4. laanwj commented at 12:28 PM on February 4, 2021: member

    Concept ACK, trimming the dependency chain for the specific thing that is being built makes sense.

  5. practicalswift commented at 9:04 AM on February 5, 2021: contributor

    Concept ACK

  6. dongcarl commented at 4:29 PM on February 8, 2021: member

    Concept ACK!

    Perhaps we could set a local host_commonname variable that maps darwin -> osx, mingw -> win, linux -> linux, and just do:

    make -C "${PWD}/depends" -j"$MAX_JOBS" download-"$host_commonname" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}

    I think I need that variable later for yet-unpublished functionality (signing/verifying) anyway

  7. fanquake commented at 3:50 AM on February 10, 2021: member

    Perhaps we could set a local host_commonname variable that maps darwin -> osx, mingw -> win, linux -> linux, and just do:

    Unless I'm misunderstanding, I don't think that will work the way we want it to. If you have HOSTS=x86_64-apple-darwin18 x86_64-w64-mingw32, I'll end up with make download-win, which wont grab the macOS dependencies?

    Or, do you mean determine the host, and then invoke make download-"$host_commonname", which would still have the same number of make invocations, but we don't have essentially the same line copy-pasted 3 times?

  8. dongcarl commented at 9:25 PM on February 18, 2021: member

    Yeah I meant the latter :-) Something like the following:

    diff --git a/contrib/guix/guix-build.sh b/contrib/guix/guix-build.sh
index a90547af0a..c296e65b88 100755
--- a/contrib/guix/guix-build.sh
+++ b/contrib/guix/guix-build.sh
@@ -136,20 +136,23 @@ done
 # environment)
 MAX_JOBS="${MAX_JOBS:-$(nproc)}"
 
+# Usage: host_to_commonname HOST
+#
+#   HOST: The current platform triple we're building for
+#
+host_to_commonname() {
+    case "$1" in
+        *darwin*) echo osx ;;
+        *mingw*)  echo win ;;
+        *linux*)  echo linux ;;
+        *)        exit 1 ;;
+    esac
+}
+
 # Download the depends sources now as we won't have internet access in the build
 # container
 for host in $HOSTS; do
-  case "$host" in
-      *darwin*)
-          make -C "${PWD}/depends" -j"$MAX_JOBS" download-osx ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
-          ;;
-      *mingw*)
-          make -C "${PWD}/depends" -j"$MAX_JOBS" download-win ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
-          ;;
-      *)
-          make -C "${PWD}/depends" -j"$MAX_JOBS" download-linux ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
-          ;;
-  esac
+    make -C "${PWD}/depends" -j"$MAX_JOBS" download-"$(host_to_commonname "$host")" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
 done
 
 # Determine the reference time used for determinism (overridable by environment)
  9. guix: only download sources for hosts being built
    If a user is only interested in building for Linux, this saves
downloading the macOS compiler and additional dependencies.

This will result in a few additional `make` invocations, for the Linux
hosts, however this is quite low overhead.

Co-authored-by: Carl Dong <contact@carldong.me>
    a6a1b106dc
  10. fanquake force-pushed on Feb 22, 2021
  11. fanquake commented at 3:02 AM on February 22, 2021: member

    @dongcarl Yes. That's better, and if you're going to use this later, that's good. Have tested it's working as expected:

    bash-5.1# HOSTS="x86_64-linux-gnu" ./contrib/guix/guix-build.sh
make: Entering directory '/bitcoin/depends'
Fetching boost_1_71_0.tar.bz2 from https://dl.bintray.com/boostorg/release/1.71.0/source/
Fetching release-2.1.11-stable.tar.gz from https://github.com/libevent/libevent/archive/
/bitcoin/depends/work/download/libevent-2.1.11-stable/release-2.1.11-stable.tar.gz.temp: OK
Fetching zlib-1.2.11.tar.gz from https://www.zlib.net
Fetching qtbase-opensource-src-5.9.8.tar.xz from https://download.qt.io/official_releases/qt/5.9/5.9.8/submodules
Fetching qttranslations-opensource-src-5.9.8.tar.xz from https://download.qt.io/official_releases/qt/5.9/5.9.8/submodules
Fetching qttools-opensource-src-5.9.8.tar.xz from https://download.qt.io/official_releases/qt/5.9/5.9.8/submodules
Fetching expat-2.2.7.tar.bz2 from https://github.com/libexpat/libexpat/releases/download/R_2_2_7/
Fetching libxcb-1.10.tar.bz2 from https://xcb.freedesktop.org/dist
Fetching xcb-proto-1.10.tar.bz2 from https://xcb.freedesktop.org/dist
Fetching libXau-1.0.8.tar.bz2 from https://xorg.freedesktop.org/releases/individual/lib/
Fetching xproto-7.0.26.tar.bz2 from https://xorg.freedesktop.org/releases/individual/proto
Fetching freetype-2.7.1.tar.bz2 from https://download.savannah.gnu.org/releases/freetype
Fetching fontconfig-2.12.1.tar.bz2 from https://www.freedesktop.org/software/fontconfig/release/
Fetching qrencode-3.4.4.tar.bz2 from https://fukuchi.org/works/qrencode/
Fetching db-4.8.30.NC.tar.gz from https://download.oracle.com/berkeley-db
Fetching sqlite-autoconf-3320100.tar.gz from https://sqlite.org/2020/
Fetching miniupnpc-2.0.20180203.tar.gz from https://miniupnp.tuxfamily.org/files/
Fetching libnatpmp-20150609.tar.gz from https://miniupnp.tuxfamily.org/files/
Fetching zeromq-4.3.1.tar.gz from https://github.com/zeromq/libzmq/releases/download/v4.3.1/
make[1]: Leaving directory '/bitcoin/depends'
make: Leaving directory '/bitcoin/depends'

    bash-5.1# HOSTS="x86_64-w64-mingw32" ./contrib/guix/guix-build.sh
make: Entering directory '/bitcoin/depends'
make: Leaving directory '/bitcoin/depends'

    bash-5.1# HOSTS="x86_64-apple-darwin18" ./contrib/guix/guix-build.sh
Fetching v1.3.0.tar.gz from https://github.com/al45tair/ds_store/archive/
Fetching v2.1.1.tar.gz from https://github.com/al45tair/mac_alias/archive/
Fetching 55562e4073dea0fbfd0b20e0bf69ffe6390c7f97.tar.gz from https://github.com/tpoechtrager/cctools-port/archive
Fetching clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz from https://releases.llvm.org/8.0.0
Fetching 3efb201881e7a76a21e0554906cf306432539cef.tar.gz from https://github.com/tpoechtrager/apple-libtapi/archive
Fetching 7ac55ec64c96f7800d9818ce64c79670e7f02b67.tar.gz from https://github.com/planetbeing/libdmg-hfsplus/archive
make[1]: Leaving directory '/bitcoin/depends'
make: Leaving directory '/bitcoin/depends'
  12. fanquake commented at 4:46 AM on February 22, 2021: member

    Guix build at a6a1b106dcc4350e420c461171c47e4934087175:

    bash-5.1# find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
5543b4bac07dff9d1f0e5ff0e2bf795960fccdead040b8d46e7d2019153ec3f4  output/bitcoin-a6a1b106dcc4-aarch64-linux-gnu-debug.tar.gz
970486a947fa5806d5be8a7b55ff2c17d7d1310f42621a9e16f110ec2a3949f2  output/bitcoin-a6a1b106dcc4-aarch64-linux-gnu.tar.gz
9ad2b32ca11a6fca7018a78ab7a34be15053a8eb9c8dd74c31facfbb385ab5f8  output/bitcoin-a6a1b106dcc4-arm-linux-gnueabihf-debug.tar.gz
6a91ce3a596fe3a5e121eff7915f90964eb04dee6ad99e2737e42efec895ea69  output/bitcoin-a6a1b106dcc4-arm-linux-gnueabihf.tar.gz
1d9d7ac4eb28b2978ce31dbf9708c721e905818286be80b9230e1cf0b7ecb286  output/bitcoin-a6a1b106dcc4-osx-unsigned.dmg
ac946a95ce57dab97783fa1bc155272d9d94457daa973758de25c1ac3411f2db  output/bitcoin-a6a1b106dcc4-osx-unsigned.tar.gz
31a7bc326987ecbe99068515328edf604f621b109f2e70ded39c48d9623e646a  output/bitcoin-a6a1b106dcc4-osx64.tar.gz
a2d76d0e81f4d421dc23e4682c1097efa043d72f77b727b078027babc9ee5086  output/bitcoin-a6a1b106dcc4-riscv64-linux-gnu-debug.tar.gz
79a97584d2c4d4a7569538f1be4dd33c8bc82c373f39a4338ec22fbc3a80e294  output/bitcoin-a6a1b106dcc4-riscv64-linux-gnu.tar.gz
bc8c2d54ed90f4f15c7808e46d73c3caea4ff456a6628f3f6c164e94b78a6ce4  output/bitcoin-a6a1b106dcc4-win-unsigned.tar.gz
1d61d1e4b44abc50a3a9fcb2d9f211f719c15153ab052cc7b4e0211bc43f5025  output/bitcoin-a6a1b106dcc4-win64-debug.zip
995efc683457e4106b88e2d234598d860c62abe5e0fdee98883aca12e32e85ab  output/bitcoin-a6a1b106dcc4-win64-setup-unsigned.exe
825fcfa8a4b39459bddbb3f40138701964eb19297a8e8d2dfdcc51a3a06459f3  output/bitcoin-a6a1b106dcc4-win64.zip
ae3d2aaadabc0bcec63beea3612ed43d468f01f989f16fd6c6c3cf495e72f2bf  output/bitcoin-a6a1b106dcc4-x86_64-linux-gnu-debug.tar.gz
7f0f0592eeadf32253fab4323a4e81d7ebe832de0c47f793b3eb53c576c373e9  output/bitcoin-a6a1b106dcc4-x86_64-linux-gnu.tar.gz
4064bef6d7f8c3f5ed792b58959d7767d9b26f8bb8bdd12041585c6e012c74d1  output/src/bitcoin-a6a1b106dcc4.tar.gz
  13. laanwj commented at 1:02 PM on February 22, 2021: member

    Code review ACK a6a1b106dcc4350e420c461171c47e4934087175

  14. fanquake merged this on Feb 23, 2021
  15. fanquake closed this on Feb 23, 2021
  16. fanquake deleted the branch on Feb 23, 2021
  17. sidhujag referenced this in commit f53f3c3d68 on Feb 23, 2021
  18. DrahtBot locked this on Aug 16, 2022
Contributors
fanquakelaanwjpracticalswiftdongcarl


dongcarl

Labels
Build system
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-26 06:14 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me