build: Use guix within gitian #22439

pull achow101 wants to merge 2 commits into bitcoin:master from achow101:guix-in-gitian changing 5 files +555 −0
  1. achow101 commented at 10:07 PM on July 13, 2021: member

    To somewhat ease the transition from gitian to guix, this PR replaces the previous gitian descriptors with ones that perform the guix build.

    Due to the way guix works, some changes were needed to gitian itself, so builders must make sure that their gitian is up to date before trying this. Additionally, the base gitian images may need to be rebuilt.

    I have only tested this using docker (with some changes to gitian: https://github.com/devrandom/gitian-builder/pull/251). It is my understanding that @luke-jr has been able to get these to work using kvm. I have tried with lxc but was unable to make it work.

    The previous gitian descriptors used Ubuntu Focal as the image base. However because gitian is unable to make such images for lxc and kvm, the base image used here is Bionic.

  2. achow101 marked this as a draft on Jul 13, 2021
  3. luke-jr commented at 10:11 PM on July 13, 2021: member

    I have failed to get a completed build so far, but have kept retrying.

    It is very resource hungry, and the absence of apt-cacher means it has to restart downloads every time one fails.

    I need to use my disksize patch for gitian to have enough space. https://github.com/devrandom/gitian-builder/pull/172

  4. DrahtBot added the label Build system on Jul 13, 2021
  5. DrahtBot added the label Scripts and tools on Jul 13, 2021
  6. DrahtBot commented at 5:26 AM on July 14, 2021: member

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    No conflicts as of last run.

  7. luke-jr commented at 4:36 PM on July 14, 2021: member 
    fab02ca332c5d79bcf1d62455a5e47e1b022522fb78ca81f2f07bde53d889a9f  aarch64-linux-gnu/bitcoin-21.99-guixtest1-aarch64-linux-gnu-debug.tar.gz
83c2e2a991bd2ae635e24667a1048e1735b77a887dc5a5f424f09b774c5bfb61  aarch64-linux-gnu/bitcoin-21.99-guixtest1-aarch64-linux-gnu.tar.gz
8d9c44cf254760000a8c62db00d919cc2e9488f6619b12123fb55a444870ce84  aarch64-linux-gnu/inputs.SHA256SUMS
db2d740831bc70cb443f2b855c076a905c99ed2567a3240d89c99f036dba1cf6  arm-linux-gnueabihf/bitcoin-21.99-guixtest1-arm-linux-gnueabihf-debug.tar.gz
92e3c65cacff6e522d8daf2bf71ca3eb2c62da67557990c4ee24ab9e17142a39  arm-linux-gnueabihf/bitcoin-21.99-guixtest1-arm-linux-gnueabihf.tar.gz
8d9c44cf254760000a8c62db00d919cc2e9488f6619b12123fb55a444870ce84  arm-linux-gnueabihf/inputs.SHA256SUMS
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  dist-archive/SKIPATTEST.TAG
10d67c880c5d1753ca2dc667241089d2e8a155282aabd17e483838239120cc87  dist-archive/bitcoin-21.99-guixtest1.tar.gz
cea5acf03c64d1f6958ee7ea5572b4ccbaee3525dde372f2e751e53bccd53137  powerpc64-linux-gnu/bitcoin-21.99-guixtest1-powerpc64-linux-gnu-debug.tar.gz
d71b0bc9f3ee82c5edf334547eb5c4383e51fc440192f95c8a1e0ab008668906  powerpc64-linux-gnu/bitcoin-21.99-guixtest1-powerpc64-linux-gnu.tar.gz
8d9c44cf254760000a8c62db00d919cc2e9488f6619b12123fb55a444870ce84  powerpc64-linux-gnu/inputs.SHA256SUMS
0e6cf8800b577a717177dedfc2f527522c8afbb1243688465747a0c1395883c9  powerpc64le-linux-gnu/bitcoin-21.99-guixtest1-powerpc64le-linux-gnu-debug.tar.gz
59d6d8f630d77676bcd88ecb28d8f6acd9e859ff16013b4b76e9033709755cd3  powerpc64le-linux-gnu/bitcoin-21.99-guixtest1-powerpc64le-linux-gnu.tar.gz
8d9c44cf254760000a8c62db00d919cc2e9488f6619b12123fb55a444870ce84  powerpc64le-linux-gnu/inputs.SHA256SUMS
e8e0622c8e748edaafeb38c7621980d9e29bebefe9fd25ff5dbd4d687e75ed08  riscv64-linux-gnu/bitcoin-21.99-guixtest1-riscv64-linux-gnu-debug.tar.gz
c45b797ff1092e015de43dfbeb8b911f0febc5d4c11f695d2ef9dc5118626059  riscv64-linux-gnu/bitcoin-21.99-guixtest1-riscv64-linux-gnu.tar.gz
8d9c44cf254760000a8c62db00d919cc2e9488f6619b12123fb55a444870ce84  riscv64-linux-gnu/inputs.SHA256SUMS
da94fccd6278cbd116d218f74150b0c1230bb40e1ac68bb0cba52503801b2366  x86_64-linux-gnu/bitcoin-21.99-guixtest1-x86_64-linux-gnu-debug.tar.gz
b925b9444131b1da245d5dd314040085f4dd9e9e513c28cfb6ae12fb0c5e6ba4  x86_64-linux-gnu/bitcoin-21.99-guixtest1-x86_64-linux-gnu.tar.gz
8d9c44cf254760000a8c62db00d919cc2e9488f6619b12123fb55a444870ce84  x86_64-linux-gnu/inputs.SHA256SUMS
5f79485f8299691166e860036149bc6d600b9d25db1e2d3a0a75a7bb18e4cbc3  x86_64-w64-mingw32/bitcoin-21.99-guixtest1-win-unsigned.tar.gz
157b6457bc8d4fafd680144fd684905a934985cdb8997e074c57b6134e5eb081  x86_64-w64-mingw32/bitcoin-21.99-guixtest1-win64-debug.zip
49ef90b5e9124203770c6c2dc154cbdab4098675fe4db93592bfef60eb31fb4a  x86_64-w64-mingw32/bitcoin-21.99-guixtest1-win64-setup-unsigned.exe
f4a8a7e079326a8500536eee5dfab38cb34a554f216a4aa2e7ab6d230b0dc813  x86_64-w64-mingw32/bitcoin-21.99-guixtest1-win64.zip
8d9c44cf254760000a8c62db00d919cc2e9488f6619b12123fb55a444870ce84  x86_64-w64-mingw32/inputs.SHA256SUMS
26f2f35486684b4b323359d11897a17e69a20a46a3524683f7e2bbc3391b1e07  bitcoin-core-guix-linux-win-res.yml
  8. achow101 force-pushed on Jul 15, 2021
  9. achow101 force-pushed on Jul 16, 2021
  10. achow101 force-pushed on Jul 20, 2021
  11. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:65 in f019d68d5a outdated 
      60 | +  if test -n "$GBUILD_CACHE_ENABLED"; then
  61 | +    mkdir -p ${GUIX_CACHE}
  62 | +    KEYFILE="${GUIX_CACHE}/signing-key.pub"
  63 | +    STORE_FILE="${GUIX_CACHE}/guix_store.nar"
  64 | +    if [[ -f "$KEYFILE" && -f "$STORE_FILE" ]]; then
  65 | +      sudo cat file | sudo guix archive --authorize
    luke-jr commented at 7:51 PM on July 20, 2021:

    Shouldn't file be "${KEYFILE}" still? (and do we actually need sudo to read it?)

    achow101 commented at 8:04 PM on July 20, 2021:

    Oops.

    sudo is not needed but there was a linter complaint about not having it.

    luke-jr commented at 8:28 PM on July 20, 2021:

    Can we fix (or disable) the linter instead?

    achow101 commented at 8:59 PM on July 20, 2021:

    We don't need sudo to read the keyfile, but we need to be able to pass it into sudo guix archive. The linter does not like cat ${KEYFILE} | ... and suggested that a redirect be used ( ... < ${KEYFILE}). But when I did that, it complained about the sudo and suggested this line.

    achow101 commented at 9:28 PM on July 20, 2021:

    I disabled the linter warnings for these lines.

  12. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:28 in f019d68d5a outdated 
      23 | +  export TZ="UTC"
  24 | +  export BUILD_DIR="$PWD"
  25 | +  if test -n "$GBUILD_CACHE_ENABLED"; then
  26 | +    export SOURCES_PATH=${GBUILD_COMMON_CACHE}
  27 | +    export BASE_CACHE=${GBUILD_PACKAGE_CACHE}
  28 | +    mkdir -p ${BASE_CACHE} ${SOURCES_PATH}
    luke-jr commented at 7:56 PM on July 20, 2021:

    For safety, there should be double quotes around the variables

        mkdir -p "${BASE_CACHE}" "${SOURCES_PATH}"
    achow101 commented at 8:04 PM on July 20, 2021:

    Done

  13. luke-jr changes_requested
  14. achow101 force-pushed on Jul 20, 2021
  15. luke-jr commented at 8:29 PM on July 20, 2021: member

    (Prefer to keep the old gitian yml removals in a separate PR)

  16. hebasto commented at 8:58 PM on July 20, 2021: member

    (Prefer to keep the old gitian yml removals in a separate PR)

    #22418

  17. luke-jr commented at 9:21 PM on July 20, 2021: member

    @hebasto Yes, I mean not bundled in this PR

  18. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:29 in 82bbfc38b8 outdated 
      24 | +  export BUILD_DIR="$PWD"
  25 | +  if test -n "$GBUILD_CACHE_ENABLED"; then
  26 | +    export SOURCES_PATH=${GBUILD_COMMON_CACHE}
  27 | +    export BASE_CACHE=${GBUILD_PACKAGE_CACHE}
  28 | +    mkdir -p "${BASE_CACHE}" "${SOURCES_PATH}"
  29 | +    GUIX_CACHE=${GBUILD_COMMON_CACHE}/guix
    luke-jr commented at 9:22 PM on July 20, 2021: 
        GUIX_CACHE="${GBUILD_COMMON_CACHE}/guix"
    achow101 commented at 9:50 PM on July 20, 2021:

    Done here and elsewhere.

  19. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:31 in 82bbfc38b8 outdated 
      26 | +    export SOURCES_PATH=${GBUILD_COMMON_CACHE}
  27 | +    export BASE_CACHE=${GBUILD_PACKAGE_CACHE}
  28 | +    mkdir -p "${BASE_CACHE}" "${SOURCES_PATH}"
  29 | +    GUIX_CACHE=${GBUILD_COMMON_CACHE}/guix
  30 | +
  31 | +    GUIX_SCRIPT=${GUIX_CACHE}/guix-install.sh
    luke-jr commented at 9:22 PM on July 20, 2021: 
        GUIX_SCRIPT="${GUIX_CACHE}/guix-install.sh"
  20. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:33 in 82bbfc38b8 outdated 
      28 | +    mkdir -p "${BASE_CACHE}" "${SOURCES_PATH}"
  29 | +    GUIX_CACHE=${GBUILD_COMMON_CACHE}/guix
  30 | +
  31 | +    GUIX_SCRIPT=${GUIX_CACHE}/guix-install.sh
  32 | +    if [[ -f "${GUIX_SCRIPT}" ]]; then
  33 | +      cp ${GUIX_SCRIPT} $BUILD_DIR
    luke-jr commented at 9:22 PM on July 20, 2021: 
          cp "${GUIX_SCRIPT}" "$BUILD_DIR"
  21. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:35 in 82bbfc38b8 outdated 
      30 | +
  31 | +    GUIX_SCRIPT=${GUIX_CACHE}/guix-install.sh
  32 | +    if [[ -f "${GUIX_SCRIPT}" ]]; then
  33 | +      cp ${GUIX_SCRIPT} $BUILD_DIR
  34 | +    fi
  35 | +    GUIX_TARBALL=${GUIX_CACHE}/guix-tarball-pack-x86_64-linux.tar.gz
    luke-jr commented at 9:23 PM on July 20, 2021: 
        GUIX_TARBALL="${GUIX_CACHE}/guix-tarball-pack-x86_64-linux.tar.gz"
  22. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:43 in 82bbfc38b8 outdated 
      38 | +    fi
  39 | +  fi
  40 | +  export JOBS=${NUM_PROCS}
  41 | +
  42 | +  # Install guix with the installer script
  43 | +  wget -nc https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
    luke-jr commented at 9:23 PM on July 20, 2021:

    Can we verify the SHA256 of this? (and peg it to the current version)

    achow101 commented at 9:50 PM on July 20, 2021:

    Done

  23. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:61 in 82bbfc38b8 outdated 
      56 | +    sudo /var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"&
  57 | +  fi
  58 | +
  59 | +  # Import guix cache
  60 | +  if test -n "$GBUILD_CACHE_ENABLED"; then
  61 | +    mkdir -p ${GUIX_CACHE}
    luke-jr commented at 9:24 PM on July 20, 2021: 
        mkdir -p "${GUIX_CACHE}"
  24. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:65 in 82bbfc38b8 outdated 
      60 | +  if test -n "$GBUILD_CACHE_ENABLED"; then
  61 | +    mkdir -p ${GUIX_CACHE}
  62 | +    KEYFILE="${GUIX_CACHE}/signing-key.pub"
  63 | +    STORE_FILE="${GUIX_CACHE}/guix_store.nar"
  64 | +    if [[ -f "$KEYFILE" && -f "$STORE_FILE" ]]; then
  65 | +      sudo cat $KEYFILE | sudo guix archive --authorize
    luke-jr commented at 9:24 PM on July 20, 2021: 
          sudo cat "$KEYFILE" | sudo guix archive --authorize
  25. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:67 in 82bbfc38b8 outdated 
      61 | +    mkdir -p ${GUIX_CACHE}
  62 | +    KEYFILE="${GUIX_CACHE}/signing-key.pub"
  63 | +    STORE_FILE="${GUIX_CACHE}/guix_store.nar"
  64 | +    if [[ -f "$KEYFILE" && -f "$STORE_FILE" ]]; then
  65 | +      sudo cat $KEYFILE | sudo guix archive --authorize
  66 | +      guix archive --import < ${STORE_FILE}
    luke-jr commented at 9:25 PM on July 20, 2021: 
          guix archive --import < "${STORE_FILE}"
  26. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:84 in 82bbfc38b8 outdated 
      77 | +  # Do the build
  78 | +  export HOSTS="x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu x86_64-w64-mingw32"
  79 | +  ./contrib/guix/guix-build
  80 | +
  81 | +  # Move all of the guix output to our output dir
  82 | +  mv guix-build-${VERSION}/output/* ${OUTDIR}
    luke-jr commented at 9:25 PM on July 20, 2021: 
      mv "guix-build-${VERSION}/output"/* "${OUTDIR}"
  27. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:87 in 82bbfc38b8 outdated 
      80 | +
  81 | +  # Move all of the guix output to our output dir
  82 | +  mv guix-build-${VERSION}/output/* ${OUTDIR}
  83 | +
  84 | +  if test -n "$GBUILD_CACHE_ENABLED"; then
  85 | +    cd ${BUILD_DIR}
    luke-jr commented at 9:25 PM on July 20, 2021: 
        cd "${BUILD_DIR}"
  28. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:90 in 82bbfc38b8 outdated 
      83 | +
  84 | +  if test -n "$GBUILD_CACHE_ENABLED"; then
  85 | +    cd ${BUILD_DIR}
  86 | +
  87 | +    # Cache guix installer
  88 | +    cp guix-install.sh ${GUIX_SCRIPT}
    luke-jr commented at 9:26 PM on July 20, 2021: 
        cp guix-install.sh "${GUIX_SCRIPT}"
  29. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:93 in 82bbfc38b8 outdated 
      86 | +
  87 | +    # Cache guix installer
  88 | +    cp guix-install.sh ${GUIX_SCRIPT}
  89 | +    guix pack --profile-name=current-guix --localstatedir guix
  90 | +    GUIX_PACK_TARBALL=$(find /gnu/store -name "*-tarball-pack.tar.gz" -type f)
  91 | +    sudo cp ${GUIX_PACK_TARBALL} ${GUIX_TARBALL}
    luke-jr commented at 9:26 PM on July 20, 2021: 
        sudo cp "${GUIX_PACK_TARBALL}" "${GUIX_TARBALL}"
  30. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:98 in 82bbfc38b8 outdated 
      91 | +    sudo cp ${GUIX_PACK_TARBALL} ${GUIX_TARBALL}
  92 | +
  93 | +    # Find all of the items in the guix store and export them to the cache
  94 | +    STORE_FILES=$(find /gnu/store -maxdepth 1 -mindepth 1 -type d -not -name ".links" | sort)
  95 | +    sudo guix archive --generate-key
  96 | +    sudo cp /etc/guix/signing-key.pub ${KEYFILE}
    luke-jr commented at 9:26 PM on July 20, 2021: 
        sudo cp /etc/guix/signing-key.pub "${KEYFILE}"
  31. in contrib/gitian-descriptors/gitian-guix-linux-win.yml:99 in 82bbfc38b8 outdated 
      92 | +
  93 | +    # Find all of the items in the guix store and export them to the cache
  94 | +    STORE_FILES=$(find /gnu/store -maxdepth 1 -mindepth 1 -type d -not -name ".links" | sort)
  95 | +    sudo guix archive --generate-key
  96 | +    sudo cp /etc/guix/signing-key.pub ${KEYFILE}
  97 | +    guix archive --export --recursive $STORE_FILES > ${STORE_FILE}
    luke-jr commented at 9:26 PM on July 20, 2021: 
        guix archive --export --recursive "$STORE_FILES" > "${STORE_FILE}"
  32. luke-jr changes_requested
  33. achow101 force-pushed on Jul 20, 2021
  34. achow101 force-pushed on Jul 20, 2021
  35. achow101 commented at 9:49 PM on July 20, 2021: member

    (Prefer to keep the old gitian yml removals in a separate PR)

    Dropped that commit

  36. achow101 force-pushed on Jul 21, 2021
  37. achow101 force-pushed on Jul 21, 2021
  38. achow101 force-pushed on Jul 22, 2021
  39. achow101 force-pushed on Jul 23, 2021
  40. achow101 force-pushed on Jul 25, 2021
  41. achow101 force-pushed on Jul 25, 2021
  42. achow101 force-pushed on Jul 26, 2021
  43. achow101 force-pushed on Jul 27, 2021
  44. Add gitian descriptors for guix building in gitian 44fb9c77c0
  45. Add instructions for gitian in guix ebda046374
  46. achow101 force-pushed on Jul 28, 2021
  47. achow101 marked this as ready for review on Aug 3, 2021
  48. fanquake commented at 5:05 AM on September 2, 2021: member

    Concept NACK - I think if someone would like to maintain these descriptors for some sort of transitional period, they are free to do so, but I'm not sure they need to live here. From what I can see we've already got a fairly solid number of builders for the 22.0 rcs we've had (especially given it's a new system), so Guix doesn't seem to be scaring anyone away. Will be interesting to see how many more we end up with for final.

    Ideally we'll go ahead with removing gitian from this repo shortly: #22418.

  49. laanwj commented at 1:11 PM on September 9, 2021: member

    I agree with @fanquake above. It might make sense to maintain this somewhere else (could even be in the docs repository which historically covers more specific building instructions), but I don't think it makes sense to have it (or anything referring to gitian) as part of this repository. It's not the recommended way to build.

  50. fanquake commented at 5:23 AM on September 10, 2021: member

    Going to close this for now then. #22418 has been merged as well.

    We currently have 13 builders who have submitted sigs for the v22.0 tag, and I'm expecting a number more.

  51. fanquake closed this on Sep 10, 2021
  52. DrahtBot locked this on Oct 30, 2022
Contributors
achow101luke-jrDrahtBothebastofanquakelaanwj
Labels
Build systemScripts and tools
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-19 00:14 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me