Change gitian-descriptors to use bionic instead #13171

pull ken2812221 wants to merge 1 commits into bitcoin:master from ken2812221:gitian changing 6 files +20 −19

ken2812221 commented at 3:44 AM on May 5, 2018: contributor
I have tested this with Ubuntu Bionic host with lxc 3.0.0, the gitian-build for Windows and MacOSX work fine, but there is an issue about it for Linux. Failed at check-symbol:
```
test/test_bitcoin: symbol __divmoddi4 from unsupported version GCC_7.0.0
test/test_bitcoin: symbol log2f from unsupported version GLIBC_2.27
qt/bitcoin-qt: symbol __divmoddi4 from unsupported version GCC_7.0.0
qt/bitcoin-qt: symbol log2f from unsupported version GLIBC_2.27
```
I think this should be fixed in ./configure --enable-glibc-back-compat

Should not be merged before #13177 devrandom/gitian-builder#178

Close #12511
fanquake added the label Build system on May 5, 2018
fanquake requested review from theuni on May 5, 2018
MarcoFalke commented at 2:16 PM on May 5, 2018: member

Thanks! Concept ACK.
ken2812221 force-pushed on May 6, 2018
laanwj commented at 12:56 PM on May 7, 2018: member

Concept ACK
MarcoFalke added this to the milestone 0.17.0 on May 31, 2018
luke-jr commented at 1:35 AM on May 31, 2018: member

Need to be sure bionic images can be made for LXC, VirtualBox, and KVM. Last I checked there was an issue doing it for KVM :/
ken2812221 commented at 3:05 AM on May 31, 2018: contributor

It seems there is no more updates for python-vm-builder. @achow101 introduced a way to do gitian-build with docker, so I thought we could drop the KVM way if we couldn't find any patch to build KVM image. See #13368
ken2812221 renamed this:
~~[WIP] Change gitian-descriptors to use bionic instead~~
Change gitian-descriptors to use bionic instead
on Jun 2, 2018
MarcoFalke commented at 2:24 PM on July 11, 2018: member

utACK 997816bcc7d59a3bdb68c677ef0a6bce00210aa0

This hopefully fixes the currently broken gitian windows cross compiles of the gui (qt 5.9.6)
MarcoFalke assigned theuni on Jul 11, 2018
in contrib/gitian-descriptors/gitian-win.yml:98 in 997816bcc7 outdated
```
  94 | @@ -93,6 +95,9 @@ script: |
  95 |      done
  96 |    done
  97 |    }
  98 | +  for i in $HOSTS; do
```
theuni commented at 6:54 PM on July 12, 2018:

It should suffice to add a symlink from ${WRAP_DIR}/${i}/${i}-g++ to /usr/bin/${i}-g++-posix``` instead. Since this is in our path, that means that it would be picked up before the generic one.

in doc/release-process.md:121 in 997816bcc7 outdated

 117 | @@ -118,7 +118,7 @@ The gbuild invocations below <b>DO NOT DO THIS</b> by default.
 118 |      ./bin/gsign --signer "$SIGNER" --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
 119 |      mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../
 120 |  
 121 | -    ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
 122 | +    ./bin/gbuild --allow-sudo --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml

theuni commented at 6:54 PM on July 12, 2018:

This shouldn't be needed if the symlink trick works.

in contrib/gitian-build.sh:284 in 997816bcc7 outdated

 280 | @@ -281,7 +281,7 @@ then
 281 |  	    echo ""
 282 |  	    echo "Compiling ${VERSION} Windows"
 283 |  	    echo ""
 284 | -	    ./bin/gbuild -j ${proc} -m ${mem} --commit bitcoin=${COMMIT} --url bitcoin=${url} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
 285 | +	    ./bin/gbuild --allow-sudo -j ${proc} -m ${mem} --commit bitcoin=${COMMIT} --url bitcoin=${url} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml

theuni commented at 6:55 PM on July 12, 2018:

This also shouldn't be needed if the symlink trick works.

ken2812221 force-pushed on Jul 12, 2018
DrahtBot added the label Needs rebase on Jul 12, 2018
in contrib/gitian-descriptors/gitian-win.yml:8 in 6475f1db7a outdated
```
   1 | @@ -2,7 +2,7 @@
   2 |  name: "bitcoin-win-0.17"
   3 |  enable_cache: true
   4 |  suites:
   5 | -- "trusty"
   6 | +- "bionic"
   7 |  architectures:
   8 |  - "amd64"
   9 |  packages:
```
MarcoFalke commented at 7:44 PM on July 12, 2018:

Should rename "git-core" to "git"?

ken2812221 commented at 7:56 PM on July 12, 2018:

Yes, I already do.

MarcoFalke commented at 7:59 PM on July 12, 2018:

In line 11 of this file in the current version it still says "git-core", no?

ken2812221 commented at 8:04 PM on July 12, 2018:

Oh, I missed that one.
ken2812221 force-pushed on Jul 12, 2018
ken2812221 force-pushed on Jul 12, 2018
ken2812221 force-pushed on Jul 12, 2018
DrahtBot removed the label Needs rebase on Jul 13, 2018
MarcoFalke commented at 1:48 AM on July 13, 2018: member

re-utACK 5eaee74b6c2acae33f806fa60da757ea6fcbc689
theuni commented at 4:09 AM on July 13, 2018: member

ACK 5eaee74b6c2acae33f806fa60da757ea6fcbc689. I haven't tested the resulting binaries yet, but I can at least confirm that linux/macos/win build successfully.

I wrestled with Gitian and KVM/LXC all day to get it built, I'll write up some notes and work-arounds.
fanquake commented at 12:35 PM on July 15, 2018: member

utACK 5eaee74

LGTM There's a lot of work building on/ issues solved by this.
fanquake commented at 7:16 AM on July 16, 2018: member

@ken2812221 Could you please rebase this.
Change gitian-descriptors to use bionic instead c1afe3232f
ken2812221 force-pushed on Jul 16, 2018
ken2812221 commented at 10:43 AM on July 16, 2018: contributor

@fanquake Rebased
fanquake commented at 10:45 AM on July 16, 2018: member

re-utACK c1afe32
laanwj merged this on Jul 16, 2018
laanwj closed this on Jul 16, 2018
fanquake referenced this in commit 19d8ca5cc1 on Jul 16, 2018
ken2812221 deleted the branch on Jul 16, 2018
Sjors commented at 6:39 PM on July 17, 2018: member

On Debian 9 I'm getting: "No such script: /usr/share/debootstrap/scripts/bionic"

As discussed on IRC, it's because debootstrap 1.0.92+ is required, but Debian 9 uses 1.0.89.

This means the current Gitian instructions for Debian 9 need to be updated for 0.17, e.g. to use Bionic instead, but there's some problems with that as well, see #13688.
MarcoFalke commented at 7:19 PM on July 17, 2018: member

Indeed, would be nice to mention the minimum version of debbootstrap required for lxc builds.
theuni commented at 7:30 PM on July 17, 2018: member

@MarcoFalke As @achow101 pointed out on IRC, LXC usage doesn't require debootstrap.

The complication with LXC is that Bionic has moved to LXC3, and that causes problems if the Gitian builder is using LXC2. Gitian has very recent support for that thanks to @ken2812221, so make sure you upgrade Gitian to latest master before anything else.

The solution for me, on Xenial, was to enable xenial-backports and allow LXC3 to be installed from there.
MarcoFalke commented at 7:52 PM on July 17, 2018: member

Ah, thanks for the input. In that case it should be removed from the programs that are installed https://github.com/bitcoin/bitcoin/blob/fa7f8a7769a4acf4368c9d59b852065a9596a079/contrib/gitian-build.py#L16 and probably a check added to assert lxc --version starts with 3, no?
theuni commented at 8:07 PM on July 17, 2018: member

Well, it kinda depends. lxc v3 isn't strictly required, it's only needed when building on Bionic if the host's lxc is < 2.1.1.

There's a bug in lxc 2.1.0 and below that causes incompatibility between v2 hosts and v3 guests. I believe it was fixed upstream by https://github.com/lxc/lxc/commit/2c4bb8e39675fb9bd95959d95367dba54fbbedfd.

See the ubuntu bug report here: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1760848

Unfortunately, those packages haven't been updated for Xenial, and likely won't be: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1776214

I have no clue about Debian or other distros.

So... I think the check should essentially be lxc-execute --version >= 2.1.1
jonasschnelli commented at 8:53 PM on July 19, 2018: contributor

I tried to upgrade my build host which is based on Debian and failed.

Since this requires LXC 2.1.1 or greater, it is currently not possible to run this on the newest version of Debian (9/stretch which currently has LXC in version 2.0.7).

Seems to be odd to drop out Debian as build host.... but maybe there are solutions to work around this (help welcome).
codablock referenced this in commit f411823c9a on Aug 13, 2018
codablock referenced this in commit 5c5bea3d72 on Aug 13, 2018
UdjinM6 referenced this in commit c09f57bd78 on Aug 13, 2018
MarcoFalke referenced this in commit 32647db362 on Sep 7, 2018
karelbilek commented at 2:44 PM on October 14, 2018: contributor

I can confirm that because of this PR, you cannot gitian build on debian but you need to use ubuntu.

You can however use this

https://gist.github.com/laanwj/c62e101bfd68718f0686926dfd10666b

There is this PR that changes the build docs into ubuntu, which is another way of fixing that

https://github.com/bitcoin-core/docs/pull/41

With either the ubuntu PR or the gist, it works (with some more changes)
alejandromgk referenced this in commit eae1493884 on Feb 20, 2019
alejandromgk referenced this in commit 7aea08a255 on Apr 3, 2019
CryptoCentric referenced this in commit b1e5fac728 on Apr 25, 2019
CryptAxe referenced this in commit 240b580985 on Jun 10, 2019
fanquake referenced this in commit ac85e81685 on Aug 20, 2019
fanquake referenced this in commit bd3f5a90ec on Sep 23, 2019
MarcoFalke referenced this in commit 4765b91f50 on Oct 14, 2019
sidhujag referenced this in commit a24510d22d on Oct 15, 2019
Fuzzbawls referenced this in commit ce80c763f6 on Jun 23, 2021
DrahtBot locked this on Sep 8, 2021

Contributors

Labels

Milestone
0.17.0

Linked (view graph)

#12511 Switch to Ubuntu 18.04 for gitian building #13177 GCC-7 and glibc-2.27 back compat code #13688 Gitian with Bionic host #16667 build: remove mingw linker workaround from win gitian descriptor