Change gitian-descriptors to use bionic instead

ken2812221 commented at 3:44 am on May 5, 2018: contributor

I have tested this with Ubuntu Bionic host with lxc 3.0.0, the gitian-build for Windows and MacOSX work fine, but there is an issue about it for Linux. Failed at check-symbol:

0test/test_bitcoin: symbol __divmoddi4 from unsupported version GCC_7.0.0
1test/test_bitcoin: symbol log2f from unsupported version GLIBC_2.27
2qt/bitcoin-qt: symbol __divmoddi4 from unsupported version GCC_7.0.0
3qt/bitcoin-qt: symbol log2f from unsupported version GLIBC_2.27

I think this should be fixed in ./configure --enable-glibc-back-compat

Should not be merged before #13177 devrandom/gitian-builder#178

Close #12511

fanquake added the label Build system on May 5, 2018

fanquake requested review from theuni on May 5, 2018

MarcoFalke commented at 2:16 pm on May 5, 2018: member

Thanks! Concept ACK.

ken2812221 force-pushed on May 6, 2018

laanwj commented at 12:56 pm on May 7, 2018: member

Concept ACK

MarcoFalke added this to the milestone 0.17.0 on May 31, 2018

luke-jr commented at 1:35 am on May 31, 2018: member

Need to be sure bionic images can be made for LXC, VirtualBox, and KVM. Last I checked there was an issue doing it for KVM :/

ken2812221 commented at 3:05 am on May 31, 2018: contributor

It seems there is no more updates for python-vm-builder. @achow101 introduced a way to do gitian-build with docker, so I thought we could drop the KVM way if we couldn’t find any patch to build KVM image. See #13368

ken2812221 renamed this:
~~[WIP] Change gitian-descriptors to use bionic instead~~
Change gitian-descriptors to use bionic instead
on Jun 2, 2018

MarcoFalke commented at 2:24 pm on July 11, 2018: member

utACK 997816bcc7d59a3bdb68c677ef0a6bce00210aa0

This hopefully fixes the currently broken gitian windows cross compiles of the gui (qt 5.9.6)

MarcoFalke assigned theuni on Jul 11, 2018

in contrib/gitian-descriptors/gitian-win.yml:98 in 997816bcc7 outdated

94@@ -93,6 +95,9 @@ script: |
95     done
96   done
97   }
98+  for i in $HOSTS; do

theuni commented at 6:54 pm on July 12, 2018:

It should suffice to add a symlink from ${WRAP_DIR}/${i}/${i}-g++ to /usr/bin/${i}-g++-posix``` instead. Since this is in our path, that means that it would be picked up before the generic one.

in doc/release-process.md:121 in 997816bcc7 outdated

117@@ -118,7 +118,7 @@ The gbuild invocations below <b>DO NOT DO THIS</b> by default.
118     ./bin/gsign --signer "$SIGNER" --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
119     mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../
120 
121-    ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
122+    ./bin/gbuild --allow-sudo --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml

theuni commented at 6:54 pm on July 12, 2018:

This shouldn’t be needed if the symlink trick works.

in contrib/gitian-build.sh:284 in 997816bcc7 outdated

280@@ -281,7 +281,7 @@ then
281 	    echo ""
282 	    echo "Compiling ${VERSION} Windows"
283 	    echo ""
284-	    ./bin/gbuild -j ${proc} -m ${mem} --commit bitcoin=${COMMIT} --url bitcoin=${url} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
285+	    ./bin/gbuild --allow-sudo -j ${proc} -m ${mem} --commit bitcoin=${COMMIT} --url bitcoin=${url} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml

theuni commented at 6:55 pm on July 12, 2018:

This also shouldn’t be needed if the symlink trick works.

ken2812221 force-pushed on Jul 12, 2018

DrahtBot added the label Needs rebase on Jul 12, 2018

in contrib/gitian-descriptors/gitian-win.yml:8 in 6475f1db7a outdated

1@@ -2,7 +2,7 @@
2 name: "bitcoin-win-0.17"
3 enable_cache: true
4 suites:
5-- "trusty"
6+- "bionic"
7 architectures:
8 - "amd64"
9 packages:

MarcoFalke commented at 7:44 pm on July 12, 2018:

Should rename “git-core” to “git”?

ken2812221 commented at 7:56 pm on July 12, 2018:

Yes, I already do.

MarcoFalke commented at 7:59 pm on July 12, 2018:

In line 11 of this file in the current version it still says “git-core”, no?

ken2812221 commented at 8:04 pm on July 12, 2018:

Oh, I missed that one.

ken2812221 force-pushed on Jul 12, 2018

DrahtBot removed the label Needs rebase on Jul 13, 2018

MarcoFalke commented at 1:48 am on July 13, 2018: member

re-utACK 5eaee74b6c2acae33f806fa60da757ea6fcbc689

theuni commented at 4:09 am on July 13, 2018: member

ACK 5eaee74b6c2acae33f806fa60da757ea6fcbc689. I haven’t tested the resulting binaries yet, but I can at least confirm that linux/macos/win build successfully.

I wrestled with Gitian and KVM/LXC all day to get it built, I’ll write up some notes and work-arounds.

fanquake commented at 12:35 pm on July 15, 2018: member

utACK 5eaee74

LGTM There’s a lot of work building on/ issues solved by this.

fanquake commented at 7:16 am on July 16, 2018: member

@ken2812221 Could you please rebase this.

Change gitian-descriptors to use bionic instead c1afe3232f

ken2812221 force-pushed on Jul 16, 2018

ken2812221 commented at 10:43 am on July 16, 2018: contributor

@fanquake Rebased

fanquake commented at 10:45 am on July 16, 2018: member

re-utACK c1afe32

laanwj merged this on Jul 16, 2018

laanwj closed this on Jul 16, 2018

fanquake referenced this in commit 19d8ca5cc1 on Jul 16, 2018

ken2812221 deleted the branch on Jul 16, 2018

Sjors commented at 6:39 pm on July 17, 2018: member

On Debian 9 I’m getting: “No such script: /usr/share/debootstrap/scripts/bionic”

As discussed on IRC, it’s because debootstrap 1.0.92+ is required, but Debian 9 uses 1.0.89.

This means the current Gitian instructions for Debian 9 need to be updated for 0.17, e.g. to use Bionic instead, but there’s some problems with that as well, see #13688.

MarcoFalke commented at 7:19 pm on July 17, 2018: member

Indeed, would be nice to mention the minimum version of debbootstrap required for lxc builds.

theuni commented at 7:30 pm on July 17, 2018: member

@MarcoFalke As @achow101 pointed out on IRC, LXC usage doesn’t require debootstrap.

The complication with LXC is that Bionic has moved to LXC3, and that causes problems if the Gitian builder is using LXC2. Gitian has very recent support for that thanks to @ken2812221, so make sure you upgrade Gitian to latest master before anything else.

The solution for me, on Xenial, was to enable xenial-backports and allow LXC3 to be installed from there.

MarcoFalke commented at 7:52 pm on July 17, 2018: member

Ah, thanks for the input. In that case it should be removed from the programs that are installed https://github.com/bitcoin/bitcoin/blob/fa7f8a7769a4acf4368c9d59b852065a9596a079/contrib/gitian-build.py#L16 and probably a check added to assert lxc --version starts with 3, no?

theuni commented at 8:07 pm on July 17, 2018: member

Well, it kinda depends. lxc v3 isn’t strictly required, it’s only needed when building on Bionic if the host’s lxc is < 2.1.1.

There’s a bug in lxc 2.1.0 and below that causes incompatibility between v2 hosts and v3 guests. I believe it was fixed upstream by https://github.com/lxc/lxc/commit/2c4bb8e39675fb9bd95959d95367dba54fbbedfd.

See the ubuntu bug report here: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1760848

Unfortunately, those packages haven’t been updated for Xenial, and likely won’t be: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1776214

I have no clue about Debian or other distros.

So… I think the check should essentially be lxc-execute --version >= 2.1.1

jonasschnelli commented at 8:53 pm on July 19, 2018: contributor

I tried to upgrade my build host which is based on Debian and failed.

Since this requires LXC 2.1.1 or greater, it is currently not possible to run this on the newest version of Debian (9/stretch which currently has LXC in version 2.0.7).

Seems to be odd to drop out Debian as build host…. but maybe there are solutions to work around this (help welcome).

codablock referenced this in commit f411823c9a on Aug 13, 2018

codablock referenced this in commit 5c5bea3d72 on Aug 13, 2018

UdjinM6 referenced this in commit c09f57bd78 on Aug 13, 2018

MarcoFalke referenced this in commit 32647db362 on Sep 7, 2018

karelbilek commented at 2:44 pm on October 14, 2018: contributor

I can confirm that because of this PR, you cannot gitian build on debian but you need to use ubuntu.

You can however use this

https://gist.github.com/laanwj/c62e101bfd68718f0686926dfd10666b

There is this PR that changes the build docs into ubuntu, which is another way of fixing that

https://github.com/bitcoin-core/docs/pull/41

With either the ubuntu PR or the gist, it works (with some more changes)

alejandromgk referenced this in commit eae1493884 on Feb 20, 2019

alejandromgk referenced this in commit 7aea08a255 on Apr 3, 2019

CryptoCentric referenced this in commit b1e5fac728 on Apr 25, 2019

CryptAxe referenced this in commit 240b580985 on Jun 10, 2019

fanquake referenced this in commit ac85e81685 on Aug 20, 2019

fanquake referenced this in commit bd3f5a90ec on Sep 23, 2019

MarcoFalke referenced this in commit 4765b91f50 on Oct 14, 2019

sidhujag referenced this in commit a24510d22d on Oct 15, 2019

Fuzzbawls referenced this in commit ce80c763f6 on Jun 23, 2021

DrahtBot locked this on Sep 8, 2021

Change gitian-descriptors to use bionic instead #13171