Scripts and tools: fix gitian build related to Debian/MacOSX-SDK/curl #14176

cisba commented at 11:00 AM on September 9, 2018: contributor

The documentation suggest to use debian 8.x, but the script bin/make-base-vm is invoked with '--suite bionic’ that on debian rises an error (not easy to understand for a newby). It is solved detecting automatically the distro and the suite-codename invoking lsb_release.

The documentation instruct to create the directory ‘gitian-builder/inputs’ with the MacOSX SDK inside. The presence of this path does not consent to clone the gitian-builder repos and rises a subsequent error not clearly related to the issue. It is solved checking the presence of the ‘.git’ inside and moving the SDK temporary outside, relocating at his place after cloning.

The curl package is required for the build process as described here.

Scripts and tools: fix two gitian build issues related to Debian and MacOSX SDK

The [documentation](https://github.com/bitcoin-core/docs/blob/master/gitian-building.md) suggest to use debian 8.x, but the script bin/make-base-vm is invoked with '--suite bionic’ that on debian rises an error (not easy to understand for a newby). It is solved detecting automatically the distro and the suite-codename invoking lsb_release.

The [documentation](https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-mac-os-sdk.md) instrunct to create the directory ‘gitian-builder/inputs’ with the MacOSX  SDK inside. The presence of this path does not consent to clone the gitian-builder repos and rises a subsequent error not clearly related to the issue. It is solved checking the presence of the ‘.git’ inside and moving the SDK temporary outside, relocating at his place after cloning.

1b9dc53402

fanquake added the label Scripts and tools on Sep 9, 2018

in contrib/gitian-build.py:41 in 1b9dc53402 outdated

  38 |          subprocess.check_call(['git', 'clone', 'https://github.com/bitcoin/bitcoin.git'])
  39 |      os.chdir('gitian-builder')
  40 | -    make_image_prog = ['bin/make-base-vm', '--suite', 'bionic', '--arch', 'amd64']
  41 | +    distro = subprocess.check_output(['lsb_release', '-is']).decode("utf-8").strip('\n').lower()
  42 | +    suite = subprocess.check_output(['lsb_release', '-cs']).decode("utf-8").strip('\n')
  43 | +    make_image_prog = ['bin/make-base-vm', '--distro', distro, '--suite', suite, '--arch', 'amd64']

hebasto commented at 11:44 AM on September 9, 2018:

Do not move from bionic. Ref: #12511

cisba commented at 2:12 PM on September 9, 2018:

Ah... ok, then I will PR to correct the documentation to use ubuntu/bionic. Thanks @hebasto ! And debian/stretch?

ken2812221 commented at 2:14 PM on September 9, 2018:

The documentation suggest to use debian 8.x, but the script bin/make-base-vm is invoked with '--suite bionic’ that on debian rises an error

You should upgrade your lxc >= 2.1.1 and debootstrap >= 1.0.92

achow101 commented at 4:05 PM on September 9, 2018:

Do not change this line, revert your changes. The VM that is built is for the gitian build and it's distro and suite have nothing to do with the host OS. The same distro and suite must be used in order to have the same binaries be built.

cisba commented at 5:51 PM on September 9, 2018:

gitianuser@debian:~$ dpkg-query -W lxc debootstrap debootstrap 1.0.89 lxc 1:2.0.7-2+deb9u2

gitianuser@debian:~$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.5 (stretch) Release: 9.5 Codename: stretch @ken2812221 It seem that debian 9.x does not provide it... I will quit for a while building on debian/stratch and I'm moving to ubuntu/bionic

MarcoFalke commented at 5:58 PM on September 9, 2018:

@cisba You can use ubuntu bionic or self-compile and install them from source.

MarcoFalke commented at 5:58 PM on September 9, 2018:

See the note in section https://github.com/bitcoin-core/docs/blob/master/gitian-building.md#preparing-the-gitian-builder-host

cisba commented at 6:11 PM on September 9, 2018:

@MarcoFalke I followed it but does not work because bionic is hard coded in gitian scripts. I tried changing it but others errors rise so I quit for a wile debian vm.

cisba commented at 6:14 PM on September 9, 2018:

@hebasto comment is the more remarkable IMHO

Scripts and tools: fix missed curl in setup

The curl package is required for the build process as described [here](https://github.com/bitcoin/bitcoin/blob/master/depends/README.md).

97d9bf88a7

cisba renamed this:
~~Scripts and tools: fix two gitian build issues related to Debian and …~~
Scripts and tools: fix gitian build related to Debian/MacOSX-SDK/curl
on Sep 9, 2018

achow101 commented at 4:13 PM on September 9, 2018: member

NACK. The changes constitute a misunderstanding of how Gitian works and are not an improvement.

The documentation suggest to use debian 8.x, but the script bin/make-base-vm is invoked with '--suite bionic’ that on debian rises an error (not easy to understand for a newby). It is solved detecting automatically the distro and the suite-codename invoking lsb_release.

As mentioned in the comments, this is not how gitian works. The Gitian VM must use specifically Ubuntu Bionic. It is completely independent of the host OS.

The documentation instruct to create the directory ‘gitian-builder/inputs’ with the MacOSX SDK inside. The presence of this path does not consent to clone the gitian-builder repos and rises a subsequent error not clearly related to the issue. It is solved checking the presence of the ‘.git’ inside and moving the SDK temporary outside, relocating at his place after cloning.

The Gitian environment should be setup before fetching the SDK. The documentation should be fixed to reflect that.

The curl package is required for the build process as described here.

That is done inside of the VM itself. Curl is not needed for setting up and starting gitian.

DrahtBot commented at 4:59 PM on September 9, 2018: member

Note to reviewers: This pull request conflicts with the following ones:

#13998 (Scripts and tools: gitian-build.py improvements and corrections by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

cisba commented at 7:55 AM on September 10, 2018: contributor

Thanks @achow101 for pointing me in the right direction. All the issues are caused by documentation. I'll try to fix docs.

cisba closed this on Sep 10, 2018

hebasto commented at 9:22 PM on September 10, 2018: member

@achow101

Curl is not needed for setting up and starting gitian.

curl indeed is needed for gitian building. Without curl we get:

make: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Entering directory '/home/gitianuser/bitcoin/depends'
Fetching boost_1_64_0.tar.bz2 from https://dl.bintray.com/boostorg/release/1.64.0/source/
/bin/sh: 1: curl: not found
Fetching boost_1_64_0.tar.bz2 from https://bitcoincore.org/depends-sources
/bin/sh: 1: curl: not found
funcs.mk:242: recipe for target '/home/gitianuser/gitian-builder/cache/common/download-stamps/.stamp_fetched-boost-boost_1_64_0.tar.bz2.hash' failed

DrahtBot locked this on Sep 8, 2021