Easy to use gitian building script #8566

achow101 commented at 8:24 PM on August 23, 2016: member

I've written a script that automates the setup and building of binaries with gitian. All of the commands are pulled from various documentation on gitian building.

The script allows you to specify unsigned or signed builds, which os to build for, the signer, the version (or commit/branch), and build arguments (threads and memory).

Right now setup is designed for debian based systems.

It runs a few commands that require sudo. For best automation, add apt-get, lxc-start, lxc-execute, and ifconfig to your sudoers file.

achow101 force-pushed on Aug 23, 2016

jonasschnelli added the label Build system on Aug 24, 2016

jonasschnelli commented at 6:39 AM on August 24, 2016: contributor

Nice! Concept ACK.

isle2983 commented at 2:30 AM on August 25, 2016: contributor

Comment spelling/grammar nits:

s/OtherBasic/Other basic/ s/Debian based/Debian-based/ s/arguement/argument/ s/# Window/# Windows/ s/# Sign Sindows/# Sign Windows/

achow101 force-pushed on Aug 25, 2016

achow101 commented at 3:04 AM on August 25, 2016: member

Nits addressed

laanwj commented at 8:43 AM on August 25, 2016: member

Concept ACK, haven't tested

btcdrak commented at 9:59 AM on August 25, 2016: contributor

Concept ACK, I'm testing this out.

sipa commented at 10:17 AM on August 25, 2016: member

One feature I've always wanted (but never bothered to add to my own build scripts) is detached signing: I don't want my GPG key on the build machine. However, if you don't use gsign, the assert file is overwritten for every build. So if possible I'd like to have a means to just copy the assert file to the right location in gitian.sigs, but without adding the gpg signature yet.

in contrib/gitian-build.sh:None in 6358aed004 outdated

  36 | +-B|--buildsign	Build both signed and unsigned binaries
  37 | +-o|--os		Specify which Operating Systems the build is for. Default is lwx. l for linux, w for windows, x for osx
  38 | +-j		Number of processes to use. Default 4
  39 | +-m		Memory to allocate in MiB. Default 2000
  40 | +--lxc           Use LXC instead of KVM
  41 | +--setup         Setup the gitian building environment. Uses KVM. If you want to use lxc, use the --lxc option. Only works on Debian-based systems (Ubuntu, Debian)

btcdrak commented at 11:12 AM on August 25, 2016:

This should default to lxc because it is the default according to the instructions we ship.

in contrib/gitian-build.sh:None in 6358aed004 outdated

 172 | +fi
 173 | +
 174 | +# Check that a signer is specified
 175 | +if [[ $SIGNER == "" ]]
 176 | +then
 177 | +    echo "Must specify a signer"

btcdrak commented at 11:20 AM on August 25, 2016:

if signer is missing you should display the usage

for example the ln command spits out

$ ln
ln: missing file operand
Try 'ln --help' for more information.

So I suggest:

$0: Missing signer
Usage: $0 [-c|u|v|b|s|B|o|h|j|m|] signer version

or

$0: Missing signer
Try $0 -h

in contrib/gitian-build.sh:None in 6358aed004 outdated

 179 | +fi
 180 | +
 181 | +# Check that a version is specified
 182 | +if [[ $VERSION == "" ]]
 183 | +then
 184 | +    echo "Must specify version"

btcdrak commented at 11:20 AM on August 25, 2016:

Suggest

$0: Missing version
Usage: $0 [-c|u|v|b|s|B|o|h|j|m|] signer version

or

$0: Missing version
Try $0 -h

in contrib/gitian-build.sh:None in 6358aed004 outdated

  17 | +proc=4
  18 | +mem=4000
  19 | +
  20 | +# Help Message
  21 | +read -d '' usage <<- EOF
  22 | +Usage: gitian-build.sh [-c|u|v|b|s|B|o|h|j|m|] signer version

btcdrak commented at 11:22 AM on August 25, 2016:

Consider using $0 in place of gitian-build.sh

in contrib/gitian-build.sh:None in 6358aed004 outdated

 227 | +	echo "Building Dependencies"
 228 | +	echo ""
 229 | +	pushd ./gitian-builder	
 230 | +	mkdir -p inputs
 231 | +	wget -N -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch
 232 | +	wget -N -P inputs http://downloads.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz

btcdrak commented at 11:25 AM on August 25, 2016:

Can I suggest we make these two URLs a variable defined at the beginning of the file. If this dep is updated in the future it will make it easier to spot.

btcdrak commented at 11:28 AM on August 25, 2016: contributor

I agree with @sipa, we need a "--sign-detached" option. In my case my signing keys are on a hardware device which is not available to the VM so I have to copy (scp) the gitian.sigs/* out to my main machine to detach sign. Certainly running the --sign command works, you end up with the asserts in the gitian.sigs folder but of course it generates error messages.

btcdrak commented at 11:30 AM on August 25, 2016: contributor

This could really do with checking that the MacOS SDK file exists in the gitian-builder/inputs/ folder

btcdrak commented at 11:49 AM on August 25, 2016: contributor

Tested various options, works, makes me happy.

achow101 force-pushed on Aug 25, 2016

achow101 commented at 7:29 PM on August 25, 2016: member

Addressed btcdrak's concerns.

I will see what I can do regarding detached signing.

achow101 commented at 7:35 PM on August 25, 2016: member

So what exactly do you guys want with detached signing?

sipa commented at 7:37 PM on August 25, 2016: member

By detached signing I mean it behaves exactly as if it's signing, and the .assert file appears inside the gitian.sigs dir, but no GPG signature for it is created.

Thanks a lot for looking into this wishlist item :)

achow101 commented at 11:36 PM on August 25, 2016: member

@sipa @btcdrak I think I got it to create just the assert files for detached signing. I also added an option to not automatically commit the sigs.

laanwj commented at 9:56 AM on August 26, 2016: member

By detached signing I mean it behaves exactly as if it's signing, and the .assert file appears inside the gitian.sigs dir, but no GPG signature for it is created.

My favorite hack for this is:

gsign -p true ...

(This uses the program "true" to sign, which effectively skips the step and returns success)

laanwj commented at 9:58 AM on August 26, 2016: member

Oh that's what you did already, great!

ghost commented at 8:03 PM on August 26, 2016: none

"--setup" has failed because it is missing the clone bitcoin command:

git clone https://github.com/bitcoin/bitcoin

Update: Fair point as I had it from somewhere else; should probably mention that somewhere for newcomers attempting this. The script itself seems to be properly working, although it won't work in a VM due to other reasons.

achow101 commented at 8:07 PM on August 26, 2016: member

The assumption is that you already have that

On August 26, 2016 4:03:39 PM Lauda notifications@github.com wrote:

"--setup" has failed because it is missing the clone bitcoin command:

git clone https://github.com/bitcoin/bitcoin

You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #8566 (comment)

achow101 force-pushed on Aug 26, 2016

Create an easy to use gitian building script

I've written a script that automates the setup and building of binaries with gitian. All of the commands are pulled from various documentation on gitian building.

eda4cfb992

Check for OSX SDK 498d8dadbf

Create option to detach sign gitian builds and not commit the files in the script

Gitian builds can be done where the assert files are made but no signing is done. The sigs can also optionally not be committed automatically.

6ffd6b4a57

achow101 force-pushed on Aug 27, 2016

in contrib/gitian-build.sh:None in 6ffd6b4a57 outdated

  40 | +-b|--build	Do a gitiain build
  41 | +-s|--sign	Make signed binaries for Windows and Mac OSX
  42 | +-B|--buildsign	Build both signed and unsigned binaries
  43 | +-o|--os		Specify which Operating Systems the build is for. Default is lwx. l for linux, w for windows, x for osx
  44 | +-j		Number of processes to use. Default 4
  45 | +-m		Memory to allocate in MiB. Default 2000

fanquake commented at 4:31 AM on August 28, 2016:

nit: looks like the default is actually 4000.

luke-jr commented at 4:33 AM on August 28, 2016:

The default probably should be 2000. 32-bit KVM has a limit of 2047...

MarcoFalke commented at 5:43 AM on August 28, 2016:

@luke-jr this would require setting -j to 1 as well. I don't think those are sensible defaults for gitian building.

btcdrak commented at 9:54 AM on August 28, 2016:

It's time @luke-jr upgraded to 64 bits :8ball:

achow101 commented at 1:37 PM on August 28, 2016:

The default from gitian is -j 2 and -m 2000, so I will set that to be the default.

sipa commented at 1:38 PM on August 28, 2016:

I would expect -j2 -m2000 to OOM.

achow101 commented at 1:41 PM on August 28, 2016:

It doesn't (at least, I haven't seen it do that). I have run gitian with the defaults before and it works fine, albeit rather slowly.

MarcoFalke commented at 3:16 PM on August 28, 2016:

It was causing OOM for me (on kvm, though).

luke-jr commented at 4:49 PM on August 28, 2016:

I have no problems building with -j2 -m2047 at least (KVM).

laanwj commented at 12:02 PM on August 28, 2016: member

doc/gitian-building.md should be updated to use this script. (not necessarily in this pull, but it makes sure that we have a "supported" build process that uses this script)

btcdrak commented at 12:50 PM on August 28, 2016: contributor

Tested ACK 6ffd6b4a573e00644b8985c825357a7aa7b8ce88 (documentation nit aside)

Set defaults to gitian defaults

Set the default -j and -m to 2 and 2000 respectively, matching gitian's default values.

3fe0b68fb4

laanwj merged this on Aug 30, 2016

laanwj closed this on Aug 30, 2016

laanwj referenced this in commit 7b98895865 on Aug 30, 2016

codablock referenced this in commit abecf550b1 on Sep 19, 2017

codablock referenced this in commit 3a1b273ba6 on Jan 9, 2018

codablock referenced this in commit fc34ec4b3f on Jan 9, 2018

andvgal referenced this in commit 18baf7332d on Jan 6, 2019

DrahtBot locked this on Sep 8, 2021