Migrate gitian-build.sh to python #13623

Fixes #13620

• Rename Mac OSX to MacOS, rename option from 'x' to 'm'
• Fix a bug from https://github.com/bitcoin/bitcoin/blob/b641f60425674d737d77abd8c49929d953ea4154/contrib/gitian-build.sh#L338-L342

Thanks // Concept ACK, code looks good to me at first glance but ofcourse needs manual testing (as it's not tested by the test framework).

Could remove the bash script in the same commit?

<!--e57a25ab6845829454e8d69fc972939a-->Note to reviewers: This pull request conflicts with the following ones:

• #13643 (Switch to NSIS 3.03 to avoid DLL hijacking by h4x3rotab)
• #13368 (Update gitian-build.sh for docker by achow101)
• #13171 (Change gitian-descriptors to use bionic instead by ken2812221)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Concept ACK

As gitian-build.sh is mostly calling other executables, I think bash/sh is still the most sensible/readable/least-dependencies choice. According to #13620 there are edge cases to fix, sure, but I'm not convinced that there's a need for a python rewrite.

Static analysis tools run against shell scripts in the check pipeline can eliminate/mitigate edge cases. I'd recommend: https://github.com/koalaman/shellcheck

As gitian-build.sh is mostly calling other executables, I think bash/sh is still the most sensible/readable/least-dependencies choice.

The point is that most of the developers in this project have a much easier time reviewing and maintaining python code. Shell script has turned out to be a continuous struggle against shell differences, bad performance, bugs and other edge cases due to obscure quoting practices, and so on.

You might disagree personally--obviously everything is possible with shell script as well, but porting this to python is more of a social consensus. (as for dependencies, I don't think that is a big thing here, Python 3 is already required to build the project and this requires no external modules)

Also, the resulting python script is a lot shorter. Means less code weight to maintain in the future :)

Update: exit 1 if signer or version is empty.

I only tested some of the basic ergonomics since I don't have the gitian builder set up on my machine yet.

Using a67812d if I pass --setup and already have one of the required repos cloned I see:

Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for ufw (0.35-5) ...
fatal: destination path 'gitian.sigs' already exists and is not an empty directory.
Traceback (most recent call last):
  File "./gitian-build.py", line 188, in <module>
    main()
  File "./gitian-build.py", line 171, in main
    setup()
  File "./gitian-build.py", line 16, in setup
    subprocess.check_call(['git', 'clone', 'https://github.com/bitcoin-core/gitian.sigs.git'])
  File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/bitcoin-core/gitian.sigs.git']' returned non-zero exit status 128.
ubuntu@ubuntu:~$ ls

Removed the repos and re-ran with --setup:

|               E |
+----[SHA256]-----+
sudo: debootstrap: command not found
Traceback (most recent call last):
  File "./gitian-build.py", line 188, in <module>
    main()
  File "./gitian-build.py", line 171, in main
    setup()
  File "./gitian-build.py", line 23, in setup
    subprocess.check_call(make_image_prog)
  File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bin/make-base-vm', '--suite', 'trusty', '--arch', 'amd64', '--lxc']' returned non-zero exit status 1.

--setup --kvm

invoke-rc.d: policy-rc.d denied execution of restart.

Configuration file '/etc/sudoers'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sudoers (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package sudo (--configure):
 EOF on stdin at conffile prompt
invoke-rc.d: policy-rc.d denied execution of restart.
Errors were encountered while processing:
 sudo
E: Sub-process /usr/bin/dpkg returned an error code (1)

Traceback (most recent call last):
  File "./gitian-build.py", line 188, in <module>
    main()
  File "./gitian-build.py", line 171, in main
    setup()
  File "./gitian-build.py", line 23, in setup
    subprocess.check_call(make_image_prog)
  File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bin/make-base-vm', '--suite', 'trusty', '--arch', 'amd64']' returned non-zero exit status 1.

I don't know how to deal with KVM errors since I haven't used KVM ever, I just follow the same command with the original script. Can anyone give a hint?

@ken2812221 Now that #13368 has been merged, can you update the new Python script to add Docker support. I will retest the changes here shortly.

<!--cf906140f33d8803c4a75a2196329ecb-->Needs rebase

This is great, especially not having to worry about the exact sequence of arguments (the bash script had some quirks).

I'm trying to sign master on a Bionic VM (haven't tried a Debian 9 host).

# git clone bitcoin, gitian.sigs, etc
cp bitcoin/contrib/gitian-build.py .
# add MacOS stuff 
python3 gitian-build.py --setup sjors -c master
python3 gitian-build.py --detach-sign --no-commit -b sjors -c master

This throws an error for me, see #13623. The original bash script also throws that error, so probably not related. Just means I can't test the rest of the process on this VM.

When running --setup --docker it throws an error:

ERRO[0000] failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: permission denied 

docker ps also throws a permission error; sudo docker ps works. sudo usermod -aG docker $USER solves the problem. It creates an image base-bionic-amd64.

Trying to build with Docker complains:

Compiling master Linux
--- Building for bionic amd64 ---
Stopping target if it is up
Making a new image copy
Error: No such container: gitian-target

The bash script throws the same complaint, so that's not related.

When running gitian-build.py without arguments, it should probably just return --help (or otherwise point out that this exists).

Is there a corresponding PR to update the instructions? (mostly just renaming .sh to .py)

Can you add make to the dependencies installed during --setup?

For another PR: having a config flag to get a Trusty base image can be useful when building backports.

Compiling master Linux --- Building for bionic amd64 --- Stopping target if it is up Making a new image copy Error: No such container: gitian-target

I believe the docker problem is already exist on gitian-build.sh because I saw the same error trying to use gitian-build.sh to create docker image.

Can you add make to the dependencies installed during --setup?

Why?

Correct, it happens both with bash and python.

Without "make" setup throws an error on a fresh Bionic machine (unless you install all dependencies recommended in the docs, which is overkill).

utACK 78f06e4af76b6e4550b8eed5a521684140a4

On Debian 9, when trying to --setup with --docker, I'm getting Package 'docker.io' has no installation candidate.

So I installed Docker manually, but get the same error. Disabling programs += ['docker.io'] does the trick.