Add scripts for doing gitian builds on any platform using VirtualBox + Vagrant + Packer #12656

maaku commented at 6:51 PM on March 9, 2018: contributor

This enables gitian builds from any operating system supporting Oracle VirtualBox and HashiCorp's vagrant and packer VM-creation utilities. Simply checkout a copy of the bitcoin source tree with this PR applied from an internet-connected machine, and do cd contrib/vagrant && make. Dependencies and build products are properly cached to provide faster subsequent builds. Builds from dirty working trees are also supported, with the uncommitted changes applied to the source tree.

Builds are nearly identical to the usual gitian process. At this time the only difference is due to a separate bug that makes the length of the GIT_COMMIT_ID string exported by git-archive dependent on the state of the local git repository, which has the further implication separate from this use case that over time it would prevent reproducibility of historical releases as well. That issue is logically separate and should be fixed on its own.

This PR is a re-opening of PR #1597 from July of 2012. Unfortunately GitHub forced creation of a new PR.

maaku force-pushed on Mar 9, 2018

in contrib/vagrant/Makefile:148 in 253390a6ba outdated

 143 | +endef
 144 | +check-requirements: .stamp-check-requirements.h
 145 | +.stamp-check-requirements.h: $(MAKEFILE)
 146 | +	@echo '[check requirements]'
 147 | +	$(foreach REQUIREMENT,$(REQUIREMENTS),$(call CHECK_REQUIREMENT,$(REQUIREMENT)))
 148 | +	$(call CHECK_REQUIREMENT_VERSION,vagrant,[2-9]\.[0-9]\.[0-9])

instagibbs commented at 7:35 PM on March 9, 2018:

fwiw Ubuntu 16.04 LTS fails this check, with 1.8.1.

maaku commented at 7:53 PM on March 9, 2018:

Vagrant 2.0+ is required.

maaku commented at 7:15 PM on March 11, 2018:

To be clear, yes Ubuntu 16.04 LTS fails this check. That’s why the check exists :) I updated the docs to have directions for installing from the source.

maaku commented at 7:16 PM on March 11, 2018:

To be clear, yes Ubuntu 16.04 LTS fails this check. That’s why the check exists :) I updated the docs to have directions for installing from the source.

in contrib/vagrant/README.md:71 in 253390a6ba outdated

  66 | +    $ make check-requirements
  67 | +
  68 | +If you cannot find `vagrant` in your distribution's package
  69 | +repositories, you can install it using ruby gems:
  70 | +
  71 | +    $ sudo gem install vagrant

instagibbs commented at 7:36 PM on March 9, 2018:

"Vagrant is no longer distributed as a RubyGem."

maaku force-pushed on Mar 9, 2018

fanquake added the label Scripts and tools on Mar 9, 2018

maaku force-pushed on Mar 9, 2018

setpill commented at 2:25 PM on March 14, 2018: contributor

Running into the following error trying to run this on Ubuntu 17.10:

[..]
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default: 
    default: Guest Additions Version: 5.0.18_Ubuntu r106667
    default: VirtualBox Version: 5.1
==> default: Mounting shared folders...
    default: /vagrant => [redacted]/bitcoin/contrib/vagrant
Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel module.
Please verify that these guest additions are properly installed in the
guest. This is not a bug in Vagrant and is usually caused by a faulty
Vagrant box. For context, the command attempted was:

mount -t vboxsf -o uid=1000,gid=1000 vagrant /vagrant

The error output from the command was:

mount: unknown filesystem type 'vboxsf'

Makefile:220: recipe for target '.stamp-gitian-host.manifest' failed
make: *** [.stamp-gitian-host.manifest] Error 1

Which is curious, since I can see earlier in the log it is downloading version 5.1.30 of the guest additions iso, and the system-installed version of the virtualbox-guest-additions-iso package is 5.1.30-1.

maaku commented at 3:54 PM on March 14, 2018: contributor

Right. For that reason I’m pretty sure this is an upstream issue with vagrant and/or virtualbox on your platform, like it is unable to communicate with the host support daemon or something. There isn’t anything special that these scripts do with respect to vbox configuration.

setpill commented at 4:49 PM on March 14, 2018: contributor

I reinstalled virtualbox using virtualbox.org's binary release channel, vagrant was already installed from vagrantup (since Ubuntu 17.10 is still on 1.9.x). Same error (but with VirtualBox Version: 5.2 instead of VirtualBox Version: 5.1).

It seems like 5.0.18 corresponds to the virtualbox-guest-additions-iso package in xenial's repositories. Out of curiosity, what host platform are you running it successfully on? What is the virtualbox version installed there?

maaku commented at 4:55 PM on March 14, 2018: contributor

Most recently on macOS 10.13.3 with VirtualBox 5.2.8 r121009. But I previously had this running on a 16.04 LTS integration server, before making some changes for the PR that should be unrelated. I'll try again on both 16.04 and 17.10.

While I'm doing that, maybe try upgrading to the vbox 5.2 series?

maaku commented at 4:56 PM on March 14, 2018: contributor

Also you can get Ubuntu binaries from Oracle's (non-Launchpad) PPA here:

https://www.virtualbox.org/wiki/Linux_Downloads

setpill commented at 4:57 PM on March 14, 2018: contributor

That's exactly where I reinstalled it from :), edited comment to clarify I am on vbox 5.2 series.

maaku commented at 5:00 PM on March 14, 2018: contributor

@setpill are you running into this bug?

https://github.com/hashicorp/vagrant/issues/8502

maaku commented at 5:05 PM on March 14, 2018: contributor

@setpill Maybe I was misreading the output you posted above. Is this a failure at the packer stage or with vagrant up and provisioning? Did packer complete construction of the box template?

setpill commented at 5:08 PM on March 14, 2018: contributor

Considering that that is a bug introduced in guest-additions version 5.1.20 and fixed in 5.1.22, and I am either running 5.1.30/5.2.8 or 5.0.18, I would suspect not.

I will run make again and capture a full log for reference.

setpill commented at 5:33 PM on March 14, 2018: contributor

Log here

setpill commented at 9:11 PM on March 14, 2018: contributor

vagrant plugin install vagrant-vbguest fixed the problem; possibly an idea to include in the readme. Building from the vagrant-0.16 branch did not generate matching out_manifests for me but this could be ascribed to the separate bug in the first comment.

maaku commented at 9:32 PM on March 14, 2018: contributor

Thanks for the update. The vagrant-X branches are private branches not meant for consumption by others. I’m surprised it worked at all.

On Mar 14, 2018, at 2:13 PM, setpill notifications@github.com wrote:

vagrant plugin install vagrant-vbguest fixed the problem; possibly an idea to include in the readme. Building from the vagrant-0.16 branch did not generate matching out_manifests for me but this could be ascribed to the separate bug in the first comment.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Add gitian-based, vagrant VM build system to enable building on any operating system platform. 65418eaffc

maaku force-pushed on Mar 14, 2018

maaku commented at 12:58 AM on March 15, 2018: contributor

Documentation has been updated.

fanquake commented at 7:37 AM on March 20, 2018: member

Started testing, looks like the build failed just after finishing the depends downloads:

Fetching xtrans-1.3.4.tar.bz2 from http://xorg.freedesktop.org/releases/individual/lib/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   357  100   357    0     0    517      0 --:--:-- --:--:-- --:--:--   517
100  178k  100  178k    0     0  38427      0  0:00:04  0:00:04 --:--:-- 59783
/Users/fanquake/Downloads/bitcoin-vagrant/depends/work/download/xtrans-1.3.4/xtrans-1.3.4.tar.bz2.temp: OK
mkdir -p output
vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
vagrant ssh --command "rm -rf ~/'bitcoin' || true"; vagrant ssh --command "mkdir -p ~/'bitcoin'" || exit 1; vagrant ssh --command "cd ~/'bitcoin' && tar -xf /vagrant/cache/src-local.tar.xz" || exit 1; vagrant ssh --command "cd ~/'bitcoin' && grep -v 'contrib/vagrant/Makefile' configure.ac >.configure.ac.2 && mv .configure.ac.2 configure.ac'"; vagrant ssh --command "cd ~/'bitcoin' && git init" || exit 1; vagrant ssh --command "cd ~/'bitcoin' && git config user.name 'Gitian Builder'" || exit 1; vagrant ssh --command "cd ~/'bitcoin' && git config user.email 'vagrant@vagrant'" || exit 1; vagrant ssh --command "cd ~/'bitcoin' && find . -not \\( -path ./.git -prune \\) -type f -print0 | xargs -0 git add -f" || exit 1; vagrant ssh --command "cd ~/'bitcoin' && env GIT_COMMITTER_NAME='`cat 'cache'/src-local.committer-name`' GIT_COMMITTER_EMAIL='`cat 'cache'/src-local.committer-email`' GIT_COMMITTER_DATE='`cat 'cache'/src-local.committer-date`' git commit --author='`cat 'cache'/src-local.author`' --date='`cat 'cache'/src-local.author-date`' -m tip" || exit 1
Connection to 127.0.0.1 closed.
Connection to 127.0.0.1 closed.
Connection to 127.0.0.1 closed.
bash: -c: line 0: unexpected EOF while looking for matching `''
bash: -c: line 1: syntax error: unexpected end of file
Connection to 127.0.0.1 closed.
Initialized empty Git repository in /home/vagrant/bitcoin/.git/
Connection to 127.0.0.1 closed.
Connection to 127.0.0.1 closed.
Connection to 127.0.0.1 closed.
Connection to 127.0.0.1 closed.
fatal: invalid date format: date
Connection to 127.0.0.1 closed.
make: *** [output/bitcoin-linux-0.17-gitian.zip] Error 1

Also, I wasn't able to use the github-merge script to merge these changes:

fanquake:bitcoin $ merge 12656
ERROR: File contrib/vagrant/README was a symlink
fanquake:bitcoin $

in contrib/vagrant/Makefile:2 in 65418eaffc

   0 | @@ -0,0 +1,516 @@
   1 | +#
   2 | +# Copyright © 2011-2018 Bitcoin Developers.

fanquake commented at 7:38 AM on March 20, 2018:

in contrib/vagrant/README.md:53 in 65418eaffc

  48 | +
  49 | +And the Packer binary needs to be placed in your path:
  50 | +
  51 | +    https://www.packer.io
  52 | +
  53 | +Use MacPorts or homebrew to install any missing dependencies, for example:

fanquake commented at 7:39 AM on March 20, 2018:

I don't think this instruction is required.

setpill commented at 2:31 PM on March 23, 2018: contributor

Windows and OSX builds both fail for me with error cp: cannot stat 'base-trusty-amd64': No such file or directory.

in contrib/vagrant/Makefile:223 in 65418eaffc

 218 | +	.stamp-check-requirements.h \
 219 | +	$(foreach FILE,$(GITIAN_HOST_FILES),$(shell echo $(FILE) | $(SED) 's:^contrib/vagrant/::g'))
 220 | +	
 221 | +	-$(call VAGRANT_DESTROY)
 222 | +	
 223 | +	# Remove any temporary files from interupted builds that would

practicalswift commented at 6:02 PM on September 3, 2018:

Typo found by codespell: interupted

in contrib/vagrant/Makefile:274 in 65418eaffc

 269 | +	
 270 | +	vagrant ssh --command "\
 271 | +	    cd ~/gitian-builder; \
 272 | +	    if [ ! -f $(GUEST_CACHE_DIR)/lxc-base/$(LXC_BASE_MANIFEST)/base-vm.tar.gz ]; then \
 273 | +	        env LXC_BRIDGE=lxcbr0 LXC_GUEST_IP=10.0.3.2 _HOST_IP=10.0.3.1 \
 274 | +	            bin/make-base-vm --lxc --suite trusty --arch amd64 || exit 1; \

MarcoFalke commented at 7:09 PM on September 4, 2018:

Should this say bionic?

maaku commented at 8:25 PM on September 4, 2018:

Only if you want to rewrite the scripts to work with bionic and test it. This is not a typo if that is the question.

MarcoFalke commented at 9:32 PM on September 4, 2018:

Sure, but on master the gitian descriptors use bionic instead of trusty, which is why I left the note.

maaku commented at 11:15 PM on September 4, 2018:

Ah, I see. Should I close this PR but leave the fork available, for now, for others to work off of?

in contrib/vagrant/config/cleanup.sh:48 in 65418eaffc

  43 | +apt-get -y autoclean
  44 | +
  45 | +echo "==> Installed packages"
  46 | +dpkg --get-selections | grep -v deinstall
  47 | +
  48 | +DISK_USAGE_BEFORE_CLEANUP=$(df -h)

practicalswift commented at 2:42 PM on September 11, 2018:

job=2018-09-11-16-26-26 pr=12656 shellcheck: DISK_USAGE_BEFORE_CLEANUP=$(df -h)
job=2018-09-11-16-26-26 pr=12656 shellcheck: ^-- SC2034: DISK_USAGE_BEFORE_CLEANUP appears unused. Verify it or export it.

in contrib/vagrant/config/cleanup.sh:59 in 65418eaffc

  54 | +
  55 | +# Clean up log files
  56 | +find /var/log -type f | while read f; do echo -ne '' > "${f}"; done;
  57 | +
  58 | +echo "==> Clearing last login information"
  59 | +>/var/log/lastlog

practicalswift commented at 2:43 PM on September 11, 2018:

job=2018-09-11-16-26-26 pr=12656 shellcheck: >/var/log/lastlog
job=2018-09-11-16-26-26 pr=12656 shellcheck: ^-- SC2188: This redirection doesn't have a command. Move to its command (or use 'true' as no-op).

in contrib/vagrant/config/cleanup.sh:53 in 65418eaffc

  48 | +DISK_USAGE_BEFORE_CLEANUP=$(df -h)
  49 | +
  50 | +# Remove Bash history
  51 | +unset HISTFILE
  52 | +rm -f /root/.bash_history
  53 | +rm -f /home/${SSH_USER}/.bash_history

practicalswift commented at 2:43 PM on September 11, 2018:

Quote with " to prevent globbing, etc. Fix throughout in this PR :-)

in contrib/vagrant/config/vagrant.sh:34 in 65418eaffc

  29 | +    fi
  30 | +
  31 | +    echo "==> Installing vagrant key"
  32 | +    mkdir $SSH_USER_HOME/.ssh
  33 | +    chmod 700 $SSH_USER_HOME/.ssh
  34 | +    cd $SSH_USER_HOME/.ssh

practicalswift commented at 2:44 PM on September 11, 2018:

job=2018-09-11-16-26-26 pr=12656 shellcheck:     cd $SSH_USER_HOME/.ssh
job=2018-09-11-16-26-26 pr=12656 shellcheck:     ^-- SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

fanquake commented at 4:07 AM on October 26, 2018: member

Closing with "Up for Grabs". However it would seem like this is more suited to the https://github.com/bitcoin-core/packaging in line with #13542?

fanquake closed this on Oct 26, 2018

fanquake added the label Up for grabs on Oct 26, 2018

maaku commented at 9:26 AM on October 26, 2018: contributor

What's the relationship to packaging?

MarcoFalke locked this on Sep 8, 2021

MarcoFalke removed the label Up for grabs on Mar 22, 2022