depends: Gitian 0.10 fixes #5465

theuni commented at 12:22 AM on December 12, 2014: member

Among other things, fixes #5428. Since this will cause all dependencies to rebuild, I cleaned up a few other things while I was in there:

make download now downloads sources for all OSs.
Updated docs to add make download before gitian build
fixed the comparisontool install process, which was moving the downloaded jar file
moved download stamps into the SOURCES_PATH dir, so they're always in sync
make build-id's deterministic by normalizing sha256 output for patches
other misc fixes and doc cleanups

Verified that when fetching the sources before building, gitian does not attempt to download anything.

depends: teach 'make download' to download sources for all hosts 422f873ef3

depends: Move source stamps to source dir and misc cleanups

Since the last commit will force rebuilds of all depends, take the opportunity
to clean up a few other things that would trigger rebuilds as well.

- Move source stamps to the sources dir so that SOURCES_PATH is respected for
  "make download".
- Only print "fetching..." when actually downloading a file.
- Avoid using non-deterministic paths for the recipe hash (patch location).
  This should ensure that all builders get the same resulting build-ids.
- Use a per-package source paths. This will allow for removing old source files
  in the future.
- Use a host-agnostic path for downloads which gets cleaned up properly.

d546191dc2

depends: fix packages with hard-coded SOURCES_PATH

Also fixes a nasty bug that removes the downloaded comparisontool jar file.

d7db4b6317

docs: release process fixups

Add instructions for manually fetching sources, as well as some misc. fixes.

e27d7cb248

theuni commented at 12:23 AM on December 12, 2014: member

Assuming this works as intended, obviously needs backport to 0.10

laanwj commented at 10:01 AM on December 12, 2014: member

Tested ACK commithash e27d7cb24835c6ce16c12eb758c2d75806103c4a pull #5465. (signature)

I reproduced the issue by isolating gitian in a virtual network with only access to the host:

# create host-only network interface tap0 for VM
sudo ip tuntap add dev tap0 mode tap user orion
# configure it to use the default 10.0.2.2 range
sudo ifconfig tap0 10.0.2.2 netmask 255.255.255.0
# run a dhcp server on the new interface to serve a fixed IP address
touch dummy-dnsmasq.conf
sudo dnsmasq -F 10.0.2.10,10.0.2.10 -a 10.0.2.2 -i tap0 -9 -p 0 -z --conf-file=dummy-dnsmasq.conf --keep-in-foreground

This needs a special version of gitian that configures qemu differently (changes in https://github.com/laanwj/gitian-builder/commit/6c8131ca866103aeab5a80d85610e05d706e9725)

Without this pull, var/build.log stops at fetching the native_comparisontool

Fetching native_comparisontool...
wget: unable to resolve host address `github.com'
wget: unable to resolve host address `bitcoincore.org'

With this pull, pre-downloading the sources with:

make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common

Successfully builds output, in the case of Linux:

d465839ef122e5776d8bde9c519416a97b337b005f4acfd4ee154584548cdf47  bitcoin-0.10.99-linux32.tar.gz
de435c80e6cdc6d1925923f2aecb00f2dc33453187ccf6419da42425494e9168  bitcoin-0.10.99-linux64.tar.gz
870b49de42d17d43a6b30b4909f88ac65682bbbb67b36be71f85c562d5768ac3  src/bitcoin-0.10.99.tar.gz
632762424db261bf7d9c5c0df2ce3d8d23b71cf9f2b1175b334199b4b80e9365  bitcoin-linux-0.10-res.yml

Windows:

e125fe1d2ff47a1ddeedebd511e94cb9ac1d8afc9b112a6d4d10d02144aa15f9  bitcoin-0.10.99-win32-setup.exe
6b62a5d1675dc48d6d201278faebd0a86d14972de772cbf834b6066329f6ebd6  bitcoin-0.10.99-win32.zip
ad58db3ba684e3eb136fb76e0d33df976c22b7ac3eb20fd9cf2bdeb1e8f137e2  bitcoin-0.10.99-win64-setup.exe
639abaa44677157e92b69d57590baf562b6980ed4fe50e6b5e08ceba4479d0be  bitcoin-0.10.99-win64.zip
870b49de42d17d43a6b30b4909f88ac65682bbbb67b36be71f85c562d5768ac3  src/bitcoin-0.10.99.tar.gz
e58f9ad337d60151d5e515e4d0feb3f269eb98ffec6e8d734fc2d93d497a4a81  bitcoin-win-0.10-res.yml

MacOSX:

8bad112c68cb5e9bad679c1dfe81ba9862f8cf0f29a8f40c39d71208c0089313  bitcoin-0.10.99-osx-unsigned.dmg
bba27015a6d2666dd6a7cdac1473cb402bcf7a70d0d84753489637f68fd71213  bitcoin-0.10.99-osx-unsigned.tar.gz
b42288b6e1b408fe84d5cef83b9d2f1b14e3a5c96e9c52f7dbd8ad34e640a5f5  bitcoin-0.10.99-osx64.tar.gz
870b49de42d17d43a6b30b4909f88ac65682bbbb67b36be71f85c562d5768ac3  src/bitcoin-0.10.99.tar.gz
7bf6b8afcf7675959ce63d4f69957ad35a7ada8a67aba4845b663f7df2f80bc3  bitcoin-osx-0.10-res.yml

laanwj added the label Build system on Dec 12, 2014

laanwj added this to the milestone 0.10.0 on Dec 12, 2014

laanwj merged this on Dec 12, 2014

laanwj closed this on Dec 12, 2014

laanwj referenced this in commit 1bcb52143a on Dec 12, 2014

laanwj referenced this in commit 0133e4fb48 on Dec 12, 2014

btcdrak commented at 8:54 AM on December 13, 2014: contributor

I get the following error with make download

http://distro.ibiblio.org/fatdog/source/c/cdrkit-1.1.11.tar.bz2:
2014-12-13 03:47:37 ERROR 404: Not Found.
ERROR: The certificate of `bitcoincore.org' is not trusted.
make[1]: *** [/home/debian/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_cdrkit-cdrkit-1.1.11.tar.bz2] Error 5
make: *** [download-osx] Error 2

btcdrak commented at 9:01 AM on December 13, 2014: contributor

The link has changed to http://distro.ibiblio.org/fatdog/source/600/c/cdrkit-1.1.11.tar.bz2

laanwj commented at 9:02 AM on December 13, 2014: member

Thanks for the new link @btcdrak, will update

btcdrak commented at 9:03 AM on December 13, 2014: contributor

I've just made a patch #5471

btcdrak commented at 9:09 AM on December 13, 2014: contributor

It also looks like bitcoincore.org needs a properly signed SSL cert?

laanwj commented at 9:54 AM on December 13, 2014: member

It has, try going to https://bitcoincore.org/ in a recent browser. The problem is that the root certificates of your OS (let me guess: debian) are outdated. It is a problem that should fix itself in time as packages are updated.

btcdrak commented at 10:12 AM on December 13, 2014: contributor

@laanwj Doesnt help when we are recommending Debian as the gitian VM :-P

laanwj commented at 10:23 AM on December 13, 2014: member

Switching it over to Debian Jessie would likely work, or maybe even Ubuntu.

btcdrak commented at 10:27 AM on December 13, 2014: contributor

Well, if it produces a hard error we have to do something. Maybe the easier route would be to update the CA bundle?

laanwj commented at 10:33 AM on December 13, 2014: member

Sure, instead of bumping the entire OS you may be able to install just the CA bundle from testing and/or experimental. According to @harding on IRC that is possible as it doesn't have any special dependencies.

OTOH hanging the recommended OS will result in less steps overall. For example the explicit step to instal python-vm-builder would not be needed on Ubuntu. Maybe a package for it exists in Jessie too.

Edit: another possibility is to add an option to depends to provide --no-check-certificate. I wouldn't like that to be the default, but it could be provided to work around this. The hashes of the packages are checked so it'd be safe.

theuni commented at 1:30 AM on February 10, 2015: member

Documenting here for posterity. The certificate issue on bitcoincore.org has been resolved. It turned out to be a server issue related to certificate order. Some clients fail when the order is switched, some don't.

The order is now correct on the server.

MarcoFalke locked this on Sep 8, 2021