depends: some base fixes/changes #7809

theuni commented at 11:48 PM on April 4, 2016: member

Fix the annoying "unexpected operator" warning that shows up sometimes.
Fix download fallbacks. Fixes #7627.
Make sure to check all necessary checksums for cctools/qt.
Ensure (make a good guess) that the host/build toolchains haven't changed since the previous build. If they have, invalidate the necessary packages.
Add a salt option for adding extra info to 4. This would allow us to (for example) seed gitian packages with dpkg info, in order to invalidate the cache if certain conditions were met.

I'm pushing these all in one PR because most of these changes will cause full cache invalidation, so we may as well try to only do it once.

depends: create a hostid and buildid and add option for salts

These add very simple sanity checks to ensure that the build/host toolchains
have not changed since the last run. If they have, all ids will change and
packages will be rebuilt.

For more complicated usage (like parsing dpkg), HOST_ID_SALT/BUILD_ID_SALT may
be used to introduce arbitrary data to the ids.

dc4ec6d3db

depends: fix fallback downloads

In some cases, failed downloads wouldn't trigger a fallback download attempt.
Namely, checksum mismatches.

fe740f1469

depends: fix "unexpected operator" error during "make download" bb717f4375

depends: qt/cctools: fix checksum checksum tests

Checksums were being verified after download, but not again before extraction

11d9f6b8b8

laanwj added the label Build system on Apr 5, 2016

MarcoFalke commented at 9:14 AM on April 5, 2016: member

So the patch to the fallback logic will fix checksum issues like https://travis-ci.org/bitcoin/bitcoin/jobs/112855470#L831 ?

But does it also fix the issues where the fallback logic itself appeared to fail? (c.f. https://travis-ci.org/bitcoin/bitcoin/jobs/112846086#L4033 )

laanwj commented at 1:47 PM on April 5, 2016: member

Concept ACK, nice!

theuni commented at 5:17 PM on April 5, 2016: member

Yes, it should fix both. It broadens the retry logic, so that a "try" becomes fetch+validate. If the first try fails, it's retried with the fallback url. In the first example, the download appears to succeed but the checksum doesn't match. Since the download was "successful", it doesn't retry. Now it retries the whole thing with the fallback url. A hacked test (i purposely broke the checksum):

cory@cory-i7:~/dev/bitcoin/depends(depends-updates)$ make download
Checksum missing or mismatched for qrencode source. Forcing re-download.
Fetching qrencode-3.4.4.tar.bz2 from https://fukuchi.org/works/qrencode/
2016-04-05 13:20:14 URL:https://fukuchi.org/works/qrencode//qrencode-3.4.4.tar.bz2 [369136/369136] -> "/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp" [1]
/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
Fetching qrencode-3.4.4.tar.bz2 from https://bitcoincore.org/depends-sources
2016-04-05 13:20:15 URL:http://dev.bitcoincore.org/depends-sources/qrencode-3.4.4.tar.bz2 [369136/369136] -> "/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp" [1]
/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
make[1]: *** [/home/cory/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-qrencode-qrencode-3.4.4.tar.bz2.hash] Error 1
make: *** [download-osx] Error 2

For the second, I'm not sure why that was a problem before. As-is, it looks like it should try to re-download. But as a test, I've forced a similar failure, and it works as intended.

cory@cory-i7:~/dev/bitcoin/depends(depends-updates)$ make download
Checksum missing or mismatched for qrencode source. Forcing re-download.
Fetching qrencode-3.4.4.tar.bz2 from https://fukuchi.org/works/qrencode/
OpenSSL: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
Unable to establish SSL connection.
Fetching qrencode-3.4.4.tar.bz2 from https://bitcoincore.org/depends-sources
OpenSSL: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
Unable to establish SSL connection.
make[1]: *** [/home/cory/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-qrencode-qrencode-3.4.4.tar.bz2.hash] Error 4
make: *** [download-osx] Error 2

laanwj commented at 12:35 PM on April 6, 2016: member

Tested ACK dc4ec6d with my crosstool-ng ARM build

in depends/Makefile:None in 11d9f6b8b8

  74 | @@ -73,6 +75,20 @@ include builders/$(build_os).mk
  75 |  include builders/default.mk
  76 |  include packages/packages.mk
  77 |  
  78 | +build_id_string:=$(BUILD_ID_SALT)
  79 | +build_id_string+=$(shell $(build_CC) --version 2>/dev/null)
  80 | +build_id_string+=$(shell $(build_AR) --version 2>/dev/null)
  81 | +build_id_string+=$(shell $(build_CXX) --version 2>/dev/null)
  82 | +build_id_string+=$(shell $(build_RANLIB) --version 2>/dev/null)
  83 | +build_id_string+=$(shell $(build_STRIP) --version 2>/dev/null)

laanwj commented at 12:39 PM on April 6, 2016:

Do we also want to get the linker info?

theuni commented at 4:08 PM on April 8, 2016:

That's tricky because we don't use the linker directly. It's not specified in depends, because there's no generic way to handle it. I think in 99% of the cases, if the linker changed, one of the above would've changed as well.

laanwj commented at 6:54 AM on April 15, 2016:

True - we never use ld directly, and we shouldn't, for example it would be quite annoying to figure out what the linker is (and what plugin it is using - turns out it is loading /opt/clang39/lib/LLVMgold.so) in my lto setup testing the clang master branch:

   CC="${CLANGPATH}/bin/clang " 
   CXX="${CLANGPATH}/bin/clang++" 
   CFLAGS="-flto -O2 -g" CXXFLAGS="-flto -O2 -g" LDFLAGS="-flto -O2 -fuse-ld=gold"
   RANLIB="${CLANGPATH}/bin/llvm-ranlib"

So yes just wanted to ask to be sure.

laanwj merged this on Apr 15, 2016

laanwj closed this on Apr 15, 2016

laanwj referenced this in commit bbd210d927 on Apr 15, 2016

codablock referenced this in commit c9914d9477 on Sep 16, 2017

codablock referenced this in commit f4f75d097d on Sep 19, 2017

codablock referenced this in commit b2eabb0974 on Dec 20, 2017

zkbot referenced this in commit a96942e809 on Dec 11, 2019

MarcoFalke locked this on Sep 8, 2021