depends: Fix id_string invocations #21427

dongcarl commented at 10:41 PM on March 12, 2021: member

Reproduced from depends/Makefile comment:

When invoking a shell, GNU Make special-cases exit code 127 (command not
found) by not capturing the output but instead passing it through. This
is not done for any other exit code.

Therefore, we require a "|| true" to avoid this behaviour when in an
environment where the build_* or host_* may not exist yet.

hebasto commented at 10:42 PM on March 12, 2021: member

Concept ACK.

DrahtBot added the label Build system on Mar 13, 2021

MarcoFalke added the label Needs gitian build on Mar 13, 2021

MarcoFalke added the label Needs Guix build on Mar 13, 2021

in depends/Makefile:128 in 70ef58f466 outdated

 130 | +# found) by not capturing the output but instead passing it through. This is not
 131 | +# done for any other exit code.
 132 | +#
 133 | +# Therefore, we require a "|| true" to avoid this behaviour when in an
 134 | +# environment where the build_* or host_* may not exist yet.
 135 | +build_id_string+=$(shell $(build_CC) -v 2>&1 || true) $(shell $(build_CC) -v -E -o /dev/null - < /dev/null 2>&1 || true)

hebasto commented at 2:29 PM on March 13, 2021:

Why adding -o /dev/null (here and other places)?

From GCC docs:

The output is in the form of preprocessed source code, which is sent to the standard output. Input files that don’t require preprocessing are ignored.

That means no output for -E when passing /dev/null as an input, right?

dongcarl commented at 4:14 PM on March 15, 2021:

When I execute: gcc -E -v -xc++ - < /dev/null | xclip -sel clip, here's what I get in my clipboard:

# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "<stdin>"

hebasto approved

hebasto commented at 2:31 PM on March 13, 2021: member

ACK 70ef58f466ae0efec0123dc39009b2ff5de95326, tested on Linux Mint 20.1 (x86_64) with the print-build_id_string target:

$ make -C depends print-build_id_string HOST=...

This PR adds -o /dev/null substring to the COLLECT_GCC_OPTIONS value.

DrahtBot commented at 4:55 PM on March 14, 2021: member

Guix builds

File	commit e0bc27a14ceb18bfe3dd9ebdae6bef447c4ec922<br>(master)	commit 8d549770606d969875a5f1c60fcb3babfc5dced5<br>(master and this pull)
*-aarch64-linux-gnu-debug.tar.gz	`3b30f4198b68fadf...`	`d8635db4270e720c...`
*-aarch64-linux-gnu.tar.gz	`176d561bd8a2aa0d...`	`4024f5420394acdf...`
*-arm-linux-gnueabihf-debug.tar.gz	`ea017c7dff2b99be...`	`c0d9623c2363710e...`
*-arm-linux-gnueabihf.tar.gz	`3ce0b418bce0f580...`	`8c66d08e50611dd9...`
*-osx-unsigned.dmg	`772d2aa9989e8d40...`	`0f85f18488fdf3a8...`
*-osx-unsigned.tar.gz	`0d8003ee2c879e13...`	`f25baf7518594848...`
*-osx64.tar.gz	`135d20c9d45b8193...`	`040f16800a82a5d4...`
*-powerpc64-linux-gnu-debug.tar.gz	`0f86fbb3da88d75b...`	`2504d9df701e4ed8...`
*-powerpc64-linux-gnu.tar.gz	`8fffb547da793622...`	`47013dc92358fe04...`
*-powerpc64le-linux-gnu-debug.tar.gz	`25bc9f40d7593a0e...`	`92c4a06880353518...`
*-powerpc64le-linux-gnu.tar.gz	`00ffc2de444f9f45...`	`c28babdbf84f7294...`
*-riscv64-linux-gnu-debug.tar.gz	`13259ba2232c7908...`	`64519ae5f24463ea...`
*-riscv64-linux-gnu.tar.gz	`277fa4286cd1acc3...`	`75af72fffd19bdc1...`
*-win-unsigned.tar.gz	`586f395e11db7d10...`	`19e50aa95b72b755...`
*-win64-debug.zip	`a47083334cbdcfa3...`	`167a36779732ef60...`
*-win64-setup-unsigned.exe	`926f391d3ea1f345...`	`f072bbf246b670d9...`
*-win64.zip	`cbfc8629325db892...`	`668522103b6534b3...`
*-x86_64-linux-gnu-debug.tar.gz	`ed183a137ae13cd4...`	`8a3b38b384974ef6...`
*-x86_64-linux-gnu.tar.gz	`f24558a571c69c68...`	`b2b95c889b0d7ff3...`
*.tar.gz	`86c28332470cb476...`	`e041c4234d1d666d...`
guix_build.log	`456643eedd9954da...`	`3eeaf71f58124147...`
guix_build.log.diff		`b47bce39a36e6c53...`

DrahtBot removed the label Needs Guix build on Mar 14, 2021

DrahtBot commented at 12:10 PM on March 22, 2021: member

Gitian builds

File	commit 63952f73b3041468fe3b25fa54858db7899273fa<br>(master)	commit c23ecbb9b18a302090179d32310d671d4ca5ac4a<br>(master and this pull)
*-aarch64-linux-gnu-debug.tar.gz	`5484237ee5c3fa99...`	`a8dd70a492556156...`
*-aarch64-linux-gnu.tar.gz	`2a05550165b38652...`	`f39e2cac79366108...`
*-arm-linux-gnueabihf-debug.tar.gz	`bfcf00e42246eae3...`	`ed1e635eff7baa16...`
*-arm-linux-gnueabihf.tar.gz	`49401c03ceb458b0...`	`cb4d08186e36ff0b...`
*-osx-unsigned.dmg	`bb46816571411f20...`	`7703a11c5bad3471...`
*-osx64.tar.gz	`87297fdb1bd1e468...`	`cca589ea2df2edd6...`
*-powerpc64-linux-gnu-debug.tar.gz	`ef0bc03e9dc3bc9b...`	`ff7bee2f90bde272...`
*-powerpc64-linux-gnu.tar.gz	`8b994ad2faf8d00f...`	`ab70f9e86a364747...`
*-powerpc64le-linux-gnu-debug.tar.gz	`a21bb50f2a1c6444...`	`c59bf89ae6af1303...`
*-powerpc64le-linux-gnu.tar.gz	`ac3bbefb3bb763cf...`	`5c603a9300e12d3b...`
*-riscv64-linux-gnu-debug.tar.gz	`d1127ea1c91d13d2...`	`7c09946bb7dbe454...`
*-riscv64-linux-gnu.tar.gz	`2a1d70a226aea599...`	`00fff4c9b50afa7a...`
*-win64-debug.zip	`49bb0d62d4d992b1...`	`e11836e2c4e16650...`
*-win64-setup-unsigned.exe	`2e31a6942609ae70...`	`5a2fb41e00980f97...`
*-win64.zip	`eea2a3d2ae87bd94...`	`e7123166429c6f18...`
*-x86_64-linux-gnu-debug.tar.gz	`c6ec5e1f93179f02...`	`902d81a6077a94ae...`
*-x86_64-linux-gnu.tar.gz	`9ec8794caea11319...`	`fc2d4e8b516dbb68...`
*.tar.gz	`debbf7a0bedc3e02...`	`bee0152a527955d1...`
bitcoin-core-linux-22-res.yml	`e8cbff85e5387c08...`	`7971905853963431...`
bitcoin-core-osx-22-res.yml	`d17361a4da1f9904...`	`0d504e85f83187bb...`
bitcoin-core-win-22-res.yml	`fbc466bbb012a14d...`	`30f112efcd4303c1...`
linux-build.log	`bda29ffb8f627075...`	`5bfddcf6a68432c1...`
osx-build.log	`4464f7ec164817d1...`	`311d15b522f011a2...`
win-build.log	`d947d26c42344b85...`	`6e785ed6217b2a54...`
bitcoin-core-linux-22-res.yml.diff		`c107065cace2deef...`
bitcoin-core-osx-22-res.yml.diff		`687812500fad3812...`
bitcoin-core-win-22-res.yml.diff		`c194e61558ccce7f...`
linux-build.log.diff		`685f872514a5dd19...`
osx-build.log.diff		`761cec8620a98a12...`
win-build.log.diff		`976c5e2e5e5018f5...`

DrahtBot removed the label Needs gitian build on Mar 22, 2021

hebasto commented at 5:19 PM on March 28, 2021: member

Guix builds:

$ find output -type f -name *$(git rev-parse --short HEAD)*.* -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
f75ddfc911ea50f5c2a77af45f3fb82a84457a3731c48d0f91657220185072ff  output/bitcoin-70ef58f466ae-aarch64-linux-gnu-debug.tar.gz
517a2acf63aaeb65549a9f09d854b7496d362016861e73203c936b2d4a59a18f  output/bitcoin-70ef58f466ae-aarch64-linux-gnu.tar.gz
d73ff950233e4e05356c2972a7d9d3868e67b6cdfd210982513ab7a4b448e267  output/bitcoin-70ef58f466ae-arm-linux-gnueabihf-debug.tar.gz
9545d86fac168b8450505d8aa8fdd34a141f377f2954bd89d6520effccbe7082  output/bitcoin-70ef58f466ae-arm-linux-gnueabihf.tar.gz
b3b1dd0040c9b26fad6c7099700bfbf10b39c9d5a1090a6bfbdfe31cb8baad6e  output/bitcoin-70ef58f466ae-osx-unsigned.dmg
95e5183081a979b4f935f4c99f48bedf8aa1fd942ac46e35d1b1d6b580f10d7a  output/bitcoin-70ef58f466ae-osx-unsigned.tar.gz
c3df5ecfbb8673894c020be1b59c628966ba34fc2e72580ad48c8b043e96caf3  output/bitcoin-70ef58f466ae-osx64.tar.gz
3f39adcfdc358ed6c07b71dc97b1e619298d2cede0359433396c5f76fcb18f7b  output/bitcoin-70ef58f466ae-powerpc64-linux-gnu-debug.tar.gz
0f49e635716e066f57fcf2128ca55371cc5d817b65d8cdf33694c20e8c5842ba  output/bitcoin-70ef58f466ae-powerpc64-linux-gnu.tar.gz
aec7a8d6212847125346e172677d28d77d9c88b4f5bfdd16cbff76c55a964622  output/bitcoin-70ef58f466ae-powerpc64le-linux-gnu-debug.tar.gz
1361824fce6525f81b41649d2d7c6d9dcdcf7047e988a5729021ac7a426a6fbc  output/bitcoin-70ef58f466ae-powerpc64le-linux-gnu.tar.gz
10b96de825bc35ca1d0cffa8a399c4c9647165580857a118dae46a8fca2c39ef  output/bitcoin-70ef58f466ae-riscv64-linux-gnu-debug.tar.gz
fdd4ab4014fd136f68dc4814c0e1e5c6f4f84ea80859f574f6aaf65807035ab8  output/bitcoin-70ef58f466ae-riscv64-linux-gnu.tar.gz
2e9d2a36656f8a43fedff9a09e60076996db4351065d2f62af810679509e7282  output/bitcoin-70ef58f466ae-win-unsigned.tar.gz
dd68149c6a261406a85e27b55721f73779e880301cb961cc91c68060db9b40e3  output/bitcoin-70ef58f466ae-win64-debug.zip
f2413090495ebb91d260999224db698c649ddd8d16912a8b7c75c988587b21fa  output/bitcoin-70ef58f466ae-win64-setup-unsigned.exe
89aae9f78371bef17f961d28c562f53ab43bf7830ce02e1fa7c65acaf9253069  output/bitcoin-70ef58f466ae-win64.zip
64c493e52746308300677d125b4685bbfc506bfa74b9063dd3b9f2f5af0371ee  output/bitcoin-70ef58f466ae-x86_64-linux-gnu-debug.tar.gz
67b50aeff37bd5cadf82106da3fcbdaba013814fb497d020092b45813249648a  output/bitcoin-70ef58f466ae-x86_64-linux-gnu.tar.gz
fd8714d2d7e13fd21ccbb5c8daf41410d7f37ac6a2fa502ec56922cfd09b6f4d  output/src/bitcoin-70ef58f466ae.tar.gz

sipa commented at 12:03 AM on March 29, 2021: member

$ sha256sum $(find -type f | fgrep 70ef58f466ae | LANG=C sort)
f75ddfc911ea50f5c2a77af45f3fb82a84457a3731c48d0f91657220185072ff  ./bitcoin-70ef58f466ae-aarch64-linux-gnu-debug.tar.gz
517a2acf63aaeb65549a9f09d854b7496d362016861e73203c936b2d4a59a18f  ./bitcoin-70ef58f466ae-aarch64-linux-gnu.tar.gz
d73ff950233e4e05356c2972a7d9d3868e67b6cdfd210982513ab7a4b448e267  ./bitcoin-70ef58f466ae-arm-linux-gnueabihf-debug.tar.gz
9545d86fac168b8450505d8aa8fdd34a141f377f2954bd89d6520effccbe7082  ./bitcoin-70ef58f466ae-arm-linux-gnueabihf.tar.gz
b3b1dd0040c9b26fad6c7099700bfbf10b39c9d5a1090a6bfbdfe31cb8baad6e  ./bitcoin-70ef58f466ae-osx-unsigned.dmg
95e5183081a979b4f935f4c99f48bedf8aa1fd942ac46e35d1b1d6b580f10d7a  ./bitcoin-70ef58f466ae-osx-unsigned.tar.gz
c3df5ecfbb8673894c020be1b59c628966ba34fc2e72580ad48c8b043e96caf3  ./bitcoin-70ef58f466ae-osx64.tar.gz
3f39adcfdc358ed6c07b71dc97b1e619298d2cede0359433396c5f76fcb18f7b  ./bitcoin-70ef58f466ae-powerpc64-linux-gnu-debug.tar.gz
0f49e635716e066f57fcf2128ca55371cc5d817b65d8cdf33694c20e8c5842ba  ./bitcoin-70ef58f466ae-powerpc64-linux-gnu.tar.gz
aec7a8d6212847125346e172677d28d77d9c88b4f5bfdd16cbff76c55a964622  ./bitcoin-70ef58f466ae-powerpc64le-linux-gnu-debug.tar.gz
1361824fce6525f81b41649d2d7c6d9dcdcf7047e988a5729021ac7a426a6fbc  ./bitcoin-70ef58f466ae-powerpc64le-linux-gnu.tar.gz
10b96de825bc35ca1d0cffa8a399c4c9647165580857a118dae46a8fca2c39ef  ./bitcoin-70ef58f466ae-riscv64-linux-gnu-debug.tar.gz
fdd4ab4014fd136f68dc4814c0e1e5c6f4f84ea80859f574f6aaf65807035ab8  ./bitcoin-70ef58f466ae-riscv64-linux-gnu.tar.gz
2e9d2a36656f8a43fedff9a09e60076996db4351065d2f62af810679509e7282  ./bitcoin-70ef58f466ae-win-unsigned.tar.gz
dd68149c6a261406a85e27b55721f73779e880301cb961cc91c68060db9b40e3  ./bitcoin-70ef58f466ae-win64-debug.zip
f2413090495ebb91d260999224db698c649ddd8d16912a8b7c75c988587b21fa  ./bitcoin-70ef58f466ae-win64-setup-unsigned.exe
89aae9f78371bef17f961d28c562f53ab43bf7830ce02e1fa7c65acaf9253069  ./bitcoin-70ef58f466ae-win64.zip
64c493e52746308300677d125b4685bbfc506bfa74b9063dd3b9f2f5af0371ee  ./bitcoin-70ef58f466ae-x86_64-linux-gnu-debug.tar.gz
67b50aeff37bd5cadf82106da3fcbdaba013814fb497d020092b45813249648a  ./bitcoin-70ef58f466ae-x86_64-linux-gnu.tar.gz
fd8714d2d7e13fd21ccbb5c8daf41410d7f37ac6a2fa502ec56922cfd09b6f4d  ./src/bitcoin-70ef58f466ae.tar.gz

fanquake commented at 5:01 AM on March 29, 2021: member

This doesn't quite work properly when building depends on Darwin:

/bin/sh: -c: line 0: unexpected EOF while looking for matching `''
/bin/sh: -c: line 1: syntax error: unexpected end of file
gmake: *** [Makefile:5: print-build_id_string] Error 2

ranlib and strip don't actually accept a --version argument:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib --version            
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: unknown option character `-' in: --version
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib [-sactfqLT] [-] archive [...]

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip --version
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip: unrecognized option: --version
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip [-AnuSXx] [-] [-d filename] [-s filename] [-R filename] [-o output] file [...]

and in the ranlilb case, it outputs a stray `....

If you remove both from the build_id_string construction, with this change we get:

build_id_string = salt Apple clang version 12.0.0 (clang-1200.0.32.29) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin Apple clang version 12.0.0 (clang-1200.0.32.29) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin  "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.15.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return -masm-verbose -munwind-tables -fcompatibility-qualified-id-block-type-checking -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 609.8 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -fdebug-compilation-dir /Users/michael/github/fanquake-bitcoin/depends -ferror-limit 19 -fmessage-length 0 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fobjc-runtime=macosx-10.15.0 -fmax-type-align=16 -fdiagnostics-show-option -o /dev/null -x c - clang -cc1 version 12.0.0 (clang-1200.0.32.29) default target x86_64-apple-darwin19.6.0 ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include" ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks" #include "..." search starts here: #include <...> search starts here:  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks (framework directory) End of search list. usage:  ar -d [-TLsv] archive file ... 	ar -m [-TLsv] archive file ... 	ar -m [-abiTLsv] position archive file ... 	ar -p [-TLsv] archive [file ...] 	ar -q [-cTLsv] archive file ... 	ar -r [-cuTLsv] archive file ... 	ar -r [-abciuTLsv] position archive file ... 	ar -t [-TLsv] archive [file ...] 	ar -x [-ouTLsv] archive [file ...]  ZERO_AR_DATE= Apple clang version 12.0.0 (clang-1200.0.32.29) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin Apple clang version 12.0.0 (clang-1200.0.32.29) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin  "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.15.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return -masm-verbose -munwind-tables -fcompatibility-qualified-id-block-type-checking -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 609.8 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -fdebug-compilation-dir /Users/michael/github/fanquake-bitcoin/depends -ferror-limit 19 -fmessage-length 0 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fobjc-runtime=macosx-10.15.0 -fmax-type-align=16 -fdiagnostics-show-option -o /dev/null -x c - clang -cc1 version 12.0.0 (clang-1200.0.32.29) default target x86_64-apple-darwin19.6.0 ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include" ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks" #include "..." search starts here: #include <...> search starts here:  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks (framework directory) End of search list. GUIX_ENVIRONMENT=

Note that ar doesn't really seem to accept --version either, but using it also doesn't throw an error.

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar --version
usage:  ar -d [-TLsv] archive file ...
	ar -m [-TLsv] archive file ...
	ar -m [-abiTLsv] position archive file ...
	ar -p [-TLsv] archive [file ...]
	ar -q [-cTLsv] archive file ...
	ar -r [-cuTLsv] archive file ...
	ar -r [-abciuTLsv] position archive file ...
	ar -t [-TLsv] archive [file ...]
	ar -x [-ouTLsv] archive [file ...]

dongcarl marked this as a draft on Mar 29, 2021

dongcarl commented at 11:51 PM on March 29, 2021: member

I'm switching this to draft because upon further inspection, the situation is more complicated than I thought. I want to thank fanquake's rigorous testing for inspiring me to check everything.

At first, I realized that the cause of the error message below:

/bin/sh: -c: line 0: unexpected EOF while looking for matching `''
/bin/sh: -c: line 1: syntax error: unexpected end of file
gmake: *** [Makefile:5: print-build_id_string] Error 2

Is because the print-% target did not escape its variables for printing on the shell, which is easily fixed by:

diff --git a/depends/Makefile b/depends/Makefile
index 7743e5d94..76156182f 100644
--- a/depends/Makefile
+++ b/depends/Makefile
@@ -1,8 +1,10 @@
 .NOTPARALLEL :

+escape = $(subst ','\'',$(1))
+
 # Pattern rule to print variables, e.g. make print-top_srcdir
 print-%:
- [@echo](/bitcoin-bitcoin/contributor/echo/) '$*' = '$($*)'
+ [@echo](/bitcoin-bitcoin/contributor/echo/) '$(call escape,$*)=$(call escape,$($*))'

 # When invoking a sub-make, keep only the command line variable definitions
 # matching the pattern in the filter function.

However, that led me to check all the other places where we use *_id_string in a $(shell)-like context, and I saw the following line: https://github.com/bitcoin/bitcoin/blob/1c7be9ab90af14d24f4668b02d9f07cec6f88a78/depends/funcs.mk#L52

According to a naive reading of this line, $(qt_build_id_long) should be suffixed by $(x86_64_darwin_id_string) (note that $($(1)_type) is x86_64_darwin). However, reality is strange:

$ gmake -C depends print-qt_build_id_long
gmake: Entering directory '/Users/dongcarl/src/bitcoin/depends'
qt_build_id_long=qt-5.9.8-f5d007fddab07fc3ede3ebf61125a551fb1cea5d43eb493878b4e693d9b6aa37-release zlib-1.2.11-22f2d8df2731c0cc7f40f462c2a27a6d87e34baf177a9cddb3f26d44bc5bbbc4 salt Apple clang version 12.0.0 (clang-1200.0.32.28) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin clang: warning: argument unused during compilation: '-mmacosx-version-min=10.14' [-Wunused-command-line-argument] Apple clang version 12.0.0 (clang-1200.0.32.28) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin  "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.14.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return -masm-verbose -munwind-tables -fcompatibility-qualified-id-block-type-checking -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 609.8 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -fdebug-compilation-dir /Users/dongcarl/src/bitcoin/depends -ferror-limit 19 -fmessage-length 0 -stack-protector 1 -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fobjc-runtime=macosx-10.14.0 -fmax-type-align=16 -fdiagnostics-show-option -o /dev/null -x c - clang -cc1 version 12.0.0 (clang-1200.0.32.28) default target x86_64-apple-darwin19.6.0 ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include" ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks"
gmake: Leaving directory '/Users/dongcarl/src/bitcoin/depends'

$ gmake -C depends print-x86_64_darwin_id_string
gmake: Entering directory '/Users/dongcarl/src/bitcoin/depends'
x86_64_darwin_id_string=salt Apple clang version 12.0.0 (clang-1200.0.32.28) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin clang: warning: argument unused during compilation: '-mmacosx-version-min=10.14' [-Wunused-command-line-argument] Apple clang version 12.0.0 (clang-1200.0.32.28) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin  "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.14.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return -masm-verbose -munwind-tables -fcompatibility-qualified-id-block-type-checking -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 609.8 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -fdebug-compilation-dir /Users/dongcarl/src/bitcoin/depends -ferror-limit 19 -fmessage-length 0 -stack-protector 1 -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fobjc-runtime=macosx-10.14.0 -fmax-type-align=16 -fdiagnostics-show-option -o /dev/null -x c - clang -cc1 version 12.0.0 (clang-1200.0.32.28) default target x86_64-apple-darwin19.6.0 ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include" ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks" #include "..." search starts here: #include <...> search starts here:  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks (framework directory) End of search list. usage:  ar -d [-TLsv] archive file ...      ar -m [-TLsv] archive file ...  ar -m [-abiTLsv] position archive file ...      ar -p [-TLsv] archive [file ...]    ar -q [-cTLsv] archive file ...         ar -r [-cuTLsv] archive file ...        ar -r [-abciuTLsv] position archive file ...    ar -t [-TLsv] archive [file ...]        ar -x [-ouTLsv] archive [file ...]  ZERO_AR_DATE= Apple clang version 12.0.0 (clang-1200.0.32.28) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin clang: warning: argument unused during compilation: '-mmacosx-version-min=10.14' [-Wunused-command-line-argument] clang: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument] Apple clang version 12.0.0 (clang-1200.0.32.28) Target: x86_64-apple-darwin19.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin  "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.14.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return -masm-verbose -munwind-tables -fcompatibility-qualified-id-block-type-checking -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 609.8 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -fdebug-compilation-dir /Users/dongcarl/src/bitcoin/depends -ferror-limit 19 -fmessage-length 0 -stack-protector 1 -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fobjc-runtime=macosx-10.14.0 -fmax-type-align=16 -fdiagnostics-show-option -o /dev/null -x c - clang -cc1 version 12.0.0 (clang-1200.0.32.28) default target x86_64-apple-darwin19.6.0 ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include" ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks" #include "..." search starts here: #include <...> search starts here:  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include  /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks (framework directory) End of search list. error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: unknown option character `-' in: --version Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib [-sactfqLT] [-] archive [...]  error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip: unrecognized option: --version Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip [-AnuSXx] [-] [-d filename] [-s filename] [-R filename] [-o output] file [...]   GUIX_ENVIRONMENT=
gmake: Leaving directory '/Users/dongcarl/src/bitcoin/depends'

As you can observe, $(qt_build_id_long) is not suffixed by $(x86_64_darwin_id_string), and is in fact even shorter than $(x86_64_darwin_id_string). If you examine $(qt_build_id_long) carefully, you'll find that it actually includes a portion of $(x86_64_darwin_id_string), all the way up to the first #.

That's the key to this puzzle: $(qt_build_id_long) was defined in an eval, so any # in its RHS expansion will be interperted as the comment symbol rather than a literal #.

I'm not sure what the best solution to this is, but I think it's a very good indication that Make is not designed to having arbitrary strings in its variables.

dongcarl force-pushed on Apr 8, 2021

dongcarl commented at 5:34 PM on April 8, 2021: member

I believe the problem described in #21427 (comment) is now fixed. Thanks to theuni for his help!

dongcarl marked this as ready for review on Apr 8, 2021

dongcarl force-pushed on Apr 8, 2021

depends: Fix id_string invocations

We now use a script named gen_id to generate the base build_id/host_id.
This solves 2 problems:

1. GNU Make special-casing exit code 127 (command not found) meant that
   warnings about missing tools would propagate to the user's terminal
   and broke our opportunistic build_id construction.
2. This change ensures that we don't have arbitrary characters in our
   make variables that would be misinterpreted by Make.

See comments in depends/Makefile and depends/gen_id for more
information.

fa872c9af3

dongcarl force-pushed on Apr 9, 2021

dongcarl commented at 12:19 AM on April 9, 2021: member

Made the gen_id script recognize the DEBUG environment variable and output its hash preimage to stderr so that we can debug weird caching issues easier in the future

jarolrod commented at 8:17 AM on April 14, 2021: member

contributing GUIX hashes for fa872c9af397837bb17859b5f43adec71239682a:

find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum

c42b06d1ce34d2f606bd8f7acecdc1fbcbd3ede094ee3006ef05025074dedea8  guix-build-fa872c9af397/output/aarch64-linux-gnu/bitcoin-fa872c9af397-aarch64-linux-gnu-debug.tar.gz
29922bcafa59dd2b27090a7e58fc5e3d45ee1bf120f831b3d970bc431a50282b  guix-build-fa872c9af397/output/aarch64-linux-gnu/bitcoin-fa872c9af397-aarch64-linux-gnu.tar.gz
64f197993919a222135eaaf2be6fa5a3c5d4b8587b4240d5319ed47321cb1249  guix-build-fa872c9af397/output/arm-linux-gnueabihf/bitcoin-fa872c9af397-arm-linux-gnueabihf-debug.tar.gz
1d2aa72db92ba9b920365eab4d84162678153ae924d1b17a4f4d4fc8106711ad  guix-build-fa872c9af397/output/arm-linux-gnueabihf/bitcoin-fa872c9af397-arm-linux-gnueabihf.tar.gz
4d3419774c0349fe432aebcf68bd78413a7928e4d11f1794f98602b5807cb96a  guix-build-fa872c9af397/output/dist-archive/bitcoin-fa872c9af397.tar.gz
2c60bf42ef6e36e63ce0dbca7787a675ec788801cefb1d97aa349f9aecd82ea5  guix-build-fa872c9af397/output/powerpc64-linux-gnu/bitcoin-fa872c9af397-powerpc64-linux-gnu-debug.tar.gz
bb73a2d91dde1970f103cb42612987ae4a9405f7df5b61d2e35816f253895ac4  guix-build-fa872c9af397/output/powerpc64-linux-gnu/bitcoin-fa872c9af397-powerpc64-linux-gnu.tar.gz
7c0b6d046501141170349592bd773193fa1f3718169379e08604728ae149b55c  guix-build-fa872c9af397/output/powerpc64le-linux-gnu/bitcoin-fa872c9af397-powerpc64le-linux-gnu-debug.tar.gz
4689de97324f515d56e31789705e6efbd783cae41a6a3f55136838f0298db006  guix-build-fa872c9af397/output/powerpc64le-linux-gnu/bitcoin-fa872c9af397-powerpc64le-linux-gnu.tar.gz
e40d5547591c64fa8ec7a5ff9521e4d4b3bc3b72b7a6c65ad12b0d01af19d64d  guix-build-fa872c9af397/output/riscv64-linux-gnu/bitcoin-fa872c9af397-riscv64-linux-gnu-debug.tar.gz
b882a86ffc419528b3a9ae4283ebedb343ebfb14c2264aa2704c60c46b7e032c  guix-build-fa872c9af397/output/riscv64-linux-gnu/bitcoin-fa872c9af397-riscv64-linux-gnu.tar.gz
0e398bd14446ce8e4df508de50e92621919e7ef30e7b02040f6c3b6db69d22fa  guix-build-fa872c9af397/output/x86_64-apple-darwin18/bitcoin-fa872c9af397-osx-unsigned.dmg
bdf1ab2b9f7dc189eab6ffde3f08ae13a1298eeffb76a97593c5384a3686c8df  guix-build-fa872c9af397/output/x86_64-apple-darwin18/bitcoin-fa872c9af397-osx-unsigned.tar.gz
e932b1e8a8df05afc72ba92acd6599b464a73a8b0b4ee94f9e1b9604b71046e0  guix-build-fa872c9af397/output/x86_64-apple-darwin18/bitcoin-fa872c9af397-osx64.tar.gz
191ee9a1fb4f3fe6742b34cf112b6934126e70e8bd4dac736ec22b9611cbfa1b  guix-build-fa872c9af397/output/x86_64-linux-gnu/bitcoin-fa872c9af397-x86_64-linux-gnu-debug.tar.gz
7159eecc9ee56534c90f269907f39f2fb0bbc0e35eca074de4d35553e335cb0a  guix-build-fa872c9af397/output/x86_64-linux-gnu/bitcoin-fa872c9af397-x86_64-linux-gnu.tar.gz
e6fe7306191269d6fb9603e15c9fe192520806ea0fc48117f3f6cdf96c9bc155  guix-build-fa872c9af397/output/x86_64-w64-mingw32/bitcoin-fa872c9af397-win-unsigned.tar.gz
88a75379b0dad6a1e49815dfe2cbde73cbd6ab155a47f64477566f21f4b20f73  guix-build-fa872c9af397/output/x86_64-w64-mingw32/bitcoin-fa872c9af397-win64-debug.zip
7cf3fc8317b25bb94ddd1a2a9f12d2470402d472697e37f2d3af37dd3dc9c257  guix-build-fa872c9af397/output/x86_64-w64-mingw32/bitcoin-fa872c9af397-win64-setup-unsigned.exe
476d4c777ea9a3980d86fed34520a39411a752d984e43674b448c3ad831dd43f  guix-build-fa872c9af397/output/x86_64-w64-mingw32/bitcoin-fa872c9af397-win64.zip

laanwj commented at 12:18 PM on May 5, 2021: member

Concept and light code review ACK fa872c9af397837bb17859b5f43adec71239682a

Also checked that the messages mentioned in #21242 are gone now.

I see no downsides to this approach. Factoring the generation of the ID string out to an external script makes it easier to understand what is happening, too.

laanwj merged this on May 5, 2021

laanwj closed this on May 5, 2021

sidhujag referenced this in commit 005744c270 on May 5, 2021

gwillen referenced this in commit 37ae37747f on Jun 1, 2022

DrahtBot locked this on Aug 18, 2022