build: Automatically include both `git`-tracked and bootstrapped files.

dongcarl commented at 0:05 am on March 31, 2020: member

 0This allows us to:
 1
 21. Retain the automatic inclusion of bootstrapped files inside dist
 3   source tarballs
 42. Also automatically include all git-tracked files in EXTRA_DIST
 5
 6Notes:
 7
 81. We already rely on git for `make dist` anyway (see dist-hook).
 92. As of automake:77d39959511295f5a30332d5d03f0a6956bd9460 (this is just
10   the latest master I have so there's a point of reference for the
11   future), we can observe the behaviour of `make dist` in
12   `lib/am/distdir.am`. Search for the `distdir-am` target within that
13   file. You will see that if a file was already added to the distdir,
14   it won't be added again.

I also performed a gitian build, here’s a diff of the source tarball and one produced with simply git archive, as expected, the difference is only in the bootstrapped files.

 0$ diff -rl gitian git
 1Only in gitian: aclocal.m4
 2Only in gitian/build-aux: compile
 3Only in gitian/build-aux: config.guess
 4Only in gitian/build-aux: config.sub
 5Only in gitian/build-aux: depcomp
 6Only in gitian/build-aux: install-sh
 7Only in gitian/build-aux: ltmain.sh
 8Only in gitian/build-aux/m4: libtool.m4
 9Only in gitian/build-aux/m4: lt~obsolete.m4
10Only in gitian/build-aux/m4: ltoptions.m4
11Only in gitian/build-aux/m4: ltsugar.m4
12Only in gitian/build-aux/m4: ltversion.m4
13Only in gitian/build-aux: missing
14Only in gitian/build-aux: test-driver
15Only in gitian: configure
16Only in gitian/doc/man: Makefile.in
17Only in gitian: Makefile.in
18Only in gitian/src/config: bitcoin-config.h.in
19Only in gitian/src: Makefile.in
20Only in gitian/src/secp256k1: aclocal.m4
21Only in gitian/src/secp256k1/build-aux: compile
22Only in gitian/src/secp256k1/build-aux: config.guess
23Only in gitian/src/secp256k1/build-aux: config.sub
24Only in gitian/src/secp256k1/build-aux: depcomp
25Only in gitian/src/secp256k1/build-aux: install-sh
26Only in gitian/src/secp256k1/build-aux: ltmain.sh
27Only in gitian/src/secp256k1/build-aux/m4: libtool.m4
28Only in gitian/src/secp256k1/build-aux/m4: lt~obsolete.m4
29Only in gitian/src/secp256k1/build-aux/m4: ltoptions.m4
30Only in gitian/src/secp256k1/build-aux/m4: ltsugar.m4
31Only in gitian/src/secp256k1/build-aux/m4: ltversion.m4
32Only in gitian/src/secp256k1/build-aux: missing
33Only in gitian/src/secp256k1/build-aux: test-driver
34Only in gitian/src/secp256k1: configure
35Only in gitian/src/secp256k1: Makefile.in
36Only in gitian/src/secp256k1/src: libsecp256k1-config.h.in
37Only in gitian/src/univalue: aclocal.m4
38Only in gitian/src/univalue/build-aux: compile
39Only in gitian/src/univalue/build-aux: config.guess
40Only in gitian/src/univalue/build-aux: config.sub
41Only in gitian/src/univalue/build-aux: depcomp
42Only in gitian/src/univalue/build-aux: install-sh
43Only in gitian/src/univalue/build-aux: ltmain.sh
44Only in gitian/src/univalue/build-aux/m4: libtool.m4
45Only in gitian/src/univalue/build-aux/m4: lt~obsolete.m4
46Only in gitian/src/univalue/build-aux/m4: ltoptions.m4
47Only in gitian/src/univalue/build-aux/m4: ltsugar.m4
48Only in gitian/src/univalue/build-aux/m4: ltversion.m4
49Only in gitian/src/univalue/build-aux: missing
50Only in gitian/src/univalue/build-aux: test-driver
51Only in gitian/src/univalue: configure
52Only in gitian/src/univalue: Makefile.in
53Only in gitian/src/univalue: univalue-config.h.in

dongcarl added the label Needs gitian build on Mar 31, 2020

fanquake added the label Build system on Mar 31, 2020

DrahtBot commented at 8:19 am on March 31, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#18349 (build: Fix quick hack for version string in releases by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

hebasto commented at 9:04 am on March 31, 2020: member

Meta question: could we described the “users” of bootstrapped files? What systems they are use? What are their workflows? etc.

I suppose a user of the Bitcoin Core, who is willing to build his binaries from sources, runs a non-unusual Linux distro, and is able to start build with ./autogen.sh. If we assume that non-unusual system has pkg-config installed, why we couldn’t make the same assumption about autotools itself?

dongcarl commented at 5:14 pm on March 31, 2020: member

In my mind, including bootstrapped files (and effectively pinning the autotools version) is better than not including them. Why would we not also include bootstrapped files if it’s easy to do?

0fc

MarcoFalke commented at 5:23 pm on March 31, 2020: member

Concept ACK. Travis out-of-tree builds are failing.

dongcarl force-pushed on Mar 31, 2020

build: Drop workaround for version string cbf2418925

automake: Include all tracked files as EXTRA_DIST

This allows us to:

1. Retain the automatic inclusion of bootstrapped files inside dist
   source tarballs
2. Also automatically include all git-tracked files in EXTRA_DIST

Notes:

1. We already rely on git for `make dist` anyway (see dist-hook).
2. As of automake:77d39959511295f5a30332d5d03f0a6956bd9460 (this is just
   the latest master I have so there's a point of reference for the
   future), we can observe the behaviour of `make dist` in
   `lib/am/distdir.am`. Search for the `distdir-am` target within that
   file. You will see that if a file was already added to the distdir,
   it won't be added again.

dcee758eec

gitian: Restore usage of `make dist` source tarballs

Some of this file ordering may not be necessary anymore, we can
investigate further and come back to this.

1b91adeea8

dongcarl force-pushed on Mar 31, 2020

ryanofsky commented at 7:00 pm on March 31, 2020: member

Why would we not also include bootstrapped files if it’s easy to do?

I don’t care about this very much, but will answer the question.

Advantages of including bootstrapped files:

Person building from source no longer has to install autotools and m4, just a c++ compiler, linker, and gnu make.

Disadvantages of including bootstrapped files:

Adds to differences between tarball distribution and git checkout. Different instructions for building one vs the other. Different failures that can occur. Possible lack of testing with normal developers running autotools themselves and only less experienced people using the generated scripts
Bigger tarball
Noisier diffs. Without generated files, someone could reasonably diff two point releases and get a sense of what changed. With generated files, such as the 35,000 line 1.1M top-level configure script, actual changes are easily drowned in sea of noise
Aesthetics, simplicity. This is subjective, but personally I like simplicity of source distributions that stick to delivering actual sources and don’t mix up sources and intermediate files without clear reason for doing so. Is there a clear reason why it’s good to include these intermediate files, and but not include intermediate files like precompiled headers or test datadirs, or maybe even object files or binaries? It’s nice to not think about these things and just stick to sources

DrahtBot commented at 7:04 pm on March 31, 2020: member

Gitian builds

File	commit 965c0c37d54c727bee50920fdfa62f8174a94280(master)	commit 81df43f232237a11c9cb4bf9b4c0418066313d2f(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`57aab7e587f3ea0a...`	`2d249de922ed2266...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`5bb0f02f1749f913...`	`19e71105df4d217a...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`3b670b4a99116f5f...`	`5822a29e3bf6f70e...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`aad52c43bf0d86ab...`	`188e7f77118b4c3f...`
bitcoin-0.19.99-osx-unsigned.dmg	`938cc8fcbcda261c...`	`e8465963adc6bf1c...`
bitcoin-0.19.99-osx64.tar.gz	`e5cd3dd49d87787c...`	`bedec5880b84464b...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`c5eadbd98b89a453...`	`11f1e2a05a9d30d3...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`585e4150e48a03f7...`	`8b92f8fb7d723ad0...`
bitcoin-0.19.99-win64-debug.zip	`46a08a338aa679ce...`	`f2fe736893f394b0...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`4e794768b43a81bc...`	`f6b50d45230fde1d...`
bitcoin-0.19.99-win64.zip	`a15127e0ed4fd3d6...`	`8b8509f17e657c49...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`7994d9e40a424497...`	`32be44e4b180200c...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`c3b2d37b3df511c4...`	`9b779e630f165842...`
bitcoin-0.19.99.tar.gz	`a39aee426533d6c0...`	`831780d143ac839d...`
bitcoin-core-linux-0.20-res.yml	`7cf55f32bec5f8be...`	`6226c038cdae9425...`
bitcoin-core-osx-0.20-res.yml	`6b594bf6aad6c167...`	`2ce1f60873083d02...`
bitcoin-core-win-0.20-res.yml	`e6738ec03bc3e6b6...`	`28dda787c7ef5167...`
linux-build.log	`057410ad898e35c5...`	`e3b323907c11b6d3...`
osx-build.log	`156f4048415dd68c...`	`3e35a12d69c14ea5...`
win-build.log	`4c690476ddd8905d...`	`1277bce228b24178...`
bitcoin-core-linux-0.20-res.yml.diff		`e4e2c39386cf2b77...`
bitcoin-core-osx-0.20-res.yml.diff		`ed8bf1b2681b5f83...`
bitcoin-core-win-0.20-res.yml.diff		`89e776f97eefcc71...`
linux-build.log.diff		`203043a1de64b2d4...`
osx-build.log.diff		`cfe1ff3dd568248f...`
win-build.log.diff		`2c4011876e5c05d3...`

DrahtBot removed the label Needs gitian build on Mar 31, 2020

laanwj commented at 11:35 am on April 1, 2020: member

I personally prefer using git archive here. It’s nice and simple and straightforward.

Calling git ls-files in the makefile is imo a bit strange, given tarballs etc don’t build from git.

dongcarl commented at 5:30 pm on April 1, 2020: member

Noted, closing.

Followups:

We might need the no-dist automake option, as I don’t think make dist makes sense anymore
We should clean up dist related targets in Makefile.am

dongcarl closed this on Apr 1, 2020

luke-jr commented at 7:52 pm on April 1, 2020: member

make dist should be used by gitian.

git archive would omit important files (like configure!).

DrahtBot locked this on Feb 15, 2022

build: Automatically include both git-tracked and bootstrapped files. #18478

Conflicts

Gitian builds

build: Automatically include both `git`-tracked and bootstrapped files. #18478