build: Automatically include both `git`-tracked and bootstrapped files. #18478

This allows us to:

1. Retain the automatic inclusion of bootstrapped files inside dist
   source tarballs
2. Also automatically include all git-tracked files in EXTRA_DIST

Notes:

1. We already rely on git for `make dist` anyway (see dist-hook).
2. As of automake:77d39959511295f5a30332d5d03f0a6956bd9460 (this is just
   the latest master I have so there's a point of reference for the
   future), we can observe the behaviour of `make dist` in
   `lib/am/distdir.am`. Search for the `distdir-am` target within that
   file. You will see that if a file was already added to the distdir,
   it won't be added again.

I also performed a gitian build, here's a diff of the source tarball and one produced with simply git archive, as expected, the difference is only in the bootstrapped files.

$ diff -rl gitian git
Only in gitian: aclocal.m4
Only in gitian/build-aux: compile
Only in gitian/build-aux: config.guess
Only in gitian/build-aux: config.sub
Only in gitian/build-aux: depcomp
Only in gitian/build-aux: install-sh
Only in gitian/build-aux: ltmain.sh
Only in gitian/build-aux/m4: libtool.m4
Only in gitian/build-aux/m4: lt~obsolete.m4
Only in gitian/build-aux/m4: ltoptions.m4
Only in gitian/build-aux/m4: ltsugar.m4
Only in gitian/build-aux/m4: ltversion.m4
Only in gitian/build-aux: missing
Only in gitian/build-aux: test-driver
Only in gitian: configure
Only in gitian/doc/man: Makefile.in
Only in gitian: Makefile.in
Only in gitian/src/config: bitcoin-config.h.in
Only in gitian/src: Makefile.in
Only in gitian/src/secp256k1: aclocal.m4
Only in gitian/src/secp256k1/build-aux: compile
Only in gitian/src/secp256k1/build-aux: config.guess
Only in gitian/src/secp256k1/build-aux: config.sub
Only in gitian/src/secp256k1/build-aux: depcomp
Only in gitian/src/secp256k1/build-aux: install-sh
Only in gitian/src/secp256k1/build-aux: ltmain.sh
Only in gitian/src/secp256k1/build-aux/m4: libtool.m4
Only in gitian/src/secp256k1/build-aux/m4: lt~obsolete.m4
Only in gitian/src/secp256k1/build-aux/m4: ltoptions.m4
Only in gitian/src/secp256k1/build-aux/m4: ltsugar.m4
Only in gitian/src/secp256k1/build-aux/m4: ltversion.m4
Only in gitian/src/secp256k1/build-aux: missing
Only in gitian/src/secp256k1/build-aux: test-driver
Only in gitian/src/secp256k1: configure
Only in gitian/src/secp256k1: Makefile.in
Only in gitian/src/secp256k1/src: libsecp256k1-config.h.in
Only in gitian/src/univalue: aclocal.m4
Only in gitian/src/univalue/build-aux: compile
Only in gitian/src/univalue/build-aux: config.guess
Only in gitian/src/univalue/build-aux: config.sub
Only in gitian/src/univalue/build-aux: depcomp
Only in gitian/src/univalue/build-aux: install-sh
Only in gitian/src/univalue/build-aux: ltmain.sh
Only in gitian/src/univalue/build-aux/m4: libtool.m4
Only in gitian/src/univalue/build-aux/m4: lt~obsolete.m4
Only in gitian/src/univalue/build-aux/m4: ltoptions.m4
Only in gitian/src/univalue/build-aux/m4: ltsugar.m4
Only in gitian/src/univalue/build-aux/m4: ltversion.m4
Only in gitian/src/univalue/build-aux: missing
Only in gitian/src/univalue/build-aux: test-driver
Only in gitian/src/univalue: configure
Only in gitian/src/univalue: Makefile.in
Only in gitian/src/univalue: univalue-config.h.in

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #18349 (build: Fix quick hack for version string in releases by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Meta question: could we described the "users" of bootstrapped files? What systems they are use? What are their workflows? etc.

I suppose a user of the Bitcoin Core, who is willing to build his binaries from sources, runs a non-unusual Linux distro, and is able to start build with ./autogen.sh. If we assume that non-unusual system has pkg-config installed, why we couldn't make the same assumption about autotools itself?

In my mind, including bootstrapped files (and effectively pinning the autotools version) is better than not including them. Why would we not also include bootstrapped files if it's easy to do?

Concept ACK. Travis out-of-tree builds are failing.

Why would we not also include bootstrapped files if it's easy to do?

I don't care about this very much, but will answer the question.

Advantages of including bootstrapped files:

• Person building from source no longer has to install autotools and m4, just a c++ compiler, linker, and gnu make.

Disadvantages of including bootstrapped files:

• Adds to differences between tarball distribution and git checkout. Different instructions for building one vs the other. Different failures that can occur. Possible lack of testing with normal developers running autotools themselves and only less experienced people using the generated scripts

• Bigger tarball

• Noisier diffs. Without generated files, someone could reasonably diff two point releases and get a sense of what changed. With generated files, such as the 35,000 line 1.1M top-level configure script, actual changes are easily drowned in sea of noise

• Aesthetics, simplicity. This is subjective, but personally I like simplicity of source distributions that stick to delivering actual sources and don't mix up sources and intermediate files without clear reason for doing so. Is there a clear reason why it's good to include these intermediate files, and but not include intermediate files like precompiled headers or test datadirs, or maybe even object files or binaries? It's nice to not think about these things and just stick to sources

<!--a722867cd34abeea1fadc8d60700f111-->

Gitian builds

I personally prefer using git archive here. It's nice and simple and straightforward.

Calling git ls-files in the makefile is imo a bit strange, given tarballs etc don't build from git.

Noted, closing.

Followups:

• We might need the no-dist automake option, as I don't think make dist makes sense anymore
• We should clean up dist related targets in Makefile.am

make dist should be used by gitian.

git archive would omit important files (like configure!).