Fix release tarball generated by gitian #18818

Without losing the new and improved git-archive method of ensuring we get the entire source included, this fixes two one remaining regressions to standard source code distribution expectations:

1. The tarball should be entirely within a single directory (not extract directly to the working directory). (merged via #20318)
2. The tarball should be pre-autogen'd, so the user can begin with ./configure and not need maintainer tools.

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #21036 (gitian: Bump descriptors to Focal for 22.0 by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

<!--a722867cd34abeea1fadc8d60700f111-->

Gitian builds

cc @dongcarl @fanquake

Does this break guix builds in any way?

• #18741 (guix: Make source tarball using git-archive by dongcarl)

<!--9cd9c72976c961c55c7acef8f6ba82cd-->

Guix builds

I think we previously decided specifically that we're not going to pre-autogen our tarballs.

See context:

• #18331
• #18478
• #18547

100% on board with restoring the tar leading directory though.

This doesn't have the complexity of git ls-files etc in #18478. I see no decision-making relating to intentionally excluding configure (which is a standard/expected part of source code distributions), only a few people being okay with losing it as a side effect.

Okay, at this point, I don't have an opinion either way anymore. Will give it a review after others chime in!

Why not follow @MarcoFalke's suggestion on IRC:

<MarcoFalke> luke-jr: Maybe split out the first commit into a pull, so that gitian builds can be performed on it more easily?

?

http://www.erisian.com.au/bitcoin-core-dev/log-2020-04-30.html#l-380

Rebased

So if this is backported, the only additional commit that also needs backport is 395c1137f630dc495ffb2752a23bc1dfd470ee53, I think (haven't checked).

For the record, my thinking in #18478 originally was this:

• make dist seems to be designed to know exactly which bootstrapped files to include (or not include) in its archive
• git archive or git ls-files seems to be good at knowing all the other files we want to include in its archive

Which means that if we combine these two, we'll get the best of both worlds, without having to list anything. This would also be immune to future changes in how bootstrapped files are named, and additional "bootstrapped but shouldn't be included" files like .deps or autom4te.cache.

Does this mean we'll have to keep listing files for make dist?

No it doesn't, because we only care that make dist produces an archive that includes the bootstrapped files, it can miss all the files in share, contrib, etc. and that would be fine, as we'll just either:

1. Add them in via a git ls-files in EXTRA_DIST like in #18478, or
2. Append the git archive archive

Both of which will include all the tracked files

@dongcarl Your (1) would miss the substitutions git-archive makes in src/clientversion.cpp. I don't see a problem with (2), although it's slightly more complex and requires care I don't know if we want to spend time reviewing (in particular, tar is picky about specific tar file formats when concatenating, and even if we just use --update, there's a risk that the git archive timestamp is after the gitian reference timestamp, which might make problems)

The solution I implemented here aims to keep it simple and obviously correct.

@luke-jr

2. The tarball should be pre-autogen'd, so the user can begin with ./configure and not need maintainer tools.

I have no experience as a package maintainer so I kindly ask you to share the real-world cases when a user cannot begin with ./autogen.sh. It seems having maintainer tools is not a much burden for a user, no?

I have no experience as a package maintainer so I kindly ask you to share the real-world cases when a user cannot begin with ./autogen.sh

It used to be common to want to distribute source packages for unix systems that might not have gnu tools installed (solaris, irix, hpux, bsds) but would have a bare-bones make, c compiler, and posix shell.

To easily support building packages on these systems, the packages would not just include human readable and writable source code, but would also come with big generated configure, libtool, etc, scripts generated by the autotools pipeline and m4.

Today since we already require gnu make, a c++11 compiler, boost, libevent, berkeley db, bash, and python for many practical use cases of the source tarball, it does not seem like it is asking to much to require autotools as well. I think the main benefits of dropping the autotools generated files are having fewer differences between git checkout builds and tarball builds, not having unfathomably long megabyte shell scripts mixed in with real source code, and having simpler diffs between releases. This opinion might be a personal heresy, though. I wouldn't want to deprive anyone of 30,000 line barely readable shell scripts if it's what people are clamoring for.

I don't know of any case where the user can't install autotools, but this isn't about "can't"...

Including the configure script, and starting with just ./configure is standard, and what people expect both in general and even specifically for Bitcoin Core. There are no real benefits to suddenly not including it, and doing so this way does not add much complexity.

Furthermore, I don't see any way to actually fix #18349 deterministically without some preprocessing as in #18902, so it's not like we could simply use git-archive alone in any case.

Including the configure script, and starting with just ./configure is standard, and what people expect both in general and even specifically for Bitcoin Core. There are no real benefits to suddenly not including it, and doing so this way does not add much complexity.

From UNIX BUILD NOTES -- To Build:

./autogen.sh ./configure make make install # optional

Docs seem to prevail over standard :)

just ./configure is standard

Not an actual standard, just a common practice that's become less common because of new build systems and languages and git, and because it doesn't have logical or practical benefits

Nothing here stops people from using autogen, as documented.

Again, there is no benefit to removing configure and breaking the de facto standard practice.

Again, there is no benefit to removing configure

It's fine to disagree and I'm not interested in delaying this PR if it contains an essential fix and other desired changes. I just don't want to to be misrepresented. So from #18818 (comment), "I think the main benefits of dropping the autotools generated files are having fewer differences between git checkout builds and tarball builds, not having unfathomably long megabyte shell scripts mixed in with real source code, and having simpler diffs between releases."

tACK 3897f3a2ec077f2be8f0d08f28e79165a6cae6d4

Concept ACK on maintaining the behaviour from earlier releases, if it's not too much trouble. This is even an improvement, because it adds depends to the source tarball.

I never used the source tarball, and can only imagine needing it if Github disappeared and there's no reliable place to git clone from.

On the one hand I share @ryanofsky's preference for keeping the tarball small and having the user run ./autogen.sh themselves. On the other hand it's nice to have as few dependencies on top of the vanilla OS as possible, since everything in the tar file was created in a reproducible way.

Suggestions for followup:

• document installation steps with minimal packages

Current process

I created a fresh Ubuntu 20.04 (virtual) machine and downloaded the v0.19.1 release source code, which extracts itself into bitcoin-0.19.1.

There's no documentation that points this out, but I can install fewer packages than docs/build-unix suggest, i.e. skip libtool autotools-dev automake. I can also skip the ./autogen step:

apt install build-essential pkg-config libdb-dev libdb++-dev libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev libssl-dev libevent-dev
./configure --with-incompatible-bdb

(this is without the GUI, also libssl-dev is not needed anymore for v0.20)

This PR

First I made a Gitian Linux build for this PR, using my regular Gitian machine:

cd bitcoin
git remote add luke-jr https://github.com/luke-jr/bitcoin.git
cd ..
./gitian-build-0.20.py -c -u https://github.com/luke-jr/bitcoin -o l -b -j 10 -m 20000 -d -D -n sjors fix_gitian_src_202004

(gitian-build-0.20.py is a copy of bitcoin/contrib/gitian-build.py)

5a1d50cb6ae06ac2f38f7a2882886a4286b388b9e2c1a09dda1f41921170476f  src/bitcoin-3897f3a2ec07.tar.gz
b5626bc1b1b59379e657974ef1a6994aed49f1814f5c0d74cc022b762d18f699  bitcoin-core-linux-0.21-res.yml

I then downloaded the source package to my VM. This extracts into bitcoin-3897f3a2ec07.

I removed all apt packages except build-essential pkg-config, installed automake libtool autotools-dev, and built with depends.

Including the configure script, and starting with just ./configure is standard, and what people expect both in general and even specifically for Bitcoin Core. There are no real benefits to suddenly not including it, and doing so this way does not add much complexity.

From UNIX BUILD NOTES -- To Build:

./autogen.sh ./configure make make install # optional

Docs seem to prevail over standard :)

This is standard procedure for the project's developers. Please don't conflate that with end-user procedure. We bootstrap because distro's and end-user-code-builders (read: not us) expect it. I don't see any reason to discontinue.

Honestly I really can't follow this anymore. I agree with @luke-jr's goals.

-0.

Just a data point: I haven't had, or otherwise heard of, any complaints about the 0.20.x tarball aren't bootstrapped.

Well, it's not like we've ever had complete/usable tarballs before either... Maybe people are just used to needing to use git? Dunno.

Removed from 0.21 milestone for now

(The autogen.sh part was controversial and is not a regression-bugfix)

<!--cf906140f33d8803c4a75a2196329ecb-->

🐙 This pull request conflicts with the target branch and needs rebase.

_{Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a "draft".}

Closing this, as at this point, we aren't likely going back on the build / dist changes. I also don't think I'm aware of any complaints in regards to the release tarball being broken (that's for 0.20 and 0.21).