build: Use git archive as source tarball

hebasto commented at 10:26 am on March 12, 2020: member

This PR:

is an alternative to #17104
closes #16734
closes #6753

The idea is clear described by some developers:

MarcoFalke:

This whole concept of explicitly listing each and every file manually (or with a fragile wildcard) is an obvious sisyphean task. I’d say all we need to do is run git archive and be done with it forever, see #16734, #6753, #11530 …

laanwj:

I agree, I’ve never been a fan of it. I don’t think we have any files in the git repository we don’t want to ship in the source tarball.

The suggested changes have a downside which is pointed by luke-jr:

… but the distfile needs to include autogen-generated files.

This means that a user is not able to run ./configure && make right away. One must run ./autogen.sh at first.

Here are opinions about mandatory use of ./autogen.sh:

ryanofsky:

It’s probably ok to require autogen. I think historically configure scripts were supposed to work on obscure unix systems that would just have a generic shell + make tool + c compiler, and not necessarily need gnu packages like m4 which are needed for autogen.

laanwj:

I also think it’s fine to require autogen. What is one dependency more, if you’re building from source.

~Also this PR provides Windows users with ZIP archives of the sources. Additionally the commit ID is stored in these ZIP files as a file comment:~

Note for reviewers: please verify is git archive output deterministic?

build: Use git archive as source tarball 5e6b8b3912

build: Drop SOURCEDIST reordering

Making SOURCEDIST deterministic is needless since a git archive is used
as the source tarball.

6c4da59f5b

fanquake added the label Build system on Mar 12, 2020

fanquake added this to the milestone 0.20.0 on Mar 12, 2020

hebasto force-pushed on Mar 12, 2020

laanwj commented at 12:33 pm on March 12, 2020: member

Thank you for picking this up.

It’s nice that this mostly removes lines, it’s a cleanup too. Adding a .zip source archive for Windows users is nice, probably, though I’m not sure if we can accomodate this in the website easily.

hebasto renamed this:
~~[WIP] build: Use git archive as source tarball~~
build: Use git archive as source tarball
on Mar 12, 2020

hebasto marked this as ready for review on Mar 12, 2020

in contrib/gitian-descriptors/gitian-linux.yml:154 in f05f8a49e0 outdated

146@@ -147,13 +147,6 @@ script: |
147   SOURCEDIST=$(echo bitcoin-*.tar.gz)
148   DISTNAME=${SOURCEDIST/%.tar.gz}
149 
150-  # Correct tar file order
151-  mkdir -p temp
152-  pushd temp
153-  tar -xf ../$SOURCEDIST
154-  find bitcoin-* | sort | tar --mtime="$REFERENCE_DATETIME" --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ../$SOURCEDIST

laanwj commented at 3:02 pm on March 12, 2020:

So this is not needed for the git archive output? That’s nice!

Oh, you mention this in the OP. I cannot find anything about this in the documentation, but looking at the output of git archive:

Files are sorted in alphabetic order
Files are owned by root/root
Permissions seem to be consistent:

0-rw-rw-r--  for normal files
1drwxrwxr-x  for directories
2-rwxrwxr-x  for executable scripts

The timestamp seems to be the same for all the files, but I don’t know where it comes from. Oh it’s the “CommitDate” of the last commit!

0-rw-rw-r-- root/root      4705 2020-03-11 16:45 .appveyor.yml

0$ git show --pretty=fuller HEAD
1commit 24a727e23e4143bcc4e5dfac536bae8d8261d32a (HEAD -> master)
2Merge: 3b21a7863411b4b0391313e765f23b2664d4540a faae5a9a356d821f0cbdea32030b0ce356351a1d
3Author:     Wladimir J. van der Laan <laanwj@protonmail.com>
4AuthorDate: Wed Mar 11 16:32:59 2020 +0100
5Commit:     Wladimir J. van der Laan <laanwj@protonmail.com>
6CommitDate: Wed Mar 11 16:45:47 2020 +0100

So yes, that looks deterministic to me (assuming the timezone doesn’t have a part in it, but tar stores UNIX timestamps so probably not).

hebasto commented at 3:22 pm on March 12, 2020:

The timestamp seems to be the same for all the files, but I don’t know where it comes from. Oh it’s the “CommitDate” of the last commit!

Yes. From git archive docs:

git archive behaves differently when given a tree ID versus when given a commit ID or tag ID. In the first case the current time is used as the modification time of each file in the archive. In the latter case the commit time as recorded in the referenced commit object is used instead.

laanwj commented at 3:40 pm on March 12, 2020:

Seems this is the same for the mtime in the global header.

laanwj commented at 3:31 pm on March 12, 2020: member

Also this PR provides Windows users with ZIP archives of the sources. Additionally the commit ID is stored in these ZIP files as a file comment:

It looks like the generated .tar archive also has a comment embedded with the commit ID. I do not know any way (besides hexdump) of getting this information.

000000200  35 32 20 63 6f 6d 6d 65  6e 74 3d 32 34 61 37 32  |52 comment=24a72|
100000210  37 65 32 33 65 34 31 34  33 62 63 63 34 65 35 64  |7e23e4143bcc4e5d|
200000220  66 61 63 35 33 36 62 61  65 38 64 38 32 36 31 64  |fac536bae8d8261d|
300000230  33 32 61 0a 00 00 00 00  00 00 00 00 00 00 00 00  |32a.............|
400000240  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Oh, this is also mentioned in the manpage:

Additionally the commit ID is stored in a global extended pax header if the tar format is used; it can be extracted using git get-tar-commit-id. In ZIP files it is stored as a file comment.

0git get-tar-commit-id < test.tar
124a727e23e4143bcc4e5dfac536bae8d8261d32a

hebasto commented at 3:42 pm on March 12, 2020: member

It looks like the generated .tar archive also has a comment embedded with the commit ID. I do not know any way (besides hexdump) of getting this information.

Yes. From git archive docs:

Additionally the commit ID is stored in a global extended pax header if the tar format is used; it can be extracted using git get-tar-commit-id.

EDIT: Sorry, slow typing…

MarcoFalke added the label Needs gitian build on Mar 14, 2020

MarcoFalke commented at 4:17 pm on March 14, 2020: member

ACK 86d9fae2943ad0d65ced2737f205db33f0bdc324 🤞

Signature:

 0-----BEGIN PGP SIGNED MESSAGE-----
 1Hash: SHA512
 2
 3ACK 86d9fae2943ad0d65ced2737f205db33f0bdc324 🤞
 4-----BEGIN PGP SIGNATURE-----
 5
 6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
 7pUgJ3Av+JDgUre1dSaM2RmQQMxS3Qs1MeW1A2byPrGwosk53SkUWU/Pbxls2Bnl4
 8W2yF7J5HV8C95/864vUi9b1GRpE5Q3H/khgD1jqXUTXZW1VfQ8OtDImh9K8kNqf0
 9JvtucQyMoyiZXN1Enh+ck4/BRhEOD1aSGHQHW0lPpSDTZfh2oEhHA5dpzSYpOv+K
10/pJxHHNNVGBaGaVuwVbyhAAievXySr9uS86w/q0e0NVnS4k2Jpac/e80hNVP6YK0
11U4FgWCmoAl9Zxoc73YkBSPZpoajrN8vqbH7/DUIFR8tzBTIiAl3/yQI6gK97JH/r
12v1HJ9gDgqa5euZncu536jsHwOhh6KWIC7D1XBbEVKL1k1Tr3gioI6F6Su1A/2OTR
1375RQGFb99MxRONLI+Kf4iy34hG//8EE/R6yRMRNp28Lxx4nUp+i0ZhgxSwOrsiz7
14Qlt14ejMnYC728yED74nsqNgVPqZbp5n5yn3M3G9Ip5/Xchc542UwuOuwLyluNgl
15ZqFZrTHX
16=63lK
17-----END PGP SIGNATURE-----

Timestamp of file with hash 70fc07cf6b63dd0d692f649fe3b0e90da30ba9c76c335ff9c9c3897f09d7551b -

in contrib/gitian-descriptors/gitian-win.yml:159 in f05f8a49e0 outdated

155@@ -156,7 +156,7 @@ script: |
156   done
157 
158   mkdir -p ${OUTDIR}/src
159-  git archive --output=${OUTDIR}/src/${DISTNAME}.tar.gz HEAD
160+  git archive --output=${OUTDIR}/src/${DISTNAME}.zip HEAD

MarcoFalke commented at 4:20 pm on March 14, 2020:

Why?

The tar is provided so that users can compile from source (without having to clone from GitHub and checkout the tag). I don’t think we really support compiling on Windows. And when we do, the extra dependency to extract a tar file is probably not too much to ask for from a windows user.

Finally, as pointed out by @laanwj, this probably does need adjustment on the Website. Not sure if it is worth it.

hebasto commented at 4:59 pm on March 14, 2020:

Commit dropped (https://github.com/bitcoin/bitcoin/pull/18331#issuecomment-599098374).

hebasto force-pushed on Mar 14, 2020

hebasto commented at 4:58 pm on March 14, 2020: member

Updated f05f8a49e02b49b6bcbdb82681b4c67509e8db73 -> 86d9fae2943ad0d65ced2737f205db33f0bdc324 (pr18331.01 -> pr18331.02, compare):

dropped “build: Suggest source as zip archive for Windows” commit

DrahtBot commented at 3:50 am on March 15, 2020: member

Gitian builds

File	commit 7f8176a1ebd00679596971fd01012cf61743b826(master)	commit 6391f76132f23ee87e90a496270ee21333ffc928(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`4c336065e794b9e1...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`bbf9ad146865c1e6...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`1383fcfcbb2db8f4...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`6ce0cccb1785c346...`
bitcoin-0.19.99-osx-unsigned.dmg	`2a808ed2d1574120...`	`b88fa3caccdb546c...`
bitcoin-0.19.99-osx64.tar.gz	`6bd11e0fe31a8380...`	`7ddbd5b0c53b4e94...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`7ce346c1502553c5...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`1afacc60029cc676...`
bitcoin-0.19.99-win64-debug.zip	`52e7784561b63b13...`	`69e9cbe82f0afbc2...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`ea46b617c4e5e1db...`	`a3789647f95c0bf1...`
bitcoin-0.19.99-win64.zip	`d321828fb88693ea...`	`60b39d946c3c2e64...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`911222a3da31873e...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`f86f8f63b76f2988...`
bitcoin-0.19.99.tar.gz	`b44bb765db3d7361...`	`f4705976b2e1f607...`
bitcoin-core-linux-0.20-res.yml	`7de5bda558602555...`
bitcoin-core-osx-0.20-res.yml	`40e50bfc96a0d76c...`	`014344aeee881717...`
bitcoin-core-win-0.20-res.yml	`7d76bbad6182235c...`	`42369c283707f147...`
linux-build.log	`b6798173d2724836...`	`1fef95497c0caaf9...`
osx-build.log	`ef5c29eaa0e5787b...`	`1ff1d8f5edea1c3b...`
win-build.log	`caf14e04ba8efa7c...`	`926fc96737bf6363...`
bitcoin-core-osx-0.20-res.yml.diff		`17014bfc1b175e31...`
bitcoin-core-win-0.20-res.yml.diff		`a4e13111f03578b3...`
linux-build.log.diff		`a02e18a16d31038b...`
osx-build.log.diff		`cb78ed91e77fc215...`
win-build.log.diff		`0f93154e9dce4282...`

DrahtBot removed the label Needs gitian build on Mar 15, 2020

in contrib/gitian-descriptors/gitian-linux.yml:189 in 86d9fae294 outdated

188@@ -196,5 +189,6 @@ script: |
189     cd ../../

MarcoFalke commented at 5:01 am on March 15, 2020:

3 lines up:

0+ cp ../README.md bitcoin-0.19.99/
1cp: cannot stat '../README.md': No such file or directory

hebasto commented at 10:56 am on March 15, 2020:

Fixed: #18331 (comment)

hebasto force-pushed on Mar 15, 2020

hebasto commented at 10:55 am on March 15, 2020: member

Updated 86d9fae2943ad0d65ced2737f205db33f0bdc324 -> c3c8d3cb06c72e14b5e264ca1492ce8a758814c7 (pr18331.02 -> pr18331.03, compare):

reworked docs processing

MarcoFalke added the label Needs gitian build on Mar 15, 2020

MarcoFalke removed the label Needs gitian build on Mar 15, 2020

build: Drop needless EXTRA_DIST content

Some EXTRA_DIST content is needless since a git archive is used as the
source tarball.

e4d366788b

hebasto force-pushed on Mar 15, 2020

hebasto commented at 5:14 pm on March 15, 2020: member

Updated c3c8d3cb06c72e14b5e264ca1492ce8a758814c7 -> e4d366788bc2e8dce8e6ca572fce08d913d15d6b (pr18331.03 -> pr18331.04, diff; diff 18331.02..pr18331.04):

fixed windows installer (was broken on the previous push)

MarcoFalke added the label Needs gitian build on Mar 15, 2020

DrahtBot commented at 1:07 pm on March 16, 2020: member

Gitian builds

File	commit d402c1e4d3e812341176e0948d64ff126f9ec3c1(master)	commit 31681d5bff526f2b765bbfb45e63707a108fab79(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`957b5705c1a66fa5...`	`cd999c2a7936cf3f...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`62975329879acd4e...`	`8550cab537180d49...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`ea509ada0d57b8ef...`	`2c3eb5fe0d286139...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`fac6f7543092aa06...`	`2956a97545c47b8a...`
bitcoin-0.19.99-osx-unsigned.dmg	`ed2e0fa5deb625ec...`	`36ed6ec5cdd32e82...`
bitcoin-0.19.99-osx64.tar.gz	`f21abdcdad8b2e1d...`	`96f00bfda76411a0...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`1cd66acbf12dc018...`	`d59dbea52b534f46...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`4e8cf7359e33b3f6...`	`6b334b7185014427...`
bitcoin-0.19.99-win64-debug.zip	`d15df67686062c17...`	`7d815bef058a1e22...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`0f5d898f95bb7ff4...`	`251387dd7e9ebdd3...`
bitcoin-0.19.99-win64.zip	`611679bd7cd092f5...`	`a399d6667bdf9c69...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`2c506693103f8df2...`	`57d687c44ed64976...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`75a57fd92a81d0e3...`	`de7fd1fae799662a...`
bitcoin-0.19.99.tar.gz	`20be7528cd98d6ae...`	`765577ae20e0dbcc...`
bitcoin-core-linux-0.20-res.yml	`d9419f4daaae64c7...`	`98fe2e965c067f0d...`
bitcoin-core-osx-0.20-res.yml	`03b7b79c9f7e54c1...`	`99dcca110ee4e5c6...`
bitcoin-core-win-0.20-res.yml	`a3dee819ac2e27d4...`	`033ee52173bad551...`
linux-build.log	`7a6ae533983e04f4...`	`a1bfb1e8c901cd7d...`
osx-build.log	`655ecded8a6a657e...`	`01d21686698f0c09...`
win-build.log	`d24ce786c98003a7...`	`028b4a331d0964ac...`
bitcoin-core-linux-0.20-res.yml.diff		`f86d1e0d7812c6d9...`
bitcoin-core-osx-0.20-res.yml.diff		`34cb8e0b6ab1a6bb...`
bitcoin-core-win-0.20-res.yml.diff		`ece3f4b0b1954de2...`
linux-build.log.diff		`e71ad7f13be23708...`
osx-build.log.diff		`cf611086e4e14f07...`
win-build.log.diff		`7d74addde228ca80...`

DrahtBot removed the label Needs gitian build on Mar 16, 2020

MarcoFalke commented at 2:28 pm on March 16, 2020: member

For archival purposes, since DrahtBot wipes the builds after a while:

bitcoin-0.19.99.tar.gz

bitcoin-0.19.99_new.tar.gz

re-ACK e4d366788bc2e8dce8e6ca572fce08d913d15d6b, only change is adding two dots in a the path 🛳

Signature:

 0-----BEGIN PGP SIGNED MESSAGE-----
 1Hash: SHA512
 2
 3re-ACK e4d366788bc2e8dce8e6ca572fce08d913d15d6b, only change is adding two dots in a the path 🛳
 4-----BEGIN PGP SIGNATURE-----
 5
 6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
 7pUirRgv/YnwsyN8A8pf5eyxqgi5qb9voP10DH/iI0wI47kkRfgNxHNuJE9hWIQMR
 88DSE3rwNHWr9R1cJrJqV6Gy4xeLXsmE6VGsW1VXc8NeKg8D+UZAgDkMJ/onVPilb
 9RNp8sVRluXvr+NO+rh3Id3gYkz5MNHm9MIz+Na9UZYEDTlByed8DhJq6IhmqCFjc
10gcdU4iPFNkJG+grPRR5++hFy4PcgOjIbHpThuZPzqaxM4hK83Clbucj0ICAIviRG
11zdSLm/i9YXzQZNYHDEIHGePVChJ8iYGgpSNIRo2z3m1XsNqGQvcG6wdGMkF1K8Ju
12k+Yazrr6aWy+mxwJRb621xUVCQlJrBfVE0/xTNT5uioBveJrQR95eTrP20oV0GjI
13lmisdFa03kmpiHkHWBQ5wMFslITPNe/4enAvukOlMvexsuHZig2sBEB9EoG+Gfzc
14WGjXGekdXal3RLhrm+tsNCjeyrUkFaFAaMDg4TTEVFHfkXNyI9Yp5gOVwS5Pr9IP
15Fsh+65rN
16=lSr7
17-----END PGP SIGNATURE-----

Timestamp of file with hash 885ed0aba5f83eaf085d59c35d2e5f2c7257ad642ea0929b4023d0c90888c9b0 -

MarcoFalke commented at 2:38 pm on March 16, 2020: member

Why is GIT_ARCHIVE set in src/clientversion.cpp even on current master? I thought we don’t use git archive yet?!

MarcoFalke commented at 3:45 pm on March 16, 2020: member

To answer my own question, we do use git archive:

0$ git grep -W 'GIT) archive'
1Makefile.am=dist-hook:
2Makefile.am:    -$(GIT) archive --format=tar HEAD -- src/clientversion.cpp | $(AMTAR) -C $(top_distdir) -xf -

MarcoFalke commented at 3:46 pm on March 16, 2020: member

@fanquake @laanwj Anything left to do here? I think this is ready

hebasto commented at 3:48 pm on March 16, 2020: member

@MarcoFalke

Why is GIT_ARCHIVE set in src/clientversion.cpp even on current master? I thought we don’t use git archive yet?!

This is the way to define GIT_ARCHIVE, GIT_COMMIT_ID, and GIT_COMMIT_DATE macros: https://github.com/bitcoin/bitcoin/blob/86623873095f8d73fd28ad323ed3d06e20433176/src/clientversion.cpp#L44-L48

using git export-subst attribute.

https://github.com/bitcoin/bitcoin/blob/86623873095f8d73fd28ad323ed3d06e20433176/.gitattributes#L1

EDIT: Sorry for slow typing. Again.

laanwj commented at 1:46 pm on March 25, 2020: member

ACK e4d366788bc2e8dce8e6ca572fce08d913d15d6b

laanwj merged this on Mar 25, 2020

laanwj closed this on Mar 25, 2020

hebasto deleted the branch on Mar 25, 2020

theuni commented at 9:42 pm on March 25, 2020: member

Sorry I got to this too late, but… I think this is a bad idea.

Requiring autogen essentially defeats the point of using autotools, which is to generate a portable configure script. This will likely be a painful move for downstream maintainers and packagers, who will now have to adjust their build dependencies.

Enumerating each file seems like a perfectly reasonable ask for a project with precision requirements as this one. Imo this is a move in the wrong direction.

ryanofsky commented at 10:45 pm on March 25, 2020: member

Requiring autogen essentially defeats the point of using autotools

This seems like a real concern. Does this PR now require running autogen when it wasn’t required before? If so, it seems like it should have release notes and documentation updates.

MarcoFalke commented at 1:57 pm on March 26, 2020: member

painful move for downstream maintainers and packagers, who will now have to adjust their build dependencies.

For a major version bump they’d need to adjust the build dependencies anyway (e.g. remove openssl and boost chrono, …). I don’t really see how adding autotools is going to be painful. Though, if it really is that painful, we could run it in the gitian vm and append the resulting configure script to the archive.

it should have release notes and documentation updates.

We don’t really have any documentation about our release tarball, so there is nothing to update I think. I am happy to leave a note in the release notes.

ryanofsky commented at 2:23 pm on March 26, 2020: member

We don’t really have any documentation about our release tarball, so there is nothing to update I think. I am happy to leave a note in the release notes.

I probably should familiarize myself with documentation more. I wrote my comment above assuming this PR required a new install step that was optional before, so installation instructions would be updated.

I don’t think we should hesitate to write release notes for user visible changes. PR descriptions are often unhelpful because they focus on implementation details, while release notes force you to summarzie what the actual effect of a change is (it’s still unclear here what files this PR removes from the tarball).

theuni commented at 2:33 pm on March 26, 2020: member

For a major version bump they’d need to adjust the build dependencies anyway (e.g. remove openssl and boost chrono, …). I don’t really see how adding autotools is going to be painful.

I’m not complaining about forcing downstreams to update. I’m complaining about the contents of that update. They’re going to have to add a pile of new build-time dependencies (autotools dev packages), which may or may not pose further complications.

hebasto commented at 2:41 pm on March 26, 2020: member

Release notes added in #18439.

MarcoFalke commented at 2:51 pm on March 26, 2020: member

@ryanofsky Our only instruction of building are here: https://github.com/bitcoin/bitcoin/blob/master/doc/build-unix.md#to-build . The first step for them is ./autogen.sh (so nothing changed for them). There is also a source code download link on https://bitcoincore.org/en/download/ . This is what is going to change after this pull request. None of our own packaging scripts should change as an effect of that. They either use the static binaries or the git archive already https://github.com/bitcoin-core/packaging/blob/2c1d6a4cc67411beb3e035d126b56bef3cf7e1f1/debian/rules#L30

I can’t comment on how hard it will be for other third party package maintainers to run autogen, but at least the debian one has a full copy of the git repo: https://sources.debian.org/src/bitcoin/0.18.1%7Edfsg-1/ . So I am not sure if there is anyone who depends on this targz and depends on this targz having the configure script included. My opinion is to wait for complaints before changing things again…

ryanofsky commented at 2:57 pm on March 26, 2020: member

I’m not complaining about forcing downstreams to update. I’m complaining about the contents of that update. They’re going to have to add a pile of new build-time dependencies (autotools dev packages), which may or may not pose further complications.

Are there particular operating systems / distros you’re concerned about here? I haven’t had the most autotools experience in the world, but I haven’t personally run into cases where earlier autoreconf steps fail, it’s always the later config / make stages that seem more fragile.

Or maybe there are systems where autotools and m4 are flat out broken or unavailable?

theuni commented at 3:12 pm on March 26, 2020: member

I’m not aware of any. Practically, this will likely simply mean that maintainers add deps to a list and move on.

But at a higher level, we can offer a pre-processed source tarball which eases the burden on the builder (as well as adding improving determinism by pegging build tools at tag time) or we could not.

After this change, our source tarball is simply a shortcut to a git clone and checkout. Maybe that’s ok, but it just doesn’t seem worth the hassle of finding out to me.

But sure, now that it’s in, we can just wait and see if it causes any real-world issues at release time.

MarcoFalke commented at 3:17 pm on March 26, 2020: member

We couldn’t keep up listing all needed files (or even establish guidelines about what files are needed) reliably, so this was the primary reason to change the source tarball. If the missing configure script is a burden or causes issues otherwise, what about appending it to the git archive after running autogen in the gitian vm?

This seems to solve both concerns?

theuni commented at 3:22 pm on March 26, 2020: member

We couldn’t keep up listing all needed files (or even establish guidelines about what files are needed) reliably, so this was the primary reason to change the source tarball. If the missing configure script is a burden or causes issues otherwise, what about appending it to the git archive after running autogen in the gitian vm?

This seems to solve both concerns?

This is exactly what gitian did before this change. It’s more than just ‘configure’ though, ‘make dist’ appends all bootstrapped files to the source tarball.

It sounds like you would just want an additional step of adding missing git files to that tarball.

Edit: I think we’re saying the same thing now, we just have different ideas of how to accomplish it.

sidhujag referenced this in commit 56ca0913fc on Mar 28, 2020

fanquake referenced this in commit 65fb3dfc8d on Apr 28, 2020

sidhujag referenced this in commit 3d6b60d09e on Apr 28, 2020

fanquake referenced this in commit 0b294c0ba8 on Mar 2, 2021

UdjinM6 referenced this in commit 98417ecad2 on Oct 23, 2021

UdjinM6 referenced this in commit 306cb1b47a on Oct 23, 2021

UdjinM6 referenced this in commit 9e85411dc7 on Oct 23, 2021

UdjinM6 referenced this in commit ad65a4bed0 on Oct 23, 2021

UdjinM6 referenced this in commit 83cbc3c811 on Dec 4, 2021

UdjinM6 referenced this in commit a0d3f37d3a on Dec 4, 2021

UdjinM6 referenced this in commit 632c759352 on Dec 5, 2021

DrahtBot locked this on Feb 15, 2022

build: Use git archive as source tarball #18331

Gitian builds

Gitian builds