Add source tarballs to release process #2804

As git provides a way to export a tarball given a commit, it's very easy to add the creation of source releases, IMHO. We very much like people to build from git tags, but apparently this is hard or at least inconvenient for some infrastructure out there. I hope the overhead is minimal enough for those doing the release...

This also disables the use of git-describe in exported source archives, as it results in a spurious error otherwise.

I hope this closes #2476.

Tested that an (Ubuntu) build of the resulting tar's content works, and results in a meaningful version being reported.

Will the resulting tarballs be generated deterministically by the gitian build process as well so we can get signatures that the tarball matches the git tag?

@johndillon The tar file is generated by git, and there's no source of randomness (the owner is set to root, the timestamps are equal to that of the last commit, and files are sorted alphabetically) as far as I can see - though this can obviously change in newer git versions. So it could be done inside gitian, I suppose, as that guarantees a consistent version of git to generate it.

@sipa Provided that is true I'll ACK this patch.

In some ways making it easier for distribution managers to distribute Bitcoin anything bothers me in that they represent a very real risk of compromise with very high rewards, but if they are going to do so this is a reasonable thing to provide.

Ok, I've moved the construction of the tar file inside gitian, so hopefully it is deterministic now.

A few notes:

• Currently, the gitian build script doesn't know the "name" of the version being built, so the tarball uses a subdir bitcoin/ rather than bitcoin-$VERSION/; I don't know how much that is uncommon/unexpected.
• I've put it inside the Windows build, as I didn't want to create a full separate gitian build for just the source, and the Linux build gets run twice (32-bit, 64-bit) so it would be double work and the result would perhaps depend on the order in which you run the builds.

Thanks for taking this on.

It would be best for a number of reasons if the sourceforge download was identical to the github generated file, at least at the time of release. We can then use github as a backup to sourceforge, or vice versa. If the files are different, then we have to choose between the two. Sourceforge is faster and way more reliable, with their endless array of mirrors.

Mind you, I am torn on this one - not having the github hash in the directory makes things a lot simpler where I am concerned - although most of that complexity is managed by someone else.

On 30 June 2013 22:15, Pieter Wuille notifications@github.com wrote:

Ok, I've moved the construction of the tar file inside gitian, so hopefully it is deterministic now.

A few notes:

• Currently, the gitian build script doesn't know the "name" of the version being built, so the tarball uses a subdir bitcoin/ rather than bitcoin-$VERSION/; I don't know how much that is uncommon/unexpected.
• I've put it inside the Windows build, as I didn't want to create a full separate gitian build for just the source, and the Linux build gets run twice (32-bit, 64-bit) so it would be double work and the result would perhaps depend on the order in which you run the builds.

— Reply to this email directly or view it on GitHubhttps://github.com/bitcoin/bitcoin/pull/2804#issuecomment-20246227 .

With the current pullreq, there is no way that the hash will match what github builds. They have the tag name in the directory, and their compression seems worse (even for .tar.gz). However, you do get signed hashes of the tarball from every gitian builder.

Currently, even our binary tarballs and ZIPs are outside the signature mechanism, so I don't know how important doing a source tarball deterministically is. The end-user can always compare with the signed git tag, after all.

In any case, I think deterministically having gitian generate the same tar.gz as github is possible anyway.

@luke-jr But the released .zip and .tar.gz files have hashes signed by gitian. True, the entire source tree is signed as well, but being able to just compare the hash on a single .tar.xz is certainly easier.

Updated again. The created tarballs should now be identical to github's at the tar level. I'm not able to produce a binary identically .gz file, however.

Got it, added -cn to gzip, and now they match the github .tar.gz files.

Note that #2805 handles bootstrapping and tarball generation, which I believe should make this PR unnecessary. Instead, gitian would have a separate descriptor for source generation (running 'make dist') which is then fed into the other builds.

Closing in favor of #2805 or its successor, as the plan seems to be to generate a source tarball through make dist running in gitian.