no separate source code tarball download available #2476

on http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.1/ there's no source code tarball which i need to get reproducible results on my build-it-yourself distro (package management system is laid out for source tarballs with checksums, not git checkouts)

apparently the linux download has the source included, but also comes with prebuilt binaries (unusable in my case, since i use a different libc). this bloats up the tarball to twice the needed size or more, which affects download time and hd usage.

the argument that 18 MB is nothing compared to 7G of blockdata, is not valid for people who just fetch all tarballs in one go (so they have them already in case they need it), but in the end do not use the bitcoin client.

the other argument that github provides downloads is also <b>not valid</b>, since

1. they generate tarballs on the fly so the checksum changes whenever it is removed from cache (gz uses a random seed, so every archive produced is unique, even with the same content)
2. github redirects from http to http<b>s</b>, making it impossible to download anything with busybox' wget program. they did not even reply to my complaints about that.
3. github has no redundancy, when the frontend server(s) is down, everything is down
4. SF.net (which is already used for binary downloads) has at least 20 mirrors in different countries, and provides much better download speeds. additionally it lets the user choose if he wants HTTP or HTTPS downloads.

so please in the future release a separate source code tarball and make it available from the default download page. thanks.

please also consider using the .xz compressor since it's about 40-60% better in compression and only slightly slower in decompression speed than .gz (but 3-4x as fast as bz2) and available on any linux distribution that came out in the last 5 years. even busybox supports it by default.

A source code tarball is really needed for packagers.

The linux tarball is the source tarball.

Our experience with packagers has not been inspiring. What system do you plan on packaging bitcoin for?

"unusable in my case, since i use a different libc", do you actually have Bitcoin running on this system?

Yes, I see. Not having a source tarball is weird. Normally that is "the release" and the binaries are supplemental.

I'm working on a Fedora rpm.

Bitcoin cannot currently be usefully packaged for Fedora because the distribution patches out all ECC support from OpenSSL, which we currently depend on.

Yes, I have an alternate openssl rpm as well.

@gmaxwell my system is https://github.com/rofl0r/sabotage based on musl libc. and no, currently i dont have bitcoin running there due to the weird release situation for the official client. but i'd like to add it, once a regular source tarball is available.

I had fairly little luck getting boost to work on alpine a year or two ago. A separated binary and source tarball is not your biggest hurdle by far. (This is all an aside mostly but— maintaining more files to release is more work, and seems kinda pointless if it's only to appease some sense of rightfulness for some that doesn't even have Bitcoin working for them already; Not that I disagree that it would be better to have a separate freestanding source tarball.)

i have both qt4 and boost installed, the only thing missing is bitcoin. https://github.com/rofl0r/sabotage/blob/master/pkg/boost https://github.com/rofl0r/sabotage/blob/master/pkg/qt4

https://github.com/bitcoin/bitcoin/archive/v0.8.3.tar.gz ?

@sipa please read before post

Sorry, ignore me :)

@rofl0r if your response is that you want a non-SSL source for it, I'm going to have to WONTFIX that, cause thats kinda nuts. Should we ever get around to running our own resources for bitcoin.org and move all the hosting to it that site will be SSL only.

i didnt say anything about SSL i simply request a source tarball as any other project offers (without useless glibc binary crap)

Then grab it of the tag based URL that sipa pointed you to. Thats currently our supported method for providing source only files. I agree it's less good than freestanding file, but I don't see a justification for additional work considering that we have a costless workable alternative.

(Gzip is also not randomized in any way, though differences in zlib version can change the output, if the github snapshots are not deterministic thats due to something else)

The github files are determinstic for me: $ sha256sum v0.8.3.tar.gz* b56de5bde38714e83f69a4daf81c9a5577f151beda60427381770a1a4819af15 v0.8.3.tar.gz b56de5bde38714e83f69a4daf81c9a5577f151beda60427381770a1a4819af15 v0.8.3.tar.gz.1

as i said in the introductory text, HTTPS cannot be fetched by busybox wget

you're offering downloads for anything, why do you dig your heels to not offer a proper source download ?

it's about adding two lines in your make-release.sh: make clean; tar cJf bitcoin-$VER.tar.xz source/

• one line in your sourceforge upload code

As I said above, we will not commit to supporting a non-SSL download method for anything we provide on an ongoing basis. There is no sense in doing additional work for every release when doing so won't solve anyone's problems. You will not be satisfied because it won't stay useful to you, no one else will be satistfied because they're already happy with the two source download options we already have.

so why do you offer non-SSL downloads for windows and mac binaries via sourceforge ?

won't solve anyone's problem - so you're saying that i and @jeffmendoza are nobody ?

@rofl0r Because of legacy inertia— thats what sourceforge does. There is another issue where yet someone else is yelling at us over the non-use of SSL. There is absolutely no way that we're going to commit to providing non-ssl downloads for your busybox environment.

just because busybox' wget does not link to openssl does not mean that openSSL is not available on the host. it's just not being linked against because of huge bloat. the packaging system uses the host's wget, which could either be busybox wget or gnu wget depending on the user's preferences, but i have to make sure that both work.

@gmaxwell sourceforge is providing both HTTP and HTTPS, so that's completely irrevelant

won't solve anyone's problem - so you're saying that i and @jeffmendoza are nobody ?

I don't know who you are. I know that you are asking for something that some additional work on an already strained release process that seemingly almost no one else cares about, on the basis of apparently incorrect claims (github's tag links being non-determinstic), with additional preconditions which we will not commit to supporting (non-ssl downloads).

Do you think that responding aggressively and arguing every point is going to get you your way here? I might have made a commitment to do some extra work to support someone's particular requirements— if I understood them and thought it would help— but I'm certainly not inclined after this interaction. :(

As far as Jeff goes, he hasn't said why one of the two options we already have isn't sufficient for his usage.

what about FreeBSD, PCBSD, etc users ? shall they download a "linux" tarball full of binary linked glibc stuff just to get the source ? linux != GLIBC, altho thats a common misconception.

As you can see by my mention of Alpine above, I'm aware that Linux!=GLIBC (and... even my first linux system was not glibc). They can download the Linux release package— at a moderate size tax—, or they can grab a tag source URL, or they can fetch using git.

the problem is not so much the binary clutter in the tarball, although it is highly annoying. the main problem is that if someone goes on your site to search for a source code download, he doesnt find anything, except a link on github. my knowledge that the linux binary download also contains the source is from IRC

what this project is currently doing is turning a source code user into a second-class citizen while prefering binary junk consumers (altho only for popular platforms).

needless to say that the binary stuff in the linux tarball is useless for any user of a non-x86 glibc linux system, whereas the source code is universal and works everywhere.

As a FreeBSD maintainer, i would like to see the following:

• add a source code only tarball, which can be used by GNU/Linux, BSD, MAC, etc etc
• and optional an additional GNU/Linux 32 bits binary download

BSD users generally compile our stuff ourselves, so there's no need for a binary tarball.

nemysis: Here ya go, https://github.com/bitcoin/bitcoin/archive/v0.8.3.tar.gz

This is not as one stable SF tarball, please upload a stable tarball as bitcoin-0.8.3.tar.gz to SF, thanks

@nemysis I'm afraid you're going to have to explain what you need thats missing here.

It seems reasonable to me to put a source tarball on sourceforge along with release binaries. As things are, I agree it seems we're somewhat discriminating against source builds, which IMHO is something we should encourage (it removes the need for gitian-like trust in release binaries).

The person doing releases may not like the extra work though...

@gmaxwell BSD and GNU/Linux users need a stable Source tarball on SF.

@sipa Better to encourage builds from GIT, since then it leaves people in a position to send patches. ;) But indeed, it shouldn't be a big deal. I'm concerned by the weirdness of the responses here. @nemysis On SF? They do? Really? Why? You're just repeating yourself— please help me understand. The tar I linked to is a stable tarball.

@gmaxwell yes is stable here, but is not good to download as bitcoin-0.8.3.tar.gz with good checksum what is needed for BSD, GNU/Linux source tarball. This is only granted from SF.

@nemysis I understand your concern about a stable download - I assume the fact that checksums can be published and verified, and are guaranteed not to change (and I indeed don't know to what extent github provides this).

But just so I understand correctly: there is nothing specifically offered by SF that you need, right? It's just the fact that github doesn't guarantee stability of its source downloads?

@gmaxwell yes for a FreeBSD Port is needed a stable tarball with good checksums, which GitHub not offer. Here when a Port use GitHub must download a snapshots, and in distinfo is then a checksum which is reference, but this is not very good, because GH sometimes change download and have same name.

not to talk about redundancy offered by sf.net and download speeds.

@nemysis I'm sorry, I still must be confused. Whats wrong with USE_GITHUB and GH_TAGNAME in ports? I had thought many ports already pulled from github archive URLs just like ours.

@gmaxwell this is true, i have too some FreeBSD Ports which use GH, but problem is that GH change very oft his rules and then must FreeBSD portmgr@ change bsd.sites.mk that download works from GH with this new rules and all Ports must be sometimes changed to this new rules. And when is a source tarball on SF then is all good.

@nemysis OK! Thats good to know.

@gmaxwell good that you this now know, Could you now upload a stable bitcoin-0.8.3.tar.gz and future Versions to SF site, please?

@gmaxwell If we're going to move to automake, we'll end up needing to do source releases anyway. To take advantage of automake's functionality, programs need to distribute source with the configure script already generated.

Yes, please put the release source on Sourceforge. Github generated distfiles change frequently, for no good reason, and any build system relying on them is going to need constant fiddling. As the maintainer of the FreeBSD port, SourceForge distribution would make a lot of sense. It would also reduce load on github's servers, which cannot be a bad thing.

As Gentoo maintainer, I have never seen the GitHub tarballs change... Gentoo's build system checks the SHA256 of all source files, so it would have picked up on anything like this.

FreeBSD and other BSD checks too SHA256 but from downloaded tarball and all downloads when user wish to install Port check if this SHA256 from distinfo is the same what have Port maintainer make with 'make makesum', also this is verified install.

On Thu, Jun 27, 2013 at 2:33 PM, nemysis notifications@github.com wrote:

FreeBSD and other BSD checks too SHA256 but from downloaded tarball and all downloads when user wish to install Port check if this SHA256 from distinfo is the same what have Port maintainer make with 'make makesum', also this is verified install.

The same holds for Gentoo, and apparently Luke has no problems.

By the way, I have no objection against source tarballs as part of the release, but I guess it's up to the person doing the release.

@luke-jr does that mean gentoo checks the extracted files as opposed to the tarball itself ? that would explain why you didnt notice the changed hash i got this information on irc from a packager: <redacted> Github sometimes changes the way it rolls the distfile. The last time they switched from the hash for the last commit to the tree before it was tagged, to the hash that represented the tagging of the release.

Well, it happened less than a month ago.

http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179435

Nothing in this thread refutes the argument rofl0r made when he started the issue, being

the other argument that github provides downloads is also not valid, since

1. they generate tarballs on the fly so the checksum changes whenever it is removed from cache (gz uses a random seed, so every archive produced is unique, even with the same content)

This obviously affects all source based distributions that use the gz's SHA256/MD5/size, so whenever github changes a minor bit, all source distributions that use the checksums for verifying are affected.

Which is why exporting source based archives to sourceforge is a good idea.

@xmj What you're saying there is untrue to the best of my ability to determine. There is no random seed in gzip or zlib. There may be other causes of non-determinism (like the timestamp) which is why I was asking more questions (and regardless of what we do here, if github is actually non-determinstic we should get that fixed).

I'm sorry, it's just my nature to ask a lot of questions when people show up with incorrect arguments and perplexing desires like wanting to use sourceforge voluntarily. :) @robbak Thanks for the information, thats great!

@rofl0r Gentoo checks the tarball's hash itself.

@gmaxwell

/dev/shm $ mkdir zip-test
/dev/shm $ cd zip-test/
/dev/shm/zip-test $ echo "test bar foo baz" > test.txt
/dev/shm/zip-test $ cd ..
/dev/shm $ tar czf zip-test.tar.gz zip-test/
/dev/shm $ sha256sum zip-test.tar.gz 
d4b554db02cbb7f91660adbdabf94ecae07a4e65c213b557324fd480d16fca00  zip-test.tar.gz
/dev/shm $ rm zip-test.tar.gz 
/dev/shm $ tar czf zip-test.tar.gz zip-test/
/dev/shm $ sha256sum zip-test.tar.gz 
81f05ba5d208dc1731cde883d46439e3b5e8c026b8f298bb4bebd4b0c6eb0b4c  zip-test.tar.gz

@rofl0r Thats the timestamp.

[gmaxwell@helmholtz tmp]$ tar cf zip-test.tar.gz zip-test/ | gzip -cn - > zip-test.tar.gz ; sha256sum zip-test.tar.gz 7abee1d0f56eb5e4405b2f4b15d24a94ff7b3103b0d2f4ba63c11e68305e9df5 zip-test.tar.gz [gmaxwell@helmholtz tmp]$ tar cf zip-test.tar.gz zip-test/ | gzip -cn - > zip-test.tar.gz ; sha256sum zip-test.tar.gz 7abee1d0f56eb5e4405b2f4b15d24a94ff7b3103b0d2f4ba63c11e68305e9df5 zip-test.tar.gz

Github tarballs don't store it.

@gmaxwell tar cf zip-test.tar.gz zip-test/ doesnt output to stdout, so your pipe does effectively nothing

Sorry, wasting my time on things you could trivially check yourself isn't conducive to the greatest care.

[gmaxwell@helmholtz tmp]$ tar cf - zip-test/ | gzip -cn - > zip-test.tar.gz ; sha256sum zip-test.tar.gz 6b75d74dc0e312b466f305180d3b5fb930ac70f7f28775ae8498ac5a20f679b4 zip-test.tar.gz [gmaxwell@helmholtz tmp]$ tar cf - zip-test/ | gzip -cn - > zip-test.tar.gz ; sha256sum zip-test.tar.gz 6b75d74dc0e312b466f305180d3b5fb930ac70f7f28775ae8498ac5a20f679b4 zip-test.tar.gz [gmaxwell@helmholtz tmp]$ tar cf - zip-test/ | gzip -cn - > zip-test.tar.gz ; sha256sum zip-test.tar.gz 6b75d74dc0e312b466f305180d3b5fb930ac70f7f28775ae8498ac5a20f679b4 zip-test.tar.gz

and what makes you think github uses your way and not mine ?

btw, i get the hash 7aec0b60aae8eaf740b5dfbb802858d9122bdd5cf556ec38107ae1f1ef660686 with your command. so different machines give different hashes for some reasons. so depending on the mood of the github servers the hash will differ. why not just use a proper, manually generated tarball instead of the laughable download service that github shoehorned on top of their git repos (so they could stop the download service they offered previously, but then stopped due to cost reasons) ?

Thankfully, git archive already provides a deterministic tar creation.

Of course you get a different hash, the data you're tarring up is different, unless you happen to have the same UID as me, same file timestamps, etc.

I checked the github files, I checked them before I even responded to this stupid point. I'm getting a little tired of people stating as fact things they haven't even tried to check. (Or ignoring Luke's gentoo point)

did you even see this (quote of robbak):

Well, it happened less than a month ago.

http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179435

@rofl0r I did, did you? It had nothing to do with github file determinism, that was actually changing to a different commit id. (Interestingly: the one it changed to is completely invisible to me: nothing in the reflog on my checkouts with that hash. Nothing in the github UI, etc.)

Edit: I've decoded what appears to have happened in robbak's case: Originally he created the port using the ID of the ultimate commit on that version, but we subsequently added the digitally signed tag as part of the release process. The path in the tarball uses the ID of the tag (94933c3) instead of the commit.

Closing before my inbox overflows. Consensus seems to be github tarballs are Just Fine For Now.

@gavinandresen apparently you cant read, that is only the opinion of @gmaxwell and nobody else here

@gavinandresen ulrich drepper would be proud of you.

github tarballs are fine for now, as stated.

As @luke-jr indicated, this may change once automake support is complete and merged.

GitHub tarballs are not fine, Github generated distfiles change frequently, for no good reason, and any build system relying on them is going to need constant fiddling. Please upload to SF stable bitcoin-0.8.3.tar.gz. Thanks in advance

Github tarballs are not fine.

Please start posting source tarballs with the change to automake, sounds great to me. (yay autotools!) You're going to start using pkg-config too, I hope?

@jeffmendoza And what problems do the github tarballs cause for you? (I'm collecting problems to report to github: So far I have none and regardless of what we do in the future it would be nice if the functionality worked right for people.)

You can see here for some details. It will work, but a real release tarball would be much preferred. http://fedoraproject.org/wiki/Packaging:SourceURL#Github

@jeffmendoza is unability to download from HTTPS nothing ? this causes a huge problem for me and other busybox users. edit: oops, that should have been @gmaxwell

@gmaxwell Not quite. We always retrieved the tag, v8.0.2. Github (or maybe a bitcoin dev) changed how it determined what that tag meant. (Probably github - there was a rash of github-sourced ports that needed changing at around that time.)

@robbak Can you explain to me how a gentoo ebuild from two years ago: (http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-p2p/bitcoind/Manifest?revision=1.1&view=markup)

Has correct SHA256s for files I can download for github today?

E.g. "DIST bitcoin-v0.5.1.tgz 1007787 RMD160 107944a SHA1 2718930 SHA256 e33279066298e0a29e1dcca566d0a0bbcd89ef6e02d43bdbbeaf046f5faf66fe"

$ wget https://github.com/bitcoin/bitcoin/tarball/v0.5.1 $ sha256sum v0.5.1 e33279066298e0a29e1dcca566d0a0bbcd89ef6e02d43bdbbeaf046f5faf66fe v0.5.1 $ tar -ztf v0.5.1 bitcoin-bitcoin-5623ee7/ bitcoin-bitcoin-5623ee7/.gitignore bitcoin-bitcoin-5623ee7/COPYING [...]

I note that 5623ee7 is the ID of the signed tag, just like the current one that confounded you. So there can be no doubt that the same behavior was in place two years ago.

Maybe you were being MITMed? :p

I note that 5623ee7 is the ID of the signed tag, just like the current one that confounded you. So there can be no doubt that the same behavior was in place two years ago.

Maybe you were being MITMed? :package:

Or maybe a tag was just overwritten?

I'm reopening this issue.

I think this discussion got derailed into a question of why the OP asked for these tarballs (and I agree with gmaxwell about that mostly: we need to find out if and why github's tarballs aren't deterministic, and we need to encourage building from git tags/commits). Nonetheless, the request for source releases is very reasonable IMHO, and very simple to accomplish.

This should be easier now that the source tarballs produced by the gitian builds are deterministic.

@laanjw

could you please clarify for non-insiders where to get these tarballs?

They are not available as there has been no 0.9 release yet. I'm just saying that releasing deterministic source tarballs (that everyone with gitian builder can check) became possible.

Tarballs can be found along with the rest of the downloads, e.g. https://bitcoin.org/bin/bitcoin-core-0.10.1/bitcoin-0.10.1.tar.gz

the /bin/ in the URL suggests its a binary (compiled). this issue is about source tarballs

Now that's a dumb comment. /bin/ is just the internal site structure used for uploading releases and has no relevance here. The file I linked to is a source tarball. Please in the future try to do the minimum amount of research before you comment.