Generate release tarball with git archive #16734

1. 
   MarcoFalke commented at 5:09 pm on August 27, 2019: member

   A lot of files are missing from the release tarball (see #6753), so it would probably be easier and more consistnet to just replace it with a simple call to git archive than to enumerate every single file and add it to the dist target in the Makefile.
2. MarcoFalke added the label Brainstorming on Aug 27, 2019
3. MarcoFalke added the label Build system on Aug 27, 2019
4. 
   ch4ot1c commented at 12:17 pm on September 17, 2019: contributor

   So should we avoid make dist in the gitian builds, or put a git archive step in the dist-hook? The latter would make two tarballs & have issues with gitignore, which is undesirable. But at the same time, not using make dist goes against autotools principles.

   Also, the generated build files are automatically added to the tarball in make dist. If we plan to git archive the original source, they’d still have to be generated and appended. EXTRA_DIST directories might just be easier for standard make targets.

   Generated files that we want to include (I think):

    0aclocal.m4
    1Makefile.in
    2configure
    3config.guess
    4config.sub
    5(recursive)
    6
    7ltmain.sh
    8install-sh
    9depcomp
   10(recursive)
   11
   12src/config/bitcoin-config.h.in
   13src/secp2561/src/libsecp256k1-config.h.in
   

   Related info: https://www.gnu.org/software/automake/faq/autotools-faq.html#Which-files-are-hand_002dwritten-and-which-generated-_0028and-how_0029_003f

5. 
   MarcoFalke commented at 1:45 pm on September 23, 2019: member

   The autotools faq just say that configure (et al.) are “Generated on the developer system”, but don’t give a reason for it. I think it is fine and not too much to ask of a user that wishes to compile from source to call ./autogen.sh && ./configure && make instead of ./configure && make.
6. 
   ryanofsky commented at 3:01 pm on September 23, 2019: member

   but don’t give a reason for it

   It’s probably ok to require autogen. I think historically configure scripts were supposed to work on obscure unix systems that would just have a generic shell + make tool + c compiler, and not necessarily need gnu packages like m4 which are needed for autogen.

7. 
   MarcoFalke commented at 6:21 pm on October 10, 2019: member

   comment by @laanwj

   I agree, I’ve never been a fan of it. I don’t think we have any files in the git repository we don’t want to ship in the source tarball.

   from #17097 (comment)

8. MarcoFalke removed the label Brainstorming on Oct 10, 2019
9. MarcoFalke added the label Up for grabs on Oct 10, 2019
10. 
    laanwj commented at 7:01 pm on October 10, 2019: member

    Yes, concept ACK, I think. I also think it’s fine to require autogen. What is one dependency more, if you’re building from source.

    I suppose the practical issue is: is git archive output deterministic?

11. 
    ch4ot1c commented at 8:35 pm on October 10, 2019: contributor

    @laanwj I think we’ll still have to use faketime, even though the archive doesn’t include the .git directory. Nearly deterministic.

    Here’s what I’ve come up with so far, based on above discussion:

    0dist-hook:
    1	- $(GIT) archive --format=tar HEAD -- src/clientversion.cpp | $(AMTAR) -C $(top_distdir) -xf -
    2	- $(GIT) rm ".gitignore"; $(GIT) add ".gitignore"; $(GIT) stash --all;
    3	- $(GIT) archive --prefix="$(PACKAGE)-$(VERSION)/" stash@{0} -- . >"$(PACKAGE)-$(VERSION)-archive.tar.gz
    4	- $(GIT) stash pop
    5	- $(GIT) checkout -- ".gitignore"
    

    Remaining problems:

    1. Creates two archives, since make dist creates one automatically (swallow somehow?). Therefore, for now the git archive output has -archive as a suffix.
    2. Not deterministic
    3. Compatibility with existing gitian descriptors

    I’m less familiar with the Gitian system, so I’ll read up on it, but feel free to run with this example.

12. 
    MarcoFalke commented at 8:42 pm on October 10, 2019: member

    The hook is run after dist, so you could just overwrite the previous archive?
13. 
    ch4ot1c commented at 10:39 pm on October 10, 2019: contributor

    Been trying - it seems to always make the very last step its tarball creation, so it’s troublesome to overwrite.

    Efforts at https://github.com/bitcoin/bitcoin/pull/17104

14. MarcoFalke referenced this in commit f4d93078dd on Oct 11, 2019
15. sidhujag referenced this in commit bba1991b59 on Oct 11, 2019
16. MarcoFalke removed the label Up for grabs on Oct 12, 2019
17. 
    laanwj commented at 11:58 pm on November 7, 2019: member

    The hook is run after dist, so you could just overwrite the previous archive?

    I don’t think it’s desirable for it to do the work twice.

    If it’s too involved to override autotool’s own machinery, an alternative would be to stop using make dist to generate the tarball and use git archive directly from the descriptors. That’s where the tarballs for distribution are produced.

18. laanwj closed this on Mar 25, 2020

    ---

19. sidhujag referenced this in commit 56ca0913fc on Mar 28, 2020
20. UdjinM6 referenced this in commit 98417ecad2 on Oct 23, 2021
21. UdjinM6 referenced this in commit 9e85411dc7 on Oct 23, 2021
22. UdjinM6 referenced this in commit 83cbc3c811 on Dec 4, 2021
23. DrahtBot locked this on Feb 15, 2022

Contributors
MarcoFalke ch4ot1c ryanofsky laanwj

Labels
Build system