Contrib: Introduce script to tag compiled binaries for convenience (py) #7665

jtimon commented at 3:55 AM on March 10, 2016: contributor

The basic goal is to make it easier to avoid recompiling branches that have been compiled already.

I felt I needed to incorporate something like this to my workflow. If it is not useful for bitcoin, I'm happy to propose it on elements, freicoin or just keep it to myself. Still putting it here just in case.

Example usage (with the appropriate bitcoin/.git/config):

git fetch origin
git checkout origin/v0.12.0
make check -j21
python contrib/devtools/tag-binaries.py v0.12.0

git checkout origin/master
make check -j21
python contrib/devtools/tag-binaries.py master

Non-bitcoin examples:

git fetch elements
git checkout elements/alpha
make check -j21
python contrib/devtools/tag-binaries.py alpha

git checkout jtimon/jt
make clean
make check -j21
python contrib/devtools/tag-binaries.py jt

git fetch freicoin
git checkout freicoin/master
git clean -fdx
./autogen.sh
./configure
make check -j21
python contrib/devtools/tag-binaries.py freicoin

git fetch elements
git checkout elements/pre-gamma
git clean -fdx
./autogen.sh
./configure
make check -j21
python contrib/devtools/tag-binaries.py pre-gamma

jonasschnelli commented at 8:05 AM on March 10, 2016: contributor

I think this can be useful and should not hurt if its placed in devtools. The problem with devtools is a missing documentation/overview. Therefore I would recommend to add a short documentation including an use case & example in devtools/README.me (or similar).

jonasschnelli added the label Dev Scripts on Mar 10, 2016

in contrib/devtools/tag-binaries.py:None in 25623894ed outdated

   0 | @@ -0,0 +1,39 @@
   1 | +#!/usr/bin/env python

MarcoFalke commented at 9:07 AM on March 10, 2016:

Bikeshed: archive-binaries.py?

jtimon commented at 10:13 PM on March 10, 2016:

As said the tagging (with [<petname>-]<commit_id>) seems the important part to me, but I wouldn't oppose to archive if more people prefer it.

MarcoFalke commented at 9:11 AM on March 10, 2016: member

I guess this is useful. (I mostly fetch the gitian binaries from somewhere because I don't archive my own compiles.) Though, I wonder if this script would be still useful after #7466.

jtimon force-pushed on Mar 10, 2016

jtimon commented at 10:09 PM on March 10, 2016: contributor

@jonasschnelli oops, I missed the README, I added a longer explanation there (longer than the one in the internal code documentation). @MarcoFalke re #7466: the way I see it, the more useful feature of this script is renaming the files with commit ids and something else rather than just moving the binaries (which doesn't require much effort anyway). Probably shorter versions of the commit ids would do as well (just the first X chars or ssomething).

jtimon force-pushed on Mar 10, 2016

Contrib: Introduce script to tag compiled binaries for convenience (py) b5a6eb3be3

jtimon force-pushed on Mar 10, 2016

jtimon commented at 10:44 PM on March 10, 2016: contributor

Updated again renaming the script from tag-binaries to archive-binaries as suggested by @MarcoFalke and only using the first 7 chars of the commit id (github seems to shorten commit ids to first 7 too, no other reason).

EDIT: here's an example after playing with it a little bit:

    bin/alpha-cli-alpha-574817b
    bin/alpha-tx-alpha-574817b
    bin/alphad-alpha-574817b
    bin/bitcoin-cli-master-c8d2473
    bin/bitcoin-cli-v0.12.0-188ca9c
    bin/bitcoin-tx-master-c8d2473
    bin/bitcoin-tx-v0.12.0-188ca9c
    bin/bitcoind-master-c8d2473
    bin/bitcoind-v0.12.0-188ca9c
    bin/freicoin-cli-freicoin-0.9-188ca9c
    bin/freicoind-freicoin-416e028
    bin/qt/alpha-qt-alpha-574817b
    bin/qt/bitcoin-qt-master-c8d2473
    bin/qt/bitcoin-qt-v0.12.0-188ca9c

fixup! support for elements/alpha, etc c63f192420

jtimon force-pushed on Mar 10, 2016

laanwj commented at 8:33 AM on March 11, 2016: member

Wouldn't using make install to a commit-dependent prefix be more consistent to use here?

I mean, that is already the build system's own way to copy the executables, and potentially, other files that have to be installed (e.g. the consensus library). It also means the script doesn't need to be updated when new executables are added.

laanwj commented at 8:34 AM on March 11, 2016: member

Ping @theuni ^^

jtimon commented at 11:42 AM on March 11, 2016: contributor

The make install idea makes sense. Maybe the whole script could be replaced with a "make tag" or "make archive" option? Anyway, I'll wait for @theuni to chime in.

theuni commented at 7:23 PM on March 11, 2016: member

Sure, something like 'make stash' make sense, since it can be smart about which binaries to save. This works for me (add to Makefile.am).

stashbin:
        $(MAKE) $(AM_MAKEFLAGS) -o install-libLTLIBRARIES install-binPROGRAMS bindir=$(abs_builddir)/bin transform="s&\$$\$$&-`$(GIT) log -n 1 --format="%h"`&"

Might make sense to move it to bin/$hash/files though?

MarcoFalke commented at 7:28 PM on March 11, 2016: member

Might make sense to move it to bin/$hash/files though?

What about bin/$version/$hash/?

Anyway, I like the make stash way of doing this.

jonasschnelli commented at 7:35 PM on March 11, 2016: contributor

+1 for make stash.

Now someone should combine this with a modified version of github-merge.py so we can build and stash by running something like buildstash.py <PR number>.

jtimon commented at 9:22 PM on March 11, 2016: contributor

Many good ideas. I'll leave this opened until there's a make-based replacement. I will be happy to test that too. To be clear, I just want something that it's git ignored and I kno how to https://github.com/bitcoin/bitcoin/pull/7665/files#diff-cac857715e8776162045b7b41f9ccc87R27 . If it only works for %sd, %s-tx, etc when s% is 'bitcoin', then I will still need something external, but it would still use the only-bitcoin stuff.

Any volunteers to take over this?

in contrib/devtools/README.md:None in c63f192420

  22 | +make check -j21
  23 | +python contrib/devtools/archive-binaries.py v0.12.0
  24 | +./bin/bitcoind-v0.12.0-188ca9c
  25 | +
  26 | +git checkout origin/master
  27 | +git clean -fdx -e bin/

MarcoFalke commented at 3:13 PM on March 12, 2016:

Isn't make clean enough? (#7504)

jtimon commented at 3:19 PM on March 12, 2016:

It seems sometimes it isn't. But it's not clear to me when (most times clean is fine). Maybe @laanwj or @theuni can clarify more.

MarcoFalke commented at 4:05 PM on March 13, 2016:

It's only cleaning up its own mess. If you create junk files, make clean won't delete them afaiu.

laanwj commented at 10:09 AM on March 14, 2016:

try make distclean, this should leave the tree clean as it was in git. If not, that's a bug.Of course it will only clean up files it created itself.

jtimon commented at 10:26 AM on March 14, 2016:

Does this include files that it created itself in older versions?

laanwj commented at 10:50 AM on March 14, 2016:

No, it can't know about those anymore. You should run make distclean with the same version as you did configure.

jtimon commented at 10:15 PM on March 14, 2016:

I wonder if we could start maintaining a file for those to later git clean -e cat src/.git_clean_exclude_files. It wouldn't need to be complete for older versions, just start now and if people find time to find more stuff to include for the older versions, you can add that too. Oh, sorry, actually what we want is to explicitly include files for cleaning, not exclude them. Is there an option for that? To be clear, the use case for this cleanup is time travelling to very old versions and coming back, in case anybody else wants to try what cleaning problems you may encounter. Maybe my way of solving them is not the right one, I just needed to do that at some point, for something related to libsecp if I remember correctly. IIRC, it was @laanwj or @theuni who told me about git clean -fdx, but could have been also anybody else in #bitcoin-dev . I can guarantee I didn't discovered -fdx by myself.

laanwj commented at 8:39 AM on March 15, 2016:

I can guarantee I didn't discovered -fdx by myself.

Hah! Using git clean -fdx is kind of black magic, you should first sign a contract in blood before you're allowed to use it. Don't ever recommend anyoen to use it without strong warning. Some may store ancillary development scripts, even wallets in the git tree. Personally I have it aliased to git napalm.

Oh, sorry, actually what we want is to explicitly include files for cleaning, not exclude them. Is there an option for that?

I don't know, I think it could be useful in some cases to add some extra files to delete. On the other hand, I build system changes happen so fast, I somehow doubt whether anyone is going to maintain a list of old files (unless you're volunteering).

laanwj commented at 8:40 AM on March 16, 2016: member

Any volunteers to take over this?

This makes me wonder what the bar should be here for inclusion - I have tons of personal development scripts, but what should be the bar for including them in contrib/devtools?

I house my dev scripts in their own repository, sometimes post them as gist, and only if I think it's helpful to at least one other dev regularly, or are critical to the release process, I include them in the repository itself.

I'm not worried about wasting a few bytes in the repository, but what I want to avoid is accruing unused and unmaintained scripts.

jtimon commented at 9:58 AM on March 16, 2016: contributor

That's why I explain I'm not sure it will be useful for the repo. Anyway, when I'm asking about taking over this, is in the form of the makefile option people were talking about, not another python script.

jtimon commented at 8:15 PM on March 16, 2016: contributor

To me clear, the biggest value for me in getting this merged is adding bin/ to .gitignore, but I can just use an external directory. At the same time, if it has up to 3 names, it seems like tagging/archiving/stashing the binaries for several random branches may actually not be such a bad idea idea after all.

It seems like doing this in the makefile would be a better option (ie make stash), but I'm not going to do that at this point. If someone creates a make-based replacement I'm happy to close this in favor of that. Another option is to close this in favor of an issue to do the same thing using make. Another option is simply closing this and keep using the script privately.

laanwj commented at 11:39 AM on March 17, 2016: member

To me clear, the biggest value for me in getting this merged is adding bin/ to .gitignore, but I can just use an external directory.

Why not use a local gitignore? The ideal git feature to use here would be a repository-local gitignore: See https://help.github.com/articles/ignoring-files/#explicit-repository-excludes

jtimon commented at 12:37 PM on March 17, 2016: contributor

Yeah, thanks. Anyway, if anyone goes with '''make tag" or some other similar thing, I'm happy to test that.

jtimon closed this on Mar 17, 2016

DrahtBot locked this on Sep 8, 2021