Add dependencies builder for pull-tester, gitian, easy cross-dev #4592

This is the first (main) part of a major rewrite of our build and release process: the dependencies builder. See the README in the depends folder for more detailed info. It unifies many of our build procedures so that they may be re-used for similar purposes. The major points are:

• Pull-tester and Gitian build processes have been unified.
• Pull-tester can test all possible configs on all supported OS’s.
• All dependencies are now built with unified recipes. No more hackish per-target shell scripts. This also means that release binaries will share the exact same dependency versions, rather than a random assortment as they are now.
• Gitian builds are now painless and require no intervention since the results are chained together.
• Dependencies are no longer a local issue for the the pull-tester. Everything can be handled in git alone. Anyone can deploy their own pull-tester if desired.
• Devs may elect to use these dependencies rather than installing them via their OS. This is especially helpful on OSX as it avoids the need for homebrew/macports.

Linux x64 and Mac x64 are supported build environments. Building should work from anywhere, it just hasn’t been a priority to actively work outside of those two yet, so the paths haven't been tested.

Build-side tools may either be installed locally or built as native dependencies. For example, the linux/win builds use locally installed gcc/mingw, but the OSX build builds and installs its own clang-based toolchain. Ideally all local build-side dependencies would be removed, but that can come down the road if it proves necessary.

The dependencies builder creates a cached payload for each package that it builds. Each is automatically rebuilt as needed, either when the build recipe or its dependencies are changed. Old payloads are automatically cleaned. The result is a set of files that are automatically glued together into a prefix suitable for building Bitcoin. This process is entirely automated and deterministic, suitable for running before each Bitcoin build. If nothing has changed, it completes in less than a second.

A POC pull-tester has been setup to use them: https://travis-ci.org/theuni/bitcoin/builds . It caches these packages and re-uses them as necessary (note that caching is not currently happening due to some Travis account restrictions, I’m actively working with the Travis guys on this. Once this is in place builds will only take a few min). The end-result is that Bitcoin is always built with a known set of dependencies, which are entirely controlled by us. No files from the build-machine are ever used. This means that the pull-tester can be OS/distro/architecture agnostic, so we don’t end up in a situation (as we are currently) where the pull-tester is a static environment that can’t be touched.

Gitian has been made to use these dependencies as well. Now, Gitian releases are built with the exact same procedure as the pull-tester, making dogfooding much easier. As mentioned for the pull-tester above, this lifts the OS restriction on Gitian, so we’re free to move to more recent distros/releases. Builds have been verified as deterministic locally, but I haven’t yet compared with non-local builds. Also worth mentioning, because we build all dependencies ourselves in a sandbox, a single base image can be used for all builds.

The new Gitian descriptors have not been included here, they will come in as a separate PR. However, the changes to enable our build-cache have been accepted upstream: https://github.com/devrandom/gitian-builder/pull/62

With this new version, the sources are fetched by the dependencies builder, and cached for re-use. Existing files in the inputs folder will not be used, so the first build will take a good big longer.

TODO: Enable the travis builds. I'm currently working with Travis on this, I hope to have it ready in the next week.

Big NACK here... Especially on creating our own package system. This also makes a major step backwards for gitian, by undoing the split yml files.

If you want a new package manager, please do it outside the main repository. It doesn't have any need to be so closely tied.

Please read the entire description. This is in no way a step backwards. Split yml files are a goal of yours that I have never shared. The package cache negates the need entirely. It will only rebuild the parts that are necessary. If a dep is changed, that dep gets rebuilt (and any anything that depends on it). That is superior to split yml's in every way.

Gitian is a mechanism for regulating the build environment. Using its descriptors as build recipes has always been fundamentally broken.

Like it or not, we have to build the dependencies for releases. For that, some sort of package manager must be used. Until now, it was a hacked together set of scripts with no continuity. Don't pretend that it wasn't a package manager, it was just a really poor one. So it's a false argument to say "we don't need a package manager". Now, it's a coherent system. And since it's a coherent system, it makes sense to share it with other things that care about dependencies.

Decoupling from the main repo would be senseless and self-defeating. In-repo, code in all branches is tied to the process it's built with, as it should be. There's no synchronization to worry about, or outside forces to break it. Automated systems can rely on this process. We'll know right away from the pull-tester if a new feature is incompatible with our release libs, just as we'll know that our released binaries will be functionally identical to what comes out of the pull-tester, just with the non-deterministic bits stripped out.

This fixes lots of real problems in a cohesive and maintainable way. I'm very willing to discuss substantive criticisms, but please be sure to attach actual arguments to them.

Works perfectly here (Ubuntu 14.04).

I agree it makes sense to share it with other systems, but putting it in the main repo accomplishes the opposite of that. Nothing else can practically use this as proposed since it is tightly tied to Bitcoin Core.

@luke-jr By other systems, i meant other Core subsystems. My mistake. Unified deps are useful for building bitcoin for dev (as @sipa probably just tested), for c-i, for release processes, etc. It's tightly tied to Bitcoin Core because it was designed to fit Bitcoin Core's needs - it sacrifices build-speed for determinism, and targets our release goals in terms of static/shared dependencies.

Sharing it outside of Core is uninteresting to me because it's designed to solve Core problems. Other projects could always fork the dir and patch to meet their requirements, but you can bet that they would choose to integrate it into their repositories because that's what makes sense. If there are direct benefits for us having this out-of-repo, please list them.

@theuni The goal of the project is to be modular, splitting Core into multiple independent components. The more we throw in this one repo, the harder that becomes. It also increases the burden on those who are not using and/or don't care about the static binaries. Furthermore, there is no benefit to including a package manager in Core itself. Finally, Bitcoin is more than just Bitcoin Core, and we should be encouraging and making it simple for all Bitcoin projects to use deterministically built binaries - this should not require forking anything.

I'm certainly in favor of modularizing the code. Initially inside the repository, and hopefully eventually into separate repositories once APIs are sufficiently stable.

Still, we'll want some way of easily building the release binaries, which will be built from some predetermined set of sources. It can even be a separate repository in its own right - bitcoin-release for example, which you're free to ignore if you're just using the code as a library, but is very useful for both release building and testing (by guaranteeing that all supported combinations are built and tested). This dependency building system can then move to bitcoin-release, and perhaps the Bitcoin source code can become a 'package' inside it?

My only real requirement is that we remain capable of building gitian binaries when this gets merged.

@luke-jr To your points

• If anything, this makes modularization easier. For example, with this in place, we could use libsecp256k1 and leveldb much more easily as external libraries, because we could be sure of how they were built for release. Same goes for new modules as we continue to break them out.
• This does not increase anyone's burden. The Core build-system remains exactly as-is, this just replaces the way we build our binaries. To repeat myself: we already had a system for this in place, it's just been upgraded.
• See above.
• I don't see how Bitcoin Core could possibly have a goal of helping others build deterministic binaries. That's 100% beyond our scope. If that's your goal, contribute to the upstream tools that break determinism. I've already done this for binutils where it was needed. If there are other tools that need fixes, I encourage you to help fix them. @sipa That's a reasonable suggestion, but it breaks features with (seemingly) no gain. When in-repo, all branches, tags, and commits will have their dependencies and build-process tied to them. This is a huge benefit as it allows for pull-tester to verify them in real-time and for gitian to reproduce them. If out-of-repo, there'd be a constant synchronization game to deal with.

For the sake of discussion, if you're going to advocate moving this stuff out of core, please elaborate on what the actual benefits of that would be. Otherwise, I just can't see how the proper response isn't "meh, just ignore the folder".

@theuni I'm not arguing for that now, just suggesting it as a possible way forward for if/when we're actually at the stage where we want separate repositories.

Releases are code. Binaries are just a convenience for people too lazy to build it. Deterministic binaries is outside the scope of Bitcoin Core in general, which is to be a full node and wallet implementation. If you want to improve the process for deterministic binaries, I agree you should be contributing to the upstream tools (gitian) designed for that, rather than trying to throw an entire package manager and package repository into Bitcoin Core where they don't belong.

Improving Bitcoin Core at the expense of leaving everything else (including other Bitcoin nodes) in the dark for no reason, is not a sensible goal. There is no benefit to embedding more functionality into Bitcoin Core when the functionality could just as well be in its own repository usable to others.

These changes make sense for the gitian project, not Bitcoin Core.

Please stop focusing on Gitian. This work was done primarily to replace the current pull-tester. Gitian can use the results as a side-effect. It would be silly not to. I'll not argue these non-points anymore.

@sipa @luke-jr A true CI system does all the testing and building of binary releases every time a pull is merged. What @theuni is creating here is a proper CI system. What he is doing is absolutely the right thing and very standard in commercial software development. It makes testing painless, building painless. Of course you can still ask others to build to check the final builds and this would make it even easier to deploy.

Once you have true continuous integration, you'll never look back, and anything else will look like the stone age.

@drak Nobody builds every dependency for every test. And if they do, they don't put the build scripts for every dependency in the main code repo.

Testing everything does not mean including everythings' build scripts in every end-project repo.

@drak So in your world (which seems quite different from what I've seen in reality..), every single program's source code has its own unique copy of the build scripts for every library it uses? Why?

@luke-jr In the case of projects that are happy to rely on a given build environment, say the standard packages of a known linux distribution then you would just accept the VM state. In a project that does not trust the dependencies except those which it compiles itself, you recreate your deps. It doesn't sound like you have much experience of CI in the real world. It's very common to have multiple CI projects interacting, each project has it's own CI process, various super projects are triggered when child projects trigger.

CI is simply the automation of everything you would and should do to test your code quality all the way through to the final build. Often those tasks are so onerous and time-consuming, so us humans do them only occasionally. Building and compiling the gitian sources packages is one of those tasks. So in CI you must replicate all the step, which include fetching the sources, compiling them and building the build environment. And you shouldn't care, since the process is automated, unless you are worried about carbon footprint, which frankly is a joke compared to the Bitcoin network's carbon footprint ;)

I don't understand your argument, nor why you would complain about such a monumental leap in progress for Bitcoin. This project has been crying out for decent CI. It will make everyone's life better so in the end, we can spend more time coding and testing features than struggling. With full CI it becomes possible to download the binaries for every single patch made - now anyone, including normal users can test out new GUI features etc (remember the whole watch wallets PR? think how many more people could have participated in the testing if we had proper CI running then?).

Finally, CI means that when Wladamir pushes an RC, the only thing we need to do here to sign it off is compare out gitian builds. If we have a CI system anyone can replicate, verifying builds are in fact as they should be (hashes match) will be a trivial exercise.

@drak That's my point: Bitcoin Core should be the super-project, while dependencies should live outside it in (potentially reused) child projects.

As for "now anyone, including normal users can test out new GUI features etc" and "verifying builds are in fact as they should be (hashes match) will be a trivial exercise", we already have both of these with the pulltester and gitian. This PR in fact makes the latter harder, as every single gitian build needs to recompile everything from scratch, rather than being able to reuse unmodified dependencies as-is.

@luke-jr seriously, read the description before commenting any further. This was written meticulously to avoid rebuilding every dependency for each build. 99% of the time, none will be built. Gitian as well. A full pull-tester build, or Gitian build, will complete in 3-4 minutes. I repeat: Nothing will be rebuilt if it hasn't been modified.

If you're not going to bother reading the description and README, please refrain from commenting. @drak: Thanks for looking at the big picture.

@theuni I don't see that in the code.

@luke-jr The entirety of the work is based on avoiding rebuilding. You've now confirmed that you've neither read the description nor the readme, and that you drew your conclusions based on the file structure alone. As I said earlier, I'm happy to have a spirited debate about the merits or implementation of the work. But I'm not going to argue your preconceived notions, there's nothing to be gained by that. Please just don't be so vocal about them, it only serves to derail the discussion.

If you'd like to give it a go, check it out, hit a 'make' in the depends dir, and wait a while. You'll notice that all builds are now cached in the 'built' folder. Hit make again, and nothing is rebuilt. Now bump a dependency's revision and 'make', and it'll be rebuilt alone (along with anything that depends on it).

Travis understands how to use this cache, once our account is setup correctly. Jenkins could too, or whatever ci is desired. I pasted a link in the summary to the PR that teaches gitian to understand it as well. With this implementation, at any given time, a full Gitian build boils down to a single command that completes in 3-4 min.

Ok, I see the gitian patch. But the other points still stand (although perhaps this PR doesn't make it much worse than it previously was in light of the gitian patch). I suppose things can be improved later on if people are dead-set on having this merged.

The PR that I sent upstream to gitian introduces 2 caches: one for common (shared) artifacts, and one for project-specific artifacts. This means that all projects can share a single generic cache pool, as well as maintaining one of their own.

It's a very primitive patch, and I only meant for it to be a starting point for the discussion. But it does work, and achieves what I set out to do. It implements a (stupidly simple) form of build-chaining. I would love for someone to take the concept and run with it, but I wanted to be sure to have the fundamentals worked out here before starting down that path.

Looks good to me. Worked fine the last time I tried and I really like how it works.

BTW: If we can get this to work on msys, we can throw out the incomplete steps in doc/build-msw.md and make building on windows easy.

How to configure/build with the dependencies that I've built in depends/ ? Are they picked up automatically? I think this should be documented.

@laanwj See the readme here: https://github.com/theuni/bitcoin/blob/depends-travis/depends/README

Real docs are high on my todo, once the major build issues are fixed.

Quick update: See https://travis-ci.org/coryfields/bitcoin/builds/31201853 for most recent run. Temporarily, only Linux is enabled on the current build with caching. Completes in ~2.5 minutes, including ComparisonTool checks

• Caching has been enabled, so builds are very speedy now. I hooked up ccache as well, so the actual 'make' is down to a few seconds.
• As suggested by @sipa, the comparison tool is fetched as a dependency for now.
• Also suggested by @sipa, build revisions have been eliminated in favor of hashing the package recipes and patches themselves.
• Build on OSX works.

All that remains now is a quick fix for OSX packaging, then onto cleanup and documentation.

@theuni Thanks, that was exactly what I was looking for.

A small nit, from that doc

A prefix will be generated that's suitable for plugging into Bitcoin's configure. In the above example, a dir named i686-w64-mingw32 will be created. To use it for Bitcoin:

./configure --prefix=pwd/depends/i686-w64-mingw32

I'm not sure that using '--prefix' for this is the right way here. Prefix is the install prefix of bitcoin itself, which is not necessarily the same as the temporary prefix of the dependencies.

The --prefix thing is an autoconf trick, and I'm afraid there's probably no avoiding it (if you want the convenience of it, anyway). When --prefix is used, it checks $prefix/share/config.site for data that can be used to seed configure. One slightly possible alternative is using a config.cache instead, but I think that would be quite a bit of trouble.

The cost of not using the --prefix trick is having to manually specify all of the variables needed (host, toolchain, boost, qt, protoc, flags, etc). Personally I don't think using --prefix is too bad, since end-user builders are unlikely to ever build this way, so running the binaries from the srcroot is the most likely outcome (if not building with a pull-tester or gitian anyway).

Thanks, it sounds completely sensible now, I had no idea configure looked for a configuration file in the --prefix.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/p4592_1dec09b341f61836147d87656aea7f7be02aab6d/ for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

Updated title and description.

tl;dr: Ready for review and (I believe) usage. I added some docs in depends, please fire away with additional questions and I'll add answers there.

Pulling this in won't change anything, as Travis and Gitian are not yet hooked up. Once merged, I'm hoping to have Travis setup and running along-side the current pull-tester a few days later. After that, I'll PR the overhauled gitian descriptors that take advantage of the dependencies builder.