gitian: Add OSX build descriptors #4185

This is a set of descriptors for building an OSX DMG without the need for OSX. The descriptors are quite large and unruly, but this seemed like the most reasonable first step towards releasing builds created this way.

These have been ported to monolithic descriptors from the work done in my repo here: https://github.com/theuni/osx-cross-depends/

I've included a Readme that attempts to cover the most obvious questions. After some major changes in the last week, the process is actually rather clean now and contains very few nasty hacks. The exception to that is cctools, which is... gross. See the readme for details.

Unfortunately my macbook completely died yesterday (new one is on the way), so I'm unable to help with testing until next week. Though I've made no major functional changes since then (worked fine as of yesterday), so I'm confident that the resulting installer is in good shape. Much testing is needed, of course.

There are still plenty of warts in the process, and I have plans for addressing them. As discussed with @laanwj, this is intended to be only the first step.

As I said before, this should be split up more (but need not block this "first step"):

• Ideally, the step requiring non-free inputs should be split from the rest of the process.
• Each dependency should produce its own binary tarball (and eventually use its own yml file).

Sidestepping the splitting discussion for now since we discussed it somewhat on IRC last night.

As for the non-free input, there's no way to split it from the rest of the process because every build requires it. If its contents were packaged into the resulting builds, that would mean that every output tarball would end up tainted. Instead, I chose to add it as an input requirement for each step so that there's no risk of output contamination.

With the understanding that it's needed for each dependency (except for dmg/genisoimage i suppose), do you have a way in mind to wall it off further?

Looks good to me, haven't tested yet.

One of the intermediate deps is non-deterministic. He's getting back to this tomorrow.

Note that it isn't of utmost importance that the intermediate deps are deterministic. Requirement for 0.9.2 is just that the end product is deterministic. The rest can be done later.

@laanwj Ack on your comments, will do.

As for the intermediate deps: I agree it's not critical (for .9.2) that they're deterministic. But I'd like to make sure they are eventually. If at some point the resulting dmg isn't deterministic, it's a huge help to be able to compare the intermediates to begin tracking down the reason.

I've addressed @laanwj's comments, I believe. I updated some of the docs to reflect the OSX process, but some of it will need to wait until after merge (checksums likely won't be the same by the time it's merged).

Also fixed the last of (I believe) the determinism issues.

A few comments:

• Could we possibly rename 10cc648683617cca8bcbeae507888099b41b530 c.tar.gz to give an indication of what it is?
• Some of the indentations are off.
• You added mkdir -p inputs; cd inputs/ after pushd ./gitian-builder, but it already appears at the top of the deps section. Following exactly, you'd get gitian-builder/inputs/inputs/.
• The actual mac build part is missing from release-process.md.
• How come this uses tarballs for the intermediate inputs (deps) and not zips like the rest of them? I mean, I assume it doesn't matter in theory, but a) consistency is nice, and b) it gives an easy way to distinguish at a glance between sources (tarballs) and built deps (zips).
• How come protobuf and boost aren't their own descriptors like they are for Windows?
• It would probably be preferable to name the descriptors and intermediate files similar to the current ones: deps-osx.yml, qt-osx.yml, bitcoin-deps-*, etc.
  • Maybe using "osx32" in the file names to match linux/win?

Right now, as of ebcf375e84a90c22ff00de28697eaf32a60ddc43, I'm getting these hashes: (will update as more builds finish)

bfbf001582145b5af296f61dd1e7aa79128309a5239e35c068cc766f0a1d4bef  osx-native-depends-r2.tar.gz
0b8f9d25eb3a38f07fcbd0ed98596443961e97a5cd5ba891b3d968cd07b3fd31  osx-depends-r2.tar.gz
deba5750b02b6f1677b9c0749b25b7ec2b7bb5851ae03b277f5b22c06bbbb3bc  osx-depends-qt-5.2.0-r2.tar.gz

Thanks. I'm headed out of town until Monday. I'll address your notes then. Fwiw: 4ed5663bfcfd1e9525616242c878128069e0a3280ff5245b6be5a2bc5d4e69a6 Bitcoin-Qt.dmg

gitian-osx-qt.yml creates

osx-depends-qt-5.2.0-r2.tar.gz

gitian-osx-bitcoin.yml wants

osx-depends-qt-5.2.1-r2.tar.gz

qt-5.2.0 has a strange character glitch where the decimal point within numbers should be. Building with qt-5.2.1 fixes this issue.

Please see #4189 which makes this PR buildable. For the most part it works except there is a remaining determinism issue and on rare occasions it gets stuck in a configure loop.

@wtogami is the configure loop perhaps related to building Qt? I remember having the same issue in Windows. For Qt, FAKE_TIME interferes with the build process and causes the same to be done time after time, so I had to do without it and create determinism in other ways.

It is intermittently during the Bitcoin-Qt.dmg build, not of qt, if that's what you are asking.

ACK, although folks need to be aware of two issues discovered in #4189.

• Your gitian base image must be clean. Cory decided for 0.9.2 it will not support a build root containing other tools.
• Occasionally the build may get stuck in a perpetual configure loop due to a race. If that happens just try again.

I've bumped qt to 5.2.1 here to fix the build issue and the font issue that @wtogami pointed out. I've asked him to create a separate PR for bumping the win32 version to match. @Michagogo I addressed some of your comments, but have left some alone in the interest of saving time. It's going to take a release process to get the real instructions nailed down, so I'd rather not focus on that just yet. It'd be great if you could PR any fixups you think are necessary after the bulk of these changes go in.

The determinism issues that warren pointed out in #4189 were related to mixing osx/win32/linux environments, which is beyond this scope of this PR. These descriptors work and supplements to address that can come later.

The configure loop is caused by a rare race, and it can be fixed as @laanwj suggested by moving the faketime to a more narrow scope. I'm aware of the issue and will address it soon. It doesn't affect the end-result though, so I don't think it's necessary to hold on the fix.

The last piece in the puzzle is codesigning. We discussed a few possibilities at the conference and I'm working on that now.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/397e9b88ef06db54d905c5bcac7bc081e2a6e9c1 for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

This should be ready for merge after squash.

Successfully built commit a99f9be (the commit on master which this was branched from) using this instructions and descriptor:

adabfb6fdb41bc4757f0ff67e06665519989acc35740e18caed74228d220ac82  Bitcoin-Qt.dmg
0fd3a204a1f0629f5edae842159133bcf12d2558a60d7b0a2611cfaee2f8cba8  bitcoin-res.yml

Merged through 1a97b22