gitian: Make linux build of dependencies deterministic #3622

laanwj commented at 10:11 AM on February 4, 2014: member

OpenSSL was embedding a timestamp causing its build to be non-deterministic. Change deps-linux to be deterministic by using FAKETIME as needed and disabling it when it gets in the way.

Outputs when built w/ KVM (and repeated with LXC), built a few times and they match every time:

Deps:

05fe8e9aef00d295f24a94deef7d3a918af5aeef371ba57fdd5a6acd8c51f6cb  bitcoin-deps-linux32-gitian-r3.zip
4227aa9d9fedbb4265b8d10a4f78b7435f34b00a54eb4d662bf78f59c6e70c27  bitcoin-deps-linux64-gitian-r3.zip

Boost:

149a464f02aa809a340306d13fc0ecb1a347057ca99daf846100e7e3c53d85c2  boost-linux32-1.55.0-gitian-r1.zip
c47bc1a5ffa5097c10280fb2fffb462a925cac5873c4ce7268e404281bddd133  boost-linux64-1.55.0-gitian-r1.zip

Result from building 0.9.0rc1 (15ec451) using the descriptors in this pull:

20b208827262ad4a4afb2419e60bf27c16e96406e3df518f77d56771e22ab13f  bin/32/bitcoin-cli
5d1f2a19bad40fb1c7f6ea0edaddf750820d8d1f5e17ab6071f88df8e174ca34  bin/32/bitcoin-qt
260d5b0cdc10a7b7ea81f5cc51a8fe1533b3ca43e5470c78b830683e7d0ec354  bin/32/bitcoind
557671b4ae41fb4561d83360e9da917a5b29a1c315fcf470d16230e4a5e64548  bin/32/test_bitcoin
ab9b1078841c5d25cb9c05b3baf214e789b1640d5fab535c86191be4cccd0221  bin/32/test_bitcoin-qt
20a93099050df6176afa3a5745b15e54be88831bd7deb91398038d6b57013917  bin/64/bitcoin-cli
685784a29cd9d33b28c23cb1690e1bca4ae820c42c2f3a3dab8592236a10c050  bin/64/bitcoin-qt
fafa628df8411c5b44bc770010ae577377d812548c10ef9e06f8b7f9a916b1e7  bin/64/bitcoind
b62e830e6560fba96075964d406f9948b53fafd6c9009cd98caa1b87c4e87171  bin/64/test_bitcoin
1427bba689501d22e16f2c380d7b45a247afbf84be222f692354eee4a7f5038e  bin/64/test_bitcoin-qt
848875dca917077a5419e876e0ef234cd6b375a55904d08f8706d861eaa57f6b  bin/bitcoin-0.9.0.tar.gz

laanwj commented at 12:51 PM on February 4, 2014: member

~~Shit, I'm getting different (but deterministic) outputs for boost with KVM.~~

[snip]

~~Looks like an ordering difference within the .zip. The files themselves are exactly the same. Getting exactly matching intermediate outputs will be difficult.~~

~~In any case ordering of files within the zip shouldn't affect determinism of the end result.~~

~~But let's see if we can get zip to use an deterministic ordering.~~

Edit: woohoo, got LXC and KVM outputs to match

gavinandresen commented at 3:31 PM on February 4, 2014: contributor

With this pull, my boost builds using VirtualBox do not match:

e63b19db6efc3075a59f8dd152e28dbc602aa4baaea1cf696222a7f0ae1f63ae  boost-linux32-1.55.0-gitian-r1.zip
80ca140ba009221a54c829d5cc2e68c71f5eac638df5866be95079070d4408cb  boost-linux64-1.55.0-gitian-r1.zip

I'll run again and see if I get a self-match.

laanwj commented at 3:48 PM on February 4, 2014: member

Ouch, can you send me one of those files, I'd like to check what the difference is.

gavinandresen commented at 4:02 PM on February 4, 2014: contributor

I do get a self-match rebuilding boost.

Uploaded to https://bitcoincore.org/~gavin/boost-linux32-1.55.0-gitian-r1.zip

My bitcoin-deps checksums don't match yours, either:

840fd30788e64e58d2961b25fd93bfafc035a3ba66faaf2a2d1c1e666c40b459  bitcoin-deps-linux32-gitian-r3.zip
59462e03bb085dceb68c87177f27358dcd7e08929e0872cad4e01ed5c8383911  bitcoin-deps-linux64-gitian-r3.zip

I uploaded those to bitcoincore.org/~gavin/ too.

laanwj commented at 4:29 PM on February 4, 2014: member

The difference seems to concern timezones.

unzip -l on your .zips shows:

   ... 2011-01-30 06:00   lib/libboost_wserialization-mt.a

And on mine:

   ...  2011-01-30 01:00   lib/libboost_wserialization-mt-s.a

All the .ymls export TZ=UTC, but that may not be enough.

gavinandresen commented at 4:33 PM on February 4, 2014: contributor

Ah! Darn timezones... I'll reset my VirtualBox base images so they are UTC and build again after lunch.

laanwj commented at 4:58 PM on February 4, 2014: member

The Boost linux depenendency was actually missing TZ=UTC. Should be better now.

I'm seeing a stranger difference in the bitcoin-deps:

Binary files mine/bitcoin-deps/host/bin/protoc and gavin/bitcoin-deps/host/bin/protoc differ
Only in gavin/bitcoin-deps/include/google/protobuf/io: gzip_stream.h
Binary files mine/bitcoin-deps/lib/libprotobuf.a and gavin/bitcoin-deps/lib/libprotobuf.a differ
diff -dur mine/bitcoin-deps/lib/libprotobuf.la gavin/bitcoin-deps/lib/libprotobuf.la
--- mine/bitcoin-deps/lib/libprotobuf.la    2013-06-01 02:00:00.000000000 +0200
+++ gavin/bitcoin-deps/lib/libprotobuf.la   2013-06-01 02:00:00.000000000 +0200
@@ -17,7 +17,7 @@
 inherited_linker_flags=' -pthread'

 # Libraries that this one depends upon.
-dependency_libs=' -lpthread'
+dependency_libs=' -lpthread -lz'

 # Names of additional weak libraries provided by this library
 weak_library_names=''
diff -dur mine/bitcoin-deps/lib/libprotobuf-lite.la gavin/bitcoin-deps/lib/libprotobuf-lite.la
--- mine/bitcoin-deps/lib/libprotobuf-lite.la   2013-06-01 02:00:00.000000000 +0200
+++ gavin/bitcoin-deps/lib/libprotobuf-lite.la  2013-06-01 02:00:00.000000000 +0200
 @`@ -17,7 +17,7 @@
 inherited_linker_flags=' -pthread'

 # Libraries that this one depends upon.
-dependency_libs=' -lpthread'
+dependency_libs=' -lpthread -lz'

 # Names of additional weak libraries provided by this library
 weak_library_names=''
diff -dur mine/bitcoin-deps/lib/libprotoc.la gavin/bitcoin-deps/lib/libprotoc.la
--- mine/bitcoin-deps/lib/libprotoc.la  2013-06-01 02:00:00.000000000 +0200
+++ gavin/bitcoin-deps/lib/libprotoc.la 2013-06-01 02:00:00.000000000 +0200
...

Your protoc is linking against libz whereas mine is not. Looks like the VMs are set up differently causing different library to be linked. Let's see if there is some protobuf configure option to force use/non-use of libz.

laanwj commented at 5:03 PM on February 4, 2014: member

I added --without--zlib to the build options for protobuf. Hopefully this is enough to equalize the builds.

gavinandresen commented at 7:27 PM on February 4, 2014: contributor

Building with these fixes (and VMs set to UTC) now. FYI: I setup my VirtualBox base image VMs with all the packages needed already installed, so they are not constantly re-downloaded-- that would explain why my protoc was linking against a system zlib.

gavinandresen commented at 10:47 PM on February 4, 2014: contributor

I rebuilt my VirtualBox VMs using vagrant (INSANELY COOL, by the way).

... and they match my old VMs, but not @laanwj

56f6752c9df169f98c3dd970beaed9026dfb0e2beb7bffb0d655a87746eba326  boost-linux32-1.55.0-gitian-r1.zip
1964ae04273e69c06d8ef93caf9a8967e86e6b5b03f008e6bb7f7aa16d29986a  boost-linux64-1.55.0-gitian-r1.zip
840fd30788e64e58d2961b25fd93bfafc035a3ba66faaf2a2d1c1e666c40b459  bitcoin-deps-linux32-gitian-r3.zip
59462e03bb085dceb68c87177f27358dcd7e08929e0872cad4e01ed5c8383911  bitcoin-deps-linux64-gitian-r3.zip

I'll build v0.9.0rc1 with those dependencies; maybe final output will gitian match even if the dependencies don't....

gavinandresen commented at 12:18 AM on February 5, 2014: contributor

Linux gitian build checksums:

20b208827262ad4a4afb2419e60bf27c16e96406e3df518f77d56771e22ab13f  bin/32/bitcoin-cli
9c30a979a2282e8326a2bec06352a3bf67f141c019d833bd1039d5de7e37cdae  bin/32/bitcoin-qt
4c98b5904854b0616938f53dbb682559b4f583ff3aea18353cf86725013c1cc7  bin/32/bitcoind
557671b4ae41fb4561d83360e9da917a5b29a1c315fcf470d16230e4a5e64548  bin/32/test_bitcoin
9c364ebb6ccad2c0944ebc6f07dfc0d7ef5c61a06045d107cb3dd9f477fb8b83  bin/32/test_bitcoin-qt
20a93099050df6176afa3a5745b15e54be88831bd7deb91398038d6b57013917  bin/64/bitcoin-cli
7a5dcc2344558f751499e57412145a286127f2c4a3922618023b6606a28cc5f9  bin/64/bitcoin-qt
6fe77a9610c6b85c550a970e51c65b21bb2b23ba05837f6091cd27840605fb45  bin/64/bitcoind
b62e830e6560fba96075964d406f9948b53fafd6c9009cd98caa1b87c4e87171  bin/64/test_bitcoin
b0470e352c303409135894783cb7cd1f99c79e14b56582218d25dd7f9c7f7587  bin/64/test_bitcoin-qt
1124101cd256f7353a662cdeb1dcdc68761e3d7cf416a49530707860c539da23  src/bitcoin-0.9.0.tar.gz

laanwj commented at 7:32 AM on February 5, 2014: member

Seemingly the TZ=UTC change to boost did change something as boost-linux* do not match your previous builds. But they still don't match. Can you upload the boost files so I can check the difference?

Regarding bitcoin-deps: Oops, i passed the --without-zlib to the wrong configure. Should be better now.

I'll build the final output and see if they match.

laanwj commented at 11:50 AM on February 5, 2014: member

@michagogo can you have a try at re-building linux gitian with this commit? I'm also interested in the intermediate outputs. If they don't match with the OP can you upload them or send them to me?

gavinandresen commented at 12:18 PM on February 5, 2014: contributor

I'm traveling, can't upload or gitian build until I'm back in two days.

laanwj commented at 12:19 PM on February 5, 2014: member

Understood, have a good trip, hopefully someone else can help replicating results.

laanwj commented at 1:52 PM on February 5, 2014: member

Matches @michagogo's result here: http://paste.ubuntu.com/6879066/ APART from src/bitcoin-0.9.0.tar.gz Well at least the binaries are deterministic.

laanwj commented at 5:28 PM on February 5, 2014: member

I've also added steps to make the source .tar.gz deterministic. The output here is:

f3ca4ba934e056d5dedd5946735518baeea48f935f856d2f5579be39df749e0e  src/bitcoin-0.9.0.tar.gz

Edit: also added permission normalization now: --mode='u+rw,go+r-w,a+X' Edit.2: Needed gzip -n to avoid gzip embedding a timestamp

laanwj commented at 2:55 PM on February 6, 2014: member

Ok, the generated .a's should be deterministic now. Umask and user/group should no longer make a difference.

gitian: Make linux build of OpenSSL deterministic

OpenSSL was embedding a timestamp causing its build to be
non-deterministic.
Change deps-linux to be deterministic by using FAKETIME
as needed and disabling it when it gets in the way.

aa9348563c

gitian: Make linux boost dependency completely deterministic

It appears that the output was different every time.

This doesn't affect the final bitcoind/bitcoin-qt, but is confusing
nevertheless.

Fix it by using FAKETIME and zipping files in deterministic order.

aabcd11ba6

gitian: Sort generated source distribution archive

Sort the filenames in the resulting tar, normalize the time/date
and user/group information.

1552145ae5

gitian: add -D flag to ar for deterministic output for linux deps

ar -D: Operate in deterministic mode. When adding files and the archive
index use zero for UIDs, GIDs, timestamps, and use consistent file modes
for all files.  When this option is used, if ar is used with identical
options and identical input files, multiple runs will create identical
output files regardless of the input files' owners, groups, file modes,
or modification times.

c13a13efec

BitcoinPullTester commented at 3:49 PM on February 10, 2014: none

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/c13a13efec0051e1fe84749c36c6676dc6e267af for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

laanwj referenced this in commit d5fa3eff03 on Feb 10, 2014

laanwj merged this on Feb 10, 2014

laanwj closed this on Feb 10, 2014

laanwj deleted the branch on Apr 9, 2014

DrahtBot locked this on Sep 8, 2021