build: add --enable-determinism configure flag

fanquake commented at 7:01 am on February 13, 2020: member

This adds a --enable-determinsm configure flag, which if used, will enable additional compile / link time flags to make subsequent builds of bitcoind deterministic.

The first flag enabled is --no-insert-timestamp. This prevents the linker from embedding timestamps, and makes consecutive builds of bitcoind.exe deterministic. This will likely also be used for Guix Windows builds.

--no-insert-timestamp:

The option –no-insert-timestamp can be used to insert a zero value for the timestamp, this ensuring that binaries produced from identical sources will compare identically.

Diff of consecutive builds of master:

 0--- bitcoind.exe.1
 1+++ bitcoind.exe.2
 2@@ -2,20 +2,20 @@
 3 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
 4 00000070: 6d6f 6465 2e0d 0d0a 2400 0000 0000 0000  mode....$.......
 5-00000080: 5045 0000 6486 1400 57e8 445e 00da 6900  PE..d...W.D^..i.
 6+00000080: 5045 0000 6486 1400 e3e9 445e 00da 6900  PE..d.....D^..i.
 7 00000090: e015 0100 f000 2600 0b02 021f 00de 4900  ......&.......I.
 8 000000a0: 00b0 5b00 008a 0000 e014 0000 0010 0000  ..[.............
 9 000000b0: 0000 4000 0000 0000 0010 0000 0002 0000  ..@.............
10 000000c0: 0400 0000 0000 0000 0500 0200 0000 0000  ................
11-000000d0: 00f0 6a00 0006 0000 bd31 af00 0300 6001  ..j......1....`.
12+000000d0: 00f0 6a00 0006 0000 d434 af00 0300 6001  ..j......4....`.
13 000000e0: 0000 2000 0000 0000 0010 0000 0000 0000  .. .............
14@@ -373594,15 +373594,15 @@
15 005b35f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
16-005b3600: 0000 0000 57e8 445e 0000 0000 7ce1 5b00  ....W.D^....|.[.
17+005b3600: 0000 0000 e2e9 445e 0000 0000 7ce1 5b00  ......D^....|.[.
18 005b3610: 0100 0000 2200 0000 2200 0000 28e0 5b00  ...."..."...(.[.

fanquake added the label Build system on Feb 13, 2020

fanquake added the label Needs gitian build on Feb 13, 2020

practicalswift commented at 7:10 am on February 13, 2020: contributor

Concept ACK: deterministic is better than non-deterministic

Thanks for doing this!

practicalswift commented at 2:14 pm on February 13, 2020: contributor

ACK 73b33c116486f6d052e6ddff2e6e95c7d70addc9 – patch looks correct

fanquake commented at 3:45 am on February 14, 2020: member

Might roll this into an --enable-determinism type build option instead.

DrahtBot commented at 7:53 am on February 14, 2020: member

Gitian builds

File	commit 0c20809da85ac708386fa1ca80f7242917f90761(master)	commit 7657ee6643d90a336fc971355eb145711a46ce1f(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`ba02e2ad0f12fdda...`	`f81ad882b99a2290...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`a632796ad6b48bbe...`	`b9a7a832858e81cc...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`c00a51909a01a49e...`	`36bf3af01812ea6d...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`0176bfb927f79d55...`	`ddcc8e5373cb5484...`
bitcoin-0.19.99-osx-unsigned.dmg	`49a0d8cdd5d04ab5...`	`0272d60588119a88...`
bitcoin-0.19.99-osx64.tar.gz	`d0d87df494449017...`	`fde68b669577ff9f...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`19f571edf6bc3393...`	`91301dbec97ff3d9...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`732d73778a5d58a7...`	`e11593d94d84d855...`
bitcoin-0.19.99-win64-debug.zip	`bca1edb8df891e2b...`	`3de3ba9488007ada...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`9556eee3b4eec400...`	`5d3b932c69bc2560...`
bitcoin-0.19.99-win64.zip	`05d88d65048ae275...`	`7b9346f2fccb94d8...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`2955b63489a3dbff...`	`69e7cce1edffd916...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`d2d7f7c6ac6ab0cd...`	`2a4c2b24f93583c8...`
bitcoin-0.19.99.tar.gz	`d5d44d1204506fa6...`	`52e69fdd6dd9c8e0...`
bitcoin-core-linux-0.20-res.yml	`9b2a3218a8e38d83...`	`d064ff329211c46c...`
bitcoin-core-osx-0.20-res.yml	`4ef8e194ef35b518...`	`f2b0cdb9f43b9eb7...`
bitcoin-core-win-0.20-res.yml	`86e22c0457a236ca...`	`0ee661f3fd690e26...`
linux-build.log	`93d975c20c7af9a9...`	`a6df9fade59471d3...`
osx-build.log	`196af88ce6d3ba83...`	`ae527c4d2157cf38...`
win-build.log	`903e3b3f2dcb7361...`	`26d679c40b7f97d0...`
bitcoin-core-linux-0.20-res.yml.diff		`c4b736da08afab29...`
bitcoin-core-osx-0.20-res.yml.diff		`f4fcd7bdf63f5fdb...`
bitcoin-core-win-0.20-res.yml.diff		`5aeaa9e1e48dde95...`
linux-build.log.diff		`584a960110476668...`
osx-build.log.diff		`6852c48171fa0d88...`
win-build.log.diff		`521e12e99c87fc90...`

DrahtBot removed the label Needs gitian build on Feb 14, 2020

practicalswift commented at 1:04 pm on February 14, 2020: contributor

@fanquake Don’t we want determinism by default? :)

luke-jr commented at 5:12 pm on February 15, 2020: member

Concept NACK, build preferences should be set by the user.

(--enable-determinism seems fair though)

build: add --enable-determinism configure flag

If used, this will enable additional compile / link time flags
that will make subsequent builds of bitcoind determinisitic.

3d9b41ecc0

fanquake force-pushed on Feb 27, 2020

fanquake renamed this:
~~build: add --no-insert-timestamp to Windows linker flags~~
build: add --enable-determinism configure flag
on Feb 27, 2020

fanquake commented at 4:55 am on February 27, 2020: member

I’ve updated this to be an --enable-determinism configure flag instead.

practicalswift commented at 6:51 am on February 27, 2020: contributor

ACK 3d9b41ecc02a69daa19df2bc850d60bf9917b724 – patch looks correct

DrahtBot commented at 7:10 am on February 27, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#12557 ([WIP] 64 bit iOS device support by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

MarcoFalke commented at 1:59 pm on February 27, 2020: member

How is this different from libfaketime, which we use in the gitian builds?

fanquake commented at 0:20 am on February 28, 2020: member

How is this different from libfaketime,

This bypasses the need for libfaketime, and I’d much prefer be able to pass an additional configure flag, than having to install, setup and use it. There are a couple other flags I plan on adding here too, such as passing -D to libtool on macOS, rather than having to set ZERO_AR_DATE.

laanwj commented at 5:54 pm on February 28, 2020: member

Yes, the idea is to not need faketime anymore at some point. It was always a hack because so many of the GNU and Qt tools embedded timestamps in all kind of unexpected places. We’ve slowly been working to get rid of those.

I like putting this under an option instead of making it configure default.

ACK 3d9b41ecc02a69daa19df2bc850d60bf9917b724

laanwj merged this on Feb 28, 2020

laanwj closed this on Feb 28, 2020

sidhujag referenced this in commit c13128fba0 on Feb 28, 2020

MarcoFalke commented at 9:25 pm on February 28, 2020: member

So we can remove libfaketime from gitian as well and replace it with this configure flag and/or appropriate SOURCE_DATE_EPOCH exports?

fanquake deleted the branch on Feb 28, 2020

hebasto commented at 2:33 pm on July 30, 2020: member

@MarcoFalke

So we can remove libfaketime from gitian as well and replace it with this configure flag and/or appropriate SOURCE_DATE_EPOCH exports?

Mind looking into #19627?

sidhujag referenced this in commit ca286d5015 on Nov 10, 2020

MarcoFalke locked this on Feb 15, 2022

build: add –enable-determinism configure flag #18135

Gitian builds

Conflicts