Reproducibility issue with 0.18.0rc1

fanquake commented at 11:05 pm on March 5, 2019: member

Now that gitian sigs are available for 0.18.0rc1, we’ve got at least one set of reproducibly non-deterministic sigs (for the riscv and aarch64 builds):

laanwj:

0b9b5b4e8aca22e55743b028a8d55bf54c0364a61210b472f0d8e19efd8d23896  bitcoin-0.18.0rc1-aarch64-linux-gnu-debug.tar.gz
105192e13f191c076af9dbc0554bb75f2c09084b634ad7323c08d1c124d17c709  bitcoin-0.18.0rc1-aarch64-linux-gnu.tar.gz
20b1b1f4a36c6523cb54cefe0089566d2a5a3b24ad7c5625d00e76fbd0f430aac  bitcoin-0.18.0rc1-riscv64-linux-gnu-debug.tar.gz
32c8ad57b3a0b9a21c920216150475138674eddfd1beca33600814435c2ce8cd2  bitcoin-0.18.0rc1-riscv64-linux-gnu.tar.gz

laanwj tried rebuilding as well as rebuilding after clearing the cache (219), and is still seeing the same results (326).

fanquake, achow101, jonasschnelli, hebasto:

03ee74310d156136b9ac4cd8da67c7ffc738ba3134ab4fcc225781f177721b075  bitcoin-0.18.0rc1-aarch64-linux-gnu-debug.tar.gz
148dcd8bf5ce6d2d9d590f377f70d4ae4cb725a6f76a26d5b5c35c297d9396194  bitcoin-0.18.0rc1-aarch64-linux-gnu.tar.gz
261f3a578a4ff5a5b0b1be5052426d0aa357df6254e105dbc317342c84143522f  bitcoin-0.18.0rc1-riscv64-linux-gnu-debug.tar.gz
3726e923b00c59ce68d84e4ef580cd95758c7e4124e216417c9e9807a33bd4043  bitcoin-0.18.0rc1-riscv64-linux-gnu.tar.gz

achow build results jonas linux build results

fanquake added the label Build system on Mar 5, 2019

fanquake added this to the milestone 0.18.0 on Mar 5, 2019

laanwj commented at 8:40 am on March 6, 2019: member

I’ve also tried rebuilding the base image. Interestingly this gives me a different output for both divergent platforms, but still different from the rest.

It looks like the test_bitcoin is the only diverging file for both platforms:

0Binary files achow101/aarch/bitcoin-0.18.0rc1/bin/test_bitcoin and laanwj/aarch/bitcoin-0.18.0rc1/bin/test_bitcoin differ
1Binary files achow101/riscv/bitcoin-0.18.0rc1/bin/test_bitcoin and laanwj/riscv/bitcoin-0.18.0rc1/bin/test_bitcoin differ

0-rwxr-xr-x 1 user user 13791384 Mar  4 15:55 achow101/aarch/bitcoin-0.18.0rc1/bin/test_bitcoin
1-rwxr-xr-x 1 user user 12640464 Mar  4 15:55 achow101/riscv/bitcoin-0.18.0rc1/bin/test_bitcoin
2-rwxr-xr-x 1 user user 13791272 Mar  4 15:55 laanwj/aarch/bitcoin-0.18.0rc1/bin/test_bitcoin
3-rwxr-xr-x 1 user user 12640360 Mar  4 15:55 laanwj/riscv/bitcoin-0.18.0rc1/bin/test_bitcoin

laanwj commented at 8:46 am on March 6, 2019: member

With my old (initial) build, bitcoin-qt was the one that differed

0Binary files achow101/aarch/bitcoin-0.18.0rc1/bin/bitcoin-qt and laanwj2/aarch/bitcoin-0.18.0rc1/bin/bitcoin-qt differ
1Binary files achow101/riscv/bitcoin-0.18.0rc1/bin/bitcoin-qt and laanwj2/riscv/bitcoin-0.18.0rc1/bin/bitcoin-qt differ

test_bitcoin curiously matches @achow101’s

laanwj commented at 9:29 am on March 6, 2019: member

the difference seems to be minimal: presence/absence of .gnu_debuglink section

 0-achow101/aarch/bitcoin-0.18.0rc1/bin/test_bitcoin:     file format elf64-little
 1+laanwj/aarch/bitcoin-0.18.0rc1/bin/test_bitcoin:     file format elf64-little
 2
 3 Contents of section .interp:
 4  0270 2f6c6962 2f6c642d 6c696e75 782d6161  /lib/ld-linux-aa
 5@@ -861626,6 +861626,3 @@
 6  0000 4743433a 20285562 756e7475 20382e32  GCC: (Ubuntu 8.2
 7  0010 2e302d31 7562756e 7475327e 31382e30  .0-1ubuntu2~18.0
 8  0020 34292038 2e322e30 00                 4) 8.2.0.
 9-Contents of section .gnu_debuglink:
10- 0000 74657374 5f626974 636f696e 2e646267  test_bitcoin.dbg
11- 0010 00000000 bee7a613                    ........

 0-achow101/riscv/bitcoin-0.18.0rc1/bin/test_bitcoin:     file format elf64-little
 1+laanwj/riscv/bitcoin-0.18.0rc1/bin/test_bitcoin:     file format elf64-little
 2
 3 Contents of section .interp:
 4  0270 2f6c6962 2f6c642d 6c696e75 782d7269  /lib/ld-linux-ri
 5@@ -789682,6 +789682,3 @@
 6  0000 4743433a 20285562 756e7475 20382e32  GCC: (Ubuntu 8.2
 7  0010 2e302d31 7562756e 7475327e 31382e30  .0-1ubuntu2~18.0
 8  0020 34292038 2e322e30 00                 4) 8.2.0.
 9-Contents of section .gnu_debuglink:
10- 0000 74657374 5f626974 636f696e 2e646267  test_bitcoin.dbg
11- 0010 00000000 5c067f87                    ....\...

laanwj commented at 9:41 am on March 6, 2019: member

Same for the difference between my own two builds, but for bitcoin-qt !

 0-laanwj/aarch/bitcoin-0.18.0rc1/bin/bitcoin-qt:     file format elf64-little
 1+laanwj2/aarch/bitcoin-0.18.0rc1/bin/bitcoin-qt:     file format elf64-little
 2
 3 Contents of section .interp:
 4  0270 2f6c6962 2f6c642d 6c696e75 782d6161  /lib/ld-linux-aa
 5@@ -2158828,6 +2158828,3 @@
 6  0000 4743433a 20285562 756e7475 20382e32  GCC: (Ubuntu 8.2
 7  0010 2e302d31 7562756e 7475327e 31382e30  .0-1ubuntu2~18.0
 8  0020 34292038 2e322e30 00                 4) 8.2.0.
 9-Contents of section .gnu_debuglink:
10- 0000 62697463 6f696e2d 71742e64 62670000  bitcoin-qt.dbg..
11- 0010 9347b4af                             .G..
12
13-laanwj/riscv/bitcoin-0.18.0rc1/bin/bitcoin-qt:     file format elf64-little
14+laanwj2/riscv/bitcoin-0.18.0rc1/bin/bitcoin-qt:     file format elf64-little
15
16 Contents of section .interp:
17  0270 2f6c6962 2f6c642d 6c696e75 782d7269  /lib/ld-linux-ri
18@@ -1846573,6 +1846573,3 @@
19  0000 4743433a 20285562 756e7475 20382e32  GCC: (Ubuntu 8.2
20  0010 2e302d31 7562756e 7475327e 31382e30  .0-1ubuntu2~18.0
21  0020 34292038 2e322e30 00                 4) 8.2.0.
22-Contents of section .gnu_debuglink:
23- 0000 62697463 6f696e2d 71742e64 62670000  bitcoin-qt.dbg..
24- 0010 ea8590d6                             ....

So the good news is: there’s no divergence in the actual code generation. It looks like a race condition around adding this section? Sometimes it’s added, sometimes it isn’t. This is supposed to happen in contrib/devtools/split-debug.sh.in:

0[@OBJCOPY](/bitcoin-bitcoin/contributor/objcopy/)@ --enable-deterministic-archives -p --add-gnu-debuglink=$3 $2

laanwj commented at 12:19 pm on March 6, 2019: member

I’m working on a patch so that failures in the contrib/devtools/split-debug.sh.in script will cause the build to fail. Currently due to find -exec semantics these go silently ignored (well maybe not silently, it will likely log something, but who digs into that every time).

laanwj commented at 3:53 pm on March 6, 2019: member

Ok that worked, it turns out it was indeed a local issue (it’s strange as there’s 16G free on the disk, but ok… maybe some VM restriction, yes I see the default size for the LXC disk is 10240MB maybe that’s no longer enough):

0+ xargs -0 -n1 '-I{}' ../contrib/devtools/split-debug.sh '{}' '{}' '{}.dbg'
1/usr/bin/aarch64-linux-gnu-objcopy:bitcoin-0.18.0rc1/bin/test_bitcoin.dbg[.debug_info]: No space left on device
2/usr/bin/aarch64-linux-gnu-objcopy:bitcoin-0.18.0rc1/bin/stIxfQNI: cannot fill debug link section `bitcoin-0.18.0rc1/bin/test_bitcoin.dbg': No such file or directory

Will submit a PR that adds proper error handling, so that these kinds of issues won’t be a hidden problem in the future.

laanwj commented at 5:54 pm on March 6, 2019: member

Ok, managed to reproduce the common gitian sigs now in https://github.com/bitcoin-core/gitian.sigs/commit/1b4b22304860476b7151757a25c5ae281bb7b19e

See #15549 for changes that would make this failure more apparent.

laanwj closed this on Mar 6, 2019

laanwj referenced this in commit 12408d33c6 on Mar 9, 2019

PastaPastaPasta referenced this in commit 7e64f9a6fe on Jun 27, 2021

PastaPastaPasta referenced this in commit f8134bc157 on Jun 28, 2021

PastaPastaPasta referenced this in commit 6ae817a5b7 on Jun 29, 2021

PastaPastaPasta referenced this in commit 009543c188 on Jul 1, 2021

PastaPastaPasta referenced this in commit 9a144a5e55 on Jul 1, 2021

PastaPastaPasta referenced this in commit 2a9cc9afdc on Jul 8, 2021

PastaPastaPasta referenced this in commit a33756c72e on Jul 10, 2021

DrahtBot locked this on Dec 16, 2021

Reproducibility issue with 0.18.0rc1 #15541