guix: Linux and macOS builds are not cross-arch reproducible with powerpc64le build arch

JeremyRand commented at 3:46 pm on November 3, 2024: contributor

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

The Guix output hashes for Windows and the source tarball match between my powerpc64le build machine and the official binaries, but the hashes for macOS and Linux do not match.

This seems to be a partial regression from 27.1, where Windows, Linux, and the source tarball all matched for me. macOS did not even build without errors for 27.1 on powerpc64le (that is at least fixed in 28.0), so I have nothing to compare there.

It is not clear to me whether the root cause might be the same as the converse problem referenced in #27897 (powerpc64le hashes didn’t match between x86_64 and aarch64 build machines).

(Maybe I’ve done something stupid here, but I definitely can’t tell what I would have done wrong, and the different behavior from 27.1 certainly seems like I’ve hit a real bug.)

Expected behaviour

All Guix output hashes should match between my powerpc64le build machine and the official binaries.

Steps to reproduce

Build Bitcoin Core via ./contrib/guix/guix-build
Compare results against the guix-sigs repo (I used pinheadmz’s Guix sigs as a reference).

Relevant log output

Hashes from my powerpc64le build machine:

 03908cbe1339fc10a220e9634629b6937dba77de926bf8dd618283fd090c61f5e  bitcoin-28.0-aarch64-linux-gnu-debug.tar.gz
 11e58898a1b48296fd7e31ddf96c248a0a45c07709d980e3dadfd66ab9e5d2845  bitcoin-28.0-aarch64-linux-gnu.tar.gz
 2dbbb90b05c2e390ba6130cac074135a52c83dcc762a28257de1abfe89d69675f  bitcoin-28.0-arm-linux-gnueabihf-debug.tar.gz
 354a30faa5a61f628d890f464c48def63858eca9b655873f1e60c4e220af41c2e  bitcoin-28.0-arm-linux-gnueabihf.tar.gz
 4da85a7758d272ca5ea345a2b549147acfbbdf61c556c98dc4f49e8f7a27af30d  bitcoin-28.0-arm64-apple-darwin-unsigned.tar.gz
 5c24ff655b0cbbdc116b19474fa621b1c803e3af25f3e540ea10d53a9ba9c4a2d  bitcoin-28.0-arm64-apple-darwin-unsigned.zip
 617033ff17cb91c09cf27fefb42801a584844385b2c6c50902ae63cebb7fc632e  bitcoin-28.0-arm64-apple-darwin.tar.gz
 7700ae2d1e204602eb07f2779a6e6669893bc96c0dca290593f80ff8e102ff37f  bitcoin-28.0.tar.gz
 8a182b39beb4603c3cad55f9f45eaae730eaa1a8c5ee26718707a35554d28654a  bitcoin-28.0-powerpc64-linux-gnu-debug.tar.gz
 9252a1e89b369ce1895a58ae4425c765e6c654214550e14245f6bdb84ec161c2b  bitcoin-28.0-powerpc64-linux-gnu.tar.gz
10cf4cbfa876142bfdb4c781271cd9a1a5410e6cb6e7f0e3996eb67bf08909d4ef  bitcoin-28.0-riscv64-linux-gnu-debug.tar.gz
11fd91dee4518444891549d41f995ef007ae81a426cc5914a4767f1d4410eb1103  bitcoin-28.0-riscv64-linux-gnu.tar.gz
12a98b0fdddcd71fdc16d29d3c4cc95baf408cc2210e23aa96fa1b808267a25387  bitcoin-28.0-x86_64-apple-darwin-unsigned.tar.gz
13898f790d4941e106158ba2c38f8cb439a6e1e4f58cad8c9cd86dd0ddff08d117  bitcoin-28.0-x86_64-apple-darwin-unsigned.zip
14d2c74eebfa6ac1e7ce64b986cbbd28e4838bd8abf2de70493d971b373d559044  bitcoin-28.0-x86_64-apple-darwin.tar.gz
153b5808f3070e88b5f6eee25678d80f4159827ea391abb0ee25ba96af2a598f75  bitcoin-28.0-x86_64-linux-gnu-debug.tar.gz
16ffa78c8e87ea20f13780e780589351484fd22defdabb6aa3234d48f9518104a1  bitcoin-28.0-x86_64-linux-gnu.tar.gz
178990def2e611323d4c7a8cf17187a138dca64f98fc0ecebda0a3e999dbdd083d  bitcoin-28.0-win64-debug.zip
18d8170c342ac049fab953f87841cbbba6c0e3f277703ddc29c678b6ab93dae966  bitcoin-28.0-win64-setup-unsigned.exe
198ec39e7bf66ea419ea79e5f1b7bee1b03a28b51ddd1daa6e167bff6abac0a5d2  bitcoin-28.0-win64-unsigned.tar.gz
2085282f4ec1bcb0cfe8db0f195e8e0f6fb77cfbe89242a81fff2bc2e9292f7acf  bitcoin-28.0-win64.zip

For comparison, see https://github.com/bitcoin-core/guix.sigs/blob/main/28.0/pinheadmz/noncodesigned.SHA256SUMS

I would be happy to upload my binaries and/or run diffoscope on request. I would also be happy to run Guix builds of additional commits on my powerpc64le machine if there are commits that seem likely to be the culprit.

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v28.0

Operating system and version

Debian 12.7 running inside KVM; host is also Debian 12.7

Machine specifications

powerpc64le; HDD; wired network on residential Internet connection.

maflcko commented at 7:52 am on November 4, 2024: member

I would be happy to upload my binaries and/or run diffoscope on request. I would also be happy to run Guix builds of additional commits on my powerpc64le machine if there are commits that seem likely to be the culprit.

I’d say all of that would be useful. You link to the GCC bump to 12, which seems one of the more significant changes to the Linux build, so testing commit e1ce5b8ae9124717c00dca71a5c5b43a7f5ad177 (and e1ce5b8ae9124717c00dca71a5c5b43a7f5ad177~1), or otherwise bisecting could be useful as well.

maflcko added the label Build system on Nov 4, 2024

laanwj commented at 11:07 am on November 14, 2024: member

I would be happy to upload my binaries

Yes, please do. i’m happy to try to help find the difference but i don’t have access to a ppc64 system.

fanquake commented at 3:21 pm on November 14, 2024: member

Last I was looking at this, if you guix built (master) for powerpc64le-linux-gnu, once on x86_64 and once on aarch64, and compared the resulting binaries, you’d end up with (a small number) of differences where two instructions are swapped. For example:

 0--- a/x86_64.txt
 1+++ b/aarch64.txt
 2@@ -176,8 +176,8 @@
 3    24ea8:      78 1b 77 7c     mr      r23,r3
 4    24eac:      66 01 17 7c     mtfprd  f0,r23
 5    24eb0:      14 f2 37 7d     add     r9,r23,r30
 6-   24eb4:      d0 00 b8 3a     addi    r21,r24,208
 7-   24eb8:      10 00 40 39     li      r10,16
 8+   24eb4:      10 00 40 39     li      r10,16
 9+   24eb8:      d0 00 b8 3a     addi    r21,r24,208
10    24ebc:      00 00 00 60     nop
11    24ec0:      10 8e 22 f9     std     r9,-29168(r2)
12    24ec4:      50 00 00 f0     xxspltd vs0,vs0,0

In this case, in:

00000000000024bf0 <_GLOBAL__sub_I_descriptor.cpp>:
1   24bf0:	4e 00 4c 3c 	addis   r2,r12,78
2   24bf4:	10 32 42 38 	addi    r2,r2,12816
3   24bf8:	a6 02 08 7c 	mflr    r0
4<snip>
5   24eb0:	14 f2 37 7d 	add     r9,r23,r30
6   24eb4:	d0 00 b8 3a 	addi    r21,r24,208
7   24eb8:	10 00 40 39 	li      r10,16
8   24ebc:	00 00 00 60 	nop

laanwj commented at 5:12 pm on November 14, 2024: member

Last I was looking at this, if you guix built (master) for powerpc64le-linux-gnu, once on x86_64 and once on aarch64

It might be, but it’s unclear to me if that is the same issue, this OP talks about building on PPC64.

(that’s a pretty awful issue btw! apparently the optimizer is non-deterministic?)

JeremyRand commented at 2:35 am on November 16, 2024: contributor

I would be happy to upload my binaries

Yes, please do. i’m happy to try to help find the difference but i don’t have access to a ppc64 system.

I’ve uploaded my non-debug binaries here:

https://ynqrnzvuotdgvqj2g73f5tyj4rcvwutdemu23jgadawihbkc3lxkbwid.onion/bitcoin-core/28.0-guix-binaries-built-on-powerpc64le/

If the debug variants would be helpful, I don’t mind uploading them too. I don’t have any plans to delete these files or decommission this server, but it’s personal hobbyist infrastructure, so I cannot make any guarantees about uptime or permanence.

laanwj assigned laanwj on Nov 25, 2024

laanwj commented at 12:48 pm on November 25, 2024: member

i’ve compared against the binaries from bitcoincore.org and looked at the ELF metadata and disassembly and it’s a significant difference, not just a few different instructions. The entire .text section ends up with a different size, which shifts the offsets, making it very hard to determine what the source of the discrepancy is.

Debug symbols could help but ideally i’d need a linker map.

Can you please build the following branch: https://github.com/laanwj/bitcoin/tree/2024-11-linker-map-28.0 — it is v28.0 with a patch to generate .map files. And then upload the .maps for x86_64-linux-gnu (i think this is enough).

guix: Linux and macOS builds are not cross-arch reproducible with powerpc64le build arch #31207