guix: Windows build is non-deterministic across build architectures #32923

issue fanquake openend this issue on July 8, 2025
  1. fanquake commented at 1:45 pm on July 8, 2025: member

    This has been the case since https://github.com/bitcoin/bitcoin/commit/8578fabb95face25d9fa7dfaad38d0a2857c2481.

    Guix build on x86_64 & aarch64 @ 01f908195589 (prior merge):

    00324606fdfc525e2acf6ffc5b5327021df6b47de107dbeb51f54162a3dead570  guix-build-01f908195589/output/dist-archive/bitcoin-01f908195589.tar.gz
1e9be0c0c6ab0d856c2176ff0da9c4fb3599f8bd0d578138262fde7307691c28b  guix-build-01f908195589/output/x86_64-w64-mingw32/SHA256SUMS.part
2b27afe43704fa3045f1acebd967f282781f5d1907b48fbc1455c69a7f148ab1a  guix-build-01f908195589/output/x86_64-w64-mingw32/bitcoin-01f908195589-win64-codesigning.tar.gz
3b445befd6dbe5668d9e6f1f2722fba0ed9b01c33c12271b790befb4856b988a9  guix-build-01f908195589/output/x86_64-w64-mingw32/bitcoin-01f908195589-win64-debug.zip
4addb8b7c4ebefeeef5ece568cdc021bc98f5c5e8a57bfac7dd12fc41df487bda  guix-build-01f908195589/output/x86_64-w64-mingw32/bitcoin-01f908195589-win64-setup-unsigned.exe
5472d798ca94c12331b4a0c5f3d7160b11d7b7a7f18776a2edb8220fffe9f9fa9  guix-build-01f908195589/output/x86_64-w64-mingw32/bitcoin-01f908195589-win64-unsigned.zip

    Guix Build @ 8578fabb95face25d9fa7dfaad38d0a2857c2481:

     0# x86_64
 1a993b92027df5cc5ff6ee2355fb4214fe566769917abcedd91b2c4c01814891f  guix-build-8578fabb95fa/output/dist-archive/bitcoin-8578fabb95fa.tar.gz
 226912d42f5228c146ee27a533daf4d111f164a34e77b6ab68a15b25c3f89d37d  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/SHA256SUMS.part
 3756656f917c0025b5b921f3fbd20f0f0815867b85c692c0bdaf6ec139adb88a3  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-codesigning.tar.gz
 49c0b6af68e716767a3ab83c3c22aae4074c26afa0ca01ad9d16ccbe90d219148  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-debug.zip
 5a103dba4ba354ed2f3ca73dfa19e86a525ba40d9d6c7baf67f663623e02449a1  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-setup-unsigned.exe
 6f77d0ef89287a5233c2904e266d3e93f779665cfa3d10586339df053bebdab84  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-unsigned.zip
 7
 8# aarch64
 9a993b92027df5cc5ff6ee2355fb4214fe566769917abcedd91b2c4c01814891f  guix-build-8578fabb95fa/output/dist-archive/bitcoin-8578fabb95fa.tar.gz
10f3bb452cb21520cd31c962a2c54866f0a0176ac2f6e6848ac6926b5c0a7b4131  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/SHA256SUMS.part
1119c98f753129ef55c920ab5dbb93b23de5164d43bd2654533c0cce8783b1406d  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-codesigning.tar.gz
129c0b6af68e716767a3ab83c3c22aae4074c26afa0ca01ad9d16ccbe90d219148  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-debug.zip
1378ace262a902030421b0748252701d4c948f72458ec7f1dddd84492ef620b53d  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-setup-unsigned.exe
148262620ed8029972b991e79e65c446cc0bae7d15f6db8c4f78dc7e0aff173900  guix-build-8578fabb95fa/output/x86_64-w64-mingw32/bitcoin-8578fabb95fa-win64-unsigned.zip

    I originally thought this was somehow introduced in #32837. However this was the difference in the binaries:

     0--- a.txt
 1+++ b.txt
 2@@ -1,9 +1,9 @@
 3 
 4-bitcoin-qt.exe_built_on_x86_64_unstripped:	file format coff-x86-64
 5+bitcoin-qt.exe_built_on_aarch64_unstripped:	file format coff-x86-64
 6 
 7 Disassembly of section .text:
 8 
 9 0000000140b1b070 <_ZN11WalletModel18prepareTransactionER22WalletModelTransactionRKN6wallet12CCoinControlE>:
10 140b1b070: f3 0f 1e fa                 	endbr64
11 140b1b074: 41 57                       	pushq	%r15
12 140b1b076: 41 56                       	pushq	%r14
13@@ -607,15 +607,15 @@
14 140b1bb7d: b8 ff ff ff ff              	movl	$0xffffffff, %eax       # imm = 0xFFFFFFFF
15 140b1bb82: e9 1f fb ff ff              	jmp	0x140b1b6a6 <_ZN11WalletModel18prepareTransactionER22WalletModelTransactionRKN6wallet12CCoinControlE+0x636>
16 140b1bb87: 89 56 1c                    	movl	%edx, 0x1c(%rsi)
17 140b1bb8a: 4c 01 fa                    	addq	%r15, %rdx
18 140b1bb8d: 49 89 d0                    	movq	%rdx, %r8
19 140b1bb90: 31 c0                       	xorl	%eax, %eax
20 140b1bb92: 4d 29 f8                    	subq	%r15, %r8
21-140b1bb95: 4c 39 fa                    	cmpq	%r15, %rdx
22+140b1bb95: 49 39 d7                    	cmpq	%rdx, %r15
23 140b1bb98: 0f 84 68 fc ff ff           	je	0x140b1b806 <_ZN11WalletModel18prepareTransactionER22WalletModelTransactionRKN6wallet12CCoinControlE+0x796>
24 140b1bb9e: 66 90                       	nop
25 140b1bba0: 41 0f b6 14 07              	movzbl	(%r15,%rax), %edx
26 140b1bba5: 88 14 01                    	movb	%dl, (%rcx,%rax)
27 140b1bba8: 48 83 c0 01                 	addq	$0x1, %rax
28 140b1bbac: 49 39 c0                    	cmpq	%rax, %r8
29 140b1bbaf: 75 ef                       	jne	0x140b1bba0 <_ZN11WalletModel18prepareTransactionER22WalletModelTransactionRKN6wallet12CCoinControlE+0xb30>
  2. fanquake added the label Build system on Jul 8, 2025
  3. fanquake added the label Windows on Jul 9, 2025
  4. achow101 commented at 5:55 am on July 10, 2025: member

    #32597 being the cause of this non-determinism doesn’t make any sense to me.

    The function with the non-determinsm is WalletModel::prepareTransaction which is a GUI only function that was not modified by #32597. That PR didn’t change anything remotely near that, so it being the root cause makes no sense.

    addr2line indicates that the problematic line corresponds to https://github.com/bitcoin/bitcoin/blob/a40e9536588c366886de4f4b9d67b8665a509929/src/prevector.h#L174 After unwrapping all of the inlines, the offending caller is https://github.com/bitcoin/bitcoin/blob/a40e9536588c366886de4f4b9d67b8665a509929/src/qt/walletmodel.cpp#L183

    This issue being in prevector is also very weird. This code is called all over place throughout the project, and it’s only in this one particular instance where it is compiled differently?

  5. maflcko commented at 6:47 am on July 10, 2025: member

    The function with the non-determinsm is WalletModel::prepareTransaction which is a GUI only function that was not modified by #32597. That PR didn’t change anything remotely near that, so it being the root cause makes no sense.

    My guess would be that one of the O2 passes (or the linker) is confused by the std::map changes in the header. (wallet.h is included in that gui module)

  6. dstadulis commented at 6:48 am on July 10, 2025: none

    Nice catch!

    tldr: seems there should be no logical difference between the two – but CMIIW:

    Assessing the differences

    Two changes exist between the two segments of machine code:

    1. Registry Extension prefix of the cmpq instruction
    2. The encoding of the operands are switched, in x86 assembly syntax convention.

    Let’s try to decode the logical effect of the changes!

    A consequential, control-flow op code cmpq which sets “condition codes” value(s) in EFLAGS registers. Added after IA32, for the ability to compare quad words on x86-64, a prefix is used to maintain compatability with prior x86 versions.

    Let’s try to decode the logical effect of these two changes!

    Conditional Jumps

    140b1bb98: 0f 84 68 fc ff ff je 0x140b1b806

    0F 84 hex indicates a Jcc “Jump if conditions met” predicate, followed by the bytes 68 fc ff ff i.e. the code doubleword (a signed, relative 32 byte offset – indicated the jump destinaton address). 0F 84, Specifically indicates the je predicate which checks if the zero-flag ZF has been set by the CC values in EFLAGs.

    What sets the zero-flag CC EFLAG value?

    cmpq!

    address: opcode mnemonic

    -140b1bb95: 4c 39 fa cmpq %r15, %rdx +140b1bb95: 49 39 d7 cmpq %rdx, %r15

    Let’s decode the machine code: 0x39 corresponds to the cmp instruction, while 0x4C and 0x49 are the Registry Extension (REX) prefixes. The software that translated machine code to mnemonic attested that the operands on {source, destination} have been flipped (Maybe the source of the build’s discrepancy is a NASM / GAS AT&T syntax difference between the two build environments?)

    Establishing equivalent between machine code and mnemonic: Decoding operand encoding

    Now, dear bit lovers, the time has come to decode the ModR/M byte to determine the operand order and registers involved.

    Documentation specifies that the registers – used as operands to cmp – are specified with two, 3 bit segements of the machine code. Hence if the two 3 bit sequences match, the logical result of of 0x39 FA and 0x39 D7 should be equivalent. Let’s check:

     0def compare_binary_bits(hex_value1, hex_value2):
 1    # Convert hexadecimal to binary and remove '0b' prefix
 2    bin_value1 = bin(int(hex_value1, 16))[2:].zfill(8)
 3    print(f"bin_value1: {bin_value1}")
 4    bin_value2 = bin(int(hex_value2, 16))[2:].zfill(8)
 5    print(f"bin_value2: {bin_value2}")
 6
 7    # Extract last 3 bits and 4th from the last - 7th from the last bits (0-indexed)
 8    last_3_bits1 = bin_value1[-3:]
 9    print(f"last_3_bits1: {last_3_bits1}")
10    middle_bits1 = bin_value1[-6:-3]
11    print(f"middle_bits1: {middle_bits1}")
12
13    last_3_bits2 = bin_value2[-3:]
14    print(f"last_3_bits2: {last_3_bits2}")
15    middle_bits2 = bin_value2[-6:-3]
16    print(f"middle_bits2: {middle_bits2}")
17
18    # Assert transposition of operands / registers
19    assert (last_3_bits2 == middle_bits1) and (last_3_bits1 == middle_bits2), "Bits do not match"
20
21# Test with given hex values
22compare_binary_bits('FA', 'D7')

    Indeed, the operands match – as attested to in the mnemonic. 😮‍💨

    Decoding the Operands’ Prefix:

    Specification:

    In 64-bit mode, the same opcodes represent the instruction prefix REX and are not treated as individual instructions. From: 325383-088-sdm-vol-2abcd.pdf

    Given the operands’ prefixes of

    0x4C => 0b01001100 0x49 => 0b01001001

    And the prefix decoding map…

    The x86-64 specification differences between to the two binaries is found:

    • the binary, compiled on x86-64, has set the R extention which affects the MODRM.reg field.
    • while the binary, compiled on aarch64, has set the B extention which affects either the MODRM.rm field or the SIB.base field.

    IIUC, these REX settings only affect the size of the registers of the source operrands and not the logical result of the specification – which is likely why the difference is elided during machine-code-to-mnemonic translation.

    Reasoning about potential consequences

    Given both prefixes set the REX.W extension, the cmp instruction should run the comparison on the two operands equivalently as cmp function is symetrical / equivalent, with transposed operands. cmp sets the zero-flag in EFLAGS which then je continues on as normal. Thus: there is no logical difference in the result of how the machine code should functioning, according to x86-64 specification ✅

    Intel x86-64 Documentation:

    jcc

    https://c9x.me/x86/html/file_module_x86_id_146.html https://www.felixcloutier.com/x86/jcc

    https://asm-docs.microagi.org/x86/cmp.html https://tizee.github.io/x86_ref_book_web/instruction/cmp.html#compare-two-operands https://c9x.me/x86/html/file_module_x86_id_35.html

    Decoding

    Intel® X86 Encoder Decoder (Intel® XED) is a software library: https://intelxed.github.io decoding 4C39FA and 4939D7

    0$ echo "obase=2; ibase=16; 4C" | bc
11001100
2$ echo "obase=2; ibase=16; 49" | bc
31001001

    radare2

    Color coding mnemonics from associated machine code

    0r2 -a x86 =(echo -en "\x49\x39\xd7")
  7. Sjors commented at 6:57 am on July 10, 2025: member

    The most recent change in prevector.h, and the only one that hasn’t been shipped in a release yet, was a40e9536588c366886de4f4b9d67b8665a509929. But that just removed the reverse iterator, which this doesn’t use afaik.

    You could try initiazing std::vector<CRecipient> vecSend; with the size of recipients, which might result in different code paths. Though even if that restored determinism it wouldn’t fix the underlying issue.

    My own aarch64 builder is broken #32759, so I can’t test it. Though I might just nuke it and try again.

  8. achow101 commented at 7:00 am on July 10, 2025: member

    #32930 seems like it results in a different enough code path to workaround this issue.

    Ultimately, this seems like an upstream compiler bug.


fanquake achow101 maflcko dstadulis Sjors

Labels
Windows Build system

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-07-11 03:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me