ci: use POSIX threads for Nix Windows builds #36316

pull willcl-ark wants to merge 3 commits into bitcoin:master from willcl-ark:win-cross-threads changing 3 files +26 −8
  1. willcl-ark commented at 4:18 PM on September 22, 2026: member

    Guix release binaries use GCC’s POSIX thread model, while the pinned Nixpkgs toolchain defaults to mcf. This makes POSIX the default for Nix Windows cross builds. Nightly builds can select mcf or win32 with the threadModel argument.

    The Nixpkgs threads setting selects the model before GCC is built. Only the mcf model pulls in mcfgthreads. For POSIX, GCC’s target libraries need winpthreads headers and libraries during the compiler build. The EXTRA_*_FOR_TARGET flags provide those paths without exposing target headers to GCC’s native build tools, which caused the earlier build failure.

    The compiler wrapper includes winpthreads through extraPackages, so the shell no longer needs to export pthread include and library flags manually. CI also prints GCC’s configured thread model to make mismatches visible.

    Fixes #36314

  2. DrahtBot added the label Tests on Sep 22, 2026
  3. DrahtBot commented at 4:18 PM on September 22, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/36316.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    ACK hebasto

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36271 (Revert #36057 & document mingw-w64 requirement by fanquake)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. willcl-ark commented at 4:18 PM on September 22, 2026: member

    Added a second commit we may want to drop to print/verify first in the CI run.

  5. hebasto commented at 4:22 PM on September 22, 2026: member

    Concept ACK.

    3edcc510370c7fd804060960a0a5e16bbe356c96

    Could we make the thread model configurable? This would help with testing MCF in nightly builds.

  6. DrahtBot added the label CI failed on Sep 22, 2026
  7. DrahtBot commented at 5:28 PM on September 22, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task lint: https://github.com/bitcoin/bitcoin/actions/runs/35753251918/job/106832510905</sub> <sub>LLM reason (✨ experimental): CI failed because the shell lint step (shellcheck) reported SC2016 in ci/test/01_base_install.sh line 55 (expressions inside single quotes).</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  8. willcl-ark force-pushed on Sep 22, 2026
  9. willcl-ark force-pushed on Sep 22, 2026
  10. willcl-ark commented at 10:22 PM on September 22, 2026: member

    Could we make the thread model configurable? This would help with testing MCF in nightly builds.

    I’ve made the thread model configurable through a Nix argument. e.g. to test mcf:

    HOST=x86_64-w64-mingw32ucrt nix-shell contrib/devtools/shell-win64-cross.nix --argstr threadModel mcf
    
  11. DrahtBot removed the label CI failed on Sep 22, 2026
  12. in ci/test/01_base_install.sh:58 in 903d6d3eb9
      51 | @@ -52,7 +52,10 @@ fi
      52 |  
      53 |  if [[ ${HOST:-} == x86_64-w64-mingw32* ]]; then
      54 |    # Install Nix packages.
      55 | -  NIX_BUILD_SHELL=bash nix-shell "${BASE_ROOT_DIR}/contrib/devtools/shell-win64-cross.nix" --run true
      56 | +  NIX_BUILD_SHELL=bash nix-shell "${BASE_ROOT_DIR}/contrib/devtools/shell-win64-cross.nix" --run "
      57 | +    echo 'Windows cross compiler:'
      58 | +    \"\$CXX\" -v
      59 | +  "
    


    maflcko commented at 7:03 AM on September 23, 2026:

    nit: on GHA (or I guess on docker?) This just prints #12 608.5 [output clipped, log limit 2MiB reached]


    hebasto commented at 7:40 AM on September 23, 2026:

    Claude suggests:

    --- a/.github/actions/configure-docker/action.yml
    +++ b/.github/actions/configure-docker/action.yml
    @@ -18,6 +18,10 @@ runs:
         - name: Set up Docker Buildx
           if: ${{ inputs.provider != 'warp' }}
           uses: docker/setup-buildx-action@v4
    +      with:
    +        driver-opts: |
    +          env.BUILDKIT_STEP_LOG_MAX_SIZE=-1
    +          env.BUILDKIT_STEP_LOG_MAX_SPEED=-1
     
         # This is required when using the gha cache backend with a manual docker buildx invocation.
         # Docker will check for variables $ACTIONS_CACHE_URL, $ACTIONS_RESULTS_URL and $ACTIONS_RUNTIME_TOKEN
    

    willcl-ark commented at 7:48 AM on September 23, 2026:

    Do you think it might be wise to keep it bounded, but increase it. Perhaps use 10MB or something?


    maflcko commented at 7:49 AM on September 23, 2026:

    Doesn't have depends an option to print all build system details? If we care about them, we should probably use that instead of a build-only step (which isn't shown after the build)? But not sure how verbose we want it to be.


    hebasto commented at 7:56 AM on September 23, 2026:

    Doesn't have depends an option to print all build system details?

    When using DEBUG=1, I guess.


    willcl-ark commented at 10:00 AM on September 23, 2026:

    OK I've half taken your clanker's suggestion hebasto, and bumped the log limit to 5MiB. This should give us a enough headroom without going unlimited.

    Also, this will only be apparent on PRs which rebuild the compilers (like this one), and not 99% of runs.

    I also moved the compiler identification step into the container run, so it should show up in logs even if we hit 5MiB of logs during the nix shell init/build.


    willcl-ark commented at 11:18 AM on September 23, 2026:

    Do we think this does the job? https://github.com/bitcoin/bitcoin/actions/runs/35846084247/job/107132587688?pr=36316#step:8:18504

     + docker exec --env DANGER_RUN_CI_ON_HOST=1 0fc35186b941b60640b8e132a841499b818a7c3b73966aeeea07da5a231312aa env NIX_BUILD_SHELL=bash nix-shell /home/runner/work/_temp/contrib/devtools/shell-win64-cross.nix --run 'echo '"'"'Windows cross compiler:'"'"' && "$CXX" -v && exec bash /home/runner/work/_temp/ci/test/03_test_script.sh'
    Windows cross compiler:
    Using built-in specs.
    COLLECT_GCC=/nix/store/ys6kn7afpgjk1jd5yklch9j2j9wd1j3b-x86_64-w64-mingw32-gcc-14.3.0/bin/x86_64-w64-mingw32-g++
    COLLECT_LTO_WRAPPER=/nix/store/ys6kn7afpgjk1jd5yklch9j2j9wd1j3b-x86_64-w64-mingw32-gcc-14.3.0/libexec/gcc/x86_64-w64-mingw32/14.3.0/lto-wrapper
    Target: x86_64-w64-mingw32
    Configured with: ../gcc-14.3.0/configure --prefix=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-x86_64-w64-mingw32-gcc-14.3.0 --with-gmp-include=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0-dev/include --with-gmp-lib=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0/lib --with-mpfr-include=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mpfr-4.2.2-dev/include --with-mpfr-lib=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mpfr-4.2.2/lib --with-mpc=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libmpc-1.4.0 --with-native-system-header-dir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mingw-w64-x86_64-w64-mingw32-13.0.0-dev/include --with-build-sysroot=/ --with-gxx-include-dir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-x86_64-w64-mingw32-gcc-14.3.0/include/c++/14.3.0/ --program-prefix=x86_64-w64-mingw32- --enable-lto --disable-libstdcxx-pch --without-included-gettext --with-system-zlib --enable-static --enable-languages=c,c++ --disable-multilib --enable-plugin --with-isl=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-isl-0.20 --with-as=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-x86_64-w64-mingw32-binutils-wrapper-2.46/bin/x86_64-w64-mingw32-as --with-headers=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mingw-w64-x86_64-w64-mingw32-13.0.0-dev/include --enable-__cxa_atexit --enable-long-long --enable-threads=posix --enable-nls --disable-bootstrap --enable-default-pie --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=x86_64-w64-mingw32
    Thread model: posix
    Supported LTO compression algorithms: zlib
    gcc version 14.3.0 (GCC) 
    
  13. willcl-ark marked this as ready for review on Sep 23, 2026
  14. ci: raise BuildKit step log limit
    The Windows cross-toolchain build can exceed BuildKit's default 2 MiB
    per-step log limit, which hides useful output.
    
    Raise the limit to 5 MiB for both CI builder configurations. This
    retains substantially more diagnostics without enabling unlimited logs.
    96d487c11e
  15. willcl-ark force-pushed on Sep 23, 2026
  16. hebasto commented at 11:49 AM on September 23, 2026: member

    Tested c1052aa56ed5b851e9e388fc2b86c0d766acd12c on Fedora 44.

    Could we make the thread model configurable? This would help with testing MCF in nightly builds.

    I’ve made the thread model configurable through a Nix argument. e.g. to test mcf:

    HOST=x86_64-w64-mingw32ucrt nix-shell contrib/devtools/shell-win64-cross.nix --argstr threadModel mcf
    
    $ HOST=x86_64-w64-mingw32ucrt nix-shell contrib/devtools/shell-win64-cross.nix --argstr threadModel mcf
    $ $CXX -v
    Using built-in specs.
    COLLECT_GCC=/nix/store/i67bfgf5xy32wzh6cck9jpxcjkvnq9k5-x86_64-w64-mingw32-gcc-14.3.0/bin/x86_64-w64-mingw32-g++
    COLLECT_LTO_WRAPPER=/nix/store/i67bfgf5xy32wzh6cck9jpxcjkvnq9k5-x86_64-w64-mingw32-gcc-14.3.0/libexec/gcc/x86_64-w64-mingw32/14.3.0/lto-wrapper
    Target: x86_64-w64-mingw32
    Configured with: ../gcc-14.3.0/configure --prefix=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-x86_64-w64-mingw32-gcc-14.3.0 --with-gmp-include=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0-dev/include --with-gmp-lib=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0/lib --with-mpfr-include=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mpfr-4.2.2-dev/include --with-mpfr-lib=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mpfr-4.2.2/lib --with-mpc=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libmpc-1.4.0 --with-native-system-header-dir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mingw-w64-x86_64-w64-mingw32-13.0.0-dev/include --with-build-sysroot=/ --with-gxx-include-dir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-x86_64-w64-mingw32-gcc-14.3.0/include/c++/14.3.0/ --program-prefix=x86_64-w64-mingw32- --enable-lto --disable-libstdcxx-pch --without-included-gettext --with-system-zlib --enable-static --enable-languages=c,c++ --disable-multilib --enable-plugin --with-isl=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-isl-0.20 --with-as=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-x86_64-w64-mingw32-binutils-wrapper-2.46/bin/x86_64-w64-mingw32-as --with-headers=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-mingw-w64-x86_64-w64-mingw32-13.0.0-dev/include --enable-__cxa_atexit --enable-long-long --enable-threads=mcf --enable-nls --disable-bootstrap --enable-default-pie --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=x86_64-w64-mingw32
    Thread model: mcf
    Supported LTO compression algorithms: zlib
    gcc version 14.3.0 (GCC) 
    
    $ gmake -C depends -j 16
    <snip>
    /home/hebasto/dev/bitcoin/depends/work/build/x86_64-w64-mingw32ucrt/zeromq/4.3.5-cbb97251921/src/thread.cpp:13:10: fatal error: pthread.h: No such file or directory
       13 | #include "pthread.h"
          |          ^~~~~~~~~~~
    compilation terminated.
    <snip>
    
  17. willcl-ark commented at 12:22 PM on September 23, 2026: member

    Hmmm, we are starting to veer into quite a lot of extra code needed to support this on-the-fly switching:

    commit 78331095c68b8df4cf34f3b7b761506af6239915
    Author: will <will@256k1.dev>
    Date:   Wed Sep 23 13:06:56 2026 +0100
    
        ci: expose winpthreads for all Windows thread models
        
        ZeroMQ includes pthread.h even when GCC uses the MCF thread model.
        The POSIX-only wrapper exposed winpthreads, so MCF builds failed
        despite GCC itself selecting mcf successfully.
        
        Add winpthreads to every wrapper while retaining mcfgthreads for MCF.
        Keep the target-only GCC build flags specific to POSIX, where GCC needs
        them while building its runtime libraries.
    
    diff --git a/contrib/devtools/shell-win64-cross.nix b/contrib/devtools/shell-win64-cross.nix
    index f61d3e93497..8b22d55ebbf 100644
    --- a/contrib/devtools/shell-win64-cross.nix
    +++ b/contrib/devtools/shell-win64-cross.nix
    @@ -25,20 +25,20 @@ let
       toolchain = crossPkgs.stdenv.cc.targetPrefix;
       pthreads = crossPkgs.windows.pthreads;
       crossGcc = crossPkgs.buildPackages.gcc14;
    -  gcc = if threadModel == "posix" then
    -    crossGcc.override {
    -      cc = crossGcc.cc.overrideAttrs (old: {
    -        # Keep target pthread headers out of GCC's native build tools.
    -        env = old.env // {
    -          EXTRA_FLAGS_FOR_TARGET =
    -            "${old.env.EXTRA_FLAGS_FOR_TARGET} -idirafter ${pthreads}/include -B${pthreads}/lib";
    -          EXTRA_LDFLAGS_FOR_TARGET =
    -            "${old.env.EXTRA_LDFLAGS_FOR_TARGET} -Wl,-L${pthreads}/lib";
    -        };
    -      });
    -      extraPackages = [ pthreads ];
    -    }
    -  else crossGcc;
    +  gcc = crossGcc.override {
    +    cc = if threadModel == "posix" then crossGcc.cc.overrideAttrs (old: {
    +      # Keep target pthread headers out of GCC's native build tools.
    +      env = old.env // {
    +        EXTRA_FLAGS_FOR_TARGET =
    +          "${old.env.EXTRA_FLAGS_FOR_TARGET} -idirafter ${pthreads}/include -B${pthreads}/lib";
    +        EXTRA_LDFLAGS_FOR_TARGET =
    +          "${old.env.EXTRA_LDFLAGS_FOR_TARGET} -Wl,-L${pthreads}/lib";
    +      };
    +    }) else crossGcc.cc;
    +    # ZeroMQ needs pthread.h even when GCC uses a different thread model.
    +    extraPackages =
    +      [ pthreads ] ++ pkgs.lib.optional (threadModel == "mcf") crossPkgs.threads.package;
    +  };
     in
     
     pkgs.mkShellNoCC {
    

    Are we sure we want' need this?

  18. hebasto commented at 1:26 PM on September 23, 2026: member

    Are we sure we want' need this?

    We can drop non-posix thread model support altogether. I can use customized scripts in the nightly builds.

  19. ci: use POSIX threads in Nix Windows cross shell
    Guix release binaries use the POSIX GCC thread model, but the pinned
    Nixpkgs cross compiler defaults to MCF. Select POSIX through the
    Nixpkgs threads hook so GCC and its target libraries agree.
    
    Leave threads.package null because making winpthreads a target
    dependency exposes its headers to native GCC build tools. Supply its
    headers and libraries through EXTRA_*_FOR_TARGET for target libraries
    instead.
    
    Add winpthreads to the compiler wrapper so builds in the shell get
    its include and library paths without manual NIX_CFLAGS_COMPILE or
    NIX_LDFLAGS exports.
    1d59d5ad25
  20. ci: print Windows cross compiler thread model
    BuildKit can truncate the compiler build log, hiding its configure
    flags. Print the compiler version and thread model when CI enters the
    Nix test shell so a mismatch is visible in the job output.
    9fd18aedb0
  21. willcl-ark force-pushed on Sep 23, 2026
  22. willcl-ark commented at 3:13 PM on September 23, 2026: member

    We can drop non-posix thread model support altogether. I can use customized scripts in the nightly builds.

    Dropped in the latest push.

  23. hebasto approved
  24. hebasto commented at 4:01 PM on September 23, 2026: member

    ACK 9fd18aedb088b5ca6a7c424aab2a4a1ec3ef4eeb.

  25. fanquake commented at 11:16 AM on September 24, 2026: member

    Need to backport this to 32.x as well.

  26. fanquake merged this on Sep 24, 2026
  27. fanquake closed this on Sep 24, 2026

  28. fanquake referenced this in commit c65060e6d3 on Sep 24, 2026
  29. fanquake referenced this in commit 6854ead11c on Sep 24, 2026
  30. fanquake referenced this in commit d06f51300e on Sep 24, 2026
  31. fanquake commented at 11:25 AM on September 24, 2026: member

    Backported to 32.x in #36300.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-10-14 03:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me