`-fstack-protector-all` triggers crashes in mingw-w64 5.3.1

laanwj commented at 6:01 pm on September 14, 2016: member

If -fstack-protector-all (part of default hardening flags) is used with the mingw-w64 compiler that is part of Ubuntu 16.04, the tests crash with:

00000000000ac1200 <boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&)>:
100000000007cdbd0 <boost::execution_monitor::execute(boost::function<int ()> const&)>:
200000000007ce1e0 <boost::execution_monitor::vexecute(boost::function<void ()> const&)>:
300000000007ba280 <boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**)>:
400000000004011b0 <__tmainCRTStartup>:
50000000000401514 <.l_start>:

This is before even reaching our code! Also bitcoin-qt crashes at startup if the wallet is enabled. This problem is reproducible both on Windows as on Wine.

The depends system was used to build the dependencies.

There is no such issue with Trusty’s compiler, as used for Travis and the current gitian builds, I don’t know if this is a compiler bug or a bug in how we use the compiler.

Bare -fstack-protectordoes not appear to trigger this issue.

Was initially discussed in #8653.

laanwj added the label Windows on Sep 14, 2016

laanwj added the label Build system on Sep 14, 2016

fanquake commented at 9:10 am on October 8, 2016: member

I’ve consolidated the important info from the original issue here.

To see if I could reproduce I did this:

Create a brand-new Ubuntu Xenial (16.04) VM, from today’s (2015-09-14) cloud image
Install the necessary dependencies, and build requirements from these documents:

0sudo apt-get install g++-mingw-w64-i686 mingw-w64-i686-dev g++-mingw-w64-x86-64 mingw-w64-x86-64-dev curl
1sudo apt-get install build-essential libtool autotools-dev automake pkg-config libssl-dev libevent-dev bsdmainutils

While doing this I indeed saw some logging output about gcc alternatives: I wonder which one was selected by default. Apparently the -win32 variant, not the POSIX one.

Follow the instructions for building for windows 64-bit:

0    cd depends
1    make HOST=x86_64-w64-mingw32 -j4
2    cd ..
3    ./autogen.sh
4    ./configure --prefix=`pwd`/depends/x86_64-w64-mingw32
5    make

It fails with the following error, the same error that you get:

0/usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/bits/unique_ptr.h:49:28: note: declared here
1   template<typename> class auto_ptr;
2                            ^
3httpserver.cpp:69:10: error: ‘mutex’ in namespace ‘std’ does not name a type
4     std::mutex cs;
5          ^
6httpserver.cpp:70:10: error: ‘condition_variable’ in namespace ‘std’ does not name a type
7     std::condition_variable cond;

I changed the alternatives to -posix under sudo update-alternatives --config ... as suggested in this patch.
Cleaned and re-built bitcoin (but not the dependencies - maybe I should), and this time it passed.
Status (Windows 10):

0bitcoin-test.exe: crashes on start
1bitcoin-qt.exe: crashes on start
2bitcoind.exe: seems to work, but throws bad_cast on exit
3bitcoin-cli.exe: seems to work, but cannot connect to server?
4bitcoin-tx.exe: seems to work

Going to clean and re-build the dependencies, as well as bitcoin entirely with the -posix compiler.
This didn’t change anything, still same status as above.

OK I’ve done some research. In principle, std::mutex is supported in the -win32 compiler variant (which we’ve always used, and are using in Trusty). No POSIX emulation layer is needed for c++11 threads. However the problem with <mutex> and friends on newer Ubuntu’s mingw-w64 seems to be the following:

Firstly, there is a /usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/mutex. However the definition of the actual class is guarded by _GLIBCXX_HAS_GTHREADS.
However it requires building glibcxx with a certain library (not to be confused with GLIB’s “libgthread”):
- _GLIBCXX_HAS_GTHREADS is supposed to be defined in /usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/x86_64-w64-mingw32/bits/c++config.h when this has been done

This is an Ubuntu-specific problem: the “gthreads” (GCC threads) library is not linked into libcxx.

On Trusty this is not an issue as gthreads is provided:

/usr/include/c++/4.8/x86_64-w64-mingw32/bits/c++config.h defines _GLIBCXX_HAS_GTHREADS.

There is a https://github.com/meganz/mingw-std-threads which some people use to work around this problem. But in principle I think Ubuntu should solve this upstream.

I have verified that after reverting the C++11 threads changes (https://github.com/laanwj/bitcoin/tree/2016_09_windows_hell), it builds normally with the default -win32 compiler. ~~bitcoin-qt then works, without having to disable hardening.~~ False positive: I was building without wallet, with the wallet it still crashes. What a shit.

However:

bitcoin-test.exe: crashes on start
bitcoin-qt.exe: crashes on start
bitcoind.exe: seems to work
bitcoin-cli.exe: seems to work, but cannot connect to server?
bitcoin-tx.exe: seems to work

The issue with bitcoin-cli looks like https://sourceforge.net/p/levent/bugs/363/ which is linked to mingw (but not its POSIX mode specifically)

This issue doesn’t go away either, unfortunately.

Looks like -fstack-protector-all is the specific hardening flag that exposes problems. Apparently it affects use of BerkeleyDB and boost::test. The tests pass (at least in wine) without --stack-protector-all. It also makes Bitcoin-Qt with wallet start again.

So apparently there are three, at most tangentially related issues regarding Windows:

The std::mutex std::condition_variable etc problem on Ubuntu 15.10+. Can apparently be solved by enabling POSIX compatibility, or by using something like https://github.com/meganz/mingw-std-threads.
-fstack-protector-all issue (-fstack-protector doesn’t have this problem, neither do the other hardening flags like nxcompat) - causes the tests to crash at startup (before even getting to our code in BasicTestingSetup!), as well as bitcoin-qt to crash at startup if the wallet is enabled.
libevent ‘zu’ header issue - causes bitcoin-cli and bitcoind to be unable to connect over RPC (which as mentioned by @sstone, looks like https://sourceforge.net/p/levent/bugs/363/) (fixed in #8730)

I wonder when each of these issues started. My only recommendation for now is: use an Ubuntu Trusty VM for windows cross-compilation

droark commented at 10:36 pm on December 24, 2016: contributor

Hello. Just FYI, I stumbled upon this while tinkering with some stuff. Ubuntu 16.10 (I don’t think anybody has mentioned this version yet) has the same mutex issues when building. Man, what a mess….

moonss777 commented at 2:12 am on June 14, 2017: none

Is there a workaround for the following error? Is there a workaround for the following error? Is there a workaround for the following error?

In file included from support/allocators/secure.h:9:0, from key.h:11, from base58.h:18, from httprpc.cpp:7: ./support/lockedpool.h:195:18: error: ‘mutex’ in namespace ‘std’ does not name a type mutable std::mutex mutex; ^ ./support/lockedpool.h:228:17: error: ‘once_flag’ in namespace ‘std’ does not name a type static std::once_flag init_flag; ^ ./support/lockedpool.h: In static member function ‘static LockedPoolManager& LockedPoolManager::Instance()’: ./support/lockedpool.h:215:9: error: ‘call_once’ is not a member of ‘std’ std::call_once(LockedPoolManager::init_flag, LockedPoolManager::CreateInstance); ^ ./support/lockedpool.h:215:24: error: ‘init_flag’ is not a member of ‘LockedPoolManager’ std::call_once(LockedPoolManager::init_flag, LockedPoolManager::CreateInstance);

eMosbat commented at 12:24 pm on August 21, 2017: none

I have same problem!

laanwj commented at 8:41 am on August 26, 2017: member

I re-tested cross-depends build with the most recent Ubuntu at this time (17.04, Zesty):

Had to select the ‘posix’ variant, to prevent errors about std::mutex. It looks like mingw-w64, as of 6.2.1, still has no std::mutex implementation for win32 threads.

0sudo update-alternatives --config x86_64-w64-mingw32-g++
1sudo update-alternatives --config x86_64-w64-mingw32-gcc

All depends, as well as the whole project built without errors.
test_bitcoin.exe runs successfully with default hardening settings, all tests pass
bitcoin-qt.exe runs successfully, gets synchronizing

So good news: the problem I’ve reported here is gone with newest Ubuntu. I haven’t tried any other versions, but clearly 16.04 (and possible 16.10) was a dud in this regard.

We should update build-windows.md as such, and add the part about POSIX alternatives from #8653 (but not recommend --disable-hardening!).

laanwj commented at 9:03 am on August 26, 2017: member

So TL;DR. Do not use Ubuntu 16.04 for cross-building Bitcoin Core to Windows (reflected in build guide since #11119) use:

14.04 (recommended, gitian does this - no need to set alternatives)
17.04 (as tested above - need to set alternatives to posix)

This will likely never change. Ubuntu doesn’t make major changes to compiler versions after the fact, even with compiler bugs like this. With this, I’m closing project https://github.com/bitcoin/bitcoin/projects/1 (but leaving this issue open, as will remain an issue).

laanwj added the label Upstream on Aug 26, 2017

Bushstar commented at 4:21 pm on September 16, 2017: contributor

I’ve actually got that problem compiling 0.14 on 32-bit on Ubuntu 17.04 running in Linux Subsystem for Windows, the issue was present in 16.04 so I upgraded and switched to gcc and gcc POSIX alternatives. Recompiled deps and run make clean but compiling still gives the following error.

 0In file included from support/allocators/secure.h:9:0,
 1                 from key.h:13,
 2                 from base58.h:21,
 3                 from alert.cpp:11:
 4./support/lockedpool.h:195:18: error: ‘mutex’ in namespace ‘std’ does not name a type
 5     mutable std::mutex mutex;
 6                  ^~~~~
 7./support/lockedpool.h:228:17: error: ‘once_flag’ in namespace ‘std’ does not name a type
 8     static std::once_flag init_flag;
 9                 ^~~~~~~~~
10./support/lockedpool.h: In static member function ‘static LockedPoolManager& LockedPoolManager::Instance()’:
11./support/lockedpool.h:215:9: error: ‘call_once’ is not a member of ‘std’
12         std::call_once(LockedPoolManager::init_flag, LockedPoolManager::CreateInstance);
13         ^~~
14./support/lockedpool.h:215:24: error: ‘init_flag’ is not a member of ‘LockedPoolManager’
15         std::call_once(LockedPoolManager::init_flag, LockedPoolManager::CreateInstance);

To build everything…

0    export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
1    cd depends
2    make HOST=i686-w64-mingw32 V=1 -j4 AUTOCONF=: AUTOHEADER=: AUTOMAKE=: ACLOCAL=:
3    cd ..
4    ./autogen.sh
5    ./configure --prefix=`pwd`/depends/i686-w64-mingw32 --with-miniupnpc --with-qrencode --disable-tests --disable-gui-tests --disable-dependency-tracking --disable-bench
6    make

gcc version output.

0gcc -v
1Using built-in specs.
2COLLECT_GCC=gcc
3COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper
4Target: x86_64-linux-gnu
5Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.3.0-12ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
6Thread model: posix
7gcc version 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2)

I’ll just go use 14.04 for now, I seem to be the odd one out having this with 32-bit.

sipsorcery commented at 12:54 pm on September 29, 2017: member

Had to select the ‘posix’ variant, to prevent errors about std::mutex. It looks like mingw-w64, as of 6.2.1, still has no std::mutex implementation for win32 threads.

It might be worth noting that the mingw-w64 win32 compiler option will probably never work with std::mutex etc, at least not without adding workarounds to the bitcoin source code.

The mingw-w64 win32 option was added to allow the Win32 thread API to be used instead of the winpthread wrapper library. I assume because the performance of using the Win32 threading API directly was significant enough to go to all the hassle of creating two different mingw-w64 cross compilers.

The problem for bitcoin is that the std::mutex etc decalrations have been excluded from the mingw-w64 headers via a pre-processor directive when using the mingw-w64 win32 cross compiler because they conflict with Win32 thread headers.

The fix is simply to set the default mingw-w64 cross compiler to posix as per the command mentioned previously:

sudo update-alternatives –config x86_64-w64-mingw32-g++

It’d be super if that single got added to the build-windows.md file to save a lot of frustration for people building on Windows Subsystem for Linux (which most Windows devs will now be doing since it’s so handy).

laanwj commented at 1:17 pm on September 29, 2017: member

It’d be super if that single got added to the build-windows.md file to save a lot of frustration for people building on Windows Subsystem for Linux (which most Windows devs will now be doing since it’s so handy).

If WFL indeed supports ubuntu 17.04, instructions for that could be added that mention doing that (feel free to do so). Currently the instructions internationally only mention 14.04 because using 16.04 is dangerous, and that used to be the only more recent version available. (and 14.04 doesn’t need the update-alternatives)

fanquake commented at 1:20 pm on September 29, 2017: member

@laanwj I added some notes in #11244 (comment), and can create a PR shortly.

laanwj commented at 1:38 pm on September 29, 2017: member

Where 16.04 is LTS, why is it dangerous?

Because of the stack protector issue with the mingw compiler you’re posting this in (see above). As far as I know, unless you or someone else can confirm otherwise, it still exists with 16.04.

@laanwj I added some notes in #11244 (comment), and can create a PR shortly.

Thanks!

avila-ivan commented at 9:50 pm on October 1, 2017: none

Hi guys

My attempt to compile core failed at the end because of this WSL problem.

As discussed here by the WSL guys, you are forced to use Xenial and there is no option to install an earlier version and they say it has to be resolved from the Bitcoin side. https://github.com/Microsoft/BashOnWindows/issues/2321

If it can’t be fixed from the bitcoin side I would propose that the windows build notes (build-windows.md) get modified to remove the WSL section or add a note until a compatible WSL release is available. https://github.com/bitcoin/bitcoin/blob/master/doc/build-windows.md#compiling-with-windows-subsystem-for-linux That same guide could use additional dependency details like the BerkeleyDB 4.8 dependency for wallets and QT which are present on the build-unix.md file.

Would save some frustration from other newcomers. Thanks!

sipsorcery commented at 11:45 pm on October 1, 2017: member

With a bit of ubuntu package wrangling I’ve been able to successfully build on WSL and an Ubuntu 16.04.3 Server virtual machine (Vmware Workstation Pro 12.5.7). The steps below install the mingw packages from zesty onto xenial. I’m sure this is not recommended/supported so if the instance you’re building on is used for other purposes be aware you might break something.

Out of interest I’m still trying to track down the change related to fstack-protector-all that broke the mingw cross compiled binaries. It’s proving difficult to track down the source of the debian mingw packages. If anybody knows where they can be located please let me know.

WSL & Ubuntu 16.04.3 Server VM (vmware 12.5.7):

$ sudo apt update
$ sudo apt upgrade
$ lsb_release -a Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial
$ sudo apt install build-essential libtool autotools-dev automake pkg-config bsdmainutils curl git
$ sudo add-apt-repository “deb http://archive.ubuntu.com/ubuntu zesty universe” [if not found: sudo apt-get install software-properties-common]
$ sudo apt update
$ sudo apt upgrade
$ sudo wget http://archive.ubuntu.com/ubuntu/pool/main/b/binutils/binutils_2.28-3ubuntu1_amd64.deb
$ sduo dpkg -i binutils_2.28-3ubuntu1_amd64.deb
$ sudo apt install -t zesty binutils-mingw-w64-x86-64 gcc-mingw-w64-x86-64 g++-mingw-w64-x86-64 mingw-w64-x86-64-dev
$ apt search mingw binutils-mingw-w64-x86-64/zesty,now 2.28-1ubuntu1+7.4ubuntu1 amd64 [installed] g++-mingw-w64-x86-64/zesty,now 6.2.1-4ubuntu1+19.2 amd64 [installed] gcc-mingw-w64-base/zesty,now 6.2.1-4ubuntu1+19.2 amd64 [installed,automatic] gcc-mingw-w64-x86-64/zesty,now 6.2.1-4ubuntu1+19.2 amd64 [installed] mingw-w64-common/zesty,now 5.0.1-1 all [installed,automatic] mingw-w64-x86-64-dev/zesty,now 5.0.1-1 all [installed]
$ sudo update-alternatives –config x86_64-w64-mingw32-g++ [select posix option]
$ sudo git clone https://github.com/bitcoin/bitcoin.git
$ chmod -R a+rw /usr/src/bitcoin
$ cd bitcoin/depends
$ Check path using $ echo $PATH, if any windows dirs included with spaces breaks build scripts, set using $ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
$ make HOST=x86_64-w64-mingw32
$ cd ..
$ ./autogen.sh
$ CONFIG_SITE=$PWD/depends/x86_64-w64-mingw32/share/config.site ./configure –prefix=/
$ make
$ sudo make install

laanwj commented at 8:13 am on October 2, 2017: member

If it can’t be fixed from the bitcoin side I would propose that the windows build notes (build-windows.md) get modified to remove the WSL section or add a note until a compatible WSL release is available.

Please file a PR to update the guide, then. It won’t be updated if no one does it, and none of the regular contributors uses windows, it makes sense if it is written by someone that actually uses WSL.

The steps below install the mingw packages from zesty onto xenial. I’m sure this is not recommended/supported so if the instance you’re building on is used for other purposes be aware you might break something. @sipsorcery Thanks for figuring this out!

laanwj referenced this in commit becbd71b0c on Oct 5, 2017

laanwj referenced this in commit 7fbf3c638f on Nov 13, 2017

shereda commented at 6:58 pm on November 29, 2017: none

@sipsorcery Have you made any progress in tracking down the fstack-protector-all (or related) changes in mingw that allowed it to run? I’m a former compiler support person and I might try to track down the issue if you haven’t already. Alternatively if someone has more information or a more complete explanation of what precisely is causing the problem, I’m interested in learning. My concern is that without a deeper understanding of the cause this could crop up again later.

sipsorcery commented at 9:45 pm on November 29, 2017: member

@shereda no I didn’t manage to track down the root cause of the corrupt exe’s from the mingw toolchain.

I did spend some time looking at the difference the Bitcoin Core –enable-hardening configure flag did to the compiler and linker options used in the BTC build:

HARDENED_CPPFLAGS = -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 HARDENED_CXXFLAGS = -Wstack-protector -fstack-protector-all HARDENED_LDFLAGS = -Wl,–dynamicbase -Wl,–nxcompat -Wl,–high-entropy-va

My notes are a little bit vague after that but from memory the options above set the Data Execution Protection (DEP) and Automatic Space Layout Randomisation (ASLR) flags on the Portable Executable binary. I checked the flags on the corrupt binary vs a known good binary and didn’t find a smoking gun.

It turned out the flags could probably be dispensed with for Windows as they are the default options but there could be other things –enable-hardening is doing that I missed.

I started to look into the change log for gcc and mingw64 but then started to go down the rabbit hole a bit and once the packages from Zesty were found to fix the problem I didn’t look into it any further.

I’d certainly still be interested in the root cause if you did manage to find it.

shereda commented at 11:23 pm on November 29, 2017: none

@sipsorcery Ok, thanks for sharing that. I’ll take a look and see if I can get any further with it.

MarcoFalke commented at 10:54 pm on July 18, 2018: member

I believe this can be closes, since Windows cross build are no longer done with xenial.

laanwj closed this on Aug 15, 2018

laanwj commented at 2:31 pm on August 15, 2018: member

Agree, this is not going to ever be solved, but it’s not relevant to us anymore.

codablock referenced this in commit 08513bfffd on Sep 25, 2019

barrystyle referenced this in commit 785278ecd1 on Jan 22, 2020

MarcoFalke locked this on Sep 8, 2021

-fstack-protector-all triggers crashes in mingw-w64 5.3.1 #8732

WSL & Ubuntu 16.04.3 Server VM (vmware 12.5.7):

`-fstack-protector-all` triggers crashes in mingw-w64 5.3.1 #8732