Serious incompatibility problems w/ newer mingw-64 on Ubuntu

sstone commented at 1:23 pm on September 2, 2016: contributor

you may have to select the posix version of gcc-mingw-w64 and g++-mingw-w64 to cross-build the latest code on ubuntu

doc (trivial): add tip for cross-builds on ubuntu

you may have to select the posix version of gcc-mingw-w64 and g++-mingw-w64 to cross-build the latest code on ubuntu

f17d19e2b0

in doc/build-windows.md: in f17d19e2b0 outdated

37@@ -38,5 +38,12 @@ To build executables for Windows 64-bit:
38     ./configure --prefix=`pwd`/depends/x86_64-w64-mingw32
39     make
40 
41+On Ununtu 15.10 and above, if your build fails because the cross-compiler does not seem to support C++11 threads (you get errors such as "httpserver.cpp:69:10: error: ‘mutex’ in namespace ‘std’ does not name a type) you may have to select the posix versions of gcc-mingw-w64 and g++-mingw-w64:

paveljanik commented at 1:45 pm on September 2, 2016:

Ubuntu

fanquake added the label Docs and Output on Sep 2, 2016

fix typo be5b9f47dd

sstone renamed this:
~~doc (trivial): add tip for cross-builds on ubuntu~~
[Trivial] doc: add tip for cross-builds on ubuntu
on Sep 5, 2016

NicolasDorier commented at 9:02 am on September 6, 2016: contributor

tested ACK it solved my issue #8511

sstone commented at 4:06 pm on September 6, 2016: contributor

There might still be a problem with bitcoin-qt.exe which I’m investigating. bitcoind.exe builds and works fine (tested on a VM and several windows boxes). Sorry about that. Update: I can build and run both bitcoin-qt.exe and bitcoind.exe if I use:

0configure --disable-hardening --prefix=`pwd`/depends/x86_64-w64-mingw32

So far I have been able to test on Windows 10 only, so more feedback is welcome

fanquake added the label Windows on Sep 9, 2016

cross-compiling: disable hardening if needed

When cross compiling for Windows on Ubuntu 15.10 and above, you may need to disable hardening if bitcoin-qt.exe fails to start properly

d70597f44d

in doc/build-windows.md: in d70597f44d

43+    sudo update-alternatives --config x86_64-w64-mingw32-gcc
44+    sudo update-alternatives --config x86_64-w64-mingw32-g++
45+
46+And select the posix version of the compiler.
47+
48+You may also need to disable "hardening" of compiled binaries if your bitcoin-qt.exe does not start properly by adding the --disable-hardening option when calling the configure script:

laanwj commented at 8:21 am on September 13, 2016:

What, are you sure about this?

Can you be more specific about this problem? I’d prefer figuring out what the issue is and never telling people to disable hardening. The situation on windows is bad enough as it is (#8248). Note that the gitian executables are built with hardening enabled.

sstone commented at 4:08 pm on September 13, 2016:

Sorry for not being specific enough: with hardening enabled, bitcoint-qt.exe does not start (it crashes on startup), but bitcoind.exe seems to work fine. With hardening disabled, bitcoin-qt.exe starts and seems to work fine AFAICT.

laanwj commented at 9:08 am on September 14, 2016:

Do remove this recommendation. You should never tell anyone in good conscience to disable hardening on a bitcoin wallet. Those security precautions exists with a good reason. Oh absolutely it will seem to “work fine” but you lose any protection against exploitation, if, say, another buffer overflow turns up in UPnP or openssl or any other library.

If it doesn’t work with hardening then you need to debug why it doesn’t work.

sstone commented at 11:28 am on September 14, 2016:

You’re right. I was thinking mostly about people who might need to cross-build quickly for testing purposes and overlooked the fact that people might use the binaries as their real wallet instead of downloading an official release or using gitian to build. I should probably close this PR and open a new specific issue ?

laanwj commented at 8:22 am on September 13, 2016: member

Very strong NACK on telling people to disable hardening, ACK on the posix alternatives.

jonasschnelli commented at 8:47 am on September 13, 2016: contributor

Agree with @laanwj. Also, the disable hardening recommendation comes without a specific reason. Does the posix compatible mingw compiler breaks the Qt build? If so, why exactly?

in doc/build-windows.md: in d70597f44d

45+
46+And select the posix version of the compiler.
47+
48+You may also need to disable "hardening" of compiled binaries if your bitcoin-qt.exe does not start properly by adding the --disable-hardening option when calling the configure script:
49+
50+    ./configure --disable-hardening --prefix=`pwd`/depends/x86_64-w64-mingw32

btcdrak commented at 8:51 am on September 13, 2016:

Why --disable-hardening ?

laanwj commented at 10:39 am on September 14, 2016: member

To see if I could reproduce I did this:

Create a brand-new Ubuntu Xenial (16.04) VM, from today’s (2015-09-14) cloud image
Install the necessary dependencies, and build requirements from these documents:

0sudo apt-get install g++-mingw-w64-i686 mingw-w64-i686-dev g++-mingw-w64-x86-64 mingw-w64-x86-64-dev curl
1sudo apt-get install build-essential libtool autotools-dev automake pkg-config libssl-dev libevent-dev bsdmainutils

While doing this I indeed saw some logging output about gcc alternatives: I wonder which one was selected by default. Apparently the -win32 variant, not the POSIX one.

Follow the instructions for building for windows 64-bit:

0    cd depends
1    make HOST=x86_64-w64-mingw32 -j4
2    cd ..
3    ./autogen.sh
4    ./configure --prefix=`pwd`/depends/x86_64-w64-mingw32
5    make

It fails with the following error, the same error that you get:

0/usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/bits/unique_ptr.h:49:28: note: declared here
1   template<typename> class auto_ptr;
2                            ^
3httpserver.cpp:69:10: error: ‘mutex’ in namespace ‘std’ does not name a type
4     std::mutex cs;
5          ^
6httpserver.cpp:70:10: error: ‘condition_variable’ in namespace ‘std’ does not name a type
7     std::condition_variable cond;

I changed the alternatives to -posix under sudo update-alternatives --config ... as suggested in this patch.
Cleaned and re-built bitcoin (but not the dependencies - maybe I should), and this time it passed.
Status (Windows 10):

0bitcoin-test.exe: crashes on start
1bitcoin-qt.exe: crashes on start
2bitcoind.exe: seems to work, but throws bad_cast on exit
3bitcoin-cli.exe: seems to work, but cannot connect to server?
4bitcoin-tx.exe: seems to work

Going to clean and re-build the dependencies, as well as bitcoin entirely with the -posix compiler.
This didn’t change anything, still same status as above.

laanwj commented at 1:19 pm on September 14, 2016: member

Whoa this even worse than I imagined. Above I already report that bitcoin-cli doesn’t seem to work. Well here’s why. TCPdump output of what comes back from the server:

015:17:02.157987 IP Acer-PC.lan.8332 > vm07.57962: Flags [P.], seq 1:446, ack 151, win 260, options [nop,nop,TS val 10525409 ecr 2673459], length 445
1E...b.@...........z. ..j..m.$|.............
2.....(.3HTTP/1.1 200 OK
3Content-Type: application/json
4Date: Wed, 14 Sep 2016 13:17:06 GMT
5Content-Length: zu
6Connection: close
7
8{"result":{"version":139900,"protocolversion":70014,"blocks":418786,"timeoffset":-25,"connections":8,"proxy":"","difficulty":209453158595.381,"testnet":false,"relayfee":0.00001000,"errors":"This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"},"error":null,"id":1}

Yes, you read that correctly: the content length is ‘zu’. ZU!

Also, bitcoind isn’t stable, it crashed at least once. Unfortunately I didn’t get a useful traceback.

~~I wouldn’t recommend using POSIX emulation mode on Windows as a workaround here. It introduces more problems than it solves~~ (see later, is not related at all)

laanwj renamed this:
~~[Trivial] doc: add tip for cross-builds on ubuntu~~
Serious incompatibility problems w/ newer mingw-64 on Ubuntu
on Sep 14, 2016

sstone commented at 1:53 pm on September 14, 2016: contributor

The issue with bitcoin-cli looks like https://sourceforge.net/p/levent/bugs/363/ which is linked to mingw (but not its POSIX mode specifically)

laanwj commented at 2:42 pm on September 14, 2016: member

OK I’ve done some research. In principle, std::mutex is supported in the -win32 compiler variant (which we’ve always used, and are using in Trusty). No POSIX emulation layer is needed for c++11 threads. However the problem with <mutex> and friends on newer Ubuntu’s mingw-w64 seems to be the following:

Firstly, there is a /usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/mutex. However the definition of the actual class is guarded by _GLIBCXX_HAS_GTHREADS.
However it requires building glibcxx with a certain library (not to be confused with GLIB’s “libgthread”):
- _GLIBCXX_HAS_GTHREADS is supposed to be defined in /usr/lib/gcc/x86_64-w64-mingw32/5.3-win32/include/c++/x86_64-w64-mingw32/bits/c++config.h when this has been done

This is an Ubuntu-specific problem: the “gthreads” (GCC threads) library is not linked into libcxx.

On Trusty this is not an issue as gthreads is provided:

/usr/include/c++/4.8/x86_64-w64-mingw32/bits/c++config.h defines _GLIBCXX_HAS_GTHREADS.

There is a https://github.com/meganz/mingw-std-threads which some people use to work around this problem. But in principle I think Ubuntu should solve this upstream.

laanwj commented at 2:53 pm on September 14, 2016: member

I have verified that after reverting the C++11 threads changes (https://github.com/laanwj/bitcoin/tree/2016_09_windows_hell), it builds normally with the default -win32 compiler. ~~bitcoin-qt then works, without having to disable hardening.~~ False positive: I was building without wallet, with the wallet it still crashes. What a shit.

However:

bitcoin-test.exe: crashes on start
bitcoin-qt.exe: crashes on start
bitcoind.exe: seems to work
bitcoin-cli.exe: seems to work, but cannot connect to server?
bitcoin-tx.exe: seems to work

The issue with bitcoin-cli looks like https://sourceforge.net/p/levent/bugs/363/ which is linked to mingw (but not its POSIX mode specifically)

This issue doesn’t go away either, unfortunately.

sstone commented at 4:02 pm on September 14, 2016: contributor

FWIW, applying the libevent patch above does seem to fix bitcoin-cli.exe

laanwj commented at 4:05 pm on September 14, 2016: member

Looks like -fstack-protector-all is the specific hardening flag that exposes problems. Apparently it affects use of BerkeleyDB and boost::test. The tests pass (at least in wine) without --stack-protector-all. It also makes Bitcoin-Qt with wallet start again.

So apparently there are three, at most tangentially related issues regarding Windows:

The std::mutex std::condition_variable etc problem on Ubuntu 15.10+. Can apparently be solved by enabling POSIX compatibility, or by using something like https://github.com/meganz/mingw-std-threads.
-fstack-protector-all issue (-fstack-protector doesn’t have this problem, neither do the other hardening flags like nxcompat) - causes the tests to crash at startup (before even getting to our code in BasicTestingSetup!), as well as bitcoin-qt to crash at startup if the wallet is enabled.
libevent ‘zu’ header issue - causes bitcoin-cli and bitcoind to be unable to connect over RPC (which as mentioned by @sstone, looks like https://sourceforge.net/p/levent/bugs/363/) (fixed in #8730)

I wonder when each of these issues started. My only recommendation for now is: use an Ubuntu Trusty VM for windows cross-compilation

laanwj commented at 4:13 pm on September 14, 2016: member

FWIW, applying the libevent patch above does seem to fix bitcoin-cli.exe

Interesting. I wonder what happened in the meantime, I just checked and the 0.13.0 executables don’t suffer from this issue. There have been no libevent version changes in depends since then (we’ve always been using 2.0.22). So it must be an issue with the combination of the new compiler and libevent.

I’m a bit afraid that the patch, if we were to apply it to depends, will introduce a similar issue (but backwards) on Trusty instead. But we can try and test.

achow101 commented at 4:14 pm on September 14, 2016: member

@laanwj where can I get a copy of your patches? I can build them in Ubuntu 16.04 and then test them on Windows 10.

laanwj referenced this in commit 64047f8e7f on Sep 14, 2016

theuni commented at 4:27 pm on September 19, 2016: member

FYI, I’m working on a large-scale toolchain refactor for depends so that we have more control over these things. Will report here when it’s ready for testing.

fanquake commented at 9:10 am on October 8, 2016: member

Closing this, I’ve copied all the important info into the #8732

fanquake closed this on Oct 8, 2016

laanwj referenced this in commit becbd71b0c on Oct 5, 2017

laanwj referenced this in commit 7fbf3c638f on Nov 13, 2017

lateminer referenced this in commit 964f326bb1 on Jan 22, 2019

codablock referenced this in commit 08513bfffd on Sep 25, 2019

barrystyle referenced this in commit 785278ecd1 on Jan 22, 2020

sstone deleted the branch on May 28, 2021

DrahtBot locked this on Aug 16, 2022

Serious incompatibility problems w/ newer mingw-64 on Ubuntu #8653