bitcoin-qt hangs in futex() / __cxa_guard

mbuesch commented at 4:42 pm on September 30, 2018: none

When I run bitcoin-qt it does not properly start up and hangs in futex():

The last calls shown in strace are:

0futex(0x7f3f7bfd77e4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
1futex(0x7f3f7d812c60, FUTEX_WAIT, 65792, NULL

gdb shows:

 0(gdb) bt
 1[#0](/bitcoin-bitcoin/0/)  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 2[#1](/bitcoin-bitcoin/1/)  0x00007ffff5e26d2f in __cxa_guard_acquire () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 3[#2](/bitcoin-bitcoin/2/)  0x00007ffff7e06dcc in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
 4[#3](/bitcoin-bitcoin/3/)  0x00007ffff7e0b706 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
 5[#4](/bitcoin-bitcoin/4/)  0x00007ffff7e2af21 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
 6[#5](/bitcoin-bitcoin/5/)  0x00007ffff7e32080 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
 7[#6](/bitcoin-bitcoin/6/)  0x00007ffff7df1c67 in QSslCertificate::QSslCertificate(QByteArray const&, QSsl::EncodingFormat) ()                                                                                        
 8   from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
 9[#7](/bitcoin-bitcoin/7/)  0x00007ffff7e07fe6 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5
10[#8](/bitcoin-bitcoin/8/)  0x00007ffff7dfb09d in QSslConfiguration::defaultConfiguration() () from /usr/lib/x86_64-linux-gnu/libQt5Network.so.5                                                                      
11[#9](/bitcoin-bitcoin/9/)  0x000055555564841f in main () at qt/bitcoin.cpp:579
12[#10](/bitcoin-bitcoin/10/) 0x00007ffff5a49b17 in __libc_start_main (main=0x555555648380 <main>, argc=1, argv=0x7fffffffdcc8, init=<optimized out>, fini=<optimized out>,                                             
13    rtld_fini=<optimized out>, stack_end=0x7fffffffdcb8) at ../csu/libc-start.c:310
14[#11](/bitcoin-bitcoin/11/) 0x0000555555657dca in _start () at /usr/include/c++/8/ext/new_allocator.h:116

I understand that this might also be a problem in Qt rather than bitcoin-qt. But other Qt applications do not show this problem. So I’d be happy to get any hint to debug this further.

Thanks!

mbuesch commented at 5:03 pm on September 30, 2018: none

This works around the issue:

 0diff --git a/src/qt/bitcoin.cpp b/src/qt/bitcoin.cpp
 1index 1e950e268..5cdcc52c5 100644
 2--- a/src/qt/bitcoin.cpp
 3+++ b/src/qt/bitcoin.cpp
 4@@ -573,7 +573,7 @@ int main(int argc, char *argv[])
 5 #ifdef Q_OS_MAC
 6     QApplication::setAttribute(Qt::AA_DontShowIconsInMenus);
 7 #endif
 8-#if QT_VERSION >= 0x050500
 9+#if QT_VERSION >= 0x050500 && 0
10     // Because of the POODLE attack it is recommended to disable SSLv3 (https://disablessl3.com/),
11     // so set SSL protocols to TLS1.0+.
12     QSslConfiguration sslconf = QSslConfiguration::defaultConfiguration();

With this patch bitcoin-qt does not deadlock anymore and starts properly.

I’m wondering what the correct fix would be. Any help is appreciated. Thanks.

fanquake added the label GUI on Sep 30, 2018

fanquake commented at 10:55 pm on September 30, 2018: member

@mbuesch What OS is this running on? Are you using Qt from depends or installed via a package manager (if so what version)?

Possibly related to #14273.

real-or-random commented at 12:06 pm on October 1, 2018: member

I’m pretty sure that this is exactly #14273. I use Arch Linux and have the same issue, i.e., this is the same OS, same symptoms, same line of code as #14273.

I haven’t had time to verify the workaround, though.

mbuesch commented at 5:19 pm on October 1, 2018: none

Yes, this is indeed a duplicate of #14273. I am running Debian Sid with libqt5network5:amd64 5.11.1+dfsg-9

I can also confirm that the sslLibraryVersionString() call from #14273 fixes the problem. Can we have that change to bitcoin-core? I understand that this is not a bitcoin-core bug, but rather a Qt bug. But the workaround seems simple enough. It might not be an option for people to update Qt easily.

 0diff --git a/src/qt/bitcoin.cpp b/src/qt/bitcoin.cpp
 1index 1e950e268..38e412fcd 100644
 2--- a/src/qt/bitcoin.cpp
 3+++ b/src/qt/bitcoin.cpp
 4@@ -576,6 +576,7 @@ int main(int argc, char *argv[])
 5 #if QT_VERSION >= 0x050500
 6     // Because of the POODLE attack it is recommended to disable SSLv3 (https://disablessl3.com/),
 7     // so set SSL protocols to TLS1.0+.
 8+    QSslSocket::sslLibraryVersionString(); // Call to sslLibraryVersionString is a workaround for a Qt bug. See bitcoin issue [#14273](/bitcoin-bitcoin/14273/) / [#14359](/bitcoin-bitcoin/14359/)
 9     QSslConfiguration sslconf = QSslConfiguration::defaultConfiguration();
10     sslconf.setProtocol(QSsl::TlsV1_0OrLater);
11     QSslConfiguration::setDefaultConfiguration(sslconf);

If that change is acceptable, I’ll send a pull request.

Thanks for your comments.

MarcoFalke commented at 10:57 pm on October 1, 2018: member

I am not sure if this is already filed against qt 5.11, but we should wait for a fix upstream or at least file it against upstream, if not done already.

zquestz commented at 6:45 am on October 3, 2018: none

@mbuesch thanks a bunch for the patch. Worked for me. =)

mbuesch commented at 6:36 pm on October 3, 2018: none

@MarcoFalke I did not find an upstream report/fix, yet. But I can surely report this to Qt. But I think the workaround should be applied to bitcoin-core regardless of that ASAP. We are not really in control of what Qt versions are being used. And people will use that buggy version for some time to come. This bug has already been around for a couple of weeks, so I don’t really see this buggy version to go away tomorrow.

This is a very non-intrusive workaround that I think we should have it in bitcoin-core to get stuff working again. This completely renders bitcoin-qt unusable, after all, if that Qt version is installed on the user’s system.

MarcoFalke commented at 1:10 am on October 4, 2018: member

It would still be nice to know why this workaround works or if there is a more suitable workaround.

real-or-random commented at 8:13 am on October 4, 2018: member

Since this is a concurrency issue, the workaround could be spurious (e.g., just change the timing to avoid the deadlock). At least we don’t know without talking to Qt. But yes, I think we should apply the workaround because it seems to help and it’s really non-intrusive.

real-or-random commented at 9:16 am on October 4, 2018: member

It turns out that #6384 which introduced the code to disable SSLv3 was redundant because this is the default in Qt anyway due to new default flag SecureProtocols introduced in https://github.com/qt/qtbase/commit/3fd2d9eff8c1f948306ee5fbfe364ccded1c4b84, which landed in Qt 5.4.

I guess #6384 was the result of a misunderstanding: @laanwj had opened https://bugreports.qt.io/browse/QTBUG-43168 to ask for a flag to disable SSLv3 two days before Qt 5.4 was tagged, so maybe he missed the new SecureProtocols default in Qt 5.4. But the Qt developers still picked up his suggestions to include more flags for specific SSL/TLS protocol versions and implemented it in Qt 5.5. Then @laanwj created #6384.

At this time, OpenSSL was the only TLS backend in Qt. Now there are more backend, and I checked that they all disable SSLv3 in the default configuration with SecureProtocols.

So I think the simplest way to get rid of this issue here is to remove the code disabling SSLv3, because Qt 5.4 enforces this anyway without any configuration tweaks.

real-or-random referenced this in commit 592f3e4caf on Oct 5, 2018

mbuesch commented at 12:50 pm on October 5, 2018: none

I created a Qt issue: https://bugreports.qt.io/browse/QTBUG-70956

real-or-random commented at 4:30 pm on October 5, 2018: member

Hm, I suspect that is in fact our issue. We create an OpenSSL singleton in util.cpp: The bug disappears when I remove the call to OPENSSL_no_config() here: https://github.com/bitcoin/bitcoin/blob/2c3eade704f63b360926de9e975ce80143781679/src/util.cpp#L115 .

That means that there is probably a race condition in general and we should delay the Qt stuff until the singleton is created.

mbuesch commented at 4:55 pm on October 5, 2018: none

Please also see the full backtrace with debugging symbols I posted here: https://bugreports.qt.io/secure/attachment/77027/full-bt.txt This looks like a recursive locking of Qt’s internal global data structure.

mbuesch commented at 5:05 pm on October 5, 2018: none

@real-or-random Thanks, that was the key. I created a minimal example here: https://bugreports.qt.io/secure/attachment/77028/77028_qttest.tar.gz

MarcoFalke added the label Upstream on Oct 5, 2018

real-or-random commented at 4:22 pm on October 8, 2018: member

Upstream has responded and essentially postponed the issue. They note that this happens only with OpenSSL 1.1.1, not with 1.1.0.

real-or-random referenced this in commit 7d173c4cd1 on Oct 9, 2018

laanwj closed this on Oct 16, 2018

laanwj referenced this in commit 2468471e13 on Oct 16, 2018

laanwj referenced this in commit 0242b5afa4 on Nov 6, 2018

jfhk referenced this in commit 15ba9231eb on Nov 14, 2018

JeremyRubin referenced this in commit c6461409bd on Nov 24, 2018

HashUnlimited referenced this in commit f0d47184d1 on Nov 26, 2018

lateminer referenced this in commit cff76d9eff on Nov 26, 2018

UdjinM6 referenced this in commit f84d5d46d3 on Jun 18, 2019

barrystyle referenced this in commit c5239bb5b0 on Jan 22, 2020

bitbandi referenced this in commit 4683cca29a on Feb 20, 2021

DrahtBot locked this on Sep 8, 2021

bitcoin-qt hangs in futex() / __cxa_guard_acquire() #14359