bitcoind aborts with boost exception in WSL guest

ashishchandr70 commented at 8:53 pm on September 11, 2018: none

Running bitcoind results in a Boost related exception and bitcoind aborts.

bitcoind to start and wait for further input

Here is the output of running the bitcoind command

 0$ bitcoind -printtoconsole
 1
 22018-09-11 20:39:22
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
222018-09-11 20:39:22 Bitcoin Core version v0.16.2 (release build)
232018-09-11 20:39:22 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
242018-09-11 20:39:22 Assuming ancestors of block 0000000000000000005214481d2d96f898e3d5416e43359c145944a909d242e0 have valid signatures.
252018-09-11 20:39:22 Setting nMinimumChainWork=000000000000000000000000000000000000000000f91c579d57cad4bc5278cc
262018-09-11 20:39:22 Using the 'sse4' SHA256 implementation
272018-09-11 20:39:22 Using RdRand as an additional entropy source
282018-09-11 20:39:22 Default data directory /home/achandra/.bitcoin
292018-09-11 20:39:22 Using data directory /home/achandra/.bitcoin
302018-09-11 20:39:22 Using config file /home/achandra/.bitcoin/bitcoin.conf
312018-09-11 20:39:22 Using at most 125 automatic connections (1024 file descriptors available)
322018-09-11 20:39:22 Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
332018-09-11 20:39:22 Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
342018-09-11 20:39:22 Using 2 threads for script verification
352018-09-11 20:39:22 scheduler thread start
362018-09-11 20:39:22 HTTP: creating work queue of depth 16
372018-09-11 20:39:22 No rpcpassword set - using random cookie authentication
382018-09-11 20:39:22 Generated RPC authentication cookie /home/achandra/.bitcoin/.cookie
392018-09-11 20:39:22 HTTP: starting 4 worker threads
402018-09-11 20:39:22 Using wallet directory /home/achandra/.bitcoin/wallets
412018-09-11 20:39:22 init message: Verifying wallet(s)...
422018-09-11 20:39:22 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
432018-09-11 20:39:22 Using wallet wallet.dat
442018-09-11 20:39:22 CDBEnv::Open: LogDir=/home/achandra/.bitcoin/wallets/database ErrorFile=/home/achandra/.bitcoin/wallets/db.log
452018-09-11 20:39:22 Cache configuration:
462018-09-11 20:39:22
47
48************************
49EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_15condition_errorEEEEE
50boost::condition_variable::do_wait_until failed in pthread_cond_timedwait: Invalid argument
51bitcoin in scheduler
52
532018-09-11 20:39:22 * Using 2.0MiB for block index database
54
55
56************************
57EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_15condition_errorEEEEE
58boost::condition_variable::do_wait_until failed in pthread_cond_timedwait: Invalid argument
59bitcoin in scheduler
60
612018-09-11 20:39:22 * Using 8.0MiB for chain state database
62terminate called after throwing an instance of '2018-09-11 20:39:22 * Using 440.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
63boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::condition_error> >2018-09-11 20:39:22 init message: Loading block index...
64'
652018-09-11 20:39:22 Opening LevelDB in /home/achandra/.bitcoin/blocks/index
66  what():  boost::condition_variable::do_wait_until failed in pthread_cond_timedwait: Invalid argument
67Aborted (core dumped)

Can produce it very reliably. Just keep running the same command i.e. bitcoind -printtoconsole

Ubuntu 16.04 running on Windows 10. Intel i5 processor and 8 GB RAM. Processor: Intel(R) Core(TM) i5-3340M CPU @ 2.70GHz, 2701 Mhz, 2 Core(s), 4 Logical Processor(s)

laanwj added the label Bug on Sep 13, 2018

laanwj commented at 8:52 am on September 13, 2018: member

~~this is strange; from the manpage of pthread_cond_timedwait:~~ (ok see below, apparently I have a linuxthreads-specific man page, not the POSIX one)

0       The pthread_cond_timedwait function returns the following error codes on error:
1              ETIMEDOUT
2                     the condition variable was not signaled until the timeout specified by abstime
3              EINTR  pthread_cond_timedwait was interrupted by a signal

It’s not supposed to ever return EINVAL (“invalid argument”).

As you say you’re running this on windows, my guess is that it has something to do with the WFL, emulating a POSIX function incorrectly.

Edit: if I have to guess, one possibility would be that pthread_cond_timedwait returns EINVAL on windows if the time is already in the past. This would be hard to work around without creating race conditions. It’s also possible that either the cond or mutex argument is considered invalid, but this is wrapped in so many layers of boost:: that I can’t see how that could happen.

luke-jr commented at 8:58 am on September 13, 2018: member

Is this a WFL build or a MingW build?

laanwj commented at 8:59 am on September 13, 2018: member

Well,

Ubuntu 16.04 running on Windows 10

that seriously suggest WFL, not MingW

luke-jr commented at 8:59 am on September 13, 2018: member

FWIW, my manpage has quite a bit more in possible errors:

 0       ENOTRECOVERABLE
 1              The state protected by the mutex is not recoverable.
 2
 3       EOWNERDEAD
 4              The mutex is a robust mutex and the process containing the previous owning thread terminated while hold‐
 5              ing the mutex lock. The mutex lock shall be acquired by the calling thread and it is up to the new owner
 6              to make the state consistent.
 7
 8       EPERM  The mutex type is PTHREAD_MUTEX_ERRORCHECK or the mutex is a robust mutex, and the current thread does
 9              not own the mutex.
10
11       The pthread_cond_timedwait() function shall fail if:
12
13       ETIMEDOUT
14              The time specified by abstime to pthread_cond_timedwait() has passed.
15
16       EINVAL The abstime argument specified a nanosecond value less than zero or greater than or equal to 1000 mil‐
17              lion.
18
19       These functions may fail if:
20
21       EOWNERDEAD
22              The mutex is a robust mutex and the previous owning thread terminated while holding the mutex lock. The
23              mutex lock shall be acquired by the calling thread and it is up to the new owner to make the state con‐
24              sistent.
25       ENOTRECOVERABLE
26              The state protected by the mutex is not recoverable.
27
28       EOWNERDEAD
29              The mutex is a robust mutex and the process containing the previous owning thread terminated while hold‐
30              ing the mutex lock. The mutex lock shall be acquired by the calling thread and it is up to the new owner
31              to make the state consistent.
32
33       EPERM  The mutex type is PTHREAD_MUTEX_ERRORCHECK or the mutex is a robust mutex, and the current thread does
34              not own the mutex.
35
36       The pthread_cond_timedwait() function shall fail if:
37
38       ETIMEDOUT
39              The time specified by abstime to pthread_cond_timedwait() has passed.
40
41       EINVAL The abstime argument specified a nanosecond value less than zero or greater than or equal to 1000 mil‐
42              lion.
43
44       These functions may fail if:
45
46       EOWNERDEAD
47              The mutex is a robust mutex and the previous owning thread terminated while holding the mutex lock. The
48              mutex lock shall be acquired by the calling thread and it is up to the new owner to make the state con‐
49              sistent.

laanwj commented at 9:06 am on September 13, 2018: member

Okay, so the time passed in through const struct timespec *abstime is invalid. I wonder in what sense, or at least how it ends up that way…

luke-jr commented at 9:15 am on September 13, 2018: member

@ashishchandr70 Does WSL support GDB? Can you get a full backtrace?

ashishchandr70 commented at 3:30 pm on September 13, 2018: none

@luke-jr here you go. Let me know if you need any more info:

 0**(gdb) run**
 1Starting program: /usr/local/bin/bitcoind
 2[Thread debugging using libthread_db enabled]
 3Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 4[New Thread 0x7ffffa670700 (LWP 27880)]
 5[New Thread 0x7ffff9e60700 (LWP 27881)]
 6[New Thread 0x7ffff9650700 (LWP 27882)]
 7[New Thread 0x7ffff8e40700 (LWP 27883)]
 8[New Thread 0x7ffff3ff0700 (LWP 27884)]
 9[New Thread 0x7ffff37e0700 (LWP 27885)]
10[New Thread 0x7ffff2fd0700 (LWP 27886)]
11
12
13************************
14EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_15condition_errorEEEEE
15boost::condition_variable::do_wait_until failed in pthread_cond_timedwait: Invalid argument
16bitcoin in scheduler
17
18terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::condition_error> >'
19  what():  boost::condition_variable::do_wait_until failed in pthread_cond_timedwait: Invalid argument
20
21Thread 3 "bitcoin-schedul" received signal SIGABRT, Aborted.
22[Switching to Thread 0x7ffff9e60700 (LWP 27881)]
230x00007ffffce35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
2454      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
25**(gdb) bt full**
26[#0](/bitcoin-bitcoin/0/)  0x00007ffffce35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
27        resultvar = 0
28        pid = 27876
29        selftid = 27881
30[#1](/bitcoin-bitcoin/1/)  0x00007ffffce3702a in __GI_abort () at abort.c:89
31        save_stage = 2
32        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0, 0,
33              140737436421119, 140737439880512, 1, 140737439880643, 140737385987728, 140737385987056, 140737436427273,
34              140737385989856, 1058, 2, 145847040, 140737287031616, 140737439885168, 140737385989888}},
35          sa_flags = -40187776, sa_restorer = 0x7ffff4000c90}
36        sigs = {__val = {32, 0 <repeats 15 times>}}
37[#2](/bitcoin-bitcoin/2/)  0x00007ffffd9af84d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
38No symbol table info available.
39[#3](/bitcoin-bitcoin/3/)  0x00007ffffd9ad6b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
40No symbol table info available.
41[#4](/bitcoin-bitcoin/4/)  0x00007ffffd9ad701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
42No symbol table info available.
43[#5](/bitcoin-bitcoin/5/)  0x00007ffffd9ad969 in __cxa_rethrow () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
44No symbol table info available.
45[#6](/bitcoin-bitcoin/6/)  0x00000000080714f5 in TraceThread<std::function<void ()> >(char const*, std::function<void ()>) (
46    name=0x84b7b5e "scheduler", func=...) at ./util.h:334
47        e = <optimized out>
48        s = "bitcoin-scheduler"
49[#7](/bitcoin-bitcoin/7/)  0x00000000080731a1 in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > >::operator()<void (*)(char const*, std::function<void ()>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(char const*, std::function<void ()>), boost::_bi::list0&, int) (f=<optimized out>, a=<synthetic pointer>,
50    this=<optimized out>) at /home/achandra/boost/boost_1_68_0/boost/bind/bind.hpp:319
51No locals.
52[#8](/bitcoin-bitcoin/8/)  boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char con---Type <return> to continue, or q <return> to quit---
53st*>, boost::_bi::value<std::function<void ()> > > >::operator()() (this=<optimized out>)
54    at /home/achandra/boost/boost_1_68_0/boost/bind/bind.hpp:1294
55No locals.
56[#9](/bitcoin-bitcoin/9/)  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, std::function<void ()>), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<std::function<void ()> > > > >::run() (this=0x8b177e0)
57    at /home/achandra/boost/boost_1_68_0/boost/thread/detail/thread.hpp:117
58No locals.
59[#10](/bitcoin-bitcoin/10/) 0x00007ffffeb23e3d in thread_proxy () from /home/achandra/boost/boost_1_68_0/stage/lib/libboost_thread.so.1.68.0
60No symbol table info available.
61[#11](/bitcoin-bitcoin/11/) 0x00007ffffd1d76ba in start_thread (arg=0x7ffff9e60700) at pthread_create.c:333
62        __res = <optimized out>
63        pd = 0x7ffff9e60700
64        now = <optimized out>
65        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737385989888, -6477113758224924360, 0, 140737488272895,
66                140737385990592, 140737488280144, 6477118070383264056, 6477110163119489336}, mask_was_saved = 0}},
67          priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
68        not_first_call = <optimized out>
69        pagesize_m1 = <optimized out>
70        sp = <optimized out>
71        freesize = <optimized out>
72        __PRETTY_FUNCTION__ = "start_thread"
73[#12](/bitcoin-bitcoin/12/) 0x00007ffffcf0741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
74No locals.
75(gdb)

laanwj added the label Windows on Sep 15, 2018

laanwj renamed this:
~~bitcoind aborts with boost exception~~
bitcoind aborts with boost exception in WSL guest
on Sep 15, 2018

sipsorcery commented at 7:14 am on September 21, 2018: member

@ashishchandr70 I’ve tested bitcoind v0.16.3 on WSL with Ubuntu 18.04 and don’t get this abort.

I’m (re)installing WSL with Ubuntu 16 to see if I can reproduce.

sipsorcery commented at 7:44 am on September 21, 2018: member

@ashishchandr70 tested v0.16.3 on WSL with Ubuntu 16.04.5 and don’t get the abort.

Release v0.16.2 isn’t available for download anymore. Can you check if you have the same problem with v0.16.3?

pstratem commented at 1:01 pm on June 6, 2019: contributor

possibly when the system is put to sleep the abstime parameter is stale and pthread_cond_timedwait is called with an abstime in the past?

laanwj closed this on Mar 6, 2020

PastaPastaPasta referenced this in commit 417099870a on Sep 28, 2021

PastaPastaPasta referenced this in commit 9b8884b430 on Sep 28, 2021

kittywhiskers referenced this in commit efb9cdad7a on Oct 12, 2021

DrahtBot locked this on Feb 15, 2022

gades referenced this in commit 65ec4496fb on May 2, 2022

Munkybooty referenced this in commit 308aa902f3 on Nov 8, 2022

Munkybooty referenced this in commit 229a901ad8 on Nov 15, 2022

Munkybooty referenced this in commit 0b453d5296 on Nov 25, 2022

Munkybooty referenced this in commit e08534c56e on Nov 25, 2022

bitcoind aborts with boost exception in WSL guest #14200