Segmentation fault on Ubuntu Linux with trickle #15674

<!-- This issue tracker is only for technical issues related to Bitcoin Core. General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange at https://bitcoin.stackexchange.com. For reporting security issues, please read instructions at https://bitcoincore.org/en/contact/. If the node is "stuck" during sync or giving "block checksum mismatch" errors, please ensure your hardware is stable by running memtest and observe CPU temperature with a load-test tool such as linpack before creating an issue! -->

<!-- Describe the issue -->

Desription

After running bitcoind for a while, the program terminated with segmentation fault.

Console Output

trickle -s -u 150 ./bitcoind -datadir=/btc-data/bitcoin/data

...

19-02-06T16:24:35Z' progress=0.974263 cache=10.4MiB(78248txo)
2019-03-27T03:19:33Z UpdateTip: new best=0000000000000000001c1d06f652792455f24b63ca66c16e2ba1cc12940af93f height=561832 version=0x20000000 log2_work=90.318593 tx=380346564 date='2019-02-06T16:27:37Z' progress=0.974264 cache=11.1MiB(84392txo)
2019-03-27T03:19:36Z UpdateTip: new best=00000000000000000002371a6adffdcf79ca3a4c99a10d11d8dc9c28355b176c height=561833 version=0x20000000 log2_work=90.318616 tx=380349444 date='2019-02-06T16:37:25Z' progress=0.974268 cache=11.7MiB(89201txo)
2019-03-27T03:19:42Z UpdateTip: new best=000000000000000000077cbc0d1845b47a0a1f4e8d269e469496bbbb03239439 height=561834 version=0x20c00000 log2_work=90.318639 tx=380350543 date='2019-02-06T16:38:29Z' progress=0.974268 cache=12.4MiB(94784txo)
2019-03-27T03:20:48Z Pre-allocating up to position 0x7000000 in blk01519.dat
2019-03-27T03:21:52Z UpdateTip: new best=0000000000000000000d4b0b3bd61bf105a06f0e5c9f06f4a4c85af8e7da200e height=561835 version=0x20000000 log2_work=90.318663 tx=380353291 date='2019-02-06T16:46:25Z' progress=0.974270 cache=13.0MiB(99572txo)
2019-03-27T03:24:11Z Pre-allocating up to position 0xa00000 in rev01519.dat
2019-03-27T03:24:11Z UpdateTip: new best=0000000000000000000e61f02e2aab67704eecb035e02460c78b7aa212bc66c2 height=561836 version=0x20000000 log2_work=90.318686 tx=380354077 date='2019-02-06T16:47:27Z' progress=0.974270 cache=13.8MiB(106454txo)
Segmentation fault (core dumped)

<!--- What behavior did you expect? -->

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

Reproduce the Issue

System: Ubuntu 16.04LTS, AWS Instance Steps: Just run it with sudo trickle -s -u 150 ./bitcoind -datadir=/btc-data/bitcoin/data and wait Version: bitcoin-0.17.0, downloaded form the official website Disk Type: SSD, 1T

Core Dump Analysis

sudo gdb ./bitcoind core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bitcoind...(no debugging symbols found)...done.
[New LWP 2756]
[New LWP 2715]
[New LWP 2714]
[New LWP 2716]
[New LWP 2711]
[New LWP 2727]
[New LWP 2712]
[New LWP 2713]
[New LWP 2717]
[New LWP 2718]
[New LWP 2755]
[New LWP 2760]
[New LWP 2719]
[New LWP 2758]
[New LWP 2720]
[New LWP 2759]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./bitcoind -datadir=/btc-data/bitcoin/data'.
Program terminated with signal SIGSEGV, Segmentation fault.
[#0](/bitcoin-bitcoin/0/)  _int_malloc (av=av@entry=0x7f215c000020, bytes=bytes@entry=288) at malloc.c:3516
3516	malloc.c: No such file or directory.
[Current thread is 1 (Thread 0x7f2163569700 (LWP 2756))]
(gdb) list
3511	in malloc.c
(gdb) bt
[#0](/bitcoin-bitcoin/0/)  _int_malloc (av=av@entry=0x7f215c000020, bytes=bytes@entry=288) at malloc.c:3516
[#1](/bitcoin-bitcoin/1/)  0x00007f21fa707184 in __GI___libc_malloc (bytes=288) at malloc.c:2913
[#2](/bitcoin-bitcoin/2/)  0x000055f69179b3b8 in ?? ()
[#3](/bitcoin-bitcoin/3/)  0x000055f691185d41 in ?? ()
[#4](/bitcoin-bitcoin/4/)  0x000055f691195c4d in ?? ()
[#5](/bitcoin-bitcoin/5/)  0x000055f69115b567 in ?? ()
[#6](/bitcoin-bitcoin/6/)  0x000055f69119b4cb in ?? ()
[#7](/bitcoin-bitcoin/7/)  0x000055f6917d6b3f in ?? ()
[#8](/bitcoin-bitcoin/8/)  0x00007f21fb17b6ba in start_thread (arg=0x7f2163569700) at pthread_create.c:333
[#9](/bitcoin-bitcoin/9/)  0x00007f21fa78a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)
[#0](/bitcoin-bitcoin/0/)  _int_malloc (av=av@entry=0x7f215c000020, bytes=bytes@entry=288) at malloc.c:3516
[#1](/bitcoin-bitcoin/1/)  0x00007f21fa707184 in __GI___libc_malloc (bytes=288) at malloc.c:2913
[#2](/bitcoin-bitcoin/2/)  0x000055f69179b3b8 in ?? ()
[#3](/bitcoin-bitcoin/3/)  0x000055f691185d41 in ?? ()
[#4](/bitcoin-bitcoin/4/)  0x000055f691195c4d in ?? ()
[#5](/bitcoin-bitcoin/5/)  0x000055f69115b567 in ?? ()
[#6](/bitcoin-bitcoin/6/)  0x000055f69119b4cb in ?? ()
[#7](/bitcoin-bitcoin/7/)  0x000055f6917d6b3f in ?? ()
[#8](/bitcoin-bitcoin/8/)  0x00007f21fb17b6ba in start_thread (arg=0x7f2163569700) at pthread_create.c:333
[#9](/bitcoin-bitcoin/9/)  0x00007f21fa78a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) exit
Undefined command: "exit".  Try "help".
(gdb) quit

This might have something to do with "trickle". I am using the latest release of trickle on Ubuntu. "trickle" works fine with my other programs. Bitcoind is the only one to have experienced this issue.

Memory should not be an issue here since I have 16GB RAM and 10GB Swap space.

Have you tried to report this bug upstream?

@lxjhk

Thanks a lot for the bug report!

Are you able to reproduce this issue against a bitcoind binary compiled from current master?

If so, can you post the corresponding gdb output?

@lxjhk Can you follow up here? Was this determined to be an issue with trickle ?

Hi sorry for late reply. I have since switched to the GoLang implementation of Bitcoin because I can't solve that issue. It does not seem to be a trickle issue because when I use trickle with other applications, everything seemed to be fine.

I am quite busy recently. I might be able to try reproducing the bug next week.

I can trivially reproduce on a fresh Ubuntu Bionic with:

sudo trickle -s -d 50 -u 50 ./bitcoin-0.18.0/bin/bitcoind

I can confirm the bug with official build of bitcoin-qt 0.20.0 on Linux Mint 19.3.