v0.14.0 shutdown hangs #9950

After updating to bitcoin-0.14.0-x86_64-linux-gnu, bitcoind will no longer shut down cleanly with a bitcoin-cli stop or the kill command. A kill -9 is required to make the process terminate.

Tested on three separate fully updated CentOS 6 installations with kernel 2.6.32-642.15.1.el6.x86_64, all of which were behaving normally with 0.13.2. Two of the nodes have pruning enabled with prune=8192, while the third is a non-pruning node. The configuration is otherwise identical:

maxconnections=512 disablewallet=1 dbcache=1000 upnp=0 dns=0 dnsseed=0

Debug log with debug=1 ends with:

2017-03-08 13:51:49 Received a POST request for / from 127.0.0.1:49740 2017-03-08 13:51:49 ThreadRPCServer method=stop 2017-03-08 13:51:50 Interrupting HTTP server 2017-03-08 13:51:50 Interrupting HTTP RPC server 2017-03-08 13:51:50 Exited http event loop 2017-03-08 13:51:50 Interrupting RPC 2017-03-08 13:51:50 tor: Thread interrupt 2017-03-08 13:51:50 torcontrol thread exit 2017-03-08 13:51:50 scheduler thread interrupt 2017-03-08 13:51:50 Shutdown: In progress... 2017-03-08 13:51:50 Stopping HTTP RPC server 2017-03-08 13:51:50 Unregistering HTTP handler for / (exactmatch 1) 2017-03-08 13:51:50 Stopping RPC 2017-03-08 13:51:50 RPC stopped. 2017-03-08 13:51:50 Stopping HTTP server 2017-03-08 13:51:50 Waiting for HTTP worker threads to exit 2017-03-08 13:51:50 Waiting for HTTP event thread to exit 2017-03-08 13:51:50 Stopped HTTP server 2017-03-08 13:51:50 net thread exit 2017-03-08 13:51:50 msghand thread exit 2017-03-08 13:51:53 opencon thread exit

Half an hour later, bitcoind is still running, and does not seem to be using any CPU or generating any IO. It does not terminate active connections, and the socket is still listening and accepting new connections, but no data appears to be exchanged. getpeerinfo from another node for a connection established after the shutdown says:

"lastsend": 1488982201,
"lastrecv": 0,
"bytessent": 126,
"bytesrecv": 0,
"conntime": 1488982201,

The issue seems to be ~80% reproducible with the given setup and configuration if bitcoind has been running for a few minutes, otherwise it seems to always shut down normally.

Can you get a traceback of where it hangs?

Sure, but without debug symbols compiled in I'm not sure how useful this is:

Thread 3 (Thread 0x7faed8e28700 (LWP 15272)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183 #1 0x00007faf08195f18 in ?? () #2 0x00007faf081d738b in ?? () #3 0x00007faf081dcb3a in ?? () #4 0x00007faf081db2cf in ?? () #5 0x00007faf0870f090 in ?? () #6 0x00007faf07ce1aa1 in start_thread (arg=0x7faed8e28700) at pthread_create.c:301 #7 0x00007faf0738caad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7faecb3f1700 (LWP 15280)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183 #1 0x00007faf0845c74b in ?? () #2 0x00007faf07ce1aa1 in start_thread (arg=0x7faecb3f1700) at pthread_create.c:301 #3 0x00007faf0738caad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7faf0810d720 (LWP 15258)): #0 0x00007faf07ce22fd in pthread_join (threadid=140388939761408, thread_return=0x0) at pthread_join.c:89 #1 0x00007faf0870efd7 in ?? () #2 0x00007faf081d944e in ?? () #3 0x00007faf081d94ea in ?? () #4 0x00007faf0819b70a in ?? () #5 0x00007faf08191923 in ?? () #6 0x00007faf081852bf in ?? () #7 0x00007faf072c2d1d in __libc_start_main (main=0x7faf08185290, argc=4, ubp_av=0x7ffcc59debe8, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7ffcc59debd8) at libc-start.c:226 #8 0x00007faf08190bd7 in ?? ()

Note that I amended the original bug report, as pruning seems to have been a red herring. The non-pruning node also ran into the same hang when I tried shutting it down right now, so it seems contingent on having been running for a few minutes rather than the difference in configuration.

Do you happen to have >= 8 addnode's set?

(If so #9953 should fix your issue).

Sure, but without debug symbols compiled in I'm not sure how useful this is

If these are the official binaries, then along with the bitcoind mapping start address (this may differ due to ASLR) and addr2line I can resolve the functions/line numbers.

Do you happen to have >= 8 addnode's set?

Dropping below 8 addnodes does seem to have resolved this. Thanks for the fix.

I've been having a shutdown hang on Windows with 0.14.0 where I'm not using the addnodes command. It could be a separate bug, given that Celean's problem was reportedly fixed by lowering the number of addnodes. Mine gets stuck at the "do not shutdown your computer" window for hours upon hours, where previous versions (except possibly 0.13.2) did not, with no noticeable CPU, hard drive, or memory activity, before it finally closes itself.

@mmortal03 since you don't have >=8 addnodes in your bitcoin.conf, you must be hitting a different problem. Can you open a new github issue, provide the answers to the issue template and paste you bitcoin.conf and the tail of your debug log?

Yep, will do. I compiled the master through #9953, and still see the issue, so it's not the same bug. I'm working on a way to reproduce the issue, and will create new issue once I narrow it down.

@jnewbery , I've created a new issue for the issue I'm experiencing here: #10025

Closing this one as #9953 fixed this specific issue and a new issue has been openened for @mmortal03 's problem