deadlock node crash while relaying wallet transaction v0.12.99.0-7f96671 #7645

mruddy commented at 8:59 PM on March 4, 2016: contributor

I self-compiled a clean copy of master (v0.12.99.0-7f96671 (64-bit), more commit detail below) and was running a two node regtest network on my local machine.

I was running various commands with the GUI command prompts and went away for a few minutes to try and research why none of the unconfirmed transactions in the mempool were getting included in blocks when I ran generate. Just before I went away, I checked that the nodes were connected with netstat. When I came back, I noticed that one of the two nodes had crashed. The terminal looked like this:

$ sudo netstat -pantu | grep bitcoin
tcp        0      0 127.0.0.1:2222          0.0.0.0:*               LISTEN      454/bitcoin-qt  
tcp        0      0 127.0.0.1:1111          0.0.0.0:*               LISTEN      453/bitcoin-qt  
tcp        0      0 127.0.0.1:47068         127.0.0.1:1111          ESTABLISHED 454/bitcoin-qt  
tcp        0      0 127.0.0.1:1111          127.0.0.1:47068         ESTABLISHED 453/bitcoin-qt  
$ bitcoin-qt: sync.cpp:112: void potential_deadlock_detected(const std::pair<void*, void*>&, const LockStack&, const LockStack&): Assertion `onlyMaybeDeadlock' failed.

This is the last little bit of /tmp/nodes/2/regtest/debug.log -- the node that failed:

2016-03-04 19:46:40 UpdateTip: new best=4e2da67ace13cd3d348d579f78058583a7af6579a2e577a5cf46207b4f4fc277  height=623 bits=545259519 log2_work=10.285402  tx=624  date=2016-03-04 19:46:53 progress=1.000000  cache=0.2MiB(623tx)
2016-03-04 19:46:40 UpdateTip: new best=074d7982b6dfa72fb905c0524e81ebceb742895ba168ab18c997ccab11fe8e7b  height=624 bits=545259519 log2_work=10.287712  tx=625  date=2016-03-04 19:46:53 progress=1.000000  cache=0.2MiB(624tx)
2016-03-04 19:46:40 UpdateTip: new best=62ae97d172aaeadacf8701fe9c95a899cf8c41f79d4809ef33bab2c38bdec556  height=625 bits=545259519 log2_work=10.290019  tx=626  date=2016-03-04 19:46:53 progress=1.000000  cache=0.2MiB(625tx)
2016-03-04 19:46:40 UpdateTip: new best=77cab76e795cd8cac27d5e9a9cbe71b45be56a5991f74b268a4ab741230c6960  height=626 bits=545259519 log2_work=10.292322  tx=627  date=2016-03-04 19:46:53 progress=1.000000  cache=0.2MiB(626tx)
2016-03-04 19:46:40 UpdateTip: new best=3f815a8ae2e107657a0c1f465f8ffc57be5a6bd57d87811025092eef7b2cd6a3  height=627 bits=545259519 log2_work=10.294621  tx=628  date=2016-03-04 19:46:53 progress=1.000000  cache=0.2MiB(627tx)
2016-03-04 19:46:41 UpdateTip: new best=0c15e4d403d5cd2cfd0be0c0f1e691c301a61eb30953f9512876c91386323eca  height=628 bits=545259519 log2_work=10.296916  tx=629  date=2016-03-04 19:46:54 progress=1.000000  cache=0.2MiB(628tx)
2016-03-04 19:46:41 UpdateTip: new best=0213034a837dd2e165cb326a16e11b756b7346d41939c2812e99360d08fc95fb  height=629 bits=545259519 log2_work=10.299208  tx=630  date=2016-03-04 19:46:54 progress=1.000000  cache=0.2MiB(629tx)
2016-03-04 19:46:41 UpdateTip: new best=086294ac122c4962b71fbdc8e6a8b6b5d5fa9c4027db01d300985fe1380a6787  height=630 bits=545259519 log2_work=10.301496  tx=631  date=2016-03-04 19:46:54 progress=1.000000  cache=0.2MiB(630tx)
2016-03-04 19:46:41 UpdateTip: new best=41a614038ff29c0501c27079796b4f0017cdfa140121b922dfc0194f08e8c3b9  height=631 bits=545259519 log2_work=10.303781  tx=632  date=2016-03-04 19:46:54 progress=1.000000  cache=0.2MiB(631tx)
2016-03-04 19:47:57 Relaying wtx cdfc78b3973b8b2419262164c7418798a07026ea550262b16c1dd42dacd62d3f
2016-03-04 19:47:57 POTENTIAL DEADLOCK DETECTED
2016-03-04 19:47:57 Previous lock order was:
2016-03-04 19:47:57  (1) pnode->cs_vSend  net.cpp:1765 (TRY)
2016-03-04 19:47:57  cs_main  main.cpp:5465 (TRY)
2016-03-04 19:47:57  cs_wallet  wallet/wallet.cpp:1500
2016-03-04 19:47:57  (2) cs_mapRelay  net.cpp:2083
2016-03-04 19:47:57 Current lock order is:
2016-03-04 19:47:57  pnode->cs_vRecvMsg  net.cpp:1746 (TRY)
2016-03-04 19:47:57  cs_main  main.cpp:4220
2016-03-04 19:47:57  (2) cs_mapRelay  main.cpp:4313
2016-03-04 19:47:57  (1) cs_vSend  net.cpp:2470

This is the startup script I was using:

#!/bin/bash
BITCOIN_QT="~/Desktop/bitcoin/src/qt/bitcoin-qt"
rm -rf /tmp/nodes/
mkdir -p /tmp/nodes/{1,2}
$BITCOIN_QT -regtest -datadir=/tmp/nodes/1/ -bind=127.0.0.1:1111 &
$BITCOIN_QT -regtest -datadir=/tmp/nodes/2/ -bind=127.0.0.1:2222 -connect=127.0.0.1:1111 &

OS: Ubuntu 15.10
Self-build of commit 7f966713a4133caccc54e6aa0f70ff80826cd3b2 with typical:
./autogen.sh
./configure --enable-debug
make check

MarcoFalke commented at 5:31 PM on March 5, 2016: member

Related or duplicate: #7470

mruddy commented at 5:45 PM on March 5, 2016: contributor

I don't think it's a dupe because the locks noted in the "POTENTIAL DEADLOCK DETECTED" message are different.

jonasschnelli commented at 5:49 PM on March 5, 2016: contributor

Agree with @mruddy. This looks after a real deadlock prevention.

laanwj commented at 1:04 PM on March 14, 2016: member

I don't think it's a dupe because the locks noted in the "POTENTIAL DEADLOCK DETECTED" message are different.

it's still the same underlying issue though: locking in the network code is a mess.

So I'm closing this as duplicate of #7470, even though it may not be an exact duplicate I don't think there's any advantage to tracking this separately.

laanwj closed this on Mar 14, 2016

MarcoFalke locked this on Sep 8, 2021