Deadlock in net code #8297

I'm running into this deadlock all the time:

2016-06-29 15:41:50 POTENTIAL DEADLOCK DETECTED
2016-06-29 15:41:50 Previous lock order was:
2016-06-29 15:41:50  pnode->cs_vRecvMsg  net.cpp:1828 (TRY)
2016-06-29 15:41:50  cs_main  main.cpp:4798
2016-06-29 15:41:50  (1) pfrom->cs_filter  main.cpp:4856
2016-06-29 15:41:50  (2) cs_vSend  net.cpp:2537
2016-06-29 15:41:50 Current lock order is:
2016-06-29 15:41:50  (2) pnode->cs_vSend  net.cpp:1847 (TRY)
2016-06-29 15:41:50  cs_main  main.cpp:6373 (TRY)
2016-06-29 15:41:50  pto->cs_inventory  main.cpp:6593
2016-06-29 15:41:50  (1) pto->cs_filter  main.cpp:6616

I'm running 63fbdbc94d76e586b2fb0754a9a69acfc4b622d3.

It's the machine with 1GB/s SSD and 32GB RAM Xeon CPU E3-1275 v5 @ 3.60GHz

2016-07-01 01:36:18 POTENTIAL DEADLOCK DETECTED
2016-07-01 01:36:18 Previous lock order was:
2016-07-01 01:36:18  (1) pnode->cs_vSend  net.cpp:1847 (TRY)
2016-07-01 01:36:18  (2) cs_main  main.cpp:6372 (TRY)
2016-07-01 01:36:18  (2) cs_main  main.cpp:3501
2016-07-01 01:36:18 Current lock order is:
2016-07-01 01:36:18  pnode->cs_vRecvMsg  net.cpp:1828 (TRY)
2016-07-01 01:36:18  (2) cs_main  main.cpp:5437
2016-07-01 01:36:18  (1) cs_vSend  net.cpp:2537

I'm running 6a87eb0

Can you verify that this is actually deadlocking?

I believe this is a false positive. On Jun 30, 2016 12:55 PM, "Jonas Schnelli" notifications@github.com wrote:

I'm running into this deadlock all the time:

2016-06-29 15:41:50 POTENTIAL DEADLOCK DETECTED 2016-06-29 15:41:50 Previous lock order was: 2016-06-29 15:41:50 pnode->cs_vRecvMsg net.cpp:1828 (TRY) 2016-06-29 15:41:50 cs_main main.cpp:4798 2016-06-29 15:41:50 (1) pfrom->cs_filter main.cpp:4856 2016-06-29 15:41:50 (2) cs_vSend net.cpp:2537 2016-06-29 15:41:50 Current lock order is: 2016-06-29 15:41:50 (2) pnode->cs_vSend net.cpp:1847 (TRY) 2016-06-29 15:41:50 cs_main main.cpp:6373 (TRY) 2016-06-29 15:41:50 pto->cs_inventory main.cpp:6593 2016-06-29 15:41:50 (1) pto->cs_filter main.cpp:6616

I'm running 63fbdbc https://github.com/bitcoin/bitcoin/commit/63fbdbc94d76e586b2fb0754a9a69acfc4b622d3 .

It's the machine with 1GB/s SSD and 32GB RAM Xeon CPU E3-1275 v5 @ 3.60GHz

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub #8297, or mute the thread https://github.com/notifications/unsubscribe/AAl4Q7wqDEvSwyh5ntHH5Y32hK5WNe5dks5qRB8tgaJpZM4JCiHn .

In my case it seemed like a true deadlock. I only built with -DDEBUG_LOCKORDER after my node seemed to not sync newer blocks after a reindex on testnet.

Neither of them looks like a false positive to me, they concern the same locks in different orders at the same time.

My nodes is now running a couple of days in -disable-debug mode and there where no signs of deadlocks... maybe this is a false positive?

I haven't had any for a few days now (with debug). The whole reason I turned on debug in the first place was a suspected deadlock and it occurred under a similar set of circumstances. Seemed to occur just after reindexing and/or disconnecting peers. It doesn't seem to happen to me during normal operation and I haven't tried to replicate again.

I see these in my log too. While they're scary, after tracing through for a while, I don't think they're dangerous. (I do think we should fix up to avoid them, though).

cs_vSend is held for the duration of SendMessages(), which, after b5599147533103efea896a1fc4ff51f2d3ad5808, sometimes locks cs_filter. Additionally, cs_filter is sometimes held while calling PushMessage, which locks cs_vSend.

However, all of that happens on the message handling thread. The net thread locks cs_vSend, but never cs_filter.

All of that could be worked around by making sure not to hold cs_filter while sending.

As for the other potential deadlock, it's a similar story. When processing incoming messages, cs_main is sometimes held while calling PushMessage (which locks cs_vSend). When processing outgoing messages though, cs_vSend is always held, followed sometimes by locking cs_main in SendMessages. But the net thread that contends for cs_vSend never locks cs_main.

The one exception is the optimistic send on new outgoing connections, which does lock cs_main and cs_vSend on the net thread, but not simultaneously.

tl;dr: Both lock order reversals are happening on the message handling thread, and other threads don't attempt to lock both mutexes. I don't think either one presents an obvious deadlock risk. We should still strive to fix by minimizing overlapping locks though.

Possibly the deadlock detection code should include the calling thread in it's tuple?

Thanks for investigating @theuni. @pstratem Well, the deadlock detection code keeps track of all orders in which locks are taken, and remembers that for later. Hacking in an exception for locks taken by the same thread (e.g. remembering the thread) would result in quite muddled logic, and this code isn't the most pretty already.

@pstratem For reporting purposes, or for actual detection?

I agree with @laanwj that we should not add exceptions for locks taken by only one thread. Either we treat that as a given and remove the lock entirely, or we treat it as a temporary situation in the current codebase, in which case the deadlock code should report it.

Is this still observed?

错误： 2019-02-26T08:18:21Z POTENTIAL DEADLOCK DETECTED 2019-02-26T08:18:21Z Previous lock order was: 2019-02-26T08:18:21Z cs_main net_processing.cpp:2462 2019-02-26T08:18:21Z (1) pool.cs validation.cpp:398 2019-02-26T08:18:21Z cs_main wallet/wallet.cpp:1279 2019-02-26T08:18:21Z (2) cs_wallet wallet/wallet.cpp:1279 2019-02-26T08:18:21Z Current lock order is: 2019-02-26T08:18:21Z cs_main wallet/wallet.cpp:1800 2019-02-26T08:18:21Z (2) cs_wallet wallet/wallet.cpp:1800 2019-02-26T08:18:21Z (1) g_mempool.cs wallet/wallet.cpp:1821

环境：Ubuntu16.04 Bitcoin ABC RPC client version v0.19.0.0-e619210

只要启动环境就报上面的错，请问该怎么解决？QQ：707170913 解决有红包