Unnecessary disconnections due to PONG delays during the initial block download #35761

issue abitmore opened this issue on July 21, 2026
  1. abitmore commented at 2:58 AM on July 21, 2026: none

    Is there an existing issue for this?

    • I have searched the existing issues

    Current behaviour

    During the initial block download, if the node does not receive a response PONG message within 20 minutes of sending a PING message, it will simply disconnect from the peer node, regardless of how many blocks it has downloaded from that peer node within those 20 minutes. Code: https://github.com/bitcoin/bitcoin/blob/6f1c56f03a00a3b5f521ec824b965f910ef9151b/src/net_processing.cpp#L5455-L5464

    This does not cause a problem for nodes that download data fast. However, for nodes with slow but stable network connections, or nodes connected to too many peers, this may cause them to repeatedly disconnect from established peers and connect to new peers, resulting in slower download speeds.

    Expected behaviour

    During the initial block download, if the node has already received a block (or other thresholds) from a peer after sending a PING message, do not disconnect even if it has not yet received a PONG message.

    Or perhaps better make the 20-minute ping-pong timeout (currently ‎TIMEOUT_INTERVAL is reused) configurable, or make the queue length for each peer (MAX_BLOCKS_IN_TRANSIT_PER_PEER, which is 16 blocks) configurable.

    Steps to reproduce

    Relevant log output

    2026-07-20T16:27:29Z New outbound-full-relay peer connected: transport: v2, version: 70016, blocks=958908 peer=173
    2026-07-20T16:27:29Z [net] sending sendcmpct (9 bytes) peer=173
    2026-07-20T16:27:29Z [net] sending ping (8 bytes) peer=173
    2026-07-20T16:27:29Z [net] sending getheaders (1029 bytes) peer=173
    2026-07-20T16:27:29Z [net] initial getheaders (958903) to peer=173 (startheight:958908)
    2026-07-20T16:27:30Z [net] received: sendcmpct (9 bytes) peer=173
    2026-07-20T16:27:30Z [net] received: ping (8 bytes) peer=173
    2026-07-20T16:27:30Z [net] sending pong (8 bytes) peer=173
    2026-07-20T16:27:30Z [net] received: getheaders (1029 bytes) peer=173
    2026-07-20T16:27:30Z [net] getheaders -1 to end from peer=173
    2026-07-20T16:27:30Z [net] sending headers (1 bytes) peer=173
    2026-07-20T16:27:30Z [net] received: feefilter (8 bytes) peer=173
    2026-07-20T16:27:30Z [net] received: feefilter of 0.00000100 BTC/kvB from peer=173
    2026-07-20T16:27:30Z [net] received: addrv2 (16 bytes) peer=173
    2026-07-20T16:27:30Z [net] Received addr: 1 addresses (1 processed, 0 rate-limited) from peer=173
    2026-07-20T16:27:34Z [net] received: addrv2 (24155 bytes) peer=173
    2026-07-20T16:27:34Z [net] Received addr: 1000 addresses (1000 processed, 0 rate-limited) from peer=173
    2026-07-20T16:27:34Z [net] received: pong (8 bytes) peer=173
    2026-07-20T16:27:34Z [net] received: headers (406 bytes) peer=173
    2026-07-20T16:27:34Z Synchronizing blockheaders, height: 958908 (~100.00%)
    2026-07-20T16:27:34Z [net] Protecting outbound peer=173 from eviction
    2026-07-20T16:27:34Z [net] sending sendheaders (0 bytes) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 00000000000000000001dbf30957f829770640898f3c7be187995a5e1645cbe8 (950790) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 00000000000000000000adde76f46bcb48ebf7f4d2200d7f6c2bf33a362aa710 (950791) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 0000000000000000000168678dce9be556e151a7970210bffd1d73502397c7d3 (950792) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 00000000000000000000c986b6845f6b9993968ded37984d52ae31a523291ba4 (950793) peer=173 
    2026-07-20T16:27:34Z [net] Requesting block 000000000000000000017618b68ddff1cb1d5a47c9b66bfad39d7b21bb3c6a7d (950794) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 00000000000000000001aeaf74473ff2118975400967d114347a78bf7a5b935e (950795) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 00000000000000000000bde0bfb52a4fc28892cc70c18212d924d2291e456a07 (950796) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 00000000000000000001bc7dabf0bc458079e4845e7c1c5ddba07e3141fb289b (950797) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 000000000000000000020e6b2d990d1f06883ecad42b6b5b2dd14de726d71ac1 (950798) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 000000000000000000008c1e5f7fc35119de11a6bc17a0d76b8431fb2c4724ec (950799) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 0000000000000000000090515c970d809424fcbd01712853e82a52bde9225709 (950800) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 0000000000000000000027f680a3c9fa271f3cf6ef7d3995794ba7d6f90ba90a (950801) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 0000000000000000000181f2ce4ebb6c0f17bdaa17df2ac20eaa49d1c4253950 (950802) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 000000000000000000008fcc5a9e6e48d957d25e2705c369c9f0c8039a375294 (950803) peer=173 
    2026-07-20T16:27:34Z [net] Requesting block 000000000000000000000ecae92f3e870c772d1fe76c97624aa11c75741d2661 (950804) peer=173
    2026-07-20T16:27:34Z [net] Requesting block 00000000000000000001d88d656d01973b7ab455bfb5600f8e3cd35196bd01df (950805) peer=173
    2026-07-20T16:27:34Z [net] sending getdata (577 bytes) peer=173
    
    2026-07-20T16:29:15Z [net] received: block (1688724 bytes) peer=173
    2026-07-20T16:29:15Z [net] received block 00000000000000000001dbf30957f829770640898f3c7be187995a5e1645cbe8 peer=173
    2026-07-20T16:29:15Z UpdateTip: new best=00000000000000000001dbf30957f829770640898f3c7be187995a5e1645cbe8 height=950790 version=0x2369e000 log2_work=96.220744 tx=13633
    59646 date='2026-05-24T08:32:23Z' progress=0.980754 cache=747.8MiB(5298112txo)
    2026-07-20T16:29:15Z [net] Requesting block 0000000000000000000007a5b47c1ce69ba9cd72805094619f599c7c26379d13 (950806) peer=173
    2026-07-20T16:29:15Z [net] sending getdata (37 bytes) peer=173
    2026-07-20T16:29:29Z [net] sending ping (8 bytes) peer=173
    
    2026-07-20T16:49:28Z [net] received: block (1658390 bytes) peer=173
    2026-07-20T16:49:28Z [net] received block 000000000000000000000ecae92f3e870c772d1fe76c97624aa11c75741d2661 peer=173
    2026-07-20T16:49:29Z UpdateTip: new best=000000000000000000000ecae92f3e870c772d1fe76c97624aa11c75741d2661 height=950804 version=0x26d66000 log2_work=96.220872 tx=13634
    29972 date='2026-05-24T10:46:38Z' progress=0.980781 cache=747.8MiB(5335066txo)
    2026-07-20T16:49:29Z [net] Requesting block 00000000000000000000e484e6f695fa3e56a7cc8d743b3f65af94d4e7ea9105 (950820) peer=173
    2026-07-20T16:49:29Z [net] sending getdata (37 bytes) peer=173
    2026-07-20T16:49:29Z [net] received: addrv2 (31 bytes) peer=173
    2026-07-20T16:49:29Z [net] Received addr: 2 addresses (2 processed, 0 rate-limited) from peer=173
    2026-07-20T16:49:29Z [net] received: headers (82 bytes) peer=173
    2026-07-20T16:49:29Z Synchronizing blockheaders, height: 958910 (~100.00%)
    2026-07-20T16:49:29Z [net] received: addrv2 (28 bytes) peer=173
    2026-07-20T16:49:29Z [net] Received addr: 1 addresses (1 processed, 0 rate-limited) from peer=173
    2026-07-20T16:49:29Z [net] received: addrv2 (44 bytes) peer=173
    2026-07-20T16:49:29Z [net] Received addr: 1 addresses (1 processed, 0 rate-limited) from peer=173
    2026-07-20T16:49:29Z [net] ping timeout: 1200.043308s, disconnecting peer=173
    2026-07-20T16:49:29Z [net] Resetting socket for peer=173
    2026-07-20T16:49:29Z [net] Cleared nodestate for peer=173
    

    How did you obtain Bitcoin Core

    Pre-built binaries

    What version of Bitcoin Core are you using?

    v31.1

    Operating system and version

    Ubuntu 26.04 LTS

    Machine specifications

    No response

  2. maflcko added the label P2P on Jul 21, 2026
  3. maflcko added the label Resource usage on Jul 21, 2026
  4. freemanrecovery224477 commented at 10:48 AM on July 23, 2026: none

    Interesting issue. During the initial block download, a node is already handling a significant amount of network and process activity, so delayed PONG responses may not necessarily indicate that a peer is unhealthy or unresponsive. If those delays are treated the same way as they would be during normal operation, the node could disconnect from otherwise useful peers unnecessary. It seems like one possible solution could be to account for IBD-specific conditions when evaluating PONG timeouts, while still ensuring genuinely unresponsive peers are eventually removed. I’m curious what others think: should IBD use more flexible timeouts handling, or could that introduce security or resource-management concerns?

  5. davidgumberg commented at 12:57 AM on August 13, 2026: contributor

    My opinion is that this is not likely to be an issue in Bitcoin Core.

    A connection that does not respond to a PONG after 20 minutes either:

    1) Has not implemented BIP31 at all.

    In this case, we won't disconnect them after a timeout: https://github.com/bitcoin/bitcoin/blob/2f72123f613f3aa1484ed06344039be78dedcb33/src/net_processing.cpp#L5715-L5722

    https://github.com/bitcoin/bitcoin/blob/2f72123f613f3aa1484ed06344039be78dedcb33/src/net_processing.cpp#L5685-L5695

    (since m_ping_nonce_sent == false)

    2) Has a RTT > 20 minutes

    This connection is not useful and we should disconnect.

    3) Has implemented BIP31 and is a good connection but their implementation or OS is buggy and never processed our PING or failed to send a PONG.

    We should also probably disconnect these nodes.


    Since you are running Bitcoin Core v31.1 and your peer advertises a version high enough to support BIP31, and the connection seems to be functioning otherwise, the only explanation for the logs you've provided is 3), looking at Bitcoin Core's implementation of PING processing, I find it unlikely that a Bitcoin Core node would not respond with a PONG:

    https://github.com/bitcoin/bitcoin/blob/2f72123f613f3aa1484ed06344039be78dedcb33/src/net_processing.cpp#L5209-L5227

    unless there is a bug somewhere else in the network message layer that causes received messages to fail to be processed or sent messages to never get sent.

    The only other possibilities are that there is a bug in your peer's OS TCP implementation, or your peer is running another implementation which has a bug in its BIP31 implementation that causes it to never respond with a PONG.

    Everything else in the description about a connection that's kind of slow but still kind of useful is not relevant, a PONG is a few bytes, a block is a few million bytes.

  6. abitmore commented at 9:46 AM on August 13, 2026: none

    a PONG is a few bytes, a block is a few million bytes.

    From the logs in OP, we can see:

    1. my node requested 15+ blocks from a peer, and then sent a PING,
    2. in 20 minutes, my node downloaded 10+ blocks from the peer, which means the peer's RTT is acceptable,
    3. after 20 minutes, my node disconnected from the peer because it hadn't received a PONG from the peer.

    My guess is that the (tiny) PONG messages were queued in the peer node after the (large) block messages, perhaps MakeAndPushMessage() appends messages to a response queue one by one, and PONG messages don't have a higher priority to be sent out sooner, which is an intentional design.

  7. mzumsande commented at 10:12 AM on August 13, 2026: contributor

    My opinion is that this is not likely to be an issue in Bitcoin Core.

    Not entirely sure, imagine this: we are slow, peer is fast, both are running latest bitcoin core.

    So we will think the peer didn't answer our ping and disconnect them.

    Now we are not so extremely slow that a roundtrip takes 20minutes, but we are slow enough that downloading 16 blocks (while simultaneously loading blocks from 9 other outbound peers) is so slow that the pong times out. This seems a realistic scenario to me, or am I missing something?

  8. mzumsande commented at 12:52 PM on August 13, 2026: contributor

    I think that the best way to remediate this is to not do the check for failed pings when there are blocks inflight with the peer. Since the last_recv timeout in net would still apply, I don't think that there would be major downsides. https://github.com/bitcoin/bitcoin/blob/11090c8bb359f894ef7d97b65aff52fe8191aec1/src/net.cpp#L2073 I'll draft up a PR for that in the next days.

  9. davidgumberg commented at 11:37 PM on August 20, 2026: contributor

    My guess is that the (tiny) PONG messages were queued in the peer node after the (large) block messages, perhaps MakeAndPushMessage() appends messages to a response queue one by one, and PONG messages don't have a higher priority to be sent out sooner, which is an intentional design.

    Now we are not so extremely slow that a roundtrip takes 20minutes, but we are slow enough that downloading 16 blocks (while simultaneously loading blocks from 9 other outbound peers) is so slow that the pong times out. This seems a realistic scenario to me, or am I missing something?

    Ah, makes sense! I was wrong, I think I must have misinterpreted or misread the logs as having the PING before the GETDATA.

  10. mzumsande commented at 3:59 PM on August 25, 2026: contributor

    See #36080 for a fix.

  11. sipa commented at 4:37 PM on August 25, 2026: member

    Would an option be to let the network layer store the receive-complete timestamp for each message, and have the disconnect logic use those instead of the time they're processed?

  12. mzumsande commented at 4:51 PM on August 25, 2026: contributor

    Would an option be to let the network layer store the receive-complete timestamp for each message, and have the disconnect logic use those instead of the time they're processed?

    I don't think so: The message that is "on hold" is the original PING from node A (IBD node) to node B (serving blocks, so it doesn't process other messages). But it's node A who disconnects because of the missing PONG (which is actually never received as a result of node B not processing the PING in time).

  13. sipa commented at 4:52 PM on August 25, 2026: member

    Right, of course. This only addresses the problem in the other direction.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-08-31 18:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me