very slow block download due to socket 10054 errors

rebroad commented at 10:24 pm on April 6, 2012: contributor

On “internet” connections from ISPs who throttle bandwidth, RST resets are often used, which show up as 10054 errors in debug.log. If this happens during a block download, the client waits until the next block is generated before resuming downloading of blocks. This can cause significant delay in catching up.

Diapolo commented at 10:25 pm on April 6, 2012: none

Is this a select() failed message?

luke-jr commented at 10:28 pm on April 6, 2012: member

This is a broken “internet” connection. I expect you probably have grounds for a class-action lawsuit if it’s intentional.

luke-jr commented at 10:28 pm on April 6, 2012: member

BTW, you might consider: iptables -I INPUT -p tcp –tcp-flags RST RST -j DROP

rebroad commented at 12:20 pm on April 7, 2012: contributor

@luke-jr they sent the RSTs to both sides of the connection, so this would only work if the other side had this rule too, which is unlikely… class action lawsuit sounds a good idea, but I’d rather not admin it..!

gavinandresen commented at 8:28 pm on April 12, 2012: contributor

Closing this– sorry, downloading the entire blockchain from slow throttled ISP connections is not something we’re going to test/support.

And there is already an issue open to support “lightweight” headers-only mode.

gavinandresen closed this on Apr 12, 2012

rebroad commented at 11:49 pm on April 12, 2012: contributor

I still do think the network would be healthier if it were to be taken into consideration that some nodes will be running from behind these “crippled” ISP networks. It has some impact, although perhaps considered negligible. Perhaps these type of ISP services will become a thing of the past. Perhaps they will become more mainstream. I guess we shall see.

rebroad commented at 1:53 pm on May 8, 2012: contributor

Hi @gavinandresen , do you consider the ability to resume downloads to be useful? e.g. “wget -c”? Given that it seems to be fairly standard on the internet these days, is there a good reason that bitcoin should not support resume-able block downloads? Could at least there be an option feature that can be supported, such that if I create a fork that does support this, that there can be a way to identify other peers which support it, without adding to the already unnecessary traffic that this proposed patch intends to reduce? I’d like to explore this, and would be happy for it not to enter the master branch (even though I think it would be an overall advantage if it eventually did).

sipa commented at 2:13 pm on May 8, 2012: member

All blocks are downloaded one at a time anyway, if the connection fails, it will eventually continue from another peer. I don’t think it’s worth adding the complexity to handle partial downloads to the protocol - especially now when blocks are at most 1 MB.

There is an independent useful improvement possible, where the client re-issues a getblocks if the connection from which the current sync is being done closes. I consider it low priority though, and I advise you get an internet connection or use a light-weight or SPV client.

rebroad commented at 2:31 pm on May 8, 2012: contributor

@sipa, I’ve already modified the software to re-issue getblocks, but haven’t gotten around to re-basing it and submitting a pull request for it. My internet connection disconnects connections at around the 300kB mark, so a 1MB block is often a problem unless download it via tor. I won’t be the only person world-wide with such a connection to the bitcoin network but who also wants to run a full-node. I think bitcoin should support such users, and have yet to hear an argument why they should be excluded. Are there any disadvantages to including the ability to resume block downloads, for example?

gavinandresen commented at 3:09 pm on May 8, 2012: contributor

Network-facing code is the most security-critical code; a mistake in handling data from the network can easily lead to a catastrophic compromise.

So keeping it as simple as possible is a high priority.

rebroad commented at 8:53 pm on May 8, 2012: contributor

@gavinandresen I agree with the importance of keeping it simple. I was wondering if perhaps then the ability to respond to to peers which request partial blocks might be added, (in a similar way to header functionality being added), but that wasn’t used by default. This would allow it to be tested by a minority of nodes who use this feature, without impacting on the majority of nodes which do not use the feature. Would this be something you might consider as a way forward?

I’d be willing to contribute to the coding of this, but would rather get an amber light first before putting too much effort into it…

sipa commented at 9:06 pm on May 8, 2012: member

I am afraid there is no way to get a decent user experience from a bitcoin node running on a crippled internet connection, anyway.

rebroad commented at 11:00 pm on May 8, 2012: contributor

@sipa - what is a decent user experience? With the changes I’ve made to my fork, I’m now downloading the blockchain much faster, but it’s very inefficient in terms of bandwidth due to often downloading the same block from many peers, and often the connection breaking before the block completes. My proposals would fix both these issues. I’m regularly getting a memory pool of 400 or so between block updates, so I think my node is contributing to the network fine for transactions so far, just not so well for blocks until these changes can be implemented. (By the way, there is still a bug in my fork where it loses track of “block providers” and doesn’t getblocks as early as it should - stil to fix).

rebroad commented at 4:49 pm on May 10, 2012: contributor

I’m about (when I get time) to clean-up my patches and start from scratch on improvements to downloading blocks. I’ll probably completely separate the code for downloading blocks from the code for downloading txs. My thinking is that it can getblocks to a number of nodes and keep a list of the blocks discovered and who has them, then based on measurements it takes of network speed in general and individual speed of each connected peer, it can download a number of different blocks at once from different peers, keeping track of any peers that disconnect before a block is downloaded so that it knows to request it again, and keeping track of nodes that go quiet or get stuck, perhaps requesting the same block from more than one node, but keeping track of the sizes of blocks (revealed in the headers) so that it can tell if a previously quiet/stuck node starts to send a block it already has and can disconnect it to save the bandwidth of both peers. In order to reduce the chances of blocks of the same size, it will limit the number of getdata blocks to a smaller number than the existing 500.

suprnurd referenced this in commit ae48630dfb on Dec 5, 2017

sanch0panza referenced this in commit 8b186a651e on May 17, 2018

sanch0panza referenced this in commit f2678b29cb on May 17, 2018

sanch0panza referenced this in commit 8afbc146e5 on May 17, 2018

lateminer referenced this in commit 6aaa531ec1 on Oct 30, 2019

DrahtBot locked this on Sep 8, 2021

very slow block download due to socket 10054 errors #1053