Local Node Discovery Mechanism

spinza commented at 12:16 pm on March 5, 2014: none

I have seen in lots of guys on forums needing a second node at home and complaining about the massive download time it will take to add that node. Now, if they thought that far, most of them could manually add an existing LAN node to their conf, but the truth is not many people dig under the hood enough. Plus the ip changes etc etc.

I’m not sure what the security implications would be of a node broadcasting it’s availability on the LAN and hooking up with other nodes automatically. I’d be happy as long as the node also has a connection with other nodes on the greater internet to make sure it’s not being sent a bunch of lies locally.

Surely this will have savings for the network in general as some of the bandwidth usage gets moved to the LAN rather than the greater internet. This should also save individual users’ bandwidth.

Perhaps using something like Simple Service Discovery Protocol.

If I were a coder I’d submit some more details but my skills are limited.

bf0d7998-81ec-48d1-b236-076ed3c77581 commented at 9:34 am on March 6, 2014: none

To expand on this, -addnode doesn’t work how you’d expect and neither does -connect really. If I want my two local nodes to communicate I either have to forge every connection manually with -connect (external ones, and internal LAN ones) or just hope that the -addnode actually works. Ideally addnode would whitelist/add priority to the connection but it doesn’t seem to, often the node will just ignore them and connect to random external peers.

spinza commented at 9:43 am on March 6, 2014: none

Good point. You could leave only one node open to the internet and have all others connect to that one. There is some logic with addnode. I don’t recall what it is at present. But you may get into issues when the node you are adding is on max connections. Also if that node have multiple peers in the same /16 IP block. Don’t know if this limit is applied on private IPs.

bf0d7998-81ec-48d1-b236-076ed3c77581 commented at 10:39 am on March 6, 2014: none

Looks like Bitcoind won’t want to connect to any more than one local IP address as they’re all in the same “bucket”, doesn’t seem to apply to addnode though. Not sure how it would find out about a local address though (surely they wouldn’t be published in addr messages?).

https://github.com/bitcoin/bitcoin/blob/master/src/netbase.cpp#L828

The comment in the addnode section suggests that it will aggressively try to connect to every specified node (one of each of the addnode is a DNS record with multiple entries). I suppose the correct behavior would be to ignore the maximum outgoing peers for these connections, otherwise they just fill with normal connections and the -addnode doesn’t succeed.

https://github.com/bitcoin/bitcoin/blob/master/src/netbase.cpp#L828

At any rate it doesn’t act as everybody expects. @gmaxwell made a comment earlier about not being able to connect to the public mining backbone with -addnode occasionally, so it’s presumably an issue others are encountering.

gmaxwell commented at 10:44 am on March 6, 2014: contributor

The network group limitation does not apply to addnoded nodes.

The outgoing connection limitation isn’t the issue there (the outgoing semaphores are) but we should not uncap the outgoing connections in any case (because that makes it very easy for people to waste a lot of network resources uninteintionally), we should just make the addnoded nodes take priority over other outgoing nodes. I discussed this and the cause for the behavior in the same discussion you’re mentioning there. :) (What I suggested is that if there are unconnected addnodes and connections to non-addnoded peers we should permit one more connection, and if we find ourselves with 9 outbound connections drop one of them (some kind of priority order))

bf0d7998-81ec-48d1-b236-076ed3c77581 commented at 10:58 am on March 6, 2014: none

Makes sense.

Not sure why people insist on trying to get more than 8 connections, there’s no real benefit to it unless you’re supplying a listening socket for other users. Is there some sort of misconception that people’s transactions will go faster if they have more or something?

spinza commented at 11:05 am on March 6, 2014: none

I don’t see much risks of a local node discovery mechanism?

The risks are less on a private LAN however (assuming the node is also talking to random nodes on the internet). But a laptop hopping on wifi there may be issues.

If the owner of the hotspot is bad, he can intercept your bitcoin traffic in any case? With or without local node discovery. If the internet on a hotspot is not working and/or bitcoin traffic is blocked in some way. E.g. before you pay for a paywall hotspot. Someone with bad motives could be running a node that is found by your laptop. Then in theory he could send you money in exchange for something and the tx will appear but may not be visible on the greater network. If you wait for it to confirm though, he’d have to find a block at current difficulty to “fake” confirm it which would be … difficult. This also applies to the first case.

gmaxwell commented at 11:06 am on March 6, 2014: contributor

Yep, people erroneously think their transactions will go through faster or that they’ll sync the blockchain faster with more connections.

gmaxwell commented at 11:07 am on March 6, 2014: contributor

@lrossouw it may make sense to have ultimately, but right now it likely wouldn’t make any difference at all in the behavior. To get faster syncup you really need to connect exclusively to a local node.

bf0d7998-81ec-48d1-b236-076ed3c77581 commented at 11:11 am on March 6, 2014: none

@gmaxwell Should add a note above the 8 outgoing limit constant that it won’t help them out if they change it, I’ve seen a few people modify that to insane levels.

0// Modifying this variable to gain more outgoing connections will not do as you expect!
1// * Downloading blocks occurs from a single peer, adding more will not let you sync faster. 
2// * More peers will not make your transactions confirm faster, the block time and transaction 
3//   propagation will not be positively impacted. 
4// * Adding more than 8 outgoing connections negatively impacts the network by using 
5//   up listening capacity

gmaxwell commented at 11:16 am on March 6, 2014: contributor

Sometimes a documentation limit is a feature, not a bug… :-/

bf0d7998-81ec-48d1-b236-076ed3c77581 commented at 12:17 pm on March 6, 2014: none

Bitcoin QT has some impressive features then (and man is that wiki out of date).

Point taken though.

bardiharborow commented at 1:12 am on March 12, 2014: contributor

BitTorrent uses a local IP multicast system to detect local peers. Just throwing it out there.

spinza commented at 9:05 am on March 27, 2014: none

Yes I had something like that in mind.

rebroad commented at 1:34 am on April 1, 2014: contributor

I would argue that if you trust the node on your LAN, that you don’t need to perform all the validations that are normally done, so perhaps adding a “trustednode=” option into bitcoin.conf would be the answer to this. Anyway, why wouldn’t you prefer to just store the blockchain once on your LAN, and just have multiple wallets using the blockchain database?

darkhosis commented at 4:12 am on April 1, 2014: none

re:

// Modifying this variable to gain more outgoing connections will not do as you expect! // * Downloading blocks occurs from a single peer, adding more will not let you sync faster. // * More peers will not make your transactions confirm faster, the block time and transaction // propagation will not be positively impacted. // * Adding more than 8 outgoing connections negatively impacts the network by using // up listening capacity

1st point, yes, when doing the initial blockchain download it’s best to just connect to one fast node using connect=.

2nd point, for the majority of people, yes. I started 12 servers all w/ 600 outbound connections & maxconnections of 900. After running them for 3 days, the transaction propagation time on http://bitcoinstats.com/network/propagation/ dropped from 1.2 or 1.3s to 0.8s… the block times dropped too, but couldn’t come to any conclusions on that without getting an avg size of block for each day.

3rd, only if you’re adding them w/o sufficient bandwidth and/or processing power (like a shared 1 core vps).

bardiharborow commented at 10:02 am on April 1, 2014: contributor

@rebroad, In the interests of saving dumb end-users from themselves, I’d have to disagree with the “trustednode=” idea. The Local Node Discovery system would allow you lots of use cases to operate more efficiently. For example, at work your co-worker’s machines could be helping you sync. You would still want your own copy (and verify it) but you don’t want to put too much load on your workplace’s network connection (multiple people downloading the blockchain). In the end it would be good to have local nodes prioritize each other (while checking with the external network to make sure a malicious LAN node couldn’t feed you off main chain blocks).

laanwj added the label Improvement on Apr 1, 2014

laanwj added the label Priority Low on Apr 1, 2014

spinza commented at 10:18 am on April 1, 2014: none

I agree with @bardiharborow . The trustednode/connect idea and the idea of manually ensuring that different wallets on different PCs share the same blockchain data is possible (see @rebroad). However it is fairly technical for most people. With a auto discovery feature some of this will automatically happen (though the blockchain data will be stored on each pc).

The problem with going ahead with this is because the blockchain download only uses one peer at a time so along with discovery we’d need a change to make sure that it either prefers local nodes for the download or a change to ensure that it downloads from multiple nodes at once as @darkhosis pointed out.

bardiharborow commented at 12:57 pm on April 2, 2014: contributor

I’ve put together a test Local Peer Discovery System using IP Multicast in Python. I have no idea how we convert that to C++, but I can work through this with someone who knows C++ better.

laanwj added the label P2P on May 9, 2014

martindale commented at 3:32 pm on June 12, 2014: none

+1 for using IP Multicast as a solution for this.

Azulan commented at 7:57 am on March 23, 2015: none

I would like to endorse the idea that local nodes should have precedence over others. I try to get my machines to talk to each other and they only stay connected for a short while. One is open to the internet and the other is not. Would really like there to be a config setting or something that induces a local priority mode or something to that effect.

spinza commented at 8:21 am on March 23, 2015: none

@Azulan you can add something like connect=[ip of online machine] in the bitcoin.conf of the offline machine. That way the offline machine will only connect to online machine. Probably also whitelist=[ip] on both bitcoin.conf so that they whitelist connections from each other.

spinza commented at 10:23 am on May 27, 2015: none

@lrossouw it may make sense to have ultimately, but right now it likely wouldn’t make any difference at all in the behavior. To get faster syncup you really need to connect exclusively to a local node.

With headers-first sync this does make a difference now. The node will sync mostly with the fastest nodes which should in most case be the LAN node(s).

Azulan commented at 3:18 am on May 30, 2015: none

@spinza I thought so too, but as I reported earlier, it never seems to prefer staying connected to the local peer, which is why I asked for local peer preference in the first place.

jgarzik commented at 4:46 am on May 30, 2015: contributor

Although not directly related, it seems useful to reference NODE_EXT_SERVICES, PR #4657

spinza commented at 10:41 am on June 2, 2015: none

@Azulan I’ll test that out at some point. I’ll addnode a local peer on a new install and see if it pulls over the inet or from the local node mainly.

Azulan commented at 8:10 am on June 8, 2015: none

@spinza I am currently running one up to date 10.2 node and one new 11.0rc node on my lan. They are peers, but the new node continues to prefer other nodes instead of the local and up to date node. The 10.2 node is port forwarded to the internet, the 11.0rc node is not.

laanwj added the label Feature on Feb 9, 2016

laanwj removed the label Refactoring on Feb 9, 2016

rebroad commented at 4:31 pm on March 1, 2016: contributor

@bardiharborow I think we might be talking about different features. Just because a node is on the LAN doesn’t mean it is necessarily to be trusted. For example with a trusted node, headers received could be regarded as checkpointed and therefore it could skip the validation that the trusted node has already done.

laanwj commented at 4:55 pm on March 1, 2016: member

The point of local node discovery would be that a LAN node is probably faster, and less costly to use (no external bandwidth usage). Trust should be separate from it.

Azulan commented at 3:02 am on December 2, 2016: none

Just want to comment that I am once again having a problem with local nodes not talking to each other and therefore not syncing faster over a local network. This is a very frustrating experience, even for an advanced user like me.

Azulan commented at 1:57 pm on January 11, 2017: none

I got it to work! I just had to change from a bitcoin core node to a bitcoin unlimited node. Much better!

rebroad commented at 4:13 am on January 26, 2017: contributor

@Azulan What does unlimited do differently in this regard?

laanwj removed the label Priority Low on Dec 6, 2017

MarcoFalke commented at 6:06 pm on May 8, 2020: member

The feature request didn’t seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

Closing due to lack of interest. Pull requests with improvements are always welcome.

MarcoFalke closed this on May 8, 2020

DrahtBot locked this on Feb 15, 2022

Local Node Discovery Mechanism #3802