[net] Don't use third-party "what is my IP" services. #3088

This patch eliminates the privacy and reliability problematic use of centralized web services for discovering the node's addresses for advertisement.

The Bitcoin protocol already allows your peers to tell you what IP they think you have, but this data isn't trustworthy since they could lie. So the challenge is using it without creating a DOS vector.

To accomplish this we adopt an approach similar to the one used by P2Pool: If we're announcing and don't have a better address discovered (e.g. via UPNP) or configured we just announce to each peer the address that peer told us. Since peers could already replace, forge, or drop our address messages this cannot create a new vulnerability... but if even one of our peers is giving us a good address we'll eventually make a useful advertisement.

This is not ready for merging yet. I've done some basic testing— instrumenting announcement and seeing that it worked like expected, but before I put a lot more time into testing it (setting up a simulated network and such) I wanted to get some feedback on the general approach of giving the peer back the address it told us.

(also, this patch was bluntly forward ported from the older version that I tested— I've not run this particular version, though it does build)

There is a bunch of code motion I've included here for some of the heuristics in deciding to use the our current estimate vs the peers report. I'm not married to it, and perhaps it would be better removed and a more simpler behavior adopted.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/a851bf84f6a2ff95bd86b23e55bb0647f5f47188 for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

This seems like a priority because of the privacy and reliability implications. Will donate boxes and/or cloud time for testing. Is there anything in the request that let's the service know that its a Bitcoin node asking for an IP?

What will happen if both of the listed services 'checkip.dyndns.org' and 'www.showmyip.com' are down?

Bitcoind advertizes as 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)', so no - it doesn't really let the site know anything. Of course, there could be other data that gives it away (combination with http protocol, closing of connections, timings, ...).

If the external ip services are down, then we don't know our own IP and can't advertize it. No big deal, but makes your node hard or impossible to find for others.

Well since these seem to be well used services that should make it fairly difficult. Volunteer services won't help either because they will only be used Bitcoin (which makes it easy to just run some honeypot nodes), and will require maintaining some sort of list of available look-up services.

Seems like the immediate band-aid would be adding some more services until this method is readying for merging.

@super3 We haven't considered it super high priority in the past because there are many other signals that give away a Bitcoin node on the network... and if you turn off listening or switch to using tor you turn off most (all in the case of Tor, hopefully) of them as as well as this one.

There were several other points of network centralization before, and we've been chewing away at them. This is the next obvious one to get rid of...

I need to look closer at the code changes, but I have little doubt this will be ready for 0.9.

This is nice to have, if we don't need to depend on centralized services we shouldn't.

Approach sounds good. Can you write a test plan?

Lying about a user-agent isn't gonna help anyway (trust me, I spent years shooting ua-forging fish in a barrel). The decentralisation aspects are more important.

ACK on approach.

Though it's worth considering how we could have had a system of signed per-node identities where you would sign your advertisement - including what you think is your IP/tor/whatever - and in addition to that nodes that relayed your advertisement could sign their own messages saying what they thought your globally reachable address was. But that's just food-for-thought - what you're doing is the right thing to do for now.

Why would one want signed per-node identities? For what would nodes want to identify themselves at all, isn't the idea that nodes are as indistinguishable and exchangeable as possible?

@laanwj One example is for anti-sybil protection by associating an identity with p2pool proofs-of-work, or by simply purchasing a fidelity bond. Once you make the identity expensive to obtain, you can then use fraud proofs - for instance rather than having an expensive UTXO commitment system and associated soft-fork you can just as easily ask nodes to make signed statements as to what transactions match a given filter, and if a different peer gives a different answer, construct a compact proof that one of them was lying and thus destroy that identity. This will work particularly well in conjunction with micropayment systems for SPV node services.

Would this be more informative if it logged when it transmits an advertisement? Otherwise it is difficult to tell if it is working at all. The log could also be informative in showing you if a peer lied to you.

Does bitcoin currently handle multi-homing? The logging would also be informative here to show advertisements of multiple external IP's.

https://github.com/wtogami/bitcoin/commits/0.8.5-externalip Backport to Bitcoin 0.8.5.

receive version message: version 70001, blocks=266272, us=XX.XX.XX.XX:8333, them=0.0.0.0:0, peer=YY.YY.YY.YY:60127

The 0.0.0.0 is expected and seemingly harmless when the remote host is using the externalip patch. But if they explicitly set their own public address with externalip= it continues to be self-reported as 0.0.0.0. Harmless but expected?

Both. The design here is specifically to not start advertising whatever addresses untrusted peers give us to other peers.

https://bitcointalk.org/index.php?topic=320695.0 If end-users want to help testing of this patch, a backport is included in this build of Bitcoin 0.8.5

Rebased version: https://github.com/laanwj/bitcoin/tree/2013_11_rebase_no_thirdparty_ip

https://bitcointalk.org/index.php?topic=320695.0 I have been testing a backport of this patch against Bitcoin 0.8.5. It seems to advertise as expected if the node has a public IP address, but if behind NAT in RFC1918 space, the remote peer does not seem to receive any addr advertisement. It is possible that my backport was wrong or I'm testing it incorrectly.

A peer running this patch for 3 days with port 8333 forwarded to a fresh IPv4 address has zero incoming connections, further suggesting it is not working.

Why was this merged? The comment by @wtogami suggests it doesn't even work as intended.

It was not merged.

Be nice to separate out, and go ahead and push, the checkpoints:: changes inside this. Shrink the patch, make it easier to read, and get the obvious cleanups upstream immediately.

I'm all for cherry picking the sane changes and get them in also.

All the changes are sane, but we need to debug @wtogami's problem first before everything can be merged.

I'll have a try at rebasing this and splitting this up between checkpoints changes and the rest.

See #3461