net: Hardcoded seeds update for 0.21

laanwj commented at 1:32 pm on October 25, 2020: member

Stats:

 0  IPv4   IPv6  Onion Pass
 1426728  59523   7900 Initial
 2426728  59523   7900 Skip entries with invalid address
 3426728  59523   7900 After removing duplicates
 4426727  59523   7900 Skip entries from suspicious hosts
 5123226  51785   7787 Enforce minimal number of blocks
 6121710  51322   7586 Require service bit 1
 7  4706   1427   3749 Require minimum uptime
 8  4124   1098   3681 Require a known and recent user agent
 9  4033   1075   3681 Filter out hosts with multiple bitcoin ports
10   512    140    512 Look up ASNs and limit results per ASN and per net

I’ve credited RandyMcMillan for the first commit because of #20190.

There are at least enough onions this time! Number of IPv6 nodes that pass all the requirements seems similar to last time in #18506.

For the next major release we’ll want TORv3 hardcoded peers as well. This makes no sense now as there are hardly any. But it’d make sense to think about how to collect them because they cannot come from the DNS seeds.

Reviewing

02020-10-28 12:04:45     jnewbery  wumpus: Do you have any suggestions for how to review [#20237](/bitcoin-bitcoin/20237/) ?
12020-10-28 12:28:37     wumpus  jnewbery: previous PRs like it might be a guide there (#18506, [#16999](/bitcoin-bitcoin/16999/)), e.g. people could try to repeat the last step in https://github.com/bitcoin/bitcoin/tree/master/contrib/seeds#seeds and see if it ends up with the same .h file, you could also repeat the entire process but as the list of peers from the seeder will be different every time that will give a (slightly, hopefully)
22020-10-28 12:28:37     wumpus  different output
32020-10-28 12:49:40     wumpus  testing what part of the peers are connectable is also useful
42020-10-28 12:51:05     wumpus  or to go deeper, whether most part of the nodes are 'good nodes' and not say spy nodes, but i don't know what means of testing

contrib: Add new versions to makeseeds.py and update gitignore 36e875b4c5

net: Hardcoded seeds update for 0.21

Stats:

```
  IPv4   IPv6  Onion Pass
426728  59523   7900 Initial
426728  59523   7900 Skip entries with invalid address
426728  59523   7900 After removing duplicates
426727  59523   7900 Skip entries from suspicious hosts
123226  51785   7787 Enforce minimal number of blocks
121710  51322   7586 Require service bit 1
  4706   1427   3749 Require minimum uptime
  4124   1098   3681 Require a known and recent user agent
  4033   1075   3681 Filter out hosts with multiple bitcoin ports
   512    140    512 Look up ASNs and limit results per ASN and per net
```

6866259fab

laanwj added the label P2P on Oct 25, 2020

laanwj added this to the milestone 0.21.0 on Oct 25, 2020

Saibato commented at 3:14 pm on October 25, 2020: contributor

This makes no sense now as there are hardly any

There are many nodes that have there onion v3 address linked to 8333 created in /etc/tor/torrc but probably won’t tell and stay onlynet onion or don’t enable service calls port 9051 and cant use the additional v3 support from 0.21 by auto creating onions anyway.

But since almost all old nodes versions can do connect and addnode with long v3 address since years, if thre Tor software is above Tor 0.3.2.2-alpha (October 2017 ), it might be useful to have a short list core approved list they could add manual to there old nodes,

And btw. why not add the Tor v3 addr from the known seeders ( if they have Tor nodes ) in the vseeds list, i.e. when the user selects onlynet=onion to make sure we don’t seed by ADDR messages via Tor exit nodes and stay at least for virgin bootstrap in Tor, i somewhere posted a patch for chainparms.cpp but forgot where. That way. nodes with 0.21 update would fast learn from the seeders ADDRv2 messages new v3 and stay in pure in Tor if they wish. If u like i can dig for that?

 0diff --git a/src/chainparams.cpp b/src/chainparams.cpp
 1index ffd2076c9..4903e3bda 100644
 2--- a/src/chainparams.cpp
 3+++ b/src/chainparams.cpp
 4@@ -10,6 +10,7 @@
 5 #include <tinyformat.h>
 6 #include <util/system.h>
 7 #include <util/strencodings.h>
 8+#include <util/system.h>
 9 #include <versionbitsinfo.h>
10 
11 #include <assert.h>
12@@ -113,6 +114,7 @@ public:
13         // This is fine at runtime as we'll fall back to using them as an addrfetch if they don't support the
14         // service bits we want, but we should get them updated to support all service bits wanted by any
15         // release ASAP to avoid it where possible.
16+        if (gArgs.GetArg("-onlynet", "") != "onion") {
17         vSeeds.emplace_back("seed.bitcoin.sipa.be"); // Pieter Wuille, only supports x1, x5, x9, and xd
18         vSeeds.emplace_back("dnsseed.bluematt.me"); // Matt Corallo, only supports x9
19         vSeeds.emplace_back("dnsseed.bitcoin.dashjr.org"); // Luke Dashjr
20@@ -122,6 +124,11 @@ public:
21         vSeeds.emplace_back("seed.bitcoin.sprovoost.nl"); // Sjors Provoost
22         vSeeds.emplace_back("dnsseed.emzy.de"); // Stephan Oeste
23         vSeeds.emplace_back("seed.bitcoin.wiz.biz"); // Jason Maurice
24+        };
25+
26+        if (gArgs.GetArg("-onlynet", "") == "onion") {
27+        vSeeds.emplace_back("seedersv3onionaddressasdasdasdasdad.onion:8333"); // Add seeder onion v3 address
28+       // since we can not seed over Tor UDP DNS and call anyway the first node from the return +       // the exit socket  the exit Tor node give us for the FQDN of normal seeders, so why not call    
29+      //if onlynet=onion the Tor seeder node and gossip from ADDRv2 or ADDR,
30+        }
31 
32         base58Prefixes[PUBKEY_ADDRESS] = std::vector<unsigned char>(1,0);
33         base58Prefixes[SCRIPT_ADDRESS] = std::vector<unsigned char>(1,5);

Saibato commented at 10:36 pm on October 25, 2020: contributor

concept ptACK Tested only the Tor nodes and so far all on this new list encoded and decoded at boostrap and where except two reachable.

Tyi, those two on the list

04vorvtoyegh4zbvr.onion 8333
1h6a32n4blbwwyn4d.onion 8333

i was unable to reach.

laanwj commented at 10:16 am on October 27, 2020: member

@Saibato Thanks for testing! Yes, I’d assume a part of them will be unreachable at any single time, I’m surprised the other 510 were connectable.

Saibato commented at 1:39 pm on October 27, 2020: contributor

I’m surprised the other 510 were connectable

Me 2 , i did run the test twice, at first thought also impossible, but that was the case with a ten second delay between next address connect in the list. But its v2 so there could be nodes doubles we wont be aware with the easy connect 8333 check i did.

Btw. a big improvement compared to the 3 working fixed that we only have now in 0.20

jonatack commented at 5:33 pm on November 2, 2020: member

The changes look fine. I’m now running the instructions in contrib/seeds/README.md to see what I get. Edit: just saw the review tips in the PR description.

 0  IPv4   IPv6  Onion Pass                                               
 1427708  59843   8114 Initial
 2427708  59843   8114 Skip entries with invalid address
 3427708  59843   8114 After removing duplicates
 4427707  59843   8114 Skip entries from suspicious hosts
 5124199  52104   8001 Enforce minimal number of blocks
 6122607  51619   7794 Require service bit 1
 7  4769   1456   3830 Require minimum uptime
 8  4180   1126   3762 Require a known and recent user agent
 9  4089   1103   3762 Filter out hosts with multiple bitcoin ports
10   512    143    512 Look up ASNs and limit results per ASN and per net

Ran makeseeds.py a second time to see the variance.

 0  IPv4   IPv6  Onion Pass                                               
 1427693  59849   8131 Initial
 2427693  59849   8131 Skip entries with invalid address
 3427693  59849   8131 After removing duplicates
 4427692  59849   8131 Skip entries from suspicious hosts
 5124185  52110   8018 Enforce minimal number of blocks
 6122597  51627   7810 Require service bit 1
 7  4752   1456   3842 Require minimum uptime
 8  4165   1126   3774 Require a known and recent user agent
 9  4076   1103   3774 Filter out hosts with multiple bitcoin ports
10   512    147    512 Look up ASNs and limit results per ASN and per net

jonatack commented at 7:25 pm on November 2, 2020: member

ACK 6866259fabd8580c92693703d80773428aca8175

reviewed the changes locally
ran python3 generate-seeds.py . > ../../src/chainparamsseeds.h from /contrib/seeds on top of this patch and it made no change
ran all the steps in /contrib/seeds/README.md, which created this diff: https://github.com/jonatack/bitcoin/commit/084717e259005d201b152bd1a90eb1ae852ecac6
returned to the original patchset and repeated the generate seeds step again to be sure

fanquake deleted a comment on Nov 2, 2020

vasild commented at 9:31 am on November 3, 2020: member

I confirm that running python3 generate-seeds.py . > ../../src/chainparamsseeds.h locally generates identical result.

Re-downloading the seeds and running python3 makeseeds.py < seeds_main.txt > nodes_main.txt changes nodes_main.txt about 12% (nodes_main.txt is 1167 lines and that command removed 137 lines and added 140).

Maybe as a future improvement we may devise some deterministic way to generate the seeds, so that everybody can generate seeds.txt and confirm http://bitcoin.sipa.be/seeds.txt.gz is not compromised. Or at least that there is a significant overlap between that file from bitcoin.sipa.be and a locally generated.

laanwj commented at 2:00 pm on November 3, 2020: member

I confirm that running python3 generate-seeds.py . > ../../src/chainparamsseeds.h locally generates identical result. @jonatack @vasild Thanks for checking !

Maybe as a future improvement we may devise some deterministic way to generate the seeds, so that everybody can generate seeds.txt and confirm http://bitcoin.sipa.be/seeds.txt.gz is not compromised. Or at least that there is a significant overlap between that file from bitcoin.sipa.be and a locally generated.

Yes, maybe. Given the same seeds.txt.gz it should be deterministic already. Well, I guess the ASN mapping could change… I could upload mine somewhere but I don’t think it would prove anything. The thing is that the seeds.txt.gz on the server changes continuously, and there is no way to prove that mine was ever the current state there and hasn’t been tampered with.

The topic of improvements to the seed generation comes up every time just before the release and then dies down immediately after it. I’m happy for someone to put effort in this though. See also #17020 which ahs quite the TODO list alrady :smile:

laanwj merged this on Nov 3, 2020

laanwj closed this on Nov 3, 2020

laanwj referenced this in commit cddcd22ab3 on Nov 19, 2020

sidhujag referenced this in commit 0c1c7970e8 on Nov 19, 2020

DrahtBot locked this on Feb 15, 2022

net: Hardcoded seeds update for 0.21 #20237

Reviewing