net: Hardcoded seeds update for 0.20 #18506

1. 
   laanwj commented at 2:26 pm on April 2, 2020: member

   Update hardcoded seeds from http://bitcoin.sipa.be/seeds.txt.gz, according to release process.

   Output from makeseeds.py:

    0  IPv4   IPv6  Onion Pass                                               
    11364173 244127   2454 Initial
    21364173 244127   2454 Skip entries with invalid address
    31129552 213117   2345 After removing duplicates
    41129548 213117   2345 Skip entries from suspicious hosts
    5338216 191944   2249 Enforce minimal number of blocks
    6336851 188993   2189 Require service bit 1
    7  6998   1520    150 Require minimum uptime
    8  5682   1290     89 Require a known and recent user agent
    9  5622   1279     89 Filter out hosts with multiple bitcoin ports
   10   512    146     89 Look up ASNs and limit results per ASN and per net
   

2. laanwj added the label P2P on Apr 2, 2020
3. laanwj added this to the milestone 0.20.0 on Apr 2, 2020
4. 
   laanwj commented at 2:43 pm on April 2, 2020: member

   Oops, need to update the user agent strings. done
5. 
   hebasto commented at 2:45 pm on April 2, 2020: member

   Are non-standard ports ok?
6. laanwj force-pushed on Apr 2, 2020
7. 
   laanwj commented at 2:48 pm on April 2, 2020: member

   Are non-standard port ok?

   Yes, that’s ok. Might help when ISPs block the standard port.

8. 
   jonasschnelli commented at 3:29 pm on April 2, 2020: contributor

   utACK 5dc165fe5be12b7a43d89137dbfe64b806aaf310 - manually verified some added, removed and kept IPv4 and IPv6 against my seed dump.
9. 
   sipa commented at 4:53 pm on April 2, 2020: member

   It is suspicious that the change in thd set of onion hosts is almost entirely red.
10. 
    laanwj commented at 5:11 pm on April 2, 2020: member

    Last time there was also a problem with onions and the DNS seed (#16999). I guess we can skip the seeds update if this is strictly worse.
11. 
    laanwj commented at 5:19 pm on April 2, 2020: member

    @sipa Now that you say, I was surprised that adding 0.19.x to the user agents didn’t result in any more onion addresses in the final result (still only 70, it did incerase the number for IPv4 though). As if the list is stale?

    Maybe if other people with DNS seeds post their list, I can combine them again and see if that gives more active onion addresses.

12. 
    sipa commented at 5:33 pm on April 2, 2020: member

    @laanwj Ugh. It seems my crawler wasn’t running with Tor enabled. Fixed.

    This probably means we’ll want to use someone else’s data (or even better, combine the results).

13. 
    laanwj commented at 5:42 pm on April 2, 2020: member

    @TheBlueMatt @luke-jr @cdecker @Sjors @petertodd @Emzy @jonasschnelli if possible please upload your DNS seed dump and post it here,especially if your crawler is running with Tor
14. 
    RandyMcMillan commented at 5:46 pm on April 2, 2020: contributor

    Are non-standard ports ok?

    I believe non-standard ports should be encouraged as a fail over. Port blocking on the backbone should be anticipated.

15. 
    sipa commented at 5:56 pm on April 2, 2020: member

    Our outgoing connection selection is very strongly avoiding non-standard ports (something that’s been in the codebase for ages BTW; the original intent was to avoid turning the Bitcoin network into a DDoS service for non-Bitcoin ports).
16. 
    luke-jr commented at 6:27 pm on April 2, 2020: member

    http://luke.dashjr.org/programs/bitcoin/files/charts/seeds.txt (162 MB, no Tor)
17. 
    Emzy commented at 7:41 pm on April 2, 2020: contributor

    @TheBlueMatt @luke-jr @cdecker @Sjors @petertodd @Emzy @jonasschnelli if possible please upload your DNS seed dump and post it here,especially if your crawler is running with Tor

    seeds-emzy.txt.gz

    I have deleted the dnsseed.dump an hour ago. So there are only 23 onion addresses in the dump. I will upload the file again later if I got more.

18. 
    Emzy commented at 10:02 pm on April 2, 2020: contributor

    seeds-emzy.txt.gz with 67 .onion
19. 
    Emzy commented at 11:21 am on April 3, 2020: contributor

    seeds-emzy.txt.gz now with 369 .onion
20. 
    Sjors commented at 12:47 pm on April 3, 2020: member

    dnsseed.dump.tar.gz with 1394 onions
21. laanwj force-pushed on Apr 3, 2020
22. net: Hardcoded seeds update for 0.20

    Update hardcoded seeds from seeds_emzy.txt seeds_lukejr.txt
    seeds_sipa.txt seeds_sjors.txt, according to release process.
    
    Output from makeseeds.py:
    ```
      IPv4   IPv6  Onion Pass
    1364173 244127   2454 Initial
    1364173 244127   2454 Skip entries with invalid address
    1129552 213117   2345 After removing duplicates
    1129548 213117   2345 Skip entries from suspicious hosts
    338216 191944   2249 Enforce minimal number of blocks
    336851 188993   2189 Require service bit 1
      6998   1520    150 Require minimum uptime
      5682   1290     89 Require a known and recent user agent
      5622   1279     89 Filter out hosts with multiple bitcoin ports
       512    146     89 Look up ASNs and limit results per ASN and per net
    ```

    0eeb0468e7
23. laanwj force-pushed on Apr 3, 2020
24. 
    laanwj commented at 2:30 pm on April 3, 2020: member

    Ok, re-did the process with the concatenated files and updated the statistics. Hopefully it’s better now (there’s a few more onions).
25. 
    Sjors commented at 6:01 pm on April 3, 2020: member

    I’m only seeing two green onions in a see of red onions. Maybe loosen the criteria for onion nodes a bit? They don’t have amazing up time as seen from my seed, but Tor itself (or my connection to it) might be flaky too.
26. 
    laanwj commented at 6:05 pm on April 3, 2020: member

    The uptime requirements for onions are already much more lenient than for IPv4 peers (10% instead of 50%). I could bring them down even more, but, do we need that many hardcoded onion seeds? Seems 89 is pretty ok to me.

    (Also, can we do changes to the seeds scripts between release please? the issue for that is #17020. Everyone seems to forget about it every time until the hurry before a release branch-off)

27. 
    Sjors commented at 6:12 pm on April 3, 2020: member

    5% uptime instead of 10% would 3x the number of nodes from my list, but I agree 89 sounds good enough, and 10% is quite bad already.

    Everyone seems to forget about it every time until the hurry before a release branch-off

    Guilty as charged :-)

28. laanwj merged this on Apr 6, 2020
29. laanwj closed this on Apr 6, 2020

    ---

30. laanwj referenced this in commit 95bde34a71 on Nov 3, 2020
31. DrahtBot locked this on Feb 15, 2022

Contributors
laanwj hebasto jonasschnelli sipa RandyMcMillan luke-jr Emzy Sjors

Labels
P2P

Milestone
0.20.0