I’ve followed the steps in https://github.com/bitcoin/bitcoin/blob/master/contrib/seeds/README.md but it seems all the hardcoded onion peers except for two are being erased, for some reason.
Do not merge as-is (might be ready after including results from multiple seeds)
Added some debugging of the funnel
0Initial: IPv4 407890, IPv6 56492, Onion 715
1Skip entries with valid address: IPv4 407890, IPv6 56492, Onion 715
2Skip entries from suspicious hosts: IPv4 407889, IPv6 56492, Onion 715
3Enforce minimal number of blocks: IPv4 104845, IPv6 48821, Onion 617
4Require service bit 1: IPv4 104650, IPv6 48759, Onion 607
5Require at least 50% 30-day uptime: IPv4 4961, IPv6 966, Onion 19
6Require a known and recent user agent: IPv4 3408, IPv6 628, Onion 2
7Filter out hosts with multiple bitcoin ports: IPv4 3402, IPv6 624, Onion 2
8ERR: Could not resolve ASN for "91.195.42.134"
9ERR: Could not resolve ASN for "150.187.36.233"
10Look up ASNs and limit results, both per ASN and globally: IPv4 512, IPv6 624, Onion 2
These are the onion-killers:
seeds.txt.gz files from the vintage point of other crawlers, to see if this drop in onion nodes and their reliability is an actual thing
@TheBlueMatt @luke-jr @cdecker @jonasschnelli @petertodd @Sjors @emzy
Mine is here: https://luke.dashjr.org/programs/bitcoin/files/charts/data/seeds.txt (161 MB)
(And mine has never done onion)
Widen range of versions.
Simple way to handle concatenating the output of multiple DNS seeds.
Follow steps in `contrib/seeds/README.md`
Thanks, combining all those, removing duplicates, and running them through ./makeseeds.py ends up with: IPv4 512, IPv6 781, Onion 171
This is okay—the old list had 176 onion hosts, and 1092 IPv4+IPv6.
Pushed the new list.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
My seed found 587 onion nodes in the past couple of hours, 26 of which are marked “good”. Should I just upload the list here, or give it a bit more time? Would those new nodes even get added given the 30 day criterion?
Update: here you go: dnsseed.dump.gz. I made a note to make a PR in a month or so for my onion nodes.
29@@ -30,7 +30,7 @@
30 PATTERN_IPV4 = re.compile(r"^((\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})):(\d+)$")
31 PATTERN_IPV6 = re.compile(r"^\[([0-9a-z:]+)\]:(\d+)$")
32 PATTERN_ONION = re.compile(r"^([abcdefghijklmnopqrstuvwxyz234567]{16}\.onion):(\d+)$")
33-PATTERN_AGENT = re.compile(r"^(/Satoshi:0.14.(0|1|2|99)/|/Satoshi:0.15.(0|1|2|99)|/Satoshi:0.16.(0|1|2|99)/)$")
34+PATTERN_AGENT = re.compile(r"^(/Satoshi:0.14.(0|1|2|3|99)/|/Satoshi:0.15.(0|1|2|99)|/Satoshi:0.16.(0|1|2|3|99)/|/Satoshi:0.17.(0|1|2|99)/|/Satoshi:0.18.99/)$")
There is one:
Be sure to update
PATTERN_AGENTinmakeseeds.pyto include the current version, and remove old versions as necessary (at a minimum when GetDesirableServiceFlags changes its default return value, as those are the services which seeds are added to addrman with).
Should we update the 4 year old SUSPICIOUS_HOSTS list?
Maybe use a scripted-diff for the generate-seeds.py command (which produces the hard to read .h file)?
would be useful to have seeds.txt.gz files from the vintage point of other crawlers, to see if this drop in onion nodes and their reliability is an actual thing
Since my seed uses a custom crawler and DNS implementation I can’t provide the file in question, but I’m more than happy to check proposed nodes for uptime in my database :-)
-o
dnsseed.dump.zip
Should we update the 4 year old SUSPICIOUS_HOSTS list?
Possibly. I think maintaining this as part of the repository at all is a bit questionable, I don’t think bitcoin core development should focus on judging individual nodes. But if you have any suggestions, feel free to add them.
Maybe use a scripted-diff for the generate-seeds.py command (which produces the hard to read .h file)?
I think this is a good idea but maybe do this for a future release, after updating the instructions how to do this?
For now, can verify the list by running the python3 generate-seeds.py . > ../../src/chainparamsseeds.h command yourself.
Apart from that, for review (and testing) please focus on the contrib/seeds/nodes_main.txt list.
Concept ACK on removing SUSPICIOUS_HOSTS
Given the current number of nodes we should be robust also in the presence of a few bad IP:s (currently 16 listed suspicious IP:s).
Concept ACK on removing SUSPICIOUS_HOSTS
Yes—I think this is a valid discussion but please move it to a new issue. Here we just want the things for rc4 (and this was a part of the release process that was forgotten) in so that we can continue with the release :-)
-o option:
2019-04_dnsseed_0-18-0-rc3.dump.gz