seeds: Pull additional nodes from my seeder and update fixed seeds

achow101 commented at 8:05 pm on April 30, 2024: member

The DNS seeder that I wrote collects statistics on node reliability in the same way that sipa’s seeder does, and also outputs this information in the same file format. Thus it can also be used in our fixed seeds update scripts. My seeder additionally crawls onion v3, i2p, and cjdns, so will now be able to set those fixed seeds automatically rather than curating manual lists.

In doing this update, I’ve found that makeseeds.py is missing newer versions from the regex as well as cjdns support; both of these have been updated.

I also noticed that the testnet fixed seeds are all manually curated and sipa’s seeder does not appear to publish any testnet data. Since I am also running the seeder for testnet, I’ve added the commands to generate testnet fixed seeds from my seeder’s data too.

Lastly, I’ve updated all of the fixed seeds. However, since my seeder has not found any cjdns nodes that met the reliability criteria (possibly due to connectivity issues present in those networks), I’ve left the previous manual seeds for that network.

DrahtBot commented at 8:05 pm on April 30, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	fjahr, virtu
Concept ACK	jaonoctus

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#30695 ([WIP] seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes by virtu)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

laanwj added the label P2P on Apr 30, 2024

in contrib/seeds/nodes_main_manual.txt:513 in ef78deb080 outdated

509@@ -510,518 +510,6 @@ zvchlrjuzqdlx37fhibhnym4y6p56vtlymujjuzhh2cp34yqfrtq.b32.i2p:0
510 zxsd3fqczh6ddgejc24nnmb3ww7nalieq3a7cs2mqiy6tmff3wia.b32.i2p:0
511 zy2ywvyqds5bgdoo4tgbu3bwjp3ygyn3zfuby44jemc6xa6fbwta.b32.i2p:0
512 zzre44vh766jgfordw2ehu2r6p44j23uyovgvm7iwuhp3g5iz4ca.b32.i2p:0
513-ycdw2e4ufgfwhcqna4g3m2qsvaly23ozaexawcj3x4gtgcehgwujjgid.onion:8333

jonatack commented at 8:27 pm on April 30, 2024:

It looks like something odd happened to the manual onion and i2p seeds. Only a small range of first letters were present, and seeds run by colleagues and the bitcoin community were no longer present.

achow101 commented at 9:05 pm on April 30, 2024:

They were updated in #29561, and the addresses were pulled from my node’s addrman. Some sorting happened somewhere, and because makeseeds.py doesn’t shuffle (this PR adds a commit that does that), when it applied the max node count, it ended up with the tail end of that list.

There seem to be sufficient i2p and onion nodes now that there is no need to specifically include nodes run by known people. We don’t do this for IPv4 or IPv6.

jonatack commented at 9:51 pm on June 26, 2024:

Yes, I see that your script removed my onion and i2p nodes then and also here.

DrahtBot added the label Needs rebase on May 13, 2024

achow101 force-pushed on May 14, 2024

achow101 commented at 3:45 am on May 14, 2024: member

My seeder has now found several i2p nodes, so I’ve gone ahead and removed the manually curated ones for both mainnet and testnet. These are now filled in by the script. The only manual ones remaining are cjdns. However, as my seeder has also found cjdns nodes, they have been added, but are only a couple.

DrahtBot removed the label Needs rebase on May 14, 2024

achow101 force-pushed on Jun 5, 2024

makeseeds: Update user agent regex

Update the user agent regex to match all 3 digits of the version number,
not just the first 2 digits.

Also updates it to include 24.2, 25.2, 26.1, 27.0, 27.1, 27.99, 28.0 and
28.99.

d5a8c4c4bd

makeseeds: Support CJDNS af550b3a0f

makeseeds: Shuffle ips after parsing

The crawlers are not guaranteed to output nodes in a random order, so
shuffle the ips list after parsing to break any biasing that may be
caused by the output order.

d2465dfac6

achow101 force-pushed on Aug 14, 2024

achow101 commented at 5:22 pm on August 14, 2024: member

Updated for testnet4. I don’t think there are any seeders publishing testnet4 data yet, so I’ve just used the fixed seeds in chainparamsseeds.h and turned them into a nodes_testnet4.txt, in addition to adding instructions.

Also refreshed the seeds for pre-28.0.

achow101 added this to the milestone 28.0 on Aug 14, 2024

in contrib/seeds/makeseeds.py:201 in 71c91db172 outdated

197@@ -198,6 +198,7 @@ def parse_args():
198     argparser = argparse.ArgumentParser(description='Generate a list of bitcoin node seed ip addresses.')
199     argparser.add_argument("-a","--asmap", help='the location of the asmap asn database file (required)', required=True)
200     argparser.add_argument("-s","--seeds", help='the location of the DNS seeds file (required)', required=True)
201+    argparser.add_argument("-m", "--minblocks", help="The minimum number of blocks each node must have", default=MIN_BLOCKS, type=int)

fjahr commented at 9:59 am on August 16, 2024:

nit: If we want this MIN_BLOCKS to default to mainnet then you could update it to 840_000 while you make edits here. But maybe it would be better to have a more concise behavior here in the future. I guess the file could have a header with the network, number of min blocks that makes sense etc. based on the numbers the seeder node has.

achow101 commented at 3:30 pm on August 16, 2024:

Updated MIN_BLOCKS.

The seeders don’t necessarily have their own node.

in contrib/seeds/nodes_testnet4.txt:1 in 6c2a87fb1c outdated

0@@ -0,0 +1,9 @@
1+18.189.156.102:48333 # AS16509

fjahr commented at 10:01 am on August 16, 2024:

nit: Typo in the commit message of 6c2a87fb1ccf26daa754e6d5a4c4ae687da562e3: “tesnet4”

achow101 commented at 3:30 pm on August 16, 2024:

Fixed

in contrib/seeds/README.md:22 in 1e0c478889 outdated

13@@ -14,8 +14,11 @@ data. Run the following commands from the `/contrib/seeds` directory:
14 ```
15 curl https://bitcoin.sipa.be/seeds.txt.gz | gzip -dc > seeds_main.txt
16 curl https://mainnet.achownodes.xyz/seeds.txt.gz | gzip -dc >> seeds_main.txt
17+curl https://testnet.achownodes.xyz/seeds.txt.gz | gzip -dc > seeds_test.txt
18 curl https://bitcoin.sipa.be/asmap-filled.dat > asmap-filled.dat
19 python3 makeseeds.py -a asmap-filled.dat -s seeds_main.txt > nodes_main.txt
20 cat nodes_main_manual.txt >> nodes_main.txt
21+python3 makeseeds.py -a asmap-filled.dat -s seeds_test.txt > nodes_test.txt
22+python3 makeseeds.py -a asmap-filled.dat -s seeds_test.txt -m 30000 > nodes_testnet4.txt

fjahr commented at 10:12 am on August 16, 2024:

Hm, either I’m missing something or this would mean that the testnet3 nodes would still be included here? Since you say this isn’t really working yet (how I understand this here) maybe rather put a TODO here instead?

achow101 commented at 3:30 pm on August 16, 2024:

I’ve commented it out and added a todo.

in contrib/seeds/README.md:18 in 7f55140007 outdated

14 
15 ```
16 curl https://bitcoin.sipa.be/seeds.txt.gz | gzip -dc > seeds_main.txt
17+curl https://mainnet.achownodes.xyz/seeds.txt.gz | gzip -dc >> seeds_main.txt
18+curl https://testnet.achownodes.xyz/seeds.txt.gz | gzip -dc > seeds_test.txt
19 curl https://bitcoin.sipa.be/asmap-filled.dat > asmap-filled.dat

fjahr commented at 10:16 am on August 16, 2024:

How up-to-date is that ASMap file? If it’s not up updated regularly it might be better to use one of the ones available at https://github.com/fjahr/asmap-data. Maybe this is a change in process that requires more eyes on it and I wouldn’t call this a blocker then but I would like to propose this as a goal for the next release to use something more recent.

cc: @sipa

sipa commented at 1:02 pm on August 16, 2024:

The file is from May 2022. No reason to keep using it; there is no inherent reason why it should be considered more reliable than yours.

achow101 commented at 3:31 pm on August 16, 2024:

Yes, if it’s old, we should use a more up to date one. Can that repo somehow have a permalink to the most recent asmap? Otherwise we would run into the same problem with outdated asmaps.

fjahr commented at 4:11 pm on August 16, 2024:

Can that repo somehow have a permalink to the most recent asmap? Otherwise we would run into the same problem with outdated asmaps.

I hadn’t really thought about this so far. This is a bit hacky but for now I copied the latest asmap file there to latest_asmap.dat and I would update that file as we create new maps so this link should always point to the latest file. I will look into making this a bit nicer by having latest.asmap.org (page is still WIP) link to that file automatically but that shouldn’t block this PR here. I can update this as a follow-up.

achow101 commented at 4:35 pm on August 16, 2024:

I think you can make a symlink in the repo and just make sure it’s updated every time.

achow101 commented at 4:53 pm on August 16, 2024:

Changed to use that asmap, although did not regenerate the seeds.

fjahr commented at 2:05 pm on August 18, 2024:

I think you can make a symlink in the repo and just make sure it’s updated every time.

I tried this but it seems like it’s not supported. I didn’t find an official statement but here is a discussion that seems to confirm there is not way to use symlinks for this: https://github.com/dear-github/dear-github/issues/156

fjahr commented at 10:38 am on August 16, 2024: contributor

tACK 7f55140007186cda876ad0a5da812e391cddbcc4

I reviewed the code and the changes look correct to me. I tested that the updated instructions in the README work as expected and they did (though see my comment on the testnet4 line). The resulting files showed some reasonable differences from the files included here, which is expected. I also confirmed that generating chainparamsseeds.h from the txt files included here yields the same result.

makeseeds: Configurable minimum blocks for testnet4's smaller chain 5bab3175a6

seeds: Also pull from achow101 seeder 0676515397

seeds: Add testnet instructions ed5b86cbe4

seeds: Remove manual onion and i2p seeds

The seeders now produce onion and i2p seeds, so there is no need to keep these
in the manual list.

Although should also be produced, there are not enough
good ones detected by the seeder, so we keep the manual seeds for them.

8ace71c737

seeds: Add testnet4 fixed seeds file f1f24d7214

seeds: Fixed seeds update

Update the fixed seeds for both mainnet and testnet

d8fd1e0faf

achow101 force-pushed on Aug 16, 2024

seeds: Use fjahr's more up to date asmap 41ad84a00c

fjahr commented at 9:55 pm on August 17, 2024: contributor

re-ACK 41ad84a00c20f54b520aab7f6f975231da0ee2d0

Only changes were addressing above (minor) comments: https://github.com/bitcoin/bitcoin/compare/7f55140007186cda876ad0a5da812e391cddbcc4..41ad84a00c20f54b520aab7f6f975231da0ee2d0

virtu commented at 9:48 am on August 19, 2024: contributor

ACK 41ad84a

Reviewed the code; changes look fine. Also in favor of using the regularly-updated asmap from collaborative runs.

I noticed the seeds.txt.gz file I used (~2024-08-19T08:30Z) file did not contain any good I2P nodes.

Also, since I’ve been working on on exporting my crawler’s results as well, I noticed the Onion node numbers seem rather low. Here’s some statistics that I generated by skipping the final stage of the makeseeds.py script (so as to not apply the per-AS and per-network limit, thus retaining all viable nodes), applying the script to all input sources individually, and comparing the resulting addresses.

Network	Addresses (before limits)	Overlap
IPv4	sipa=1714, ava=2886, virtu=4554	sipa-ava=704, sipa-virtu=1684, ava-virtu=2355, sipa-ava-virtu=687
IPv6	sipa=469, ava=583, virtu=1203	sipa-ava=178, sipa-virtu=460, ava-virtu=523, all=176
Onion	ava=443, virtu=11384	ava-virtu=388
CJDNS	ava=2, virtu=9	ava-virtu=2

achow101 commented at 3:36 pm on August 19, 2024: member

I noticed the seeds.txt.gz file I used (~2024-08-19T08:30Z) file did not contain any good I2P nodes.

I think it got disconnected from I2P and started marking every I2P node as down when it was actually that the crawler’s I2P connection was down. That’s something I’ll need to fix.

jaonoctus approved

jaonoctus commented at 6:17 pm on August 26, 2024: none

utACK

achow101 referenced this in commit 17071e47f5 on Aug 26, 2024

achow101 merged this on Aug 26, 2024

achow101 closed this on Aug 26, 2024

virtu commented at 5:53 am on August 27, 2024: contributor

just rebased #30695 since #30008 got merged and noticed it accidentally removes all ~hardcoded~ onion and i2p seeds in src/chainparamsseeds.h (and seeds_main.txt).

[edited for clarity]

fjahr commented at 10:39 am on August 27, 2024: contributor

it accidentally removes all hardcoded onion and i2p seeds

Wasn’t that the plan? From the description:

My seeder additionally crawls onion v3, i2p, and cjdns, so will now be able to set those fixed seeds automatically rather than curating manual lists.

virtu commented at 10:46 am on August 27, 2024: contributor

it accidentally removes all hardcoded onion and i2p seeds

Wasn’t that the plan? From the description:

My seeder additionally crawls onion v3, i2p, and cjdns, so will now be able to set those fixed seeds automatically rather than curating manual lists.

“hardcoded” was a poor choice, I guess. Updated my OP to avoid further confusion.

I wasn’t referring to the removal of the manually curated addresses in nodes_main_manual.txt but to the fact that there won’t be any Onion or I2P seeds hardcoded into the binary at all (via src/chainparamsseeds.h), which I believe is an accident.

achow101 commented at 3:19 pm on August 27, 2024: member

I wasn’t referring to the removal of the manually curated addresses in nodes_main_manual.txt but to the fact that there won’t be any Onion or I2P seeds hardcoded into the binary at all (via src/chainparamsseeds.h), which I believe is an accident.

Yes, just noticed that too.

I’ve added #30695 to the milestone to get those added back in.

seeds: Pull additional nodes from my seeder and update fixed seeds #30008

Code Coverage

Reviews

Conflicts