contrib: Use asmap for ASN lookup in makeseeds

laanwj commented at 3:28 pm on April 15, 2022: member

Add an argument -a to provide a asmap file to do the IP to ASN lookups.

This speeds up the script greatly, and makes the output deterministic. Also removes the dependency on dns.lookup.

I’ve annotated the output with ASxxxx comments to provide a way to verify the functionality.

For now I’ve added instructions in README.md to download and use the demo.map from the asmap repository. When we have some other mechanism for distributing asmap files we could switch to that.

This continues #24824. I’ve removed the fallbacks and extra complexity, as everyone will be using the same instructions anyway.

Co-authored-by: Pieter Wuille pieter.wuille@gmail.com Co-authored-by: russeree reese.russell@ymail.com

laanwj added the label Scripts and tools on Apr 15, 2022

jonatack commented at 3:37 pm on April 15, 2022: contributor

Concept ACK.

(Python linter says to remove unused headers.)

laanwj force-pushed on Apr 15, 2022

DrahtBot commented at 6:38 am on April 16, 2022: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

in contrib/seeds/makeseeds.py:154 in 08ecc04e5d outdated

161+        if asn is None or asn_count[ip['net'], asn] == max_per_asn[ip['net']]:
162             # do not add this ip as we already have too many
163             # ips from this ASN on this network
164             continue
165-        asn_count[asn] += 1
166+        asn_count[ip['net'], asn] += 1

laanwj commented at 10:51 am on April 18, 2022:

btw: This fixes a bug that was exposed by the changes in #24818, where it was possible for a limit to be exceeded because max_per_asn differs per network but asn_count was not counted per network (in combination with ==). Making the counting per ipv4/ipv6 avoids this, and makes sense to me conceptually. So after this, a single ASN can have both 2 IPv4 seeds and 6 IPv6 seeds.

jonatack commented at 4:05 pm on April 18, 2022:

With this change I only see 12 entries in nodes_main.txt though (2 IPv4, 10 IPv6, all AS30765). Haven’t reviewed deeply enough yet to see if that’s a bug and where.

laanwj commented at 5:57 pm on April 21, 2022:

Which asmap are you using? It seems more likely to me that your asmap is wrong, or something is wrong with parsing, causing all IPs to be assigned to one AS, than that this change is at fault.

laanwj commented at 6:00 pm on April 21, 2022:

FWIW this is my map file:

0sha256sum demo.map
194b6142b574d468e73c71ed9fb411683d295c37b756f1f735013f736ee647330  demo.map

Output:

 0Loading asmap database demo.map... Done.
 1  IPv4   IPv6  Onion Pass
 2470789  73296      0 Initial
 3470789  73296      0 Skip entries with invalid address
 4470789  73296      0 After removing duplicates
 5  6318   1747      0 Enforce minimal number of blocks
 6  5456   1504      0 Require service bit 1
 7  3866    887      0 Require minimum uptime
 8  3789    864      0 Require a known and recent user agent
 9  3760    856      0 Filter out hosts with multiple bitcoin ports
10   512    301      0 Look up ASNs and limit results per ASN and per net

russeree commented at 4:51 am on April 23, 2022:

I currently am having this same bug using the asmap from the readme.md included. curl https://github.com/sipa/asmap/raw/master/demo.map > demo.map

0sha256sum demo.map
172034b228f2b91d8c0864d27d5f8cbbad4b34f1b0948fd2bc9f1acf66aaf168c  demo.map

 0Loading asmap database demo.map... Done.
 1  IPv4   IPv6  Onion Pass
 2470888  73307      0 Initial
 3470888  73307      0 Skip entries with invalid address
 4470888  73307      0 After removing duplicates
 5470887  73307      0 Skip entries from suspicious hosts
 6  6502   1787      0 Enforce minimal number of blocks
 7  5610   1523      0 Require service bit 1
 8  3862    896      0 Require minimum uptime
 9  3777    869      0 Require a known and recent user agent
10  3753    861      0 Filter out hosts with multiple bitcoin ports
11     2     10      0 Look up ASNs and limit results per ASN and per net
12
13real    0m11.036s
14user    0m10.079s
15sys     0m0.907s

The time to parse is insanely fast but the included file is only 2.5kb which to me seems incredibly small as an uncompressed asmap is nearly 27MB.

sipa commented at 5:09 am on April 23, 2022:

@russeree If you use curl, it’ll just store “You are being redirected to …” in the file.

That demo.dap file is also very old, and wasn’t even intended back then to be complete. Try https://bitcoin.sipa.be/asmap-filled.dat.

russeree commented at 5:20 am on April 23, 2022:

0cat demo.map
1<html><body>You are being <a href="https://raw.githubusercontent.com/sipa/asmap/master/demo.map">redirected</a>.</body></html>

well then … after using the correct file things looked much better.

 0`Loading asmap database demo.map... Done.
 1  IPv4   IPv6  Onion Pass
 2470888  73307      0 Initial
 3470888  73307      0 Skip entries with invalid address
 4470888  73307      0 After removing duplicates
 5470887  73307      0 Skip entries from suspicious hosts
 6  6502   1787      0 Enforce minimal number of blocks
 7  5610   1523      0 Require service bit 1
 8  3862    896      0 Require minimum uptime
 9  3777    869      0 Require a known and recent user agent
10  3753    861      0 Filter out hosts with multiple bitcoin ports
11   512    291      0 Look up ASNs and limit results per ASN and per net
12
13real    0m15.295s
14user    0m14.423s
15sys     0m0.808s`

using a broken file presents an interesting note to me. When using the broken file the script just used “You are being redirected to …” as the ASN for ipv4 with a limit of 2. Then does ipv6 also use “You are being redirected to …” for the 10 items per ASN? So the max allocated ips per ASN is actually 12 including ipv4 and ipv6?

laanwj commented at 8:58 am on April 23, 2022:

So the max allocated ips per ASN is actually 12 including ipv4 and ipv6?

Yes. The maximum number of seeds per ASN is 12: 2 IPv4 and 10 IPv6.

I"ll update the doc, sorry for providing a curl line that doesn’t work. I thought it worked for me!

in contrib/seeds/asmap.py:10 in 08ecc04e5d outdated

0@@ -0,0 +1,90 @@
1+#!/usr/bin/env python3
2+# Copyright (c) 2013-2020 The Bitcoin Core developers
3+# Distributed under the MIT software license, see the accompanying
4+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
5+import ipaddress

laanwj commented at 10:58 am on April 18, 2022:

BTW I didn’t realize before that Python had built-in functionality to parse/manipulate IP addresses (both v4 and v6). Might make sense (but not in this PR) to use this in more places and replace all the ad-hoc hacks.

in contrib/seeds/asmap.py:11 in 08ecc04e5d outdated

 6+
 7+# Convert a byte array to a bit array
 8+def DecodeBytes(byts):
 9+    return [(byt >> i) & 1 for byt in byts for i in range(8)]
10+
11+def DecodeBits(stream, bitpos, minval, bit_sizes):

laanwj commented at 11:02 am on April 18, 2022:

The code from here came from testmap which doesn’t come with a license. It looks like @sipa and @jamesob contributed to it, so to include it we would need you to sign off that it’s acceptable to include this here under the MIT license.

sipa commented at 1:43 pm on April 18, 2022:

ACK MIT license from me.

Also, I’m currently rewriting the Python asmap code from scratch actually (to have a single class that can do creation of asmap files, decoding, lookup, diffing, have tests, …), and it’s close to being done.

jamesob commented at 1:50 pm on April 18, 2022:

ACK MIT license

laanwj commented at 4:07 pm on April 18, 2022:

Thank you!

Also, I’m currently rewriting the Python asmap code from scratch actually (to have a single class that can do creation of asmap files, decoding, lookup, diffing, have tests, …), and it’s close to being done.

OK, we can wait for that, there’s no hurry here. It would be nice to have this before 0.24 branch-off.

dunxen commented at 6:50 pm on May 22, 2022:

Also, I’m currently rewriting the Python asmap code from scratch actually (to have a single class that can do creation of asmap files, decoding, lookup, diffing, have tests, …), and it’s close to being done.

/start slightly off-topic:

Nice! Is the diffing for encoded mappings or just output from asmap-rs? Or would this even make asmap-rs obsolete? I’m in the process of making some improvements to UX there to show progress on RIPE data dump downloads and finding the ASN bottlenecks. I was also thinking of including a command there to compare non-encoded maps. I was going to use that to run some sort of monthly historical analysis over the past year to get an idea of how much the maps tend to change. Just don’t want to duplicate effort.

Just some of the things to help get asmaps distributed with releases and enabled by default. @sipa, just let me know if you want to continue this chat off GitHub or on some other issue. :)

/end slightly off-topic

sipa commented at 3:24 pm on May 31, 2022:

@dunxen Yeah, possibly. At this point the new asmap python code can do everything asmap-rs can and more (encode, decode, diff, bottleneck, …), but there is some more work around making it nicer to use, and of course questions around review and integration into processes.

jonatack commented at 4:11 pm on April 18, 2022: contributor

Tested that both -a and --asmap run, but if makeseeds no longer works without the added arg, can it be removed?

0$ python3 makeseeds.py < seeds_main.txt > nodes_main.txt
1usage: makeseeds.py [-h] -a ASMAP
2makeseeds.py: error: the following arguments are required: -a/--asmap

in contrib/seeds/README.md:11 in 08ecc04e5d outdated

10@@ -11,18 +11,7 @@ to addrman with).
11 The seeds compiled into the release are created from sipa's DNS seed data, like this:

jonatack commented at 4:13 pm on April 18, 2022:

Maybe update here

0The seeds compiled into the release are created from sipa's DNS seed and AS map data, like this:

in contrib/seeds/makeseeds.py:191 in 08ecc04e5d outdated

186 def main():
187+    args = parse_args()
188+
189+    print(f'Loading asmap database {args.asmap}... ', end='', file=sys.stderr, flush=True)
190+    asmap = ASMap(args.asmap)
191+    print(f'Done.', file=sys.stderr)

jonatack commented at 4:15 pm on April 18, 2022:

a few nitty suggestions here

0-    print(f'Loading asmap database {args.asmap}... ', end='', file=sys.stderr, flush=True)
1+    print(f'Loading asmap database "{args.asmap}"... ', end='', file=sys.stderr, flush=True)
2     asmap = ASMap(args.asmap)
3-    print(f'Done.', file=sys.stderr)
4+    print('done.\n', file=sys.stderr)

laanwj commented at 4:20 pm on April 18, 2022: member

Tested that both -a and –asmap run, but if makeseeds no longer works without the added arg, can it be removed?

I considered this, but I have a slight preference for explicit named instead of positional arguments that’s why I kept the -a/--asmap.

fanquake added this to the milestone 24.0 on Apr 18, 2022

in contrib/seeds/makeseeds.py:183 in 08ecc04e5d outdated

177@@ -201,7 +178,18 @@ def ip_stats(ips: List[Dict]) -> str:
178 
179     return f"{hist['ipv4']:6d} {hist['ipv6']:6d} {hist['onion']:6d}"
180 
181+def parse_args():
182+    argparser = argparse.ArgumentParser(description=f'Generates a list of bitcoin node seed ip addresses.')
183+    argparser.add_argument("-a","--asmap", help=f'The location of the asmap asn database file (required)', required=True)

jonatack commented at 4:35 pm on April 18, 2022:

nits

0-    argparser = argparse.ArgumentParser(description=f'Generates a list of bitcoin node seed ip addresses.')
1-    argparser.add_argument("-a","--asmap", help=f'The location of the asmap asn database file (required)', required=True)
2+    argparser = argparse.ArgumentParser(description='Generates a list of bitcoin node seed ip addresses.')
3+    argparser.add_argument("-a", "--asmap", help='the location of the asmap asn database file (required)', required=True)

0$  python3 makeseeds.py -h
1usage: makeseeds.py [-h] -a ASMAP
2
3Generates a list of bitcoin node seed ip addresses.
4
5optional arguments:
6  -h, --help            show this help message and exit
7  -a ASMAP, --asmap ASMAP
8                        the location of the asmap asn database file (required)

sipa commented at 6:10 pm on April 21, 2022: member

I’ve put more up-to-date and generic asmap files on https://bitcoin.sipa.be/asmap-unfilled.dat and https://bitcoin.sipa.be/asmap-filled.dat (the latter one has subnets with no actual ASN assigned to nearby ASNs to minimize the file size). The input data was sourced through https://github.com/rrybarczyk/asmap-rs, and compiled to asmap format by new code I’m working on at https://github.com/sipa/asmap/tree/nextgen. I hope that’s in a presentable state in the next few days somewhere, but I’m put it online already to experiment with.

laanwj force-pushed on Apr 23, 2022

laanwj commented at 9:14 am on April 23, 2022: member

Re-pushed:

Addressed @jonatack ’s comments
Changed curl URL to @sipa’s new “asmap-filled.dat” asmap database (sorry for providing link to a corrupted one)
Mention @jamesob as co-author

in contrib/seeds/README.md:12 in 4f33193970 outdated

 7@@ -8,21 +8,11 @@ and remove old versions as necessary (at a minimum when GetDesirableServiceFlags
 8 changes its default return value, as those are the services which seeds are added
 9 to addrman with).
10 
11-The seeds compiled into the release are created from sipa's DNS seed data, like this:
12+The seeds compiled into the release are created from sipa's DNS seed and AS map
13+data, like this:

jonatack commented at 11:10 am on April 25, 2022:

nit, maybe be explicit regarding the directory to run these commands from

0data. Run the following commands from the `/contrib/seeds` directory:

laanwj commented at 6:50 pm on April 25, 2022:

Makes sense.

in contrib/seeds/README.md:15 in 4f33193970 outdated

12+The seeds compiled into the release are created from sipa's DNS seed and AS map
13+data, like this:
14 
15     curl https://bitcoin.sipa.be/seeds.txt.gz | gzip -dc > seeds_main.txt
16-    python3 makeseeds.py < seeds_main.txt > nodes_main.txt
17+    curl https://bitcoin.sipa.be/asmap-filled.dat > asmap-filled.dat

jonatack commented at 11:12 am on April 25, 2022:

Should this filename be made consistent (one way or the other and .gitignore too) with DEFAULT_ASMAP_FILENAME?

0src/init.cpp:129:static const char* DEFAULT_ASMAP_FILENAME="ip_asn.map";

laanwj commented at 6:50 pm on April 25, 2022:

I don’t know, I think that use of asmap is completely unrelated. Using the same name would make sure if we were to use the one from the user’s bitcoin directory. But we want to use a specific, deterministic asmap here so everyone gets the same result.

jonatack commented at 11:22 am on April 26, 2022:

Good point, agree.

jonatack commented at 11:18 am on April 25, 2022: contributor

Lightly tested ACK 4f331939706fddd210d910ed71a9368747a48dae per git range-diff 10a626a1 08ecc04 4f33193

 0Loading asmap database "asmap-filled.dat"…Done.
 1  IPv4   IPv6  Onion Pass                                               
 2470769  73264      0 Initial
 3470769  73264      0 Skip entries with invalid address
 4470769  73264      0 After removing duplicates
 5  6321   1728      0 Enforce minimal number of blocks
 6  5492   1496      0 Require service bit 1
 7  3861    872      0 Require minimum uptime
 8  3782    849      0 Require a known and recent user agent
 9  3757    841      0 Filter out hosts with multiple bitcoin ports
10   512    281      0 Look up ASNs and limit results per ASN and per net

laanwj commented at 10:55 am on April 26, 2022: member

Cherry-picked a commit from @jonatack with improvements.

laanwj force-pushed on Apr 26, 2022

jonatack commented at 12:28 pm on April 26, 2022: contributor

re-ACK 7edacdc6952a4327b01dec34d664450f055a10d0

dunxen approved

dunxen commented at 6:35 pm on May 22, 2022: contributor

tACK 7edacdc

With this version of the map:

0₿ sha256sum asmap-filled.dat 
1f479906ce1731281616a235a85a79c9c5085c36d88689939ef7fcc5196d30874  asmap-filled.dat

I get:

 0₿ python3 makeseeds.py -a asmap-filled.dat < seeds_main.txt > nodes_main.txt
 1Loading asmap database "asmap-filled.dat"…Done.
 2Loading and parsing DNS seeds…Done.
 3  IPv4   IPv6  Onion Pass                                               
 4471267  73410      0 Initial
 5471267  73410      0 Skip entries with invalid address
 6471267  73410      0 After removing duplicates
 7  6927   1908      0 Enforce minimal number of blocks
 8  5967   1617      0 Require service bit 1
 9  3636    856      0 Require minimum uptime
10  3244    771      0 Require a known and recent user agent
11  3221    767      0 Filter out hosts with multiple bitcoin ports
12   512    238      0 Look up ASNs and limit results per ASN and per net

laanwj commented at 10:26 am on May 23, 2022: member

I’m kind of in doubt here, with the testing and review this has got, should we go ahead and merge this and leave updating to asmap-nextgen for a follow-up PR?

dunxen commented at 3:32 pm on May 23, 2022: contributor

should we go ahead and merge this and leave updating to asmap-nextgen for a follow-up PR?

I think this is fair :) Definitely adds a lot of value already.

sipa commented at 3:50 pm on May 23, 2022: member

The current module there in asmap.py has an ASMap class with tests, and a from_binary and lookup method, which should be sufficient for the purposes here. I don’t think much is going to change there, and it’s already much better at handling error conditions than the current code, so feel free to switch to that.

I’m going to still clean things up more and have some cli tools using it, but the asmap module is pretty much done.

w0xlt approved

w0xlt commented at 3:56 pm on May 23, 2022: contributor

tACK https://github.com/bitcoin/bitcoin/pull/24864/commits/7edacdc6952a4327b01dec34d664450f055a10d0

dunxen approved

laanwj commented at 8:48 am on May 25, 2022: member

I’m going to still clean things up more and have some cli tools using it, but the asmap module is pretty much done.

I’ve replaced asmap.py with the new one. I didn’t find a way to look up an IP address directly so I did need to port over the decode_ip function. Please let me know if anything can be done more efficiently.

Result is the same, but it’s a bit slower (I think the bottleneck is in the loading phase):

Before:

 0$ time python3 makeseeds.py -a asmap-filled.dat < seeds_main.txt > /dev/null 
 1Loading asmap database "asmap-filled.dat"…Done.
 2Loading and parsing DNS seeds…Done.
 3  IPv4   IPv6  Onion Pass                                               
 4470789  73296      0 Initial
 5470789  73296      0 Skip entries with invalid address
 6470789  73296      0 After removing duplicates
 7  6318   1747      0 Enforce minimal number of blocks
 8  5456   1504      0 Require service bit 1
 9  3866    887      0 Require minimum uptime
10  3789    864      0 Require a known and recent user agent
11  3760    856      0 Filter out hosts with multiple bitcoin ports
12   512    295      0 Look up ASNs and limit results per ASN and per net
13
14real    0m9.826s
15user    0m9.349s
16sys     0m0.468s

After:

 0Loading asmap database "asmap-filled.dat"…Done.
 1Loading and parsing DNS seeds…Done.
 2  IPv4   IPv6  Onion Pass                                               
 3470789  73296      0 Initial
 4470789  73296      0 Skip entries with invalid address
 5470789  73296      0 After removing duplicates
 6  6318   1747      0 Enforce minimal number of blocks
 7  5456   1504      0 Require service bit 1
 8  3866    887      0 Require minimum uptime
 9  3789    864      0 Require a known and recent user agent
10  3760    856      0 Filter out hosts with multiple bitcoin ports
11   512    295      0 Look up ASNs and limit results per ASN and per net
12
13real    0m35.717s
14user    0m35.050s
15sys     0m0.628s

sipa commented at 12:22 pm on May 25, 2022: member

@laanwj Ah yes, there are only functions for converting between net ranges and prefixes, not addresses. I’ll add functions for those too. I don’t think that’s holding up anything here though; I can PR an update when it’s done.

The slowdown is indeed somewhat expected, as it’s converting the whole map to a lookup tree format rather than interpreting the binary data directly, which is faster per lookup but slower to load. Is that a concern? I could add interpretation based logic too.

laanwj commented at 8:23 am on May 26, 2022: member

Is that a concern? I could add interpretation based logic too.

No, not for me at least. It’s not a script to be run often and ~30s still a lot better than when it had to DNS query for every address. I also think the error checking implied by parsing instead of interpretation is useful here. And maybe there’s scope for optimization of the loading (if anyone does care).

Edit: our linter did find a few problems, not sure you care about these as our coding style isn’t necessarily the asmap one, but here goes:

0contrib/seeds/asmap.py:5:1: F407 future feature annotations is not defined
1contrib/seeds/asmap.py:284:9: E306 expected 1 blank line before a nested definition, found 0
2contrib/seeds/asmap.py:349:9: E306 expected 1 blank line before a nested definition, found 0
3contrib/seeds/asmap.py:370:9: E306 expected 1 blank line before a nested definition, found 0
4contrib/seeds/asmap.py:383:17: E125 continuation line with same indent as next logical line
5contrib/seeds/asmap.py:471:13: E306 expected 1 blank line before a nested definition, found 0
6contrib/seeds/asmap.py:532:9: E306 expected 1 blank line before a nested definition, found 0
7contrib/seeds/asmap.py:638:9: E306 expected 1 blank line before a nested definition, found 0
8^---- failure generated from lint-python.py

jonatack commented at 3:48 pm on May 27, 2022: contributor

Between 17 and 20 seconds total makeseeds.py time for me, which seems fine. The slower parts are preceded by a “Loading…” message, so all good.

laanwj force-pushed on May 30, 2022

laanwj commented at 4:31 pm on May 30, 2022: member

Re-pushed to fix linter errors (upstream in sipa/asmap#5).

However there’s still one left which I don’t really know how to get rid of:

0contrib/seeds/asmap.py:5:1: F407 future feature annotations is not defined

I don’t get this locally. It must have to do with Python version differences.

Edit: trying to simply remove the from __future__ import annotatinos line.

Edit.2: that gives me an error while running

0Traceback (most recent call last):
1  File "…/bitcoin/contrib/seeds/makeseeds.py", line 16, in <module>
2    from asmap import ASMap
3  File "…/bitcoin/contrib/seeds/asmap.py", line 171, in <module>
4    class _BinNode:
5  File "…/bitcoin/contrib/seeds/asmap.py", line 179, in _BinNode
6    def __init__(self, ins: _Instruction, arg1: _BinNode, arg2: _BinNode): ...
7NameError: name '_BinNode' is not defined

laanwj force-pushed on May 30, 2022

sipa commented at 4:39 pm on May 30, 2022: member

FWIW the asmap code runs significantly faster inside of pypy3.

jonatack commented at 4:40 pm on May 30, 2022: contributor

Was just wondering about this. I’m not seeing the issue locally either with Python 3.10.4.

contrib: Use asmap for ASN lookup in makeseeds

Add an argument `-a` to provide a asmap file to do the IP to ASN
lookups.

This speeds up the script greatly, and makes the output deterministic.
Also removes the dependency on `dns.lookup`.

I've annotated the output with ASxxxx comments to provide a way to
verify the functionality.

For now I've added instructions in README.md to download and use the
`demo.map` from the asmap repository. When we have some other mechanism
for distributing asmap files we could switch to that.

This continues #24824. I've removed all the fallbacks and extra
complexity, as everyone will be using the same instructions anyway.

Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: James O'Beirne <james.obeirne@pm.me>
Co-authored-by: russeree <reese.russell@ymail.com>

b54180303d

contrib: add seeds progress indicator and remove asmap one in makeseeds script ae00b9e02c

laanwj force-pushed on May 31, 2022

laanwj commented at 10:00 am on May 31, 2022: member

I added a commit that disables the Python linter for asmap.py.

0    test: Exclude asmap.py from Python linter checks
1    
2    asmap.py, a third-party dependency, is excluded from Python linter checks
3    because of `from __future__ import annotations` (PEP 563 postponed annotations)
4    this workaround can be removed when the Python version used is upgraded to 3.7+.

I decided to do this instead of removing the annotations because I think the postponed annotations are useful and we should use them as well (in the future, when the Python version allows for this).

This should hopefully make this pass the CI again.

laanwj force-pushed on May 31, 2022

in contrib/seeds/makeseeds.py:130 in 10383422a8 outdated

150-        return asn
151-    except Exception as e:
152-        sys.stderr.write(f'ERR: Could not resolve ASN for "{ip}": {e}\n')
153-        return None
154+def decode_ip(ip: str) -> int:
155+    addr = ipaddress.ip_address(ip)

sipa commented at 2:03 pm on May 31, 2022:

You can instead use asmap.net_to_prefix(ipaddress.ip_network(ip)) for this whole function. Functionally that means the result will also accept netmasks (“1.2.0.0/16” e.g.). asmap.lookup will then return None if not the entire range maps to a single ASN. I don’t believe this matters for the use case here.

laanwj commented at 2:50 pm on May 31, 2022:

Thanks. I looked at the code briefly but wasn’t under the impression net_to_prefix would accept individual addresses as well. Will try!

in contrib/seeds/makeseeds.py:160 in 10383422a8 outdated

161             # ips from this network
162             continue
163-        asn = lookup_asn(ip['net'], ip['ip'])
164-        if asn is None or asn_count[asn] == max_per_asn[ip['net']]:
165+        asn = asmap.lookup(decode_ip(ip['ip']))
166+        if asn is None or asn_count[ip['net'], asn] == max_per_asn[ip['net']]:

sipa commented at 2:20 pm on May 31, 2022:

You may want to add a condition for asn==0 here (0 is returned in case the IP has no corresponding ASN; None is returned if it’s ambiguous - which is only possible for ranges, not individual IPs).

laanwj commented at 2:51 pm on May 31, 2022:

Will do!

laanwj force-pushed on May 31, 2022

laanwj commented at 3:34 pm on May 31, 2022: member

Re-pushed, updated for @sipa’s suggestions.

jamesob commented at 3:38 pm on May 31, 2022: member

Concept ACK, will review soon.

sipa commented at 4:03 pm on May 31, 2022: member

@laanwj I’ve pushed another change, dropping the future annotations. It turns out you can use the string name of forward-declared types as well as typing annotations, which suffices here. I’ve also merged your https://github.com/sipa/asmap/pull/5. Hopefully it now passes all linter requirements?

contrib: Update makeseeds to asmap-nextgen 667e316bcb

laanwj force-pushed on Jun 1, 2022

laanwj commented at 12:40 pm on June 1, 2022: member

Ok, thanks for adding a license too, updated to the new asmap.py.

sipa commented at 6:12 pm on June 1, 2022: member

ACK 667e316bcb300eec131727a7ce54dd038031e267

dunxen approved

dunxen commented at 9:40 am on June 3, 2022: contributor

re-ACK 667e316

laanwj merged this on Jun 16, 2022

laanwj closed this on Jun 16, 2022

sidhujag referenced this in commit 3cdf40d6d3 on Jun 17, 2022

brunoerg commented at 2:38 pm on December 17, 2022: contributor

For now I’ve added instructions in README.md to download and use the demo.map from the asmap repository. When we have some other mechanism for distributing asmap files we could switch to that.

https://github.com/brunoerg/asmapy can be a good alternative!

bitcoin locked this on Dec 17, 2023

contrib: Use asmap for ASN lookup in makeseeds #24864

Conflicts