contrib: asmap-tool - Compare ASMaps with respect to specific addresses #30246

pull virtu wants to merge 1 commits into bitcoin:master from virtu:2024-06-compare-asmaps changing 1 files +41 −0
  1. virtu commented at 12:59 PM on June 7, 2024: contributor

    Right now, we have no way to quantify the "degradation" of an ASMap over time in the context of Bitcoin's P2P network in a meaningful way. However, such data would be useful for:

    1. Making sure the minimum shelf life of ASMaps is compatible with the release cycle (we wouldn't want to start shipping ASMaps with releases before making sure ASMaps typically do not become obsolete before the time of the next release)
    2. Node operators eager to keep their ASMaps up-to-date between releases.

    While asmap-tool.py has a diff command to perform a prefix-based comparison of two ASMaps, it is hard to reason about whether an old ASMap still is "good enough" or should be replaced with a newer one based on a prefix-based diff such as the following:

    $ ./asmap-tool.py diff 1704463200_asmap.dat 1710770400_asmap.dat
    [...]
    # 2c0f:fc98::/32 was AS37282
    # 2c0f:fcb8::/32 was AS37323
    2c0f:ff18::/32 AS37044 # was unassigned
    2c0f:ff98::/32 AS37113 # was unassigned
    2c0f:ffa0::/32 AS37273 # was unassigned
    # 76082350 (2^26.18) IPv4 addresses changed; 834271985742505274886878979424260 (2^109.36) IPv6 addresses changed
    

    One option for a more Bitcoin-centric ASMap comparison comprises comparing ASNs for the addresses of Bitcoin nodes and reporting on the number/share of addresses of nodes with disagreeing ASNs. By applying this approach to a node's set of known peers, a node operator can estimate how many of the node's peers are mapped to out-of-date AS when using the currently deployed and an up-to-date ASMap as input.

    This PR adds this functionality to asmap-tool.py by introducing a diff_addrs subcommand. In addition to two ASMaps, the subcommand reads addresses from a (getnodeaddresses-compatible) file, and computes statistics for those addresses:

    $ ./asmap-tool.py diff_addrs 1704463200_asmap.dat 1710770400_asmap.dat <(bitcoin-cli getnodeaddresses 0)
    275 address(es) reassigned from unassigned to AS51167
    84 address(es) reassigned from AS198949 to AS15557
    66 address(es) reassigned from AS45758 to AS45629
    33 address(es) reassigned from AS174 to AS212238
    [...]
    1 address(es) reassigned from unassigned to AS399619
    Summary: 919 (1.67%) of 54,901 addresses were reassigned.
    

    When the -s / --show-addresses flag is used, addresses subject to reassignment are included in the output.

  2. DrahtBot commented at 12:59 PM on June 7, 2024: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK fjahr, brunoerg, achow101
    Concept ACK sipa

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

  3. DrahtBot added the label Scripts and tools on Jun 7, 2024
  4. DrahtBot added the label CI failed on Jun 7, 2024
  5. DrahtBot commented at 5:22 PM on June 7, 2024: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    <sub>Debug: https://github.com/bitcoin/bitcoin/runs/25942564631</sub>

  6. fjahr commented at 9:19 AM on June 8, 2024: contributor

    Concept ACK

  7. DrahtBot removed the label CI failed on Jun 9, 2024
  8. brunoerg commented at 12:26 PM on June 10, 2024: contributor

    Concept ACK

  9. brunoerg commented at 12:43 PM on June 21, 2024: contributor

    I haven't looked at the code yet but I just tested it with two asmap files a few months apart and with 25k addresses, looks fine:

    78 address(es) reassigned from unassigned to AS51167
    47 address(es) reassigned from AS198949 to AS15557
    38 address(es) reassigned from AS45758 to AS45629
    18 address(es) reassigned from AS12715 to AS12479
    15 address(es) reassigned from AS57269 to AS8708
    12 address(es) reassigned from AS174 to AS212238
    9 address(es) reassigned from AS41378 to AS60024
    7 address(es) reassigned from unassigned to AS212238
    6 address(es) reassigned from AS212238 to AS174
    6 address(es) reassigned from AS9009 to AS51167
    6 address(es) reassigned from unassigned to AS141995
    6 address(es) reassigned from AS64286 to AS137409
    5 address(es) reassigned from AS834 to AS51167
    5 address(es) reassigned from AS62240 to AS206092
    5 address(es) reassigned from AS42366 to AS147049
    5 address(es) reassigned from AS202685 to AS51396
    4 address(es) reassigned from AS8972 to AS20773
    4 address(es) reassigned from AS3758 to AS9506
    4 address(es) reassigned from AS20838 to AS12479
    3 address(es) reassigned from AS51167 to AS40021
    3 address(es) reassigned from AS28220 to AS53059
    3 address(es) reassigned from AS8972 to AS29066
    3 address(es) reassigned from AS60808 to AS38136
    3 address(es) reassigned from AS9664 to AS38136
    3 address(es) reassigned from unassigned to AS39351
    3 address(es) reassigned from unassigned to AS136787
    3 address(es) reassigned from AS30860 to AS6698
    3 address(es) reassigned from AS22394 to AS6167
    2 address(es) reassigned from AS34549 to AS212238
    2 address(es) reassigned from AS61317 to AS62240
    2 address(es) reassigned from AS12638 to AS6805
    2 address(es) reassigned from AS57169 to AS9009
    2 address(es) reassigned from unassigned to AS41745
    2 address(es) reassigned from AS17554 to AS24334
    2 address(es) reassigned from AS42366 to AS136787
    2 address(es) reassigned from AS43180 to AS6698
    2 address(es) reassigned from AS43146 to AS197695
    2 address(es) reassigned from AS135097 to AS142286
    2 address(es) reassigned from AS264228 to AS28656
    2 address(es) reassigned from AS56902 to AS35197
    1 address(es) reassigned from unassigned to AS47583
    1 address(es) reassigned from AS35154 to AS12668
    1 address(es) reassigned from AS49276 to AS47987
    1 address(es) reassigned from unassigned to AS40021
    1 address(es) reassigned from AS8781 to AS42298
    1 address(es) reassigned from unassigned to AS50666
    1 address(es) reassigned from AS20454 to AS49037
    1 address(es) reassigned from unassigned to AS215623
    1 address(es) reassigned from AS6128 to AS53488
    1 address(es) reassigned from AS63023 to AS400175
    1 address(es) reassigned from AS207375 to AS206238
    1 address(es) reassigned from unassigned to AS203446
    1 address(es) reassigned from AS6167 to AS22394
    1 address(es) reassigned from AS31257 to AS204137
    1 address(es) reassigned from AS14277 to AS31769
    1 address(es) reassigned from AS26479 to AS397872
    1 address(es) reassigned from AS63023 to AS61112
    1 address(es) reassigned from AS25577 to unassigned
    1 address(es) reassigned from AS21769 to AS215659
    1 address(es) reassigned from AS61317 to AS47583
    1 address(es) reassigned from AS48728 to AS19905
    1 address(es) reassigned from AS10429 to AS26599
    1 address(es) reassigned from AS203963 to AS212238
    1 address(es) reassigned from unassigned to AS216071
    1 address(es) reassigned from AS6703 to AS21497
    1 address(es) reassigned from unassigned to AS152485
    1 address(es) reassigned from AS49055 to AS198610
    1 address(es) reassigned from AS3356 to unassigned
    1 address(es) reassigned from AS12389 to AS21378
    1 address(es) reassigned from AS59077 to AS58466
    1 address(es) reassigned from AS47869 to AS6206
    1 address(es) reassigned from AS206003 to AS206264
    1 address(es) reassigned from AS834 to AS47583
    1 address(es) reassigned from AS174 to AS400618
    1 address(es) reassigned from AS4134 to AS58466
    1 address(es) reassigned from AS834 to AS216183
    1 address(es) reassigned from AS51765 to AS136258
    1 address(es) reassigned from AS13672 to AS13977
    1 address(es) reassigned from AS59711 to AS215467
    1 address(es) reassigned from AS15542 to AS15435
    1 address(es) reassigned from AS55577 to AS18209
    1 address(es) reassigned from AS834 to AS134143
    1 address(es) reassigned from AS14080 to AS10620
    1 address(es) reassigned from AS136787 to AS62240
    1 address(es) reassigned from AS8851 to unassigned
    1 address(es) reassigned from AS28968 to AS216363
    1 address(es) reassigned from AS2542 to AS25472
    1 address(es) reassigned from AS31034 to AS202613
    1 address(es) reassigned from AS21586 to AS27425
    1 address(es) reassigned from AS49348 to AS3280
    1 address(es) reassigned from unassigned to AS57043
    1 address(es) reassigned from AS174 to AS3257
    1 address(es) reassigned from AS61317 to AS212238
    1 address(es) reassigned from AS209984 to AS203936
    1 address(es) reassigned from AS133199 to AS133861
    1 address(es) reassigned from AS57427 to AS57523
    1 address(es) reassigned from unassigned to AS197695
    1 address(es) reassigned from AS203020 to AS136787
    1 address(es) reassigned from AS208599 to AS199087
    1 address(es) reassigned from AS25086 to AS8359
    Summary: 391 (1.51%) of 25,875 addresses were reassigned
    

    Looking at the ASes into peering db, the reassignments make sense.

  10. brunoerg approved
  11. brunoerg commented at 12:54 PM on June 24, 2024: contributor

    ACK 42eb7b144c09f0404487f2f5d17ca1a3e91c9365

  12. DrahtBot requested review from fjahr on Jun 24, 2024
  13. in contrib/asmap/asmap-tool.py:7 in 42eb7b144c outdated
       5 | @@ -6,7 +6,9 @@
       6 |  import argparse
       7 |  import sys
    


    fjahr commented at 1:36 PM on June 24, 2024:

    nit: sys isn't in the right alphabetical order here. You could reorder it while you are at it.


    virtu commented at 2:47 PM on June 27, 2024:

    Moving sys doesn't fully fix the import block because it is separated from the next function definition by only one newline. If I added another newline below the import block fix it, I should probably also add additional newlines between function definitions for consistency's sake. Not sure if any/how much refactoring should be included here. wdyt?


    fjahr commented at 12:27 PM on July 8, 2024:

    It's fine to leave it as is. It was just a nit :)

  14. in contrib/asmap/asmap-tool.py:127 in 42eb7b144c outdated
     122 | +    parser_diff_addrs.add_argument("infile1", type=argparse.FileType("rb"),
     123 | +                                   help="first file to compare (text or binary)")
     124 | +    parser_diff_addrs.add_argument("infile2", type=argparse.FileType("rb"),
     125 | +                                   help="second file to compare (text or binary)")
     126 | +    parser_diff_addrs.add_argument("addrs_file", type=argparse.FileType("r"),
     127 | +                                   help="address file generated with getnodeaddresses to use in the comparison")
    


    fjahr commented at 1:39 PM on June 24, 2024:

    You should name the specific settings to run getnodeaddresses with so that the result is as expected. I guess that would be 0 and all (default) right? I fear that if people do it wrong they could get a result like "nothing was reassigned" and then think: oh great, the internet is so stable nowadays ;)

  15. in contrib/asmap/asmap-tool.py:186 in 42eb7b144c outdated
     181 | +            new_asn = f"AS{new_asn}" if new_asn else "unassigned"
     182 | +            opt = ": " + ", ".join(reassigned_addrs) if args.show_addresses else ""
     183 | +            print(f"{num_reassigned} address(es) reassigned from {old_asn} to {new_asn}{opt}")
     184 | +        num_reassignments = sum(len(addrs) for _, addrs in reassignments)
     185 | +        share = num_reassignments / len(addrs) if len(addrs) > 0 else 0
     186 | +        print(f"Summary: {num_reassignments:,} ({share:.2%}) of {len(addrs):,} addresses were reassigned.")
    


    fjahr commented at 1:44 PM on June 24, 2024:

    Maybe add the count of addresses that were unassigned here specifically in the summary. Those may be among the most interesting to look at so I think it could be highlighted.


    virtu commented at 2:49 PM on June 27, 2024:

    Good idea. Total change is now broken down into migrations (from one AS to another), assignments (previously unassigned, now assigned to an AS) and unassignments (vice versa).

    Summary: 371 (1.37%) of 27,050 addresses were reassigned (migrations=242, assignments=125, unassignments=4)

  16. fjahr commented at 1:46 PM on June 24, 2024: contributor

    Looks good to me, just some additional documentation on documentation on the addrs input file would be good to avoid people getting strange results.

  17. DrahtBot requested review from fjahr on Jun 24, 2024
  18. sipa commented at 4:56 PM on June 24, 2024: member

    Concept ACK

  19. Compare ASMaps with respect to specific addresses
    Introduce diff_addrs subcommand as means for a Bitcoin-centric
    comparison of two ASMaps.
    
    In addition to two ASMaps, the subcommand reads addresses from
    a (getnodeaddresses-compatible) file, and provides information on
    addresses that have mismatching ASN according to the two ASMaps.
    5215c925d1
  20. virtu force-pushed on Jun 27, 2024
  21. fjahr commented at 12:47 PM on July 8, 2024: contributor

    tACK 5215c925d1382e71c9e1d642fced8a152c629c7f

    I played around with a few different input files, results look reasonable and I think this is a very helpful for analysis of generated asmap files.

    For anyone who wants to test but is missing asmap files, you can use the ones here: https://github.com/fjahr/asmap-data

  22. DrahtBot requested review from sipa on Jul 8, 2024
  23. DrahtBot requested review from brunoerg on Jul 8, 2024
  24. brunoerg approved
  25. brunoerg commented at 1:47 PM on July 8, 2024: contributor

    reACK 5215c925d1382e71c9e1d642fced8a152c629c7f

    verified the changes since my last review, lgtm.

  26. achow101 commented at 8:16 PM on August 12, 2024: member

    ACK 5215c925d1382e71c9e1d642fced8a152c629c7f

  27. achow101 merged this on Aug 12, 2024
  28. achow101 closed this on Aug 12, 2024

  29. virtu deleted the branch on Sep 11, 2024
  30. bitcoin locked this on Sep 11, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-02 12:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me