[RFC] bitcoin-asmap utility

sipa commented at 8:38 am on April 9, 2020: member

This adds a new binary, bitcoin-asmap that can be used to construct, inspect, and query asmap files. It replaces the Python scripts I had at https://github.com/sipa/asmap with something more performant, efficient, and auditable.

There are some rough edges still (in particular documenting the code and a number of edge cases), but I’d like to offer it for comments here. I’m not sure we want this code to be part of the Bitcoin Core codebase, but it certainly has advantages (in terms of code reuse and having consistent encoder/decoder). I think it may also be beneficial that users have a way to audit these files using code that’s subject to our review and testing process.

This PR includes a fuzz test that verifies (for small, reduced inputs) that encoding produces asmap files consistent with their input, that invoking the interpreter on them works as expected, and that decoding and re-encoding does not change their behavior.

Given a text file asmap.txt with lines like these:

0193.189.95.0/24 AS34906
1193.189.96.0/24 AS20850
2193.189.98.0/23 AS33925
3193.190.0.0/15 AS2611
4193.19.102.0/23 AS20485
5193.19.103.0/24 AS8342
6193.19.106.0/23 AS3257

You can run bitcoin-asmap encode asmap.dat <asmap.txt to construct an asmap file in a few seconds. bitcoin-asmap decode asmap.dat will decode it back to the original format (note however that the result may look very different, as unassigned IP ranges get remapped for efficiency). bitcoin-asmap lookup asmap.dat 193.190.253.208 will look up one entry in it.

There is code at https://github.com/rrybarczyk/asmap-rs (by @rrybarczyk and @naumenkogs) to build input for this tool using RIPE’s dumps of Internet routing tables.

fanquake added the label P2P on Apr 9, 2020

fanquake added the label Needs Conceptual Review on Apr 9, 2020

jonatack commented at 9:37 am on April 9, 2020: member

Concept ACK (leaving aside for now the question of inclusion in the codebase).

naumenkogs commented at 1:59 pm on April 9, 2020: member

Concept ACK. I think it makes sense for us to have this tool inside the repo, if we want to encourage using asmap, and improve the auditability/maintenance.

Did some slight code review, will do more if/when we get enough Concept ACKs.

Also, I will probably post this separately, but anyone who speaks rust is welcome to review our tool (https://github.com/i7i/asmap-rs) for aggregating routing info from various sources :)

DrahtBot commented at 3:33 pm on April 9, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#18512 (Improve asmap checks and add sanity check by sipa)
#18450 (util: Use locale independent ToString(…) instead of locale dependent strprintf(…) for low-level string formatting by practicalswift)
#18317 (Serialization improvements step 6 (all except wallet/gui) by sipa)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

sipa force-pushed on Apr 10, 2020

sipa force-pushed on Apr 11, 2020

sipa commented at 0:49 am on April 12, 2020: member

Added lots of comments, and improved the encoder algorithm. I believe it will now always find the smallest encoding (in a few seconds, for realistic inputs).

Introduce Instruction enum in asmap d69f51b925

Deal with decoding failures explicitly in asmap Interpret 9940be6f67

Improve asmap Interpret checks and document failures 72a20f6db6

Add asmap sanity checker 8689555287

Add additional effiency checks to sanity checker d01cfa80ac

Make asmap Interpreter errors fatal and fuzz test it 9c31523e9e

Add asmap_direct fuzzer that tests Interpreter directly 9e54347e69

[REFACTOR] Add CSubNet::GetCIDR e6712455df

Implement asmap decoder fc06b253cb

Implement asmap encoder 26f5a7e8c1

Add fuzzer for asmap encoder/decoder fdebc68cd9

Add bitcoin-asmap tool 3d435b1061

sipa force-pushed on Apr 12, 2020

practicalswift commented at 5:26 pm on April 13, 2020: contributor

Concept ACK

jb55 commented at 9:20 pm on April 16, 2020: member

+0 for having it in the repo, Concept ACK

luke-jr commented at 4:35 am on April 23, 2020: member

Ideally it should be in another repo, but maybe we’re not ready for that yet if it shares a lot of code.

DrahtBot added the label Needs rebase on May 6, 2020

DrahtBot commented at 1:48 pm on May 6, 2020: member

🐙 This pull request conflicts with the target branch and needs rebase.

Sjors commented at 4:54 pm on November 16, 2020: member

It might be worth building this on top of #19937 as part of bitcoin-util.

Concept ACK on including something this into the codebase, ideally along with some instructions which are a bit sparse now. Unless you suspect this needs tons of tweaking in the future, in which case perhaps a separate repo is better. Or if the output can be used by other software than Bitcoin Core.

lestephane commented at 8:58 pm on March 26, 2021: none

@Sjors I had the same problem with the lack of clear instructions so I created a Vagrantfile that performs all the steps in one go.

It’s located at https://github.com/lestephane/vagrant-asmap

You run vagrant up, and after a while, if your machine has the minimum specs to run the bottleneck analysis (12GB RAM seems to be needed for me), you end up with an ip_asn.map file.

DrahtBot commented at 11:22 am on December 15, 2021: member

Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
Is it no longer relevant? ➡️ Please close.
Did the author lose interest or time to work on this? ➡️ Please close it and mark it ‘Up for grabs’ with the label, so that it can be picked up in the future.

DrahtBot commented at 1:07 pm on March 21, 2022: member

Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
Is it no longer relevant? ➡️ Please close.
Did the author lose interest or time to work on this? ➡️ Please close it and mark it ‘Up for grabs’ with the label, so that it can be picked up in the future.

dunxen commented at 8:59 am on April 7, 2022: contributor

Concept ACK

I’m working on some tooling to help with map comparisons between releases and some user experience improvements to asmap-rs.

Although certainly we’ll have an encoded map included with every release when asmap is enabled by default, I think it still makes sense to include bitcoin-asmap in this repo for easier maintenance and discoverability. It’s just less hassle for node operators who may be interested generating and encoding their own mappings too.

sipa commented at 6:27 pm on June 8, 2022: member

Closing. I’m working on external tooling for working with asmap files in https://github.com/sipa/asmap/tree/nextgen (encode, decode, diff, bottleneck, …).

sipa closed this on Jun 8, 2022

DrahtBot locked this on Jun 8, 2023

[RFC] bitcoin-asmap utility #18573

Conflicts