This updates the currently embedded data ahead of the v31 release.
It currently uses the map from the 1772726400 run
The attestation process to accompany these runs is still rather new but sigs are collected here: https://github.com/asmap/asmap.sigs/tree/main/1772726400
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
See the guideline for information on the review process.
| Type | Reviewers |
|---|---|
| ACK | sedited, hodlinator, achow101 |
If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.
@achow101 @fanquake looking to confirm a few things about attestations. The ASmap flow is currently:
Generate ASmap with kartograf
-> use the one with the most matches
-> encode it via contrib/asmap/asmap-tool.py
-> open PR to bitcoin-core/asmap-data
-> people reproduce the hashes for the encoded files (filled and unfilled)
-> merge PR to bitcoin-core/asmap-data
Does it make sense to have only the people with the “winning hash” attest to their output file hash, and its encoded variants? So if 6 people have a matching hash, they become the subset of participants that must submit attestations.
Other users could validate that the encoded ASmap included in bitcoin-core/asmap-data has the expected / attested hash, and could decode the ASmap to get the initial file hash too.
Or is it enough to attest on the hashes of the ASmap with the most matches in a collaborative run’s output? As in, we add the “winning” ASmap to the bitcoin-core/asmap-data and anyone can drive by, recreate the encoded files from the uploaded file, and attest to them.
Does it make sense to have only the people with the “winning hash” attest to their output file hash, and its encoded variants?
It’s a tricky question, but I feel like it only really makes sense if you have the people reproducing the full data set post their signed hashes. You don’t really gain much from many people attesting to the encoding only, since the source data is the real unknown factor, and the encoding is easily reproduced.
I feel like it only really makes sense if you have the people reproducing the full data set
yes, though since the outputs are deterministic based on the inputs, we could imagine the collab runs be only to fetch the inputs, compare them amongst ourselves and decide what the “right” ones are. We could then post the inputs, and have everyone reproduce the outputs from them and attest. For now, maybe we keep doing the collab runs, agree on a “winning” hash, post the input data, then have people reproduce with that input and attest?
I am ok with allowing anyone to attest to the asmap files so long as they are reproducible from easily examinable files.
You don’t really gain much from many people attesting to the encoding only, since the source data is the real unknown factor, and the encoding is easily reproduced.
This the conundrum, the first intermediate product that I would call easily examinable is the result file but at that point all that is left is the encoding and that is very easy to verify by encoding yourself, checking the sigs is probably even more work unless we provide a specialized script.
In GUIX the source is unambiguous, everyone starts from the same commit hash. The attestation for the reproducible build lends trust to the build process which is very complex and could have interference. In asmap the source is kind of ambiguous, every collaborative runner who got the matching hash could have slightly different raw download data. The encoding process afterwards is very simple, no compilers or dependencies involved.
Ideally we would get a signature from everyone who got a match in the collaborative run but requiring the collaborative runners to only be able to participate in a meaningful way if they are able to sign also isn’t great and unnecessarily restrict the pool of participants.
I guess reproducing from a raw data set which results in the hash that got the most matches and then letting everyone sign would be a good first step. This would mean leaving audit of the raw inputs to the people themselves if they haven’t gotten a match. We could develop better tooling to compare at let people decide if they got a mismatch or for general analysis of the data if they didn’t participate at all (though kartograf already has some of that tooling like coverage etc.).
we could imagine the collab runs be only to fetch the inputs, compare them amongst ourselves and decide what the “right” ones are
I don’t understand this, how is this different from what we are doing today? How would we compare the inputs without processing them?
contrib/asmap and do the encoding for you with asmap-tool.py.
@fjahr could you upload the inputs for 1772726400 to releases as you did here ? so we can reproduce from inputs.
could you upload the inputs for 1772726400 to releases as you did here ? so we can reproduce from inputs.
Done: https://github.com/bitcoin-core/asmap-data/releases What’s a bit weird is that we have to attach releases to a commit hash in github so it would make sense to attach the release to the commit that add the actual encoded result but since we need to share the inputs to get the sigs before merging that commit, the release points to an older commit now. Releases are also immutable so it seems I can’t change it later. But it’s probably still better than some awkward workaround using temp storage somewhere else outside GH.
Updated to use the new file from 1772726400. This is all a bit off-the-cuff but the different parts are coming together now:
The one potential blocker could be if maintainers would like more time to look into asmap.sigs repo. It’s essentially a very light-weight guix.sigs and currently not part of the bitcoin-core org.
The file was produced in this collaborative run: https://github.com/bitcoin-core/asmap-data/issues/44
ACK 0690a5d0f27b52435782815b9d7520684486a9ad FWIW
(I’m one of the ones attesting in asmap.sigs, so maybe my ACK shouldn’t weigh as heavily).
0₿ sha256sum src/node/data/ip_asn.dat
1478d61986c59365cf86cd244485bbbe76a9ca0c630864717286dd19949879074 src/node/data/ip_asn.dat
Hash matches my locally generated 1772726400_asmap.dat file.