net: create IP to ASN database from file

russeree commented at 3:24 am on April 11, 2022: contributor

Per task number 1 in issue #17020 , “Read ASNs from a file instead of hardcoding”

The function ’lookup_asn’ has been replaced in a drop in manner by a tab separated value (.tsv) file loader class and function set. This drop in bypasses the current REST based ASN implementation where a 3rd party is contacted per ip address to fetch the current ASN for that ip address. A REST approach is slower than the localized database search found in this PR.

Tech Details This drop in replacement utilizes a class object that upon construction loads an ASN database file into memory. This class should be instantiated in ‘makeseeds.py’. Once the class has been constructed the makeseeds.py can call asn_lookup() to get the ASN of an ip address from the now localized database. This approach affords speed and decentralization at the cost of a marginal repository size increase. As the old saying goes there is no such thing as a free lunch.

Edit: 4/11/2022 The class object has been reworked in such a manner that upon instantiation the class if the constructor is provided with a True value for ‘fetch’. The class will acquire the compressed dataset upon runtime from the web address stored in the class object (this could be passes as parameter theoretically). This fetching process removes the need to store the ip to asn database as a blob in the repository. If fetch is set to False then the class object will attempt to load the default file. If neither a asn db file is present or a fetch is requested the failover mechanism kicks in an performs ip to asn decoding using the previous method currently in master. Failover is slower in all cases and seemly doesn’t resolve ip addresses as frequently.

Datasource: https://iptoasn.com/ Combined IPv4+IPv6 to ASN map

Error messages for failed ip lookups from the previous REST are identical. Additional asserts have been added to avoid feeding the ASN database loader invalid data. These asserts should cover some of the most common issues.

IP has too many segments
File does not exist
Invalid number of columns (currently 5)
IP address has too many segments
Decimal/Hexdecimal range is invalid within a segment
ASN database has no records

Cost and Changes 2 files were added. ‘ansdecode.py’ which contains a class that loads a .tsv.gz file and has member functions that replace existing ASN lookups that are performed by parsing http://www.team-cymru.com/IP-ASN-mapping.html. The second file by default is ip2asn.tsv.gz. This filename is a class constructor parameter that can be changed if a user would like to use a custom file location. This ip2asn.tsv.gz is a compressed .tsv that has a format of [rangeStart rangeEnd asnNumber …]\n. Only the first 3 columns of the TSV are utilized additional columns can be included but will only make importing less efficient by doing so.

Advantages

Faster: on an Intel Xeon 2699v4 Linux time was reduced from ~2:40 to ~1:50
Decreases dependance on 3rd party services - Changes to the ASN allocations are infrequent.
Auditable data format Uncompressed
Asserts on class load with the ample information to identify broken/corrupt lines in the input file
IPV4/6 Detection function is included and could be used to replace the need for manual definitions in makeseeds.py

Disadvantages

Uncompressed ASN database unmodified is 26MB - https://iptoasn.com/ - Data is public domain.
Compressed ASN databased is ~7.8MB - https://iptoasn.com/ - Data is public domain.
ASN .tsv.gz database would need to be updated occasionally.

Notes

ansdecode.py can be included into the makeseeds.py at the cost of additional file length
Deliberate use of a class to make testing and auditing of IP hex strings to integer possible.
The TSV file could be converted to a CSV easily
The TSV file could be downloaded at runtime instead of being included
At the cost of complexity one could implement a parallel threaded search for the applicable ip range
Internet based fallback and failover is possible though that would be quite extreme.

DrahtBot added the label Scripts and tools on Apr 11, 2022

russeree renamed this:
~~Loading ASN directory from a .tsv file.~~
makeseeds.py : create ASN to ip database from file.
on Apr 11, 2022

russeree renamed this:
~~makeseeds.py : create ASN to ip database from file.~~
makeseeds.py : create ASN to ip database from file
on Apr 11, 2022

russeree renamed this:
~~makeseeds.py : create ASN to ip database from file~~
net: makeseeds.py - create IP to ASN database from file
on Apr 11, 2022

russeree renamed this:
~~net: makeseeds.py - create IP to ASN database from file~~
net: create IP to ASN database from file - makeseeds.py
on Apr 11, 2022

in contrib/seeds/README.md:13 in 44d316b17d outdated

 9@@ -10,9 +10,8 @@ to addrman with).
10 
11 The seeds compiled into the release are created from sipa's DNS seed data, like this:
12 
13-    curl https://bitcoin.sipa.be/seeds.txt.gz | gzip -dc > seeds_main.txt
14+    curl -s http://bitcoin.sipa.be/seeds.txt.gz | gzip -dc > seeds_main.txt

laanwj commented at 10:46 am on April 11, 2022:

Why are you changing this?

russeree commented at 11:42 am on April 11, 2022:

These issues have all been resolved. The 3 affected files are now up to date with master. These changes were unintended and were likely a byproduct of an earlier pull.

There should only be 3 files that change within this pull request.

makeseeds.py - Instantiation of ASN Decoder Class
asndecode.py - IP -> ASN Class Object
ip2asn.tsv.gz - Database File

As a technical note the contents of asndecode.py could be merged with makeseeds.py but this would reduce clarity IMO.

laanwj commented at 10:47 am on April 11, 2022: member

Concept ACK, but it looks like you’re (accidentally, I guess) reverting some changes made recently.

russeree commented at 12:09 pm on April 11, 2022: contributor

Concept ACK, but it looks like you’re (accidentally, I guess) reverting some changes made recently.

You are correct there were accidental reversions. These have all been remedied and this PR is now up to date with master. The issues pointed out in your initial review should now be resolved.

fanquake commented at 12:15 pm on April 11, 2022: member

contrib/seeds/ip2asn.tsv.gz

This should be downloaded and used as needed. We aren’t going to commit a 7mb blob to the repo.

You should try and maintain a clean commit history as you develop, rather than pushing all of the random fixups and other commits to this PR. Your commits will need to be squashed before anything is merged in any case.

russeree commented at 12:32 pm on April 11, 2022: contributor

contrib/seeds/ip2asn.tsv.gz

This should be downloaded and used as needed. We aren’t going to commit a 7mb blob to the repo.

You should try and maintain a clean commit history as you develop, rather than pushing all of the random fixups and other commits to this PR. Your commits will need to be squashed before anything is merged in any case.

Addressing the uncleanliness of this PR. Should this be PR remade? I can squash from here on out, I will learn from my mistakes but am still newish to git.

The larger issue to determine is if downloading a blob at runtime any better than parsing http://www.team-cymru.com/IP-ASN-mapping.html as makeseeds.py does right now? The other option would be to determine an acceptable database size for the ip to ASN database? I don’t foresee anything smaller than 1MB with trimming and compression.

Things to compress the ASN database would be to

Use a tree like structure where a node is an ASN and would connect to ranges it controls
Instead of writing the whole IP Address in the end range field, use the number of consecutive ip addresses owned in that block

Edit: Would a duality solution where the user could download an ASN file and if detected makeseeds would use it, If no ip->asn database file exists then the failover would be to parse http://www.team-cymru.com/IP-ASN-mapping.html as makeseeds is doing right now.

The issue as presented in #17020 suggests to not hardcode an ASN and then place it in a file in the same way as suspicious hosts are. The ASN directory is quite massive and doesn’t follow any specific patterns due to the fact it’s a registry of real world relations. Thus a hardcoded dataset with potentially billions of entries will require quite a bit of space.

DrahtBot commented at 9:50 pm on April 11, 2022: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#24818 (net: improve and address issues in makeseeds.py by RF5)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

russeree commented at 9:59 pm on April 11, 2022: contributor

I have rebased the code with the requests from fanquake. Could I close this PR and file a new one to clean up some of my earlier ‘spam’ from too many commits?

Thanks

brunoerg commented at 10:19 pm on April 11, 2022: member

Could I close this PR and file a new one to clean up some of my earlier ‘spam’ from too many commits?

You can squash the commits, no need to open a new PR. See: https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

f170fb6393

russeree force-pushed on Apr 11, 2022

russeree commented at 10:32 pm on April 11, 2022: contributor

Could I close this PR and file a new one to clean up some of my earlier ‘spam’ from too many commits?

You can squash the commits, no need to open a new PR. See: https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

I have squashed these commits, thank you so much for the reference. The PR looks so much better.

contrib/seeds/ip2asn.tsv.gz

This should be downloaded and used as needed. We aren’t going to commit a 7mb blob to the repo.

You should try and maintain a clean commit history as you develop, rather than pushing all of the random fixups and other commits to this PR. Your commits will need to be squashed before anything is merged in any case.

There is now a mechanism in the class object to fetch the ip-asn directory database .gz file at runtime. The user can also load directly from a file they have created or obtained on their own if they wish without parsing an external source. A feature has been added is that if the asn db file is not loaded or used the class fails over to a 1:1 clone of the current implementation to decode the ASN information that fetches though an api request.

Edit: Working on fixing this PR -> some branch elements are not in sync with the master. Current master implementation - time master ASN in file database time contrib

Note that the current implementation in this PR was able to resolve an IPv6 Address that was not resolvable via the current API service. If the ASN Database could not find a range for the current ip it would have failed over to the API call and the error in master would have been present in this branch.

EDIT 2: I could use some assistance fixing this branch. I have made some mistakes and I don’t know how to walk them back.

LINT fixes 2a459260d4

Squased lint changes - added features

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

LINT fixes

ef693dc18e

Merge branch 'contrib-seeds-asn' of https://github.com/russeree/bitcoin into contrib-seeds-asn a2047959a9

Python include dns.resolver d70372d925

Failover implementation complete 5814d5e918

russeree force-pushed on Apr 12, 2022

Squash

Squashed All Previous Commits

RPC: Switch getblockfrompeer back to standard param name blockhash

This commit partially reverts 923312fbf6a89efde1739da0b7209694d4f892ba.

Update RPC argument and field naming guideline in developer notes

Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>

build: fix MSVC build after subtree update

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Aaron Clauson <aaron@sipsorcery.com>

build: remove --enable-experimental from libsecp256k1 configure

build: remove some no-longer-needed var unexporting from configure

key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign

The renaming occured in
https://github.com/bitcoin-core/secp256k1/pull/1089.

Squased lint changes - added features

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

LINT fixes

LINT fixes

Squash - Too many commits

Python include dns.resolver

Failover implementation complete

Removed DNS resolver

Revert changes

623d4ff6b2

russeree force-pushed on Apr 12, 2022

Put lock logging behind DEBUG_LOCKCONTENTION preprocessor directive f6dce649ab

Add DEBUG_LOCKCONTENTION documentation to the developer notes 6afd2cbd05

Squash

Squashed All Previous Commits

RPC: Switch getblockfrompeer back to standard param name blockhash

This commit partially reverts 923312fbf6a89efde1739da0b7209694d4f892ba.

Update RPC argument and field naming guideline in developer notes

Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>

build: fix MSVC build after subtree update

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Aaron Clauson <aaron@sipsorcery.com>

build: remove --enable-experimental from libsecp256k1 configure

build: remove some no-longer-needed var unexporting from configure

key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign

The renaming occured in
https://github.com/bitcoin-core/secp256k1/pull/1089.

Squased lint changes - added features

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

LINT fixes

LINT fixes

Squash - Too many commits

Python include dns.resolver

Failover implementation complete

Removed DNS resolver

Revert changes

db0875d0a7

Merge branch 'contrib-seeds-asn' of https://github.com/russeree/bitcoin into contrib-seeds-asn cbbb36cc65

refactor: Remove deduplication of data in rollingbloom bench b73a6ed56b

lint: codespell 2.1.0 534808ecf0

lint: flake8 4.0.1 6bdf21ddbc

lint: mypy 0.942 bd77c9e8e9

refactor: fixup named args in txpackage tests

Regression in #24152.

c4e3415842

Put lock logging behind DEBUG_LOCKCONTENTION preprocessor directive ec1d3cc0b7

Add DEBUG_LOCKCONTENTION documentation to the developer notes 4bf12261d6

Squash

Squashed All Previous Commits

RPC: Switch getblockfrompeer back to standard param name blockhash

This commit partially reverts 923312fbf6a89efde1739da0b7209694d4f892ba.

Update RPC argument and field naming guideline in developer notes

Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>

build: fix MSVC build after subtree update

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Aaron Clauson <aaron@sipsorcery.com>

build: remove --enable-experimental from libsecp256k1 configure

build: remove some no-longer-needed var unexporting from configure

key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign

The renaming occured in
https://github.com/bitcoin-core/secp256k1/pull/1089.

Squased lint changes - added features

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

LINT fixes

LINT fixes

Squash - Too many commits

Python include dns.resolver

Failover implementation complete

Removed DNS resolver

Revert changes

5842e2351d

Merge branch 'contrib-seeds-asn' of https://github.com/russeree/bitcoin into contrib-seeds-asn 6673d4d90c

russeree requested review from laanwj on Apr 13, 2022

russeree commented at 10:33 pm on April 13, 2022: contributor

If this code has potential to be a candidate to solve the ASN read from file should the asnlookup.py be merged into makeseeds.py or should it remain a standalone file? Does the filename reflect the modules usage?

Should the load ASNs from file implementation be the default option instead of current REST lookup methods? Or should it be defaulted too with a REST based implementation as a failover is there is no file or a file could not be parsed?
Should there be a defined format for an ASN file? Should the ASN file be compressed? CSV,TSV,JSON,XML?
Since committing a 3MB+ blob to the core repo is not a good idea. Where should a file be hosted as a good default location to get on execution.
What validation needs to happen in regards to the ip->asn lookup. There are a few moving parts such as fetch, failover, and parsing that would likely need validation.

laanwj commented at 6:31 am on April 14, 2022: member

The idea is to eliminate run-time variance that we have now by querying the ASNs. Ideally the ASN file would come from a third party, and would not need to be managed by us. The only important thing really is that people who run the script (either to update the seeds, or to check them) can get the same file, so compare the sha256sum or so.

Since committing a 3MB+ blob to the core repo is not a good idea. Where should a file be hosted as a good default location to get on execution.

Right, maintaining a ASN database is definitely out of scope for the project, and would also needlessly clutter the repo.

russeree commented at 7:15 am on April 14, 2022: contributor

@laanwj So this is the approach that will be used. A high availability hosted source will have a copy of the ip->asn database. The SHA256 checksum for this file will be coded as default argument to the class. This will ensure no variance from run to run. In order to maintain functionality under all costs if either the file/checksum fails. The script should fall back unto the previous method of rest parsing.

Should the asndecoder class file be merged into makeseeds.py?

ASN blocks don’t frequency change but how often should the checksum / asn database combo be updated and maintained? Per release?

laanwj commented at 8:14 am on April 14, 2022: member

I’d say people would download the ASN database at the same time they now download the seeds.txt.gz, so this is for every major release. This is the only time that the hardcoded seeds are updated in the release process.

The SHA256 checksum for this file will be coded as default argument to the class.

I don’t think that’s necessary. Users can compare the ASN database they have with their own means. Say, mention the sha256sum of the file in the review post in the PR. The only functionality that’s needed is to pass in this file to makeseeds.py and use it instead of run-time querying.

russeree commented at 8:39 am on April 14, 2022: contributor

This is where this could get tricky to solve with the current implementation. Right now the limitation is that there can only be one input file to makeseeds.py persys.stdin.readlines(). This currently is the user fetched seeds.txt.gz Would it be an option to switch makeseeds.py to a python script with input parameters?

example makeseeds.py -i seeds.txt.gz -a asn.txt.gz or makeseeds.py --seeds seeds.txt.gz --asn-db asn.txt.gz

instead of

makeseeds.py < seeds.txt.gz

There is the possibility to maintain 100% backwards compatibility with the existing commands if that is desired.

laanwj commented at 11:22 am on April 14, 2022: member

Yes, adding a command line argument for the ASN database is fine! There’s no pressing reason for backwards compatibility, as far as I know no one uses this as part of automatic scripting, just make sure the documentation (the instructions in the README.md) is up to date.

sipa commented at 2:00 pm on April 14, 2022: member

Bitcoin Core itself can also use an ASN database, with -asmap=FILE argument. The asmap file format is a custom compact binary format for looking up ASN values by IP address. There has been some work around tooling and discussions about distribution that are ongoing (there is some in #18573, Python code in https://github.com/sipa/asmap, Rust code in https://github.com/rrybarczyk/asmap-rs).

Would it be reasonable to try to use the same format? There is Python lookup code for it in https://github.com/sipa/asmap/blob/master/testmap.py for example.

laanwj commented at 6:55 pm on April 14, 2022: member

@sipa Sounds like a great idea to me. I had asmap in the back of my head, but somehow thought it was solving a different issue. But it makes sense.

Merge branch 'master' into contrib-seeds-asn 96f0f21749

Merge branch 'master' into contrib-seeds-asn 475bb72701

merge completed c62070fd43

Fix merge issues 7a8150b297

parent 10f629e644819977914860d093dcd1a19313391a

author MarcoFalke <falke.marco@gmail.com> 1649237525 +0200
committer russeree <reese.russell@ymail.com> 1650013843 -0700

parent 10f629e644819977914860d093dcd1a19313391a
author MarcoFalke <falke.marco@gmail.com> 1649237525 +0200
committer russeree <reese.russell@ymail.com> 1650013815 -0700

parent 10f629e644819977914860d093dcd1a19313391a
author MarcoFalke <falke.marco@gmail.com> 1649237525 +0200
committer russeree <reese.russell@ymail.com> 1650013794 -0700

ci: Build all optional tools in tidy task

lint: remove boost::bind linter

I don't think we need to maintain a linter for reintroducing boost::bind
at this point.

doc: Convert remaining comments to clang-tidy format

[docs] package feerate

[packages/policy] use package feerate in package validation

This allows CPFP within a package prior to submission to mempool.

[validation] try individual validation before package validation

This avoids "parents pay for children" and "siblings pay for siblings"
behavior, since package feerate is calculated with totals and is
topology-unaware.

It also ensures that package validation never causes us to reject a
transaction that we would have otherwise accepted in single-tx
validation.

[unit test] package feerate and package cpfp

[validation] don't package validate if not policy or missing inputs

Package validation policy only differs from individual policy in its
evaluation of feerate. Minimize DoS surface; don't validate all over
again if we know the result will be the same.

lint: remove qt SIGNAL/SLOT lint

I think we are past the point where we need to lint for this, the CPU
can probably be better utilized.

refactor: Remove deduplication of data in rollingbloom bench

lint: codespell 2.1.0

lint: flake8 4.0.1

lint: mypy 0.942

refactor: fixup named args in txpackage tests

Regression in #24152.

Put lock logging behind DEBUG_LOCKCONTENTION preprocessor directive

Add DEBUG_LOCKCONTENTION documentation to the developer notes

Squash

Squashed All Previous Commits

RPC: Switch getblockfrompeer back to standard param name blockhash

This commit partially reverts 923312fbf6a89efde1739da0b7209694d4f892ba.

Update RPC argument and field naming guideline in developer notes

Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>

build: fix MSVC build after subtree update

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Aaron Clauson <aaron@sipsorcery.com>

build: remove --enable-experimental from libsecp256k1 configure

build: remove some no-longer-needed var unexporting from configure

key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign

The renaming occured in
https://github.com/bitcoin-core/secp256k1/pull/1089.

Squased lint changes - added features

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

LINT fixes

LINT fixes

Squash - Too many commits

Python include dns.resolver

Failover implementation complete

Removed DNS resolver

Revert changes

refactor: Remove deduplication of data in rollingbloom bench

lint: mypy 0.942

refactor: fixup named args in txpackage tests

Regression in #24152.

Put lock logging behind DEBUG_LOCKCONTENTION preprocessor directive

Add DEBUG_LOCKCONTENTION documentation to the developer notes

Squash

Squashed All Previous Commits

RPC: Switch getblockfrompeer back to standard param name blockhash

This commit partially reverts 923312fbf6a89efde1739da0b7209694d4f892ba.

Update RPC argument and field naming guideline in developer notes

Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>

build: fix MSVC build after subtree update

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Aaron Clauson <aaron@sipsorcery.com>

build: remove --enable-experimental from libsecp256k1 configure

build: remove some no-longer-needed var unexporting from configure

key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign

The renaming occured in
https://github.com/bitcoin-core/secp256k1/pull/1089.

Squased lint changes - added features

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

LINT fixes

LINT fixes

Squash - Too many commits

Python include dns.resolver

Failover implementation complete

Removed DNS resolver

Revert changes

Put lock logging behind DEBUG_LOCKCONTENTION preprocessor directive

Add DEBUG_LOCKCONTENTION documentation to the developer notes

Squash

Squashed All Previous Commits

RPC: Switch getblockfrompeer back to standard param name blockhash

This commit partially reverts 923312fbf6a89efde1739da0b7209694d4f892ba.

Update RPC argument and field naming guideline in developer notes

Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>

build: fix MSVC build after subtree update

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Aaron Clauson <aaron@sipsorcery.com>

build: remove --enable-experimental from libsecp256k1 configure

build: remove some no-longer-needed var unexporting from configure

key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign

The renaming occured in
https://github.com/bitcoin-core/secp256k1/pull/1089.

Squased lint changes - added features

Squashed to reduce spam - added features

Loading ASN directory from a .tsv file.

Added the use of a GZIP compressed ip -> ASN file.

LINT Fixes - 7 words

Fixed comparison to none using 'is' instead of '=='

Fixed unintended changes to readme.md

whitespace correction

restored generate-seeds.py

restored makeseeds.py

Fixed a missing return type within exception

Removed unused variable and if __main__

Updated to fetch and included legacy failover

LINT fixes

LINT fixes

Squash - Too many commits

Python include dns.resolver

Failover implementation complete

Removed DNS resolver

Revert changes

RPC: Switch getblockfrompeer back to standard param name blockhash

This commit partially reverts 923312fbf6a89efde1739da0b7209694d4f892ba.

Update RPC argument and field naming guideline in developer notes

Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>

build: fix MSVC build after subtree update

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Aaron Clauson <aaron@sipsorcery.com>

build: remove --enable-experimental from libsecp256k1 configure

build: remove some no-longer-needed var unexporting from configure

key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign

The renaming occured in
https://github.com/bitcoin-core/secp256k1/pull/1089.

test: compare `/chaininfo` response with `getblockchaininfo` RPC

test: use MiniWallet for feature_fee_estimation.py

This test can now be run even with the Bitcoin Core wallet disabled.

Converted lint-python-mutable-default-parameters.sh to python

Change permission

Change argument so that it's compatiable with python 3.6

Change comment to docstring

Remove .split, .append, .extend calls. Remove 'output' variable assignment

build: Do not define `PROVIDE_FUZZ_MAIN_FUNCTION` macro unconditionally

builder-keys: Add will8clark

gui: add FormatPeerAge() utility helper

Co-authored-by: randymcmillan <randy.lee.mcmillan@gmail.com>

gui: add Age column to peers tab

Co-authored-by: Jon Atack <jon@atack.com>

gui: peersWidget - ResizeToContents Age and IP/Netmask columns

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>

gui: add test runner summary

gui: count test failures in test runner summary

gui, refactor: rename fInvalid to num_test_failures in test_main.cpp

qt: Fix headers

This change is preparation for Qt 6, and it fixes an experimental build
with Qt 6.2.4.

qt: Use `|` instead of `+` for key modifiers

This change is preparation for Qt 6 where `+` has been deprecated, and
it fixes an experimental build with Qt 6.2.4.

qt: Update deprecated enum value

This change is preparation for Qt 6, and it fixes an experimental build
with Qt 6.2.4.
The `Qt::ItemIsTristate` value has been deprecated since 5.6.0 (see
ae8406d82f541f6d9112bdac192e5e4e114d56aa upstream commit).

print `(none)` if no warnings in -getinfo

build, refactor: Drop useless `call` Make function

util, refactor: Add UNIQUE_NAME helper macro

This change replaces repetitive code with a helper macro.

Replace uint256 specific implementations of base_uint::GetHex() and base_uint::SetHex() with proper ones that don't depend on uint256 and replace template methods instantiations of base_uint with template class instantiation

guix: fix GCC 10.3.0 + mingw-w64 setjmp/longjmp issues

This commit backports a patch to the GCC 10.3.0 we build for Windows
cross-compilation in Guix. The commit has been backported to the GCC
releases/gcc-10 branch, but hasn't yet made it into a release.

The patch corrects a regression from an earlier GCC commit, see:
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=357c4350680bf29f0c7a115424e3da11c53b5582
and
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=074226d5aa86cd3de517014acfe34c7f69a2ccc7,
related to the way newer versions of mingw-w64 implement setjmp/longjmp.

Ultimately this was causing a crash for us when Windows users were
viewing the network traffic tab inside the GUI. After some period, long
enough that a buffer would need reallocating, a call into FreeTypes
gray_record_cell() would result in a call to ft_longjmp (longjmp), which
would then trigger a crash.

Fixes: https://github.com/bitcoin-core/gui/issues/582.

See also:
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=e8d1ca7d2c344a411779892616c423e157f4aea8.
https://bugreports.qt.io/browse/QTBUG-93476.

doc: Remove fee delta TODO from txmempool.cpp

net: remove non-blocking bool from interface

lint: Convert lint-logs.sh to Python

test: determine path to `bitcoin-util` in test framework

The path is stored in `self.options.bitcoinutil`, points to
`src/bitcoin-util` by default and can be overrided with the
`BITCOINUTIL` environment variable.

test: add `is_bitcoin_util_compiled` helper

test: add test for signet miner script

depends: Add file-based logging for individual packages

ci: Make log verbose in error case only

This change silences depends build using LOG=1.

doc: Add pre-splitoff translation update to release-process.md

ac83524b20

Merge branch 'contrib-seeds-asn' of https://github.com/russeree/bitcoin into contrib-seeds-asn 8581cad3d2

russeree commented at 9:25 am on April 15, 2022: contributor

I know this isn’t a tech support forum but my branch is quite messed up at this point. Would it be acceptable to close/delete this PR and start over. I feel my code has great value but this PR is not following the contributing guidelines regarding squashing to any degree. At this point there doesn’t seem to be a good solution to squashing/merging to tidy this branch up.

All attempts to squash end up with a massive number of conflicts.

There was a pull request that was merged into makeseeds.py this evening and trying to pull master and merge with my branch didn’t go well at all.

Merge branch 'contrib-seeds-asn' of https://github.com/russeree/bitcoin into contrib-seeds-asn 859c6e605e

Added input arguments and enabled an ASN Database file loader. b2806f9974

fanquake commented at 11:11 am on April 15, 2022: member

Would it be acceptable to close/delete this PR and start over.

Using git is a requirement for contributing to this project. Closing and re-opening PRs is noisy and loses context / discussion. You should just fix your branch up here.

At this point there doesn’t seem to be a good solution to squashing/merging to tidy this branch up.

There are many different ways to achieve this cleanup. I performed the following; you could adapt it for working on your local branch:

0git fetch upstream pull/24824/head:24824
1git checkout 24824
2git reset origin/master
3git add contrib/seeds/*
4git commit -m "contrib: create IP to ASN database from file"
5# git stash'd anything else
6git push origin the-branch-for-this-pr -f

laanwj commented at 1:09 pm on April 15, 2022: member

I might pick this up and implement @sipa’s idea.

Edit: working on this, will open a new PR shortly.

laanwj referenced this in commit 5574de47f6 on Apr 15, 2022

laanwj commented at 3:29 pm on April 15, 2022: member

Closing in favor of #24864

laanwj closed this on Apr 15, 2022

laanwj referenced this in commit b48d2b0fa8 on Apr 15, 2022

laanwj referenced this in commit 08ecc04e5d on Apr 15, 2022

russeree deleted the branch on Apr 15, 2022

russeree commented at 6:22 pm on April 15, 2022: contributor

0git checkout 24824
1git reset origin/master
2git add contrib/seeds/*
3git commit -m "contrib: create IP to ASN database from file"

These steps did work perfectly. @laanwj did close this PR in favor of #24864. I do hope to continue to contribute.

jonatack commented at 6:33 pm on April 15, 2022: member

I do hope to continue to contribute.

:+1: If you like, reviewing and testing #24864 could be a great next step.

laanwj referenced this in commit 4f33193970 on Apr 23, 2022

jonatack referenced this in commit c7fa2a29e2 on Apr 26, 2022

laanwj referenced this in commit 2b088e2eaf on Apr 26, 2022

laanwj referenced this in commit b54180303d on May 31, 2022

laanwj referenced this in commit 7f2c983e1c on Jun 16, 2022

DrahtBot locked this on Apr 15, 2023

net: create IP to ASN database from file - makeseeds.py #24824

Per task number 1 in issue #17020 , “Read ASNs from a file instead of hardcoding”

Conflicts