tests: Allow tests to use a loopback address other than 127.0.0.1 for more test runner parallelism #26841

achow101 commented at 4:42 AM on January 7, 2023: member

I often need to run multiple test_runner.py instances in parallel and the tests will often have port conflicts with other tests being run. While setting TEST_RUNNER_PORT_MIN can help, the number of test instances that can be run is still limited by the number of free ports for the 127.0.0.1 address, and it quickly runs into ports being used by other software at the high end of the port range. However it occurred to me that 127.0.0.1 is not the only loopback address available, and having the tests use different addresses allows us to avoid port conflicts as the other addresses are pretty much unused. This PR introduces a new environment variable BITCOIN_TEST_BIND which can specify an ip address for spawned nodes to bind to. By specifying different loopback addresses for each test_runner.py instance, I can completely avoid port conflicts and run many more in parallel.

In order for the tests to work, the host needs to be setup to have the specified addresses bound to the loopback interface. This can be achieved with a command like ip addr add 127.0.0.2/32 dev lo which would allow env BITCOIN_TEST_BIND=127.0.0.2 test/functional/test_runner.py to work.

References to 127.0.0.1 have been (mostly) changed to LOCALHOST which will match the value specified in BITCOIN_TEST_BIND, with the default being 127.0.0.1.

init: Remove sensitive flag from rpcbind c5fbea97c0

test: Search for any 127.0.0.0/8 address for non-loopback ip in rpc_bind 17678beb12

test: Add LOCALHOST to test_framework/util to track localhost address 14fe3dda62

test: Import LOCALHOST to the tests that will need it

The next commit will be replacing 127.0.0.1 with LOCALHOST in the
functional tests, so the tests that will have this done need to import
it.

7353c5d985

test: Use f-strings to build ip_port in p2p_permissions and rpc_net e9b6d9c19d

test: Use addr_to_hex in feature_anchors 12d80e1522

test: Unset rpcbind and rpcallowip from conf file in rpc_bind 0638f16c52

test: Use LOCALHOST for RPC cb32e98ef8

test: Make sure to replace only "bind" in p2p_permissions 74ef279d39

scripted-diff: Replace hardcoded 127.0.0.1 with LOCALHOST

-BEGIN VERIFY SCRIPT-
sed -E -i -e ":a;s/(f[\'\"].+?)127.0.0.1/\1{LOCALHOST}/;ta" $(git grep -El "f['\"]\S+?127.0.0.1" -- "test/functional/" ":!test/functional/test_framework/util.py" ":!test/functional/rpc_bind.py")
sed -E -i -e ":a;s/['\"]127.0.0.1['\"]/LOCALHOST/;ta" $(git grep -El "['\"]127.0.0.1['\"]" -- "test/functional/*" ":!test/functional/test_framework/util.py" ":!test/functional/rpc_bind.py")
sed -E -i -e ":a;s/(['\"]\S*?)127.0.0.1/f\1{LOCALHOST}/;ta" $(git grep -El "['\"](\S*?)127.0.0.1" -- "test/functional/" ":!test/functional/test_framework/util.py" ":!test/functional/rpc_bind.py")
sed -E -i -e "s/127.0.0.1:/{LOCALHOST}:/" $(git grep -El "127.0.0.1:" -- "test/functional/" ":!test/functional/test_framework/util.py" ":!test/functional/rpc_bind.py")
-END VERIFY SCRIPT-

333d2d7221

DrahtBot commented at 4:42 AM on January 7, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#26625 (test: Run mempool_packages.py with MiniWallet by MarcoFalke)
#26604 (test: add coverage for -bantime by brunoerg)
#26586 (test: previous releases: add v24.0.1 by theStack)
#26467 (bumpfee: Allow the user to choose which output is change by achow101)
#22729 (Make it possible to disable Tor binds and abort startup on bind failure by vasild)
#20892 (tests: Run both descriptor and legacy tests within a single test invocation by achow101)
#17783 (util: Fix -norpcwhitelist, -norpcallowip, and similar corner case behavior by ryanofsky)
#17580 (refactor: Add ALLOW_LIST flags and enforce usage in CheckArgFlags by ryanofsky)
#17493 (util: Forbid ambiguous multiple assignments in config file by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Tests on Jan 7, 2023

achow101 commented at 12:30 AM on January 10, 2023: member

Someone pointed out to me that the same thing could be achieved without modifying anything by using network namespaces, so closing in favor of doing that.

For this interested, here's what I do now for each instance I want to run (scripted, of course):

ip netns add bitcoin_test_netns_1
ip netns exec bitcoin_test_netns_1 ip link set dev lo up
ip netns exec bitcoin_test_netns_1 ip link add dum0 type dummy
ip netns exec bitcoin_test_netns_1 ip addr add 10.1.1.1/32 dev dum0
ip netns exec bitcoin_test_netns_1 ip link set dum0 up
ip netns exec bitcoin_test_netns_1 sudo -u $(whoami) test/functional/test_runner.py -j 60

This results in successfully running all of the tests that would normally be run, and without interfering with each other.

achow101 closed this on Jan 10, 2023

bitcoin locked this on Jan 10, 2024