Run feature_bind_port_(discover|externalip).py in CI #33362

pull vasild wants to merge 2 commits into bitcoin:master from vasild:ci_add1111addr changing 4 files +78 −41
  1. vasild commented at 7:50 AM on September 11, 2025: contributor

    feature_bind_port_discover.py and feature_bind_port_externalip.py require a routable address on the machine to run. Since that was not predictably available on CI, those tests required a manual setting up of IP addresses (e.g. using ifconfig) and then running the tests with a command line option telling them that the addresses are set up. The tests were not run in CI and got rot.

    Change that to auto-detect, from the tests, whether the needed IP addresses are present and if yes, run the test, otherwise skip it. Also change the CI to configure the needed addresses when running the functional tests. This way the tests will be run regularly on CI.

    Fixes: #31336

  2. DrahtBot commented at 7:50 AM on September 11, 2025: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33362.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK willcl-ark

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31723 (qa: Facilitate debugging bitcoind inside functional tests by hodlinator)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. vasild force-pushed on Sep 11, 2025
  4. vasild commented at 8:39 AM on September 11, 2025: contributor

    69177404ba...f047fce8ab: fix typo

  5. in ci/test/02_run_container.sh:91 in f047fce8ab outdated
      85 | @@ -86,7 +86,15 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
      86 |    docker image prune --force --filter "label=$CI_IMAGE_LABEL"
      87 |  
      88 |    # shellcheck disable=SC2086
      89 | -  CI_CONTAINER_ID=$(docker run --cap-add LINUX_IMMUTABLE $CI_CONTAINER_CAP --rm --interactive --detach --tty \
      90 | +  CI_CONTAINER_ID=$(docker run \
      91 | +                  --cap-add LINUX_IMMUTABLE \
      92 | +                  --cap-add NET_ADMIN \
    


    willcl-ark commented at 12:15 PM on September 11, 2025:

    What is the effect on a developer running CI locally on their machine, of adding NET_ADMIN capability? Is is possible for the CI to then modify/break the developer machine networking?

       CAP_NET_ADMIN
              Perform various network-related operations:
              •  interface configuration;
              •  administration of IP firewall, masquerading, and
                 accounting;
              •  modify routing tables;
              •  bind to any address for transparent proxying;
              •  set type-of-service (TOS);
              •  clear driver statistics;
              •  set promiscuous mode;
              •  enabling multicasting;
              •  use setsockopt(2) to set the following socket options:
                 SO_DEBUG, SO_MARK, SO_PRIORITY (for a priority outside
                 the range 0 to 6), SO_RCVBUFFORCE, and SO_SNDBUFFORCE.
                 

    We use our own --network in CI:

    https://github.com/bitcoin/bitcoin/blob/ee42d59d4de970769ebabf77b89ff4269498f61e/ci/test/02_run_container.sh#L71

    But I am unsure if e.g. ip addr add "1.1.1.1/32" adds it to this network only, or the host network?

    If this is affecting (or has the potential to affect) host network, perhaps it can be gated by if $CI or similar?


    vasild commented at 1:07 PM on September 11, 2025:

    The privileges only apply to the guest, not to host. The guest ("container" in docker language) has no way to manipulate the host's networking.

    https://docs.docker.com/engine/containers/run/#container-networking https://docs.docker.com/engine/network/

    We use our own --network in CI ...

    Hmm. Maybe that could be an easier way to add the dummy routable addresses. Then they will be available for the whole duration of the entire CI run, not only during the functional tests as in this PR. I guess this should be ok, will look into this...


    vasild commented at 2:51 PM on September 12, 2025:

    We use our own --network in CI ...

    Maybe that could be an easier way to add the dummy routable addresses

    Yes, done like that. Seems to be smaller change and looks more robust.

  6. vasild force-pushed on Sep 12, 2025
  7. vasild commented at 2:50 PM on September 12, 2025: contributor

    f047fce8ab...9b30b0bea8: set up the needed addresses of the VMs outside of the VMs, when setting them up using docker run and docker network connect, instead of from inside the VM. The former seems to be simpler. Idea from the discussion above.

  8. DrahtBot added the label Needs rebase on Oct 29, 2025
  9. vasild force-pushed on Nov 6, 2025
  10. vasild commented at 4:55 PM on November 6, 2025: contributor

    9b30b0bea8...e245da32dc: rebase due to conflicts

  11. DrahtBot removed the label Needs rebase on Nov 6, 2025
  12. DrahtBot added the label Needs rebase on Dec 19, 2025
  13. in ci/test/02_run_container.py:150 in e245da32dc outdated
     146 | @@ -145,6 +147,8 @@ def main():
     147 |          ).stdout.strip()
     148 |          os.environ["CI_CONTAINER_ID"] = container_id
     149 |  
     150 | +        run(["docker", "network", "connect", "--ip=1.1.1.5", "ci-ip4net", container_id])
    


    frankomosh commented at 7:41 AM on December 31, 2025:

    The container has been attached to both the default bridge network and ci-ip4net, hence two IPv4 addresses. Do you think getifaddrs() may return the bridge IP first, and cause Discover() to find the wrong address?

    If thats the case then I think it might be worth it to disconnect from bridge after connecting to ci-ip4net.


    vasild commented at 9:25 AM on February 4, 2026:

    Discover() finds all addresses, not just the first one?

  14. in ci/test/02_run_container.py:108 in e245da32dc outdated
     104 | @@ -105,6 +105,7 @@ def main():
     105 |              CI_CCACHE_MOUNT = f"type=bind,src={os.environ['CCACHE_DIR']},dst={os.environ['CCACHE_DIR']}"
     106 |  
     107 |          run(["docker", "network", "create", "--ipv6", "--subnet", "1111:1111::/112", "ci-ip6net"], check=False)
     108 | +        run(["docker", "network", "create", "--subnet", "1.1.1.0/24", "ci-ip4net"], check=False)
    


    frankomosh commented at 7:43 AM on December 31, 2025:

    Would it make sense to use RFC-5737 here? (I think they are reserved for documentation and testing)


  15. in test/functional/feature_bind_port_externalip.py:67 in 560395103b outdated
      72 | +        for node in self.nodes:
      73 | +            node.has_explicit_bind = True
      74 | +        try:
      75 | +            self.start_nodes()
      76 | +        except FailedToStartError as e:
      77 | +            for node in self.nodes:
    


    willcl-ark commented at 2:47 PM on January 5, 2026:

    This error/cleanup code is duplicated in both files (with the exception of printing 1 or 2 addresses), would it be worth extracting this into a helper in test_node.py or somewhere?


    vasild commented at 9:29 AM on February 4, 2026:

    Maybe, do you want me to take a short at that?


    NicolaLS commented at 3:10 PM on February 12, 2026:

    Using rg -C 2 "self.start_nodes()" . it does not seem like any other tests do this (surround with try/except) so I am wondering is this cleanup even necessary?

    In case it is, is it only for this test or for all tests? In case you can't argue why it is necessary this should be removed in both files, in case it is only for these tests adding a helper is not worth it imo. in case its relevant for all tests this should be a separate PR (but you can leave the duplicated logic in this PR)


    willcl-ark commented at 8:20 PM on April 28, 2026:

    sure, something like this, I think:

    diff --git a/test/functional/feature_bind_port_discover.py b/test/functional/feature_bind_port_discover.py
    index 86eca321e5e..778fce2fd23 100755
    --- a/test/functional/feature_bind_port_discover.py
    +++ b/test/functional/feature_bind_port_discover.py
    @@ -72,24 +72,13 @@ class BindPortDiscoverTest(BitcoinTestFramework):
             try:
                 self.start_nodes()
             except FailedToStartError as e:
    -            for node in self.nodes:
    -                if node.running:
    -                    if node.rpc_connected:
    -                        node.stop_node(wait=node.rpc_timeout)
    -                    else:
    -                        node.process.kill()
    -                        node.process.wait(timeout=node.rpc_timeout)
    -                        node.process = None
    -                        node.stdout.close()
    -                        node.stderr.close()
    -                        node.running = False
    +            self.cleanup_partially_started_nodes()
                 if 'Unable to bind to ' in str(e):
                     raise SkipTest(
                         f'To run this test make sure that {ADDR1} and {ADDR2} '
                         '(routable addresses) are assigned to non-loopback '
                         'interfaces on this machine')
    -            else:
    -                raise e
    +            raise
    
         def run_test(self):
             self.log.info(
    diff --git a/test/functional/feature_bind_port_externalip.py b/test/functional/feature_bind_port_externalip.py
    index a8e3abe851d..28250611e13 100755
    --- a/test/functional/feature_bind_port_externalip.py
    +++ b/test/functional/feature_bind_port_externalip.py
    @@ -64,24 +64,13 @@ class BindPortExternalIPTest(BitcoinTestFramework):
             try:
                 self.start_nodes()
             except FailedToStartError as e:
    -            for node in self.nodes:
    -                if node.running:
    -                    if node.rpc_connected:
    -                        node.stop_node(wait=node.rpc_timeout)
    -                    else:
    -                        node.process.kill()
    -                        node.process.wait(timeout=node.rpc_timeout)
    -                        node.process = None
    -                        node.stdout.close()
    -                        node.stderr.close()
    -                        node.running = False
    +            self.cleanup_partially_started_nodes()
                 if 'Unable to bind to ' in str(e):
                     raise SkipTest(
                         f'To run this test make sure that {ADDR} '
                         '(routable address) is assigned to non-loopback '
                         'interface on this machine')
    -            else:
    -                raise e
    +            raise
    
         def run_test(self):
             self.log.info("Test the proper port is used for -externalip=")
    diff --git a/test/functional/test_framework/test_framework.py b/test/functional/test_framework/test_framework.py
    index b809e68f4df..8cc7121f102 100755
    --- a/test/functional/test_framework/test_framework.py
    +++ b/test/functional/test_framework/test_framework.py
    @@ -535,6 +535,27 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
                 # Wait for nodes to stop
                 node.wait_until_stopped()
    
    +    def cleanup_partially_started_nodes(self):
    +        """Tear down nodes left running after a failed start_nodes().
    +
    +        After start_nodes() raises (e.g. FailedToStartError), some nodes may be
    +        RPC-connected, some may have a live process without RPC, and some may
    +        already have exited. Stop the connected ones cleanly and force-kill the
    +        rest so the framework's teardown can proceed.
    +        """
    +        for node in self.nodes:
    +            if not node.running:
    +                continue
    +            if node.rpc_connected:
    +                node.stop_node(wait=node.rpc_timeout)
    +            else:
    +                node.process.kill()
    +                node.process.wait(timeout=node.rpc_timeout)
    +                node.process = None
    +                node.stdout.close()
    +                node.stderr.close()
    +                node.running = False
    +
         def restart_node(self, i, extra_args=None, clear_addrman=False, *, expected_stderr=''):
             """Stop and start a test node"""
             self.stop_node(i, expected_stderr=expected_stderr)
    

    I replaced raise e with raise, as in python3 this gets you a fuller traceback, but can drop that part if you don't think it's necessary


    vasild commented at 12:21 PM on April 29, 2026:

    Took the patch, thanks! (added you as co-author).

  16. vasild force-pushed on Feb 4, 2026
  17. vasild commented at 9:30 AM on February 4, 2026: contributor

    e245da32dc4410be684b25c197e64458378a5d11...70558c66068c977df5673adeca4d68a80a3d853f: rebase due to conflicts

  18. DrahtBot removed the label Needs rebase on Feb 4, 2026
  19. in test/functional/feature_bind_port_externalip.py:63 in ff9f1ed9ec outdated
      68 | +    def setup_nodes(self):
      69 | +        self.add_nodes(self.num_nodes, self.extra_args)
      70 | +        # TestNode.start() will add -bind= to extra_args if has_explicit_bind is
      71 | +        # False. We do not want any -bind= thus set has_explicit_bind to True.
      72 | +        for node in self.nodes:
      73 | +            node.has_explicit_bind = True
    


    NicolaLS commented at 2:55 PM on February 12, 2026:

    In case this PR is intended to build on top of #32757 you can add a commit that removes this as this is a workaround that is required before that PR.

    See: #31293 (comment)

    So, if that is fixed, then the above fix to the test will not be necessary.


    NicolaLS commented at 2:57 PM on February 12, 2026:

    Removing it also indirectly tests that the discover fix works and makes this test better.


    vasild commented at 11:16 AM on April 10, 2026:

    feature_bind_port_externalip.py fails if I remove node.has_explicit_bind = True:

    2026-04-10T11:11:30.108371Z TestFramework (ERROR): Unexpected exception:
    Traceback (most recent call last):
      File ".../test/functional/test_framework/test_framework.py", line 143, in main
        self.run_test()
        ~~~~~~~~~~~~~^^
      File ".../test/functional/feature_bind_port_externalip.py", line 95, in run_test
        assert_equal(local['port'], expected_port)
        ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File ".../test/functional/test_framework/util.py", line 83, in assert_equal
        raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
    AssertionError: not(11826 == 30001)
    

    This is because for node0 without node.has_explicit_bind = True the framework adds Command-line arg: bind="0.0.0.0:11826" which overrides the test's own Command-line arg: port="30001".


    vasild commented at 12:15 PM on April 10, 2026:

    Did you mean feature_bind_port_discover.py? That test passes if node.has_explicit_bind = True is removed, but that would cripple the test, making it to test less cases. The test has this for node0 and node1:

    ['-discover', f'-port={self.bind_ports[0]}', '-listen=1'], # Without any -bind
    ['-discover', f'-bind=0.0.0.0:{self.bind_ports[1]}'], # Explicit -bind=0.0.0.0
    

    If node.has_explicit_bind = True is removed then the first one, node0, also gets Command-line arg: bind="0.0.0.0:11894" which is then the same as for node1. In other words, then node0 and node1 will test the same case - with explicit -bind=0.0.0.0 and "without any -bind" will not be tested.


    NicolaLS commented at 4:10 PM on April 10, 2026:

    Yes sorry there were two errors in my reasoning: 1) yes I meant feature_bind_port_discover.py not the file I commented on. 2) I thought that since Discover now properly runs on explicit wildcard binds it is not required to add node.has_explicit_bind = True anymore.

    As far as I understand it now it is still required as a hatch so the testing framework does not inject/override the bind...which seems like an issue with the framework but you are right that node.has_explicit_bind = True can not be removed.

    edit: actually I might meant both files I can honestly not remember anymore but it does not matter.

  20. fanquake marked this as a draft on Feb 27, 2026
  21. fanquake commented at 1:31 PM on February 27, 2026: member

    Moved to draft, given this builds on #32757, and review should go there.

  22. achow101 referenced this in commit fe3d2be1d6 on Apr 9, 2026
  23. DrahtBot added the label Needs rebase on Apr 9, 2026
  24. vasild force-pushed on Apr 10, 2026
  25. vasild commented at 10:21 AM on April 10, 2026: contributor

    70558c66068c977df5673adeca4d68a80a3d853f...60c708c8d3dc976ab9a50339638a6a2e70868a54: rebase due to conflicts, now that #32757 is merged, remove it from this PR.

  26. DrahtBot removed the label Needs rebase on Apr 10, 2026
  27. vasild marked this as ready for review on Apr 10, 2026
  28. vasild commented at 12:15 PM on April 10, 2026: contributor

    Ready for review.

  29. sedited requested review from willcl-ark on Apr 18, 2026
  30. sedited requested review from maflcko on Apr 18, 2026
  31. in test/functional/feature_bind_port_discover.py:33 in 285a5de4b9
      39 | -ADDR2 = '2.2.2.2'
      40 | +# FreeBSD and MacOS:
      41 | +# ifconfig INTERFACE_NAME 1.1.1.5/32 alias && ifconfig INTERFACE_NAME 1111:1111::5/128 alias  # to set up
      42 | +# ifconfig INTERFACE_NAME 1.1.1.5 -alias && ifconfig INTERFACE_NAME 1111:1111::5 -alias       # to remove it, after the test
      43 | +ADDR1 = '1.1.1.5'
      44 | +ADDR2 = '1111:1111::5'
    


    willcl-ark commented at 8:24 PM on April 28, 2026:

    Worth adding a comment here (or perhaps better in in run_container.py) that these must match: https://github.com/bitcoin/bitcoin/blob/285a5de4b9457f3b385fcc04753840cacb1eb61f/ci/test/02_run_container.py#L117 to try and avoid them getting out of sync?


    vasild commented at 12:21 PM on April 29, 2026:

    Done

  32. willcl-ark commented at 8:27 PM on April 28, 2026: member

    Just one comment and a refactor patch to try and remove the duplicated code for your perusal.

  33. test: make feature_bind_port_(discover|externalip).py auto-detect the skip condition
    Instead of requiring a run with an explicit `--ihave1111and2222`, detect
    whether the routable addresses are set up and if not, then skip the test.
    
    To detect whether the addresses are set use `bitcoind` - start it
    and ask it to bind on them and see if it will error with
    "Unable to bind". Since this is what the tests do anyway, just start
    the nodes and see if an exception will be raised like
    `FailedToStartError` / "Unable to bind".
    
    This makes it possible for the CI to run
    `feature_bind_port_discover.py` and
    `feature_bind_port_externalip.py` by just setting up the
    addresses, without having to explicitly provide `--ihave1111and2222`.
    
    Co-authored-by: willcl-ark <will@256k1.dev>
    1b93983bf5
  34. ci: add one more routable address to the VMs (docker containers)
    Also explicitly specify which addresses from the docker network to
    assign to the VM.
    
    With `1.1.1.5` and `1111:1111::5` set on the machine, the tests
    `feature_bind_port_discover.py` and
    `feature_bind_port_externalip.py` will run.
    75cf9708a0
  35. vasild force-pushed on Apr 29, 2026
  36. vasild commented at 12:20 PM on April 29, 2026: contributor

    60c708c8d3dc976ab9a50339638a6a2e70868a54...75cf9708a05d08c3de5dffd91228444d942254b8: address suggestions

  37. willcl-ark approved
  38. willcl-ark commented at 1:40 PM on April 29, 2026: member

    ACK 75cf9708a05d08c3de5dffd91228444d942254b8

    Looks like CI may be stuck, so I guess we have to see why that may be when it concludes. But code looks good to me


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-02 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me