ci: detect outbound internet traffic generated while running tests #31349

pull vasild wants to merge 2 commits into bitcoin:master from vasild:test_log_internet_traffic changing 8 files +71 −4
  1. vasild commented at 1:58 pm on November 22, 2024: contributor

    Prevent generating outbound traffic on a non-loopback interface during tests.

    • Fix node_init_tests/init_test
    • Change CI to catch new regressions, including DNS traffic. DNS traffic is detected because some VMs have configured a non-loopback DNS server, [1111:1111::1]:53. This is achieved by running tcpdump during tests and inspecting its output after the tests. Add --cap-add NET_RAW to docker to be able to run tcpdump.

    Resolves #31339

  2. DrahtBot commented at 1:58 pm on November 22, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31349.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK maflcko, laanwj, jonatack, BrandonOdiwuor, sipa, Sjors, 1440000bytes
    Stale ACK 0xB10C, ryanofsky, fjahr

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #33677 (ci: Retry image building once on failure by maflcko)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Tests on Nov 22, 2024
  4. vasild force-pushed on Nov 22, 2024
  5. DrahtBot commented at 2:12 pm on November 22, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33384128561

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  6. DrahtBot added the label CI failed on Nov 22, 2024
  7. maflcko commented at 2:17 pm on November 22, 2024: member
    Nice. Conecpt ACK!
  8. laanwj commented at 8:33 am on November 25, 2024: member

    Concept ACK

    i’m slightly worried this may generate false positive. As is, this detects traffic on the entire (virtual) machine while running the tests. Are there no other daemons running on the CI instance that could interfere with this?

  9. vasild force-pushed on Nov 25, 2024
  10. vasild commented at 9:08 am on November 25, 2024: contributor

    @laanwj, Right! And ps ax in the VM looks suspiciously scarce: #31339 (comment) showing just bash and 03_test_script.sh.

    Another source of false positive could be if somebody from the outside initiates communication to the VM to which it responds. E.g. an outsider tries to connect to the VM to which it responds with an outbound packet e.g. TCP RST. At least that should be obvious from the error log, showing the incoming packet first (I just pushed a slight change for that). Maybe also the traffic-from-another-daemon could be obvious - e.g. if there is traffic to apt.update.ubuntu.com:443…

  11. vasild force-pushed on Nov 25, 2024
  12. DrahtBot removed the label CI failed on Nov 25, 2024
  13. vasild force-pushed on Nov 25, 2024
  14. DrahtBot added the label CI failed on Nov 25, 2024
  15. DrahtBot commented at 12:51 pm on November 25, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33474555794

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  16. DrahtBot removed the label CI failed on Nov 25, 2024
  17. laanwj commented at 3:07 pm on November 25, 2024: member

    Another source of false positive could be if somebody from the outside initiates communication to the VM to which it responds. E.g. an outsider tries to connect to the VM to which it responds with an outbound packet e.g. TCP RST.

    Exactly. For all we know, the CI VM is firewalled off sufficiently that this can’t happen, but we don’t know.

    At least that should be obvious from the error log, showing the incoming packet first (I just pushed a slight change for that).

    Ah yes, as long as it’s only some extra logging, having a manual factor in this is fine. It only becomes critical if network traffic would cause a CI failure.

    i’m not aware of a straightforward way to “log network traffic of this process and subproceses only”. Yes, it could be done with a linux network namespace, but that’s a lot of hassle.

    And ps ax in the VM looks suspiciously scarce: #31339 (comment) showing just bash and 03_test_script.sh.

    Seeing this, it might already be namespaced. Though a process namespace doesn’t necessarily mean the network namespace is isolated.

  18. vasild force-pushed on Nov 25, 2024
  19. vasild commented at 4:12 pm on November 25, 2024: contributor

    1592a7dad4...071e43ffae: fix feature_config_args.py and p2p_seednode.py to not generate non-loopback traffic.

    It only becomes critical if network traffic would cause a CI failure.

    My intention here is to fail the CI because otherwise the log will be buried in the CI output and nobody will notice it. It follows that if this fails randomly with false positives when one would have to investigate it manually for arbitrary PRs which is highly highly highly undesirable.

  20. in ci/test/03_test_script.sh:160 in 071e43ffae outdated 
    155+}
156+
157+function traffic_monitor_begin()
158+{
159+  for ifname in $(get_interfaces) ; do
160+    tcpdump -n -i "$ifname" -w "$(tcpdump_file_for_interface "$ifname")" &
    maflcko commented at 4:25 pm on November 25, 2024:
    The return code is ignored, which is why CI passed, despite the tests calling out
    vasild commented at 8:54 am on November 26, 2024:

    On some of the VMs this produces:

    0[09:47:01.270] + tcpdump -n -i eth0 -w /tmp/tcpdump_eth0
1[09:47:01.334] tcpdump: eth0: You don't have permission to perform this capture on that device
2[09:47:01.335] (socket: Operation not permitted)

    and then the CI passes because the return code is ignored. I think better not fail the CI when tcpdump does not work in that environment. It is ok as long as tcpdump works on at least one VM to catch problems.

    maflcko commented at 9:04 am on November 26, 2024:
    The problem is that no one will notice if this isn’t run on any machine, because it will silently pass even if there is an error.
    vasild commented at 9:11 am on November 26, 2024:
    True, if it stops working on all VMs, then nobody will notice. Any ideas how to approach this?
    maflcko commented at 9:24 am on November 26, 2024:

    I’d say it is fine to ignore it by default (if you want). However, there should be one machine in the CI matrix to run the check (and fail on any error).

    The cirrus workers are running in a user account, so they may not have the permissions (unless they are switched to @0xB10C’s workers, which are running as root?). Alternatively, you could try with --cap-add=.../--privileged, but I haven’t tried this. I guess the only task that has the required permissions right now is the ASan GHA task?

    0xB10C commented at 11:22 am on November 27, 2024:

    (unless they are switched to @0xB10C’s workers, which are running as root?)

    the runner setup I’m working on explicitly doesn’t run as root and is far from finished :)

    maflcko commented at 11:52 am on November 27, 2024:

    the runner setup I’m working on explicitly doesn’t run as root

    Are you sure, because the current CI (in this run) is run in a user account (not root), and gives a permission error. The same CI in your run does not give a permission error, so there seems to be a difference.

    The only thing I see is that you are using docker, which IIRC is running rootful by default.

    0xB10C commented at 1:30 pm on November 27, 2024:

    I’m using rootless-docker which runs dockerd as a user account. Inside the container, you’re root and can tcpdump on the containers eth0 interface, but you can’t* e.g. mount and edit the hosts /etc/passwd like you can’t with the user account.

    *until someone finds a vuln in rootless-docker

    maflcko commented at 1:59 pm on November 27, 2024:

    It may be that docker rootless has a different capabilities set, compared to podman. (Can be checked with capsh --print).

    In any case, my preference would be to explicitly list the required caps, instead of relying on a vendor default.

    vasild commented at 3:43 pm on November 27, 2024:

    The problem is that no one will notice if this isn’t run on any machine

    I changed it to insist that the tcpdump file was created on the ASAN env. So the ASAN job will be red if this stops working on it.

    maflcko commented at 7:59 am on November 28, 2024:

    I don’t think this is sufficient. The CI failure (https://github.com/bitcoin/bitcoin/pull/31349#issuecomment-2499335672) will remain, depending on a vendor default.

    Again, my preference would be to explicitly list the required (or removed) caps, instead of relying on a vendor default. Otherwise, it will become harder to run the CI locally, or lead to vendor-lock-in.

    vasild commented at 8:41 am on December 3, 2024:

    … explicitly list the required (or removed) caps …

    Done, NET_RAW is required to run tcpdump: https://www.tcpdump.org/manpages/pcap.3pcap.html

    Plus, the ASAN job requires that tcpdump -w runs and creates the file, otherwise it will be red.

    I think that resolves the concerns from this thread, so I am closing it. Feel free to comment / reopen if there is more to this.

  21. 0xB10C commented at 0:55 am on November 26, 2024: contributor

    Ran this on my CI runner which has 8.8.8.8 configured as DNS server for docker.

    https://cirrus-ci.com/task/5500763260059648?logs=ci#L1137

    0[00:46:26.215] + tcpdump -n -r /tmp/tcpdump_eth0 tcp or udp
1[00:46:26.219] 00:42:50.052764 IP 172.18.0.2.46566 > 8.8.8.8.53: 39301+ A? x9.dummySeed.invalid. (38)
2[00:46:26.219] 00:42:50.053181 IP 172.18.0.2.58686 > 8.8.8.8.53: 36487+ AAAA? x9.dummySeed.invalid. (38)
3[00:46:26.219] 00:42:50.059038 IP 8.8.8.8.53 > 172.18.0.2.46566: 39301 NXDomain 0/1/0 (113)
4[00:46:26.219] 00:42:50.060121 IP 8.8.8.8.53 > 172.18.0.2.58686: 36487 NXDomain 0/1/0 (113)
5[00:46:26.219] 00:42:50.060574 IP 172.18.0.2.34312 > 8.8.8.8.53: 25243+ A? x9.dummySeed.invalid. (38)
6[00:46:26.219] 00:42:50.060939 IP 172.18.0.2.47040 > 8.8.8.8.53: 63641+ AAAA? x9.dummySeed.invalid. (38)
7...

    Edit: My understanding is as follows: The DNS requests normally go to a local DNS resolver which then asks an upstream resolver. The upstream resolver (possibly your ISP) indirectly learns that you are running Bitcoin Core tests, even if there was no direct communication over a non-loopback interface.

      0[00:46:26.209] ++ tcpdump -n -r /tmp/tcpdump_eth0 --direction=out tcp or udp
  1[00:46:26.213] reading from file /tmp/tcpdump_eth0, link-type EN10MB (Ethernet), snapshot length 262144
  2[00:46:26.215] + '[' -n '00:42:50.052764 IP 172.18.0.2.46566 > 8.8.8.8.53: 39301+ A? x9.dummySeed.invalid. (38)
  3[00:46:26.215] 00:42:50.053181 IP 172.18.0.2.58686 > 8.8.8.8.53: 36487+ AAAA? x9.dummySeed.invalid. (38)
  4[00:46:26.215] 00:42:50.059038 IP 8.8.8.8.53 > 172.18.0.2.46566: 39301 NXDomain 0/1/0 (113)
  5[00:46:26.215] 00:42:50.060121 IP 8.8.8.8.53 > 172.18.0.2.58686: 36487 NXDomain 0/1/0 (113)
  6[00:46:26.215] 00:42:50.060574 IP 172.18.0.2.34312 > 8.8.8.8.53: 25243+ A? x9.dummySeed.invalid. (38)
  7[00:46:26.215] 00:42:50.060939 IP 172.18.0.2.47040 > 8.8.8.8.53: 63641+ AAAA? x9.dummySeed.invalid. (38)
  8[00:46:26.215] 00:42:50.066767 IP 8.8.8.8.53 > 172.18.0.2.34312: 25243 NXDomain 0/1/0 (113)
  9[00:46:26.215] 00:42:50.068273 IP 8.8.8.8.53 > 172.18.0.2.47040: 63641 NXDomain 0/1/0 (113)
 10[00:46:26.215] 00:42:50.420185 IP 172.18.0.2.55135 > 8.8.8.8.53: 9419+ A? fakenodeaddr.fakedomain.invalid. (49)
 11[00:46:26.215] 00:42:50.420589 IP 172.18.0.2.42709 > 8.8.8.8.53: 57544+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 12[00:46:26.215] 00:42:50.426139 IP 8.8.8.8.53 > 172.18.0.2.55135: 9419 NXDomain 0/1/0 (124)
 13[00:46:26.215] 00:42:50.426488 IP 8.8.8.8.53 > 172.18.0.2.42709: 57544 NXDomain 0/1/0 (124)
 14[00:46:26.215] 00:42:50.426928 IP 172.18.0.2.42643 > 8.8.8.8.53: 12212+ A? fakenodeaddr.fakedomain.invalid. (49)
 15[00:46:26.215] 00:42:50.427362 IP 172.18.0.2.33528 > 8.8.8.8.53: 41906+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 16[00:46:26.215] 00:42:50.432905 IP 8.8.8.8.53 > 172.18.0.2.42643: 12212 NXDomain 0/1/0 (124)
 17[00:46:26.215] 00:42:50.433389 IP 8.8.8.8.53 > 172.18.0.2.33528: 41906 NXDomain 0/1/0 (124)
 18[00:46:26.215] 00:42:50.785947 IP 172.18.0.2.40413 > 8.8.8.8.53: 23723+ A? x9.dummySeed.invalid. (38)
 19[00:46:26.215] 00:42:50.786307 IP 172.18.0.2.33152 > 8.8.8.8.53: 51880+ AAAA? x9.dummySeed.invalid. (38)
 20[00:46:26.215] 00:42:50.786916 IP 172.18.0.2.60591 > 8.8.8.8.53: 57214+ A? fakenodeaddr.fakedomain.invalid. (49)
 21[00:46:26.215] 00:42:50.787237 IP 172.18.0.2.51085 > 8.8.8.8.53: 45180+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 22[00:46:26.215] 00:42:50.792174 IP 8.8.8.8.53 > 172.18.0.2.33152: 51880 NXDomain 0/1/0 (113)
 23[00:46:26.215] 00:42:50.792196 IP 8.8.8.8.53 > 172.18.0.2.40413: 23723 NXDomain 0/1/0 (113)
 24[00:46:26.215] 00:42:50.794262 IP 8.8.8.8.53 > 172.18.0.2.51085: 45180 NXDomain 0/1/0 (124)
 25[00:46:26.215] 00:42:50.794281 IP 8.8.8.8.53 > 172.18.0.2.60591: 57214 NXDomain 0/1/0 (124)
 26[00:46:26.215] 00:42:50.794701 IP 172.18.0.2.53594 > 8.8.8.8.53: 7814+ AAAA? x9.dummySeed.invalid. (38)
 27[00:46:26.215] 00:42:50.794819 IP 172.18.0.2.33826 > 8.8.8.8.53: 50053+ A? x9.dummySeed.invalid. (38)
 28[00:46:26.215] 00:42:50.795297 IP 172.18.0.2.54482 > 8.8.8.8.53: 30981+ A? fakenodeaddr.fakedomain.invalid. (49)
 29[00:46:26.215] 00:42:50.795592 IP 172.18.0.2.48225 > 8.8.8.8.53: 65050+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 30[00:46:26.215] 00:42:50.800988 IP 8.8.8.8.53 > 172.18.0.2.33826: 50053 NXDomain 0/1/0 (113)
 31[00:46:26.215] 00:42:50.801160 IP 8.8.8.8.53 > 172.18.0.2.53594: 7814 NXDomain 0/1/0 (113)
 32[00:46:26.215] 00:42:50.801189 IP 8.8.8.8.53 > 172.18.0.2.54482: 30981 NXDomain 0/1/0 (124)
 33[00:46:26.215] 00:42:50.801282 IP 8.8.8.8.53 > 172.18.0.2.48225: 65050 NXDomain 0/1/0 (124)
 34[00:46:26.215] 00:42:51.151925 IP 172.18.0.2.45895 > 8.8.8.8.53: 32540+ A? x9.dummySeed.invalid. (38)
 35[00:46:26.215] 00:42:51.154708 IP 172.18.0.2.39532 > 8.8.8.8.53: 3102+ AAAA? x9.dummySeed.invalid. (38)
 36[00:46:26.215] 00:42:51.160435 IP 8.8.8.8.53 > 172.18.0.2.45895: 32540 NXDomain 0/1/0 (113)
 37[00:46:26.215] 00:42:51.160623 IP 8.8.8.8.53 > 172.18.0.2.39532: 3102 NXDomain 0/1/0 (113)
 38[00:46:26.215] 00:42:51.161037 IP 172.18.0.2.54958 > 8.8.8.8.53: 64262+ A? x9.dummySeed.invalid. (38)
 39[00:46:26.215] 00:42:51.161501 IP 172.18.0.2.55394 > 8.8.8.8.53: 38663+ AAAA? x9.dummySeed.invalid. (38)
 40[00:46:26.215] 00:42:51.167068 IP 8.8.8.8.53 > 172.18.0.2.54958: 64262 NXDomain 0/1/0 (113)
 41[00:46:26.215] 00:42:51.167562 IP 8.8.8.8.53 > 172.18.0.2.55394: 38663 NXDomain 0/1/0 (113)
 42[00:46:26.215] 00:43:13.896684 IP 172.18.0.2.38577 > 8.8.8.8.53: 37859+ A? x9.dummySeed.invalid. (38)
 43[00:46:26.215] 00:43:13.898035 IP 172.18.0.2.57872 > 8.8.8.8.53: 26852+ AAAA? x9.dummySeed.invalid. (38)
 44[00:46:26.215] 00:43:13.904071 IP 8.8.8.8.53 > 172.18.0.2.38577: 37859 NXDomain 0/1/0 (113)
 45[00:46:26.215] 00:43:13.904515 IP 8.8.8.8.53 > 172.18.0.2.57872: 26852 NXDomain 0/1/0 (113)
 46[00:46:26.215] 00:43:13.905051 IP 172.18.0.2.41442 > 8.8.8.8.53: 3978+ A? x9.dummySeed.invalid. (38)
 47[00:46:26.215] 00:43:13.905744 IP 172.18.0.2.51126 > 8.8.8.8.53: 44169+ AAAA? x9.dummySeed.invalid. (38)
 48[00:46:26.215] 00:43:13.910763 IP 8.8.8.8.53 > 172.18.0.2.41442: 3978 NXDomain 0/1/0 (113)
 49[00:46:26.215] 00:43:13.912478 IP 8.8.8.8.53 > 172.18.0.2.51126: 44169 NXDomain 0/1/0 (113)
 50[00:46:26.215] 00:43:14.345922 IP 172.18.0.2.46373 > 8.8.8.8.53: 23556+ AAAA? x9.dummySeed.invalid. (38)
 51[00:46:26.215] 00:43:14.346104 IP 172.18.0.2.55199 > 8.8.8.8.53: 59658+ A? x9.dummySeed.invalid. (38)
 52[00:46:26.215] 00:43:14.352039 IP 8.8.8.8.53 > 172.18.0.2.55199: 59658 NXDomain 0/1/0 (113)
 53[00:46:26.215] 00:43:14.352108 IP 8.8.8.8.53 > 172.18.0.2.46373: 23556 NXDomain 0/1/0 (113)
 54[00:46:26.215] 00:43:14.355678 IP 172.18.0.2.40940 > 8.8.8.8.53: 56364+ AAAA? x9.dummySeed.invalid. (38)
 55[00:46:26.215] 00:43:14.356531 IP 172.18.0.2.51128 > 8.8.8.8.53: 29229+ A? x9.dummySeed.invalid. (38)
 56[00:46:26.215] 00:43:14.361956 IP 8.8.8.8.53 > 172.18.0.2.40940: 56364 NXDomain 0/1/0 (113)
 57[00:46:26.215] 00:43:14.362506 IP 8.8.8.8.53 > 172.18.0.2.51128: 29229 NXDomain 0/1/0 (113)
 58[00:46:26.215] 00:44:27.054154 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043001621 ecr 0,nop,wscale 7], length 0
 59[00:46:26.215] 00:44:27.945389 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043002513 ecr 0,nop,wscale 7], length 0
 60[00:46:26.215] 00:44:28.096227 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043002664 ecr 0,nop,wscale 7], length 0
 61[00:46:26.215] 00:44:28.992218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043003560 ecr 0,nop,wscale 7], length 0
 62[00:46:26.215] 00:44:29.120219 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043003688 ecr 0,nop,wscale 7], length 0
 63[00:46:26.215] 00:44:30.016217 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043004584 ecr 0,nop,wscale 7], length 0
 64[00:46:26.215] 00:44:30.144232 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043004712 ecr 0,nop,wscale 7], length 0
 65[00:46:26.215] 00:44:31.040220 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043005608 ecr 0,nop,wscale 7], length 0
 66[00:46:26.215] 00:44:31.168218 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043005736 ecr 0,nop,wscale 7], length 0
 67[00:46:26.215] 00:44:32.064218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043006632 ecr 0,nop,wscale 7], length 0
 68[00:46:26.215] 00:45:32.199353 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531031147 ecr 0,nop,wscale 7], length 0
 69[00:46:26.215] 00:45:33.248224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531032196 ecr 0,nop,wscale 7], length 0
 70[00:46:26.215] 00:45:34.272224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531033220 ecr 0,nop,wscale 7], length 0
 71[00:46:26.215] 00:45:35.296253 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531034244 ecr 0,nop,wscale 7], length 0
 72[00:46:26.215] 00:45:36.320281 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531035268 ecr 0,nop,wscale 7], length 0
 73[00:46:26.215] 00:45:37.442439 IP 172.18.0.2.52570 > 8.8.8.8.53: 3832+ AAAA? some.node. (27)
 74[00:46:26.215] 00:45:37.442487 IP 172.18.0.2.60704 > 8.8.8.8.53: 1023+ A? some.node. (27)
 75[00:46:26.215] 00:45:37.448551 IP 8.8.8.8.53 > 172.18.0.2.60704: 1023 NXDomain 0/1/0 (102)
 76[00:46:26.215] 00:45:37.448911 IP 8.8.8.8.53 > 172.18.0.2.52570: 3832 NXDomain 0/1/0 (102)
 77[00:46:26.215] 00:45:37.449395 IP 172.18.0.2.53889 > 8.8.8.8.53: 63359+ AAAA? some.node. (27)
 78[00:46:26.215] 00:45:37.449419 IP 172.18.0.2.51133 > 8.8.8.8.53: 6268+ A? some.node. (27)
 79[00:46:26.215] 00:45:37.455701 IP 8.8.8.8.53 > 172.18.0.2.51133: 6268 NXDomain 0/1/0 (102)
 80[00:46:26.215] 00:45:37.455743 IP 8.8.8.8.53 > 172.18.0.2.53889: 63359 NXDomain 0/1/0 (102)
 81[00:46:26.215] 00:45:37.734057 IP 172.18.0.2.50765 > 8.8.8.8.53: 34600+ AAAA? x9.dummySeed.invalid. (38)
 82[00:46:26.215] 00:45:37.734277 IP 172.18.0.2.47474 > 8.8.8.8.53: 53797+ A? x9.dummySeed.invalid. (38)
 83[00:46:26.215] 00:45:37.740114 IP 8.8.8.8.53 > 172.18.0.2.50765: 34600 NXDomain 0/1/0 (113)
 84[00:46:26.215] 00:45:37.740144 IP 8.8.8.8.53 > 172.18.0.2.47474: 53797 NXDomain 0/1/0 (113)
 85[00:46:26.215] 00:45:37.740754 IP 172.18.0.2.33593 > 8.8.8.8.53: 4491+ A? x9.dummySeed.invalid. (38)
 86[00:46:26.215] 00:45:37.740756 IP 172.18.0.2.34759 > 8.8.8.8.53: 34698+ AAAA? x9.dummySeed.invalid. (38)
 87[00:46:26.215] 00:45:37.746602 IP 8.8.8.8.53 > 172.18.0.2.34759: 34698 NXDomain 0/1/0 (113)
 88[00:46:26.215] 00:45:37.746641 IP 8.8.8.8.53 > 172.18.0.2.33593: 4491 NXDomain 0/1/0 (113)
 89[00:46:26.215] 00:45:38.234283 IP 172.18.0.2.33343 > 8.8.8.8.53: 22856+ A? dummySeed.invalid. (35)
 90[00:46:26.215] 00:45:38.234286 IP 172.18.0.2.43752 > 8.8.8.8.53: 55883+ AAAA? dummySeed.invalid. (35)
 91[00:46:26.215] 00:45:38.240530 IP 8.8.8.8.53 > 172.18.0.2.43752: 55883 NXDomain 0/1/0 (110)
 92[00:46:26.215] 00:45:38.240809 IP 8.8.8.8.53 > 172.18.0.2.33343: 22856 NXDomain 0/1/0 (110)
 93[00:46:26.215] 00:45:38.241237 IP 172.18.0.2.35861 > 8.8.8.8.53: 14859+ A? dummySeed.invalid. (35)
 94[00:46:26.215] 00:45:38.241238 IP 172.18.0.2.51487 > 8.8.8.8.53: 46093+ AAAA? dummySeed.invalid. (35)
 95[00:46:26.215] 00:45:38.247292 IP 8.8.8.8.53 > 172.18.0.2.35861: 14859 NXDomain 0/1/0 (110)
 96[00:46:26.215] 00:45:38.247320 IP 8.8.8.8.53 > 172.18.0.2.51487: 46093 NXDomain 0/1/0 (110)
 97[00:46:26.215] 00:45:39.779784 IP 172.18.0.2.34493 > 8.8.8.8.53: 26799+ A? fakenodeaddr. (30)
 98[00:46:26.215] 00:45:39.779791 IP 172.18.0.2.56878 > 8.8.8.8.53: 61600+ AAAA? fakenodeaddr. (30)
 99[00:46:26.215] 00:45:39.785890 IP 8.8.8.8.53 > 172.18.0.2.56878: 61600 NXDomain 0/1/0 (105)
100[00:46:26.215] 00:45:39.785928 IP 8.8.8.8.53 > 172.18.0.2.34493: 26799 NXDomain 0/1/0 (105)
101[00:46:26.215] 00:45:39.786515 IP 172.18.0.2.54375 > 8.8.8.8.53: 52196+ A? fakenodeaddr. (30)
102[00:46:26.215] 00:45:39.786693 IP 172.18.0.2.53292 > 8.8.8.8.53: 56037+ AAAA? fakenodeaddr. (30)
103[00:46:26.215] 00:45:39.792298 IP 8.8.8.8.53 > 172.18.0.2.54375: 52196 NXDomain 0/1/0 (105)
104[00:46:26.215] 00:45:39.792592 IP 8.8.8.8.53 > 172.18.0.2.53292: 56037 NXDomain 0/1/0 (105)
105[00:46:26.215] 00:45:42.174906 IP 11.22.33.44.18444 > 172.18.0.2.51812: Flags [R.], seq 0, ack 3432062660, win 65535, length 0
106[00:46:26.215] 00:45:42.722951 IP 172.18.0.2.33938 > 8.8.8.8.53: 23646+ AAAA? fakeaddress1. (30)
107[00:46:26.215] 00:45:42.723043 IP 172.18.0.2.38954 > 8.8.8.8.53: 24914+ A? fakeaddress1. (30)
108[00:46:26.215] 00:45:42.728907 IP 8.8.8.8.53 > 172.18.0.2.38954: 24914 NXDomain 0/1/0 (105)
109[00:46:26.215] 00:45:42.729111 IP 8.8.8.8.53 > 172.18.0.2.33938: 23646 NXDomain 0/1/0 (105)
110[00:46:26.215] 00:45:42.729684 IP 172.18.0.2.52893 > 8.8.8.8.53: 3801+ AAAA? fakeaddress1. (30)
111[00:46:26.215] 00:45:42.729715 IP 172.18.0.2.57497 > 8.8.8.8.53: 62431+ A? fakeaddress1. (30)
112[00:46:26.215] 00:45:42.735746 IP 8.8.8.8.53 > 172.18.0.2.57497: 62431 NXDomain 0/1/0 (105)
113[00:46:26.215] 00:45:42.735771 IP 8.8.8.8.53 > 172.18.0.2.52893: 3801 NXDomain 0/1/0 (105)
114[00:46:26.215] 00:45:43.175350 IP 11.22.33.44.18444 > 172.18.0.2.51828: Flags [R.], seq 0, ack 3743466588, win 65535, length 0' ']'
115[00:46:26.215] + echo 'Outbound TCP or UDP packets on the non loopback interface generated during tests:'
116[00:46:26.215] Outbound TCP or UDP packets on the non loopback interface generated during tests:
117[00:46:26.215] + tcpdump -n -r /tmp/tcpdump_eth0 tcp or udp
118[00:46:26.218] reading from file /tmp/tcpdump_eth0, link-type EN10MB (Ethernet), snapshot length 262144
119[00:46:26.219] 00:42:50.052764 IP 172.18.0.2.46566 > 8.8.8.8.53: 39301+ A? x9.dummySeed.invalid. (38)
120[00:46:26.219] 00:42:50.053181 IP 172.18.0.2.58686 > 8.8.8.8.53: 36487+ AAAA? x9.dummySeed.invalid. (38)
121[00:46:26.219] 00:42:50.059038 IP 8.8.8.8.53 > 172.18.0.2.46566: 39301 NXDomain 0/1/0 (113)
122[00:46:26.219] 00:42:50.060121 IP 8.8.8.8.53 > 172.18.0.2.58686: 36487 NXDomain 0/1/0 (113)
123[00:46:26.219] 00:42:50.060574 IP 172.18.0.2.34312 > 8.8.8.8.53: 25243+ A? x9.dummySeed.invalid. (38)
124[00:46:26.219] 00:42:50.060939 IP 172.18.0.2.47040 > 8.8.8.8.53: 63641+ AAAA? x9.dummySeed.invalid. (38)
125[00:46:26.219] 00:42:50.066767 IP 8.8.8.8.53 > 172.18.0.2.34312: 25243 NXDomain 0/1/0 (113)
126[00:46:26.219] 00:42:50.068273 IP 8.8.8.8.53 > 172.18.0.2.47040: 63641 NXDomain 0/1/0 (113)
127[00:46:26.219] 00:42:50.420185 IP 172.18.0.2.55135 > 8.8.8.8.53: 9419+ A? fakenodeaddr.fakedomain.invalid. (49)
128[00:46:26.219] 00:42:50.420589 IP 172.18.0.2.42709 > 8.8.8.8.53: 57544+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
129[00:46:26.219] 00:42:50.426139 IP 8.8.8.8.53 > 172.18.0.2.55135: 9419 NXDomain 0/1/0 (124)
130[00:46:26.219] 00:42:50.426488 IP 8.8.8.8.53 > 172.18.0.2.42709: 57544 NXDomain 0/1/0 (124)
131[00:46:26.219] 00:42:50.426928 IP 172.18.0.2.42643 > 8.8.8.8.53: 12212+ A? fakenodeaddr.fakedomain.invalid. (49)
132[00:46:26.219] 00:42:50.427362 IP 172.18.0.2.33528 > 8.8.8.8.53: 41906+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
133[00:46:26.219] 00:42:50.432905 IP 8.8.8.8.53 > 172.18.0.2.42643: 12212 NXDomain 0/1/0 (124)
134[00:46:26.219] 00:42:50.433389 IP 8.8.8.8.53 > 172.18.0.2.33528: 41906 NXDomain 0/1/0 (124)
135[00:46:26.219] 00:42:50.785947 IP 172.18.0.2.40413 > 8.8.8.8.53: 23723+ A? x9.dummySeed.invalid. (38)
136[00:46:26.219] 00:42:50.786307 IP 172.18.0.2.33152 > 8.8.8.8.53: 51880+ AAAA? x9.dummySeed.invalid. (38)
137[00:46:26.219] 00:42:50.786916 IP 172.18.0.2.60591 > 8.8.8.8.53: 57214+ A? fakenodeaddr.fakedomain.invalid. (49)
138[00:46:26.219] 00:42:50.787237 IP 172.18.0.2.51085 > 8.8.8.8.53: 45180+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
139[00:46:26.219] 00:42:50.792174 IP 8.8.8.8.53 > 172.18.0.2.33152: 51880 NXDomain 0/1/0 (113)
140[00:46:26.219] 00:42:50.792196 IP 8.8.8.8.53 > 172.18.0.2.40413: 23723 NXDomain 0/1/0 (113)
141[00:46:26.219] 00:42:50.794262 IP 8.8.8.8.53 > 172.18.0.2.51085: 45180 NXDomain 0/1/0 (124)
142[00:46:26.219] 00:42:50.794281 IP 8.8.8.8.53 > 172.18.0.2.60591: 57214 NXDomain 0/1/0 (124)
143[00:46:26.219] 00:42:50.794701 IP 172.18.0.2.53594 > 8.8.8.8.53: 7814+ AAAA? x9.dummySeed.invalid. (38)
144[00:46:26.219] 00:42:50.794819 IP 172.18.0.2.33826 > 8.8.8.8.53: 50053+ A? x9.dummySeed.invalid. (38)
145[00:46:26.219] 00:42:50.795297 IP 172.18.0.2.54482 > 8.8.8.8.53: 30981+ A? fakenodeaddr.fakedomain.invalid. (49)
146[00:46:26.219] 00:42:50.795592 IP 172.18.0.2.48225 > 8.8.8.8.53: 65050+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
147[00:46:26.219] 00:42:50.800988 IP 8.8.8.8.53 > 172.18.0.2.33826: 50053 NXDomain 0/1/0 (113)
148[00:46:26.219] 00:42:50.801160 IP 8.8.8.8.53 > 172.18.0.2.53594: 7814 NXDomain 0/1/0 (113)
149[00:46:26.219] 00:42:50.801189 IP 8.8.8.8.53 > 172.18.0.2.54482: 30981 NXDomain 0/1/0 (124)
150[00:46:26.219] 00:42:50.801282 IP 8.8.8.8.53 > 172.18.0.2.48225: 65050 NXDomain 0/1/0 (124)
151[00:46:26.219] 00:42:51.151925 IP 172.18.0.2.45895 > 8.8.8.8.53: 32540+ A? x9.dummySeed.invalid. (38)
152[00:46:26.219] 00:42:51.154708 IP 172.18.0.2.39532 > 8.8.8.8.53: 3102+ AAAA? x9.dummySeed.invalid. (38)
153[00:46:26.219] 00:42:51.160435 IP 8.8.8.8.53 > 172.18.0.2.45895: 32540 NXDomain 0/1/0 (113)
154[00:46:26.219] 00:42:51.160623 IP 8.8.8.8.53 > 172.18.0.2.39532: 3102 NXDomain 0/1/0 (113)
155[00:46:26.219] 00:42:51.161037 IP 172.18.0.2.54958 > 8.8.8.8.53: 64262+ A? x9.dummySeed.invalid. (38)
156[00:46:26.219] 00:42:51.161501 IP 172.18.0.2.55394 > 8.8.8.8.53: 38663+ AAAA? x9.dummySeed.invalid. (38)
157[00:46:26.219] 00:42:51.167068 IP 8.8.8.8.53 > 172.18.0.2.54958: 64262 NXDomain 0/1/0 (113)
158[00:46:26.219] 00:42:51.167562 IP 8.8.8.8.53 > 172.18.0.2.55394: 38663 NXDomain 0/1/0 (113)
159[00:46:26.219] 00:43:13.896684 IP 172.18.0.2.38577 > 8.8.8.8.53: 37859+ A? x9.dummySeed.invalid. (38)
160[00:46:26.219] 00:43:13.898035 IP 172.18.0.2.57872 > 8.8.8.8.53: 26852+ AAAA? x9.dummySeed.invalid. (38)
161[00:46:26.219] 00:43:13.904071 IP 8.8.8.8.53 > 172.18.0.2.38577: 37859 NXDomain 0/1/0 (113)
162[00:46:26.219] 00:43:13.904515 IP 8.8.8.8.53 > 172.18.0.2.57872: 26852 NXDomain 0/1/0 (113)
163[00:46:26.219] 00:43:13.905051 IP 172.18.0.2.41442 > 8.8.8.8.53: 3978+ A? x9.dummySeed.invalid. (38)
164[00:46:26.219] 00:43:13.905744 IP 172.18.0.2.51126 > 8.8.8.8.53: 44169+ AAAA? x9.dummySeed.invalid. (38)
165[00:46:26.219] 00:43:13.910763 IP 8.8.8.8.53 > 172.18.0.2.41442: 3978 NXDomain 0/1/0 (113)
166[00:46:26.219] 00:43:13.912478 IP 8.8.8.8.53 > 172.18.0.2.51126: 44169 NXDomain 0/1/0 (113)
167[00:46:26.219] 00:43:14.345922 IP 172.18.0.2.46373 > 8.8.8.8.53: 23556+ AAAA? x9.dummySeed.invalid. (38)
168[00:46:26.219] 00:43:14.346104 IP 172.18.0.2.55199 > 8.8.8.8.53: 59658+ A? x9.dummySeed.invalid. (38)
169[00:46:26.219] 00:43:14.352039 IP 8.8.8.8.53 > 172.18.0.2.55199: 59658 NXDomain 0/1/0 (113)
170[00:46:26.219] 00:43:14.352108 IP 8.8.8.8.53 > 172.18.0.2.46373: 23556 NXDomain 0/1/0 (113)
171[00:46:26.219] 00:43:14.355678 IP 172.18.0.2.40940 > 8.8.8.8.53: 56364+ AAAA? x9.dummySeed.invalid. (38)
172[00:46:26.219] 00:43:14.356531 IP 172.18.0.2.51128 > 8.8.8.8.53: 29229+ A? x9.dummySeed.invalid. (38)
173[00:46:26.219] 00:43:14.361956 IP 8.8.8.8.53 > 172.18.0.2.40940: 56364 NXDomain 0/1/0 (113)
174[00:46:26.219] 00:43:14.362506 IP 8.8.8.8.53 > 172.18.0.2.51128: 29229 NXDomain 0/1/0 (113)
175[00:46:26.219] 00:44:27.054154 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043001621 ecr 0,nop,wscale 7], length 0
176[00:46:26.219] 00:44:27.945389 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043002513 ecr 0,nop,wscale 7], length 0
177[00:46:26.219] 00:44:28.096227 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043002664 ecr 0,nop,wscale 7], length 0
178[00:46:26.219] 00:44:28.992218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043003560 ecr 0,nop,wscale 7], length 0
179[00:46:26.219] 00:44:29.120219 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043003688 ecr 0,nop,wscale 7], length 0
180[00:46:26.219] 00:44:30.016217 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043004584 ecr 0,nop,wscale 7], length 0
181[00:46:26.219] 00:44:30.144232 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043004712 ecr 0,nop,wscale 7], length 0
182[00:46:26.219] 00:44:31.040220 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043005608 ecr 0,nop,wscale 7], length 0
183[00:46:26.219] 00:44:31.168218 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043005736 ecr 0,nop,wscale 7], length 0
184[00:46:26.219] 00:44:32.064218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043006632 ecr 0,nop,wscale 7], length 0
185[00:46:26.219] 00:45:32.199353 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531031147 ecr 0,nop,wscale 7], length 0
186[00:46:26.219] 00:45:33.248224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531032196 ecr 0,nop,wscale 7], length 0
187[00:46:26.219] 00:45:34.272224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531033220 ecr 0,nop,wscale 7], length 0
188[00:46:26.219] 00:45:35.296253 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531034244 ecr 0,nop,wscale 7], length 0
189[00:46:26.219] 00:45:36.320281 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531035268 ecr 0,nop,wscale 7], length 0
190[00:46:26.219] 00:45:37.442439 IP 172.18.0.2.52570 > 8.8.8.8.53: 3832+ AAAA? some.node. (27)
191[00:46:26.219] 00:45:37.442487 IP 172.18.0.2.60704 > 8.8.8.8.53: 1023+ A? some.node. (27)
192[00:46:26.219] 00:45:37.448551 IP 8.8.8.8.53 > 172.18.0.2.60704: 1023 NXDomain 0/1/0 (102)
193[00:46:26.219] 00:45:37.448911 IP 8.8.8.8.53 > 172.18.0.2.52570: 3832 NXDomain 0/1/0 (102)
194[00:46:26.219] 00:45:37.449395 IP 172.18.0.2.53889 > 8.8.8.8.53: 63359+ AAAA? some.node. (27)
195[00:46:26.219] 00:45:37.449419 IP 172.18.0.2.51133 > 8.8.8.8.53: 6268+ A? some.node. (27)
196[00:46:26.219] 00:45:37.455701 IP 8.8.8.8.53 > 172.18.0.2.51133: 6268 NXDomain 0/1/0 (102)
197[00:46:26.219] 00:45:37.455743 IP 8.8.8.8.53 > 172.18.0.2.53889: 63359 NXDomain 0/1/0 (102)
198[00:46:26.219] 00:45:37.734057 IP 172.18.0.2.50765 > 8.8.8.8.53: 34600+ AAAA? x9.dummySeed.invalid. (38)
199[00:46:26.219] 00:45:37.734277 IP 172.18.0.2.47474 > 8.8.8.8.53: 53797+ A? x9.dummySeed.invalid. (38)
200[00:46:26.219] 00:45:37.740114 IP 8.8.8.8.53 > 172.18.0.2.50765: 34600 NXDomain 0/1/0 (113)
201[00:46:26.219] 00:45:37.740144 IP 8.8.8.8.53 > 172.18.0.2.47474: 53797 NXDomain 0/1/0 (113)
202[00:46:26.219] 00:45:37.740754 IP 172.18.0.2.33593 > 8.8.8.8.53: 4491+ A? x9.dummySeed.invalid. (38)
203[00:46:26.219] 00:45:37.740756 IP 172.18.0.2.34759 > 8.8.8.8.53: 34698+ AAAA? x9.dummySeed.invalid. (38)
204[00:46:26.219] 00:45:37.746602 IP 8.8.8.8.53 > 172.18.0.2.34759: 34698 NXDomain 0/1/0 (113)
205[00:46:26.219] 00:45:37.746641 IP 8.8.8.8.53 > 172.18.0.2.33593: 4491 NXDomain 0/1/0 (113)
206[00:46:26.219] 00:45:38.234283 IP 172.18.0.2.33343 > 8.8.8.8.53: 22856+ A? dummySeed.invalid. (35)
207[00:46:26.219] 00:45:38.234286 IP 172.18.0.2.43752 > 8.8.8.8.53: 55883+ AAAA? dummySeed.invalid. (35)
208[00:46:26.219] 00:45:38.240530 IP 8.8.8.8.53 > 172.18.0.2.43752: 55883 NXDomain 0/1/0 (110)
209[00:46:26.219] 00:45:38.240809 IP 8.8.8.8.53 > 172.18.0.2.33343: 22856 NXDomain 0/1/0 (110)
210[00:46:26.219] 00:45:38.241237 IP 172.18.0.2.35861 > 8.8.8.8.53: 14859+ A? dummySeed.invalid. (35)
211[00:46:26.219] 00:45:38.241238 IP 172.18.0.2.51487 > 8.8.8.8.53: 46093+ AAAA? dummySeed.invalid. (35)
212[00:46:26.219] 00:45:38.247292 IP 8.8.8.8.53 > 172.18.0.2.35861: 14859 NXDomain 0/1/0 (110)
213[00:46:26.219] 00:45:38.247320 IP 8.8.8.8.53 > 172.18.0.2.51487: 46093 NXDomain 0/1/0 (110)
214[00:46:26.219] 00:45:39.779784 IP 172.18.0.2.34493 > 8.8.8.8.53: 26799+ A? fakenodeaddr. (30)
215[00:46:26.219] 00:45:39.779791 IP 172.18.0.2.56878 > 8.8.8.8.53: 61600+ AAAA? fakenodeaddr. (30)
216[00:46:26.219] 00:45:39.785890 IP 8.8.8.8.53 > 172.18.0.2.56878: 61600 NXDomain 0/1/0 (105)
217[00:46:26.219] 00:45:39.785928 IP 8.8.8.8.53 > 172.18.0.2.34493: 26799 NXDomain 0/1/0 (105)
218[00:46:26.219] 00:45:39.786515 IP 172.18.0.2.54375 > 8.8.8.8.53: 52196+ A? fakenodeaddr. (30)
219[00:46:26.219] 00:45:39.786693 IP 172.18.0.2.53292 > 8.8.8.8.53: 56037+ AAAA? fakenodeaddr. (30)
220[00:46:26.219] 00:45:39.792298 IP 8.8.8.8.53 > 172.18.0.2.54375: 52196 NXDomain 0/1/0 (105)
221[00:46:26.219] 00:45:39.792592 IP 8.8.8.8.53 > 172.18.0.2.53292: 56037 NXDomain 0/1/0 (105)
222[00:46:26.219] 00:45:42.174906 IP 11.22.33.44.18444 > 172.18.0.2.51812: Flags [R.], seq 0, ack 3432062660, win 65535, length 0
223[00:46:26.219] 00:45:42.722951 IP 172.18.0.2.33938 > 8.8.8.8.53: 23646+ AAAA? fakeaddress1. (30)
224[00:46:26.219] 00:45:42.723043 IP 172.18.0.2.38954 > 8.8.8.8.53: 24914+ A? fakeaddress1. (30)
225[00:46:26.219] 00:45:42.728907 IP 8.8.8.8.53 > 172.18.0.2.38954: 24914 NXDomain 0/1/0 (105)
226[00:46:26.219] 00:45:42.729111 IP 8.8.8.8.53 > 172.18.0.2.33938: 23646 NXDomain 0/1/0 (105)
227[00:46:26.219] 00:45:42.729684 IP 172.18.0.2.52893 > 8.8.8.8.53: 3801+ AAAA? fakeaddress1. (30)
228[00:46:26.219] 00:45:42.729715 IP 172.18.0.2.57497 > 8.8.8.8.53: 62431+ A? fakeaddress1. (30)
229[00:46:26.219] 00:45:42.735746 IP 8.8.8.8.53 > 172.18.0.2.57497: 62431 NXDomain 0/1/0 (105)
230[00:46:26.219] 00:45:42.735771 IP 8.8.8.8.53 > 172.18.0.2.52893: 3801 NXDomain 0/1/0 (105)
231[00:46:26.219] 00:45:43.175350 IP 11.22.33.44.18444 > 172.18.0.2.51828: Flags [R.], seq 0, ack 3743466588, win 65535, length 0
  22. 1440000bytes commented at 1:09 pm on November 26, 2024: none

    Concept ACK

    A simple solution to avoid leaking IP address when running tests locally would be to disconnect internet while running tests.

  23. jonatack commented at 1:57 pm on November 26, 2024: member
    Concept ACK. Per https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2024-11-26#1069602: “it turns out the owners of 1.2.3.4, 11.22.33.44 and 8.8.8.8, if they would bother, would know the IP address of every dev who runs the functional tests at home.”
  24. vasild force-pushed on Nov 27, 2024
  25. DrahtBot added the label CI failed on Nov 27, 2024
  26. DrahtBot commented at 4:42 pm on November 27, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33609854840

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  27. vasild force-pushed on Nov 27, 2024
  28. vasild commented at 5:36 pm on November 27, 2024: contributor

    8799018bd5...803ed4638b: include #31343 into this PR to demonstrate that #31343 works as intended and also to turn the CI here green.

    About the false positives - I think it is worth trying this in its current mode where any detected traffic is assumed to have originated from the tests and fails the CI. If this ever fails the CI for another reason (false positive), then it would be easy to turn this into a “report in the logs only but don’t fail” by removing the exit 1 line.

    About the DNS traffic - I did cat /etc/resolv.conf in the CI and the name server is indeed configured as a local one: nameserver 127.0.0.11.

  29. BrandonOdiwuor commented at 5:44 pm on November 27, 2024: contributor
    Concept ACK
  30. vasild marked this as a draft on Nov 28, 2024
  31. vasild force-pushed on Nov 28, 2024
  32. vasild commented at 5:31 pm on November 28, 2024: contributor
    Converted to draft for a while, testing docker with full privileges (need cirrus which does not run in my personal fork).
  33. vasild force-pushed on Nov 29, 2024
  34. vasild force-pushed on Nov 29, 2024
  35. DrahtBot removed the label CI failed on Nov 29, 2024
  36. vasild force-pushed on Nov 29, 2024
  37. vasild force-pushed on Dec 2, 2024
  38. vasild force-pushed on Dec 2, 2024
  39. vasild force-pushed on Dec 2, 2024
  40. vasild force-pushed on Dec 3, 2024
  41. vasild commented at 10:48 am on December 3, 2024: contributor
    Ready for review. I updated the OP with some details.
  42. vasild marked this as ready for review on Dec 3, 2024
  43. in ci/test/03_test_script.sh:160 in c88464d754 outdated 
    155+}
156+
157+function traffic_monitor_begin()
158+{
159+  for ifname in $(get_interfaces) ; do
160+    tcpdump -n -i "$ifname" -w "$(tcpdump_file_for_interface "$ifname")" &
    maflcko commented at 10:59 am on December 3, 2024:

    I don’t think this works reliably? See:

    0              This output will be buffered if written to a file or pipe, so a program reading from the file or pipe may not see packets for an arbitrary amount of time after they are received.  Use the -U flag to cause packets to be
1              written as soon as they are received.
    vasild commented at 10:28 am on December 4, 2024:

    Even better! Added -U.

    With -U is does not matter anymore, but I think it was ok before as well, see my comment below.

  44. in ci/test/03_test_script.sh:174 in c88464d754 outdated 
    165+{
166+  # Stop all tcpdump instances (we want the word splitting if "jobs -p" returns more than one PID).
167+  # shellcheck disable=SC2046
168+  while kill -SIGTERM $(jobs -p) ; do
169+    sleep 1
170+  done
    maflcko commented at 11:01 am on December 3, 2024:

    Why is this needed? The file isn’t cleared anyway (and exited if it isn’t clear), so might as well just run the processes and never kill them?

    Also, it doesn’t seem to be working anyway on some of the tasks?

    vasild commented at 10:39 am on December 4, 2024:

    To 1. flush the file and 2. so that the next set of tests can start fresh without leftover tcpdump processes. Now 1. is not needed because of -U, but I will leave this kill for 2.

    In my experiments SIGTERM would cause tcpdump to exit gracefully - flush the file and then exit. SIGKILL would result in unflushed file.

    Yes, in some tasks kill gives “permission denied” even if docker is running with --privileged or --cap-add ALL.

    maflcko commented at 10:48 am on December 4, 2024:

    Yes, in some tasks kill gives “permission denied” even if docker is running with --privileged or --cap-add ALL.

    That is interesting, because all tasks should be running on workers that are set up identically.

    tests can start fresh without leftover tcpdump processes

    My thinking was to just have a single process, which is started once and never killed. This should remove the need to kill it, and also the need (and code) to start it every time.

    vasild commented at 10:57 am on December 4, 2024:

    Then it would be unclear whether the traffic was generated by unit tests, functional tests or fuzz tests. So I start/stop a separate monitoring for each of those set of tests. I perceived this information useful when investigating. It is already tough that it is not known which test generated the traffic.

    Somehow I do not like reading from the “live” file while it is being written. -U would help not to miss data, but it might happen that by the time we read it the last packet is still being written and is half-written and the reading tcpdump is upset by that.

    vasild commented at 10:59 am on December 4, 2024:

    Yes, in some tasks kill gives “permission denied” even if docker is running with --privileged or --cap-add ALL.

    That is interesting, because all tasks should be running on workers that are set up identically.

    I noticed those tasks print:

    Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg

    Btw, now with tcpdump -U those tasks do not print “tcpdump: truncated dump file; tried to read 4 file header bytes, only got 0” anymore

    maflcko commented at 11:06 am on December 4, 2024:

    That is interesting, because all tasks should be running on workers that are set up identically.

    I noticed those tasks print:

    Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg

    That should be identical for all tasks as well. See:

    The only difference should be that one is bare metal and the other is KVM, but I don’t see how this could cause a difference here.

    vasild commented at 11:11 am on December 4, 2024:
    Hmm, right. Then I have no idea why kill does not work on some. It is ok as long as it works on at least one.
    vasild commented at 1:19 pm on December 17, 2024:
    In one of the previous pushes I changed it to use distinct tcpdump files for each test run, so 2. from #31349 (review) is not necessary any more. Removed this snippet.
  45. vasild force-pushed on Dec 4, 2024
  46. in ci/test/03_test_script.sh:185 in feabffd808 outdated 
    180+    fi
181+    # We are running as root and those files are created with owner:group =
182+    # tcpdump:tcpdump and then `tcpdump -r` refuses to read them with an error
183+    # "permission denied" if they are not owned by root:root.
184+    chown root:root "$f"
185+    if [ -n "$(tcpdump -n -r "$f" --direction=out tcp or udp)" ] ; then
    maflcko commented at 10:49 am on December 4, 2024:
    This is also ignoring the return code? So it could also lead to CI silently passing on all tasks, when it should not.
    maflcko commented at 10:53 am on December 4, 2024:
    Generally, I am not sure if bash is the right language to implement something like this. Silently ignoring possible errors doesn’t seem like a great approach. Other languages, such as Rust or Python are a bit more explicit in documenting when a return code is ignored.
    vasild commented at 12:07 pm on December 4, 2024:
    Ignoring the return status of tcpdump here is deliberate because some VMs, the ones that couldn’t stop tcpdump failed to read the unflushed file with an error like “tcpdump: truncated dump file; tried to read 4 file header bytes, only got 0”. Now they don’t due to the added -U, hmm…
    maflcko commented at 12:32 pm on December 4, 2024:

    I’d say this is unrelated to flushing. Ignoring the return code in all tasks can mean that even the tasks where the detection is working are going to silently break tomorrow. As mentioned previously in the thread you closed, I think this is brittle.

    To clarify, my feedback not just applies to the line I commented on, but it applies to all code where the return code is silently ignored and where it could lead to silent breakage.

    vasild commented at 1:46 pm on December 4, 2024:
    Ok, I changed this to fail if tcpdump gives error when it reads the file.
    maflcko commented at 3:43 pm on December 4, 2024:

    Again, my feedback not just applies to the line I commented on, but it applies to all code where the return code is silently ignored and where it could lead to silent breakage.

    For example, pipefail isn’t set, so get_interfaces can also silently fail …

    vasild commented at 4:43 pm on December 4, 2024:

    I assume ifconfig without any arguments will never fail.

    Do you think it would be better to change it to:

    0 function get_interfaces()
1 {
2+  set -o pipefail
3   ifconfig | awk -F ':| ' '/^[^[:space:]]/ { if (!match($1, /^lo/)) { print $1 } }'
4+  set +o pipefail
5 }

    or that is just clutter?

    vasild commented at 6:53 pm on December 4, 2024:
    There is also top -l 1 -s 0 | awk ' /PhysMem/ {print}' near the start of 03_test_script.sh. Maybe set -o pipefail at the start of the file? (would be somewhat out of scope of this PR, but I am fine with it if nobody objects)
    maflcko commented at 7:57 am on December 5, 2024:

    I assume ifconfig without any arguments will never fail.

    I don’t think this is right. Even if the program was a magic unicorn that never could fail for any reason at all, it will fail when it is missing.

    This can happen easily when someone removes “unused” CI packages with the proof that CI passes.

    You can also check it locally:

    0# ifconfig | cat
1sh: 13: ifconfig: not found
2# echo $?
30

    There is also top -l 1 -s 0 | awk ' /PhysMem/ {print}'

    Correct, but this is a macos-only debug-only log, unrelated to checking the behavior of Bitcoin Core or the Bitcoin Core tests. Even if it were to fail, I don’t see how it could lead to a test failure or test misbehaviour being silently ignored.

    vasild commented at 9:55 am on December 5, 2024:
    Enabled pipefail as in the above diff. Thanks!
  47. vasild force-pushed on Dec 4, 2024
  48. vasild commented at 1:46 pm on December 4, 2024: contributor
    feabffd808...925ee8707e: (hopefully) address #31349 (review)
  49. vasild commented at 3:07 pm on December 4, 2024: contributor

    I think the tcpdump approach works well enough. I am posting here just for the records - an alternative approach is to use iptables to detect traffic. I managed to get it to log the traffic, but couldn’t read that log - I could only see a summary of how many packets matched, not individual ones and their destination address.

    Below is a WIP patch which can be used as a starting point if I or somebody else wants to pursue that further.

     0diff --git a/ci/test/00_setup_env.sh b/ci/test/00_setup_env.sh
 1index 94149106ee..7889910684 100755
 2--- a/ci/test/00_setup_env.sh
 3+++ b/ci/test/00_setup_env.sh
 4@@ -61,10 +61,10 @@ export CCACHE_COMPRESS=${CCACHE_COMPRESS:-1}
 5 export CCACHE_DIR="${CCACHE_DIR:-$BASE_SCRATCH_DIR/ccache}"
 6 # Folder where the build result is put (bin and lib).
 7 export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out}
 8 # The folder for previous release binaries.
 9 # This folder exists only on the ci guest, and on the ci host as a volume.
10 export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/prev_releases}
11-export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential pkg-config curl ca-certificates ccache python3 rsync git procps bison e2fsprogs cmake net-tools tcpdump}
12+export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential pkg-config curl ca-certificates ccache python3 rsync git procps bison e2fsprogs cmake net-tools tcpdump bind9-host iptables inetutils-telnet}
13 export GOAL=${GOAL:-install}
14 export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
15 export CI_RETRY_EXE=${CI_RETRY_EXE:-"retry --"}
16diff --git a/ci/test/02_run_container.sh b/ci/test/02_run_container.sh
17index e1539e8bef..0fb5feaec5 100755
18--- a/ci/test/02_run_container.sh
19+++ b/ci/test/02_run_container.sh
20@@ -88,13 +88,24 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
21   # When detecting podman-docker, `--external` should be added.
22   docker image prune --force --filter "label=$CI_IMAGE_LABEL"
23 
24   # Append $USER to /tmp/env to support multi-user systems and $CONTAINER_NAME
25   # to allow support starting multiple runs simultaneously by the same user.
26   # shellcheck disable=SC2086
27-  CI_CONTAINER_ID=$(docker run --cap-add LINUX_IMMUTABLE --cap-add NET_RAW $CI_CONTAINER_CAP --rm --interactive --detach --tty \
28+  CI_CONTAINER_ID=$(docker run \
29+                  --cap-add LINUX_IMMUTABLE \
30+                  --cap-add NET_ADMIN \
31+                  --cap-add SYSLOG \
32+                  $CI_CONTAINER_CAP \
33+                  --cap-add NET_ADMIN \
34+                  --cap-add NET_RAW \
35+                  --privileged \
36+                  --rm \
37+                  --interactive \
38+                  --detach \
39+                  --tty \
40                   --mount "type=bind,src=$BASE_READ_ONLY_DIR,dst=$BASE_READ_ONLY_DIR,readonly" \
41                   --mount "${CI_CCACHE_MOUNT}" \
42                   --mount "${CI_DEPENDS_MOUNT}" \
43                   --mount "${CI_DEPENDS_SOURCES_MOUNT}" \
44                   --mount "${CI_PREVIOUS_RELEASES_MOUNT}" \
45                   --env-file /tmp/env-$USER-$CONTAINER_NAME \
46diff --git a/ci/test/03_test_script.sh b/ci/test/03_test_script.sh
47index 71048330e6..c4b22417a5 100755
48--- a/ci/test/03_test_script.sh
49+++ b/ci/test/03_test_script.sh
50@@ -188,12 +188,31 @@ function traffic_monitor_end()
51       tcpdump -n -r "$f" tcp or udp
52       exit 1
53     fi
54   done
55 }
56 
57+ifconfig || :
58+echo "Non-loopback interfaces: $(get_interfaces)"
59+
60+traffic_monitor_begin
61+iptables  -A OUTPUT -j LOG --log-prefix "ttttt1" --log-level emerg || :
62+ip6tables -A OUTPUT -j LOG --log-prefix "ttttt2" --log-level emerg || :
63+iptables  -A OUTPUT -m addrtype \! --dst-type LOCAL -j LOG --log-prefix "ttttt3" --log-level emerg || :
64+ip6tables -A OUTPUT -m addrtype \! --dst-type LOCAL -j LOG --log-prefix "ttttt4" --log-level emerg || :
65+id || :
66+cat /etc/resolv.conf || :
67+host bitcoin.org || :
68+host nonexistentinvalidfoobarbaz.org || :
69+telnet 50.60.1.2 3456 || :
70+dmesg |grep ttttt || :
71+grep -r ttttt /var/log/ || :
72+iptables -v -x -n -L || :
73+traffic_monitor_end
74+exit 34
75+
76 if [ "$RUN_UNIT_TESTS" = "true" ]; then
77   traffic_monitor_begin
78   DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" CTEST_OUTPUT_ON_FAILURE=ON ctest --stop-on-failure "${MAKEJOBS}" --timeout $(( TEST_RUNNER_TIMEOUT_FACTOR * 60 ))
79   traffic_monitor_end
80 fi
  50. DrahtBot added the label Needs rebase on Dec 5, 2024
  51. vasild force-pushed on Dec 5, 2024
  52. vasild commented at 8:16 am on December 5, 2024: contributor
    925ee8707e...56dbe78934: rebase due to conflicts
  53. vasild force-pushed on Dec 5, 2024
  54. DrahtBot added the label CI failed on Dec 5, 2024
  55. DrahtBot removed the label Needs rebase on Dec 5, 2024
  56. vasild commented at 9:37 am on December 5, 2024: contributor
    56dbe78934...5f3b24711e: use distinct tcpdump file per test (unitparallel, unitsequential, functional, fuzz, tidy) and log that in case of detected traffic.
  57. vasild force-pushed on Dec 5, 2024
  58. vasild commented at 9:54 am on December 5, 2024: contributor
    5f3b24711e...a6c3c9defb: #31349 (review)
  59. DrahtBot removed the label CI failed on Dec 5, 2024
  60. vasild force-pushed on Dec 11, 2024
  61. vasild commented at 5:24 am on December 11, 2024: contributor
    a6c3c9defb...46e38e2e33: rebase and remove merged #31343 which this PR included.
  62. DrahtBot added the label Needs rebase on Dec 13, 2024
  63. vasild force-pushed on Dec 17, 2024
  64. vasild commented at 1:16 pm on December 17, 2024: contributor
    46e38e2e33...95fc90610a: rebase due to conflicts and don’t stop the tcpdump processes because it is now not necessary because distinct files are used for each run, #31349 (review).
  65. DrahtBot removed the label Needs rebase on Dec 17, 2024
  66. DrahtBot added the label CI failed on Dec 17, 2024
  67. DrahtBot removed the label CI failed on Dec 17, 2024
  68. in ci/test/03_test_script.sh:175 in 95fc90610a outdated 
    170+{
171+  test_name="$1"
172+
173+  for ifname in $(get_interfaces) ; do
174+    f=$(tcpdump_file "$test_name" "$ifname")
175+    if [ ! -e "$f" ] && [ "$FILE_ENV" != "./ci/test/00_setup_env_native_asan.sh" ] ; then
    0xB10C commented at 12:24 pm on December 21, 2024:
    nit: someone renaming this file down the line might not be aware of a dependency on this exact file name
    vasild commented at 10:51 am on December 23, 2024:

    Yeah, I was thinking the same… Should I change it to the following?

    0-     if [ ! -e "$f" ] && [ "$FILE_ENV" != "./ci/test/00_setup_env_native_asan.sh" ] ; then
1+     if [ ! -e "$f" ] && [ "$CONTAINER_NAME" != "ci_native_asan" ] ; then
    0xB10C commented at 1:42 pm on December 23, 2024:

    Yet another alternative would be to set a flag like CI_FAIL_IF_NO_TCPDUMP_FILE (or similar) in 00_setup_env_native_asan.sh and add a comment mentioning the dependency:

    • if the script name or the container name are renamed, this will still work
    • if script is removed, someone might see the comment about the dependency during review
    • it’s easy to add more tasks to the list of required tasks by just adding CI_FAIL_IF_NO_TCPDUMP="true"
    vasild commented at 12:08 pm on December 24, 2024:
    Done! Much better.
  69. 0xB10C approved
  70. 0xB10C commented at 12:27 pm on December 21, 2024: contributor

    ACK 95fc90610a1162fc06e61b607488d05229c9909f

    Nice solution to use 127.0.0.1:1 as unreachable proxy for DNS requests too. I ran this on my CI runners with a 8.8.8.8 and 1.1.1.1 as DNS servers as before and the tests don’t connect out now. I ran with test: avoid generating non-loopback traffic from feature_config_args.py reverted to check that a few tasks fail and report the outgoing connections. I did a light code review, looked at e.g. the tcpdump options being used, but didn’t look at the iptables alternative.

  71. DrahtBot requested review from laanwj on Dec 21, 2024
  72. DrahtBot requested review from BrandonOdiwuor on Dec 21, 2024
  73. DrahtBot requested review from maflcko on Dec 21, 2024
  74. DrahtBot requested review from jonatack on Dec 21, 2024
  75. sipa commented at 2:24 pm on December 23, 2024: member
    Concept ACK
  76. vasild force-pushed on Dec 23, 2024
  77. vasild commented at 6:42 pm on December 23, 2024: contributor
    95fc90610a...0ac9caf7be: do #31349 (review)
  78. vasild force-pushed on Dec 23, 2024
  79. vasild commented at 6:48 pm on December 23, 2024: contributor
    0ac9caf7be...d8cd80e814: forgot to add a comment in the previous push
  80. vasild force-pushed on Dec 24, 2024
  81. vasild commented at 11:29 am on December 24, 2024: contributor
    d8cd80e...48843ec: s/=/!=/ :facepalm:
  82. in ci/test/03_test_script.sh:202 in 48843ec694 outdated 
    176+      # In some CI environments this script is not running as root and so the
177+      # tcpdump errors and does not create $f. Skip silently those, but we
178+      # need at least one where tcpdump can run and this is the ASAN one. So
179+      # treat the absence of $f as an error only on the ASAN task.
180+      continue
181+    fi
    maflcko commented at 11:42 am on December 24, 2024:

    Given that you no longer kill tcpdump, the code should work on all CI tasks, except for the GHA macos tasks?

    If there are only a few tasks that do not work, it may be better to just carve them out, instead of enumerating all the ones that work or dealing with code to ensure that at least one works for each config (fuzz/tidy/tests)?

    vasild commented at 12:29 pm on December 24, 2024:

    The table in the OP describes each task and its status wrt to this. There are 18 tasks. It is supposed to work on the top 5+4=9 tasks. The other 9 tasks are not expected to run this - e.g. Windows or tidy that does not run any tests or macOS that runs without docker and has no enough permissions.

    The intention here is to have at least one task that runs this. But it will not hurt to enforce it on more. Do you want to add CI_FAIL_IF_NO_TCPDUMP_FILE=1 to more tasks? Or reverse the logic to CI_OK_IF_NO_TCPDUMP_FILE=1 in the tasks that are expected to not run this (maybe it is the same if it is 9 vs 9 tasks)?

    maflcko commented at 12:39 pm on December 24, 2024:

    The other 9 tasks are not expected to run this - e.g. Windows or tidy that does not run any tests or macOS that runs without docker and has no enough permissions.

    If no tests are run, then no exclusion is needed as well. Though, if this is not expected to run on tidy, I wonder why traffic_monitor_begin "tidy" exists? This leaves macOS, which can be excluded. The other tasks, which do not use the CI infra anyway, so don’t need to be excluded as well.

    vasild commented at 1:41 pm on January 6, 2025:

    You are right! Flipped the logic to exclude only macOS.

    No strong opinion on “tidy” - I added it because those tests should not generate internet traffic. Would drop it if requested.

  83. fjahr commented at 5:05 pm on December 30, 2024: contributor

    Concept ACK

    The changes in the tests look good but I need a little more time with the CI stuff. Maybe the PR could have been split there to get the critical fixes in faster but I guess it’s a bit late now.

  84. vasild force-pushed on Jan 6, 2025
  85. vasild commented at 1:38 pm on January 6, 2025: contributor
    48843ec694...bbfc58a0af: rebase and “If there are only a few tasks that do not work, it may be better to just carve them out, instead of enumerating all the ones that work”, see above.
  86. fanquake commented at 3:02 pm on January 6, 2025: member

    I tried testing the asan job, via time MAKEJOBS="-j17" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" ./ci/test_run_all.sh, on a Fedora dev box, and it fails with the following output:

     0+ chown root:root /tmp/tcpdump_unitparallel_eth0
 1++ tcpdump -n -r /tmp/tcpdump_unitparallel_eth0 --direction=out tcp or udp
 2reading from file /tmp/tcpdump_unitparallel_eth0, link-type EN10MB (Ethernet), snapshot length 262144
 3+ out='14:47:32.023429 IP6 1111:1111::4.33262 > 1111:1111::1.53: 9493+ A? debuginfod.fedoraproject.org. (46)
 414:47:32.023487 IP6 1111:1111::4.33262 > 1111:1111::1.53: 1815+ AAAA? debuginfod.fedoraproject.org. (46)
 514:47:32.050396 IP6 1111:1111::4.51620 > 1111:1111::1.53: 22098+ A? debuginfod.fedoraproject.org. (46)
 614:47:32.050457 IP6 1111:1111::4.51620 > 1111:1111::1.53: 19536+ AAAA? debuginfod.fedoraproject.org. (46)
 714:47:32.059718 IP6 1111:1111::4.48563 > 1111:1111::1.53: 672+ A? debuginfod.fedoraproject.org. (46)
 814:47:32.060018 IP6 1111:1111::4.48563 > 1111:1111::1.53: 1965+ AAAA? debuginfod.fedoraproject.org. (46)
 914:47:32.069417 IP6 1111:1111::1.53 > 1111:1111::4.51620: 22098 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1014:47:32.069442 IP6 1111:1111::1.53 > 1111:1111::4.33262: 9493 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1114:47:32.069467 IP6 1111:1111::1.53 > 1111:1111::4.48563: 672 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1214:47:32.163955 IP6 1111:1111::4.41445 > 1111:1111::1.53: 34154+ A? debuginfod.fedoraproject.org. (46)
1314:47:32.164474 IP6 1111:1111::4.41445 > 1111:1111::1.53: 49518+ AAAA? debuginfod.fedoraproject.org. (46)
1414:47:32.164826 IP6 1111:1111::1.53 > 1111:1111::4.41445: 34154 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1514:47:32.184123 IP6 1111:1111::4.51525 > 1111:1111::1.53: 40013+ A? debuginfod.fedoraproject.org. (46)
1614:47:32.184205 IP6 1111:1111::4.51525 > 1111:1111::1.53: 32841+ AAAA? debuginfod.fedoraproject.org. (46)
1714:47:32.186645 IP6 1111:1111::1.53 > 1111:1111::4.51525: 40013 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
  87. vasild commented at 11:12 am on January 7, 2025: contributor

    The current CI failure is unrelated to this PR and is fixed separately in #31614. It will probably pass if the CI task is restarted because it is a race.

    14:47:32.023429 IP6 1111:1111::4.33262 > 1111:1111::1.53: 9493+ A? debuginfod.fedoraproject.org. (46) @fanquake so some piece of the command:

    0DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" CTEST_OUTPUT_ON_FAILURE=ON ctest --stop-on-failure "${MAKEJOBS}" --timeout $(( TEST_RUNNER_TIMEOUT_FACTOR * 60 ))

    Generated DNS requests for debuginfod.fedoraproject.org :eyes: :open_mouth:. It is probably not test_bitcoin because grepping for debuginfo or fedoraproject in the entire code base of Bitcoin Core yields no results. Is this running directly on the host or inside docker? Looks like some spyware that calls home when ctest or test_bitcoin is run.

    Edit: if it is running on the host without docker then it could be that processes other than ctest and test_bitcoin are run during the tests and they generate the traffic. Related: https://fedoraproject.org/wiki/Debuginfod.

  88. luke-jr commented at 8:38 pm on January 8, 2025: member

    i’m not aware of a straightforward way to “log network traffic of this process and subproceses only”.

    Not quite that, but iptables has the ability to match and log uid

  89. 0xB10C commented at 11:46 pm on January 8, 2025: contributor

    I tried testing the asan job, via time MAKEJOBS="-j17" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" ./ci/test_run_all.sh, on a Fedora dev box, and it fails with the following output.

    I guess checking that we’re in a docker container (that hopefully doesn’t have other services that does anything network related) is the only option here. Only fail in docker, otherwise just print, if possible.

    Possibly by checking that /proc/1/cgroup exists and this is not empty with cat /proc/1/cgroup | grep docker - though I haven’t tested this.

  90. maflcko commented at 12:22 pm on January 9, 2025: member

    I tried testing the asan job, via time MAKEJOBS="-j17" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" ./ci/test_run_all.sh, on a Fedora dev box, and it fails with the following output.

    I guess checking that we’re in a docker container (that hopefully doesn’t have other services that does anything network related) is the only option here. Only fail in docker, otherwise just print, if possible.

    None of the CI tasks use fedora, so the only way to run the native_asan task on Fedora is to run it in a container, which still failed here. So I don’t think your suggestion will help, but I haven’t tried it.

  91. fanquake commented at 12:39 pm on January 9, 2025: member
    Note that this isn’t Fedora specific. The exact same issue happens on Ubuntu.
  92. maflcko commented at 12:42 pm on January 9, 2025: member
    Maybe the test changes can be split up from the ci changes, given that this is still WIP?
  93. vasild commented at 9:26 am on January 13, 2025: contributor
    Extracted the changes to the tests in #31646 (suggested by @fjahr and @maflcko).
  94. vasild commented at 9:35 am on January 13, 2025: contributor
    @fanquake, is the problem you reported above #31349 (comment) when the tests run on the host, without a docker?
  95. Sjors commented at 11:01 am on January 13, 2025: member

    Concept ACK

    I tested this with https://github.com/Sjors/bitcoin/pull/76 on my own CI setup, which uses Cirrus workers configured using (roughly) the instructions here: https://github.com/bitcoin/bitcoin/blob/master/.cirrus.yml#L8 (using podman-docker)

    (I did run into an issue, left a comment there since it might be specific to my setup)

  96. maflcko commented at 11:16 am on January 13, 2025: member
    I guess an env var is leaking into the docker. To reproduce you could try setting the URL via the env var DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/ , but I haven’t tried it.
  97. maflcko commented at 11:58 am on January 13, 2025: member

    Checked locally that the following fails for me as well: time env -i HOME="$HOME" PATH="$PATH" USER="$USER" MAKEJOBS="-j$( nproc )" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" DEBUGINFOD_URLS='https://debuginfod.fedoraproject.org/' ./ci/test_run_all.sh

    In theory it is not recommended to run the tests without a clean env, according to the ci/README.md. So I am not sure if this can be left as-is, or a workaround for DEBUGINFOD_URLS is convenient .

  98. glozow referenced this in commit 7cd862aab9 on Jan 14, 2025
  99. DrahtBot added the label Needs rebase on Jan 14, 2025
  100. vasild force-pushed on Jan 15, 2025
  101. vasild commented at 8:30 am on January 15, 2025: contributor

    bbfc58a0af...69e076664d: drop the tests changes from this PR because they were already merged via #31646. So here remains a CI change to detect future regressions.

    If people want to run this manually, outside of the CI in a way that causes false positive (discussion above), what about something like this:

     0--- i/ci/test/03_test_script.sh
 1+++ w/ci/test/03_test_script.sh
 2@@ -184,13 +184,15 @@ function traffic_monitor_end()
 3     # "permission denied" if they are not owned by root:root.
 4     chown root:root "$f"
 5     out="$(tcpdump -n -r "$f" --direction=out tcp or udp)"
 6     if [ -n "$out" ] ; then
 7       echo "Error: outbound TCP or UDP packets on the non loopback interface generated during $test_name tests:" >&2
 8       tcpdump -n -r "$f" tcp or udp
 9-      exit 1
10+      if [ -z "$INTERNET_TRAFFIC_EXPECTED" ] ; then
11+        exit 1
12+      fi
13     fi
14   done
15 }
16 
17 if [ "$RUN_UNIT_TESTS" = "true" ]; then
18   traffic_monitor_begin "unitparallel"

    and then, obviously, set INTERNET_TRAFFIC_EXPECTED=1 when running this manually (in unclean environment)? Or leave it as it is?

  102. DrahtBot removed the label Needs rebase on Jan 15, 2025
  103. DrahtBot added the label Needs rebase on Jan 17, 2025
  104. vasild force-pushed on Feb 21, 2025
  105. vasild commented at 7:55 am on February 21, 2025: contributor
    69e076664d...76feae207d: rebase due to conflicts and allow to log but not treat as an error detected internet traffic ($INTERNET_TRAFFIC_EXPECTED mentioned above). Can be used when running manually outside of CI where other programs on the host can generate internet traffic.
  106. DrahtBot removed the label Needs rebase on Feb 21, 2025
  107. DrahtBot added the label Needs rebase on Apr 2, 2025
  108. vasild force-pushed on Apr 16, 2025
  109. vasild commented at 2:33 pm on April 16, 2025: contributor
    76feae207d...4652f75bbf: rebase due to conflicts
  110. DrahtBot removed the label Needs rebase on Apr 16, 2025
  111. DrahtBot added the label Needs rebase on Jul 14, 2025
  112. fanquake referenced this in commit fd813bf863 on Aug 5, 2025
  113. vasild force-pushed on Sep 4, 2025
  114. vasild commented at 8:24 am on September 4, 2025: contributor
    4652f75bbf...f400e0bb82: rebase due to conflicts
  115. DrahtBot removed the label Needs rebase on Sep 4, 2025
  116. vasild force-pushed on Sep 4, 2025
  117. DrahtBot added the label Needs rebase on Sep 4, 2025
  118. vasild force-pushed on Sep 4, 2025
  119. vasild commented at 1:12 pm on September 4, 2025: contributor

    f400e0bb82...9dcbf6ccde: fix a new case of non-loopback traffic from node_init_tests/init_test.

    This PR originally contained a few fixes of tests that generated network traffic + a CI change to catch such future cases in CI. Then the tests fixes were moved to #31646 and merged. Then the activity here waned.

    Now there is a new case of network traffic generated by a test which went in master unnoticed. I have fixed that and included it here.

    9dcbf6ccde...a9ac49c8ac: rebase due to conflicts

  120. in src/test/node_init_tests.cpp:37 in 005b8ca02c outdated 
    32@@ -32,6 +33,9 @@ class TestInit : public interfaces::Init
33 
34 BOOST_AUTO_TEST_CASE(init_test)
35 {
36+    const auto create_sock_orig = CreateSock;
37+    CreateSock = [](int, int, int) { return std::make_unique<ZeroSock>(); };
    ryanofsky commented at 1:39 pm on September 4, 2025:

    In commit “ci: detect outbound internet traffic generated while running tests” (f400e0bb82920adf50fec1b9d701e8e85e62030a)

    Thanks for the fix! I think instead of manipulating CreateSock for this specific test it would be probably better to add gArgs.ForceSetArg("-natpmp", "0"); to BasicTestingSetup::BasicTestingSetup. Reasons:

    • It would work generally for all tests, not just this one test
    • This would make unit tests more consistent with functional tests
    • CreateSock seems like a band-aid more than a cure. I think the problem here is the test trying to use the network, not just being able to use it after it tries.

    If you prefer current approach, though, it seems ok.

    vasild commented at 3:29 pm on September 4, 2025:

    Yeah, I thought about setting -natpmp to off, but decided would be better to test with the default arguments as that more closely matches the real world. That argument seems weak.

    band-aid more than a cure

    I agree. Changed to what you suggest above.

  121. ryanofsky approved
  122. ryanofsky commented at 1:53 pm on September 4, 2025: contributor

    Code review ACK a9ac49c8accffeffaae72f54021fd939c951844c

    Thanks for the fix! Note that this fix may (I’m not sure) mask an integer overflow issue #32345 (review) that seems like it is a real bug.

    The new tcpdump mechanism to detect unexpected traffic is pretty simple and seems very nice. Another idea could be to have unit test setup use CreateSock to throw errors if code attempts a remote connection to make errors a little easier to catch locally. But tcpdump solution is more general so seems like a good approach regardless.

  123. DrahtBot requested review from maflcko on Sep 4, 2025
  124. DrahtBot requested review from sipa on Sep 4, 2025
  125. DrahtBot requested review from Sjors on Sep 4, 2025
  126. DrahtBot requested review from fjahr on Sep 4, 2025
  127. DrahtBot requested review from 0xB10C on Sep 4, 2025
  128. DrahtBot removed the label Needs rebase on Sep 4, 2025
  129. vasild force-pushed on Sep 4, 2025
  130. vasild commented at 3:30 pm on September 4, 2025: contributor
    a9ac49c8ac...778675ac71: pick #31349 (review)
  131. DrahtBot added the label Needs rebase on Sep 5, 2025
  132. vasild force-pushed on Sep 8, 2025
  133. vasild commented at 10:16 am on September 8, 2025: contributor
    778675ac71...151edfaf78: rebase due to conflicts
  134. DrahtBot removed the label Needs rebase on Sep 8, 2025
  135. ryanofsky approved
  136. ryanofsky commented at 1:40 pm on September 9, 2025: contributor
    Code review ACK 151edfaf78115402c29088dabc271c2b268102a5. Just rebased since last review and replaced node_init_tests fix CreateSock fix with natpmp=0 fix
  137. in ci/test/03_test_script.sh:211 in 151edfaf78 
    206+    chown root:root "$f"
207+    out="$(tcpdump -n -r "$f" --direction=out tcp or udp)"
208+    if [ -n "$out" ] ; then
209+      echo "Error: outbound TCP or UDP packets on the non loopback interface generated during $test_name tests:" >&2
210+      tcpdump -n -r "$f" tcp or udp
211+      if [ -z "$INTERNET_TRAFFIC_EXPECTED" ] ; then
    sipa commented at 2:12 pm on September 9, 2025:
    This INTERNET_TRAFFIC_EXPECTED seems to be unused?
    vasild commented at 9:04 am on September 11, 2025:
    INTERNET_TRAFFIC_EXPECTED is a means to turn this into a non-fatal error. For folks that want to run the CI shell scripts outside of CI, in environments where other programs on the same machine create internet traffic. It came from here: #31349 (comment)
    maflcko commented at 11:32 am on September 26, 2025:

    Due to a silent merge conflict, this won’t actually work. To pass the setting into the container, the hidden setting has to be “documented” now. For context, see https://github.com/bitcoin/bitcoin/blob/65e909dfdd934f033727e5404b5616a29dc18209/ci/test/02_run_container.py#L29-L34

    However, the silent conflict should have also fixed the bug that made this setting necessary in the first place.

    It was added for #31349 (comment). However, I expect the command to now pass. (Haven’t tried)

    vasild commented at 12:45 pm on September 26, 2025:
    Removed INTERNET_TRAFFIC_EXPECTED. Maybe it was an overkill in the first place.
  138. DrahtBot requested review from sipa on Sep 9, 2025
  139. vasild commented at 9:35 am on September 11, 2025: contributor

    Another idea could be to have unit test setup use CreateSock to throw errors …

    I think that is worth exploring. Opened #33363 to track it, so that it does not get forgotten.

  140. vasild force-pushed on Sep 26, 2025
  141. vasild commented at 12:45 pm on September 26, 2025: contributor
    151edfaf78...f4fdf81d3a: rebase and remove INTERNET_TRAFFIC_EXPECTED.
  142. vasild commented at 10:11 am on September 29, 2025: contributor
    Updated the PR description. It referred to a older version of the PR.
  143. ryanofsky approved
  144. ryanofsky commented at 7:16 pm on September 29, 2025: contributor
    Code review ACK f4fdf81d3a1811561ed35d6ce2424978be284d51, just dropping INTERNET_TRAFFIC_EXPECTED variable since last review
  145. in src/test/util/setup_common.cpp:193 in bd058f1873 outdated 
    189@@ -190,6 +190,8 @@ BasicTestingSetup::BasicTestingSetup(const ChainType chainType, TestOpts opts)
190     m_args.ForceSetArg("-datadir", fs::PathToString(m_path_root));
191     gArgs.ForceSetArg("-datadir", fs::PathToString(m_path_root));
192 
193+    gArgs.ForceSetArg("-natpmp", "0");
    fjahr commented at 7:25 pm on September 29, 2025:
    nit: Would be nice to add a comment here so people don’t have to hunt down the commit description.
    vasild commented at 12:37 pm on September 30, 2025:
    Added // Avoid non-loopback network traffic during tests.
  146. in ci/test/03_test_script.sh:198 in f4fdf81d3a outdated 
    185+  for ifname in $(get_interfaces) ; do
186+    tcpdump -nU -i "$ifname" -w "$(tcpdump_file "$test_name" "$ifname")" &
187+  done
188+}
189+
190+function traffic_monitor_end()
    fjahr commented at 9:58 pm on September 29, 2025:
    nit: here I would also like a bit of documention, maybe adding a small section in ci/readme.md would be the right place for this?
    vasild commented at 12:39 pm on September 30, 2025:
    I added brief comments to the newly added functions. ci/README.md seems to me too high level for this. No strong opinion though.
  147. fjahr commented at 9:58 pm on September 29, 2025: contributor
    Hm, I was hoping I could easily test this by commenting out the change in the first commit and then running this in the CI, which I did by pushing to this branch: https://github.com/fjahr/bitcoin/commits/pr31349/ While not everything has finished, some job including unit tests succeeded and the only failing job ran out of space which seems unrelated. Am I missing something here?
  148. DrahtBot requested review from fjahr on Sep 29, 2025
  149. vasild force-pushed on Sep 30, 2025
  150. vasild commented at 12:45 pm on September 30, 2025: contributor

    f4fdf81d3a...39f90a4a78: add some comments, suggested by @fjahr above. Also, restore the exit 1 which I accidentally removed in a previous push.

    I was hoping I could easily test …

    The traffic is detected and reported in your CI jobs (search for Error: outbound), but there was no exit 1 🤦. Sorry for that :face_in_clouds: :face_with_head_bandage:

  151. in src/test/util/setup_common.cpp:193 in d8372a220f outdated 
    189@@ -190,6 +190,8 @@ BasicTestingSetup::BasicTestingSetup(const ChainType chainType, TestOpts opts)
190     m_args.ForceSetArg("-datadir", fs::PathToString(m_path_root));
191     gArgs.ForceSetArg("-datadir", fs::PathToString(m_path_root));
192 
193+    gArgs.ForceSetArg("-natpmp", "0"); // Avoid non-loopback network traffic during tests.
    ryanofsky commented at 1:07 pm on September 30, 2025:

    In commit “test: avoid non-loopback network traffic from node_init_tests/init_test” (d8372a220fb9691347d88547e381b8579ad35edb)

    Could be nice to add the same comment to functional test setup as well

    https://github.com/bitcoin/bitcoin/blob/25212dfdb4cd7291392b6a94130f658c5bfa0a48/test/functional/test_framework/util.py#L460

    vasild commented at 2:15 pm on September 30, 2025:
    Done.
  152. in ci/test/03_test_script.sh:182 in 39f90a4a78 
    177+
178+# Generate a file name for storing raw packets captured by tcpdump.
179+function tcpdump_file()
180+{
181+  test_name="$1"
182+  interface_name="$2"
    ryanofsky commented at 1:10 pm on September 30, 2025:

    In commit “ci: detect outbound internet traffic generated while running tests” (39f90a4a78020087d19491be7b315ad91f252e46)

    Not important but these are global variables. Might be a little better to set as local test_name="$1" etc

    vasild commented at 2:15 pm on September 30, 2025:
    Added local, thanks!
  153. ryanofsky approved
  154. ryanofsky commented at 1:12 pm on September 30, 2025: contributor
    Code review ACK 39f90a4a78020087d19491be7b315ad91f252e46 just documenting things better since last review and adding missing exit 1 to trigger CI failure (nice find!)
  155. fjahr commented at 2:13 pm on September 30, 2025: contributor

    tACK 39f90a4a78020087d19491be7b315ad91f252e46

    Thanks for addressing my comments! Tested again by pushing the latest version of the code with the change of the first commit commented out. I could observe the expected failure in the CI this time: https://github.com/fjahr/bitcoin/actions/runs/18131366221/job/51598667988

    I would be happy to re-review if you decide to address @ryanofsky ’s latest comments.

  156. vasild force-pushed on Sep 30, 2025
  157. vasild commented at 2:15 pm on September 30, 2025: contributor
    39f90a4a78...c652deb3c1: address suggestions
  158. fjahr commented at 9:18 pm on September 30, 2025: contributor

    re-ACK c652deb3c16b7edccb741b9b473502092c0c2638

    Just addressed @ryanofsky ’s comments.

  159. DrahtBot requested review from ryanofsky on Sep 30, 2025
  160. DrahtBot added the label Needs rebase on Oct 15, 2025
  161. maflcko commented at 6:07 pm on October 15, 2025: member

    Not sure why I get this failure, but when installing a fresh Fedora VM, then podman-docker inside that, and then running this pull rebased, I got:

     0MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh" ./ci/test_run_all.sh
 1
 2...
 3
 4100% tests passed, 0 tests failed out of 135
 5
 6Total Test time (real) =  42.30 sec
 7+ traffic_monitor_end unit
 8+ test_name=unit
 9++ get_interfaces
10++ set -o pipefail
11++ ifconfig
12++ awk -F ':| ' '/^[^[:space:]]/ { if (!match($1, /^lo/)) { print $1 } }'
13++ set +o pipefail
14+ for ifname in $(get_interfaces)
15++ tcpdump_file unit eth0
16++ local test_name=unit
17++ local interface_name=eth0
18++ echo /tmp/tcpdump_unit_eth0
19+ f=/tmp/tcpdump_unit_eth0
20+ '[' '!' -e /tmp/tcpdump_unit_eth0 ']'
21+ chown root:root /tmp/tcpdump_unit_eth0
22++ tcpdump -n -r /tmp/tcpdump_unit_eth0 --direction=out tcp or udp
23reading from file /tmp/tcpdump_unit_eth0, link-type EN10MB (Ethernet), snapshot length 262144
24+ out='17:59:15.979577 IP6 1111:1111::3.59201 > 1111:1111::1.53: 20844+ A? x9.dummySeed.invalid. (38)
2517:59:15.979644 IP6 1111:1111::3.59201 > 1111:1111::1.53: 49770+ AAAA? x9.dummySeed.invalid. (38)
2617:59:15.980117 IP6 1111:1111::1.53 > 1111:1111::3.59201: 20844 NXDomain* 0/0/0 (38)
2717:59:15.980200 IP6 1111:1111::1.53 > 1111:1111::3.59201: 49770 NXDomain* 0/0/0 (38)
2817:59:15.982301 IP6 1111:1111::3.51971 > 1111:1111::1.53: 35137+ A? x9.dummySeed.invalid. (38)
2917:59:15.982320 IP6 1111:1111::3.51971 > 1111:1111::1.53: 18242+ AAAA? x9.dummySeed.invalid. (38)
3017:59:15.983461 IP6 1111:1111::1.53 > 1111:1111::3.51971: 35137 NXDomain* 0/0/0 (38)
3117:59:15.983567 IP6 1111:1111::1.53 > 1111:1111::3.51971: 18242 NXDomain* 0/0/0 (38)'
32+ '[' -n '17:59:15.979577 IP6 1111:1111::3.59201 > 1111:1111::1.53: 20844+ A? x9.dummySeed.invalid. (38)
3317:59:15.979644 IP6 1111:1111::3.59201 > 1111:1111::1.53: 49770+ AAAA? x9.dummySeed.invalid. (38)
3417:59:15.980117 IP6 1111:1111::1.53 > 1111:1111::3.59201: 20844 NXDomain* 0/0/0 (38)
3517:59:15.980200 IP6 1111:1111::1.53 > 1111:1111::3.59201: 49770 NXDomain* 0/0/0 (38)
3617:59:15.982301 IP6 1111:1111::3.51971 > 1111:1111::1.53: 35137+ A? x9.dummySeed.invalid. (38)
3717:59:15.982320 IP6 1111:1111::3.51971 > 1111:1111::1.53: 18242+ AAAA? x9.dummySeed.invalid. (38)
3817:59:15.983461 IP6 1111:1111::1.53 > 1111:1111::3.51971: 35137 NXDomain* 0/0/0 (38)
3917:59:15.983567 IP6 1111:1111::1.53 > 1111:1111::3.51971: 18242 NXDomain* 0/0/0 (38)' ']'
40+ echo 'Error: outbound TCP or UDP packets on the non loopback interface generated during unit tests:'
41Error: outbound TCP or UDP packets on the non loopback interface generated during unit tests:
42+ tcpdump -n -r /tmp/tcpdump_unit_eth0 tcp or udp
43reading from file /tmp/tcpdump_unit_eth0, link-type EN10MB (Ethernet), snapshot length 262144
4417:59:15.979577 IP6 1111:1111::3.59201 > 1111:1111::1.53: 20844+ A? x9.dummySeed.invalid. (38)
4517:59:15.979644 IP6 1111:1111::3.59201 > 1111:1111::1.53: 49770+ AAAA? x9.dummySeed.invalid. (38)
4617:59:15.980117 IP6 1111:1111::1.53 > 1111:1111::3.59201: 20844 NXDomain* 0/0/0 (38)
4717:59:15.980200 IP6 1111:1111::1.53 > 1111:1111::3.59201: 49770 NXDomain* 0/0/0 (38)
4817:59:15.982301 IP6 1111:1111::3.51971 > 1111:1111::1.53: 35137+ A? x9.dummySeed.invalid. (38)
4917:59:15.982320 IP6 1111:1111::3.51971 > 1111:1111::1.53: 18242+ AAAA? x9.dummySeed.invalid. (38)
5017:59:15.983461 IP6 1111:1111::1.53 > 1111:1111::3.51971: 35137 NXDomain* 0/0/0 (38)
5117:59:15.983567 IP6 1111:1111::1.53 > 1111:1111::3.51971: 18242 NXDomain* 0/0/0 (38)
52+ exit 1

    For reference, that IP is from ci/test/02_run_container.sh: docker network create --ipv6 --subnet 1111:1111::/112 ci-ip6net || true

    Though, FILE_ENV="./ci/test/00_setup_env_native_previous_releases.sh" passes fine :shrug:

  162. fanquake commented at 8:36 am on October 16, 2025: member
    Yea, I have seen the same, on one of my Fedora boxes, which would be a blocker, given it breaks running the CI locally.
  163. test: avoid non-loopback network traffic from node_init_tests/init_test 
    The test calls:
`AppInitMain()` -> `StartMapPort()` -> `StartThreadMapPort()` ->
`ThreadMapPort()` -> `ProcessPCP()` -> `PCPRequestPortMap()` ->
`CreateSock()` and on the returned value from `CreateSock()` it calls
the `Connect()` method.

Thus, change `BasicTestingSetup::BasicTestingSetup()` to set `-natpmp`
to 0. This way `node_init_tests/init_test` or other tests will not do
network activity due to `ThreadMapPort()`.

Also add a comment about `natpmp=0` in
`test/functional/test_framework/util.py`.
    9d666afa20
  164. ci: detect outbound internet traffic generated while running tests 
    Resolves https://github.com/bitcoin/bitcoin/issues/31339
    6e0f3a4a58
  165. vasild force-pushed on Oct 16, 2025
  166. vasild commented at 9:01 am on October 16, 2025: contributor

    c652deb3c1...6e0f3a4a58: rebase due to conflicts

    Not sure why I get this failure …

    It is making requests to the DNS server at 1111:1111::1.53, trying to resolve x9.dummySeed.invalid.

    https://github.com/bitcoin/bitcoin/blob/e14451ac87339ed61b8c872f027184a978dd96eb/src/kernel/chainparams.cpp#L596-L598

  167. DrahtBot removed the label Needs rebase on Oct 16, 2025
  168. maflcko commented at 11:07 am on October 16, 2025: member

    Not sure why I get this failure …

    It is making requests to the DNS server at 1111:1111::1.53, trying to resolve x9.dummySeed.invalid.

    Yes, I understand this, but I don’t understand why the CI is green on this pull request here, but it fails locally when using podman. I guess it could be due to running as root inside the VM. Though, the failure seemingly not being reproducible on every run makes it even more odd.

    My cmd history today (once it passed, once it failed):

    0   1  git clone https://github.com/vasild/bitcoin/ --depth=1 --branch=test_log_internet_traffic ./test_oct_16
1   2  cd test_oct_16/
2   3  env -i RUN_FUNCTIONAL_TESTS=false HOME="$HOME" PATH="$PATH" USER="$USER" MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh" ./ci/test_run_all.sh 
3
  169. vasild commented at 2:01 pm on October 16, 2025: contributor
    I can’t reproduce locally. An attempt to resolve x9.dummySeed.invalid. during unit tests looks like one of those issues that this PR aims to uncover.
  170. maflcko commented at 9:28 am on October 21, 2025: member

    I can’t reproduce locally.

    Are you sure? Above I tried on Fedora podman-docker, and today it also failed on Ubuntu podman-docker:

    0(root)# docker --version 
1Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
2podman version 4.9.3
  171. fanquake commented at 12:11 pm on October 21, 2025: member
    Yea. Still failing for me with Podman (5.6.2) on my Fedora box.
  172. maflcko commented at 1:15 pm on October 21, 2025: member

    From the output, I think there may be several unit tests affected, and they should probably be fixed, even if the GHA CI does not catch this issue.

    Also, I tried on a fresh Ubuntu VM with a fresh user account (not root) and the issue persists. So I think the issue generally uncovers via podman.

  173. DrahtBot added the label Needs rebase on Oct 29, 2025
  174. DrahtBot commented at 10:52 am on October 29, 2025: contributor
    🐙 This pull request conflicts with the target branch and needs rebase.


vasild DrahtBot maflcko laanwj 0xB10C 1440000bytes jonatack BrandonOdiwuor sipa fjahr fanquake luke-jr Sjors ryanofsky


Sjors laanwj sipa fjahr jonatack maflcko ryanofsky BrandonOdiwuor 0xB10C

Labels
Tests Needs rebase

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-11-12 21:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me