ci: detect outbound internet traffic generated while running tests #31349

pull vasild wants to merge 1 commits into bitcoin:master from vasild:test_log_internet_traffic changing 6 files +63 −3
  1. vasild commented at 1:58 pm on November 22, 2024: contributor

    Prevent generating outbound traffic on a non-loopback interface during tests.

    • Fix all tests, including ones that generate DNS traffic
    • Change CI to catch new regressions, including DNS traffic. DNS traffic is detected because some VMs have configured a non-loopback DNS server, [1111:1111::1]:53:
      • previous releases, depends DEBUG
      • TSan, depends, gui
      • multiprocess, i686, DEBUG
      • no wallet, libbitcoinkernel
    • Required capabilities within the VM are explicit in 02_run_container.sh: --cap-add NET_RAW
    • False positives either from non-test generated outbound traffic or responses to outside-originated traffic will fail the CI. I think there is a good chance that this does not happen. If it happens then this can be revisited then and can be easily worked around by removing the exit 1 line added to 03_test_script.sh in this PR.

    Summary of each VM wrt the new check:

    VM unit (parallel) unit (sequential) functional fuzz tidy
    32-bit CentOS, dash, gui (Cirrus CI) :heavy_check_mark: - :heavy_check_mark: - -
    ARM, unit tests, no functional tests (Cirrus CI) :heavy_check_mark: - - - -
    ASan + LSan + UBSan + integer, no depends, USDT (GitHub) :heavy_check_mark: - :heavy_check_mark: - -
    MSan, depends (Cirrus CI) :heavy_check_mark: - - - -
    TSan, depends, gui (Cirrus CI) :heavy_check_mark: - :heavy_check_mark: - -
    Win64 native fuzz, VS 2022 (GitHub) - - - 2 -
    Win64 native, VS 2022 (GitHub) 2 - 2 - -
    Win64-cross (Cirrus CI) - - - - -
    fuzzer,address,undefined,integer, no depends (Cirrus CI) - - - :heavy_check_mark: -
    lint (Cirrus CI) - - - - -
    macOS 14 native, arm64, fuzz (GitHub) - - 1 1 -
    macOS 14 native, arm64, no depends, sqlite only, gui (GitHub) 1 - 1 - -
    macOS-cross, gui, no tests (Cirrus CI) - - - - -
    multiprocess, i686, DEBUG (Cirrus CI) :heavy_check_mark: - :heavy_check_mark: - -
    no wallet, libbitcoinkernel (Cirrus CI) :heavy_check_mark: - :heavy_check_mark: - -
    previous releases, depends DEBUG (Cirrus CI) - :heavy_check_mark: :heavy_check_mark: - -
    test each commit (GitHub) 2 - 2 - -
    tidy (Cirrus CI) - - - - :heavy_check_mark:

    :heavy_check_mark:: test is run under tcpdump (it works!) -: test is not run in that VM 1: tcpdump: en0: You don't have permission to capture on that device (does not work) 2: tests are run but directly from .github/workflows/ci.yml instead of from ci/test/03_test_script.sh (does not work)

    Resolves #31339

  2. DrahtBot commented at 1:58 pm on November 22, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31349.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK maflcko, laanwj, 1440000bytes, jonatack, BrandonOdiwuor, sipa, fjahr, Sjors
    Stale ACK 0xB10C

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31669 (cmake: Introduce WITH_PYTHON build option by hebasto)
    • #31651 (ci: Turn CentOS task into native one by maflcko)
    • #31621 (doc: Update dependency installation for Debian/Ubuntu by adlai)
    • #31593 (ci: Bump centos stream 10 by maflcko)
    • #30997 (build: Switch to Qt 6 by hebasto)
    • #30975 (Add multiprocess binaries to release build (except Windows) by Sjors)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Tests on Nov 22, 2024
  4. vasild force-pushed on Nov 22, 2024
  5. DrahtBot commented at 2:12 pm on November 22, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33384128561

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  6. DrahtBot added the label CI failed on Nov 22, 2024
  7. maflcko commented at 2:17 pm on November 22, 2024: member
    Nice. Conecpt ACK!
  8. laanwj commented at 8:33 am on November 25, 2024: member

    Concept ACK

    i’m slightly worried this may generate false positive. As is, this detects traffic on the entire (virtual) machine while running the tests. Are there no other daemons running on the CI instance that could interfere with this?

  9. vasild force-pushed on Nov 25, 2024
  10. vasild commented at 9:08 am on November 25, 2024: contributor

    @laanwj, Right! And ps ax in the VM looks suspiciously scarce: #31339 (comment) showing just bash and 03_test_script.sh.

    Another source of false positive could be if somebody from the outside initiates communication to the VM to which it responds. E.g. an outsider tries to connect to the VM to which it responds with an outbound packet e.g. TCP RST. At least that should be obvious from the error log, showing the incoming packet first (I just pushed a slight change for that). Maybe also the traffic-from-another-daemon could be obvious - e.g. if there is traffic to apt.update.ubuntu.com:443…

  11. vasild force-pushed on Nov 25, 2024
  12. DrahtBot removed the label CI failed on Nov 25, 2024
  13. vasild force-pushed on Nov 25, 2024
  14. DrahtBot added the label CI failed on Nov 25, 2024
  15. DrahtBot commented at 12:51 pm on November 25, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33474555794

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  16. DrahtBot removed the label CI failed on Nov 25, 2024
  17. laanwj commented at 3:07 pm on November 25, 2024: member

    Another source of false positive could be if somebody from the outside initiates communication to the VM to which it responds. E.g. an outsider tries to connect to the VM to which it responds with an outbound packet e.g. TCP RST.

    Exactly. For all we know, the CI VM is firewalled off sufficiently that this can’t happen, but we don’t know.

    At least that should be obvious from the error log, showing the incoming packet first (I just pushed a slight change for that).

    Ah yes, as long as it’s only some extra logging, having a manual factor in this is fine. It only becomes critical if network traffic would cause a CI failure.

    i’m not aware of a straightforward way to “log network traffic of this process and subproceses only”. Yes, it could be done with a linux network namespace, but that’s a lot of hassle.

    And ps ax in the VM looks suspiciously scarce: #31339 (comment) showing just bash and 03_test_script.sh.

    Seeing this, it might already be namespaced. Though a process namespace doesn’t necessarily mean the network namespace is isolated.

  18. vasild force-pushed on Nov 25, 2024
  19. vasild commented at 4:12 pm on November 25, 2024: contributor

    1592a7dad4...071e43ffae: fix feature_config_args.py and p2p_seednode.py to not generate non-loopback traffic.

    It only becomes critical if network traffic would cause a CI failure.

    My intention here is to fail the CI because otherwise the log will be buried in the CI output and nobody will notice it. It follows that if this fails randomly with false positives when one would have to investigate it manually for arbitrary PRs which is highly highly highly undesirable.

  20. in ci/test/03_test_script.sh:160 in 071e43ffae outdated 
    155+}
156+
157+function traffic_monitor_begin()
158+{
159+  for ifname in $(get_interfaces) ; do
160+    tcpdump -n -i "$ifname" -w "$(tcpdump_file_for_interface "$ifname")" &
    maflcko commented at 4:25 pm on November 25, 2024:
    The return code is ignored, which is why CI passed, despite the tests calling out
    vasild commented at 8:54 am on November 26, 2024:

    On some of the VMs this produces:

    0[09:47:01.270] + tcpdump -n -i eth0 -w /tmp/tcpdump_eth0
1[09:47:01.334] tcpdump: eth0: You don't have permission to perform this capture on that device
2[09:47:01.335] (socket: Operation not permitted)

    and then the CI passes because the return code is ignored. I think better not fail the CI when tcpdump does not work in that environment. It is ok as long as tcpdump works on at least one VM to catch problems.

    maflcko commented at 9:04 am on November 26, 2024:
    The problem is that no one will notice if this isn’t run on any machine, because it will silently pass even if there is an error.
    vasild commented at 9:11 am on November 26, 2024:
    True, if it stops working on all VMs, then nobody will notice. Any ideas how to approach this?
    maflcko commented at 9:24 am on November 26, 2024:

    I’d say it is fine to ignore it by default (if you want). However, there should be one machine in the CI matrix to run the check (and fail on any error).

    The cirrus workers are running in a user account, so they may not have the permissions (unless they are switched to @0xB10C’s workers, which are running as root?). Alternatively, you could try with --cap-add=.../--privileged, but I haven’t tried this. I guess the only task that has the required permissions right now is the ASan GHA task?

    0xB10C commented at 11:22 am on November 27, 2024:

    (unless they are switched to @0xB10C’s workers, which are running as root?)

    the runner setup I’m working on explicitly doesn’t run as root and is far from finished :)

    maflcko commented at 11:52 am on November 27, 2024:

    the runner setup I’m working on explicitly doesn’t run as root

    Are you sure, because the current CI (in this run) is run in a user account (not root), and gives a permission error. The same CI in your run does not give a permission error, so there seems to be a difference.

    The only thing I see is that you are using docker, which IIRC is running rootful by default.

    0xB10C commented at 1:30 pm on November 27, 2024:

    I’m using rootless-docker which runs dockerd as a user account. Inside the container, you’re root and can tcpdump on the containers eth0 interface, but you can’t* e.g. mount and edit the hosts /etc/passwd like you can’t with the user account.

    *until someone finds a vuln in rootless-docker

    maflcko commented at 1:59 pm on November 27, 2024:

    It may be that docker rootless has a different capabilities set, compared to podman. (Can be checked with capsh --print).

    In any case, my preference would be to explicitly list the required caps, instead of relying on a vendor default.

    vasild commented at 3:43 pm on November 27, 2024:

    The problem is that no one will notice if this isn’t run on any machine

    I changed it to insist that the tcpdump file was created on the ASAN env. So the ASAN job will be red if this stops working on it.

    maflcko commented at 7:59 am on November 28, 2024:

    I don’t think this is sufficient. The CI failure (https://github.com/bitcoin/bitcoin/pull/31349#issuecomment-2499335672) will remain, depending on a vendor default.

    Again, my preference would be to explicitly list the required (or removed) caps, instead of relying on a vendor default. Otherwise, it will become harder to run the CI locally, or lead to vendor-lock-in.

    vasild commented at 8:41 am on December 3, 2024:

    … explicitly list the required (or removed) caps …

    Done, NET_RAW is required to run tcpdump: https://www.tcpdump.org/manpages/pcap.3pcap.html

    Plus, the ASAN job requires that tcpdump -w runs and creates the file, otherwise it will be red.

    I think that resolves the concerns from this thread, so I am closing it. Feel free to comment / reopen if there is more to this.

  21. 0xB10C commented at 0:55 am on November 26, 2024: contributor

    Ran this on my CI runner which has 8.8.8.8 configured as DNS server for docker.

    https://cirrus-ci.com/task/5500763260059648?logs=ci#L1137

    0[00:46:26.215] + tcpdump -n -r /tmp/tcpdump_eth0 tcp or udp
1[00:46:26.219] 00:42:50.052764 IP 172.18.0.2.46566 > 8.8.8.8.53: 39301+ A? x9.dummySeed.invalid. (38)
2[00:46:26.219] 00:42:50.053181 IP 172.18.0.2.58686 > 8.8.8.8.53: 36487+ AAAA? x9.dummySeed.invalid. (38)
3[00:46:26.219] 00:42:50.059038 IP 8.8.8.8.53 > 172.18.0.2.46566: 39301 NXDomain 0/1/0 (113)
4[00:46:26.219] 00:42:50.060121 IP 8.8.8.8.53 > 172.18.0.2.58686: 36487 NXDomain 0/1/0 (113)
5[00:46:26.219] 00:42:50.060574 IP 172.18.0.2.34312 > 8.8.8.8.53: 25243+ A? x9.dummySeed.invalid. (38)
6[00:46:26.219] 00:42:50.060939 IP 172.18.0.2.47040 > 8.8.8.8.53: 63641+ AAAA? x9.dummySeed.invalid. (38)
7...

    Edit: My understanding is as follows: The DNS requests normally go to a local DNS resolver which then asks an upstream resolver. The upstream resolver (possibly your ISP) indirectly learns that you are running Bitcoin Core tests, even if there was no direct communication over a non-loopback interface.

      0[00:46:26.209] ++ tcpdump -n -r /tmp/tcpdump_eth0 --direction=out tcp or udp
  1[00:46:26.213] reading from file /tmp/tcpdump_eth0, link-type EN10MB (Ethernet), snapshot length 262144
  2[00:46:26.215] + '[' -n '00:42:50.052764 IP 172.18.0.2.46566 > 8.8.8.8.53: 39301+ A? x9.dummySeed.invalid. (38)
  3[00:46:26.215] 00:42:50.053181 IP 172.18.0.2.58686 > 8.8.8.8.53: 36487+ AAAA? x9.dummySeed.invalid. (38)
  4[00:46:26.215] 00:42:50.059038 IP 8.8.8.8.53 > 172.18.0.2.46566: 39301 NXDomain 0/1/0 (113)
  5[00:46:26.215] 00:42:50.060121 IP 8.8.8.8.53 > 172.18.0.2.58686: 36487 NXDomain 0/1/0 (113)
  6[00:46:26.215] 00:42:50.060574 IP 172.18.0.2.34312 > 8.8.8.8.53: 25243+ A? x9.dummySeed.invalid. (38)
  7[00:46:26.215] 00:42:50.060939 IP 172.18.0.2.47040 > 8.8.8.8.53: 63641+ AAAA? x9.dummySeed.invalid. (38)
  8[00:46:26.215] 00:42:50.066767 IP 8.8.8.8.53 > 172.18.0.2.34312: 25243 NXDomain 0/1/0 (113)
  9[00:46:26.215] 00:42:50.068273 IP 8.8.8.8.53 > 172.18.0.2.47040: 63641 NXDomain 0/1/0 (113)
 10[00:46:26.215] 00:42:50.420185 IP 172.18.0.2.55135 > 8.8.8.8.53: 9419+ A? fakenodeaddr.fakedomain.invalid. (49)
 11[00:46:26.215] 00:42:50.420589 IP 172.18.0.2.42709 > 8.8.8.8.53: 57544+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 12[00:46:26.215] 00:42:50.426139 IP 8.8.8.8.53 > 172.18.0.2.55135: 9419 NXDomain 0/1/0 (124)
 13[00:46:26.215] 00:42:50.426488 IP 8.8.8.8.53 > 172.18.0.2.42709: 57544 NXDomain 0/1/0 (124)
 14[00:46:26.215] 00:42:50.426928 IP 172.18.0.2.42643 > 8.8.8.8.53: 12212+ A? fakenodeaddr.fakedomain.invalid. (49)
 15[00:46:26.215] 00:42:50.427362 IP 172.18.0.2.33528 > 8.8.8.8.53: 41906+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 16[00:46:26.215] 00:42:50.432905 IP 8.8.8.8.53 > 172.18.0.2.42643: 12212 NXDomain 0/1/0 (124)
 17[00:46:26.215] 00:42:50.433389 IP 8.8.8.8.53 > 172.18.0.2.33528: 41906 NXDomain 0/1/0 (124)
 18[00:46:26.215] 00:42:50.785947 IP 172.18.0.2.40413 > 8.8.8.8.53: 23723+ A? x9.dummySeed.invalid. (38)
 19[00:46:26.215] 00:42:50.786307 IP 172.18.0.2.33152 > 8.8.8.8.53: 51880+ AAAA? x9.dummySeed.invalid. (38)
 20[00:46:26.215] 00:42:50.786916 IP 172.18.0.2.60591 > 8.8.8.8.53: 57214+ A? fakenodeaddr.fakedomain.invalid. (49)
 21[00:46:26.215] 00:42:50.787237 IP 172.18.0.2.51085 > 8.8.8.8.53: 45180+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 22[00:46:26.215] 00:42:50.792174 IP 8.8.8.8.53 > 172.18.0.2.33152: 51880 NXDomain 0/1/0 (113)
 23[00:46:26.215] 00:42:50.792196 IP 8.8.8.8.53 > 172.18.0.2.40413: 23723 NXDomain 0/1/0 (113)
 24[00:46:26.215] 00:42:50.794262 IP 8.8.8.8.53 > 172.18.0.2.51085: 45180 NXDomain 0/1/0 (124)
 25[00:46:26.215] 00:42:50.794281 IP 8.8.8.8.53 > 172.18.0.2.60591: 57214 NXDomain 0/1/0 (124)
 26[00:46:26.215] 00:42:50.794701 IP 172.18.0.2.53594 > 8.8.8.8.53: 7814+ AAAA? x9.dummySeed.invalid. (38)
 27[00:46:26.215] 00:42:50.794819 IP 172.18.0.2.33826 > 8.8.8.8.53: 50053+ A? x9.dummySeed.invalid. (38)
 28[00:46:26.215] 00:42:50.795297 IP 172.18.0.2.54482 > 8.8.8.8.53: 30981+ A? fakenodeaddr.fakedomain.invalid. (49)
 29[00:46:26.215] 00:42:50.795592 IP 172.18.0.2.48225 > 8.8.8.8.53: 65050+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
 30[00:46:26.215] 00:42:50.800988 IP 8.8.8.8.53 > 172.18.0.2.33826: 50053 NXDomain 0/1/0 (113)
 31[00:46:26.215] 00:42:50.801160 IP 8.8.8.8.53 > 172.18.0.2.53594: 7814 NXDomain 0/1/0 (113)
 32[00:46:26.215] 00:42:50.801189 IP 8.8.8.8.53 > 172.18.0.2.54482: 30981 NXDomain 0/1/0 (124)
 33[00:46:26.215] 00:42:50.801282 IP 8.8.8.8.53 > 172.18.0.2.48225: 65050 NXDomain 0/1/0 (124)
 34[00:46:26.215] 00:42:51.151925 IP 172.18.0.2.45895 > 8.8.8.8.53: 32540+ A? x9.dummySeed.invalid. (38)
 35[00:46:26.215] 00:42:51.154708 IP 172.18.0.2.39532 > 8.8.8.8.53: 3102+ AAAA? x9.dummySeed.invalid. (38)
 36[00:46:26.215] 00:42:51.160435 IP 8.8.8.8.53 > 172.18.0.2.45895: 32540 NXDomain 0/1/0 (113)
 37[00:46:26.215] 00:42:51.160623 IP 8.8.8.8.53 > 172.18.0.2.39532: 3102 NXDomain 0/1/0 (113)
 38[00:46:26.215] 00:42:51.161037 IP 172.18.0.2.54958 > 8.8.8.8.53: 64262+ A? x9.dummySeed.invalid. (38)
 39[00:46:26.215] 00:42:51.161501 IP 172.18.0.2.55394 > 8.8.8.8.53: 38663+ AAAA? x9.dummySeed.invalid. (38)
 40[00:46:26.215] 00:42:51.167068 IP 8.8.8.8.53 > 172.18.0.2.54958: 64262 NXDomain 0/1/0 (113)
 41[00:46:26.215] 00:42:51.167562 IP 8.8.8.8.53 > 172.18.0.2.55394: 38663 NXDomain 0/1/0 (113)
 42[00:46:26.215] 00:43:13.896684 IP 172.18.0.2.38577 > 8.8.8.8.53: 37859+ A? x9.dummySeed.invalid. (38)
 43[00:46:26.215] 00:43:13.898035 IP 172.18.0.2.57872 > 8.8.8.8.53: 26852+ AAAA? x9.dummySeed.invalid. (38)
 44[00:46:26.215] 00:43:13.904071 IP 8.8.8.8.53 > 172.18.0.2.38577: 37859 NXDomain 0/1/0 (113)
 45[00:46:26.215] 00:43:13.904515 IP 8.8.8.8.53 > 172.18.0.2.57872: 26852 NXDomain 0/1/0 (113)
 46[00:46:26.215] 00:43:13.905051 IP 172.18.0.2.41442 > 8.8.8.8.53: 3978+ A? x9.dummySeed.invalid. (38)
 47[00:46:26.215] 00:43:13.905744 IP 172.18.0.2.51126 > 8.8.8.8.53: 44169+ AAAA? x9.dummySeed.invalid. (38)
 48[00:46:26.215] 00:43:13.910763 IP 8.8.8.8.53 > 172.18.0.2.41442: 3978 NXDomain 0/1/0 (113)
 49[00:46:26.215] 00:43:13.912478 IP 8.8.8.8.53 > 172.18.0.2.51126: 44169 NXDomain 0/1/0 (113)
 50[00:46:26.215] 00:43:14.345922 IP 172.18.0.2.46373 > 8.8.8.8.53: 23556+ AAAA? x9.dummySeed.invalid. (38)
 51[00:46:26.215] 00:43:14.346104 IP 172.18.0.2.55199 > 8.8.8.8.53: 59658+ A? x9.dummySeed.invalid. (38)
 52[00:46:26.215] 00:43:14.352039 IP 8.8.8.8.53 > 172.18.0.2.55199: 59658 NXDomain 0/1/0 (113)
 53[00:46:26.215] 00:43:14.352108 IP 8.8.8.8.53 > 172.18.0.2.46373: 23556 NXDomain 0/1/0 (113)
 54[00:46:26.215] 00:43:14.355678 IP 172.18.0.2.40940 > 8.8.8.8.53: 56364+ AAAA? x9.dummySeed.invalid. (38)
 55[00:46:26.215] 00:43:14.356531 IP 172.18.0.2.51128 > 8.8.8.8.53: 29229+ A? x9.dummySeed.invalid. (38)
 56[00:46:26.215] 00:43:14.361956 IP 8.8.8.8.53 > 172.18.0.2.40940: 56364 NXDomain 0/1/0 (113)
 57[00:46:26.215] 00:43:14.362506 IP 8.8.8.8.53 > 172.18.0.2.51128: 29229 NXDomain 0/1/0 (113)
 58[00:46:26.215] 00:44:27.054154 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043001621 ecr 0,nop,wscale 7], length 0
 59[00:46:26.215] 00:44:27.945389 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043002513 ecr 0,nop,wscale 7], length 0
 60[00:46:26.215] 00:44:28.096227 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043002664 ecr 0,nop,wscale 7], length 0
 61[00:46:26.215] 00:44:28.992218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043003560 ecr 0,nop,wscale 7], length 0
 62[00:46:26.215] 00:44:29.120219 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043003688 ecr 0,nop,wscale 7], length 0
 63[00:46:26.215] 00:44:30.016217 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043004584 ecr 0,nop,wscale 7], length 0
 64[00:46:26.215] 00:44:30.144232 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043004712 ecr 0,nop,wscale 7], length 0
 65[00:46:26.215] 00:44:31.040220 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043005608 ecr 0,nop,wscale 7], length 0
 66[00:46:26.215] 00:44:31.168218 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043005736 ecr 0,nop,wscale 7], length 0
 67[00:46:26.215] 00:44:32.064218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043006632 ecr 0,nop,wscale 7], length 0
 68[00:46:26.215] 00:45:32.199353 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531031147 ecr 0,nop,wscale 7], length 0
 69[00:46:26.215] 00:45:33.248224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531032196 ecr 0,nop,wscale 7], length 0
 70[00:46:26.215] 00:45:34.272224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531033220 ecr 0,nop,wscale 7], length 0
 71[00:46:26.215] 00:45:35.296253 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531034244 ecr 0,nop,wscale 7], length 0
 72[00:46:26.215] 00:45:36.320281 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531035268 ecr 0,nop,wscale 7], length 0
 73[00:46:26.215] 00:45:37.442439 IP 172.18.0.2.52570 > 8.8.8.8.53: 3832+ AAAA? some.node. (27)
 74[00:46:26.215] 00:45:37.442487 IP 172.18.0.2.60704 > 8.8.8.8.53: 1023+ A? some.node. (27)
 75[00:46:26.215] 00:45:37.448551 IP 8.8.8.8.53 > 172.18.0.2.60704: 1023 NXDomain 0/1/0 (102)
 76[00:46:26.215] 00:45:37.448911 IP 8.8.8.8.53 > 172.18.0.2.52570: 3832 NXDomain 0/1/0 (102)
 77[00:46:26.215] 00:45:37.449395 IP 172.18.0.2.53889 > 8.8.8.8.53: 63359+ AAAA? some.node. (27)
 78[00:46:26.215] 00:45:37.449419 IP 172.18.0.2.51133 > 8.8.8.8.53: 6268+ A? some.node. (27)
 79[00:46:26.215] 00:45:37.455701 IP 8.8.8.8.53 > 172.18.0.2.51133: 6268 NXDomain 0/1/0 (102)
 80[00:46:26.215] 00:45:37.455743 IP 8.8.8.8.53 > 172.18.0.2.53889: 63359 NXDomain 0/1/0 (102)
 81[00:46:26.215] 00:45:37.734057 IP 172.18.0.2.50765 > 8.8.8.8.53: 34600+ AAAA? x9.dummySeed.invalid. (38)
 82[00:46:26.215] 00:45:37.734277 IP 172.18.0.2.47474 > 8.8.8.8.53: 53797+ A? x9.dummySeed.invalid. (38)
 83[00:46:26.215] 00:45:37.740114 IP 8.8.8.8.53 > 172.18.0.2.50765: 34600 NXDomain 0/1/0 (113)
 84[00:46:26.215] 00:45:37.740144 IP 8.8.8.8.53 > 172.18.0.2.47474: 53797 NXDomain 0/1/0 (113)
 85[00:46:26.215] 00:45:37.740754 IP 172.18.0.2.33593 > 8.8.8.8.53: 4491+ A? x9.dummySeed.invalid. (38)
 86[00:46:26.215] 00:45:37.740756 IP 172.18.0.2.34759 > 8.8.8.8.53: 34698+ AAAA? x9.dummySeed.invalid. (38)
 87[00:46:26.215] 00:45:37.746602 IP 8.8.8.8.53 > 172.18.0.2.34759: 34698 NXDomain 0/1/0 (113)
 88[00:46:26.215] 00:45:37.746641 IP 8.8.8.8.53 > 172.18.0.2.33593: 4491 NXDomain 0/1/0 (113)
 89[00:46:26.215] 00:45:38.234283 IP 172.18.0.2.33343 > 8.8.8.8.53: 22856+ A? dummySeed.invalid. (35)
 90[00:46:26.215] 00:45:38.234286 IP 172.18.0.2.43752 > 8.8.8.8.53: 55883+ AAAA? dummySeed.invalid. (35)
 91[00:46:26.215] 00:45:38.240530 IP 8.8.8.8.53 > 172.18.0.2.43752: 55883 NXDomain 0/1/0 (110)
 92[00:46:26.215] 00:45:38.240809 IP 8.8.8.8.53 > 172.18.0.2.33343: 22856 NXDomain 0/1/0 (110)
 93[00:46:26.215] 00:45:38.241237 IP 172.18.0.2.35861 > 8.8.8.8.53: 14859+ A? dummySeed.invalid. (35)
 94[00:46:26.215] 00:45:38.241238 IP 172.18.0.2.51487 > 8.8.8.8.53: 46093+ AAAA? dummySeed.invalid. (35)
 95[00:46:26.215] 00:45:38.247292 IP 8.8.8.8.53 > 172.18.0.2.35861: 14859 NXDomain 0/1/0 (110)
 96[00:46:26.215] 00:45:38.247320 IP 8.8.8.8.53 > 172.18.0.2.51487: 46093 NXDomain 0/1/0 (110)
 97[00:46:26.215] 00:45:39.779784 IP 172.18.0.2.34493 > 8.8.8.8.53: 26799+ A? fakenodeaddr. (30)
 98[00:46:26.215] 00:45:39.779791 IP 172.18.0.2.56878 > 8.8.8.8.53: 61600+ AAAA? fakenodeaddr. (30)
 99[00:46:26.215] 00:45:39.785890 IP 8.8.8.8.53 > 172.18.0.2.56878: 61600 NXDomain 0/1/0 (105)
100[00:46:26.215] 00:45:39.785928 IP 8.8.8.8.53 > 172.18.0.2.34493: 26799 NXDomain 0/1/0 (105)
101[00:46:26.215] 00:45:39.786515 IP 172.18.0.2.54375 > 8.8.8.8.53: 52196+ A? fakenodeaddr. (30)
102[00:46:26.215] 00:45:39.786693 IP 172.18.0.2.53292 > 8.8.8.8.53: 56037+ AAAA? fakenodeaddr. (30)
103[00:46:26.215] 00:45:39.792298 IP 8.8.8.8.53 > 172.18.0.2.54375: 52196 NXDomain 0/1/0 (105)
104[00:46:26.215] 00:45:39.792592 IP 8.8.8.8.53 > 172.18.0.2.53292: 56037 NXDomain 0/1/0 (105)
105[00:46:26.215] 00:45:42.174906 IP 11.22.33.44.18444 > 172.18.0.2.51812: Flags [R.], seq 0, ack 3432062660, win 65535, length 0
106[00:46:26.215] 00:45:42.722951 IP 172.18.0.2.33938 > 8.8.8.8.53: 23646+ AAAA? fakeaddress1. (30)
107[00:46:26.215] 00:45:42.723043 IP 172.18.0.2.38954 > 8.8.8.8.53: 24914+ A? fakeaddress1. (30)
108[00:46:26.215] 00:45:42.728907 IP 8.8.8.8.53 > 172.18.0.2.38954: 24914 NXDomain 0/1/0 (105)
109[00:46:26.215] 00:45:42.729111 IP 8.8.8.8.53 > 172.18.0.2.33938: 23646 NXDomain 0/1/0 (105)
110[00:46:26.215] 00:45:42.729684 IP 172.18.0.2.52893 > 8.8.8.8.53: 3801+ AAAA? fakeaddress1. (30)
111[00:46:26.215] 00:45:42.729715 IP 172.18.0.2.57497 > 8.8.8.8.53: 62431+ A? fakeaddress1. (30)
112[00:46:26.215] 00:45:42.735746 IP 8.8.8.8.53 > 172.18.0.2.57497: 62431 NXDomain 0/1/0 (105)
113[00:46:26.215] 00:45:42.735771 IP 8.8.8.8.53 > 172.18.0.2.52893: 3801 NXDomain 0/1/0 (105)
114[00:46:26.215] 00:45:43.175350 IP 11.22.33.44.18444 > 172.18.0.2.51828: Flags [R.], seq 0, ack 3743466588, win 65535, length 0' ']'
115[00:46:26.215] + echo 'Outbound TCP or UDP packets on the non loopback interface generated during tests:'
116[00:46:26.215] Outbound TCP or UDP packets on the non loopback interface generated during tests:
117[00:46:26.215] + tcpdump -n -r /tmp/tcpdump_eth0 tcp or udp
118[00:46:26.218] reading from file /tmp/tcpdump_eth0, link-type EN10MB (Ethernet), snapshot length 262144
119[00:46:26.219] 00:42:50.052764 IP 172.18.0.2.46566 > 8.8.8.8.53: 39301+ A? x9.dummySeed.invalid. (38)
120[00:46:26.219] 00:42:50.053181 IP 172.18.0.2.58686 > 8.8.8.8.53: 36487+ AAAA? x9.dummySeed.invalid. (38)
121[00:46:26.219] 00:42:50.059038 IP 8.8.8.8.53 > 172.18.0.2.46566: 39301 NXDomain 0/1/0 (113)
122[00:46:26.219] 00:42:50.060121 IP 8.8.8.8.53 > 172.18.0.2.58686: 36487 NXDomain 0/1/0 (113)
123[00:46:26.219] 00:42:50.060574 IP 172.18.0.2.34312 > 8.8.8.8.53: 25243+ A? x9.dummySeed.invalid. (38)
124[00:46:26.219] 00:42:50.060939 IP 172.18.0.2.47040 > 8.8.8.8.53: 63641+ AAAA? x9.dummySeed.invalid. (38)
125[00:46:26.219] 00:42:50.066767 IP 8.8.8.8.53 > 172.18.0.2.34312: 25243 NXDomain 0/1/0 (113)
126[00:46:26.219] 00:42:50.068273 IP 8.8.8.8.53 > 172.18.0.2.47040: 63641 NXDomain 0/1/0 (113)
127[00:46:26.219] 00:42:50.420185 IP 172.18.0.2.55135 > 8.8.8.8.53: 9419+ A? fakenodeaddr.fakedomain.invalid. (49)
128[00:46:26.219] 00:42:50.420589 IP 172.18.0.2.42709 > 8.8.8.8.53: 57544+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
129[00:46:26.219] 00:42:50.426139 IP 8.8.8.8.53 > 172.18.0.2.55135: 9419 NXDomain 0/1/0 (124)
130[00:46:26.219] 00:42:50.426488 IP 8.8.8.8.53 > 172.18.0.2.42709: 57544 NXDomain 0/1/0 (124)
131[00:46:26.219] 00:42:50.426928 IP 172.18.0.2.42643 > 8.8.8.8.53: 12212+ A? fakenodeaddr.fakedomain.invalid. (49)
132[00:46:26.219] 00:42:50.427362 IP 172.18.0.2.33528 > 8.8.8.8.53: 41906+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
133[00:46:26.219] 00:42:50.432905 IP 8.8.8.8.53 > 172.18.0.2.42643: 12212 NXDomain 0/1/0 (124)
134[00:46:26.219] 00:42:50.433389 IP 8.8.8.8.53 > 172.18.0.2.33528: 41906 NXDomain 0/1/0 (124)
135[00:46:26.219] 00:42:50.785947 IP 172.18.0.2.40413 > 8.8.8.8.53: 23723+ A? x9.dummySeed.invalid. (38)
136[00:46:26.219] 00:42:50.786307 IP 172.18.0.2.33152 > 8.8.8.8.53: 51880+ AAAA? x9.dummySeed.invalid. (38)
137[00:46:26.219] 00:42:50.786916 IP 172.18.0.2.60591 > 8.8.8.8.53: 57214+ A? fakenodeaddr.fakedomain.invalid. (49)
138[00:46:26.219] 00:42:50.787237 IP 172.18.0.2.51085 > 8.8.8.8.53: 45180+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
139[00:46:26.219] 00:42:50.792174 IP 8.8.8.8.53 > 172.18.0.2.33152: 51880 NXDomain 0/1/0 (113)
140[00:46:26.219] 00:42:50.792196 IP 8.8.8.8.53 > 172.18.0.2.40413: 23723 NXDomain 0/1/0 (113)
141[00:46:26.219] 00:42:50.794262 IP 8.8.8.8.53 > 172.18.0.2.51085: 45180 NXDomain 0/1/0 (124)
142[00:46:26.219] 00:42:50.794281 IP 8.8.8.8.53 > 172.18.0.2.60591: 57214 NXDomain 0/1/0 (124)
143[00:46:26.219] 00:42:50.794701 IP 172.18.0.2.53594 > 8.8.8.8.53: 7814+ AAAA? x9.dummySeed.invalid. (38)
144[00:46:26.219] 00:42:50.794819 IP 172.18.0.2.33826 > 8.8.8.8.53: 50053+ A? x9.dummySeed.invalid. (38)
145[00:46:26.219] 00:42:50.795297 IP 172.18.0.2.54482 > 8.8.8.8.53: 30981+ A? fakenodeaddr.fakedomain.invalid. (49)
146[00:46:26.219] 00:42:50.795592 IP 172.18.0.2.48225 > 8.8.8.8.53: 65050+ AAAA? fakenodeaddr.fakedomain.invalid. (49)
147[00:46:26.219] 00:42:50.800988 IP 8.8.8.8.53 > 172.18.0.2.33826: 50053 NXDomain 0/1/0 (113)
148[00:46:26.219] 00:42:50.801160 IP 8.8.8.8.53 > 172.18.0.2.53594: 7814 NXDomain 0/1/0 (113)
149[00:46:26.219] 00:42:50.801189 IP 8.8.8.8.53 > 172.18.0.2.54482: 30981 NXDomain 0/1/0 (124)
150[00:46:26.219] 00:42:50.801282 IP 8.8.8.8.53 > 172.18.0.2.48225: 65050 NXDomain 0/1/0 (124)
151[00:46:26.219] 00:42:51.151925 IP 172.18.0.2.45895 > 8.8.8.8.53: 32540+ A? x9.dummySeed.invalid. (38)
152[00:46:26.219] 00:42:51.154708 IP 172.18.0.2.39532 > 8.8.8.8.53: 3102+ AAAA? x9.dummySeed.invalid. (38)
153[00:46:26.219] 00:42:51.160435 IP 8.8.8.8.53 > 172.18.0.2.45895: 32540 NXDomain 0/1/0 (113)
154[00:46:26.219] 00:42:51.160623 IP 8.8.8.8.53 > 172.18.0.2.39532: 3102 NXDomain 0/1/0 (113)
155[00:46:26.219] 00:42:51.161037 IP 172.18.0.2.54958 > 8.8.8.8.53: 64262+ A? x9.dummySeed.invalid. (38)
156[00:46:26.219] 00:42:51.161501 IP 172.18.0.2.55394 > 8.8.8.8.53: 38663+ AAAA? x9.dummySeed.invalid. (38)
157[00:46:26.219] 00:42:51.167068 IP 8.8.8.8.53 > 172.18.0.2.54958: 64262 NXDomain 0/1/0 (113)
158[00:46:26.219] 00:42:51.167562 IP 8.8.8.8.53 > 172.18.0.2.55394: 38663 NXDomain 0/1/0 (113)
159[00:46:26.219] 00:43:13.896684 IP 172.18.0.2.38577 > 8.8.8.8.53: 37859+ A? x9.dummySeed.invalid. (38)
160[00:46:26.219] 00:43:13.898035 IP 172.18.0.2.57872 > 8.8.8.8.53: 26852+ AAAA? x9.dummySeed.invalid. (38)
161[00:46:26.219] 00:43:13.904071 IP 8.8.8.8.53 > 172.18.0.2.38577: 37859 NXDomain 0/1/0 (113)
162[00:46:26.219] 00:43:13.904515 IP 8.8.8.8.53 > 172.18.0.2.57872: 26852 NXDomain 0/1/0 (113)
163[00:46:26.219] 00:43:13.905051 IP 172.18.0.2.41442 > 8.8.8.8.53: 3978+ A? x9.dummySeed.invalid. (38)
164[00:46:26.219] 00:43:13.905744 IP 172.18.0.2.51126 > 8.8.8.8.53: 44169+ AAAA? x9.dummySeed.invalid. (38)
165[00:46:26.219] 00:43:13.910763 IP 8.8.8.8.53 > 172.18.0.2.41442: 3978 NXDomain 0/1/0 (113)
166[00:46:26.219] 00:43:13.912478 IP 8.8.8.8.53 > 172.18.0.2.51126: 44169 NXDomain 0/1/0 (113)
167[00:46:26.219] 00:43:14.345922 IP 172.18.0.2.46373 > 8.8.8.8.53: 23556+ AAAA? x9.dummySeed.invalid. (38)
168[00:46:26.219] 00:43:14.346104 IP 172.18.0.2.55199 > 8.8.8.8.53: 59658+ A? x9.dummySeed.invalid. (38)
169[00:46:26.219] 00:43:14.352039 IP 8.8.8.8.53 > 172.18.0.2.55199: 59658 NXDomain 0/1/0 (113)
170[00:46:26.219] 00:43:14.352108 IP 8.8.8.8.53 > 172.18.0.2.46373: 23556 NXDomain 0/1/0 (113)
171[00:46:26.219] 00:43:14.355678 IP 172.18.0.2.40940 > 8.8.8.8.53: 56364+ AAAA? x9.dummySeed.invalid. (38)
172[00:46:26.219] 00:43:14.356531 IP 172.18.0.2.51128 > 8.8.8.8.53: 29229+ A? x9.dummySeed.invalid. (38)
173[00:46:26.219] 00:43:14.361956 IP 8.8.8.8.53 > 172.18.0.2.40940: 56364 NXDomain 0/1/0 (113)
174[00:46:26.219] 00:43:14.362506 IP 8.8.8.8.53 > 172.18.0.2.51128: 29229 NXDomain 0/1/0 (113)
175[00:46:26.219] 00:44:27.054154 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043001621 ecr 0,nop,wscale 7], length 0
176[00:46:26.219] 00:44:27.945389 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043002513 ecr 0,nop,wscale 7], length 0
177[00:46:26.219] 00:44:28.096227 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043002664 ecr 0,nop,wscale 7], length 0
178[00:46:26.219] 00:44:28.992218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043003560 ecr 0,nop,wscale 7], length 0
179[00:46:26.219] 00:44:29.120219 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043003688 ecr 0,nop,wscale 7], length 0
180[00:46:26.219] 00:44:30.016217 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043004584 ecr 0,nop,wscale 7], length 0
181[00:46:26.219] 00:44:30.144232 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043004712 ecr 0,nop,wscale 7], length 0
182[00:46:26.219] 00:44:31.040220 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043005608 ecr 0,nop,wscale 7], length 0
183[00:46:26.219] 00:44:31.168218 IP 172.18.0.2.51812 > 11.22.33.44.18444: Flags [S], seq 3432062659, win 64240, options [mss 1460,sackOK,TS val 1043005736 ecr 0,nop,wscale 7], length 0
184[00:46:26.219] 00:44:32.064218 IP 172.18.0.2.51828 > 11.22.33.44.18444: Flags [S], seq 3743466587, win 64240, options [mss 1460,sackOK,TS val 1043006632 ecr 0,nop,wscale 7], length 0
185[00:46:26.219] 00:45:32.199353 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531031147 ecr 0,nop,wscale 7], length 0
186[00:46:26.219] 00:45:33.248224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531032196 ecr 0,nop,wscale 7], length 0
187[00:46:26.219] 00:45:34.272224 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531033220 ecr 0,nop,wscale 7], length 0
188[00:46:26.219] 00:45:35.296253 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531034244 ecr 0,nop,wscale 7], length 0
189[00:46:26.219] 00:45:36.320281 IP 172.18.0.2.34256 > 0.0.0.1.18444: Flags [S], seq 2197884795, win 64240, options [mss 1460,sackOK,TS val 2531035268 ecr 0,nop,wscale 7], length 0
190[00:46:26.219] 00:45:37.442439 IP 172.18.0.2.52570 > 8.8.8.8.53: 3832+ AAAA? some.node. (27)
191[00:46:26.219] 00:45:37.442487 IP 172.18.0.2.60704 > 8.8.8.8.53: 1023+ A? some.node. (27)
192[00:46:26.219] 00:45:37.448551 IP 8.8.8.8.53 > 172.18.0.2.60704: 1023 NXDomain 0/1/0 (102)
193[00:46:26.219] 00:45:37.448911 IP 8.8.8.8.53 > 172.18.0.2.52570: 3832 NXDomain 0/1/0 (102)
194[00:46:26.219] 00:45:37.449395 IP 172.18.0.2.53889 > 8.8.8.8.53: 63359+ AAAA? some.node. (27)
195[00:46:26.219] 00:45:37.449419 IP 172.18.0.2.51133 > 8.8.8.8.53: 6268+ A? some.node. (27)
196[00:46:26.219] 00:45:37.455701 IP 8.8.8.8.53 > 172.18.0.2.51133: 6268 NXDomain 0/1/0 (102)
197[00:46:26.219] 00:45:37.455743 IP 8.8.8.8.53 > 172.18.0.2.53889: 63359 NXDomain 0/1/0 (102)
198[00:46:26.219] 00:45:37.734057 IP 172.18.0.2.50765 > 8.8.8.8.53: 34600+ AAAA? x9.dummySeed.invalid. (38)
199[00:46:26.219] 00:45:37.734277 IP 172.18.0.2.47474 > 8.8.8.8.53: 53797+ A? x9.dummySeed.invalid. (38)
200[00:46:26.219] 00:45:37.740114 IP 8.8.8.8.53 > 172.18.0.2.50765: 34600 NXDomain 0/1/0 (113)
201[00:46:26.219] 00:45:37.740144 IP 8.8.8.8.53 > 172.18.0.2.47474: 53797 NXDomain 0/1/0 (113)
202[00:46:26.219] 00:45:37.740754 IP 172.18.0.2.33593 > 8.8.8.8.53: 4491+ A? x9.dummySeed.invalid. (38)
203[00:46:26.219] 00:45:37.740756 IP 172.18.0.2.34759 > 8.8.8.8.53: 34698+ AAAA? x9.dummySeed.invalid. (38)
204[00:46:26.219] 00:45:37.746602 IP 8.8.8.8.53 > 172.18.0.2.34759: 34698 NXDomain 0/1/0 (113)
205[00:46:26.219] 00:45:37.746641 IP 8.8.8.8.53 > 172.18.0.2.33593: 4491 NXDomain 0/1/0 (113)
206[00:46:26.219] 00:45:38.234283 IP 172.18.0.2.33343 > 8.8.8.8.53: 22856+ A? dummySeed.invalid. (35)
207[00:46:26.219] 00:45:38.234286 IP 172.18.0.2.43752 > 8.8.8.8.53: 55883+ AAAA? dummySeed.invalid. (35)
208[00:46:26.219] 00:45:38.240530 IP 8.8.8.8.53 > 172.18.0.2.43752: 55883 NXDomain 0/1/0 (110)
209[00:46:26.219] 00:45:38.240809 IP 8.8.8.8.53 > 172.18.0.2.33343: 22856 NXDomain 0/1/0 (110)
210[00:46:26.219] 00:45:38.241237 IP 172.18.0.2.35861 > 8.8.8.8.53: 14859+ A? dummySeed.invalid. (35)
211[00:46:26.219] 00:45:38.241238 IP 172.18.0.2.51487 > 8.8.8.8.53: 46093+ AAAA? dummySeed.invalid. (35)
212[00:46:26.219] 00:45:38.247292 IP 8.8.8.8.53 > 172.18.0.2.35861: 14859 NXDomain 0/1/0 (110)
213[00:46:26.219] 00:45:38.247320 IP 8.8.8.8.53 > 172.18.0.2.51487: 46093 NXDomain 0/1/0 (110)
214[00:46:26.219] 00:45:39.779784 IP 172.18.0.2.34493 > 8.8.8.8.53: 26799+ A? fakenodeaddr. (30)
215[00:46:26.219] 00:45:39.779791 IP 172.18.0.2.56878 > 8.8.8.8.53: 61600+ AAAA? fakenodeaddr. (30)
216[00:46:26.219] 00:45:39.785890 IP 8.8.8.8.53 > 172.18.0.2.56878: 61600 NXDomain 0/1/0 (105)
217[00:46:26.219] 00:45:39.785928 IP 8.8.8.8.53 > 172.18.0.2.34493: 26799 NXDomain 0/1/0 (105)
218[00:46:26.219] 00:45:39.786515 IP 172.18.0.2.54375 > 8.8.8.8.53: 52196+ A? fakenodeaddr. (30)
219[00:46:26.219] 00:45:39.786693 IP 172.18.0.2.53292 > 8.8.8.8.53: 56037+ AAAA? fakenodeaddr. (30)
220[00:46:26.219] 00:45:39.792298 IP 8.8.8.8.53 > 172.18.0.2.54375: 52196 NXDomain 0/1/0 (105)
221[00:46:26.219] 00:45:39.792592 IP 8.8.8.8.53 > 172.18.0.2.53292: 56037 NXDomain 0/1/0 (105)
222[00:46:26.219] 00:45:42.174906 IP 11.22.33.44.18444 > 172.18.0.2.51812: Flags [R.], seq 0, ack 3432062660, win 65535, length 0
223[00:46:26.219] 00:45:42.722951 IP 172.18.0.2.33938 > 8.8.8.8.53: 23646+ AAAA? fakeaddress1. (30)
224[00:46:26.219] 00:45:42.723043 IP 172.18.0.2.38954 > 8.8.8.8.53: 24914+ A? fakeaddress1. (30)
225[00:46:26.219] 00:45:42.728907 IP 8.8.8.8.53 > 172.18.0.2.38954: 24914 NXDomain 0/1/0 (105)
226[00:46:26.219] 00:45:42.729111 IP 8.8.8.8.53 > 172.18.0.2.33938: 23646 NXDomain 0/1/0 (105)
227[00:46:26.219] 00:45:42.729684 IP 172.18.0.2.52893 > 8.8.8.8.53: 3801+ AAAA? fakeaddress1. (30)
228[00:46:26.219] 00:45:42.729715 IP 172.18.0.2.57497 > 8.8.8.8.53: 62431+ A? fakeaddress1. (30)
229[00:46:26.219] 00:45:42.735746 IP 8.8.8.8.53 > 172.18.0.2.57497: 62431 NXDomain 0/1/0 (105)
230[00:46:26.219] 00:45:42.735771 IP 8.8.8.8.53 > 172.18.0.2.52893: 3801 NXDomain 0/1/0 (105)
231[00:46:26.219] 00:45:43.175350 IP 11.22.33.44.18444 > 172.18.0.2.51828: Flags [R.], seq 0, ack 3743466588, win 65535, length 0
  22. 1440000bytes commented at 1:09 pm on November 26, 2024: none

    Concept ACK

    A simple solution to avoid leaking IP address when running tests locally would be to disconnect internet while running tests.

  23. jonatack commented at 1:57 pm on November 26, 2024: member
    Concept ACK. Per https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2024-11-26#1069602: “it turns out the owners of 1.2.3.4, 11.22.33.44 and 8.8.8.8, if they would bother, would know the IP address of every dev who runs the functional tests at home.”
  24. vasild force-pushed on Nov 27, 2024
  25. DrahtBot added the label CI failed on Nov 27, 2024
  26. DrahtBot commented at 4:42 pm on November 27, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33609854840

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  27. vasild force-pushed on Nov 27, 2024
  28. vasild commented at 5:36 pm on November 27, 2024: contributor

    8799018bd5...803ed4638b: include #31343 into this PR to demonstrate that #31343 works as intended and also to turn the CI here green.

    About the false positives - I think it is worth trying this in its current mode where any detected traffic is assumed to have originated from the tests and fails the CI. If this ever fails the CI for another reason (false positive), then it would be easy to turn this into a “report in the logs only but don’t fail” by removing the exit 1 line.

    About the DNS traffic - I did cat /etc/resolv.conf in the CI and the name server is indeed configured as a local one: nameserver 127.0.0.11.

  29. BrandonOdiwuor commented at 5:44 pm on November 27, 2024: contributor
    Concept ACK
  30. vasild marked this as a draft on Nov 28, 2024
  31. vasild force-pushed on Nov 28, 2024
  32. vasild commented at 5:31 pm on November 28, 2024: contributor
    Converted to draft for a while, testing docker with full privileges (need cirrus which does not run in my personal fork).
  33. vasild force-pushed on Nov 29, 2024
  34. vasild force-pushed on Nov 29, 2024
  35. DrahtBot removed the label CI failed on Nov 29, 2024
  36. vasild force-pushed on Nov 29, 2024
  37. vasild force-pushed on Dec 2, 2024
  38. vasild force-pushed on Dec 2, 2024
  39. vasild force-pushed on Dec 2, 2024
  40. vasild force-pushed on Dec 3, 2024
  41. vasild commented at 10:48 am on December 3, 2024: contributor
    Ready for review. I updated the OP with some details.
  42. vasild marked this as ready for review on Dec 3, 2024
  43. in ci/test/03_test_script.sh:160 in c88464d754 outdated 
    155+}
156+
157+function traffic_monitor_begin()
158+{
159+  for ifname in $(get_interfaces) ; do
160+    tcpdump -n -i "$ifname" -w "$(tcpdump_file_for_interface "$ifname")" &
    maflcko commented at 10:59 am on December 3, 2024:

    I don’t think this works reliably? See:

    0              This output will be buffered if written to a file or pipe, so a program reading from the file or pipe may not see packets for an arbitrary amount of time after they are received.  Use the -U flag to cause packets to be
1              written as soon as they are received.
    vasild commented at 10:28 am on December 4, 2024:

    Even better! Added -U.

    With -U is does not matter anymore, but I think it was ok before as well, see my comment below.

  44. in ci/test/03_test_script.sh:174 in c88464d754 outdated 
    165+{
166+  # Stop all tcpdump instances (we want the word splitting if "jobs -p" returns more than one PID).
167+  # shellcheck disable=SC2046
168+  while kill -SIGTERM $(jobs -p) ; do
169+    sleep 1
170+  done
    maflcko commented at 11:01 am on December 3, 2024:

    Why is this needed? The file isn’t cleared anyway (and exited if it isn’t clear), so might as well just run the processes and never kill them?

    Also, it doesn’t seem to be working anyway on some of the tasks?

    vasild commented at 10:39 am on December 4, 2024:

    To 1. flush the file and 2. so that the next set of tests can start fresh without leftover tcpdump processes. Now 1. is not needed because of -U, but I will leave this kill for 2.

    In my experiments SIGTERM would cause tcpdump to exit gracefully - flush the file and then exit. SIGKILL would result in unflushed file.

    Yes, in some tasks kill gives “permission denied” even if docker is running with --privileged or --cap-add ALL.

    maflcko commented at 10:48 am on December 4, 2024:

    Yes, in some tasks kill gives “permission denied” even if docker is running with --privileged or --cap-add ALL.

    That is interesting, because all tasks should be running on workers that are set up identically.

    tests can start fresh without leftover tcpdump processes

    My thinking was to just have a single process, which is started once and never killed. This should remove the need to kill it, and also the need (and code) to start it every time.

    vasild commented at 10:57 am on December 4, 2024:

    Then it would be unclear whether the traffic was generated by unit tests, functional tests or fuzz tests. So I start/stop a separate monitoring for each of those set of tests. I perceived this information useful when investigating. It is already tough that it is not known which test generated the traffic.

    Somehow I do not like reading from the “live” file while it is being written. -U would help not to miss data, but it might happen that by the time we read it the last packet is still being written and is half-written and the reading tcpdump is upset by that.

    vasild commented at 10:59 am on December 4, 2024:

    Yes, in some tasks kill gives “permission denied” even if docker is running with --privileged or --cap-add ALL.

    That is interesting, because all tasks should be running on workers that are set up identically.

    I noticed those tasks print:

    Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg

    Btw, now with tcpdump -U those tasks do not print “tcpdump: truncated dump file; tried to read 4 file header bytes, only got 0” anymore

    maflcko commented at 11:06 am on December 4, 2024:

    That is interesting, because all tasks should be running on workers that are set up identically.

    I noticed those tasks print:

    Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg

    That should be identical for all tasks as well. See:

    The only difference should be that one is bare metal and the other is KVM, but I don’t see how this could cause a difference here.

    vasild commented at 11:11 am on December 4, 2024:
    Hmm, right. Then I have no idea why kill does not work on some. It is ok as long as it works on at least one.
    vasild commented at 1:19 pm on December 17, 2024:
    In one of the previous pushes I changed it to use distinct tcpdump files for each test run, so 2. from #31349 (review) is not necessary any more. Removed this snippet.
  45. vasild force-pushed on Dec 4, 2024
  46. in ci/test/03_test_script.sh:185 in feabffd808 outdated 
    180+    fi
181+    # We are running as root and those files are created with owner:group =
182+    # tcpdump:tcpdump and then `tcpdump -r` refuses to read them with an error
183+    # "permission denied" if they are not owned by root:root.
184+    chown root:root "$f"
185+    if [ -n "$(tcpdump -n -r "$f" --direction=out tcp or udp)" ] ; then
    maflcko commented at 10:49 am on December 4, 2024:
    This is also ignoring the return code? So it could also lead to CI silently passing on all tasks, when it should not.
    maflcko commented at 10:53 am on December 4, 2024:
    Generally, I am not sure if bash is the right language to implement something like this. Silently ignoring possible errors doesn’t seem like a great approach. Other languages, such as Rust or Python are a bit more explicit in documenting when a return code is ignored.
    vasild commented at 12:07 pm on December 4, 2024:
    Ignoring the return status of tcpdump here is deliberate because some VMs, the ones that couldn’t stop tcpdump failed to read the unflushed file with an error like “tcpdump: truncated dump file; tried to read 4 file header bytes, only got 0”. Now they don’t due to the added -U, hmm…
    maflcko commented at 12:32 pm on December 4, 2024:

    I’d say this is unrelated to flushing. Ignoring the return code in all tasks can mean that even the tasks where the detection is working are going to silently break tomorrow. As mentioned previously in the thread you closed, I think this is brittle.

    To clarify, my feedback not just applies to the line I commented on, but it applies to all code where the return code is silently ignored and where it could lead to silent breakage.

    vasild commented at 1:46 pm on December 4, 2024:
    Ok, I changed this to fail if tcpdump gives error when it reads the file.
    maflcko commented at 3:43 pm on December 4, 2024:

    Again, my feedback not just applies to the line I commented on, but it applies to all code where the return code is silently ignored and where it could lead to silent breakage.

    For example, pipefail isn’t set, so get_interfaces can also silently fail …

    vasild commented at 4:43 pm on December 4, 2024:

    I assume ifconfig without any arguments will never fail.

    Do you think it would be better to change it to:

    0 function get_interfaces()
1 {
2+  set -o pipefail
3   ifconfig | awk -F ':| ' '/^[^[:space:]]/ { if (!match($1, /^lo/)) { print $1 } }'
4+  set +o pipefail
5 }

    or that is just clutter?

    vasild commented at 6:53 pm on December 4, 2024:
    There is also top -l 1 -s 0 | awk ' /PhysMem/ {print}' near the start of 03_test_script.sh. Maybe set -o pipefail at the start of the file? (would be somewhat out of scope of this PR, but I am fine with it if nobody objects)
    maflcko commented at 7:57 am on December 5, 2024:

    I assume ifconfig without any arguments will never fail.

    I don’t think this is right. Even if the program was a magic unicorn that never could fail for any reason at all, it will fail when it is missing.

    This can happen easily when someone removes “unused” CI packages with the proof that CI passes.

    You can also check it locally:

    0# ifconfig | cat
1sh: 13: ifconfig: not found
2# echo $?
30

    There is also top -l 1 -s 0 | awk ' /PhysMem/ {print}'

    Correct, but this is a macos-only debug-only log, unrelated to checking the behavior of Bitcoin Core or the Bitcoin Core tests. Even if it were to fail, I don’t see how it could lead to a test failure or test misbehaviour being silently ignored.

    vasild commented at 9:55 am on December 5, 2024:
    Enabled pipefail as in the above diff. Thanks!
  47. vasild force-pushed on Dec 4, 2024
  48. vasild commented at 1:46 pm on December 4, 2024: contributor
    feabffd808...925ee8707e: (hopefully) address #31349 (review)
  49. vasild commented at 3:07 pm on December 4, 2024: contributor

    I think the tcpdump approach works well enough. I am posting here just for the records - an alternative approach is to use iptables to detect traffic. I managed to get it to log the traffic, but couldn’t read that log - I could only see a summary of how many packets matched, not individual ones and their destination address.

    Below is a WIP patch which can be used as a starting point if I or somebody else wants to pursue that further.

     0diff --git a/ci/test/00_setup_env.sh b/ci/test/00_setup_env.sh
 1index 94149106ee..7889910684 100755
 2--- a/ci/test/00_setup_env.sh
 3+++ b/ci/test/00_setup_env.sh
 4@@ -61,10 +61,10 @@ export CCACHE_COMPRESS=${CCACHE_COMPRESS:-1}
 5 export CCACHE_DIR="${CCACHE_DIR:-$BASE_SCRATCH_DIR/ccache}"
 6 # Folder where the build result is put (bin and lib).
 7 export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out}
 8 # The folder for previous release binaries.
 9 # This folder exists only on the ci guest, and on the ci host as a volume.
10 export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/prev_releases}
11-export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential pkg-config curl ca-certificates ccache python3 rsync git procps bison e2fsprogs cmake net-tools tcpdump}
12+export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential pkg-config curl ca-certificates ccache python3 rsync git procps bison e2fsprogs cmake net-tools tcpdump bind9-host iptables inetutils-telnet}
13 export GOAL=${GOAL:-install}
14 export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
15 export CI_RETRY_EXE=${CI_RETRY_EXE:-"retry --"}
16diff --git a/ci/test/02_run_container.sh b/ci/test/02_run_container.sh
17index e1539e8bef..0fb5feaec5 100755
18--- a/ci/test/02_run_container.sh
19+++ b/ci/test/02_run_container.sh
20@@ -88,13 +88,24 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
21   # When detecting podman-docker, `--external` should be added.
22   docker image prune --force --filter "label=$CI_IMAGE_LABEL"
23 
24   # Append $USER to /tmp/env to support multi-user systems and $CONTAINER_NAME
25   # to allow support starting multiple runs simultaneously by the same user.
26   # shellcheck disable=SC2086
27-  CI_CONTAINER_ID=$(docker run --cap-add LINUX_IMMUTABLE --cap-add NET_RAW $CI_CONTAINER_CAP --rm --interactive --detach --tty \
28+  CI_CONTAINER_ID=$(docker run \
29+                  --cap-add LINUX_IMMUTABLE \
30+                  --cap-add NET_ADMIN \
31+                  --cap-add SYSLOG \
32+                  $CI_CONTAINER_CAP \
33+                  --cap-add NET_ADMIN \
34+                  --cap-add NET_RAW \
35+                  --privileged \
36+                  --rm \
37+                  --interactive \
38+                  --detach \
39+                  --tty \
40                   --mount "type=bind,src=$BASE_READ_ONLY_DIR,dst=$BASE_READ_ONLY_DIR,readonly" \
41                   --mount "${CI_CCACHE_MOUNT}" \
42                   --mount "${CI_DEPENDS_MOUNT}" \
43                   --mount "${CI_DEPENDS_SOURCES_MOUNT}" \
44                   --mount "${CI_PREVIOUS_RELEASES_MOUNT}" \
45                   --env-file /tmp/env-$USER-$CONTAINER_NAME \
46diff --git a/ci/test/03_test_script.sh b/ci/test/03_test_script.sh
47index 71048330e6..c4b22417a5 100755
48--- a/ci/test/03_test_script.sh
49+++ b/ci/test/03_test_script.sh
50@@ -188,12 +188,31 @@ function traffic_monitor_end()
51       tcpdump -n -r "$f" tcp or udp
52       exit 1
53     fi
54   done
55 }
56 
57+ifconfig || :
58+echo "Non-loopback interfaces: $(get_interfaces)"
59+
60+traffic_monitor_begin
61+iptables  -A OUTPUT -j LOG --log-prefix "ttttt1" --log-level emerg || :
62+ip6tables -A OUTPUT -j LOG --log-prefix "ttttt2" --log-level emerg || :
63+iptables  -A OUTPUT -m addrtype \! --dst-type LOCAL -j LOG --log-prefix "ttttt3" --log-level emerg || :
64+ip6tables -A OUTPUT -m addrtype \! --dst-type LOCAL -j LOG --log-prefix "ttttt4" --log-level emerg || :
65+id || :
66+cat /etc/resolv.conf || :
67+host bitcoin.org || :
68+host nonexistentinvalidfoobarbaz.org || :
69+telnet 50.60.1.2 3456 || :
70+dmesg |grep ttttt || :
71+grep -r ttttt /var/log/ || :
72+iptables -v -x -n -L || :
73+traffic_monitor_end
74+exit 34
75+
76 if [ "$RUN_UNIT_TESTS" = "true" ]; then
77   traffic_monitor_begin
78   DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" CTEST_OUTPUT_ON_FAILURE=ON ctest --stop-on-failure "${MAKEJOBS}" --timeout $(( TEST_RUNNER_TIMEOUT_FACTOR * 60 ))
79   traffic_monitor_end
80 fi
  50. DrahtBot added the label Needs rebase on Dec 5, 2024
  51. vasild force-pushed on Dec 5, 2024
  52. vasild commented at 8:16 am on December 5, 2024: contributor
    925ee8707e...56dbe78934: rebase due to conflicts
  53. vasild force-pushed on Dec 5, 2024
  54. DrahtBot added the label CI failed on Dec 5, 2024
  55. DrahtBot removed the label Needs rebase on Dec 5, 2024
  56. vasild commented at 9:37 am on December 5, 2024: contributor
    56dbe78934...5f3b24711e: use distinct tcpdump file per test (unitparallel, unitsequential, functional, fuzz, tidy) and log that in case of detected traffic.
  57. vasild force-pushed on Dec 5, 2024
  58. vasild commented at 9:54 am on December 5, 2024: contributor
    5f3b24711e...a6c3c9defb: #31349 (review)
  59. DrahtBot removed the label CI failed on Dec 5, 2024
  60. vasild force-pushed on Dec 11, 2024
  61. vasild commented at 5:24 am on December 11, 2024: contributor
    a6c3c9defb...46e38e2e33: rebase and remove merged #31343 which this PR included.
  62. DrahtBot added the label Needs rebase on Dec 13, 2024
  63. vasild force-pushed on Dec 17, 2024
  64. vasild commented at 1:16 pm on December 17, 2024: contributor
    46e38e2e33...95fc90610a: rebase due to conflicts and don’t stop the tcpdump processes because it is now not necessary because distinct files are used for each run, #31349 (review).
  65. DrahtBot removed the label Needs rebase on Dec 17, 2024
  66. DrahtBot added the label CI failed on Dec 17, 2024
  67. DrahtBot removed the label CI failed on Dec 17, 2024
  68. in ci/test/03_test_script.sh:175 in 95fc90610a outdated 
    170+{
171+  test_name="$1"
172+
173+  for ifname in $(get_interfaces) ; do
174+    f=$(tcpdump_file "$test_name" "$ifname")
175+    if [ ! -e "$f" ] && [ "$FILE_ENV" != "./ci/test/00_setup_env_native_asan.sh" ] ; then
    0xB10C commented at 12:24 pm on December 21, 2024:
    nit: someone renaming this file down the line might not be aware of a dependency on this exact file name
    vasild commented at 10:51 am on December 23, 2024:

    Yeah, I was thinking the same… Should I change it to the following?

    0-     if [ ! -e "$f" ] && [ "$FILE_ENV" != "./ci/test/00_setup_env_native_asan.sh" ] ; then
1+     if [ ! -e "$f" ] && [ "$CONTAINER_NAME" != "ci_native_asan" ] ; then
    0xB10C commented at 1:42 pm on December 23, 2024:

    Yet another alternative would be to set a flag like CI_FAIL_IF_NO_TCPDUMP_FILE (or similar) in 00_setup_env_native_asan.sh and add a comment mentioning the dependency:

    • if the script name or the container name are renamed, this will still work
    • if script is removed, someone might see the comment about the dependency during review
    • it’s easy to add more tasks to the list of required tasks by just adding CI_FAIL_IF_NO_TCPDUMP="true"
    vasild commented at 12:08 pm on December 24, 2024:
    Done! Much better.
  69. 0xB10C approved
  70. 0xB10C commented at 12:27 pm on December 21, 2024: contributor

    ACK 95fc90610a1162fc06e61b607488d05229c9909f

    Nice solution to use 127.0.0.1:1 as unreachable proxy for DNS requests too. I ran this on my CI runners with a 8.8.8.8 and 1.1.1.1 as DNS servers as before and the tests don’t connect out now. I ran with test: avoid generating non-loopback traffic from feature_config_args.py reverted to check that a few tasks fail and report the outgoing connections. I did a light code review, looked at e.g. the tcpdump options being used, but didn’t look at the iptables alternative.

  71. DrahtBot requested review from laanwj on Dec 21, 2024
  72. DrahtBot requested review from BrandonOdiwuor on Dec 21, 2024
  73. DrahtBot requested review from maflcko on Dec 21, 2024
  74. DrahtBot requested review from jonatack on Dec 21, 2024
  75. sipa commented at 2:24 pm on December 23, 2024: member
    Concept ACK
  76. vasild force-pushed on Dec 23, 2024
  77. vasild commented at 6:42 pm on December 23, 2024: contributor
    95fc90610a...0ac9caf7be: do #31349 (review)
  78. vasild force-pushed on Dec 23, 2024
  79. vasild commented at 6:48 pm on December 23, 2024: contributor
    0ac9caf7be...d8cd80e814: forgot to add a comment in the previous push
  80. vasild force-pushed on Dec 24, 2024
  81. vasild commented at 11:29 am on December 24, 2024: contributor
    d8cd80e...48843ec: s/=/!=/ :facepalm:
  82. in ci/test/03_test_script.sh:181 in 48843ec694 outdated 
    176+      # In some CI environments this script is not running as root and so the
177+      # tcpdump errors and does not create $f. Skip silently those, but we
178+      # need at least one where tcpdump can run and this is the ASAN one. So
179+      # treat the absence of $f as an error only on the ASAN task.
180+      continue
181+    fi
    maflcko commented at 11:42 am on December 24, 2024:

    Given that you no longer kill tcpdump, the code should work on all CI tasks, except for the GHA macos tasks?

    If there are only a few tasks that do not work, it may be better to just carve them out, instead of enumerating all the ones that work or dealing with code to ensure that at least one works for each config (fuzz/tidy/tests)?

    vasild commented at 12:29 pm on December 24, 2024:

    The table in the OP describes each task and its status wrt to this. There are 18 tasks. It is supposed to work on the top 5+4=9 tasks. The other 9 tasks are not expected to run this - e.g. Windows or tidy that does not run any tests or macOS that runs without docker and has no enough permissions.

    The intention here is to have at least one task that runs this. But it will not hurt to enforce it on more. Do you want to add CI_FAIL_IF_NO_TCPDUMP_FILE=1 to more tasks? Or reverse the logic to CI_OK_IF_NO_TCPDUMP_FILE=1 in the tasks that are expected to not run this (maybe it is the same if it is 9 vs 9 tasks)?

    maflcko commented at 12:39 pm on December 24, 2024:

    The other 9 tasks are not expected to run this - e.g. Windows or tidy that does not run any tests or macOS that runs without docker and has no enough permissions.

    If no tests are run, then no exclusion is needed as well. Though, if this is not expected to run on tidy, I wonder why traffic_monitor_begin "tidy" exists? This leaves macOS, which can be excluded. The other tasks, which do not use the CI infra anyway, so don’t need to be excluded as well.

    vasild commented at 1:41 pm on January 6, 2025:

    You are right! Flipped the logic to exclude only macOS.

    No strong opinion on “tidy” - I added it because those tests should not generate internet traffic. Would drop it if requested.

  83. fjahr commented at 5:05 pm on December 30, 2024: contributor

    Concept ACK

    The changes in the tests look good but I need a little more time with the CI stuff. Maybe the PR could have been split there to get the critical fixes in faster but I guess it’s a bit late now.

  84. vasild force-pushed on Jan 6, 2025
  85. vasild commented at 1:38 pm on January 6, 2025: contributor
    48843ec694...bbfc58a0af: rebase and “If there are only a few tasks that do not work, it may be better to just carve them out, instead of enumerating all the ones that work”, see above.
  86. fanquake commented at 3:02 pm on January 6, 2025: member

    I tried testing the asan job, via time MAKEJOBS="-j17" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" ./ci/test_run_all.sh, on a Fedora dev box, and it fails with the following output:

     0+ chown root:root /tmp/tcpdump_unitparallel_eth0
 1++ tcpdump -n -r /tmp/tcpdump_unitparallel_eth0 --direction=out tcp or udp
 2reading from file /tmp/tcpdump_unitparallel_eth0, link-type EN10MB (Ethernet), snapshot length 262144
 3+ out='14:47:32.023429 IP6 1111:1111::4.33262 > 1111:1111::1.53: 9493+ A? debuginfod.fedoraproject.org. (46)
 414:47:32.023487 IP6 1111:1111::4.33262 > 1111:1111::1.53: 1815+ AAAA? debuginfod.fedoraproject.org. (46)
 514:47:32.050396 IP6 1111:1111::4.51620 > 1111:1111::1.53: 22098+ A? debuginfod.fedoraproject.org. (46)
 614:47:32.050457 IP6 1111:1111::4.51620 > 1111:1111::1.53: 19536+ AAAA? debuginfod.fedoraproject.org. (46)
 714:47:32.059718 IP6 1111:1111::4.48563 > 1111:1111::1.53: 672+ A? debuginfod.fedoraproject.org. (46)
 814:47:32.060018 IP6 1111:1111::4.48563 > 1111:1111::1.53: 1965+ AAAA? debuginfod.fedoraproject.org. (46)
 914:47:32.069417 IP6 1111:1111::1.53 > 1111:1111::4.51620: 22098 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1014:47:32.069442 IP6 1111:1111::1.53 > 1111:1111::4.33262: 9493 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1114:47:32.069467 IP6 1111:1111::1.53 > 1111:1111::4.48563: 672 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1214:47:32.163955 IP6 1111:1111::4.41445 > 1111:1111::1.53: 34154+ A? debuginfod.fedoraproject.org. (46)
1314:47:32.164474 IP6 1111:1111::4.41445 > 1111:1111::1.53: 49518+ AAAA? debuginfod.fedoraproject.org. (46)
1414:47:32.164826 IP6 1111:1111::1.53 > 1111:1111::4.41445: 34154 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
1514:47:32.184123 IP6 1111:1111::4.51525 > 1111:1111::1.53: 40013+ A? debuginfod.fedoraproject.org. (46)
1614:47:32.184205 IP6 1111:1111::4.51525 > 1111:1111::1.53: 32841+ AAAA? debuginfod.fedoraproject.org. (46)
1714:47:32.186645 IP6 1111:1111::1.53 > 1111:1111::4.51525: 40013 2/0/0 A 38.145.60.21, A 38.145.60.20 (78)
  87. vasild commented at 11:12 am on January 7, 2025: contributor

    The current CI failure is unrelated to this PR and is fixed separately in #31614. It will probably pass if the CI task is restarted because it is a race.

    14:47:32.023429 IP6 1111:1111::4.33262 > 1111:1111::1.53: 9493+ A? debuginfod.fedoraproject.org. (46) @fanquake so some piece of the command:

    0DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" CTEST_OUTPUT_ON_FAILURE=ON ctest --stop-on-failure "${MAKEJOBS}" --timeout $(( TEST_RUNNER_TIMEOUT_FACTOR * 60 ))

    Generated DNS requests for debuginfod.fedoraproject.org :eyes: :open_mouth:. It is probably not test_bitcoin because grepping for debuginfo or fedoraproject in the entire code base of Bitcoin Core yields no results. Is this running directly on the host or inside docker? Looks like some spyware that calls home when ctest or test_bitcoin is run.

    Edit: if it is running on the host without docker then it could be that processes other than ctest and test_bitcoin are run during the tests and they generate the traffic. Related: https://fedoraproject.org/wiki/Debuginfod.

  88. luke-jr commented at 8:38 pm on January 8, 2025: member

    i’m not aware of a straightforward way to “log network traffic of this process and subproceses only”.

    Not quite that, but iptables has the ability to match and log uid

  89. 0xB10C commented at 11:46 pm on January 8, 2025: contributor

    I tried testing the asan job, via time MAKEJOBS="-j17" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" ./ci/test_run_all.sh, on a Fedora dev box, and it fails with the following output.

    I guess checking that we’re in a docker container (that hopefully doesn’t have other services that does anything network related) is the only option here. Only fail in docker, otherwise just print, if possible.

    Possibly by checking that /proc/1/cgroup exists and this is not empty with cat /proc/1/cgroup | grep docker - though I haven’t tested this.

  90. maflcko commented at 12:22 pm on January 9, 2025: member

    I tried testing the asan job, via time MAKEJOBS="-j17" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" ./ci/test_run_all.sh, on a Fedora dev box, and it fails with the following output.

    I guess checking that we’re in a docker container (that hopefully doesn’t have other services that does anything network related) is the only option here. Only fail in docker, otherwise just print, if possible.

    None of the CI tasks use fedora, so the only way to run the native_asan task on Fedora is to run it in a container, which still failed here. So I don’t think your suggestion will help, but I haven’t tried it.

  91. fanquake commented at 12:39 pm on January 9, 2025: member
    Note that this isn’t Fedora specific. The exact same issue happens on Ubuntu.
  92. maflcko commented at 12:42 pm on January 9, 2025: member
    Maybe the test changes can be split up from the ci changes, given that this is still WIP?
  93. vasild commented at 9:26 am on January 13, 2025: contributor
    Extracted the changes to the tests in #31646 (suggested by @fjahr and @maflcko).
  94. vasild commented at 9:35 am on January 13, 2025: contributor
    @fanquake, is the problem you reported above #31349 (comment) when the tests run on the host, without a docker?
  95. Sjors commented at 11:01 am on January 13, 2025: member

    Concept ACK

    I tested this with https://github.com/Sjors/bitcoin/pull/76 on my own CI setup, which uses Cirrus workers configured using (roughly) the instructions here: https://github.com/bitcoin/bitcoin/blob/master/.cirrus.yml#L8 (using podman-docker)

    (I did run into an issue, left a comment there since it might be specific to my setup)

  96. maflcko commented at 11:16 am on January 13, 2025: member
    I guess an env var is leaking into the docker. To reproduce you could try setting the URL via the env var DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/ , but I haven’t tried it.
  97. maflcko commented at 11:58 am on January 13, 2025: member

    Checked locally that the following fails for me as well: time env -i HOME="$HOME" PATH="$PATH" USER="$USER" MAKEJOBS="-j$( nproc )" FILE_ENV="./ci/test/00_setup_env_native_asan.sh" DEBUGINFOD_URLS='https://debuginfod.fedoraproject.org/' ./ci/test_run_all.sh

    In theory it is not recommended to run the tests without a clean env, according to the ci/README.md. So I am not sure if this can be left as-is, or a workaround for DEBUGINFOD_URLS is convenient .

  98. glozow referenced this in commit 7cd862aab9 on Jan 14, 2025
  99. DrahtBot added the label Needs rebase on Jan 14, 2025
  100. ci: detect outbound internet traffic generated while running tests 
    Resolves https://github.com/bitcoin/bitcoin/issues/31339
    69e076664d
  101. vasild force-pushed on Jan 15, 2025
  102. vasild commented at 8:30 am on January 15, 2025: contributor

    bbfc58a0af...69e076664d: drop the tests changes from this PR because they were already merged via #31646. So here remains a CI change to detect future regressions.

    If people want to run this manually, outside of the CI in a way that causes false positive (discussion above), what about something like this:

     0--- i/ci/test/03_test_script.sh
 1+++ w/ci/test/03_test_script.sh
 2@@ -184,13 +184,15 @@ function traffic_monitor_end()
 3     # "permission denied" if they are not owned by root:root.
 4     chown root:root "$f"
 5     out="$(tcpdump -n -r "$f" --direction=out tcp or udp)"
 6     if [ -n "$out" ] ; then
 7       echo "Error: outbound TCP or UDP packets on the non loopback interface generated during $test_name tests:" >&2
 8       tcpdump -n -r "$f" tcp or udp
 9-      exit 1
10+      if [ -z "$INTERNET_TRAFFIC_EXPECTED" ] ; then
11+        exit 1
12+      fi
13     fi
14   done
15 }
16 
17 if [ "$RUN_UNIT_TESTS" = "true" ]; then
18   traffic_monitor_begin "unitparallel"

    and then, obviously, set INTERNET_TRAFFIC_EXPECTED=1 when running this manually (in unclean environment)? Or leave it as it is?

  103. DrahtBot removed the label Needs rebase on Jan 15, 2025
  104. DrahtBot added the label Needs rebase on Jan 17, 2025
  105. DrahtBot commented at 1:12 pm on January 17, 2025: contributor

    🐙 This pull request conflicts with the target branch and needs rebase.


vasild DrahtBot maflcko laanwj 0xB10C 1440000bytes jonatack BrandonOdiwuor sipa fjahr fanquake luke-jr Sjors


laanwj jonatack BrandonOdiwuor

Labels
Tests Needs rebase

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-01-21 06:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me