tests: Add script checking for deterministic line coverage in unit tests #15296

pull practicalswift wants to merge 1 commits into bitcoin:master from practicalswift:test_deterministic_coverage changing 1 files +151 −0
  1. practicalswift commented at 9:29 pm on January 30, 2019: contributor

    Add script checking for deterministic line coverage in unit tests.

    Context: #14343 (“coverage reports non-deterministic”)

    When the coverage is deterministic this script can be invoked from Travis to guard against regressions, but left inactive for now.

    Output in case of determinism:

    0$ contrib/test_deterministic_coverage.sh 2
1[2019-01-30 20:08:46] Measuring coverage, run [#1](/bitcoin-bitcoin/1/) of 2
2[2019-01-30 20:10:45] Measuring coverage, run [#2](/bitcoin-bitcoin/2/) of 2
3
4Coverage test passed: Deterministic coverage across 2 runs.

    Output in case of non-determinism:

     0$ contrib/test_deterministic_coverage.sh 2
 1[2019-01-30 20:08:46] Measuring coverage, run [#1](/bitcoin-bitcoin/1/) of 2
 2[2019-01-30 20:10:45] Measuring coverage, run [#2](/bitcoin-bitcoin/2/) of 2
 3
 4The line coverage is non-deterministic between runs.
 5
 6The test suite must be deterministic in the sense that the set of lines executed at least
 7once must be identical between runs. This is a neccessary condition for meaningful coverage
 8measuring.
 9
10--- gcovr.run-1.txt   2019-01-30 23:14:07.419418694 +0100
11+++ gcovr.run-2.txt   2019-01-30 23:15:57.998811282 +0100
12@@ -471,7 +471,7 @@
13 test/crypto_tests.cpp                        270     270   100%
14 test/cuckoocache_tests.cpp                   142     142   100%
15 test/dbwrapper_tests.cpp                     148     148   100%
16-test/denialofservice_tests.cpp               225     225   100%
17+test/denialofservice_tests.cpp               225     224    99%   363
18 test/descriptor_tests.cpp                    116     116   100%
19 test/fs_tests.cpp                             24       3    12%   14,16-17,19-20,23,25-26,29,31-32,35-36,39,41-42,45-46,49,51-52
20 test/getarg_tests.cpp                        111     111   100%
21@@ -585,5 +585,5 @@
22 zmq/zmqpublishnotifier.h                       5       0     0%   12,31,37,43,49
23 zmq/zmqrpc.cpp                                21       0     0%   16,18,20,22,33-35,38-45,49,52,56,60,62-63
24 ------------------------------------------------------------------------------
25-TOTAL                                      61561   27606    44%
26+TOTAL                                      61561   27605    44%
27 ------------------------------------------------------------------------------

    In this case line 363 of test/denialofservice_tests.cpp was executed only in the second run. Non-determinism detected!

  2. practicalswift force-pushed on Jan 30, 2019
  3. practicalswift force-pushed on Jan 30, 2019
  4. practicalswift force-pushed on Jan 30, 2019
  5. practicalswift force-pushed on Jan 30, 2019
  6. practicalswift force-pushed on Jan 30, 2019
  7. fanquake added the label Tests on Jan 30, 2019
  8. fanquake added the label Scripts and tools on Jan 30, 2019
  9. fanquake removed the label Tests on Jan 30, 2019
  10. practicalswift force-pushed on Jan 30, 2019
  11. practicalswift force-pushed on Jan 30, 2019
  12. practicalswift force-pushed on Jan 31, 2019
  13. practicalswift force-pushed on Jan 31, 2019
  14. practicalswift force-pushed on Jan 31, 2019
  15. practicalswift force-pushed on Feb 1, 2019
  16. practicalswift force-pushed on Feb 1, 2019
  17. practicalswift force-pushed on Feb 1, 2019
  18. practicalswift force-pushed on Feb 1, 2019
  19. practicalswift force-pushed on Feb 1, 2019
  20. practicalswift force-pushed on Feb 1, 2019
  21. practicalswift force-pushed on Feb 1, 2019
  22. practicalswift force-pushed on Feb 4, 2019
  23. in contrib/test_deterministic_coverage.sh:113 in 621cd3e74b outdated 
    84+    if [[ $(get_file_suffix_count gcda) != 0 ]]; then
85+        echo "Error: Stale *.gcda files found. Exiting."
86+        exit 1
87+    fi
88+    TEST_OUTPUT_TEMPFILE=$(mktemp)
89+    if ! BOOST_TEST_RUN_FILTERS="$(IFS=":"; echo "!${NON_DETERMINISTIC_TESTS[*]}" | sed 's/:/:!/g')" ${TEST_BITCOIN_BINARY} > "${TEST_OUTPUT_TEMPFILE}" 2>&1; then
    MarcoFalke commented at 8:39 pm on February 4, 2019:
    Instead of running all tests, could this take a specific test suite or test case to run, so the other tests wouldn’t have to be run if you are looking a specific case?
    practicalswift commented at 9:16 pm on February 4, 2019:
    @MarcoFalke Good idea! Now implemented. --help describes the syntax :-)
  24. practicalswift force-pushed on Feb 4, 2019
  25. practicalswift force-pushed on Feb 5, 2019
  26. MarcoFalke referenced this in commit 9e7f8f6c82 on Feb 5, 2019
  27. in contrib/test_deterministic_coverage.sh:17 in 3f6a8bd8fb outdated 
    12+# Use GCOV_EXECUTABLE="llvm-cov gcov" if compiling with clang.
13+GCOV_EXECUTABLE="gcov"
14+
15+# Disable tests known to cause non-deterministic behaviour and document the source or point of non-determinism.
16+NON_DETERMINISTIC_TESTS=(
17+    "bloom_tests/rolling_bloom"                               # bloom_tests.cpp: data[i] = RandomData(); [...] if (rb1.contains(data[i]))
    MarcoFalke commented at 10:38 pm on February 5, 2019:
    this was fixed
  28. practicalswift force-pushed on Feb 7, 2019
  29. practicalswift commented at 9:52 pm on February 7, 2019: contributor
    @MarcoFalke Removed suppressions for now fixed bloom_tests/rolling_bloom and coins_tests/updatecoins_simulation_test. Please re-review :-)
  30. MarcoFalke commented at 9:40 pm on February 14, 2019: member
    utACK 84e8413d26ae35f6bd279ff260c742334fd061d7
  31. MarcoFalke commented at 10:23 pm on February 14, 2019: member
    Should be in ./contrib/devtools/?
  32. tests: Add script checking for deterministic line coverage 43206239a8
  33. practicalswift force-pushed on Feb 15, 2019
  34. practicalswift commented at 8:38 am on February 15, 2019: contributor
    @MarcoFalke Moved! Please re-review :-)
  35. MarcoFalke commented at 3:11 pm on February 15, 2019: member
    Could this be modified to work with functional tests as well? :)
  36. practicalswift commented at 10:38 pm on February 28, 2019: contributor

    @MarcoFalke Sure, I can try to find time to add that feature in a second iteration. Perhaps we can merge this version first to limit work in progress and get the first version in?

    The non-functional tests use case should be finished in the current version and ready from merge.

  37. MarcoFalke merged this on Mar 2, 2019
  38. MarcoFalke closed this on Mar 2, 2019
  39. MarcoFalke referenced this in commit 80112b17e7 on Mar 2, 2019
  40. MarcoFalke referenced this in commit 0c9de67f34 on Apr 16, 2019
  41. practicalswift deleted the branch on Apr 10, 2021
  42. random-zebra referenced this in commit 93f43f0f81 on Apr 14, 2021
  43. PastaPastaPasta referenced this in commit 7fe7b445f2 on Jun 27, 2021
  44. PastaPastaPasta referenced this in commit ba7dc914eb on Jun 27, 2021
  45. PastaPastaPasta referenced this in commit 3e4dd8d9fc on Jun 28, 2021
  46. PastaPastaPasta referenced this in commit 3b989cd794 on Jun 28, 2021
  47. PastaPastaPasta referenced this in commit 28b30d5464 on Jun 28, 2021
  48. PastaPastaPasta referenced this in commit ebf56727ab on Jun 29, 2021
  49. PastaPastaPasta referenced this in commit 4eafb4dea6 on Jun 29, 2021
  50. PastaPastaPasta referenced this in commit fad61df3f2 on Jul 1, 2021
  51. PastaPastaPasta referenced this in commit 2624540a97 on Jul 1, 2021
  52. PastaPastaPasta referenced this in commit 963e4c3d9f on Jul 8, 2021
  53. PastaPastaPasta referenced this in commit dd2f85c084 on Jul 10, 2021
  54. Munkybooty referenced this in commit 832b2ab570 on Sep 30, 2021
  55. Munkybooty referenced this in commit 904eece82c on Oct 7, 2021
  56. Munkybooty referenced this in commit 4306ca966f on Oct 12, 2021
  57. Munkybooty referenced this in commit 0d9074977f on Oct 16, 2021
  58. Munkybooty referenced this in commit b128c108b9 on Oct 20, 2021
  59. Munkybooty referenced this in commit 57453a23b6 on Oct 21, 2021
  60. Munkybooty referenced this in commit 97f31af382 on Oct 23, 2021
  61. PastaPastaPasta referenced this in commit c916ca2a6b on Oct 24, 2021
  62. PastaPastaPasta referenced this in commit e0e2a2e3fe on Oct 25, 2021
  63. pravblockc referenced this in commit cd90eefb06 on Nov 18, 2021
  64. DrahtBot locked this on Aug 18, 2022


practicalswift MarcoFalke

Labels
Scripts and tools

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-05 01:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me