contrib: Add deterministic-unittest-coverage #31901

1. 
   maflcko commented at 8:30 pm on February 18, 2025: member

   The contrib/devtools/test_deterministic_coverage.sh script is problematic:

   • It is written in bash. This can lead to issues when running with the ancient bash version shipped by macOS by default, or can lead to other compatibility issues, such as #31588 (review). Also, pipefail isn’t set, so IO errors may be silently ignored.
   • It is based on gcov. This can lead to issues, such as #31588#pullrequestreview-2602169248 (possibly due to prefix-map), or #31588 (comment) (gcovr processing error), or #31588#pullrequestreview-2605954001 (gcovr assertion error).
   • The script is severely outdated, with the last update to NON_DETERMINISTIC_TESTS being in the prior decade.

   Instead of patching around all issues one-by-one, just provide a fresh rewrite, based on the recently added deterministic-fuzz-coverage tool based on clang, llvm-cov, and llvm-profdata. (Initial feedback indicates that this is a more promising attempt: #31588 (comment) and #31588 (comment)).

   The new tool also sets RANDOM_CTX_SEED=21 as suggested by hodlinator in #31588 (comment).

2. gitignore: target/

   Needed for the recently added
   contrib/devtools/deterministic-fuzz-coverage/target/

   fa1e0a7228
3. 
   DrahtBot commented at 8:30 pm on February 18, 2025: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Code Coverage & Benchmarks

   For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31901.

   Reviews

   See the guideline for information on the review process.

   Type	Reviewers
   Concept ACK	dergoegge, theStack

   If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #31588 (contrib: fix test_deterministic_coverage.sh script for out-of-tree builds by theStack)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

4. DrahtBot renamed this:
   contrib: Add deterministic-unittest-coverage
   contrib: Add deterministic-unittest-coverage
   on Feb 18, 2025
5. DrahtBot added the label Scripts and tools on Feb 18, 2025
6. 
   dergoegge commented at 10:10 am on February 19, 2025: member

   Concept ACK
7. in contrib/devtools/deterministic-unittest-coverage/src/main.rs:26 in bbbb6998f9 outdated

   21+    exit(1)
   22+}
   23+
   24+fn sanity_check(test_exe: &Path) {
   25+    for tool in [LLVM_PROFDATA, LLVM_COV, DIFF] {
   26+        let output = Command::new(tool).arg("--version").output();
   

   ---

   ---

   

   theStack commented at 1:32 pm on February 21, 2025:

   On the machine where I tried this yesterday quickly (OpenBSD 7.6, with LLVM 16.0.6 included in the base system), neither llvm-profdata nor diff know a --version argument, so the sanity check failed even though the binaries were there. Will try on a Linux machine with Ubuntu in a bit.

   ---

   

   maflcko commented at 1:51 pm on February 21, 2025:

   It should be possible to drop diff and use git diff --no-index (or something else). As for the llvm tool, it looks like they added the arg later:

   0# llvm-profdata-19 --version 
   1Debian LLVM version 19.1.4
   2  Optimized build.
   3# llvm-profdata-16 --version
   4llvm-profdata-16: Unknown command!
   5USAGE: llvm-profdata-16 <merge|show|overlap> [args...]
   

   ---

   

   maflcko commented at 1:53 pm on February 21, 2025:

   Looks like they added it in 17, so I could require that, or remove the --version check.

   0# llvm-profdata-17 --version 
   1llvm-profdata-17
   2Ubuntu LLVM version 17.0.6
   3  Optimized build.
   

8. in contrib/devtools/deterministic-unittest-coverage/src/main.rs:18 in bbbb6998f9 outdated

   13+const DIFF: &str = "diff";
   14+
   15+fn exit_help(err: &str) -> ! {
   16+    eprintln!("Error: {}", err);
   17+    eprintln!();
   18+    eprintln!("Usage: program ./build_dir [custom test filter (default: all but known non-deterministic tests)]");
   

   ---

   ---

   

   theStack commented at 1:34 pm on February 21, 2025:

   0    eprintln!("Usage: program ./build_dir <custom test filter>");
   

   as there is no default yet and specifying a filter is required by now?

9. 
   theStack commented at 1:36 pm on February 21, 2025: contributor

   Concept ACK
10. contrib: deterministic-fuzz-coverage fixups

    * Remove unused -fPIC in the docs. This is harmless, but no longer
      needed after commit 06b9236f4327525875768af5fc554c651c5ec3cf.
    * Name the fuzz_corpora dir after its real name.
    * Add missing cargo lock file.
    * Use git instead of diff command to increase compatibility
    * Use --help instead of --version to increase compatibility
    * Use assert consistently for unexpected errors.
    * Remove redundant Stdio::from.
    * Fix typos.

    faa6826911
11. contrib: Add deterministic-unittest-coverage

    This replaces the bash script with a tool based on clang/llvm tools.

    fafce12a48
12. maflcko force-pushed on Feb 21, 2025

Contributors
maflcko DrahtBot dergoegge theStack

Labels
Scripts and tools