fuzz: Corpus sharding in parallel runs #36121

maflcko commented at 2:07 PM on August 29, 2026: member

Currently, the fuzz runner accepts a --par option to schedule fuzz runs in parallel. This is fine. However, when only a single target is selected, --par will not speed up the run. Moreover, when multiple targets are selected, the run-time of the longest target dominates.

Fix both issues by splitting the corpus to into --par equal-sized shards by default. The sharding can be disabled, if needed.

Can be tested e.g. by running a single target:

time ./bld-cmake/test/fuzz/test_runner.py --par 1  -l DEBUG ./qa-assets/fuzz_corpora/ utxo_snapshot  # slow
time ./bld-cmake/test/fuzz/test_runner.py --par 99 -l DEBUG ./qa-assets/fuzz_corpora/ utxo_snapshot  # fast

fuzz: Corpus sharding in parallel runs

To better utilize the given --par option, split each corpus into shards by default.

The diff can be reviewed via --ignore-all-space

d0000027db

DrahtBot added the label Fuzzing on Aug 29, 2026

DrahtBot commented at 2:07 PM on August 29, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/36121.

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
Concept ACK	jeanpablojp

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

maflcko commented at 3:35 PM on August 29, 2026: member

Hmm,

macOS native, fuzz seems 3 min slower than master?? (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657343 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704437?pr=36121#logs)
asan is 9 min faster (nice) (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657348 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704584?pr=36121)
msan is 15 min faster (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657192 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704553?pr=36121)
Windows is 7 min faster (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657181 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704453?pr=36121)

jeanpablojp commented at 1:47 PM on August 31, 2026: contributor

Concept ACK

I had a look at the logs. Looks like on macOS the pool was already saturated. Summing the process times and dividing by --par gives 613s there, against 200s for the slowest single target, so there was no tail to cut. On Windows and MSan it's the other way round, and that is where the win comes from.

Almost all of the macOS side comes from one target. utxo_total_supply builds a ChainTestingSetup inside the target body, so a datadir per input, and three shards of 680 take around 240s each against 200s for all 2040 in one process. Same on Windows.

Is it worth gating this on something, or does the win on the other jobs already outweigh it?

in test/fuzz/test_runner.py:69 in d0000027db

  64 | @@ -64,6 +65,11 @@ def main():
  65 |          default=4,
  66 |          help='How many targets to merge or execute in parallel.',
  67 |      )
  68 | +    parser.add_argument(
  69 | +        '--corpus-shards',

jeanpablojp commented at 1:47 PM on August 31, 2026:

nit: --help ends up with two defaults, the one in the help string and a (default: None) that ArgumentDefaultsHelpFormatter appends.

in test/fuzz/test_runner.py:341 in d0000027db

 347 | -        ]
 348 | -        empty_dir = not any(corpus_path.iterdir())
 349 | -        if using_libfuzzer:
 350 | -            if empty_min_time and empty_dir:
 351 | -                args += [f"-max_total_time={empty_min_time}"]
 352 | +    with tempfile.TemporaryDirectory(dir=corpus.parent, prefix='.fuzz-shards-') as shard_root:

jeanpablojp commented at 1:47 PM on August 31, 2026:

This gets created before we know whether there will be any sharding, so with a read-only parent the run dies in mkdtemp, and --corpus-shards 1 doesn't avoid it. It also survives a SIGTERM, left next to the corpus.

And since the with wraps the collection loop too, the removal only happens after the last job, with everything else idle. 31s on macOS.

in test/fuzz/test_runner.py:348 in d0000027db

 354 | +            corpus_path = corpus / t
 355 | +            os.makedirs(corpus_path, exist_ok=True)
 356 | +            corpus_files = sorted(path for path in corpus_path.iterdir() if path.is_file())
 357 | +            empty_dir = not corpus_files
 358 | +
 359 | +            if corpus_shards > 1 and len(corpus_files) > 1:

jeanpablojp commented at 1:47 PM on August 31, 2026:

This hits every target that has a corpus, none has fewer than three inputs. cmpctblock sometimes does the same kind of per-input work through ResetChainmanAndMempool.

The hard links are one per corpus input, so that part doesn't shrink with more workers. On macOS that takes 155s before the first job gets collected.

in test/fuzz/test_runner.py:394 in d0000027db

 439 | +            except subprocess.CalledProcessError as e:
 440 | +                if e.stdout:
 441 | +                    logging.info(e.stdout)
 442 | +                if e.stderr:
 443 | +                    logging.info(e.stderr)
 444 | +                logging.info(f"⚠️ Failure generated from target with exit code {e.returncode}: {result.args}")

jeanpablojp commented at 1:47 PM on August 31, 2026:

nit: this path is already gone by the time anyone reads the log, it's inside the shard dir. The file name survives, the path doesn't.

And with the sys.exit(1) in here, the dir disappears under the shards that are still running, so on the standalone builds they die on the read_file Assert instead of on the input that actually broke.

in test/fuzz/test_runner.py:399 in d0000027db

 444 | +                logging.info(f"⚠️ Failure generated from target with exit code {e.returncode}: {result.args}")
 445 | +                sys.exit(1)
 446 | +            if using_libfuzzer:
 447 | +                done_stat = [l for l in output.splitlines() if "DONE" in l]
 448 | +                assert len(done_stat) == 1
 449 | +                stats.append((target, done_stat[0]))

jeanpablojp commented at 1:47 PM on August 31, 2026:

Summary goes from 237 lines to 3754 on your asan run, and cov: becomes per process. coins_view_db reports sixteen values between 12033 and 13709 where master reports 14509, and since it's a union there's no way to put it back together. So you can't compare coverage across a change any more. Could the shards be folded back into one line per target?