fuzz: Speed up `dbwrapper_concurrent

marcofleon commented at 9:12 PM on June 12, 2026: contributor

Limiting how many read queries each worker executes significantly speeds up this test, especially when running with sanitizers. This still builds the full query list, and then takes the first 512/2000 after each worker shuffles it. I think that keeps some more operation diversity vs just lowering the query max directly. It also allowed me to reuse and test against a corpus I already had. Let me know if I'm wrong, but I don't think this test needs every worker to execute an identical query list to be effective.

This PR also reverts the num_entries max from 3000 back to 5000, as that didn't have much effect on input speed and restores a bit of lost coverage.

Lastly, as #35455 (review) points out, remove the unnecessary Mutex from StartReadPoolIfNeeded(). Fuzz targets are entered sequentially within a process and parallel fuzzing uses separate processes/forks, so a mutex to prevent two in-process callers from racing to start the pool isn't needed.

DrahtBot added the label Fuzzing on Jun 12, 2026

DrahtBot commented at 9:12 PM on June 12, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35521.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	brunoerg

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

marcofleon commented at 9:28 PM on June 12, 2026: contributor

Coverage differences: current master, this PR, and then also coverage if the only change to the test was increasing num_entries to 5000. You can see some of the restored coverage, e.g. in leveldb/util/cache.cc LRUCache::Ref isn't hit if entries is 3000. These are all after a several hundred cpu-hour campaign.

As for speed, here is running an ASan + UBSan build on master:

FUZZ=dbwrapper_concurrent_reads ./asanfuzzbuild/bin/fuzz dbwrapperconcurcorp3000/ -runs=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 4170807245
INFO: Loaded 1 modules   (616070 inline 8-bit counters): 616070 [0x55bdc36dd5d0, 0x55bdc3773c56), 
INFO: Loaded 1 PC tables (616070 PCs): 616070 [0x55bdc3773c58,0x55bdc40da4b8), 
INFO:     1551 files found in dbwrapperconcurcorp3000/
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 1551 min: 1b max: 700b total: 492935b rss: 151Mb
Slowest unit: 23 s:
artifact_prefix='./'; Test unit written to ./slow-unit-8c659dab9cf2c5aee95aad5c87be59fc3f009012
Slowest unit: 29 s:
artifact_prefix='./'; Test unit written to ./slow-unit-1ebc5f38743a66bdb1618f1dbed7bfbc3ccdf7a6
Slowest unit: 33 s:
artifact_prefix='./'; Test unit written to ./slow-unit-6cf667add79b8dd38abdef5eaf143ab328f970f3
[#1024](/bitcoin-bitcoin/1024/)	pulse  cov: 5453 ft: 15429 corp: 714/131Kb exec/s: 1 rss: 1408Mb
^C==2648531== libFuzzer: run interrupted; exiting

And then this PR:

FUZZ=dbwrapper_concurrent_reads ./asanfuzzbuild/bin/fuzz dbwrapperconcurcorp5000/ -runs=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 23721510
INFO: Loaded 1 modules   (616074 inline 8-bit counters): 616074 [0x563d32d32610, 0x563d32dc8c9a), 
INFO: Loaded 1 PC tables (616074 PCs): 616074 [0x563d32dc8ca0,0x563d3372f540), 
INFO:     1567 files found in dbwrapperconcurcorp5000/
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 1567 min: 1b max: 700b total: 463623b rss: 151Mb
[#512](/bitcoin-bitcoin/512/)	pulse  cov: 5112 ft: 13609 corp: 334/4743b exec/s: 256 rss: 455Mb
[#1024](/bitcoin-bitcoin/1024/)	pulse  cov: 5414 ft: 17076 corp: 700/114Kb exec/s: 11 rss: 1075Mb
[#1575](/bitcoin-bitcoin/1575/)	INITED cov: 5469 ft: 19215 corp: 992/273Kb exec/s: 5 rss: 1225Mb
[#1575](/bitcoin-bitcoin/1575/)	DONE   cov: 5469 ft: 19215 corp: 992/273Kb lim: 664 exec/s: 5 rss: 1225Mb
Done 1575 runs in 301 second(s)

These are the two existing corpora I got from comparing the current test on master using 3000 vs 5000 entries. Because I don't change any of the fuzz input consuming in this PR, I just reused the "5000" corpus. Takes about 5 minutes with 512 max queries and it was 163 seconds with 256 queries. So, we could adjust MAX_READ_QUERIES_PER_WORKER to make this target faster or a bit slower (if we think more queries is better).

andrewtoth commented at 9:48 PM on June 12, 2026: contributor

I don't think this test needs every worker to execute an identical query list to be effective.

Correct, and the fuzzer will eventually pick overlapping lists as well if it chooses fewer queries than max.

This is still not very fast :/

How many vcpus are on the machine you are running on? Fewer than 9 and it will of course be much slower.

marcofleon commented at 10:08 PM on June 12, 2026: contributor

Realized I didn't minimize the corpus:

FUZZ=dbwrapper_concurrent_reads ./asanfuzzbuild/bin/fuzz dbwrapperconcurcorpfinal/ -runs=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 796326126
INFO: Loaded 1 modules   (616065 inline 8-bit counters): 616065 [0x559b7b9fb610, 0x559b7ba91c91), 
INFO: Loaded 1 PC tables (616065 PCs): 616065 [0x559b7ba91c98,0x559b7c3f84a8), 
INFO:      396 files found in dbwrapperconcurcorpfinalfinall/
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 396 min: 1b max: 700b total: 139898b rss: 150Mb
[#256](/bitcoin-bitcoin/256/)	pulse  cov: 5451 ft: 16949 corp: 242/50Kb exec/s: 8 rss: 993Mb
[#398](/bitcoin-bitcoin/398/)	INITED cov: 5470 ft: 18679 corp: 362/120Kb exec/s: 5 rss: 993Mb
[#398](/bitcoin-bitcoin/398/)	DONE   cov: 5470 ft: 18679 corp: 362/120Kb lim: 700 exec/s: 5 rss: 993Mb
Done 398 runs in 71 second(s)

That's with 512 queries and ASan+UBSan. Should be the same coverage, though I haven't checked yet.

This is still not very fast :/

Definitely one of our slower targets, but not having any libFuzzer slow units is a big improvement. I think this would be acceptable for CI. This fuzz harness is just inherently expensive, with what it's trying to test. As I mentioned, lowering the max to 256 seems to provide ~2x speed up. Or if anyone sees any other ways to speed it up without losing useful coverage?

How many vcpus are on the machine you are running on?

32

in src/test/fuzz/dbwrapper.cpp:173 in 184a510124 outdated

 168 | @@ -169,12 +169,13 @@ void VerifyIterator(CDBWrapper& dbw, const Oracle& oracle,
 169 |  /** Maximum number of concurrent reader threads in dbwrapper_concurrent_reads. */
 170 |  constexpr size_t MAX_READ_WORKERS{8};
 171 |  
 172 | +/** Maximum number of queries each worker executes in dbwrapper_concurrent_reads. */
 173 | +constexpr size_t MAX_READ_QUERIES_PER_WORKER{512};
 174 | +
 175 |  ThreadPool g_read_pool{"dbfuzz"};
 176 | -Mutex g_read_pool_mutex;

maflcko commented at 8:50 AM on June 15, 2026:

nit: Could also drop the other one src/test/fuzz/threadpool.cpp:Mutex g_pool_mutex; for consistency, to avoid having to do it in the future?

marcofleon commented at 1:10 PM on June 17, 2026:

Done, made it a separate commit.

fuzz: Speed up dbwrapper_concurrent_reads harness

Limit how many read queries each worker executes. This significantly
speeds up the test, as each worker runs around 75% fewer (2000 to 512)
expensive LevelDB operations (like `IteratorSeek`) but still
ends up hitting the intended target code.

Revert the `num_entries` max from 3000 back to 5000, as that didn't
have much effect on input speed and restores a bit of lost coverage.

993d0fb3b3

fuzz: Remove unnecessary thread pool mutexes

Remove the `Mutex` from the `threadpool` and `dbwrapper_concurrent_reads`
pool startup helpers. Fuzz targets are entered sequentially within a
process and parallel fuzzing uses separate processes/forks, which each
have their own copy of the global thread pool. Therefore, a mutex to
prevent two in-process callers from racing to start the pool isn't needed.

ed260393a9

in src/test/fuzz/dbwrapper.cpp:426 in 184a510124

 422 |              std::string key_str;
 423 |              start_latch.arrive_and_wait();
 424 |              const std::unique_ptr<CDBIterator> it{db.NewIterator()};
 425 |              // Every read must agree with the oracle, the source of truth.
 426 | -            for (const auto i : order) {
 427 | +            for (size_t pos{0}; pos < queries_to_run; ++pos) {

frankomosh commented at 10:40 AM on June 15, 2026:

style nit: could also use for (const auto i : std::span{order}.first(queries_to_run)) { , similar idiom as p2p_transport_serialization.cpp#L255.

marcofleon commented at 1:11 PM on June 17, 2026:

Done. Yeah that's better, thanks.

marcofleon force-pushed on Jun 17, 2026

brunoerg commented at 7:16 PM on June 17, 2026: contributor

Concept ACK

in src/test/fuzz/dbwrapper.cpp:174 in ed260393a9

 169 | @@ -169,12 +170,13 @@ void VerifyIterator(CDBWrapper& dbw, const Oracle& oracle,
 170 |  /** Maximum number of concurrent reader threads in dbwrapper_concurrent_reads. */
 171 |  constexpr size_t MAX_READ_WORKERS{8};
 172 |  
 173 | +/** Maximum number of queries each worker executes in dbwrapper_concurrent_reads. */
 174 | +constexpr size_t MAX_READ_QUERIES_PER_WORKER{512};

brunoerg commented at 7:20 PM on June 17, 2026:

993d0fb3b305f63cc22701cfd1e6e6d956b6db91: Couldn't we adopt an even lower number (e.g. 128)? Since we're trying to test concurrent read interleavings, I think 128 would give the same substantial race coverage.

marcofleon commented at 6:00 PM on June 18, 2026:

Tested with 128 and it is ~3x faster. The coverage is the same with the corpus I have. So same code paths are hit, just with less repetition. I think the speed up could be worth it, but it's hard to evaluate exactly what more read queries provides (more LevelDB stress per input?). @andrewtoth curious what you think.

I'm going to add TSan as a fuzzer instance to our infra, so the increased speed would be helpful there.

andrewtoth commented at 12:38 AM on June 19, 2026:

Lowering to 128 makes sense to me. Especially if coverage is the same. Should be more than enough for TSan to detect a race.

fuzz: Speed up `dbwrapper_concurrent_reads` harness #35521

Code Coverage & Benchmarks

Reviews