Inspired by #31132 (comment).
We currently do concurrent leveldb reads when accessing our indexes.
txindex- we callFindTx()from multiple RPC threads.blockfilterindex- we callLookupFilter/Header()concurrently frommsghandthread for p2p requests as well as RPC threads.coinstatsindex- we callLookUpStats()from multiple RPC threads.txospenderindex- we callFindSpender()from multiple RPC threads.
We also read from our chainstate and blocks index while background compactions are writing.
While OSS-Fuzz does cover leveldb (https://github.com/google/oss-fuzz/blob/master/projects/leveldb/fuzz_db.cc), it doesn’t cover multi threaded access. Without a deterministic hypervisor this fuzz harness won’t be deterministic, but we can at least run it with TSan to get a higher confidence that the synchronization code in leveldb is correct. Hopefully other reviewers find this useful.
This harness creates a global threadpool with 16 threads, and then creates an in-memory levelDB which it seeds with deterministically random values. It selects up to 16 threads and chooses a random set of keys to query. It first performs all queries on the db on a single thread to get a baseline, then synchronizes all threads on a latch so they hit the db at the same time. Each thread performs the same queries, and afterwards are all checked against the baseline.
It uses a DeterministicEnv to capture background compaction work when seeding the db, which is also run immediately after the latch is released. This causes a race between compaction and reading, ensuring we exercise many thread synchronization code paths in leveldb.
I ran both TSan and ASan/UBSan overnight with no issues.