Picking up from #21006.
After commit https://github.com/bitcoin/bitcoin/commit/ccd8ef65f93ed82a87cee634660bed3ac17d9eb5 it is no longer required to hold cs_main when calling ReadBlockFromDisk. This can be verified in master at https://github.com/bitcoin/bitcoin/blob/master/src/node/blockstorage.cpp#L755. Same can be seen for UndoReadFromDisk https://github.com/bitcoin/bitcoin/blob/master/src/node/blockstorage.cpp#L485.
The first commit moves ReadBlockFromDisk outside the lock scope in rest_block, where we can see a huge performance improvement when fetching blocks with multiple threads.
My test setup, on an Intel i7 with 8 cores (16 threads):
- Start a fully synced bitcoind, with this
bitcoin.conf:
0 rest=1
1 rpcthreads=16
2 rpcworkqueue=64
3 rpcuser=user
4 rpcpassword=password
- Run ApacheBench: 10000 requests, 16 parallel threads, fetching block nr. 750000 in binary:
0 ab -n 10000 -c 16 "http://127.0.0.1:8332/rest/block/0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e.bin"
Time per request (mean) 183 ms on master 30 ms this branch
So this can process 6.1 times as many requests, and saturates all the cores instead of keeping them partly idle waiting in the lock. With 8 threads the mean times were 90 ms on master and 19 ms on this branch, a speedup of 4.7x.
Big thanks to martinus for finding this and the original PR.
The second commit is from a suggestion on the original PR by jonatack to remove the unnecessary LOCK(cs_main) in the zmq notifier’s NotifyBlock.
I also found that this approach could be applied to rpcs getblock (including verbosity=3), getblockstats, and gettxoutproof with similar very good results. The above benchmarks steps need to be modified slightly for RPC. Run the following ApacheBench command with different request data in a file named data.json:
0ab -p data.json -n 10000 -c 16 -A user:password "http://127.0.0.1:8332/"
For getblock, use the following in data.json:
0{"jsonrpc": "1.0", "id": "curltest", "method": "getblock", "params": ["0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e"]}
master - 184 ms mean request time branch - 28 ms mean request time
For getblock with verbosity level 3, use the following in data.json:
0{"jsonrpc": "1.0", "id": "curltest", "method": "getblock", "params": ["0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e", 3]}
This verbosity level fetches an undo file from disk, so it benefits from this approach as well. However, a lot of time is spent serializing to JSON so the performance gain is not as severe. master - 818 ms mean request time branch - 505 ms mean request time
For getblockstats, use the following in data.json:
0{"jsonrpc": "1.0", "id": "curltest", "method": "getblockstats", "params": ["0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e", ["minfeerate","avgfeerate"]]}
This request used a lock on reading both a block and undo file, so the results are very good. master - 244 ms mean request time branch - 28 ms mean request time
