Add benchmarks to `bench

laanwj commented at 1:46 pm on April 15, 2016: member

A benchmarking framework was added 6 months ago in #6733, but there are still no (non-silly) benchmarks added to it. Would be nice if someone added some benchmarks, so that it is easier to see that claimed performance improvements really are such. Some ideas (some easy, some more involved):

Script validation (done in #8873)
CCoinsDBView caching
Coins database
Mempool (done in #8873)
EncodeBase58 performance see #7656 (done in #8107)
Wallet: Coin selection see #7823 (done in #8873)
SHA256 throughput and other cryptographic hashes (done in #8039)
LockedMemoryPool random allocation/deallocation performance (when #8753 merged)

laanwj added the label Tests on Apr 15, 2016

laanwj added the label Easy to implement on Apr 15, 2016

laanwj commented at 2:17 pm on April 26, 2016: member

#7934 adds a benchmark for rolling bloom filter (thanks @sipa).

rodentrabies commented at 9:11 pm on April 30, 2016: contributor

Are those benchmarks supposed to cover all use cases (for example, EncodeBase58 for char arrays and std::strings, as well as for private/public keys, pubkey/script hashes etc), or just one or two cases?

gmaxwell commented at 9:15 pm on April 30, 2016: contributor

@yurizhykin It should be whatever is “useful”… if we don’t ever use something in a way where performance matters, it’s less likely to be useful.

rodentrabies commented at 9:24 pm on April 30, 2016: contributor

@gmaxwell Ok, so I will select most used cases for each item from the list above, and adjust it based on comments for the PR, if any.

laanwj commented at 11:26 am on May 2, 2016: member

Right, I just listed some examples, if you want to be thorough it makes sense to profile (using e.g. valgrind) first and see where the bottlenecks are during various modes and operations and RPC commands. I for example never expected EncodeBase58 to be a bottleneck for anything but it turned out to be.

(for example, EncodeBase58 for char arrays and std::strings, as well as for private/public keys, pubkey/script hashes etc)

I think you should benchmark at the level of abstraction of the function. EncodeBase58 for encoding strings would be enough in this case. Size the strings to be around the size of the typical data encoded.

rodentrabies commented at 11:49 am on May 2, 2016: contributor

Great, it sound like a nice guideline, thanks.

laanwj referenced this in commit 1d580e0153 on May 10, 2016

laanwj referenced this in commit 65bc96444d on May 10, 2016

laanwj referenced this in commit 4153366030 on May 10, 2016

laanwj referenced this in commit 32114dd634 on May 11, 2016

droark commented at 5:24 pm on June 8, 2016: contributor

Hi. I’m happy to write some benchmarks. I could use a bit of guidance, though. Which of the remaining ideas is (arguably) the easiest to implement? I’m tempted to say script validation but I’m still getting to know the nuances of the code.

Thanks.

rodentrabies commented at 10:19 pm on June 8, 2016: contributor

@droark, I think you’re right about script validation. Also, coin selection seems to be not that hard. I started working on profiling approach, but had my uni exams started, so currently just read the codebase myself.

laanwj commented at 6:03 am on June 9, 2016: member

Wallet coin selection is probably the hardest, as you need a wider selection of scenarios, just testing the same one over and over isn’t too useful. Generating random isn’t useful either for measurements. Maybe they could be deterministically generated. Or maybe just hand-pick a few representative ones.

The CCoinsView/CCoinsViewDB ones aren’t too difficult. It’s easy to create a temporary in-memory database (some other tests do this) and get and put values into it. Some database benchmarks could be very straightforward. Replicating the actual usage patterns of the client is hard though, many times micro-benchmarks of the database showed completely different characteristics than e.g. reindex timings. But that’s not a requirement of every benchmark.

Basic mempool benchmarks shouldn’t be too hard either. Just keep pushing transactions into it and see what e.g. the performance of eviction is.

Script validation is also quite straightforward.

ryanofsky referenced this in commit be1bd691af on Oct 3, 2016

ryanofsky referenced this in commit 456848695f on Oct 3, 2016

ryanofsky referenced this in commit c9086edc5c on Oct 4, 2016

laanwj referenced this in commit 18dacf9bd2 on Oct 18, 2016

onurakdemir commented at 6:39 am on January 3, 2018: none

Hi,

Can you point me a bench that I can work on please? It is better if it is hard:)

MarcoFalke referenced this in commit b5e4b9b510 on Jan 22, 2018

laanwj commented at 1:46 pm on January 26, 2018: member

I think all the low-hanging fruit here is covered, it’s unclear to me whether a benchmark of database things would make much sense in this context (how to make a micro-benchmark that even remotely reflects the real workload), so I’m closing this issue.

laanwj closed this on Jan 26, 2018

lateminer referenced this in commit dc38daaf38 on Oct 14, 2018

lateminer referenced this in commit db82c9a2cf on Oct 14, 2018

PastaPastaPasta referenced this in commit e264f0e8b2 on Apr 4, 2020

PastaPastaPasta referenced this in commit a09e1c9b7e on Apr 5, 2020

random-zebra referenced this in commit eb835973d1 on May 27, 2020

random-zebra referenced this in commit 16614ab77a on May 27, 2020

random-zebra referenced this in commit 44421180bf on May 28, 2020

ckti referenced this in commit a7e4127fb3 on Mar 28, 2021

DrahtBot locked this on Sep 8, 2021

Add benchmarks to bench_bitcoin #7883

Add benchmarks to `bench_bitcoin` #7883