Simple benchmarking framework

gavinandresen commented at 2:45 pm on September 28, 2015: contributor

Benchmarking framework, loosely based on google’s micro-benchmarking library (https://github.com/google/benchmark)

Wny not use the Google Benchmark framework? Because adding Even More Dependencies isn’t worth it. If we get a dozen or three benchmarks and need nanosecond-accurate timings of threaded code then switching to the full-blown Google Benchmark library should be considered.

The benchmark framework is hard-coded to run each benchmark for one wall-clock second, and then spits out .csv-format timing information to stdout. It is left as an exercise for later (or maybe never) to add command-line arguments to specify which benchmark(s) to run, how long to run them for, how to format results, etc etc etc. Again, see the Google Benchmark framework for where that might end up.

See src/bench/Examples.cpp for a sanity-test benchmark that just benchmarks ‘sleep 100 milliseconds’ and a benchmark for math.h’s sin() function.

To compile and run benchmarks:

0  configure --enable-bench  ...other configure options
1  cd src; make bench

Sample output:

0Benchmark,count,min,max,average
1Sleep100ms,10,0.10048,0.104789,0.103251
2Trig,37748735,0,4.76837e-07,2.71684e-08

in src/bench/bench.h: in 830fec1135 outdated

14+ * Usage:
15+
16+static void CODE_TO_TIME(benchmark::State& state)
17+{
18+    ... do any setup needed...
19+    while (state.KeepRunning()) {

laanwj commented at 3:01 pm on September 28, 2015:

This works well as long as ‘stuff you want to time’ does not run too quickly - otherwise the state.KeepRunning(), which does a gettimeofday syscall, with the implied latency of that, will dominate. (an alternative to calling the time function in the benchmark loop that is commonly used, is to first time a fixed # of runs of the loop, then from the result compute a number of iterations so that it runs for approximately the allotted time)

laanwj added the label Tests on Sep 28, 2015

laanwj commented at 3:42 pm on September 28, 2015: member

Concept ACK

in src/Makefile.bench.include: in 830fec1135 outdated

13+bench_bench_bitcoin_LDADD = $(LIBBITCOIN_SERVER) $(LIBBITCOIN_CLI) $(LIBBITCOIN_COMMON) $(LIBBITCOIN_UTIL) $(LIBBITCOIN_CRYPTO) $(LIBBITCOIN_UNIVALUE) $(LIBLEVELDB) $(LIBMEMENV) \
14+  $(BOOST_LIBS) $(LIBSECP256K1)
15+if ENABLE_WALLET
16+bench_bench_bitcoin_LDADD += $(LIBBITCOIN_WALLET)
17+endif
18+bench_bench_bitcoin_LDADD += $(LIBBITCOIN_CONSENSUS) $(BDB_LIBS) $(SSL_LIBS) $(CRYPTO_LIBS) $(MINIUPNPC_LIBS) $(CURL_LIBS)

jonasschnelli commented at 3:46 pm on September 28, 2015:

rm $(CURL_LIBS)? This seems to have be sneaked in from XT.

jonasschnelli commented at 4:16 pm on September 28, 2015: contributor

Concept ACK

gavinandresen force-pushed on Sep 28, 2015

gavinandresen force-pushed on Sep 29, 2015

gavinandresen commented at 4:53 pm on September 29, 2015: contributor

Removed the $(CURL_LIBS) noticed by @jonasschnelli.

And added the new libevent and libzmq dependencies.

I also decided to make it disabled by default, so you have to configure –enable-bench, so it doesn’t slow down compile times for everybody. @laanwj : RE: gettimeofday overhead and timing things that are very fast: timing how fast gettimeofday() is and then warning the developer if their benchmarking code runs into that overhead (maybe suggest that they loop 1,000 times inside the KeepRunning loop) would be another way of preventing that problem. Feel free to improve, I’m planning on using this to benchmark things that take microseconds to run, not nanoseconds.

jgarzik commented at 5:58 pm on September 29, 2015: contributor

Disagree on default-disabled.

Principle: Code that is not built by default bitrots rapidly, is not tested as much, receives less maintenance in general by random developers updating the code.

gavinandresen commented at 6:06 pm on September 29, 2015: contributor

@jgarzik : I KNEW you were going to say that…

Would a Travis configuration that runs benchmarks be good enough to prevent code rot? Compiling and linking Yet Another Binary that approximately nobody will run (unless you’re actively working on optimizing something) tickles my “features shouldn’t cost anything unless you’re using them” sensibility.

paveljanik commented at 8:16 pm on September 29, 2015: contributor

This looks a bit strange in the benchmarking tool:

0SelectParams(CBaseChainParams::MAIN);

in src/Makefile.bench.include: in 7500d59463 outdated

0@@ -0,0 +1,45 @@
1+bin_PROGRAMS += bench/bench_bitcoin
2+BENCH_SRCDIR = bench
3+BENCH_BINARY=bench/bench_bitcoin$(EXEEXT)

paveljanik commented at 8:18 pm on September 29, 2015:

Spaces around =?

in src/Makefile.bench.include: in 7500d59463 outdated

0@@ -0,0 +1,45 @@
1+bin_PROGRAMS += bench/bench_bitcoin
2+BENCH_SRCDIR = bench
3+BENCH_BINARY=bench/bench_bitcoin$(EXEEXT)
4+
5+
6+bench_bench_bitcoin_SOURCES =\

paveljanik commented at 8:19 pm on September 29, 2015:

Space after =

paveljanik commented at 8:33 pm on September 29, 2015: contributor

I think it should stay disabled by default as it is now, because benchmarking will be done on purpose by developers only.

Concept ACK. I like it.

This is a way to benchmark parts of code. Now we should identify parts that need some love and prepare benchmarks for them specifically. With this framework, people can report speed fixes with proper benchmark code which can be tested before and after applying the fix. @laanwj @gavinandresen I think that both ways to benchmark are useful and maybe we should add the second example benchmark there to illustrate.

sipa commented at 8:40 pm on September 29, 2015: member

Compile by default but not run, seems fine to me.

gavinandresen force-pushed on Sep 29, 2015

gavinandresen commented at 8:48 pm on September 29, 2015: contributor

Fixed @paveljanik ’s spaces nits, and removed the unneeded SelectParams inherited from the unit test code.

Data on the “compile by default or not” decision:

It takes 8 seconds on my machine to compile and link the benchmarking code, single-processor, ccache cleared. 5 seconds doing parallel make. Under 2 seconds with a ‘hot’ ccache.

gavinandresen force-pushed on Sep 29, 2015

gavinandresen commented at 9:33 pm on September 29, 2015: contributor

Damn you @laanwj, you inspired me to spend another hour implementing support for really fast benchmarks….

Picked @paveljanik ’s CreateNewBlock nit (good catch, I copied this from a CreateNewBlock benchmark I haven’t finished). I decided to rename MilliSleep.cpp to Examples.cpp, and added a “see how fast sin() runs” benchmark to test the really-fast-benchmark support (works nicely).

And switched back to compile-by-default, which seems to be the consensus.

I’m really and truly done tweaking this now, assuming Travis is happy.

btcdrak commented at 10:08 pm on September 29, 2015: contributor

Concept ACK

jgarzik commented at 10:16 pm on September 29, 2015: contributor

Yes, to be specific, I meant “compile by default but not run”

dcousens commented at 11:03 pm on September 29, 2015: contributor

concept ACK

0xC2 commented at 1:17 am on September 30, 2015: none

Tested.

in src/bench/bench_bitcoin.cpp: in b3a2e05cd4 outdated

0@@ -0,0 +1,30 @@
1+// Copyright (c) 2015 The Bitcoin Core developers
2+// Distributed under the MIT software license, see the accompanying
3+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
4+
5+#include "bench.h"
6+
7+#include "chainparams.h"

paveljanik commented at 6:29 am on September 30, 2015:

You do not need chainparams.h now.

in src/bench/bench_bitcoin.cpp: in b3a2e05cd4 outdated

18+extern void noui_connect();
19+
20+int
21+main(int argc, char** argv)
22+{
23+    ECC_Start();

paveljanik commented at 6:30 am on September 30, 2015:

ECC_*() are here probably because you are testing ConnectBlock()? This should be in the specific benchmark init section IMO, not here.

paveljanik commented at 6:30 am on September 30, 2015: contributor

ACK

Simple benchmarking framework

Benchmarking framework, loosely based on google's micro-benchmarking
library (https://github.com/google/benchmark)

Wny not use the Google Benchmark framework? Because adding Even More Dependencies
isn't worth it. If we get a dozen or three benchmarks and need nanosecond-accurate
timings of threaded code then switching to the full-blown Google Benchmark library
should be considered.

The benchmark framework is hard-coded to run each benchmark for one wall-clock second,
and then spits out .csv-format timing information to stdout. It is left as an
exercise for later (or maybe never) to add command-line arguments to specify which
benchmark(s) to run, how long to run them for, how to format results, etc etc etc.
Again, see the Google Benchmark framework for where that might end up.

See src/bench/MilliSleep.cpp for a sanity-test benchmark that just benchmarks
'sleep 100 milliseconds.'

To compile and run benchmarks:
  cd src; make bench

Sample output:

Benchmark,count,min,max,average
Sleep100ms,10,0.101854,0.105059,0.103881

535ed9223d

Support very-fast-running benchmarks

Avoid calling gettimeofday every time through the benchmarking loop, by keeping
track of how long each loop takes and doubling the number of iterations done
between time checks when they take less than 1/16'th of the total elapsed time.

7072c544b5

gavinandresen force-pushed on Sep 30, 2015

in configure.ac: in 7072c544b5

90@@ -91,6 +91,11 @@ AC_ARG_ENABLE(tests,
91     [use_tests=$enableval],
92     [use_tests=yes])
93 
94+AC_ARG_ENABLE(bench,
95+    AS_HELP_STRING([--enable-bench],[compile benchmarks (default is yes)]),

paveljanik commented at 6:12 pm on October 2, 2015:

If enabled by default, please make it --disable-bench (see #6739).

in src/Makefile.am: in 7072c544b5

60@@ -61,6 +61,7 @@ endif
61 
62 bin_PROGRAMS =
63 TESTS =
64+BENCHMARKS =

paveljanik commented at 6:17 pm on October 2, 2015:

BTW - why do you have this variable here? Can it be deleted?

laanwj commented at 2:31 pm on October 6, 2015: member

ACK

laanwj merged this on Oct 6, 2015

laanwj closed this on Oct 6, 2015

laanwj referenced this in commit b7d78fd0bd on Oct 6, 2015

zkbot referenced this in commit aa225ebb0b on Jan 24, 2020

zkbot referenced this in commit 74ff73abab on Jan 24, 2020

furszy referenced this in commit 4ed15cc69d on Jun 8, 2020

DrahtBot locked this on Sep 8, 2021

Simple benchmarking framework #6733