Replace current benchmarking framework with nanobench #18011

Replace current benchmarking framework with nanobench

This replaces the current benchmarking framework with nanobench [1], an MIT licensed single-header benchmarking library, of which I am the autor. This has in my opinion several advantages, especially on Linux:

• fast: Running all benchmarks takes ~6 seconds instead of 4m13s on an Intel i7-8700 CPU @ 3.20GHz.

• accurate: I ran e.g. the benchmark for SipHash_32b 10 times and calculate standard deviation / mean = coefficient of variation:

  • 0.57% CV for old benchmarking framework
  • 0.20% CV for nanobench

  So the benchmark results with nanobench seem to vary less than with the old framework.

• It automatically determines runtime based on clock precision, no need to specify number of evaluations.

• measure instructions, cycles, branches, instructions per cycle, branch misses (only Linux, when performance counters are available)

• output in markdown table format.

• Warn about unstable environment (frequency scaling, turbo, ...)

• For better profiling, it is possible to set the environment variable NANOBENCH_ENDLESS to force endless running of a particular benchmark without the need to recompile. This makes it to e.g. run "perf top" and look at hotspots.

Here is an example copy & pasted from the terminal output:

| ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 2.52 | 396,529,415.94 | 0.6% | 25.42 | 8.02 | 3.169 | 0.06 | 0.0% | 0.03 | bench/crypto_hash.cpp RIPEMD160 | 1.87 | 535,161,444.83 | 0.3% | 21.36 | 5.95 | 3.589 | 0.06 | 0.0% | 0.02 | bench/crypto_hash.cpp SHA1 | 3.22 | 310,344,174.79 | 1.1% | 36.80 | 10.22 | 3.601 | 0.09 | 0.0% | 0.04 | bench/crypto_hash.cpp SHA256 | 2.01 | 496,375,796.23 | 0.0% | 18.72 | 6.43 | 2.911 | 0.01 | 1.0% | 0.00 | bench/crypto_hash.cpp SHA256D64_1024 | 7.23 | 138,263,519.35 | 0.1% | 82.66 | 23.11 | 3.577 | 1.63 | 0.1% | 0.00 | bench/crypto_hash.cpp SHA256_32b | 3.04 | 328,780,166.40 | 0.3% | 35.82 | 9.69 | 3.696 | 0.03 | 0.0% | 0.03 | bench/crypto_hash.cpp SHA512

[1] https://github.com/martinus/nanobench

Strong concept ACK! Seems like a big improvement.

Can you comment more on the 6 seconds claim? AFAIK each bench was supposed to target running for 1 second? Is this no longer required to reduce variance?

Secondly -- and separately -- can you comment on how this might impact the need for something like #17375? Can we add better support for benchmarks where we want to run with different scaling params and output each trial to get a sense of the complexity?

I calculate a good number of iterations based on the clock accuracy, then perform these iterations a few times and use the median to get rid of outliers. I found it actually to be more reliable with shorter runs, because there is less chance for random fluctuations to interfer. It is necessary though to disable frequency scaling etc (but this should be done with the old framework too anyways). This can be easily done with e.g pyperf

Concerning #17375, nanobench can estimate complexity, but it requires a bit of code change: https://github.com/martinus/nanobench/blob/master/docs/reference.md#asymptotic-complexity

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #19377 (bench: Add OrphanTxPool benchmark by hebasto)
• #19326 (Simplify hash.h interface using Spans by sipa)
• #19280 (Verify the block filter hash when reading from disk. by pstratem)
• #19181 (Add ASM optimizations for MuHash3072 by fjahr)
• #19145 (Add hash_type MUHASH for gettxoutsetinfo by fjahr)
• #19055 (Add MuHash3072 implementation by fjahr)
• #18815 (bench: Add logging benchmark by MarcoFalke)
• #18731 (refactor: Make CCheckQueue RAII-styled by hebasto)
• #18710 (Add local thread pool to CCheckQueue by hebasto)
• #18354 (Use shared pointers only in validation interface by bvbfan)
• #18261 (Erlay: bandwidth-efficient transaction relay protocol by naumenkogs)
• #18014 (lib: Optimizing siphash implementation by elichai)
• #17526 (Use Single Random Draw In addition to knapsack as coin selection fallback by achow101)
• #17331 (Use effective values throughout coin selection by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

The lint check currently fails with this error:

fatal: bad revision '8b138526b5dc...488d538cbf6f'

I believe the reason is some key verification check at the end of ci/lint/06_script.sh, but I can't really say why this is failing

Concept ACK

The travis failure is

The locale dependent function std::to_string(...) appears to be used:

src/bench/nanobench.h:            auto sysCpu = "/sys/devices/system/cpu/cpu" + std::to_string(id);

src/bench/nanobench.h:                warnings.emplace_back("CPU frequency scaling enabled: CPU " + std::to_string(id) + " between " +

Unnecessary locale dependence can cause bugs that are very

tricky to isolate and fix. Please avoid using locale dependent

functions if possible.

Advice not applicable in this specific case? Add an exception

by updating the ignore list in test/lint/lint-locale-dependence.sh

^---- failure generated from test/lint/lint-locale-dependence.sh

Would it be easy to hack in csv output that is somewhat similar to the existing output? The markdown table looks a little tricky to parse programmatically (though it could be done). For example, bitcoinperf (https://github.com/chaincodelabs/bitcoinperf) currently relies on this format.

The locale dependent function std::to_string(...) appears to be used:

Ah, ok I'll fix this

Would it be easy to hack in csv output that is somewhat similar to the existing output?

I think it should be easy, in nanobench I already have CSV & JSON output format using mustache-like templates, so it's possible to create a custom format. I have not exposed this feature yet though in this PR.

Strong concept ACK @martinus, thanks for your great contributions! Please keep them coming :)

Does this library also does memory clobbers and barriers? (like google's DoNotOptimize[0], ClobberMemory[1], or Rust's black_box[2])

[0] https://github.com/google/benchmark/blob/master/include/benchmark/benchmark.h#L307 [1] https://github.com/google/benchmark/blob/master/include/benchmark/benchmark.h#L326 [2] https://doc.rust-lang.org/std/hint/fn.black_box.html

Does this library also does memory clobbers and barriers? (like google's DoNotOptimize[0], ClobberMemory[1], or Rust's black_box[2])

I currently have doNotOptimizeAway, which is based on folly's benchmark. I think folly's version is based on google benchmark. I have not added doNotOptimizeAway calls in the PR because I did not want to modify each benchmark too much

I don't have clobberMemory yet, because I've never used it... What I'm also doing is I force the run(...) to be noinline to prevent some optimizations.

Is it possible to make the nanobench include a submodule (like secp256k1 or univalue) so that it's easier for us to pull in updates from upstream? If you plan on adding new features to nanobench, that should help streamline review potentially. If you think that there will be Bitcoin specific changes made to the header that you wouldn't want to upstream, then I would leave it as you've done.

Closed #17375 in favor of nanobench. When you have time would love assistance in making the asymptotic test introduced there nanobench compatible.

Being able to run asymptotic tests on the code is going to be a huge help with advocating for mempool policy changes in the future (e.g., loosening descendants limit) once the epoch mempool work is completed. This also impacts other parts of the code (e.g., wallet) where right now we don't have good insight into if we introduce a regression. I think the curve fitting is a bit less useful because we do care about the constant factors too (e.g., if we're a O(n log n) v.s. O(n) but c > log n for max n), but it's quite nifty nonetheless.

Is it possible to make the nanobench include a submodule (like secp256k1 or univalue)

I think it should be possible, I need to read up how git-subtree works... I prefer if nanobench stays generic, and try to implement bitcoin's requirement in a generic way so it's usable by others too. So no separate repository if possible.

When you have time would love assistance in making the asymptotic test introduced there nanobench compatible.

Sure, I'll have a look at #17375 when I have time! I can't predict how soon this is though.

Haven't looked at the library itself yet, but Concept ACK on replacing the current framework. (I really dislike it) Personally I also would've been fine with dynamically linking against google's benchmarking library (https://github.com/google/benchmark)

Personally I also would've been fine with dynamically linking against google's benchmarking library (https://github.com/google/benchmark)

I don't think google benchmark is viable here. It's a large dependency, and you also need to use the gtest framework for this. It would be quite a big change.

Concept ACK

We discussed nanobench today in the IRC Meeting. There's seems to be general agreement that this is a nice idea, and that the current bench framework isn't perfect.

Our current bench framework is actually based on Google's, and I think most people are opposed to linking google's whole thing.

With respect to the question of if to subtree or not: let's ignore that for now, copied in is fine, and we can deal with that in the future if we require changes to nanobench or if there are new features in nanobench we want to incorporate.

There's some concern about maintaining compatibility with existing tools. I think given that the output is fundamentally different from before (no longer reporting iterations and things like that) we can't have perfect parity. But perhaps we could:

1. "backport" on top of the last few releases (no new release) so that we have a bit more history to compare with
2. Add a compatibility mode which emits something similar to the previous output with NaN subsituted where nanobench has no equivalent value.
3. Ignore compatibility, but do support output into a nice machine-readable format.

I think most people would be satisfied with 3, as 2 can be done a script on 3's output and 1 can be done if someone has the itch for it.

Thanks for the summary! I just read through the log here and think I can add a few clarifications:

I think if we can do a cursory check it's not actually malware

It's not malware, not sure how I can help here :) I've created nanobench because I was annoyed at how difficult other benchmarking frameworks were to integrate into existing codebase because I don't like google test.

For example, a lot of tools rely on the output format of the current bench framework

I have templating support in nanobench, and I can relatively easily add another output format that resembles the current output format closely. I need to do some improvements to the available data in nanobench, then I can use a template liks this to produce practically the same output as before:

# Benchmark, evals, iterations, total, min, max, median
{{#benchmarks}} {{name}}, {{num_measurements}}, {{total_iters}}, {{total_runtime}}, {{min}}, {{max}}, {{median}}
{{/benchmarks}}

Then I can e.g. print the markdown tables to stdout, and create e.g. benchmarkresults.csv along with it based on the template format.

<jeremyrubin> I beleive nanobench autodetects variance or something

Google benchmark is quite simple: it has a fixed runtime that it wants to achieve, then finds out the number of iterations it needs to do to get there, then it measures the time for that.

In nanobench I try to be a bit smarter: I find out the clocks accuracy first, and base the target runtime on that. Since clocks are nowadays very accurate (18ns or so on my machine), I can easily perform the measurement multiple times and use the median to get rid of outliers.

The fast runtimes gives very repeatable measurements for stuff that's deterministic (e.g. SHA hashing). There I believe nanobench has a clear advantage over all other bencharking frameworks that I've tried.

When the code under test has fluctuations (e.g. because it allocates stuff, or has some randomness or lots of cache misses / branch misspredictions in it), nanobench's runtime measurement probably isn't better than google benchmark. In that case it helps to also show the numbers for branch misses and retired instruction count to get a better feeling.

I've rebased & pushed a big update to the code. In addition to the markdown output, I also generate a file benchmarkresults.csv which has practically the same content as the output had previously. This should can be used by any tools that rely on the benchmark output. On my computer, the file has this output:

# Benchmark, evals, iterations, total, min, max, median
AssembleBlock, 11, 1, 0.006106585, 0.000538949, 0.000643127, 0.000545681
Base58CheckEncode, 11, 9.81818181818182, 0.000240226, 2.06e-06, 3.02745454545455e-06, 2.07709090909091e-06
Base58Decode, 11, 27.2727272727273, 0.000245003, 8.11689655172414e-07, 8.1968e-07, 8.166e-07
Base58Encode, 11, 18.5454545454545, 0.000244574, 1.19615e-06, 1.2115e-06, 1.19763157894737e-06
Bech32Decode, 11, 42.3636363636364, 0.000250479, 4.27466666666667e-07, 1.32740740740741e-06, 4.35666666666667e-07
Bech32Encode, 11, 34.6363636363636, 0.000248104, 6.29258064516129e-07, 8.0896875e-07, 6.36314285714286e-07
BenchLockedPool, 11, 167.363636363636, 0.000264967, 1.08922651933702e-07, 1.73769662921348e-07, 1.46138888888889e-07
BenchTimeDeprecated, 11, 4587.81818181818, 0.00024425, 4.38138264341248e-09, 5.01115472009915e-09, 5.0051077059738e-09
BenchTimeMillis, 11, 237.363636363636, 0.00024483, 9.26184738955823e-08, 9.43307086614173e-08, 9.3963963963964e-08
BenchTimeMillisSys, 11, 243.272727272727, 0.000244799, 9.1088122605364e-08, 9.19113924050633e-08, 9.15038461538462e-08
BenchTimeMock, 11, 10562.9090909091, 0.000246826, 2.08633681343622e-09, 2.29533626901521e-09, 2.08746447742343e-09
BlockToJsonVerbose, 11, 1, 0.812951704, 0.072167457, 0.085656596, 0.072707134
CCheckQueueSpeedPrevectorJob, 11, 11.1818181818182, 0.199323679, 0.00153634016666667, 0.00172734963636364, 0.0016258129
CCoinsCaching, 11, 36.5454545454545, 0.000227408, 5.422e-07, 7.69351351351351e-07, 5.44605263157895e-07
CHACHA20_1MB, 11, 1, 0.023085774, 0.00201953, 0.002240195, 0.002090533
CHACHA20_256BYTES, 11, 44.5454545454545, 0.000245398, 4.99659574468085e-07, 5.02047619047619e-07, 5.00717391304348e-07
CHACHA20_64BYTES, 11, 168.363636363636, 0.000245097, 1.32019736842105e-07, 1.32545454545455e-07, 1.32360759493671e-07
CHACHA20_POLY1305_AEAD_1MB_ENCRYPT_DECRYPT, 11, 1, 0.063083902, 0.005614908, 0.006009997, 0.005679771
CHACHA20_POLY1305_AEAD_1MB_ONLY_ENCRYPT, 11, 1, 0.031591654, 0.002799809, 0.003083797, 0.002855794
CHACHA20_POLY1305_AEAD_256BYTES_ENCRYPT_DECRYPT, 11, 11.2727272727273, 0.000236947, 1.89941666666667e-06, 1.92533333333333e-06, 1.90981818181818e-06
CHACHA20_POLY1305_AEAD_256BYTES_ONLY_ENCRYPT, 11, 21.5454545454545, 0.000233245, 9.55045454545455e-07, 1.23718181818182e-06, 9.5747619047619e-07
CHACHA20_POLY1305_AEAD_64BYTES_ENCRYPT_DECRYPT, 11, 25, 0.000247072, 8.81333333333333e-07, 1.015125e-06, 8.87961538461539e-07
CHACHA20_POLY1305_AEAD_64BYTES_ONLY_ENCRYPT, 11, 48.9090909090909, 0.00024602, 4.42068181818182e-07, 5.63777777777778e-07, 4.45867924528302e-07
ComplexMemPool, 11, 1, 3.457592325, 0.313054487, 0.316239363, 0.313548617
ConstructGCSFilter, 11, 1, 0.018953542, 0.001667658, 0.001879675, 0.001676587
DeserializeAndCheckBlockTest, 11, 1, 0.068399757, 0.006003952, 0.006412799, 0.00618906
DeserializeBlockTest, 11, 1, 0.057322626, 0.005100356, 0.00547949, 0.005164525
DuplicateInputs, 11, 1, 0.082094071, 0.007329521, 0.007526674, 0.007472722
FastRandom_1bit, 11, 14773.0909090909, 0.000239545, 1.46252213259886e-09, 1.47889590295829e-09, 1.47590446579989e-09
FastRandom_32bit, 11, 2285.45454545455, 0.000242346, 9.26413255360624e-09, 1.2139653815893e-08, 9.36415362731152e-09
HASH_1MB, 11, 1, 0.037448639, 0.00333361, 0.003531469, 0.003387306
HASH_256BYTES, 11, 17.2727272727273, 0.000244883, 1.28611764705882e-06, 1.291e-06, 1.28872222222222e-06
HASH_64BYTES, 11, 32.8181818181818, 0.000244595, 6.75885714285714e-07, 6.79625e-07, 6.77514285714286e-07
MatchGCSFilter, 11, 1, 0.000303108, 2.6787e-05, 3.0751e-05, 2.7047e-05
MempoolEviction, 11, 1, 0.00035951, 2.6216e-05, 4.4312e-05, 3.0276e-05
MerkleRoot, 11, 1, 0.013989533, 0.001220381, 0.001462655, 0.001242791
POLY1305_1MB, 11, 1, 0.008822394, 0.000778289, 0.000909028, 0.000782461
POLY1305_256BYTES, 11, 106.272727272727, 0.000243963, 2.07696428571429e-07, 2.09357142857143e-07, 2.088e-07
POLY1305_64BYTES, 11, 328.636363636364, 0.000248855, 6.70127795527157e-08, 8.63612040133779e-08, 6.72832369942197e-08
PrevectorClearNontrivial, 11, 791.454545454545, 0.000372666, 2.59770491803279e-08, 1.94881395348837e-07, 2.60037735849057e-08
PrevectorClearTrivial, 11, 2630.54545454545, 0.000244958, 8.46212395795578e-09, 8.47009966777409e-09, 8.46505271378368e-09
PrevectorDeserializeNontrivial, 11, 1, 0.001192481, 0.000105425, 0.000123912, 0.00010568
PrevectorDeserializeTrivial, 11, 2, 0.000255826, 1.1333e-05, 1.2048e-05, 1.1635e-05
PrevectorDestructorNontrivial, 11, 415.272727272727, 0.000358251, 5.15516483516484e-08, 3.43633971291866e-07, 5.15764966740576e-08
PrevectorDestructorTrivial, 11, 1590.36363636364, 0.000237761, 1.31619407687461e-08, 1.39810235767683e-08, 1.3519e-08
PrevectorResizeNontrivial, 11, 811.272727272727, 0.000273009, 2.46290155440415e-08, 8.05017709563164e-08, 2.46548295454545e-08
PrevectorResizeTrivial, 11, 2536.54545454545, 0.000244927, 8.77334809892949e-09, 8.78255578093306e-09, 8.77719528178244e-09
RIPEMD160, 11, 1, 0.028423992, 0.002516573, 0.002757947, 0.00255517
RollingBloom, 11, 44.3636363636364, 0.000245064, 4.99872340425532e-07, 5.05348837209302e-07, 5.02531914893617e-07
RollingBloomReset, 11, 1, 0.000696645, 6.2348e-05, 6.9429e-05, 6.2588e-05
RpcMempool, 11, 1, 0.121633853, 0.010798163, 0.011604045, 0.010929157
SHA1, 11, 1, 0.021057291, 0.001862186, 0.002103509, 0.001878588
SHA256, 11, 1, 0.035631576, 0.00317911, 0.003383292, 0.003233048
SHA256D64_1024, 11, 1, 0.001464103, 0.000132135, 0.000138764, 0.000132205
SHA256_32b, 11, 95.6363636363636, 0.000244411, 2.31903225806452e-07, 2.32772727272727e-07, 2.32415841584158e-07
SHA512, 11, 1, 0.034134625, 0.00303295, 0.003282536, 0.003086899
SipHash_32b, 11, 776.181818181818, 0.000244754, 2.85349500713267e-08, 2.88179581795818e-08, 2.86630872483221e-08
Trig, 11, 4313.81818181818, 0.000238146, 5.01395348837209e-09, 5.03433333333333e-09, 5.01748251748252e-09
VerifyScriptBench, 11, 1, 0.001860283, 0.000155127, 0.000299556, 0.000155384

Note that "number of iterations" is now a double value, because in nanobench I automatically determine the number of iterations, and the value is the average number of iterations over the 11 evaluations. (so e.g. Base58CheckEncode as 11 evaluations and 9.81818181818182 iterations, so 11*9.818 = 108 iterations in total)

utACK 83a7839

Verified that only benchmarks are effected, checked that the high level design seems reasonable & an improvement over what we do presently.

In bf5ae5e I've added some support for asymptotes. I hope that's somewhat similar to what you did in #17375, @JeremyRubin?

Usage is e.g. like this:

./bench_bitcoin -filter=ComplexMemPool -asymptote=25,50,100,200,400,600,800

This runs the benchmark ComplexMemPool several times but with different complexityN settings. The benchmark can extract that number and use it accordingly. Here, it's used for childTxs. The output is this:

| complexityN | ns/op | op/s | err% | ins/op | cyc/op | IPC | total | benchmark |------------:|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|----------:|:---------- | 25 | 1,064,241.00 | 939.64 | 1.4% | 3,960,279.00 | 2,829,708.00 | 1.400 | 0.01 | ComplexMemPool | 50 | 1,579,530.00 | 633.10 | 1.0% | 6,231,810.00 | 4,412,674.00 | 1.412 | 0.02 | ComplexMemPool | 100 | 4,022,774.00 | 248.58 | 0.6% | 16,544,406.00 | 11,889,535.00 | 1.392 | 0.04 | ComplexMemPool | 200 | 15,390,986.00 | 64.97 | 0.2% | 63,904,254.00 | 47,731,705.00 | 1.339 | 0.17 | ComplexMemPool | 400 | 69,394,711.00 | 14.41 | 0.1% | 272,602,461.00 | 219,014,691.00 | 1.245 | 0.76 | ComplexMemPool | 600 | 168,977,165.00 | 5.92 | 0.1% | 639,108,082.00 | 535,316,887.00 | 1.194 | 1.86 | ComplexMemPool | 800 | 310,109,077.00 | 3.22 | 0.1% |1,149,134,246.00 | 984,620,812.00 | 1.167 | 3.41 | ComplexMemPool

| coefficient | err% | complexity |--------------:|-------:|------------ | 4.78486e-07 | 4.5% | O(n^2) | 6.38557e-10 | 21.7% | O(n^3) | 3.42338e-05 | 38.0% | O(n log n) | 0.000313914 | 46.9% | O(n) | 0.0129823 | 114.4% | O(log n) | 0.0815055 | 133.8% | O(1)

The best fitting curve is O(n^2), so the algorithm seems to scale quadratic with childTxs in the range 25 to 800.

utACK bf5ae5e

bravo!

ACK https://github.com/bitcoin/bitcoin/pull/18011/commits/c2e924fc046110eb7ac5ab7bf19cfaf6daf1c44b

Tested and light code review ACK c2e924fc04 -- built/tests/ran benches/tested the options (-?/-help, -asymptote=, -filter=, -list)... modulo a few minor comments below.

Runs very quickly.

Various bench runs for info and comparison:

• bench output of this PR e.g. ./src/bench/bench_bitcoin
• benchmarkresults.csv (current bench output format)
• Bench output on master to compare with the csv
• Bench with asymptotes e.g. ./src/bench/bench_bitcoin -filter=ComplexMemPool -asymptote=25,50,100,200,400,600,800

New, simplified options help:

bitcoin ((HEAD detached at origin/pr/18011))$ ./src/bench/bench_bitcoin -?
Options:

  -?
       Print this help message and exit

  -asymptote=n1,n2,n3,...
       Test asymptotic growth of the runtime of an algorithm, if supported by
       the benchmark

  -filter=<regex>
       Regular expression filter to select benchmark by name (default: .*)

  -list
       List benchmarks without executing them. Can be combined with -scaling
       and -filter

One potential concern is how this will affect the usefulness of long-term benchmarking projects like https://github.com/chaincodelabs/bitcoinperf, e.g. https://bitcoinperf.com (which seems to be down?)

I've pushed a few updates:

• updated nanobench.h with a (bit) faster RNG, and explicit constructors so extended-lint-all.h doesn't complain any more
• add command line options -output_csv to enable creation of legacy CSV file
• add command line optoin -output_json to create a big JSON with all data

Not sure if I should squash all the changes into a single commit or leave the separately?

Here is console output, .CSV file, and .json file of one run (click RAW):

https://gist.github.com/martinus/d5e596b7802199737ef38399ef749d51

Perhaps the gArgs.AddArg("-list") fix and the "comment nits" changes ought to be in 79fd93a where they are first changed rather than 51b83e1 and 9d6eb72. Alternatively, with 2 ACKs it may have been good to rebase with no changes to preserve existing review and do the rest in a follow-up PR (I'm not sure).

tACK 9d6eb7207a9baa78ce9c6231e517717a598c8d33

Went over the code changes in core, very briefly looked over nanobench.h, compared the benchmark results with the results before, and it looks ok (there are some differences but in benchmarks with high variance)

And I really like that it's human readable now :)

rebased

I tried running this locally, and it seems I am getting varying results:

mac-mini:bitcoin-core marco$ git log -1 && make -j 9 && ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
commit a841d1e25b1b26b6381f36e14307c2549a79edb4 (HEAD)
Author: Martin Ankerl <martin.ankerl@gmail.com>
Date:   Sun Mar 8 16:20:55 2020 +0100

    remove unnecessary include util/time.h
Making all in src
Making all in doc/man
make[1]: Nothing to be done for `all'.
make[1]: Nothing to be done for `all-am'.
Warning, results might be unstable:
* NDEBUG not defined, assert() macros are evaluated

Recommendations
* Make sure you compile for Release

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|          303,061.00 |            3,299.67 |    2.6% |      0.00 | `VerifyNestedIfScript`
mac-mini:bitcoin-core marco$ git log -1 && make -j 9 && ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
commit a841d1e25b1b26b6381f36e14307c2549a79edb4 (HEAD)
Author: Martin Ankerl <martin.ankerl@gmail.com>
Date:   Sun Mar 8 16:20:55 2020 +0100

    remove unnecessary include util/time.h
Making all in src
Making all in doc/man
make[1]: Nothing to be done for `all'.
make[1]: Nothing to be done for `all-am'.
Warning, results might be unstable:
* NDEBUG not defined, assert() macros are evaluated

Recommendations
* Make sure you compile for Release

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|          155,083.00 |            6,448.16 |    9.4% |      0.00 | :wavy_dash: `VerifyNestedIfScript` (Unstable with ~1.0 iters. Increase `minEpochIterations` to e.g. 10)

Notably:

• Consecutive runs are off by a factor of 2
• There is a warning Warning, results might be unstable: NDEBUG not defined, assert() macros are evaluated, but I think it is impossible to compile Bitcoin Core with assert disabled
• There is a warning from the framework itself: :wavy_dash: VerifyNestedIfScript(Unstable with ~1.0 iters. IncreaseminEpochIterations to e.g. 10)

I've finally found the time to rebased my branch with a major update of nanobench, and updated the new benchmarks too. @MarcoFalke, I am pretty sure your instability comes from CPU frequency scaling / turbo mode. For nanobench to be able to have accurate runtime results it needs a CPU locked to fixed frequency. Under Linux I print warnings & suggestions if I detect this, but I don't have such a logic in place for other operating systems (yet).

When I run the benchmark VerifyNestedIfScript a few times I get these results:

| ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 63,447.00 | 15,761.19 | 0.9% | 677,018.00 | 201,628.00 | 3.358 | 143,829.00 | 0.2% | 0.00 | VerifyNestedIfScript | 63,094.00 | 15,849.37 | 0.8% | 677,018.00 | 201,495.00 | 3.360 | 143,829.00 | 0.2% | 0.00 | VerifyNestedIfScript | 63,548.00 | 15,736.14 | 0.7% | 677,018.00 | 201,894.00 | 3.353 | 143,829.00 | 0.2% | 0.00 | VerifyNestedIfScript | 63,340.00 | 15,787.81 | 0.9% | 677,018.00 | 201,894.00 | 3.353 | 143,829.00 | 0.2% | 0.00 | VerifyNestedIfScript | 63,288.00 | 15,800.78 | 1.0% | 677,018.00 | 201,096.00 | 3.367 | 143,829.00 | 0.2% | 0.00 | VerifyNestedIfScript | 63,261.00 | 15,807.53 | 0.9% | 677,018.00 | 201,894.00 | 3.353 | 143,829.00 | 0.2% | 0.00 | VerifyNestedIfScript | 63,309.00 | 15,795.54 | 0.7% | 677,018.00 | 202,027.00 | 3.351 | 143,829.00 | 0.2% | 0.00 | VerifyNestedIfScript

So it is a very stable benchmark for me, with locked frequency scaling. When I don't lock the CPU my results too fluctuate, and the benchmark prints these warnings (I've removed the NDEBUG warning):

Warning, results might be unstable:
* CPU frequency scaling enabled: CPU 0 between 800.0 and 4,600.0 MHz
* CPU governor is 'powersave' but should be 'performance'
* Turbo is enabled, CPU frequency will fluctuate

Recommendations
* Use 'pyperf system tune' before benchmarking. See https://github.com/vstinner/pyperf

utack 78c312c. @MarcoFalke can you repeat your benchmark after running pyperf system tune?

@martinus While going thru this in July 2nd, 2020's meeting, I believe people were wondering what the support is like for non x86 architectures. Would it fail to compile? Have limited functionality? Or fail at runtime?

FWIW, I could compile and run this PR (as merged on master) on RV64. It doesn't seem there is any compatibility issue as was implied by the travis run.

Concept ACK.

@martinus While going thru this in July 2nd, 2020's meeting, I believe people were wondering what the support is like for non x86 architectures. Would it fail to compile? Have limited functionality? Or fail at runtime?

The CPU statistics like instructions, cycles, branch misspredictions are only available on Linux through perf events. But it should compile on any platform with C++11 support, then I'm only relying on the std::chrono timers.

I have now a relatively comprehensive documentation of nanobench available here: https://nanobench.ankerl.com/

Concept ACK

So far I have built the PR and run some tests without any problems.

ACK 78c312c983255e15fc274de2368a2ec13ce81cbf

I've merged this because there was unanimous agreement that we want the new benchmark framework and it works as expected here (as well as for @fjahr and @JeremyRubin ). If @MarcoFalke's issue, is still a problem we should look into it, please open a github issue for it.

👍 per my ACK a few months ago #18011#pullrequestreview-366872789

Thanks for merging! If anyone has any questions or issue with nanobench please notify me

@martinus Mind looking into #18710 if it has any performance regression on supported platforms?

@martinus How to conditionally skip a benchmark in nanobench framework (wrt #19710 (comment) and #19710 (comment))?

You can use -filter to specify a regular expression for which tests to run

You can use -filter to specify a regular expression for which tests to run

I mean in the code, e.g., skip CCheckQueueSpeedPrevectorJob if GetNumCores() < 2

You can use -filter to specify a regular expression for which tests to run

I mean in the code, e.g., skip CCheckQueueSpeedPrevectorJob if GetNumCores() < 2

Ah, of course

Before benchmark is run you can do a check and then simply return, e.g. like so:

static void CCheckQueueSpeedPrevectorJob(benchmark::Bench& bench)
{
    if (GetNumCores() < 2) {
        return;
    }

But that needs a little update in bench.cpp, because then the benchmark doesn't have any results:

        if (!bench.results().empty()) {
            benchmarkResults.push_back(bench.results().back());
        }

@martinus Thanks! I've submitted a commit (ce3e6a7cb21d1aa455513970846e1f70c01472a4) in #19710.

Not sure if this is caused by this pull, but some benchmarks changed performance quite significantly:

• BenchLockedPool went down almost 100% to approx 0 compared to before: https://codespeed.bitcoinperf.com/timeline/#/?exe=3,4,2,1,5&base=1+23&ben=micro.clang.BenchLockedPool&env=1&revs=200&equid=off&quarts=on&extr=on
• All the prevector ones as well (except the serialization one): https://codespeed.bitcoinperf.com/timeline/#/?exe=3,4,2,1,5&base=1+23&ben=micro.clang.PrevectorClearNontrivial&env=1&revs=200&equid=off&quarts=on&extr=on

Is this due to compiler optimizations or something else?

Funnily the trig dummy bench went up by ~100%: https://codespeed.bitcoinperf.com/timeline/#/?exe=3,4,2,1,5&base=1+23&ben=micro.clang.Trig&env=1&revs=200&equid=off&quarts=on&extr=on