WIP NOMERGE [bench] gitian builds for OP

MarcoFalke commented at 9:09 pm on March 14, 2020: member

Get gitian builds for two revisions, with and without #16902

MarcoFalke added the label Tests on Mar 14, 2020

MarcoFalke added the label Needs gitian build on Mar 14, 2020

DrahtBot commented at 10:29 pm on March 14, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#18571 (fuzz: Disable debug log file by MarcoFalke)
#18562 (ci: Run unit tests sequential once by MarcoFalke)
#18422 ([consensus] MOVEONLY: Move single-sig checking EvalScript code to EvalChecksig by jnewbery)
#18014 (lib: Optimizing siphash implementation by elichai)
#18000 (Index for UTXO Set Statistics by fjahr)
#17977 ([WIP] Implement BIP 340-342 validation (Schnorr/taproot/tapscript) by sipa)
#17526 (Use Single Random Draw In addition to knapsack as coin selection fallback by achow101)
#17331 (Use effective values throughout coin selection by achow101)
#13525 (policy: Report reason inputs are nonstandard from AreInputsStandard by Empact)
#13062 (Make script interpreter independent from storage type CScript by sipa)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot commented at 8:38 pm on March 15, 2020: member

Gitian builds

File	commit 67dfd18f4401986063e22c79d4d7da61f15b5cd4(master)	commit d622e3d68c68469cd73c29e05f84770a8f9702b1(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`2a8f73aeb5c13752...`	`986d23604353f7c7...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`ff068319a9e46aa4...`	`f5b8416ea3d9955e...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`a707188d3822b963...`	`54c22b01a20afaef...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`57c2972bae73eb6d...`	`b947af8230f49fa7...`
bitcoin-0.19.99-osx-unsigned.dmg	`61f4694b65f2fcfa...`	`05dd7289b0877914...`
bitcoin-0.19.99-osx64.tar.gz	`f9e740828f11f0b0...`	`15f8ae161f1a66c2...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`99dcdb32f68d93ac...`	`289f18c2619c76a1...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`ce7287fb7d6291d0...`	`93944b7ea4d0bb85...`
bitcoin-0.19.99-win64-debug.zip	`d34b8c96ab2b5b7c...`	`1e976977348cddc5...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`d4d3ada682e5f876...`	`4973394339b02606...`
bitcoin-0.19.99-win64.zip	`02d580f9a33cd7bb...`	`ecf1effa9024cb85...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`ae647e3f30c5c537...`	`4a2269a1b83f53b0...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`e44a3ac81cfc8243...`	`6942e1ac36d83d34...`
bitcoin-0.19.99.tar.gz	`1294e333e7db1345...`	`3e0e55948245659d...`
bitcoin-core-linux-0.20-res.yml	`d6aafec283db2408...`	`dda7bc29014ae53d...`
bitcoin-core-osx-0.20-res.yml	`e59580777ac8bda4...`	`41c60e64ceebb759...`
bitcoin-core-win-0.20-res.yml	`98b37da0cc4b66fc...`	`f1fb48c5c7cf6d69...`
linux-build.log	`7963474358aebaf2...`	`5e90906b4ff4ec2a...`
osx-build.log	`ed80e00b4ca0fb17...`	`5552b2c0af786db4...`
win-build.log	`c7310c9b58c22cdb...`	`f472e1b0aa5d22cc...`
bitcoin-core-linux-0.20-res.yml.diff		`7bc9acfbd2303a8e...`
bitcoin-core-osx-0.20-res.yml.diff		`1638b9afb2f81d96...`
bitcoin-core-win-0.20-res.yml.diff		`971c9b0a6f922817...`
linux-build.log.diff		`ced631e32143e460...`
osx-build.log.diff		`4da9bb70630566ef...`
win-build.log.diff		`448d3d7e5c4a04de...`

DrahtBot removed the label Needs gitian build on Mar 15, 2020

MarcoFalke commented at 6:45 pm on March 16, 2020: member

Tested on ARM:

Current master (with optimizations):

0VerifyNestedIfScript, 5, 100, 0.25091, 0.000499568, 0.000508809, 0.000500318

This pull (without the opt):

0VerifyNestedIfScript, 5, 100, 0.505166, 0.00100926, 0.0010135, 0.00100959

MarcoFalke commented at 6:46 pm on March 16, 2020: member

@fjahr , would be nice if you can test this on macOs to compare against #16902 (comment)

The bench_bitcoin binary is contained in the bitcoin-0.19.99-osx64.tar.gz archive.

MarcoFalke commented at 9:21 pm on March 18, 2020: member

@fjahr Alternatively, it would be nice to know what optimization flags you used to compile #16902 (comment)

fjahr commented at 9:25 pm on March 18, 2020: member

@fjahr Alternatively, it would be nice to know what optimization flags you used to compile #16902 (comment)

Just the defaults of ./configure but I am also compiling this PR right now and will post results soon.

MarcoFalke commented at 9:41 pm on March 18, 2020: member

compiling this PR right now

The pr was mostly to get two gitian builds for osx (bitcoin-0.19.99-osx64.tar.gz). They are using the release compile flags and a cross compiler.

fjahr commented at 9:58 pm on March 18, 2020: member

The pr was mostly to get two gitian builds for osx (bitcoin-0.19.99-osx64.tar.gz). They are using the release compile flags and a cross compiler.

Sorry, misread at first, here are results for bitcoin-0.19.99-osx64.tar.gz.

Master:

0$ bin/bench_bitcoin -filter=VerifyNestedIfScript -evals=50
1# Benchmark, evals, iterations, total, min, max, median
2VerifyNestedIfScript, 50, 100, 1.88564, 0.000253594, 0.00063017, 0.000371099
3VerifyNestedIfScript, 50, 100, 1.89325, 0.000237669, 0.000570061, 0.000369836
4VerifyNestedIfScript, 50, 100, 1.94918, 0.000245143, 0.000753701, 0.000398808

Master and this PR:

0$ bin/bench_bitcoin -filter=VerifyNestedIfScript -evals=50
1# Benchmark, evals, iterations, total, min, max, median
2VerifyNestedIfScript, 50, 100, 1.47942, 0.000233691, 0.000447447, 0.000285911
3VerifyNestedIfScript, 50, 100, 1.52134, 0.000252425, 0.000506466, 0.000287488
4VerifyNestedIfScript, 50, 100, 1.49092, 0.000252439, 0.000458893, 0.000286676

MarcoFalke commented at 10:00 pm on March 18, 2020: member

Hmmm :thinking: Now it seems to even slow it down by a bit. (Master should be the faster one)

fjahr commented at 10:08 pm on March 18, 2020: member

Hmmm 🤔 Now it seems to even slow it down by a bit. (Master should be the faster one)

Yeah, I had some direct exchanges with @ajtowns as well after my results on #16902 but we could not figure it out. Let me know if you want me to do more specific testing or investigate my system aside from the information I shared so far.

MarcoFalke commented at 11:49 pm on March 18, 2020: member

The hardware you are using and the version of the operating system would help to check if this can be reproduced or if it is only happening on your end.

fjahr commented at 0:06 am on March 19, 2020: member

I think the most relevant info is:

MacBook Pro (13-inch, 2016)
Intel(R) Core(TM) i5-6287U CPU @ 3.10GHz
macOS 10.15.3 (latest stable)

MarcoFalke commented at 6:32 pm on March 19, 2020: member

Testing on a mac-mini shows the same result: Slower on master (the optimized version) and faster with the old code:

0mac-mini:~ marco$ ./master/bitcoin-0.19.99/bin/bench_bitcoin -filter=VerifyNestedIfScript -evals=50 
1# Benchmark, evals, iterations, total, min, max, median
2VerifyNestedIfScript, 50, 100, 0.884273, 0.000143609, 0.000295676, 0.000151077
3
4mac-mini:~ marco$ ./old/bitcoin-0.19.99/bin/bench_bitcoin -filter=VerifyNestedIfScript -evals=50 
5# Benchmark, evals, iterations, total, min, max, median
6VerifyNestedIfScript, 50, 100, 0.782465, 0.000150363, 0.000179858, 0.000153508

ajtowns commented at 5:16 am on March 24, 2020: member

Testing on a mac-mini shows the same result:

Super weird. Maybe try setting nOpCount = 0; prior to the ++nOpCount > MAX_OPS_PER_SCRIPT check in script/interpreter.cpp and check to see what the behaviour is with much larger numbers of OP_IFs or OP_1s in bench/verify_script.cpp? If your script is {1 IF}*x NOP*y ENDIF*x 1 your runtime should be O(3x+y) in master, but O(3x+y+xy) with the revert, which ought to be noticible at the some point. (You need to change the 1000 OP_1’s to OP_NOP’s or you’ll also violate the stack.size() + altstack.size() > MAX_STACK_SIZE test)

Replace current benchmarking framework with nanobench

This replaces the current benchmarking framework with nanobench [1], an
MIT licensed single-header benchmarking library, of which I am the
autor. This has in my opinion several advantages, especially on Linux:

* fast: Running all benchmarks takes ~6 seconds instead of 4m13s on
  an Intel i7-8700 CPU @ 3.20GHz.

* accurate: I ran e.g. the benchmark for SipHash_32b 10 times and
  calculate standard deviation / mean = coefficient of variation:

  * 0.57% CV for old benchmarking framework
  * 0.20% CV for nanobench

  So the benchmark results with nanobench seem to vary less than with
  the old framework.

* It automatically determines runtime based on clock precision, no need
  to specify number of evaluations.

* measure instructions, cycles, branches, instructions per cycle,
  branch misses (only Linux, when performance counters are available)

* output in markdown table format.

* Warn about unstable environment (frequency scaling, turbo, ...)

* For better profiling, it is possible to set the environment variable
  NANOBENCH_ENDLESS to force endless running of a particular benchmark
  without the need to recompile. This makes it to e.g. run "perf top"
  and look at hotspots.

Here is an example copy & pasted from the terminal output:

|             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|                2.52 |      396,529,415.94 |    0.6% |           25.42 |            8.02 |  3.169 |           0.06 |    0.0% |      0.03 | `bench/crypto_hash.cpp RIPEMD160`
|                1.87 |      535,161,444.83 |    0.3% |           21.36 |            5.95 |  3.589 |           0.06 |    0.0% |      0.02 | `bench/crypto_hash.cpp SHA1`
|                3.22 |      310,344,174.79 |    1.1% |           36.80 |           10.22 |  3.601 |           0.09 |    0.0% |      0.04 | `bench/crypto_hash.cpp SHA256`
|                2.01 |      496,375,796.23 |    0.0% |           18.72 |            6.43 |  2.911 |           0.01 |    1.0% |      0.00 | `bench/crypto_hash.cpp SHA256D64_1024`
|                7.23 |      138,263,519.35 |    0.1% |           82.66 |           23.11 |  3.577 |           1.63 |    0.1% |      0.00 | `bench/crypto_hash.cpp SHA256_32b`
|                3.04 |      328,780,166.40 |    0.3% |           35.82 |            9.69 |  3.696 |           0.03 |    0.0% |      0.03 | `bench/crypto_hash.cpp SHA512`

[1] https://github.com/martinus/nanobench

4a7e92139b

Do not write benchmarkresults.csv when calling benchmark with -list e88e8a0d69

Adds support for asymptotes

This adds support to calculate asymptotic complexity of a benchmark.
This is similar to #17375, but currently only one asymptote is
supported, and I have added support in the benchmark `ComplexMemPool`
as an example.

Usage is e.g. like this:

```
./bench_bitcoin -filter=ComplexMemPool -asymptote=25,50,100,200,400,600,800
```

This runs the benchmark `ComplexMemPool` several times but with
different complexityN settings. The benchmark can extract that number
and use it accordingly. Here, it's used for `childTxs`. The output is
this:

| complexityN |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |     total | benchmark
|------------:|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|----------:|:----------
|          25 |        1,064,241.00 |              939.64 |    1.4% |    3,960,279.00 |    2,829,708.00 |  1.400 |      0.01 | `ComplexMemPool`
|          50 |        1,579,530.00 |              633.10 |    1.0% |    6,231,810.00 |    4,412,674.00 |  1.412 |      0.02 | `ComplexMemPool`
|         100 |        4,022,774.00 |              248.58 |    0.6% |   16,544,406.00 |   11,889,535.00 |  1.392 |      0.04 | `ComplexMemPool`
|         200 |       15,390,986.00 |               64.97 |    0.2% |   63,904,254.00 |   47,731,705.00 |  1.339 |      0.17 | `ComplexMemPool`
|         400 |       69,394,711.00 |               14.41 |    0.1% |  272,602,461.00 |  219,014,691.00 |  1.245 |      0.76 | `ComplexMemPool`
|         600 |      168,977,165.00 |                5.92 |    0.1% |  639,108,082.00 |  535,316,887.00 |  1.194 |      1.86 | `ComplexMemPool`
|         800 |      310,109,077.00 |                3.22 |    0.1% |1,149,134,246.00 |  984,620,812.00 |  1.167 |      3.41 | `ComplexMemPool`

|   coefficient |   err% | complexity
|--------------:|-------:|------------
|   4.78486e-07 |   4.5% | O(n^2)
|   6.38557e-10 |  21.7% | O(n^3)
|   3.42338e-05 |  38.0% | O(n log n)
|   0.000313914 |  46.9% | O(n)
|     0.0129823 | 114.4% | O(log n)
|     0.0815055 | 133.8% | O(1)

The best fitting curve is O(n^2), so the algorithm seems to scale
quadratic with `childTxs` in the range 25 to 800.

4c33547849

Don't using namespace ankerl::nanobench

Instead of `using namespace ankerl::nanobench`, just pull in ankerl::nanobench::Bench. This is the only class needed in the benchmarks anyways.

4f82f05d80

Add option -output_csv and -output_json

-output_cvs produces the (legacy) CSV format, and -output_json produces a big JSON file all the data that has been gathered.

Put all args into a separate struct to prevent argument explosion of BenchRunner::RunAll.

9eea19f82f

comment nits 17feb04349

remove unnecessary include util/time.h a841d1e25b

Revert "Implement O(1) OP_IF/NOTIF/ELSE/ENDIF logic"

This reverts commit e6e622e5a0e22c2ac1b50b96af818e412d67ac54.

cf2eb9040f

Revert "[refactor] interpreter: define interface for vfExec"

This reverts commit d0e8f4d5d8ddaccb37f98b7989fb944081e41ab8.

991966fa9b

MarcoFalke force-pushed on Mar 28, 2020

MarcoFalke added the label Needs gitian build on Mar 28, 2020

DrahtBot commented at 11:21 pm on March 29, 2020: member

Gitian builds

File	commit 27a82d347eddfe1cde0005f8a2174274ab6efe9f(master)	commit 71ede6f8080660f3ead41b985a9d06d77d22970c(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`ad1ec03659569853...`	`34f6241dae2cef42...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`5089797fd9a583bf...`	`b29448e70155ad6b...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`60300e24c9ca405b...`	`8c439396a2b24055...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`0ee586254245075a...`	`98f88af872f43bfc...`
bitcoin-0.19.99-osx-unsigned.dmg	`a33b4524614c9080...`	`0bce971031f6bda0...`
bitcoin-0.19.99-osx64.tar.gz	`b71b775bfd6f4f41...`	`9f5eda508f2049cb...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`bcf14b2564bc16f9...`	`d2d1183d1de05cb6...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`06ae9547b089b0c8...`	`2914d1724b71ab94...`
bitcoin-0.19.99-win64-debug.zip	`628c9ecca66a231e...`	`1540956088d44971...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`dbbb8776c6fb98f0...`	`ffba6abbcc2da1ff...`
bitcoin-0.19.99-win64.zip	`07a8a14315a79561...`	`56784a521238676d...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`e87328e592187294...`	`d362486c3a717f1a...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`00867ee2b9cba802...`	`bfc3ad62c6533581...`
bitcoin-0.19.99.tar.gz	`a767bd5af81be41d...`	`51a364765a9efe51...`
bitcoin-core-linux-0.20-res.yml	`b73d8723ec1f874c...`	`838c6d30b7c3d0c8...`
bitcoin-core-osx-0.20-res.yml	`261a24bb1f6fa1f9...`	`8928d2ca8272dcad...`
bitcoin-core-win-0.20-res.yml	`61556603241be0cf...`	`6f70cf660064b109...`
linux-build.log	`23a8d94233423c08...`	`7f59f0be1d229241...`
osx-build.log	`cd2ccb82220021bb...`	`8447cf2eb010ea91...`
win-build.log	`f3299bed9bb746ba...`	`45e8e423bf5bff6a...`
bitcoin-core-linux-0.20-res.yml.diff		`bda903084067b66a...`
bitcoin-core-osx-0.20-res.yml.diff		`5970450160fce12e...`
bitcoin-core-win-0.20-res.yml.diff		`f9da8367709b21af...`
linux-build.log.diff		`059f44d5da8b25a9...`
osx-build.log.diff		`513600f41413e92a...`
win-build.log.diff		`0ec42e479df98d0f...`

DrahtBot removed the label Needs gitian build on Mar 29, 2020

DrahtBot added the label Needs rebase on Apr 9, 2020

DrahtBot commented at 1:37 pm on April 9, 2020: member

🐙 This pull request conflicts with the target branch and needs rebase.

MarcoFalke commented at 2:51 pm on April 9, 2020: member

With the new bench framwork I get this:

This pull with the last two commits dropped:

 0mac-mini:bitcoin-core marco$ ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
 1Warning, results might be unstable:
 2* NDEBUG not defined, assert() macros are evaluated
 3
 4Recommendations
 5* Make sure you compile for Release
 6
 7|               ns/op |                op/s |    err% |     total | benchmark
 8|--------------------:|--------------------:|--------:|----------:|:----------
 9|          154,373.00 |            6,477.82 |    9.4% |      0.00 | :wavy_dash: `VerifyNestedIfScript` (Unstable with ~1.0 iters. Increase `minEpochIterations` to e.g. 10)
10mac-mini:bitcoin-core marco$ ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
11Warning, results might be unstable:
12* NDEBUG not defined, assert() macros are evaluated
13
14Recommendations
15* Make sure you compile for Release
16
17|               ns/op |                op/s |    err% |     total | benchmark
18|--------------------:|--------------------:|--------:|----------:|:----------
19|          141,380.00 |            7,073.14 |    0.5% |      0.00 | `VerifyNestedIfScript`
20mac-mini:bitcoin-core marco$ ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
21Warning, results might be unstable:
22* NDEBUG not defined, assert() macros are evaluated
23
24Recommendations
25* Make sure you compile for Release
26
27|               ns/op |                op/s |    err% |     total | benchmark
28|--------------------:|--------------------:|--------:|----------:|:----------
29|          141,859.00 |            7,049.25 |    0.4% |      0.00 | `VerifyNestedIfScript`

This pull:

 0mac-mini:bitcoin-core marco$ ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
 1Warning, results might be unstable:
 2* NDEBUG not defined, assert() macros are evaluated
 3
 4Recommendations
 5* Make sure you compile for Release
 6
 7|               ns/op |                op/s |    err% |     total | benchmark
 8|--------------------:|--------------------:|--------:|----------:|:----------
 9|          201,531.00 |            4,962.02 |   26.1% |      0.00 | :wavy_dash: `VerifyNestedIfScript` (Unstable with ~1.0 iters. Increase `minEpochIterations` to e.g. 10)
10mac-mini:bitcoin-core marco$ ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
11Warning, results might be unstable:
12* NDEBUG not defined, assert() macros are evaluated
13
14Recommendations
15* Make sure you compile for Release
16
17|               ns/op |                op/s |    err% |     total | benchmark
18|--------------------:|--------------------:|--------:|----------:|:----------
19|          148,760.00 |            6,722.24 |    0.3% |      0.00 | `VerifyNestedIfScript`
20mac-mini:bitcoin-core marco$ ./src/bench/bench_bitcoin --filter=VerifyNestedIfScript
21Warning, results might be unstable:
22* NDEBUG not defined, assert() macros are evaluated
23
24Recommendations
25* Make sure you compile for Release
26
27|               ns/op |                op/s |    err% |     total | benchmark
28|--------------------:|--------------------:|--------:|----------:|:----------
29|          149,385.00 |            6,694.11 |    0.3% |      0.00 | `VerifyNestedIfScript`

MarcoFalke commented at 2:52 pm on April 9, 2020: member

So it does seem to get slightly faster, which might point to a problem with the current bench framework.

MarcoFalke closed this on May 7, 2020

MarcoFalke deleted the branch on May 7, 2020

DrahtBot locked this on Feb 15, 2022

WIP NOMERGE [bench] gitian builds for OP_IF bench #18352

Conflicts

Gitian builds

Gitian builds