bench: Add SHA256 implementation specific benchmarks

hebasto commented at 3:17 pm on May 8, 2023: member

On the master branch, only the best available SHA256 implementation is being benchmarked. This PR makes bench_bitcoin benchmark all SHA256 implementations that are available on the system.

For example:

on Linux:

 0$ ./src/bench/bench_bitcoin -filter=SHA.*
 1Using the 'x86_shani(1way,2way)' SHA256 implementation
 2
 3|             ns/byte |              byte/s |    err% |     total | benchmark
 4|--------------------:|--------------------:|--------:|----------:|:----------
 5|                1.00 |    1,002,545,462.93 |    0.4% |      0.01 | `SHA1`
 6|                2.91 |      344,117,991.18 |    0.1% |      0.03 | `SHA256 using the 'standard' SHA256 implementation`
 7|                2.21 |      453,081,794.40 |    0.1% |      0.02 | `SHA256 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
 8|                2.21 |      453,396,506.58 |    0.1% |      0.02 | `SHA256 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
 9|                0.53 |    1,870,520,687.49 |    0.1% |      0.01 | `SHA256 using the 'x86_shani(1way,2way)' SHA256 implementation`
10|                7.90 |      126,627,134.33 |    0.0% |      0.01 | `SHA256D64_1024 using the 'standard' SHA256 implementation`
11|                3.94 |      253,850,206.07 |    0.0% |      0.01 | `SHA256D64_1024 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
12|                1.40 |      716,247,553.38 |    0.4% |      0.01 | `SHA256D64_1024 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
13|                1.26 |      792,706,270.13 |    0.9% |      0.01 | `SHA256D64_1024 using the 'x86_shani(1way,2way)' SHA256 implementation`
14|                6.75 |      148,172,097.64 |    0.2% |      0.01 | `SHA256_32b using the 'standard' SHA256 implementation`
15|                4.90 |      204,156,289.96 |    0.1% |      0.01 | `SHA256_32b using the 'sse4(1way),sse41(4way)' SHA256 implementation`
16|                4.90 |      204,101,274.22 |    0.1% |      0.01 | `SHA256_32b using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
17|                1.70 |      589,052,595.35 |    0.4% |      0.01 | `SHA256_32b using the 'x86_shani(1way,2way)' SHA256 implementation`
18|                2.21 |      453,441,736.14 |    1.0% |      0.02 | `SHA3_256_1M`
19|                1.92 |      521,807,101.48 |    1.0% |      0.02 | `SHA512`

on macOS (M1):

 0% ./src/bench/bench_bitcoin -filter=SHA.\*
 1Using the 'arm_shani(1way,2way)' SHA256 implementation
 2
 3|             ns/byte |              byte/s |    err% |     total | benchmark
 4|--------------------:|--------------------:|--------:|----------:|:----------
 5|                1.36 |      737,644,274.00 |    0.6% |      0.02 | `SHA1`
 6|                3.08 |      324,556,777.15 |    0.2% |      0.03 | `SHA256 using the 'standard' SHA256 implementation`
 7|                0.45 |    2,198,104,135.18 |    0.3% |      0.01 | `SHA256 using the 'arm_shani(1way,2way)' SHA256 implementation`
 8|                8.84 |      113,131,299.18 |    0.0% |      0.01 | `SHA256D64_1024 using the 'standard' SHA256 implementation`
 9|                0.94 |    1,059,406,239.36 |    0.0% |      0.01 | `SHA256D64_1024 using the 'arm_shani(1way,2way)' SHA256 implementation`
10|                6.17 |      162,050,659.51 |    0.2% |      0.01 | `SHA256_32b using the 'standard' SHA256 implementation`
11|                1.15 |      866,637,155.98 |    0.0% |      0.01 | `SHA256_32b using the 'arm_shani(1way,2way)' SHA256 implementation`
12|                1.69 |      592,636,491.59 |    0.2% |      0.02 | `SHA3_256_1M`
13|                1.89 |      528,785,775.66 |    0.0% |      0.02 | `SHA512`

Found it useful, while working on #24773.

DrahtBot commented at 3:17 pm on May 8, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	sipa, martinus, MarcoFalke
Concept ACK	luke-jr

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

DrahtBot added the label Tests on May 8, 2023

MarcoFalke commented at 3:20 pm on May 8, 2023: member

What about benchmarking always all that are available on the system?

hebasto commented at 3:22 pm on May 8, 2023: member

What about benchmarking all that are available on the system?

You mean, as separated benchmarks? Without -sha-implementation option?

hebasto force-pushed on May 9, 2023

hebasto renamed this:
~~bench: Add `-sha-implementation` command-line option~~
bench: Benchmark all `SHA256` implementations that are available on the system
on May 9, 2023

hebasto force-pushed on May 9, 2023

hebasto commented at 4:22 pm on May 9, 2023: member

Reworked according to @MarcoFalke’s suggestion.

Friendly ping @martinus :)

in src/bench/bench.cpp:133 in e5ab4a84ce outdated

146-            func(bench);
147-        } else {
148-            for (auto n : args.asymptote) {
149-                bench.complexityN(n);
150-                func(bench);
151+        if (properties.iterate_sha256_implementations) {

martinus commented at 12:49 pm on May 11, 2023:

Hm, I don’t really like that all of that SHA256 specific code is becoming part of the benchmark framework. Wouldn’t it be sufficient to just adapt the SHA benchmark in crypto_hash.cpp so that there is a separate test for each of the implementations? There could be a separate benchmark for each value of the enum UseImplementation.

fanquake commented at 12:06 pm on May 16, 2023:

Hm, I don’t really like that all of that SHA256 specific code is becoming part of the benchmark framework.

Yea. Concept NACK on the current approach.

hebasto commented at 4:28 pm on May 16, 2023:

@martinus

Thanks! Reworked.

in src/bench/bench_bitcoin.cpp:123 in e5ab4a84ce outdated

118@@ -118,6 +119,9 @@ int main(int argc, char** argv)
119         return EXIT_SUCCESS;
120     }
121 
122+    std::string sha256_algo = SHA256AutoDetect();
123+    tfm::format(std::cout, "Using the '%s' SHA256 implementation\n", sha256_algo);

martinus commented at 12:50 pm on May 11, 2023:

This always prints a SHA log even when no SHA related benchmark is run

hebasto commented at 4:29 pm on May 16, 2023:

Thanks! That change has been dropped.

hebasto force-pushed on May 16, 2023

hebasto commented at 4:28 pm on May 16, 2023: member

Reworked according to @martinus’s comments.

hebasto renamed this:
~~bench: Benchmark all `SHA256` implementations that are available on the system~~
bench: Add SHA256 implementation specific benchmarks
on May 16, 2023

hebasto force-pushed on May 18, 2023

hebasto commented at 11:06 am on May 18, 2023: member

Rebased on top of the merged #27615 as it was relevant.

luke-jr commented at 4:50 pm on June 23, 2023: member

Concept ACK

achow101 requested review from sipa on Sep 20, 2023

achow101 requested review from josibake on Sep 20, 2023

Add ability to specify SHA256 implementation for benchmark purposes 5f72417176

bench: Add SHA256 implementation specific benchmarks ce6df7df9b

hebasto force-pushed on Sep 20, 2023

hebasto commented at 8:26 pm on September 20, 2023: member

Rebased for fresh CI simultaneously with #24773 that is based on this PR.

DrahtBot added the label CI failed on Sep 21, 2023

DrahtBot removed the label CI failed on Sep 21, 2023

sipa commented at 3:12 pm on September 29, 2023: member

ACK ce6df7df9bab2405cfe7d6e382f5682cf30de476

My output (Ryzen 5950X):

ns/byte	byte/s	err%	total	benchmark
1.29	776,698,179.43	0.3%	0.01	`SHA256D64_1024_AVX2 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
0.67	1,492,914,034.43	0.1%	0.01	`SHA256D64_1024_SHANI using the 'x86_shani(1way,2way)' SHA256 implementation`
3.45	290,161,790.35	0.3%	0.01	`SHA256D64_1024_SSE4 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
6.96	143,610,645.45	0.2%	0.01	`SHA256D64_1024_STANDARD using the 'standard' SHA256 implementation`
4.51	221,653,283.35	0.1%	0.01	`SHA256_32b_AVX2 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
1.25	799,826,326.07	0.2%	0.01	`SHA256_32b_SHANI using the 'x86_shani(1way,2way)' SHA256 implementation`
4.54	220,365,747.83	0.1%	0.01	`SHA256_32b_SSE4 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
5.77	173,257,910.52	0.1%	0.01	`SHA256_32b_STANDARD using the 'standard' SHA256 implementation`
2.01	498,088,336.96	0.2%	0.02	`SHA256_AVX2 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
0.44	2,288,837,568.06	0.4%	0.01	`SHA256_SHANI using the 'x86_shani(1way,2way)' SHA256 implementation`
1.99	501,633,066.45	0.5%	0.02	`SHA256_SSE4 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
2.50	400,467,265.21	0.2%	0.03	`SHA256_STANDARD using the 'standard' SHA256 implementation`

hebasto commented at 11:23 am on October 2, 2023: member

@martinus

Mind taking another look at this PR?

martinus commented at 5:52 pm on October 2, 2023: contributor

ACK ce6df7df9bab2405cfe7d6e382f5682cf30de476. I would have created a helper function in the test to avoid the code duplication for each test, but that’s just me nitpicking. Here are results from my Ryzen 7950X, with ./src/bench/bench_bitcoin -filter="SHA256.*" -min-time=1000:

ns/byte	byte/s	err%	ins/byte	bra/byte	miss%	total	benchmark
1.08	922,442,557.83	0.0%	18.39	0.01	0.3%	1.07	`SHA256D64_1024_AVX2 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
0.59	1,696,233,239.51	0.1%	6.60	0.03	0.1%	1.10	`SHA256D64_1024_SHANI using the 'x86_shani(1way,2way)' SHA256 implementation`
2.94	339,763,959.72	0.0%	51.98	0.02	0.1%	1.10	`SHA256D64_1024_SSE4 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
5.79	172,580,960.50	0.1%	132.83	0.06	0.1%	1.10	`SHA256D64_1024_STANDARD using the 'standard' SHA256 implementation`
3.82	261,777,995.08	0.1%	82.72	1.69	0.0%	1.10	`SHA256_32b_AVX2 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
1.10	912,755,631.26	0.0%	14.09	1.53	0.0%	1.10	`SHA256_32b_SHANI using the 'x86_shani(1way,2way)' SHA256 implementation`
3.82	261,543,369.48	0.1%	82.72	1.69	0.0%	1.10	`SHA256_32b_SSE4 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
4.10	244,096,616.72	0.2%	112.94	1.53	0.0%	1.10	`SHA256_32b_STANDARD using the 'standard' SHA256 implementation`
1.75	572,561,916.85	0.1%	36.80	0.09	0.0%	1.10	`SHA256_AVX2 using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
0.38	2,639,518,254.68	0.0%	2.56	0.02	0.0%	1.10	`SHA256_SHANI using the 'x86_shani(1way,2way)' SHA256 implementation`
1.75	572,393,692.61	0.2%	36.80	0.09	0.0%	1.10	`SHA256_SSE4 using the 'sse4(1way),sse41(4way)' SHA256 implementation`
1.82	548,938,688.20	0.1%	51.91	0.02	0.0%	1.10	`SHA256_STANDARD using the 'standard' SHA256 implementation`

martinus approved

hebasto requested review from MarcoFalke on Oct 4, 2023

MarcoFalke commented at 1:53 pm on October 4, 2023: member

review ACK ce6df7df9bab2405cfe7d6e382f5682cf30de476 🏵

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK ce6df7df9bab2405cfe7d6e382f5682cf30de476 🏵
3GsNRGSniZroT9qJHZFN7xnASprLA9scBcdwWOood4TqNZX7pGDvXRFuAI0buRTCEQMVV9OT5ZVnJMymXcWDPCw==

DrahtBot removed review request from MarcoFalke on Oct 4, 2023

fanquake merged this on Oct 4, 2023

fanquake closed this on Oct 4, 2023

hebasto deleted the branch on Oct 4, 2023

bench: Add SHA256 implementation specific benchmarks #27598

Reviews

Conflicts