bench: use std::chrono rather than gettimeofday #11562

pull theuni wants to merge 2 commits into bitcoin:master from theuni:bench-clock-chrono changing 3 files +38 −29
  1. theuni commented at 9:09 pm on October 25, 2017: member

    gettimeofday has portability issues, see for example #11558.

    Regardless of large-scale clock refactors in the future, I think it’s fine for bench to just use std::chrono itself.

    Note that this may slightly improve bench accuracy and changes the display from tiny floats to nanosecond counts instead.

  2. practicalswift commented at 9:19 pm on October 25, 2017: contributor
    Concept ACK
  3. fanquake added the label Tests on Oct 25, 2017
  4. MarcoFalke commented at 10:33 pm on October 25, 2017: member
    Concept ACK. This is what the google bench does.
  5. fanquake commented at 0:23 am on October 26, 2017: member

    master https://github.com/bitcoin/bitcoin/commit/57ee73990f1ce29916adfd99f93eae1ccea1a43b

     0src/bench/bench_bitcoin
 1#Benchmark,count,min,max,average,min_cycles,max_cycles,average_cycles
 2Base58CheckEncode,294912,0.000003700712114,0.000003791210474,0.000003750498258,12553,12860,12723
 3Base58Decode,1048576,0.000000986561645,0.000001043212251,0.000001009610287,3346,3542,3425
 4Base58Encode,425984,0.000002401517122,0.000002516913810,0.000002463796521,8146,8538,8357
 5BenchLockedPool,640,0.001663804054260,0.001768421381712,0.001725013926625,5644197,5998992,5852070
 6CCheckQueueSpeed,512,0.001896254718304,0.002109408378601,0.001992908306420,6432527,7155741,6760920
 7CCheckQueueSpeedPrevectorJob,448,0.002330623567104,0.002476744353771,0.002359484455415,7906058,8401791,8004012
 8CCoinsCaching,122880,0.000008048693417,0.000008630100638,0.000008307983323,27303,29275,28184
 9CoinSelection,576,0.001785077154636,0.001916944980621,0.001805677182145,6055513,6502807,6125742
10DeserializeAndCheckBlockTest,112,0.008977383375168,0.009968042373657,0.009116992354393,30453379,33814404,30927281
11DeserializeBlockTest,144,0.007569313049316,0.007755517959595,0.007618673973613,25677256,26308849,25846183
12FastRandom_1bit,640,0.001670766621828,0.001709744334221,0.001680362224579,5671193,5799963,5700593
13FastRandom_32bit,112,0.009404510259628,0.009604752063751,0.009498813322612,31902396,32581996,32222551
14MempoolEviction,24576,0.000041037099436,0.000047371606342,0.000042666749020,139209,160698,144746
15PrevectorClear,5120,0.000209093093872,0.000216422602534,0.000211418559775,709305,734155,717233
16PrevectorDestructor,5120,0.000199727714062,0.000203179195523,0.000201328517869,677510,689255,682960
17RIPEMD160,416,0.002455778419971,0.002532064914703,0.002477367910055,8330696,8589435,8404447
18RollingBloom-refresh,1,0.000129,0.000129,0.000129
19RollingBloom-refresh,1,0.000111,0.000111,0.000111
20RollingBloom-refresh,1,0.000129,0.000129,0.000129
21RollingBloom-refresh,1,0.000112,0.000112,0.000112
22RollingBloom-refresh,1,0.000113,0.000113,0.000113
23RollingBloom-refresh,1,0.000112,0.000112,0.000112
24RollingBloom-refresh,1,0.000111,0.000111,0.000111
25RollingBloom-refresh,1,0.000113,0.000113,0.000113
26RollingBloom-refresh,1,0.000111,0.000111,0.000111
27RollingBloom-refresh,1,0.000123,0.000123,0.000123
28RollingBloom-refresh,1,0.000108,0.000108,0.000108
29RollingBloom-refresh,1,0.000127,0.000127,0.000127
30RollingBloom-refresh,1,0.000112,0.000112,0.000112
31RollingBloom-refresh,1,0.000112,0.000112,0.000112
32RollingBloom-refresh,1,0.000113,0.000113,0.000113
33RollingBloom-refresh,1,0.000111,0.000111,0.000111
34RollingBloom-refresh,1,0.000117,0.000117,0.000117
35RollingBloom-refresh,1,0.000112,0.000112,0.000112
36RollingBloom-refresh,1,0.000109,0.000109,0.000109
37RollingBloom-refresh,1,0.00011,0.00011,0.00011
38RollingBloom-refresh,1,0.000108,0.000108,0.000108
39RollingBloom-refresh,1,0.000112,0.000112,0.000112
40RollingBloom-refresh,1,0.000112,0.000112,0.000112
41RollingBloom-refresh,1,0.00011,0.00011,0.00011
42RollingBloom-refresh,1,0.000109,0.000109,0.000109
43RollingBloom-refresh,1,0.000113,0.000113,0.000113
44RollingBloom,1572864,0.000000646410626,0.000000685669875,0.000000676037568,2192,2327,2293
45SHA1,576,0.001821309328079,0.001895938068628,0.001866892394092,6178301,6431563,6333002
46SHA256,352,0.002996556460857,0.003085255622864,0.003031289712949,10165221,10466029,10283598
47SHA256_32b,6,0.215636968612671,0.217754483222961,0.216444651285807,731612744,738682787,734277071
48SHA512,384,0.002636194229126,0.002715311944485,0.002677739908298,8942669,9211179,9083624
49SipHash_32b,28,0.036945462226868,0.039195537567139,0.037574112415314,125329483,132961911,127469598
50Sleep100ms,10,0.102285981178284,0.104399442672729,0.102977490425110,347096871,354150647,349350507
51Trig,67108864,0.000000015336752,0.000000016048432,0.000000015816315,52,54,53
52VerifyScriptBench,6144,0.000164830125868,0.000173246022314,0.000169373194998,559145,587697,574596

    This PR https://github.com/bitcoin/bitcoin/pull/11562/commits/e2dc48c3e5f70c75b762be656e0c9419b2b0208f

     0src/bench/bench_bitcoin
 1#Benchmark,count,min,max,average,min_cycles,max_cycles,average_cycles. Clock precision: nanosecond
 2Base58CheckEncode,262144,3873ns,4485ns,4025ns,13138,15215,13655
 3Base58Decode,983040,1022ns,1089ns,1042ns,3467,3694,3536
 4Base58Encode,393216,2509ns,2694ns,2568ns,8511,9140,8712
 5BenchLockedPool,576,1711427ns,1831078ns,1761460ns,5805626,6211513,5975351
 6CCheckQueueSpeed,576,1894807ns,1987571ns,1943943ns,6427702,6742379,6594382
 7CCheckQueueSpeedPrevectorJob,416,2375311ns,2435702ns,2410998ns,8057696,8262562,8178757
 8CCoinsCaching,122880,8041ns,8954ns,8476ns,27277,30374,28754
 9CoinSelection,576,1883259ns,1920119ns,1896561ns,6388520,6513567,6433648
10DeserializeAndCheckBlockTest,112,9069196ns,10666107ns,9416574ns,30765154,36182297,31943563
11DeserializeBlockTest,128,7655800ns,8043978ns,7823370ns,25970543,27287306,26538987
12FastRandom_1bit,640,1677213ns,1759200ns,1725739ns,5689570,5967687,5854173
13FastRandom_32bit,104,9361699ns,10403821ns,9620188ns,31757436,35292568,32634282
14MempoolEviction,24576,41403ns,43570ns,42427ns,140452,147802,143927
15PrevectorClear,5120,208818ns,234129ns,215746ns,708367,794231,731869
16PrevectorDestructor,4608,217063ns,228702ns,221567ns,736337,775821,751618
17RIPEMD160,416,2491869ns,2678229ns,2566195ns,8453095,9085277,8705227
18RollingBloom-refresh,1,112950,112950,112950
19RollingBloom-refresh,1,109470,109470,109470
20RollingBloom-refresh,1,119208,119208,119208
21RollingBloom-refresh,1,108423,108423,108423
22RollingBloom-refresh,1,110924,110924,110924
23RollingBloom-refresh,1,107718,107718,107718
24RollingBloom-refresh,1,104799,104799,104799
25RollingBloom-refresh,1,107850,107850,107850
26RollingBloom-refresh,1,236540,236540,236540
27RollingBloom-refresh,1,110421,110421,110421
28RollingBloom-refresh,1,107586,107586,107586
29RollingBloom-refresh,1,105012,105012,105012
30RollingBloom-refresh,1,135634,135634,135634
31RollingBloom-refresh,1,121365,121365,121365
32RollingBloom-refresh,1,105733,105733,105733
33RollingBloom-refresh,1,107545,107545,107545
34RollingBloom-refresh,1,107570,107570,107570
35RollingBloom-refresh,1,113778,113778,113778
36RollingBloom-refresh,1,112316,112316,112316
37RollingBloom-refresh,1,107881,107881,107881
38RollingBloom-refresh,1,108146,108146,108146
39RollingBloom-refresh,1,107903,107903,107903
40RollingBloom-refresh,1,110589,110589,110589
41RollingBloom-refresh,1,107555,107555,107555
42RollingBloom,1441792,679ns,742ns,711ns,2305,2518,2414
43SHA1,576,1845302ns,1918492ns,1887662ns,6259766,6508049,6403460
44SHA256,352,3038257ns,3252807ns,3120301ns,10306589,11034401,10584908
45SHA256_32b,6,220640728ns,226463297ns,222662923ns,748472847,768224917,755332837
46SHA512,384,2663500ns,3062115ns,2776882ns,9035312,10387520,9419936
47SipHash_32b,28,36830074ns,39666791ns,38085206ns,124937632,134560472,129195322
48Sleep100ms,10,101757668ns,103232076ns,102764599ns,345189224,350191554,348605289
49Trig,67108864,15ns,16ns,15ns,52,57,53
50VerifyScriptBench,6144,166119ns,174340ns,169195ns,563523,591412,573955
  6. laanwj commented at 2:52 pm on October 26, 2017: member
    Does this also use a monotonic clock? That’s another issue with using plain gettimeofday for benchmarks.
  7. theuni commented at 4:38 pm on October 26, 2017: member
    @laanwj Yes, if the resolution is no worse than the high precision clock. See the last commit: https://github.com/bitcoin/bitcoin/pull/11562/commits/e2dc48c3e5f70c75b762be656e0c9419b2b0208f
  8. laanwj commented at 7:00 pm on October 26, 2017: member

    @laanwj Yes, if the resolution is no worse than the high precision clock. See the last commit: e2dc48c

    Nice! concept ACK

  9. martinus commented at 7:28 am on October 27, 2017: contributor
    I’ve also switched to std::chrono in https://github.com/bitcoin/bitcoin/pull/11517
  10. in src/bench/bench.cpp:38 in e2dc48c3e5 outdated 
    42+benchmark::BenchRunner::RunAll(benchmark::duration elapsedTimeForOne)
43 {
44     perf_init();
45     std::cout << "#Benchmark" << "," << "count" << "," << "min" << "," << "max" << "," << "average" << ","
46-              << "min_cycles" << "," << "max_cycles" << "," << "average_cycles" << "\n";
47+              << "min_cycles" << "," << "max_cycles" << "," << "average_cycles. " << "Clock precision: "
    laanwj commented at 10:21 am on November 6, 2017:
    Please use a ,, to separate here, instead of a dot, otherwise programs will get confused re: CSV parsing.
  11. in src/bench/bench.cpp:109 in e2dc48c3e5 outdated 
    106+    int64_t min_elapsed = std::chrono::duration_cast<std::chrono::nanoseconds>(minTime).count();
107+    int64_t max_elapsed = std::chrono::duration_cast<std::chrono::nanoseconds>(maxTime).count();
108+    int64_t avg_elapsed = std::chrono::duration_cast<std::chrono::nanoseconds>((now-beginTime)/count).count();
109     int64_t averageCycles = (nowCycles-beginCycles)/count;
110-    std::cout << std::fixed << std::setprecision(15) << name << "," << count << "," << minTime << "," << maxTime << "," << average << ","
111+    std::cout << std::fixed << std::setprecision(15) << name << "," << count << "," << min_elapsed << "ns," << max_elapsed << "ns," << avg_elapsed << "ns,"
    laanwj commented at 10:23 am on November 6, 2017:
    Please put the unit id in the row header, not in the data. Adding it to every record complicates parsing (for example, spreadsheets won’t see it as number anymore, and will not be able to compute with it or plot it).
  12. laanwj commented at 10:23 am on November 6, 2017: member
    utACK apart from output format nits.
  13. bench: switch to std::chrono for time measurements 
    std::chrono removes portability issues.

Rather than storing doubles, store the untouched time_points. Then
convert to nanoseconds for display. This allows for maximum precision, while
keeping results comparable between differing hardware/operating systems.

Also, display full nanosecond counts rather than sub-second floats.
    c515d266ec
  14. bench: prefer a steady clock if the resolution is no worse 24a0bddf4a
  15. theuni force-pushed on Nov 7, 2017
  16. theuni commented at 10:20 pm on November 7, 2017: member
    Fixed up formatting as @laanwj suggested. I just dropped the commit which added the clock precision display, as there’s no obvious (to me) way to show it without breaking csv.
  17. laanwj merged this on Nov 8, 2017
  18. laanwj closed this on Nov 8, 2017
  19. laanwj referenced this in commit 5776582b7f on Nov 8, 2017
  20. MarcoFalke commented at 3:53 pm on November 9, 2017: member

    post merge utACK 24a0bddf4ae13d8f1fa0436a547de67dcf6d4f2a

    Looks like a clear improvement.

  21. TheBlueMatt commented at 4:50 pm on November 9, 2017: member
    Postumous utACK 24a0bddf4ae13d8f1fa0436a547de67dcf6d4f2a modulo changes in #11646.
  22. laanwj referenced this in commit 68e021e3a3 on Dec 13, 2017
  23. MarcoFalke referenced this in commit afa9600020 on Aug 13, 2018
  24. PastaPastaPasta referenced this in commit 6d02d0b8e3 on Jul 18, 2019
  25. PastaPastaPasta referenced this in commit 6a18c3645e on Dec 22, 2019
  26. PastaPastaPasta referenced this in commit e6c46f3f02 on Dec 22, 2019
  27. PastaPastaPasta referenced this in commit 4607b9eb3c on Jan 2, 2020
  28. PastaPastaPasta referenced this in commit 4aea4b3f0d on Jan 2, 2020
  29. PastaPastaPasta referenced this in commit 55e7cfd604 on Jan 4, 2020
  30. PastaPastaPasta referenced this in commit 79723ea4da on Jan 4, 2020
  31. PastaPastaPasta referenced this in commit 66f8045fe7 on Jan 12, 2020
  32. PastaPastaPasta referenced this in commit 7610b8699f on Jan 12, 2020
  33. PastaPastaPasta referenced this in commit b8243e7644 on Jan 12, 2020
  34. PastaPastaPasta referenced this in commit 2afd200c33 on Jan 12, 2020
  35. PastaPastaPasta referenced this in commit 896e5a9717 on Jan 12, 2020
  36. PastaPastaPasta referenced this in commit b78253b2f6 on Jan 12, 2020
  37. PastaPastaPasta referenced this in commit d7c8ef3653 on Jan 12, 2020
  38. PastaPastaPasta referenced this in commit e80235ac83 on Jan 12, 2020
  39. PastaPastaPasta referenced this in commit efe3b9e1ca on Jan 12, 2020
  40. PastaPastaPasta referenced this in commit 106aab0c19 on Jan 12, 2020
  41. PastaPastaPasta referenced this in commit 97df283547 on Jan 12, 2020
  42. PastaPastaPasta referenced this in commit 01af073a40 on Jan 12, 2020
  43. PastaPastaPasta referenced this in commit 6d749f8d2a on Jan 16, 2020
  44. PastaPastaPasta referenced this in commit 6c9e49cd61 on Jan 16, 2020
  45. zkbot referenced this in commit aa225ebb0b on Jan 24, 2020
  46. zkbot referenced this in commit 74ff73abab on Jan 24, 2020
  47. PastaPastaPasta referenced this in commit 410113b7e3 on Jan 31, 2020
  48. PastaPastaPasta referenced this in commit 577f597e99 on Jan 31, 2020
  49. PastaPastaPasta referenced this in commit b12538634b on Feb 4, 2020
  50. PastaPastaPasta referenced this in commit 304f012632 on Feb 9, 2020
  51. PastaPastaPasta referenced this in commit 3b439789c6 on Feb 13, 2020
  52. PastaPastaPasta referenced this in commit 9d9d1499b0 on Feb 27, 2020
  53. PastaPastaPasta referenced this in commit 39a8e20de6 on Feb 27, 2020
  54. furszy referenced this in commit 4ed15cc69d on Jun 8, 2020
  55. ckti referenced this in commit 4a4ccdfe65 on Mar 28, 2021
  56. ckti referenced this in commit 26a91e1270 on Mar 28, 2021
  57. PastaPastaPasta referenced this in commit ffb6b7c210 on May 26, 2021
  58. gades referenced this in commit e391a0081c on Jun 26, 2021
  59. PastaPastaPasta referenced this in commit 0687c9a9f5 on Jun 27, 2021
  60. PastaPastaPasta referenced this in commit af67ca3268 on Jun 28, 2021
  61. PastaPastaPasta referenced this in commit f4d7177988 on Jun 29, 2021
  62. PastaPastaPasta referenced this in commit 2618b52d3c on Jun 29, 2021
  63. PastaPastaPasta referenced this in commit fd2b028aba on Jun 29, 2021
  64. PastaPastaPasta referenced this in commit 07e1a914af on Jun 29, 2021
  65. PastaPastaPasta referenced this in commit a9bf129d18 on Jun 29, 2021
  66. gades referenced this in commit ad3bab0111 on Jun 30, 2021
  67. linuxsh2 referenced this in commit 78e9c711a6 on Jul 29, 2021
  68. DrahtBot locked this on Sep 8, 2021


theuni practicalswift MarcoFalke fanquake laanwj martinus TheBlueMatt

Labels
Tests

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-05 01:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me