[Benchmark] replace Wall-Clock Time with Per-process CPU Time #1701

1. 
   Raimo33 commented at 4:08 pm on July 14, 2025: none

   Summary

   I propose replacing the current gettimeofday()-based timing measurements in the benchmark suite with CPU cycle counting using hardware performance counters (RDTSC). This change would significantly improve benchmark reliability, precision, and consistency.

   Current Issues with Time-Based Measurements

   The current benchmarking implementation in bench.h uses wall-clock time measurements with microsecond precision:

   0static int64_t gettime_i64(void) {
   1    struct timeval tv;
   2    gettimeofday(&tv, NULL);
   3    return (int64_t)tv.tv_usec + (int64_t)tv.tv_sec * 1000000LL;
   4}
   

   This approach has several significant limitations:

   1. Limited Precision: Microsecond resolution is insufficient for modern CPUs where cryptographic operations can complete in hundreds of nanoseconds. Requiring a lot of iterations to get an acceptable result
   2. System Interference: Wall-clock time is affected by:
      • OS scheduler interrupts and context switches
      • Other running processes competing for CPU time
      • Power management and frequency scaling
      • Thermal throttling
      • Shared caches between cores
   3. High Variability: Benchmark results can vary by 50%+ between runs due to system noise
   4. Non-deterministic: Results depend on system load, making comparisons unreliable
   5. Overhead: System call overhead affects measurement accuracy
   6. Lack of comparison: doesn’t provide a stable, reliable metric for team-wide performance discussions and comparisons.

   Proposed Implementation

   Use clocks that don’t include the time that the process is paused etc. I propose something like clock_gettime(CLOCK_PROCESS_CPUTIME_ID) or perf stat -e cpu-clock

2. Raimo33 closed this on Sep 1, 2025

   ---

3. 
   real-or-random commented at 7:23 am on September 2, 2025: contributor

   I think that’s a good proposal.
4. real-or-random reopened this on Sep 2, 2025

   ---

5. 
   Raimo33 commented at 9:22 am on September 2, 2025: none

   I think that’s a good proposal.

   Unfortunately I was mistaken about RDTSC. It is not immune from system interference. It merely addresses points 1, 5, 6.

   After careful research, the only options are clock_gettime(CLOCK_PROCESS_CPUTIME_ID) or perf stat -e cpu-clock. They would address points 1, 2 (except for shared caches), 3, 4.

   both are not portable.

6. Raimo33 renamed this:
   Benchmark CPU cycles
   [Benchmark] replace Wall-Clock Time with Per-process CPU Time
   on Sep 2, 2025

Contributors
Raimo33 real-or-random