It can be pretty convenient for certain classes of benchmark to be able to run it with dynamic parameters, to generate curves of magnitudes runtimes.
This PR adds this functionality and shows an example using it.
This functionality is a bit useless unless you also combine it with a filter to ensure the params are meaningful for the target asymptotic benchmark.
25@@ -25,6 +26,7 @@ static void SetupBenchArgs()
26 gArgs.AddArg("-evals=<n>", strprintf("Number of measurement evaluations to perform. (default: %u)", DEFAULT_BENCH_EVALUATIONS), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
27 gArgs.AddArg("-filter=<regex>", strprintf("Regular expression filter to select benchmark by name (default: %s)", DEFAULT_BENCH_FILTER), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
28 gArgs.AddArg("-scaling=<n>", strprintf("Scaling factor for benchmark's runtime (default: %u)", DEFAULT_BENCH_SCALING), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
29+ gArgs.AddArg("-asymptote=n", strprintf("Asymptotic factors for benchmark's runtime, positional (default: \"\")"), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
BENCHMARK(ComplexMemPool, 1); line and hardcode the “asymptotes”. Or if you like, make the benchmark settings dynamic, so that an asymptote can be created by running bench_bitcoin -filter=... -scale_asymptotes_factor=5 to multiply all dynamic settings in the selected/filtered bench by that factor.
@MarcoFalke that’s insufficient for the example given. In that benchmark, all different sorts of asymptotic behaviors can be triggered with different growth factors.
I typically run it something like this:
for x in 10 100 1000 10000 100000 1000000; do ./src/bench/bench_bitcoin -asymptote=100 -asymptote=100 -asymptote=$x -asymptote=10 --evals=1; done | grep Com --line-buffered | cut -d, -f6
Scaling all the factors doesn’t let you express nuances.
The other issue is that if you just copy the existing Benchmark stuff you lose the ‘runs in one second" property.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.