Change benchmark output to CSV #680

1. 
   real-or-random commented at 5:48 pm on October 28, 2019: contributor

   Not crucial but nice to have. It’s as readable as the current form for engineers but much easier to use in a spreadsheet if you want to look at the changes, which we do occasionally (that’s what benchmarks are for, hm?).

   Originally posted by @elichai in #678 (comment)

2. 
   gmaxwell commented at 8:55 pm on October 28, 2019: contributor

   I really dislike the bitcoin output, I think it’s very hard to read and make sense of… if someone does this at least put the most important measurements first. I’m dubious about the value of just bulk running these as most of their performance is a consequence of random code/data alignment and such… they jump around at random unless effort is made to deal with that.
3. real-or-random cross-referenced this on Nov 6, 2019 from issue Remove "except in benchmarks" exception for fp math by laanwj
4. 
   siv2r commented at 1:16 am on October 13, 2021: contributor

   I see running the bench_internal, bench_sign binaries produce the benchmark output to the terminal. I have the following two approaches in mind for CSV output.

   Approach 1: Modify the run_benchmark function to write to a CSV file. So, we will have a file like bench_internal.csv instead of bench_internal binary.

   Approach 2: Set the benchmark output format with a ./configure option (default will be terminal output) and create a binary or a CSV file accordingly. The google benchmark tool follows this pattern.

   Thoughts?

5. 
   real-or-random commented at 1:39 pm on October 13, 2021: contributor

   I see running the bench_internal, bench_sign binaries produce the benchmark output to the terminal. I have the following two approaches in mind for CSV output.

   There are more binaries, just look bench_*.c files in the src dir.

   Approach 1: Modify the run_benchmark function to write to a CSV file. So, we will have a file like bench_internal.csv instead of bench_internal binary.

   Approach 2: Set the benchmark output format with a ./configure option (default will be terminal output) and create a binary or a CSV file accordingly. The google benchmark tool follows this pattern.

   Thoughts?

   I think there’s a fundamental misunderstanding here. You’ll still need the binaries to produce the output. The question is just if the binaries write a table to the terminal or to a CSV file.

   So I think the right approach is to add a boolean parameter/flag to the run_benchmark function such that the caller can choose CSV output. And then the user could specify --output=foo.csv on the command-line when running the binary.

6. 
   siv2r commented at 2:11 pm on October 13, 2021: contributor

   I think there’s a fundamental misunderstanding here. You’ll still need the binaries to produce the output. The question is just if the binaries write a table to the terminal or to a CSV file.

   Makes sense. I don’t know why I thought creating a CSV output was possible without a binary.

   So I think the right approach is to add a boolean parameter/flag to the run_benchmark function such that the caller can choose CSV output. And then the user could specify --output=foo.csv on the command-line when running the binary.

   Sounds good!

   In addition to a command line and CSV output, Will a JSON output format be helpful? (like this)

7. 
   real-or-random commented at 3:11 pm on October 13, 2021: contributor

   In addition to a command line and CSV output, Will a JSON output format be helpful? (like this)

   It’d nice to have if it’s not much more work but I think it’s a stretch goal.

8. real-or-random cross-referenced this on Oct 18, 2021 from issue Shared benchmark format for command line and CSV outputs by siv2r
9. jonasnick closed this on Oct 22, 2021

   ---

Contributors
real-or-random gmaxwell siv2r