This refactors things so that eval_phase_two can expose logs even when the script succeeds.
It also enriches traces to be either Logs or Labels, so that we can tell the difference between the two when inspecting the traces.
The idea is to get a good sample of measures from running benchmarks
with various sizes, so one can get an idea of how well a function
performs at various sizes.
Given that size can be made arbitrarily large, and that we currently
report all benchmarks, I installed a fibonacci heuristic to gather
data points from 0 to the max size using an increasing stepping.
Defined as a trait as I already anticipate we might need different
sizing strategy, likely driven by the user via a command-line option;
but for now, this will do.
Signed-off-by: KtorZ <5680256+KtorZ@users.noreply.github.com>
This commit removes some duplication between bench and test runners,
as well as fixing the results coming out of running benchmarks.
Running benchmarks is expected to yield multiple measures, for each of
the iteration. For now, it'll suffice to show results for each size;
but eventually, we'll possibly try to interpolate results with
different curves and pick the best candidate.
Signed-off-by: KtorZ <5680256+KtorZ@users.noreply.github.com>
In particular, using a concrete enum instead of a string to avoid an
unnecessary incomplete pattern-match, and remove superfluous comments.
Signed-off-by: KtorZ <5680256+KtorZ@users.noreply.github.com>
1. A new option `show_json_schema` which, when enabled, will print the JSON schema of the command output if the target isn't an ANSI-capable terminal.
2. Some modifications to the help message and error handling for the new option `show_json_schema`.
This is now done to avoid flooding the help screen with an entire
JSON schema. Plus, it makes the schema more easily exportable as an
actual JSON schema.
- Ensure consistency between overall summary and individual summaries.
- Remove 'max_*' properties, since they are only padding numbers used
for formatting the terminal output.
- Rename a few fields to be closer to the existing naming (name ->
title, memory -> mem, etc..)
- Remove duplicated outputs
- Re-order fields such that summaries come first
For the program to be consistent, the 'EventListener' target that we
pass to a Project should be responsible for the output format.
Otherwise, we are contingent on developers to remember passing the
option at call-site. Plus, it overloads the project code with an extra
boolean option.
Instead, since the behaviour is solely driven by the execution
context, we can instantiate a different event target upfront, and
simply hold on to it throughout the program.
As a nice side-effect, we can gently re-organize the code to keep the
terminal printing logic and the json printing logic separate.
Pi Lanningham