trivial benchmark

3.0.0

An easy to use benchmarking system.

About Trivial-Benchmark

Frequently I want to do a quick benchmark comparison of my functions. time is nice to get some data, but it's limited to a single run so there isn't really much of a statistical value in it. Trivial-Benchmark runs a block of code many times and outputs some statistical data for it. On SBCL this includes the data from time, for all other implementations just the REAL- and RUN-TIME data.

How-To

We assume that there's a local or global nickname for org.shirakumo.trivial-benchmark called tb. You can activate the global nickname with (org.shirakumo.trivial-benchmark:add-nickname).

For basic throwaway benchmarking, the with-timing macro should suffice:

(tb:with-timing (1000)
  (+ 1 1))

However, you can also do more complex timing using your own timer and with-sampling. The former creates a new timer object (with an optional list of metrics to sample) and the latter collects one sample for each metric of the timer for the duration of the body forms.

(defvar *timer* (make-instance 'tb:timer))

(tb:with-sampling (*timer*)
  (+ 1 1))

(tb:with-sampling (*timer*)
  (expt 10 100))

(tb:report *timer*)

(tb:reset *timer*)

(tb:report *timer*)

System Information

Version: 3.0.0

Dependencies:

documentation-utils

Author: Yukari Hafner

License: zlib

Homepage: https://shinmera.github.io/trivial-benchmark/

Bug Tracker: https://github.com/Shinmera/trivial-benchmark/issues

Sources: https://github.com/Shinmera/trivial-benchmark.git

Definition Index

ORG.SHIRAKUMO.TRIVIAL-BENCHMARK

TB
TRIVIAL-BENCHMARK

No documentation provided.

EXTERNAL SPECIAL-VARIABLE

DEFAULT-COMPUTATIONS

Which computations to make by default.

See COMPUTE

EXTERNAL SPECIAL-VARIABLE

DEFAULT-METRICS

Which metrics to include by default.

If NIL, all sampled metrics are included.

See FORMAT-TIMER-STATS

EXTERNAL SPECIAL-VARIABLE

DEFAULT-SAMPLERS

Which sampler types to use by default.

See SAMPLER
See WITH-SAMPLING

EXTERNAL CLASS
BYTES-CONSED
Source
No documentation provided.
EXTERNAL CLASS
CYCLE-COUNTER
Source
No documentation provided.
EXTERNAL CLASS
EVAL-CALLS
Source
No documentation provided.
EXTERNAL CLASS
GC-RUN-TIME
Source
No documentation provided.
EXTERNAL CLASS
REAL-TIME
Source
No documentation provided.
EXTERNAL CLASS
RUN-TIME
Source
No documentation provided.

EXTERNAL CLASS

SAMPLER

Representation of a sample measuring method.

See VARIABLES
See WRAP-MEASUREMENT-FORM
See COMMIT-SAMPLES-FORM
See DEFINE-SAMPLER
See WITH-SAMPLING

EXTERNAL CLASS
SB-TIME
Source
No documentation provided.
EXTERNAL CLASS
SYSTEM-INFO
Source
No documentation provided.

EXTERNAL CLASS

TIMER

Encompasses a set of samples.

A timer can be re-used to sample in individual steps.

See FORMAT-TIMER-STATS
See SAMPLES
See METRIC-TYPES
See REPORT-TO
See RESET
See WITH-SAMPLING
See WITH-TIMING

EXTERNAL FUNCTION

ADD-NICKNAME

Adds the nikcnames TRIVIAL-BENCHMARK and TB to the package.

This is for convenience.

EXTERNAL FUNCTION

FORMAT-TIMER-STATS

STREAM
TIMER
&REST
ARGS
&KEY
COMPUTATIONS
METRICS
&ALLOW-OTHER-KEYS

Print a table of the samples contained in the timer.

STREAM should be a stream to print to.
TIMER should be a TIMER instance.
COMPUTATIONS should be the derived statistical properties to compute
METRICS should be the metrics to include. If NIL, uses all metrics the
timer recorded samples for.
Other arguments are passed on to PRINT-TABLE

See TIMER (type)
See PRINT-TABLE
See *DEFAULT-COMPUTATIONS*
See *DEFAULT-METRICS*

EXTERNAL FUNCTION

PRINT-TABLE

TABLE
&KEY
STREAM
PADDING
FORMAT

Print a table of values

TABLE should be a list of lists in row-major order. Each row must have
the same number of elements.

STREAM is the stream to print the table to.

PADDING is the amount of space to insert to the left and right of
every cell.

FORMAT is the format to print the table in. Can be one of:
  :PRINC   --- justs PRINCs the table instead
  :MINIMAL --- prints the table as small as it can
  :FANCY   --- prints the table with box drawing glyphs

EXTERNAL FUNCTION

REPORT

THING
&REST
ARGS
&KEY
STREAM
&ALLOW-OTHER-KEYS

Prints a report on the given thing.

This is a wrapper around REPORT-TO.

See REPORT-TO

EXTERNAL GENERIC-FUNCTION

COMMIT-SAMPLES-FORM

SAMPLER
COMMIT-FN

Emit a form to commit measured samples

Returns a new form.
The returned form is emitted into a lexical environment where
COMMIT-FN is bound to a macro with the following behaviour:

  (commit-fn metric sample ...)

  Each pair of METRIC and SAMPLE are recorded in the timer
  METRIC should be a symbol naming the metric to store the sample
  under, and SAMPLE should be a REAL number to record. The sample is
  coerced to a DOUBLE-FLOAT automatically.

See WITH-SAMPLING
See SAMPLER (type)

EXTERNAL GENERIC-FUNCTION

COMPUTE

THING
SAMPLES

Compute a derived statistic from a vector of samples.

You may add additional methods to this, though the following
computation types are provided by default:

  :COUNT
  :SAMPLES   --- The number of samples
  :TOTAL     --- The sum of all samples
  :MINIMUM   --- The smallest sample
  :MAXIMUM   --- The biggest sample
  :MEDIAN    --- The median of all samples
  :AVERAGE   --- The average of all samples
  :DEVIATION --- The standard deviation based on the average

See *DEFAULT-COMPUTATIONS*

EXTERNAL GENERIC-FUNCTION

METRIC-TYPES

TIMER

Returns a list of metric types included in the timer.

See TIMER (type)

EXTERNAL GENERIC-FUNCTION

REPORT-TO

STREAM
THING
&KEY
COMPUTATIONS
&ALLOW-OTHER-KEYS

Print a report about THING to STREAM.

STREAM may be a stream, T, or NIL, according to FORMAT's output
designators.

COMPUTATIONS may be a list of statistical properties to report on a
vector of samples.

Additional arguments are passed on to PRINT-TABLE

See TIMER (type)
See FORMAT-TIMER-STATS
See PRINT-TABLE

EXTERNAL GENERIC-FUNCTION

RESET

TIMER

Resets the timer and clears all samples.

See TIMER (type)

EXTERNAL GENERIC-FUNCTION

SAMPLES

TIMER
METRIC

Accesses a sample vector for the given metric.

The returned vector is adjustable and has a fill-pointer.
If no vector for the given metric existed before, it is created for
you.

See TIMER (type)

EXTERNAL GENERIC-FUNCTION

VARIABLES

SAMPLER

Return a list of variable specs the sampler needs.

Each variable spec should be a list of the following elements:
  SYMBOL  --- The symbol the variable is bound to
  DEFAULT --- The default value the variable is initialised to
  TYPE    --- The type to declare the variable as

See WITH-SAMPLING
See SAMPLER (type)

EXTERNAL GENERIC-FUNCTION

WRAP-MEASUREMENT-FORM

SAMPLER
FORM

Wrap FORM in a form to record sampling data.

Returns the new form.
The returned form is emitted into a lexical environment where
variables according to the SAMPLER's VARIBALES specs are bound.

See WITH-SAMPLING
See SAMPLER (type)

EXTERNAL MACRO

DEFINE-DELTA-SAMPLER

NAME
&BODY
SAMPLE-POINT-FORMS

Convenience macro to define a sampler type with a single delta metric.

NAME can either be the name of the sampler, or a list of the name and
a normalisation factor by which the delta is divided to convert it to
a proper sample point.

SAMPLE-POINT-FORMS should be one or more forms whose ultimate return
value evaluates to some REAL number. The forms are emitted *twice*,
the first time before the execution to be measured, and the second
time after. The sample is computed based on the difference of the two
return values.

See DEFINE-SAMPLER
See SAMPLER (type)

EXTERNAL MACRO

DEFINE-SAMPLER

NAME
VARS
&BODY
FORMS

Convenience macro to define a new sampler type.

NAME should be the name of the new sampler class.
VARS should be a list of variable specs that the sampler will use to
store data during a sampling step. Each spec can be either a symbol
naming the variable, or a variable spec list.
FORMS should match the following structures:

  (:MEASURE (FORM-VAR) BODY...)
  (:COMMIT (COMMIT-FN-VAR) BODY...)

Which are converted to methods for WRAP-MEASUREMENT-FORM and
COMMIT-SAMPLES-FORM respectively. Every variable in VARS is bound
during the body of either, so that you can conveniently emit it into
the resulting forms.

See VARIABLES
See WRAP-MEASUREMENT-FORM
See COMMIT-SAMPLES-FORM
See SAMPLER (type)
See DEFINE-DELTA-SAMPLER

EXTERNAL MACRO

WITH-SAMPLING

TIMER-FORM
&REST
SAMPLERS

&BODY
FORMS

Records sampling information about FORMS into a timer.

TIMER-FORM should evaluate to a TIMER instance, into which the
sampling data will be committed.

SAMPLERS should be a list of names of SAMPLER types that will measure
the FORMS. If none are given, *DEFAULT-SAMPLERS* are used.

The values of FORMS are returned.

During expansion of this macro each sampler in SAMPLERS is
instantiated and variable bindings according to their VARIABLES specs
are emitted. Then, the FORMS are wrapped according to each sampler's
WRAP-MEASUREMENT-FORM return value. The order here can matter, and the
first sampler specified will be the "innermost" wrapper. Finally,
each sampler's COMMIT-SAMPLES-FORM is emitted in order to record the
sampling data into the timer.

See *DEFAULT-SAMPLERS*
See SAMPLER (type)
See TIMER (type)
See VARIABLES
See WRAP-MEASUREMENT-FORM
See COMMIT-SAMPLES-FORM

EXTERNAL MACRO

WITH-TIMING

N
&REST
REPORT-ARGS
&KEY
TIMER
SAMPLERS

&BODY
FORMS

Convenience macro to do a one-off timing.

Creates a timer instance according to TIMER, runs WITH-SAMPLING N
times, and then REPORTs the timer using the additional REPORT-ARGS.

See TIMER (type)
See WITH-SAMPLING
See REPORT