# `Benchee.Statistics`
[🔗](https://github.com/bencheeorg/benchee/blob/1.5.1/lib/benchee/statistics.ex#L1)

Statistics related functionality that is meant to take the raw benchmark data
and then compute statistics like the average and the standard deviation etc.

See `statistics/1` for a breakdown of the included statistics.

# `mode`

```elixir
@type mode() :: [number()] | number() | nil
```

Careful with the mode, might be multiple values, one value or nothing.😱

# `samples`

```elixir
@type samples() :: [number()]
```

The samples a `Benchee.Collect` collected to compute statistics from.

# `t`

```elixir
@type t() :: %Benchee.Statistics{
  absolute_difference: float() | nil,
  average: float(),
  ips: float() | nil,
  lower_outlier_bound: number(),
  maximum: number(),
  median: number(),
  minimum: number(),
  mode: mode(),
  outliers: [number()],
  percentiles: %{required(number()) => float()},
  relative_less: float() | nil | :infinity,
  relative_more: float() | nil | :infinity,
  sample_size: integer(),
  std_dev: float(),
  std_dev_ips: float() | nil,
  std_dev_ratio: float(),
  upper_outlier_bound: number()
}
```

All the statistics `statistics/1` computes from the samples.

This used for run times, memory and reductions. Generally with these,
the lower the better (less run time, memory consumption or reductions).

These values mostly correspond to their cousins in `Statistex`.

Overview of all the statistics Benchee currently provides:

  * average - average of all the samples (the lower the better)
  * ips - iterations per second, how often can the given function be
  executed within one second, used only for run times (the higher the better)
  * std_dev - standard deviation, how much results vary among the samples
  (the higher the more the results vary)
  * std_dev_ratio - standard deviation expressed as how much it is relative to
  the average
  * std_dev_ips - the absolute standard deviation of iterations per second
  * median - when all measured times are sorted, this is the middle
  value (or average of the two middle values when the number of times is
  even). More stable than the average and somewhat more likely to be a
  typical value you see.
  * percentiles - a map of percentile ranks. These are the values below
    which x% of the samples lie. For example, 99% of samples are less than
    is a value for which 99% of the run times are less than it.
  * mode - the samples that occur the most. Often one value, but
    can be multiple values if they occur the same amount of times. If no value
    occurs at least twice, this value will be `nil`.
  * minimum - the smallest sample measured for the scenario
  * maximum - the biggest sample measured for the scenario
  * relative_more - relative to the reference (usually the fastest scenario) how much more
  was the average of this scenario. E.g. for reference at 100, this scenario 200 then it
  is 2.0.
  * relative_less - relative to the reference (usually the fastest scenario) how much less
  was the average of this scenario. E.g. for reference at 100, this scenario 200 then it
  is 0.5.
  * absolute_difference - relative to the reference (usually the fastest scenario) what is
  the difference of the averages of the scenarios. e.g. for reference at 100, this
  scenario 200 then it is 100.
  * sample_size - the number of measurements/samples taken into account for calculating statistics
  * outliers - if outlier exclusion was enabled, may include any samples of outliers that were found, empty list otherwise
  * lower_outlier_bound - value below which values are considered an outlier
  * upper_outlier_bound - value above which values are considered an outlier

# `statistics`

Takes a suite with scenarios and their data samples, adds the statistics to the
scenarios. For an overview of what the statistics mean see `t:t/0`.

Note that this will also sort the scenarios fastest to slowest to ensure a consistent order
of scenarios in all used formatters.

## Examples

    iex> scenarios = [
    ...>   %Benchee.Scenario{
    ...>     job_name: "My Job",
    ...>     run_time_data: %Benchee.CollectionData{
    ...>       samples: [200, 400, 400, 400, 500, 500, 500, 700, 900]
    ...>     },
    ...>     memory_usage_data: %Benchee.CollectionData{
    ...>       samples: [200, 400, 400, 400, 500, 500, 500, 700, 900]
    ...>     },
    ...>     input_name: "Input",
    ...>     input: "Input"
    ...>   }
    ...> ]
    ...>
    ...> suite = %Benchee.Suite{scenarios: scenarios}
    ...> statistics(suite, Benchee.Test.FakeProgressPrinter)
    %Benchee.Suite{
      scenarios: [
        %Benchee.Scenario{
          job_name: "My Job",
          input_name: "Input",
          input: "Input",
          run_time_data: %Benchee.CollectionData{
            samples: [200, 400, 400, 400, 500, 500, 500, 700, 900],
            statistics: %Benchee.Statistics{
              average: 500.0,
              ips: 2000_000.0,
              std_dev: 200.0,
              std_dev_ratio: 0.4,
              std_dev_ips: 800_000.0,
              median: 500.0,
              percentiles: %{25 => 400.0, 50 => 500.0, 75 => 600.0, 99 => 900.0},
              mode: [500, 400],
              minimum: 200,
              maximum: 900,
              sample_size: 9,
              outliers: [],
              lower_outlier_bound: 100.0,
              upper_outlier_bound: 900.0
            }
          },
          memory_usage_data: %Benchee.CollectionData{
            samples: [200, 400, 400, 400, 500, 500, 500, 700, 900],
            statistics: %Benchee.Statistics{
              average: 500.0,
              ips: nil,
              std_dev: 200.0,
              std_dev_ratio: 0.4,
              std_dev_ips: nil,
              median: 500.0,
              percentiles: %{25 => 400.0, 50 => 500.0, 75 => 600.0, 99 => 900.0},
              mode: [500, 400],
              minimum: 200,
              maximum: 900,
              sample_size: 9,
              outliers: [],
              lower_outlier_bound: 100.0,
              upper_outlier_bound: 900.0
            }
          }
        }
      ],
      system: nil
    }