Metrics for binary outcomes

Brier score

The Brier Score is the mean squared error between the probabilistic prediction and the observed outcome. The Brier score is a proper scoring rule. Small values are better (best is 0, the worst is 1).

$$ \textrm{Brier\_Score} = (\textrm{prediction} - \textrm{outcome})^2, $$ where \(\textrm{outcome} \in \{0, 1\}\), and \(\textrm{prediction} \in [0, 1]\) represents the probability that the outcome is equal to 1.

Log score for binary outcomes

The Log Score is the negative logarithm of the probability assigned to the observed value. It is a proper scoring rule. Small values are better (best is zero, worst is infinity).

Usage

brier_score(observed, predicted)

logs_binary(observed, predicted)

Arguments

observed

A factor of length n with exactly two levels, holding the observed values. The highest factor level is assumed to be the reference level. This means that predicted represents the probability that the observed value is equal to the highest factor level.

predicted

A numeric vector of length n, holding probabilities. Values represent the probability that the corresponding outcome is equal to the highest level of the factor observed.

Value

A numeric vector of size n with the Brier scores

A numeric vector of size n with log scores

Details

The functions require users to provide observed values as a factor in order to distinguish its input from the input format required for scoring point forecasts. Internally, however, factors will be converted to numeric values. A factor observed = factor(c(0, 1, 1, 0, 1) with two levels (0 and 1) would internally be coerced to a numeric vector (in this case this would result in the numeric vector c(1, 2, 2, 1, 1)). After subtracting 1, the resulting vector (c(0, 1, 1, 0) in this case) is used for internal calculations. All predictions are assumed represent the probability that the outcome is equal of the last/highest factor level (in this case that the outcome is equal to 1).

You could alternatively also provide a vector like observed = factor(c("a", "b", "b", "a")) (with two levels, a and b), which would result in exactly the same internal representation. Probabilities then represent the probability that the outcome is equal to "b". If you want your predictions to be probabilities that the outcome is "a", then you could of course make observed a factor with levels swapped, i.e. observed = factor(c("a", "b", "b", "a"), levels = c("b", "a"))

Input format

Overview of required input format for binary and point forecasts

See also

Other log score functions: logs_categorical(), logs_sample()

Examples

observed <- factor(sample(c(0, 1), size = 30, replace = TRUE))
predicted <- runif(n = 30, min = 0, max = 1)

brier_score(observed, predicted)
#>  [1] 0.169810872 0.319983015 0.720333826 0.136189585 0.539816982 0.573052550
#>  [7] 0.210298251 0.005442733 0.940506004 0.209025175 0.132818958 0.775957259
#> [13] 0.533583639 0.773922330 0.177404878 0.032952811 0.700006942 0.860101989
#> [19] 0.010349776 0.035994321 0.423125196 0.650581640 0.408156615 0.150229898
#> [25] 0.003399837 0.300913133 0.070701554 0.002211134 0.008862522 0.099258707
logs_binary(observed, predicted)
#>  [1] 0.53116635 0.83395161 1.88865474 0.46051080 1.32697839 1.41470349
#>  [7] 0.61356527 0.07663796 3.49981031 0.61100092 0.45325406 2.12766051
#> [13] 1.31106851 2.11800405 0.54678893 0.20031742 1.81194692 2.62302230
#> [19] 0.10728887 0.21037750 1.05119662 1.64292444 1.01852104 0.49036148
#> [25] 0.06007715 0.79530283 0.30910680 0.04816419 0.09887158 0.37841454