Brier score
The Brier Score is the mean squared error between the probabilistic prediction and the observed outcome. The Brier score is a proper scoring rule. Small values are better (best is 0, the worst is 1).
$$ \textrm{Brier\_Score} = (\textrm{prediction} - \textrm{outcome})^2, $$ where \(\textrm{outcome} \in \{0, 1\}\), and \(\textrm{prediction} \in [0, 1]\) represents the probability that the outcome is equal to 1.
Log score for binary outcomes
The Log Score is the negative logarithm of the probability assigned to the observed value. It is a proper scoring rule. Small values are better (best is zero, worst is infinity).
Arguments
- observed
A factor of length n with exactly two levels, holding the observed values. The highest factor level is assumed to be the reference level. This means that
predicted
represents the probability that the observed value is equal to the highest factor level.- predicted
A numeric vector of length n, holding probabilities. Values represent the probability that the corresponding outcome is equal to the highest level of the factor
observed
.
Details
The functions require users to provide observed values as a factor in order
to distinguish its input from the input format required for scoring point
forecasts. Internally, however, factors will be converted to numeric values.
A factor observed = factor(c(0, 1, 1, 0, 1)
with two levels (0
and 1
)
would internally be coerced to a numeric vector (in this case this would
result in the numeric vector c(1, 2, 2, 1, 1)
). After subtracting 1, the
resulting vector (c(0, 1, 1, 0)
in this case) is used for internal
calculations. All predictions are assumed represent the probability that the
outcome is equal of the last/highest factor level (in this case that the
outcome is equal to 1).
You could alternatively also provide a vector like
observed = factor(c("a", "b", "b", "a"))
(with two levels, a
and b
),
which would result in exactly the same internal representation. Probabilities
then represent the probability that the outcome is equal to "b".
If you want your predictions to be probabilities that the outcome is "a",
then you could of course make observed
a factor with levels swapped, i.e.
observed = factor(c("a", "b", "b", "a"), levels = c("b", "a"))
See also
Other log score functions:
logs_nominal()
,
logs_sample()
Examples
observed <- factor(sample(c(0, 1), size = 30, replace = TRUE))
predicted <- runif(n = 30, min = 0, max = 1)
brier_score(observed, predicted)
#> [1] 0.169810872 0.319983015 0.720333826 0.136189585 0.539816982 0.573052550
#> [7] 0.210298251 0.005442733 0.940506004 0.209025175 0.132818958 0.775957259
#> [13] 0.533583639 0.773922330 0.177404878 0.032952811 0.700006942 0.860101989
#> [19] 0.010349776 0.035994321 0.423125196 0.650581640 0.408156615 0.150229898
#> [25] 0.003399837 0.300913133 0.070701554 0.002211134 0.008862522 0.099258707
logs_binary(observed, predicted)
#> [1] 0.53116635 0.83395161 1.88865474 0.46051080 1.32697839 1.41470349
#> [7] 0.61356527 0.07663796 3.49981031 0.61100092 0.45325406 2.12766051
#> [13] 1.31106851 2.11800405 0.54678893 0.20031742 1.81194692 2.62302230
#> [19] 0.10728887 0.21037750 1.05119662 1.64292444 1.01852104 0.49036148
#> [25] 0.06007715 0.79530283 0.30910680 0.04816419 0.09887158 0.37841454