Proper Scoring Rule to score quantile predictions. Smaller values are better.
The quantile score is
closely related to the Interval score (see interval_score()) and is
the quantile equivalent that works with single quantiles instead of
central prediction intervals.
Arguments
- true_values
A vector with the true observed values of size n
- predictions
nxN matrix of predictive samples, n (number of rows) being the number of data points and N (number of columns) the number of Monte Carlo samples. Alternatively, predictions can just be a vector of size n.
- quantiles
vector of size n with the quantile values of the corresponding predictions.
- weigh
if TRUE, weigh the score by alpha / 2, so it can be averaged into an interval score that, in the limit, corresponds to CRPS. Alpha is the value that corresponds to the (alpha/2) or (1 - alpha/2) quantiles provided and will be computed from the quantile. Alpha is the decimal value that represents how much is outside a central prediction interval (E.g. for a 90 percent central prediction interval, alpha is 0.1). Default:
TRUE.
References
Strictly Proper Scoring Rules, Prediction,and Estimation, Tilmann Gneiting and Adrian E. Raftery, 2007, Journal of the American Statistical Association, Volume 102, 2007 - Issue 477
Evaluating epidemic forecasts in an interval format, Johannes Bracher, Evan L. Ray, Tilmann Gneiting and Nicholas G. Reich, https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008618
Examples
true_values <- rnorm(10, mean = 1:10)
alpha <- 0.5
lower <- qnorm(alpha / 2, rnorm(10, mean = 1:10))
upper <- qnorm((1 - alpha / 2), rnorm(10, mean = 1:10))
qs_lower <- quantile_score(true_values,
predictions = lower,
quantiles = alpha / 2
)
qs_upper <- quantile_score(true_values,
predictions = upper,
quantiles = 1 - alpha / 2
)
interval_score <- (qs_lower + qs_upper) / 2