Skip to contents

Package documentation

scoringutils scoringutils-package
scoringutils: Utilities for Scoring and Assessing Predictions

Example data

example_binary
Binary forecast example data
example_nominal
Nominal example data
example_point
Point forecast example data
example_quantile
Quantile example data
example_sample_continuous
Continuous forecast example data
example_sample_discrete
Discrete forecast example data

Create forecast objects

as_forecast_binary()
Create a forecast object for binary forecasts
as_forecast_doc_template
General information on creating a forecast object
as_forecast_generic()
Common functionality for as_forecast_<type> functions
as_forecast_nominal()
Create a forecast object for nominal forecasts
as_forecast_point()
Create a forecast object for point forecasts
as_forecast_quantile()
Create a forecast object for quantile-based forecasts
as_forecast_sample()
Create a forecast object for sample-based forecasts
set_forecast_unit()
Set unit of a single forecast manually

Validate forecast objects

assert_forecast()
Assert that input is a forecast object and passes validations
is_forecast_binary() is_forecast_nominal() is_forecast_point() is_forecast_quantile() is_forecast_sample() is_forecast()
Test whether an object is a forecast object

Gain insights about forecasts and inputs

get_forecast_counts()
Count number of available forecasts
print(<forecast>)
Print information about a forecast object
get_duplicate_forecasts()
Find duplicate forecasts
get_forecast_unit()
Get unit of a single forecast

Transform data

as_forecast_point()
Create a forecast object for point forecasts
as_forecast_quantile()
Create a forecast object for quantile-based forecasts
log_shift()
Log transformation with an additive shift
transform_forecasts()
Transform forecasts and observed values

Handling metrics/scoring functions

get_metrics()
Get metrics
get_metrics(<forecast_binary>)
Get default metrics for binary forecasts
get_metrics(<forecast_nominal>)
Get default metrics for nominal forecasts
get_metrics(<forecast_point>)
Get default metrics for point forecasts
get_metrics(<forecast_quantile>)
Get default metrics for quantile-based forecasts
get_metrics(<forecast_sample>)
Get default metrics for sample-based forecasts
get_metrics(<scores>)
Get names of the metrics that were used for scoring
select_metrics()
Select metrics from a list of functions

Evaluate forecasts

add_relative_skill()
Add relative skill scores based on pairwise comparisons
get_correlations()
Calculate correlation between metrics
get_coverage()
Get quantile and interval coverage values for quantile-based forecasts
get_pairwise_comparisons()
Obtain pairwise comparisons between models
get_pit_histogram()
Probability integral transformation histogram
score()
Evaluate forecasts
summarise_scores() summarize_scores()
Summarise scores as produced by score()

Lower-level scoring functions

ae_median_quantile()
Absolute error of the median (quantile-based version)
ae_median_sample()
Absolute error of the median (sample-based version)
bias_quantile()
Determines bias of quantile forecasts
bias_sample()
Determine bias of forecasts
crps_sample() dispersion_sample() overprediction_sample() underprediction_sample()
(Continuous) ranked probability score
dss_sample()
Dawid-Sebastiani score
interval_coverage()
Interval coverage (for quantile-based forecasts)
interval_score()
Interval score
logs_sample()
Logarithmic score (sample-based version)
mad_sample()
Determine dispersion of a probabilistic forecast
pit_histogram_sample()
Probability integral transformation for counts
quantile_score()
Quantile score
brier_score() logs_binary()
Metrics for binary outcomes
logs_nominal()
Log score for nominal outcomes
se_mean_sample()
Squared error of the mean (sample-based version)
wis() dispersion_quantile() overprediction_quantile() underprediction_quantile()
Weighted interval score (WIS)

Functions for plotting and data visualisation

plot_correlations()
Plot correlation between metrics
plot_forecast_counts()
Visualise the number of available forecasts
plot_heatmap()
Create a heatmap of a scoring metric
plot_interval_coverage()
Plot interval coverage
plot_pairwise_comparisons()
Plot heatmap of pairwise comparisons
plot_quantile_coverage()
Plot quantile coverage
plot_wis()
Plot contributions to the weighted interval score
theme_scoringutils()
Scoringutils ggplot2 theme

Internal functions

Internal input check functions

assert_dims_ok_point()
Assert Inputs Have Matching Dimensions
assert_forecast_generic()
Validation common to all forecast types
assert_forecast_type()
Assert that forecast type is as expected
assert_input_binary()
Assert that inputs are correct for binary forecast
assert_input_interval()
Assert that inputs are correct for interval-based forecast
assert_input_nominal()
Assert that inputs are correct for nominal forecasts
assert_input_point()
Assert that inputs are correct for point forecast
assert_input_quantile()
Assert that inputs are correct for quantile-based forecast
assert_input_sample()
Assert that inputs are correct for sample-based forecast
check_columns_present()
Check column names are present in a data.frame
check_dims_ok_point()
Check Inputs Have Matching Dimensions
check_duplicates()
Check that there are no duplicate forecasts
check_input_binary()
Check that inputs are correct for binary forecast
check_input_interval()
Check that inputs are correct for interval-based forecast
check_input_point()
Check that inputs are correct for point forecast
check_input_quantile()
Check that inputs are correct for quantile-based forecast
check_input_sample()
Check that inputs are correct for sample-based forecast
check_number_per_forecast()
Check that all forecasts have the same number of rows
check_numeric_vector()
Check whether an input is an atomic vector of mode 'numeric'
check_try()
Helper function to convert assert statements into checks
get_forecast_type()
Get forecast type from forecast object
get_type()
Get type of a vector or matrix of observed values or predictions
test_columns_not_present()
Test whether column names are NOT present in a data.frame
test_columns_present()
Test whether all column names are present in a data.frame
validate_metrics()
Validate metrics

Misc internal functions

apply_metrics()
Apply a list of functions to a data table of forecasts
as_scores()
Create an object of class scores from data
assert_scores()
Validate an object of class scores
bias_quantile_single_vector()
Compute bias for a single vector of quantile predictions
clean_forecast()
Clean forecast object
compare_forecasts()
Compare a subset of common forecasts
document_assert_functions
Documentation template for assert functions
document_check_functions
Documentation template for check functions
document_test_functions
Documentation template for test functions
ensure_data.table()
Ensure that an object is a data.table
forecast_types
Documentation template for forecast types
geometric_mean()
Calculate geometric mean
get_protected_columns()
Get protected columns from data
get_range_from_quantile()
Get interval range belonging to a quantile
illustration-input-metric-binary-point
Illustration of required inputs for binary and point forecasts
illustration-input-metric-nominal
Illustration of required inputs for nominal forecasts
illustration-input-metric-quantile
Illustration of required inputs for quantile-based forecasts
illustration-input-metric-sample
Illustration of required inputs for sample-based forecasts
interpolate_median()
Helper function to interpolate the median prediction if it is not available
new_forecast()
Class constructor for forecast objects
new_scores()
Construct an object of class scores
pairwise_comparison_one_group()
Do pairwise comparison for one set of forecasts
permutation_test()
Simple permutation test
quantile_to_interval() quantile_to_interval_dataframe() quantile_to_interval_numeric()
Transform from a quantile format to an interval format
run_safely()
Run a function safely
sample_to_interval_long()
Change data from a sample-based format to a long interval range format
scoringutils scoringutils-package
scoringutils: Utilities for Scoring and Assessing Predictions