Package index • scoringutils

Package documentation

scoringutils scoringutils-package: scoringutils: Utilities for Scoring and Assessing Predictions

Example data

example_binary: Binary forecast example data

example_nominal: Nominal example data

example_ordinal: Ordinal example data

example_point: Point forecast example data

example_quantile: Quantile example data

example_sample_continuous: Continuous forecast example data

example_sample_discrete: Discrete forecast example data

Create forecast objects

as_forecast_binary(): Create a forecast object for binary forecasts

as_forecast_doc_template: General information on creating a forecast object

as_forecast_generic(): Common functionality for as_forecast_<type> functions

as_forecast_nominal(): Create a forecast object for nominal forecasts

as_forecast_ordinal(): Create a forecast object for ordinal forecasts

as_forecast_point(): Create a forecast object for point forecasts

as_forecast_quantile(): Create a forecast object for quantile-based forecasts

as_forecast_sample(): Create a forecast object for sample-based forecasts

set_forecast_unit(): Set unit of a single forecast manually

Validate forecast objects

assert_forecast(): Assert that input is a forecast object and passes validations

is_forecast_binary() is_forecast_nominal() is_forecast_ordinal() is_forecast_point() is_forecast_quantile() is_forecast_sample() is_forecast(): Test whether an object is a forecast object

Gain insights about forecasts and inputs

get_forecast_counts(): Count number of available forecasts

print(<forecast>): Print information about a forecast object

get_duplicate_forecasts(): Find duplicate forecasts

get_forecast_unit(): Get unit of a single forecast

Transform data

as_forecast_point(): Create a forecast object for point forecasts

as_forecast_quantile(): Create a forecast object for quantile-based forecasts

log_shift(): Log transformation with an additive shift

transform_forecasts(): Transform forecasts and observed values

Handling metrics/scoring functions

get_metrics(): Get metrics

get_metrics(<forecast_binary>): Get default metrics for binary forecasts

get_metrics(<forecast_nominal>): Get default metrics for nominal forecasts

get_metrics(<forecast_ordinal>): Get default metrics for nominal forecasts

get_metrics(<forecast_point>): Get default metrics for point forecasts

get_metrics(<forecast_quantile>): Get default metrics for quantile-based forecasts

get_metrics(<forecast_sample>): Get default metrics for sample-based forecasts

get_metrics(<scores>): Get names of the metrics that were used for scoring

select_metrics(): Select metrics from a list of functions

Evaluate forecasts

add_relative_skill(): Add relative skill scores based on pairwise comparisons

get_correlations(): Calculate correlation between metrics

get_coverage(): Get quantile and interval coverage values for quantile-based forecasts

get_pairwise_comparisons(): Obtain pairwise comparisons between models

get_pit_histogram(): Probability integral transformation histogram

score(): Evaluate forecasts

summarise_scores() summarize_scores(): Summarise scores as produced by score()

Lower-level scoring functions

ae_median_quantile(): Absolute error of the median (quantile-based version)

ae_median_sample(): Absolute error of the median (sample-based version)

bias_quantile(): Determines bias of quantile forecasts

bias_sample(): Determine bias of forecasts

crps_sample() dispersion_sample() overprediction_sample() underprediction_sample(): (Continuous) ranked probability score

dss_sample(): Dawid-Sebastiani score

interval_coverage(): Interval coverage (for quantile-based forecasts)

interval_score(): Interval score

logs_sample(): Logarithmic score (sample-based version)

mad_sample(): Determine dispersion of a probabilistic forecast

pit_histogram_sample(): Probability integral transformation for counts

quantile_score(): Quantile score

rps_ordinal(): Ranked Probability Score for ordinal outcomes

brier_score() logs_binary(): Metrics for binary outcomes

logs_categorical(): Log score for categorical outcomes

se_mean_sample(): Squared error of the mean (sample-based version)

wis() dispersion_quantile() overprediction_quantile() underprediction_quantile(): Weighted interval score (WIS)

Functions for plotting and data visualisation

plot_correlations(): Plot correlation between metrics

plot_forecast_counts(): Visualise the number of available forecasts

plot_heatmap(): Create a heatmap of a scoring metric

plot_interval_coverage(): Plot interval coverage

plot_pairwise_comparisons(): Plot heatmap of pairwise comparisons

plot_quantile_coverage(): Plot quantile coverage

plot_wis(): Plot contributions to the weighted interval score

theme_scoringutils(): Scoringutils ggplot2 theme

Internal functions

Internal input check functions

assert_dims_ok_point(): Assert Inputs Have Matching Dimensions

assert_forecast_generic(): Validation common to all forecast types

assert_forecast_type(): Assert that forecast type is as expected

assert_input_binary(): Assert that inputs are correct for binary forecast

assert_input_interval(): Assert that inputs are correct for interval-based forecast

assert_input_nominal(): Assert that inputs are correct for nominal forecasts

assert_input_ordinal(): Assert that inputs are correct for ordinal forecasts

assert_input_point(): Assert that inputs are correct for point forecast

assert_input_quantile(): Assert that inputs are correct for quantile-based forecast

assert_input_sample(): Assert that inputs are correct for sample-based forecast

check_columns_present(): Check column names are present in a data.frame

check_dims_ok_point(): Check Inputs Have Matching Dimensions

check_duplicates(): Check that there are no duplicate forecasts

check_input_binary(): Check that inputs are correct for binary forecast

check_input_interval(): Check that inputs are correct for interval-based forecast

check_input_point(): Check that inputs are correct for point forecast

check_input_quantile(): Check that inputs are correct for quantile-based forecast

check_input_sample(): Check that inputs are correct for sample-based forecast

check_number_per_forecast(): Check that all forecasts have the same number of rows

check_numeric_vector(): Check whether an input is an atomic vector of mode 'numeric'

check_try(): Helper function to convert assert statements into checks

get_forecast_type(): Get forecast type from forecast object

get_type(): Get type of a vector or matrix of observed values or predictions

test_columns_not_present(): Test whether column names are NOT present in a data.frame

test_columns_present(): Test whether all column names are present in a data.frame

validate_metrics(): Validate metrics

Misc internal functions

apply_metrics(): Apply a list of functions to a data table of forecasts

as_scores(): Create an object of class scores from data

assert_scores(): Validate an object of class scores

bias_quantile_single_vector(): Compute bias for a single vector of quantile predictions

clean_forecast(): Clean forecast object

compare_forecasts(): Compare a subset of common forecasts

document_assert_functions: Documentation template for assert functions

document_check_functions: Documentation template for check functions

document_test_functions: Documentation template for test functions

ensure_data.table(): Ensure that an object is a data.table

forecast_types: Documentation template for forecast types

geometric_mean(): Calculate geometric mean

get_protected_columns(): Get protected columns from data

get_range_from_quantile(): Get interval range belonging to a quantile

illustration-input-metric-binary-point: Illustration of required inputs for binary and point forecasts

illustration-input-metric-nominal: Illustration of required inputs for nominal forecasts

illustration-input-metric-ordinal: Illustration of required inputs for ordinal forecasts

illustration-input-metric-quantile: Illustration of required inputs for quantile-based forecasts

illustration-input-metric-sample: Illustration of required inputs for sample-based forecasts

interpolate_median(): Helper function to interpolate the median prediction if it is not available

new_forecast(): Class constructor for forecast objects

new_scores(): Construct an object of class scores

pairwise_comparison_one_group(): Do pairwise comparison for one set of forecasts

permutation_test(): Simple permutation test

quantile_to_interval() quantile_to_interval_dataframe() quantile_to_interval_numeric(): Transform from a quantile format to an interval format

run_safely(): Run a function safely

sample_to_interval_long(): Change data from a sample-based format to a long interval range format

scoringutils scoringutils-package: scoringutils: Utilities for Scoring and Assessing Predictions