ForecastEnsembles.jl

Appearance

1

Public Documentation ​

Documentation for ForecastEnsembles's public interface.

Contents ​

Index ​

Public API ​

ForecastEnsembles.BLP Type
julia
BLP(; weights = nothing)

Beta-transformed linear pool (Gneiting & Ranjan 2013): a recalibrated mixture that corrects the linear opinion pool's underdispersion by passing the mixture CDF through a fitted Beta CDF.

fit(BLP(), training, observations) fits the Beta to the linear pool's PIT values on the training set (maximum likelihood); combine(ft, fitted) applies it by evaluating the linear pool's quantile function at Beta-remapped levels. weights are the underlying per-model pool weights (equal by default, or any per-model EnsembleWeights / fitted method); the Beta parameters are learned on top. Quantile forecasts only.

Because the fit is a Beta on PIT values rather than a full weight regression, BLP is a recalibration of the pool: weights returns nothing (the level remap is not expressible as model weights), so apply it with combine(ft, fitted).

Fields

  • weights: per-model pool weights, or nothing for equal weights.
source
ForecastEnsembles.CRPSStacking Type
julia
CRPSStacking(; dirichlet_alpha = 1.001,
               lambda = nothing, time_col = nothing,
               task_weights = nothing)

CRPS-stacked linear opinion pool. Mirrors lopensemble::crps_weights, including its time weighting.

By default every training task contributes equally to the objective. Two ways to change that:

  • task_weights: a DataFrame with the training table's task-id columns plus a :weight column — one non-negative weight per task. The general mechanism; covers recency, per-region weighting (lopensemble's gamma), down-weighting anomalous reporting weeks, and so on.

  • lambda with time_col: convenience for recency weighting. time_col names the task column that orders tasks in time; lambda is one of

    • a scalar φ ∈ (0, 1]: exponential decay, weight φ^(T − t) for the t-th of T ordered unique time values (the common forecasting- literature choice; φ = 1 recovers equal weights),

    • :lopensemble: the quadratic ramp 2 − (1 − t/T)² that lopensemble::crps_weights uses by default (oldest ≈ 1, newest 2),

    • a Vector{Float64} with one weight per ordered unique time value (lopensemble's vector form),

    • a function of the normalised time rank t/T ∈ (0, 1] returning a weight.

lambda and task_weights are mutually exclusive. The Dirichlet prior strength scales with the effective sample size (Σλ)²/Σλ² rather than the raw task count, so heavy down-weighting of history does not quietly strengthen the prior relative to the data.

source
ForecastEnsembles.EnsembleMethod Type
julia
EnsembleMethod

Top of the method-type hierarchy. Subtypes split into

  • UnfittedMethod — can be passed to combine directly.

  • TrainedMethod — must be passed through fit first to obtain a fitted counterpart, which is itself an UnfittedMethod and can then be passed to combine.

source
ForecastEnsembles.EnsembleWeights Type
julia
EnsembleWeights(data; shape = :auto)

A typed container for per-model or per-quantile ensemble weights. The underlying data::DataFrame always has columns :model_id and :weight, and optionally :output_type_id. Two shapes are supported:

  • :per_model — columns :model_id, :weight. A single weight per model applied at every quantile/sample/etc.

  • :per_quantile — columns :model_id, :output_type_id, :weight. Weights vary across quantile levels.

Construct from a DataFrame (shape inferred from columns by default) or by calling weights(m) on a fitted method. MixtureEnsemble and QuantileEnsemble accept any of: a raw DataFrame, an EnsembleWeights, or any fitted method for which weights(m) returns one. The conversion happens at construction of the method, not at combine time, so invalid inputs fail fast.

source
ForecastEnsembles.FittedBLP Type
julia
FittedBLP(alpha, beta, weights)

Output of fit(::BLP, …). Stores the fitted Beta shape parameters alpha and beta and the underlying pool weights. alpha = beta = 1 means the linear pool was already calibrated (no transform). Apply with combine(ft, fitted); weights returns nothing.

source
ForecastEnsembles.FittedCRPSStacking Type
julia
FittedCRPSStacking(weights, models)

Output of fit(::CRPSStacking, …). Stores the simplex-constrained ensemble weights as a DataFrame with columns model_id and weight, the list of models these weights refer to, and crps, the fitted objective value (the mean CRPS achieved at the optimum). Plug into combine(ft, fitted) (sample inputs) — internally a LinearPool with these weights.

source
ForecastEnsembles.FittedHedge Type
julia
FittedHedge(weights, models, trajectory)

Output of fit(::Hedge, …). Stores the final simplex weights (a DataFrame with columns model_id and weight), the component models in weight order, and the trajectory — a long DataFrame (time_col, model_id, weight) of the weights after each update, for diagnosing weight stability over time. Plug into combine(ft, fitted) — internally a LinearPool with the final weights.

source
ForecastEnsembles.FittedInverseScore Type
julia
FittedInverseScore(weights, models, scores)

Output of fit(::InverseScore, …). Stores the simplex weights (a DataFrame with columns model_id and weight), the component models in weight order, and the per-member mean scores they were derived from. Plug into combine(ft, fitted) — internally a LinearPool with these weights.

source
ForecastEnsembles.FittedPartialPooling Type
julia
FittedPartialPooling(weights, global_weights, strata, models, score_value)

Output of fit(::PartialPooling, …). Stores the per-stratum weights (a DataFrame with the strata columns plus model_id and weight), the pooled global_weights (model_id, weight) used for strata not seen in training, the strata columns, the component models, and the mean score_value at the optimum. Plug into combine(ft, fitted) — it applies each stratum's weights, falling back to global_weights for an unseen stratum. weights(fitted) returns the pooled global vector as an EnsembleWeights.

source
ForecastEnsembles.FittedQRA Type
julia
FittedQRA(coefs, intercepts, models, levels, group_cols, per_quantile_weights, enforce_normalisation, has_intercept)

Output of fit(::QRA, …). coefs is a Dict{NamedTuple => Vector{Float64}} mapping a (group..., quantile_level) key to a vector of model coefficients in the order models. When per_quantile_weights is false, all keys sharing the same group share the same coefficients (and have the same key under a sentinel :any quantile_level).

Fields

  • coefs: Dict mapping each (group, quantile_level) key to its vector of model coefficients (ordered as models).

  • intercepts: Dict mapping each (group, quantile_level) key to its intercept; empty (all zero) when the fit has no intercept.

  • models: component model ids, giving the order of the coefficient vectors.

  • levels: the quantile levels the fit was trained on.

  • group_cols: the grouping task columns used when fitting; may be empty.

  • per_quantile_weights: Bool, whether coefficients vary across quantile levels.

  • enforce_normalisation: Bool, whether the simplex constraint (non-negative coefficients summing to one) was imposed.

  • has_intercept: Bool, whether an intercept was estimated.

source
ForecastEnsembles.FittedStacking Type
julia
FittedStacking(weights, models, score_value)

Output of fit(::Stacking, …). Stores the simplex ensemble weights (a DataFrame with columns model_id and weight), the component models in weight order, and the mean score_value achieved at the optimum. Plug into combine(ft, fitted) — internally a LinearPool with these weights.

source
ForecastEnsembles.ForecastTable Type
julia
ForecastTable

A long-format probabilistic-forecast table aligned with the hubverse model_out_tbl schema.

Required columns

  • the model identifier column (default :model_id)

  • :output_type — symbol, one of ForecastEnsembles.KNOWN_OUTPUT_TYPES

  • :output_type_id — quantile level, sample index, threshold, …

  • :value — the forecast value

  • one or more task-id columns (e.g. :location, :horizon, :target_date)

Construction

julia
ForecastTable(df; task_id_cols, model_id_col = :model_id)

task_id_cols may be omitted, in which case it is inferred as every column that is not one of the required columns.

Fields

  • data: the underlying long-format DataFrame holding the forecasts.

  • task_id_cols: the task-id columns identifying a forecast target.

  • model_id_col: the column naming the model that produced each forecast.

source
ForecastEnsembles.Hedge Type
julia
Hedge(score; eta = 1.0, time_col)

Online ensemble weighting by the Hedge / exponentiated-gradient rule.

fit(Hedge(score; time_col), training, observations) walks the distinct time_col values in order and, at each step, multiplies every member's weight by exp(-eta · sₜ) — where sₜ is that member's mean score on the current step (negatively oriented, so a lower score keeps more weight) — then renormalises to the simplex. A member absent at a step keeps its weight (a "sleeping expert"). The final weights plug into combine; the full trajectory is kept for weight-stability diagnostics.

Unlike InverseScore (one pooled score per member) this adapts to when members did well, so it tracks regime change; unlike Stacking it needs no optimiser and updates incrementally. score is any negatively-oriented weighted-sample rule from ScoringRules (e.g. ScoringRules.crps), which must be loaded — it is a weak dependency.

Fields

  • score: the scoring-rule function (negatively oriented).

  • eta: learning rate. Larger values adapt faster and concentrate weight more aggressively on recent winners; as it tends to 0 the weights stay uniform. Scale it to the magnitude of your score. Must be positive.

  • time_col: the task-id column defining the update order.

source
ForecastEnsembles.InverseScore Type
julia
InverseScore(score; temperature = 1.0)

Performance weighting: score each member independently and weight the better ones more heavily — wᵢ ∝ exp(−temperature · sᵢ), where sᵢ is member i's mean score over the training set (negatively oriented, so a lower score earns more weight). No optimisation, so it is fast and robust with few observations; but — unlike Stacking — it scores each member in isolation and never sees how they combine, so it is blind to redundancy between them.

fit(InverseScore(score), training, observations) returns a FittedInverseScore that plugs into combine / weights. score is any negatively-oriented weighted-sample rule from ScoringRules (e.g. ScoringRules.crps), which must be loaded — it is a weak dependency.

Fields

  • score: the scoring-rule function (negatively oriented).

  • temperature: sharpness of the softmax over member scores. As it tends to 0 the weights approach equal; large values approach winner-take-all. Must be positive.

source
ForecastEnsembles.LinearPool Type
julia
MixtureEnsemble(; weights = nothing, n_samples = 10_000)

Mixture (linear-opinion-pool) ensemble: the ensemble distribution is the (weighted) mixture of the component distributions, F = Σᵢ wᵢ Fᵢ. The algorithm path depends on the forecast output_type:

  • :sample — weighted resample from per-model samples.

  • :cdf — pointwise weighted average of CDFs.

  • :quantile — reconstruct each model's CDF from its quantiles, draw n_samples, pool, and re-extract quantiles at the original levels.

Mixture pooling is fundamentally a per-model operation; per-quantile weights aren't meaningful here (use QuantileEnsemble for that).

source
ForecastEnsembles.MixtureEnsemble Type
julia
MixtureEnsemble(; weights = nothing, n_samples = 10_000)

Mixture (linear-opinion-pool) ensemble: the ensemble distribution is the (weighted) mixture of the component distributions, F = Σᵢ wᵢ Fᵢ. The algorithm path depends on the forecast output_type:

  • :sample — weighted resample from per-model samples.

  • :cdf — pointwise weighted average of CDFs.

  • :quantile — reconstruct each model's CDF from its quantiles, draw n_samples, pool, and re-extract quantiles at the original levels.

Mixture pooling is fundamentally a per-model operation; per-quantile weights aren't meaningful here (use QuantileEnsemble for that).

source
ForecastEnsembles.PartialPooling Type
julia
PartialPooling(score; strata, lambda = 1.0, dirichlet_alpha = 1.0)

Hierarchical (partially pooled) stacking: learn a weight vector per stratum that shrinks toward a shared global vector, so a data-sparse stratum borrows strength from the rest.

fit(PartialPooling(score; strata), training, observations) jointly optimises, in softmax space, one logit vector per distinct combination of the strata columns plus a global logit vector, minimising the mean score of each stratum's linearly-pooled forecast plus a shrinkage penalty pulling every stratum toward the global vector. The FittedPartialPooling result plugs into combine, which applies each stratum's own weights (an unseen stratum falls back to the global vector).

score is any negatively-oriented weighted-sample rule from ScoringRules (e.g. ScoringRules.crps), which must be loaded — it is a weak dependency. Generalises Stacking: a single stratum, or lambda → ∞, recovers global stacking.

Fields

  • score: the scoring-rule function to minimise (negatively oriented).

  • strata: task-id columns whose value combinations define the strata (e.g. [:location], [:location, :age_group]).

  • lambda: shrinkage strength toward the global vector. 0 fits each stratum independently; large values pool them toward one shared vector. Must be ≥ 0.

  • dirichlet_alpha: strength of a symmetric-Dirichlet prior on each stratum's weights, pulling them toward the simplex centre; 1.0 applies no prior.

source
ForecastEnsembles.QRA Type
julia
QRA(; per_quantile_weights = false, intercept = true,
      enforce_normalisation = false, noncross = false,
      group = Symbol[])

Quantile Regression Averaging. group lists task dimensions over which a separate regression is fitted. Mirrors qrensemble::qra.

source
ForecastEnsembles.QuantileEnsemble Type
julia
QuantileEnsemble(agg = :mean; weights = nothing)

Per-quantile weighted aggregation of quantile forecasts. At each task and quantile level τ, take a weighted mean (agg = :mean, also called Vincentization) or weighted median (agg = :median) of the per-model quantile values. weights may be:

  • nothing — equal weights (the "simple ensemble" of the hubverse).

  • a per-model EnsembleWeights — same weights at every τ.

  • a per-quantile EnsembleWeights — different weights per τ (e.g. from a per-τ QRA fit, or supplied externally).

  • any fitted method whose weights(m) returns one of the above (FittedCRPSStacking, FittedQRA in the right configuration, etc.).

source
ForecastEnsembles.Stacking Type
julia
Stacking(score; dirichlet_alpha = 1.0)

Score-optimal stacking against a user-supplied proper scoring rule.

fit(Stacking(score), training, observations) learns simplex ensemble weights that minimise the mean score of the linearly-pooled forecast, where score is any negatively-oriented rule from ScoringRules — e.g. ScoringRules.crps. The FittedStacking result plugs into combine, or into LinearPool/QuantileEnsemble via weights.

ScoringRules must be loaded (using ScoringRules) for fit to work. It is a weak dependency, so the MIT core does not pull in its GPL code unless you opt in. CRPSStacking and QRA remain the dependency-free closed-form specialisations for CRPS and WIS respectively.

Fields

  • score: the scoring-rule function to minimise (negatively oriented).

  • dirichlet_alpha: strength of a symmetric-Dirichlet prior on the weights, pulling them toward the simplex centre; 1.0 applies no prior.

source
ForecastEnsembles.TrainedMethod Type
julia
TrainedMethod

An ensemble method whose weights or coefficients are learned from past performance before use, for example QRA or CRPSStacking. Passing one through fit returns a fitted object (itself an UnfittedMethod) that can then be passed to combine.

source
ForecastEnsembles.TrimmedMean Type
julia
TrimmedMean(; fraction = 0.1, mode = :trim)

Robust ensemble mean. At each task and output_type_id (e.g. a quantile level τ), order the per-model values and either trim — drop the lowest and highest fraction of them, then average the rest — or winsorise — clamp those extremes to the surviving boundary values, then average all of them.

k = round(fraction · n) models are trimmed/clamped from each end, capped so at least one value always survives (so fraction → 0.5 degenerates to the median, fraction = 0 to the plain mean). This is the robust cousin of QuantileEnsemble(:mean): cheaper than a full median ensemble to reason about, and tunable in how much of the tail it discards. It aggregates values that are comparable across models at a shared output_type_id, so it supports :quantile and :cdf forecasts, not :sample (sample indices are not aligned across models — use MixtureEnsemble there).

Fields

  • fraction: proportion trimmed/clamped from each end, in [0, 0.5).

  • mode: :trim (drop the extremes) or :winsorise (clamp them).

Example

julia
using ForecastEnsembles, DataFrames
df = DataFrame(
    model_id = string.("m", 1:5),
    output_type = "quantile",
    output_type_id = 0.5,
    location = "A",
    value = [1.0, 2.0, 3.0, 4.0, 100.0]
)
ft = ForecastTable(df; task_id_cols = [:location])
combine(ft, TrimmedMean(; fraction = 0.2))
source
ForecastEnsembles.UnfittedMethod Type
julia
UnfittedMethod

An ensemble method applied directly to forecasts with no training, for example QuantileEnsemble(:mean) or MixtureEnsemble(). It carries any fixed configuration (aggregation rule, supplied weights) and can be passed straight to combine without a prior call to fit.

source
ForecastEnsembles.Windowed Type
julia
Windowed(method, window; time_col)

Wrap a TrainedMethod so it trains on only the most recent window values of time_col.

fit(Windowed(method, window; time_col), training, observations) keeps the last window distinct time_col values of training (and the matching observations), fits method on that subset, and returns method's own fitted result — so it drops straight into combine.

Useful for a rolling-window scheme in backtest: compare an expanding-window CRPSStacking() against a rolling Windowed(CRPSStacking(), 8; time_col = :date).

Fields

  • method: the inner TrainedMethod to fit on the window.

  • window: number of most-recent time_col values to train on (must be ≥ 1).

  • time_col: the task-id column defining time order.

source
ForecastEnsembles.backtest Function
julia
backtest(ft, observations, schemes; time_col, min_train = 1,
         rng = default_rng(), score_fn = <ScoringRules default>) -> DataFrame

Expanding-window backtest of ensemble schemes. The unique values of time_col are ordered; for each test time after the first min_train, every scheme is trained on the earlier times and scored on the test time out-of-sample. Returns one row per (scheme, test time) with columns scheme, the time_col, and :score (mean over that time's tasks).

schemes maps names to EnsembleMethods (a Dict or a vector of name => method pairs):

  • a TrainedMethod — CRPSStacking(), QRA(...) — is fitted on the training window each fold, then applied to the test time;

  • an UnfittedMethod — QuantileEnsemble(:mean), MixtureEnsemble() — is applied directly.

Each scheme must match the table's output_type: CRPSStacking and sample combiners need :sample data, QRA and quantile combiners need :quantile.

Scoring is injected via score_fn(forecast::ForecastTable, observations) -> Real (the mean score of one fold). Load ScoringRules (using ScoringRules) for a sensible default — CRPS for sample forecasts, the quantile score for quantile forecasts — or pass your own score_fn.

Aggregate across folds yourself, e.g.

julia
using DataFrames
res = backtest(ft, obs, schemes; time_col = :target_date)
combine(groupby(res, :scheme), :score => mean => :mean_score)

Arguments

  • ft: a ForecastTable.

  • observations: a DataFrame with the table's task-id columns and an :observed column.

  • schemes: a Dict or vector of name => EnsembleMethod pairs to compare.

Keyword Arguments

  • time_col: the column giving the time index to expand the window over.

  • min_train: number of initial times used only for training (default 1).

  • rng: RNG used by sample-based schemes (default default_rng()).

  • score_fn: a scorer (forecast, observations) -> Real; defaults to the ScoringRules-based rule for the table's output_type (needs using ScoringRules).

Examples

julia
using ForecastEnsembles, DataFrames, Random, Statistics
rng = MersenneTwister(1)
T = 12; K = 40
obs = DataFrame(t = 1:T, observed = randn(rng, T))
rows = DataFrame[]
for (mid, s) in (("m1", (y, r) -> y .+ randn(r, K)), ("m2", (y, r) -> 2 .* randn(r, K)))
    for t in 1:T
        push!(rows, DataFrame(model_id = mid, output_type = "sample",
            output_type_id = 1:K, t = t, value = s(obs.observed[t], rng)))
    end
end
ft = ForecastTable(reduce(vcat, rows); task_id_cols = [:t])
schemes = ["equal" => MixtureEnsemble(), "stack" => CRPSStacking()]

# Inject a scorer — here mean absolute error of the ensemble mean; with
# `using ScoringRules` you would instead pass a proper rule such as ScoringRules.crps.
function mae(ens, o)
    d = innerjoin(DataFrame(ens), o; on = :t)
    per = combine(groupby(d, :t),
        [:value, :observed] => ((v, y) -> abs(mean(v) - first(y))) => :e)
    return mean(per.e)
end

backtest(ft, obs, schemes; time_col = :t, min_train = 6, score_fn = mae)
source
DataFrames.combine Function
julia
combine(ft::ForecastTable, m::QuantileEnsemble) -> ForecastTable

Hub-style simple/weighted ensemble. Aggregates value across model_id within each (task, output_type, output_type_id) group, using m.agg (:mean or :median) and optional per-model weights.

Mirrors hubEnsembles::simple_ensemble. The output model_id is set to "hub-ensemble", matching the R package default.

source
julia
combine(ft::ForecastTable, m::MixtureEnsemble; rng = default_rng()) -> ForecastTable

Linear opinion pool. The kernel is dispatched on the table's output_type:

  • :sample → weighted resampling of per-model samples to give a single pooled sample set per task (the only path that uses rng).

  • :cdf → weighted pointwise average of CDFs.

  • :quantile → reconstruct a continuous distribution per model via QuantileDistribution, then invert the mixture CDF Σᵢ wᵢ Fᵢ exactly by bisection at each input level. Deterministic — no Monte Carlo error, which matters for the extreme levels (τ = 0.01, 0.99) hubs request.

source
julia
combine(ft::ForecastTable, m::TrimmedMean; rng = default_rng()) -> ForecastTable

Apply a trimmed or winsorised cross-model mean at each (task, output_type_id). See TrimmedMean. The output model_id is "hub-ensemble".

source
julia
combine(ft::ForecastTable, m::FittedQRA) -> ForecastTable

Apply fitted QRA weights to a new set of forecasts. The output has output_type = :quantile and one row per (task, quantile_level).

source
julia
combine(ft::ForecastTable, m::FittedCRPSStacking; rng = default_rng()) -> ForecastTable

Apply CRPS-stacked weights to a (sample-typed) forecast table. Equivalent to combine(ft, LinearPool(weights = m.weights)).

Arguments

  • ft: a ForecastTable of sample forecasts.

  • m: a FittedCRPSStacking holding the fitted ensemble weights.

Keyword Arguments

  • rng: random number generator used for resampling; defaults to default_rng().

Example

julia
using ForecastEnsembles, DataFrames, Random
rng = MersenneTwister(1)
T = 20; K = 50
obs = DataFrame(t = 1:T, observed = randn(rng, T))
rows = DataFrame[]
for (mid, s) in (("m1", (y, r) -> y .+ randn(r, K)), ("m2", (y, r) -> 3 .* randn(r, K)))
    for t in 1:T
        push!(rows, DataFrame(model_id = mid, output_type = "sample",
            output_type_id = 1:K, t = t, value = s(obs.observed[t], rng)))
    end
end
ft = ForecastTable(reduce(vcat, rows); task_id_cols = [:t])
fitted = fit(CRPSStacking(), ft, obs)
combine(ft, fitted)
source
julia
combine(df::AbstractDataFrame, args...;
        renamecols::Bool=true, threads::Bool=true)
combine(f::Callable, df::AbstractDataFrame;
        renamecols::Bool=true, threads::Bool=true)
combine(gd::GroupedDataFrame, args...;
        keepkeys::Bool=true, ungroup::Bool=true,
        renamecols::Bool=true, threads::Bool=true)
combine(f::Base.Callable, gd::GroupedDataFrame;
        keepkeys::Bool=true, ungroup::Bool=true,
        renamecols::Bool=true, threads::Bool=true)

Create a new data frame that contains columns from df or gd specified by args and return it. The result can have any number of rows that is determined by the values returned by passed transformations.

Below detailed common rules for all transformation functions supported by DataFrames.jl are explained and compared.

All these operations are supported both for AbstractDataFrame (when split and combine steps are skipped) and GroupedDataFrame. Technically, AbstractDataFrame is just considered as being grouped on no columns (meaning it has a single group, or zero groups if it is empty). The only difference is that in this case the keepkeys and ungroup keyword arguments (described below) are not supported and a data frame is always returned, as there are no split and combine steps in this case.

In order to perform operations by groups you first need to create a GroupedDataFrame object from your data frame using the groupby function that takes two arguments: (1) a data frame to be grouped, and (2) a set of columns to group by.

Operations can then be applied on each group using one of the following functions:

  • combine: does not put restrictions on number of rows returned per group; the returned values are vertically concatenated following order of groups in GroupedDataFrame; it is typically used to compute summary statistics by group; for GroupedDataFrame if grouping columns are kept they are put as first columns in the result;

  • select: return a data frame with the number and order of rows exactly the same as the source data frame, including only new calculated columns; select! is an in-place version of select; for GroupedDataFrame if grouping columns are kept they are put as first columns in the result;

  • transform: return a data frame with the number and order of rows exactly the same as the source data frame, including all columns from the source and new calculated columns; transform! is an in-place version of transform; existing columns in the source data frame are put as first columns in the result;

As a special case, if a GroupedDataFrame that has zero groups is passed then the result of the operation is determined by performing a single call to the transformation function with a 0-row argument passed to it. The output of this operation is only used to identify the number and type of produced columns, but the result has zero rows.

All these functions take a specification of one or more functions to apply to each subset of the DataFrame. This specification can be of the following forms: 2. standard column selectors (integers, Symbols, strings, vectors of integers, vectors of Symbols, vectors of strings, All, Cols, :, Between, Not and regular expressions)

  1. a cols => function pair indicating that function should be called with positional arguments holding columns cols, which can be any valid column selector; in this case target column name is automatically generated and it is assumed that function returns a single value or a vector; the generated name is created by concatenating source column name and function name by default (see examples below).

  2. a cols => function => target_cols form additionally explicitly specifying the target column or columns, which must be a single name (as a Symbol or a string), a vector of names or AsTable. Additionally it can be a Function which takes a string or a vector of strings as an argument containing names of columns selected by cols, and returns the target columns names (all accepted types except AsTable are allowed).

  3. a col => target_cols pair, which renames the column col to target_cols, which must be single name (as a Symbol or a string), a vector of names or AsTable.

  4. column-independent operations function => target_cols or just function for specific functions where the input columns are omitted; without target_cols the new column has the same name as function, otherwise it must be single name (as a Symbol or a string). Supported functions are:

  • nrow to efficiently compute the number of rows in each group.

  • proprow to efficiently compute the proportion of rows in each group.

  • eachindex to return a vector holding the number of each row within each group.

  • groupindices to return the group number.

  1. vectors or matrices containing transformations specified by the Pair syntax described in points 2 to 5

  2. a function which will be called with a SubDataFrame corresponding to each group if a GroupedDataFrame is processed, or with the data frame itself if an AbstractDataFrame is processed; this form should be avoided due to its poor performance unless the number of groups is small or a very large number of columns are processed (in which case SubDataFrame avoids excessive compilation)

Note! If the expression of the form x => y is passed then except for the special convenience form nrow => target_cols it is always interpreted as cols => function. In particular the following expression function => target_cols is not a valid transformation specification.

Note! If cols or target_cols are one of All, Cols, Between, or Not, broadcasting using .=> is supported and is equivalent to broadcasting the result of names(df, cols) or names(df, target_cols). This behaves as if broadcasting happened after replacing the selector with selected column names within the data frame scope.

All functions have two types of signatures. One of them takes a GroupedDataFrame as the first argument and an arbitrary number of transformations described above as following arguments. The second type of signature is when a Function or a Type is passed as the first argument and a GroupedDataFrame as the second argument (similar to map).

As a special rule, with the cols => function and cols => function => target_cols syntaxes, if cols is wrapped in an AsTable object then a NamedTuple containing columns selected by cols is passed to function. The documentation of DataFrames.table_transformation provides more information about this functionality, in particular covering performance considerations.

What is allowed for function to return is determined by the target_cols value: 2. If both cols and target_cols are omitted (so only a function is passed), then returning a data frame, a matrix, a NamedTuple, a Tables.AbstractRow or a DataFrameRow will produce multiple columns in the result. Returning any other value produces a single column.

  1. If target_cols is a Symbol or a string then the function is assumed to return a single column. In this case returning a data frame, a matrix, a NamedTuple, a Tables.AbstractRow, or a DataFrameRow raises an error.

  2. If target_cols is a vector of Symbols or strings or AsTable it is assumed that function returns multiple columns. If function returns one of AbstractDataFrame, NamedTuple, DataFrameRow, Tables.AbstractRow, AbstractMatrix then rules described in point 1 above apply. If function returns an AbstractVector then each element of this vector must support the keys function, which must return a collection of Symbols, strings or integers; the return value of keys must be identical for all elements. Then as many columns are created as there are elements in the return value of the keys function. If target_cols is AsTable then their names are set to be equal to the key names except if keys returns integers, in which case they are prefixed by x (so the column names are e.g. x1, x2, ...). If target_cols is a vector of Symbols or strings then column names produced using the rules above are ignored and replaced by target_cols (the number of columns must be the same as the length of target_cols in this case). If fun returns a value of any other type then it is assumed that it is a table conforming to the Tables.jl API and the Tables.columntable function is called on it to get the resulting columns and their names. The names are retained when target_cols is AsTable and are replaced if target_cols is a vector of Symbols or strings.

In all of these cases, function can return either a single row or multiple rows. As a particular rule, values wrapped in a Ref or a 0-dimensional AbstractArray are unwrapped and then treated as a single row.

select/select! and transform/transform! always return a data frame with the same number and order of rows as the source (even if GroupedDataFrame had its groups reordered), except when selection results in zero columns in the resulting data frame (in which case the result has zero rows).

For combine, rows in the returned object appear in the order of groups in the GroupedDataFrame. The functions can return an arbitrary number of rows for each group, but the kind of returned object and the number and names of columns must be the same for all groups, except when a DataFrame() or NamedTuple() is returned, in which case a given group is skipped.

It is allowed to mix single values and vectors if multiple transformations are requested. In this case single value will be repeated to match the length of columns specified by returned vectors.

To apply function to each row instead of whole columns, it can be wrapped in a ByRow struct. cols can be any column indexing syntax, in which case function will be passed one argument for each of the columns specified by cols or a NamedTuple of them if specified columns are wrapped in AsTable. If ByRow is used it is allowed for cols to select an empty set of columns, in which case function is called for each row without any arguments and an empty NamedTuple is passed if empty set of columns is wrapped in AsTable.

If a collection of column names is passed then requesting duplicate column names in target data frame are accepted (e.g. select!(df, [:a], :, r"a") is allowed) and only the first occurrence is used. In particular a syntax to move column :col to the first position in the data frame is select!(df, :col, :). On the contrary, output column names of renaming, transformation and single column selection operations must be unique, so e.g. select!(df, :a, :a => :a) or select!(df, :a, :a => ByRow(sin) => :a) are not allowed.

In general columns returned by transformations are stored in the target data frame without copying. An exception to this rule is when columns from the source data frame are reused in the target data frame. This can happen via expressions like: :x1, [:x1, :x2], :x1 => :x2, :x1 => identity => :x2, or :x1 => (x -> @view x[inds]) (note that in the last case the source column is reused indirectly via a view). In such cases the behavior depends on the value of the copycols keyword argument:

  • if copycols=true then results of such transformations always perform a copy of the source column or its view;

  • if copycols=false then copies are only performed to avoid storing the same column several times in the target data frame; more precisely, no copy is made the first time a column is used, but each subsequent reuse of a source column (when compared using ===, which excludes views of source columns) performs a copy;

Note that performing transform! or select! assumes that copycols=false.

If df is a SubDataFrame and copycols=true then a DataFrame is returned and the same copying rules apply as for a DataFrame input: this means in particular that selected columns will be copied. If copycols=false, a SubDataFrame is returned without copying columns and in this case transforming or renaming columns is not allowed.

If a GroupedDataFrame is passed and threads=true (the default), a separate task is spawned for each specified transformation; each transformation then spawns as many tasks as Julia threads, and splits processing of groups across them (however, currently transformations with optimized implementations like sum and transformations that return multiple rows use a single task for all groups). This allows for parallel operation when Julia was started with more than one thread. Passed transformation functions must therefore not modify global variables (i.e. they must be pure), use locks to control parallel accesses, or threads=false must be passed to disable multithreading. In the future, parallelism may be extended to other cases, so this requirement also holds for DataFrame inputs.

In order to improve the performance of the operations some transformations invoke optimized implementation, see DataFrames.table_transformation for details.

Keyword arguments

  • renamecols::Bool=true : whether in the cols => function form automatically generated column names should include the name of transformation functions or not.

  • keepkeys::Bool=true : whether grouping columns of gd should be kept in the returned data frame.

  • ungroup::Bool=true : whether the return value of the operation on gd should be a data frame or a GroupedDataFrame.

  • threads::Bool=true : whether transformations may be run in separate tasks which can execute in parallel (possibly being applied to multiple rows or groups at the same time). Whether or not tasks are actually spawned and their number are determined automatically. Set to false if some transformations require serial execution or are not thread-safe.

Metadata: this function propagates table-level :note-style metadata. Column-level :note-style metadata is propagated if: a) a single column is transformed to a single column and the name of the column does not change (this includes all column selection operations), or b) a single column is transformed with identity or copy to a single column even if column name is changed (this includes column renaming). As a special case for GroupedDataFrame if the output has the same name as a grouping column and keepkeys=true, metadata is taken from original grouping column.

Examples

julia
julia> df = DataFrame(a=1:3, b=4:6)
3×2 DataFrame
 Row │ a      b
     │ Int64  Int64
─────┼──────────────
   1 │     1      4
   2 │     2      5
   3 │     3      6

julia> combine(df, :a => sum, nrow, renamecols=false)
1×2 DataFrame
 Row │ a      nrow
     │ Int64  Int64
─────┼──────────────
   1 │     6      3

julia> combine(df, :a => ByRow(sin) => :c, :b)
3×2 DataFrame
 Row │ c         b
     │ Float64   Int64
─────┼─────────────────
   1 │ 0.841471      4
   2 │ 0.909297      5
   3 │ 0.14112       6

julia> combine(df, :, [:a, :b] => (a, b) -> a .+ b .- sum(b)/length(b))
3×3 DataFrame
 Row │ a      b      a_b_function
     │ Int64  Int64  Float64
─────┼────────────────────────────
   1 │     1      4           0.0
   2 │     2      5           2.0
   3 │     3      6           4.0

julia> combine(df, All() .=> [minimum maximum])
1×4 DataFrame
 Row │ a_minimum  b_minimum  a_maximum  b_maximum
     │ Int64      Int64      Int64      Int64
─────┼────────────────────────────────────────────
   1 │         1          4          3          6

julia> using Statistics

julia> combine(df, AsTable(:) => ByRow(mean), renamecols=false)
3×1 DataFrame
 Row │ a_b
     │ Float64
─────┼─────────
   1 │     2.5
   2 │     3.5
   3 │     4.5

julia> combine(df, AsTable(:) => ByRow(mean) => x -> join(x, "_"))
3×1 DataFrame
 Row │ a_b
     │ Float64
─────┼─────────
   1 │     2.5
   2 │     3.5
   3 │     4.5

julia> combine(first, df)
1×2 DataFrame
 Row │ a      b
     │ Int64  Int64
─────┼──────────────
   1 │     1      4

julia> df = DataFrame(a=1:3, b=4:6, c=7:9)
3×3 DataFrame
 Row │ a      b      c
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      4      7
   2 │     2      5      8
   3 │     3      6      9

julia> combine(df, AsTable(:) => ByRow(x -> (mean=mean(x), std=std(x))) => :stats,
               AsTable(:) => ByRow(x -> (mean=mean(x), std=std(x))) => AsTable)
3×3 DataFrame
 Row │ stats                    mean     std
     │ NamedTup…                Float64  Float64
─────┼───────────────────────────────────────────
   1 │ (mean = 4.0, std = 3.0)      4.0      3.0
   2 │ (mean = 5.0, std = 3.0)      5.0      3.0
   3 │ (mean = 6.0, std = 3.0)      6.0      3.0

julia> df = DataFrame(a=repeat([1, 2, 3, 4], outer=[2]),
                      b=repeat([2, 1], outer=[4]),
                      c=1:8);

julia> gd = groupby(df, :a);

julia> combine(gd, :c => sum, nrow)
4×3 DataFrame
 Row │ a      c_sum  nrow
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      6      2
   2 │     2      8      2
   3 │     3     10      2
   4 │     4     12      2

julia> combine(gd, :c => sum, nrow, ungroup=false)
GroupedDataFrame with 4 groups based on key: a
First Group (1 row): a = 1
 Row │ a      c_sum  nrow
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      6      2
⋮
Last Group (1 row): a = 4
 Row │ a      c_sum  nrow
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     4     12      2

julia> combine(gd) do d # do syntax for the slower variant
           sum(d.c)
       end
4×2 DataFrame
 Row │ a      x1
     │ Int64  Int64
─────┼──────────────
   1 │     1      6
   2 │     2      8
   3 │     3     10
   4 │     4     12

julia> combine(gd, :c => (x -> sum(log, x)) => :sum_log_c) # specifying a name for target column
4×2 DataFrame
 Row │ a      sum_log_c
     │ Int64  Float64
─────┼──────────────────
   1 │     1    1.60944
   2 │     2    2.48491
   3 │     3    3.04452
   4 │     4    3.46574

julia> combine(gd, [:b, :c] .=> sum) # passing a vector of pairs
4×3 DataFrame
 Row │ a      b_sum  c_sum
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      4      6
   2 │     2      2      8
   3 │     3      4     10
   4 │     4      2     12

julia> combine(gd) do sdf # dropping group when DataFrame() is returned
          sdf.c[1] != 1 ? sdf : DataFrame()
       end
6×3 DataFrame
 Row │ a      b      c
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     2      1      2
   2 │     2      1      6
   3 │     3      2      3
   4 │     3      2      7
   5 │     4      1      4
   6 │     4      1      8

auto-splatting, renaming and keepkeys

julia
julia> df = DataFrame(a=repeat([1, 2, 3, 4], outer=[2]),
                      b=repeat([2, 1], outer=[4]),
                      c=1:8);

julia> gd = groupby(df, :a);

julia> combine(gd, :b => :b1, :c => :c1, [:b, :c] => +, keepkeys=false)
8×3 DataFrame
 Row │ b1     c1     b_c_+
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     2      1      3
   2 │     2      5      7
   3 │     1      2      3
   4 │     1      6      7
   5 │     2      3      5
   6 │     2      7      9
   7 │     1      4      5
   8 │     1      8      9

broadcasting and column expansion

julia
julia> df = DataFrame(a=repeat([1, 2, 3, 4], outer=[2]),
                      b=repeat([2, 1], outer=[4]),
                      c=1:8);

julia> gd = groupby(df, :a);

julia> combine(gd, :b, AsTable([:b, :c]) => ByRow(extrema) => [:min, :max])
8×4 DataFrame
 Row │ a      b      min    max
     │ Int64  Int64  Int64  Int64
─────┼────────────────────────────
   1 │     1      2      1      2
   2 │     1      2      2      5
   3 │     2      1      1      2
   4 │     2      1      1      6
   5 │     3      2      2      3
   6 │     3      2      2      7
   7 │     4      1      1      4
   8 │     4      1      1      8

julia> combine(gd, [:b, :c] .=> Ref) # preventing vector from being spread across multiple rows
4×3 DataFrame
 Row │ a      b_Ref      c_Ref
     │ Int64  SubArray…  SubArray…
─────┼─────────────────────────────
   1 │     1  [2, 2]     [1, 5]
   2 │     2  [1, 1]     [2, 6]
   3 │     3  [2, 2]     [3, 7]
   4 │     4  [1, 1]     [4, 8]

julia> combine(gd, AsTable(Not(:a)) => Ref) # protecting result
4×2 DataFrame
 Row │ a      b_c_Ref
     │ Int64  NamedTup…
─────┼─────────────────────────────────
   1 │     1  (b = [2, 2], c = [1, 5])
   2 │     2  (b = [1, 1], c = [2, 6])
   3 │     3  (b = [2, 2], c = [3, 7])
   4 │     4  (b = [1, 1], c = [4, 8])

julia> combine(gd, :, AsTable(Not(:a)) => sum, renamecols=false)
8×4 DataFrame
 Row │ a      b      c      b_c
     │ Int64  Int64  Int64  Int64
─────┼────────────────────────────
   1 │     1      2      1      3
   2 │     1      2      5      7
   3 │     2      1      2      3
   4 │     2      1      6      7
   5 │     3      2      3      5
   6 │     3      2      7      9
   7 │     4      1      4      5
   8 │     4      1      8      9

protecting returned vectors from being expanded into multiple rows

julia
julia> df = DataFrame(a=[1, 1, 2, 2],
                      b=[[1, 2], [2, 3], [3, 4], [4, 5]]);

julia> gd = groupby(df, :a);

julia> combine(gd, :b => Ref∘sum)
2×2 DataFrame
 Row │ a      b_Ref_sum
     │ Int64  Array…
─────┼──────────────────
   1 │     1  [3, 5]
   2 │     2  [7, 9]
source
ForecastEnsembles.effective_num_models Function
julia
effective_num_models(m) -> Float64 or DataFrame

The effective number of models in a weight vector: the participation ratio of the normalised weights p. It is 1 when all weight sits on one model and equals the model count M when the weights are equal, so it reads as "how many models is this ensemble really using".

m may be a fitted method exposing weights (e.g. FittedCRPSStacking, FittedStacking, FittedHedge; FittedPartialPooling uses its pooled global vector), an EnsembleWeights, a weights DataFrame, or a raw weight vector. For per-quantile weights it returns a DataFrame (output_type_id, effective_num_models) — one value per quantile level; otherwise a scalar.

Example

julia
using ForecastEnsembles
effective_num_models([0.6, 0.3, 0.1])
source
StatsAPI.fit Function

Fit a statistical model.

source
ForecastEnsembles.model_ids Function
julia
model_ids(ft::ForecastTable) -> Vector

The distinct model identifiers present in ft, taken from its model-id column.

Arguments

Example

julia
using ForecastEnsembles, DataFrames
df = DataFrame(
    location = "A", horizon = 1,
    model_id = repeat(["m1", "m2", "m3"], inner = 2),
    output_type = "quantile",
    output_type_id = repeat([0.25, 0.75], 3),
    value = [1.0, 3.0, 2.0, 4.0, 0.5, 2.5]
)
ft = ForecastTable(df; task_id_cols = [:location, :horizon])
model_ids(ft)
source
ForecastEnsembles.output_type Function
julia
output_type(ft::ForecastTable) -> Symbol

The single output_type present in ft. Throws if the table mixes output types, since most ensemble methods are defined on a single type at a time.

Arguments

Examples

julia
using ForecastEnsembles, DataFrames
df = DataFrame(
    location = "A", horizon = 1,
    model_id = repeat(["m1", "m2", "m3"], inner = 2),
    output_type = "quantile",
    output_type_id = repeat([0.25, 0.75], 3),
    value = [1.0, 3.0, 2.0, 4.0, 0.5, 2.5]
)
ft = ForecastTable(df; task_id_cols = [:location, :horizon])
output_type(ft)
source
ForecastEnsembles.task_id_cols Function
julia
task_id_cols(ft::ForecastTable) -> Vector{Symbol}

The task-id columns of ft, i.e. the columns that together identify a forecast target (e.g. :location, :horizon, :target_date).

Arguments

Example

julia
using ForecastEnsembles, DataFrames
df = DataFrame(
    location = "A", horizon = 1,
    model_id = repeat(["m1", "m2", "m3"], inner = 2),
    output_type = "quantile",
    output_type_id = repeat([0.25, 0.75], 3),
    value = [1.0, 3.0, 2.0, 4.0, 0.5, 2.5]
)
ft = ForecastTable(df; task_id_cols = [:location, :horizon])
task_id_cols(ft)
source
ForecastEnsembles.weight_stability Function
julia
weight_stability(m::FittedHedge) -> DataFrame

How much each model's Hedge weight moved over the training run: the total variation   of every model's weight along the fitted trajectory. A large value flags a model whose weight swung across the history (regime change or noise); a small one flags a stable contribution. Returns a DataFrame with columns model_id and total_variation.

Example

julia
using ForecastEnsembles, DataFrames
trajectory = DataFrame(
    model_id = repeat(["m1", "m2"], inner = 2),
    weight = [0.5, 0.7, 0.5, 0.3],
    t = [1, 2, 1, 2]
)
fitted = FittedHedge(
    DataFrame(model_id = ["m1", "m2"], weight = [0.7, 0.3]), ["m1", "m2"], trajectory)
weight_stability(fitted)
source
ForecastEnsembles.weights Function
julia
weights(m) -> Union{DataFrame, Nothing}

Per-model weights estimated by a fitted method, as a DataFrame with columns :model_id and :weight. Returns nothing when the fit does not correspond to a single weight vector on the simplex (e.g. unconstrained QRA, per-quantile QRA, QRA with a non-zero intercept).

When weights(m) !== nothing, m can be passed in place of an explicit weights frame to any method that accepts one — for example MixtureEnsemble(weights = m) or QuantileEnsemble(:mean; weights = m). This is the composition path between trained and untrained methods.

Arguments

Examples

julia
using ForecastEnsembles, DataFrames, Random
rng = MersenneTwister(1)
T = 20; K = 50
obs = DataFrame(t = 1:T, observed = randn(rng, T))
rows = DataFrame[]
for (mid, s) in (("m1", (y, r) -> y .+ randn(r, K)), ("m2", (y, r) -> 3 .* randn(r, K)))
    for t in 1:T
        push!(rows, DataFrame(model_id = mid, output_type = "sample",
            output_type_id = 1:K, t = t, value = s(obs.observed[t], rng)))
    end
end
ft = ForecastTable(reduce(vcat, rows); task_id_cols = [:t])
weights(fit(CRPSStacking(), ft, obs))
source