BVDOutbreakSize

Appearance

Forecast evaluation ​

How the forecasts on the forecasts page have scored against the data that arrived afterwards. Scoring is the continuous ranked probability score against a persistence baseline, defined in the forecast scoring Methods section. The same scoring by province is on the province forecast evaluation page and by health zone on the health-zone forecast evaluation page.

Load packages, data and fitted chains
julia
# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))
validation_forecast_from (generic function with 1 method)
julia
# The fits this page reads, loaded from the cache here.
chn_joint = load_fit("joint")
frozen_lastweek = load_fit("frozen_validation")
frozen_lastweek_streams = frozen_validation_stream_fits();

Summary ​

The overall bullets come first, then each stream's own scores, from the sections further down this page. Relative skill is the model's CRPS over the persistence baseline's, so a value below one beats the baseline. Coverage is the fraction of forecasts whose observed value falls inside the 90% predictive interval, nominally 0.9.

  • Joint model across releases: beat the baseline on 1 of 5 streams.

  • Frozen joint model: beat the baseline on 2 of 5 streams.

Across releases

  • confirmed cases: relative skill 6.59, 90% coverage 0.94, bias 0.3 over 100 forecasts.

  • confirmed deaths: relative skill 9.27, 90% coverage 0.92, bias -0.17 over 98 forecasts.

  • isolation beds: relative skill 1.43, 90% coverage 0.95, bias 0.19 over 95 forecasts.

  • onset reports: relative skill 0.94, 90% coverage 0.96, bias -0.1 over 24 forecasts.

  • recovered: relative skill 1.67, 90% coverage 1.0, bias -0.14 over 14 forecasts.

Frozen fits

  • confirmed cases: relative skill 2.42, 90% coverage 0.79, bias 0.27 over 247 forecasts.

  • confirmed deaths: relative skill 0.88, 90% coverage 0.87, bias -0.05 over 191 forecasts.

  • isolation beds: relative skill 0.78, 90% coverage 0.98, bias 0.1 over 217 forecasts.

  • onset reports: relative skill 1.02, 90% coverage 0.93, bias -0.11 over 28 forecasts.

  • recovered: relative skill 5.18, 90% coverage 0.69, bias 0.38 over 97 forecasts.

Forecast validation ​

How last week's forecast held up against the data since observed, using the frozen re-fit and one-week projection defined in forecast-versus-frozen evaluation. Only the streams the situation reports are still updating are validated here. A stream that has stopped being reported carries a cumulative total that repeats its last reported value, so there is no observation for the past week to score against. The frozen fit also conditions on the isolation beds, so the projected bed occupancy is scored against the beds held a week later. The bed validation is weak at a one-week-back freeze. The reported occupancy rate starts only on 9 June, so the capacity has no implied-capacity anchor and rides its random walk back to the freeze date. Like the scores further down, the confirmed new-count rows here take out any retrospective harmonisation step the week contained. Such a step reattaches records notified earlier, so it is not something the forecast was predicting. The cumulative rows are scored against the published total, harmonisation included.

Fit one week back and validate the one-week-ahead forecast
julia
# frozen_lastweek and frozen_lastweek_streams are computed in the setup
# block above, and `validation_forecast_from` is defined there.
validation_forecast = validation_forecast_from("frozen_validation");

# Each frozen individual (single-stream) fit's own one-week-ahead new-count
# forecast at the same cut-off as `frozen_lastweek`, from
# [`forecast_stream`](@ref) (the same per-stream forecaster
# `stream_forecasts.csv` uses), so the validation plots below can show the
# individual fit alongside the joint rather than the joint alone. Recovered
# has no individual fit and is absent here, as it is throughout this report.
# Only the still-reported streams are fitted at the validation cut-off, so
# a stream the situation reports have stopped updating is absent from
# `frozen_lastweek_streams` and carries no individual series here.
function _validation_individual_new(sid, stream::Symbol)
    haskey(frozen_lastweek_streams, sid) || return nothing
    return Float64.(
        forecast_stream(
            fit_forecast("frozen_validation_$sid"), stream; horizon = 7
        )
    )
end
validation_individual = NamedTuple(
    k => v
        for (k, v) in pairs(
            (;
                cases_new = _validation_individual_new(
                    "cases", :reported_cases
                ),
                deaths_new = _validation_individual_new(
                    "deaths", :suspected_deaths
                ),
                confirmed_new = _validation_individual_new(
                    "confirmed", :confirmed_cases
                ),
                confirmed_deaths_new = _validation_individual_new(
                    "confirmed_deaths", :confirmed_deaths
                ),
            )
        )
        if !isnothing(v)
)
# The frozen individual (treatment-only) fit's own bed-occupancy forecast.
# `nothing` when the beds have stopped being reported, so the treatment fit
# is absent; the bed panel then draws the joint alone.
validation_individual_isolation = haskey(frozen_lastweek_streams, "treatment") ?
    Float64.(
        forecast_stream(
            fit_forecast("frozen_validation_treatment"), :isolation_beds;
            horizon = 7
        )
    ) : nothing

# The observed beds at the current cut-off (the forecast target), so the
# frozen-fit bed forecast is scored against what the beds actually held.
# Held back once the beds stop being reported, since the last count would
# then be carried forward rather than observed at the target date.
_obs_beds = stream_reporting(obs, :isolation_beds) ?
    obs.isolation_history.counts[end] : missing
# Same observed/baseline keying as the plot below, so the table covers every
# fitted count stream (cumulative and new-count rows) plus the bed level.
# A harmonisation-break day between the frozen cut-off and the current one
# puts records into the confirmed cumulative that were never notified in that
# week, so the new-count truth carries a step the forecast was never
# predicting. Take it out, the same correction `score_releases.jl` applies.
# Grid days are relative to a seeding date fixed by the genetic tmrca, so the
# frozen fit's own `n` and the current `obs.n` index the same grid.
validation_breaks = (
    confirmed_cum = confirmed_break_correction(
        obs, frozen_lastweek.o.n, obs.n
    ),
    confirmed_deaths_cum = confirmed_break_correction(
        obs, frozen_lastweek.o.n, obs.n; deaths = true
    ),
)

# Observed cumulative at the target date per stream, keyed by the forecast's
# cumulative column; `baseline` is each stream's origin cumulative (the
# frozen cut-off), so the new count is scored against observed minus origin,
# less any harmonisation the window carries (see `validation_breaks`). Both
# the table and the plot below take the still-reported streams
# (`reporting_cum_cols`, from the setup block): a stream the situation
# reports have stopped updating has an origin and a target reading the same
# repeated total, so its cumulative truth is stale and its new-count truth is
# a guaranteed zero.
validation_observed = (
    cases_cum = obs.reported_cases,
    deaths_cum = obs.total_deaths,
    confirmed_cum = obs.confirmed_cases,
    confirmed_deaths_cum = obs.confirmed_deaths,
    recovered_cum = obs.recovered_cases,
)
validation_baseline = (
    cases_cum = frozen_lastweek.o.reported_cases,
    deaths_cum = frozen_lastweek.o.total_deaths,
    confirmed_cum = frozen_lastweek.o.confirmed_cases,
    confirmed_deaths_cum = frozen_lastweek.o.confirmed_deaths,
    recovered_cum = frozen_lastweek.o.recovered_cases,
)

validation_table = forecast_vs_truth(
    validation_forecast;
    observed = keep_streams(validation_observed, reporting_cum_cols),
    baseline = keep_streams(validation_baseline, reporting_cum_cols),
    breaks = validation_breaks,
    isolation = _obs_beds
);
Forecast-versus-observed validation table
StreamQuantityObservedLower 90%Lower 60%Lower 30%Upper 30%Upper 60%Upper 90%Within 90% PI
DRC confirmed casescumulative by T+78067815982578315844085208683no
DRC confirmed casesnew this week3954875856437688481011no
DRC confirmed deathscumulative by T+73901393539663989402640504101no
DRC confirmed deathsnew this week202236267290327351402no
DRC recovered among confirmedcumulative by T+72070201222312308242725072742yes
DRC recovered among confirmednew this week191133352429548628863yes
DRC isolation bedsoccupancy at T+77736887768289039531054yes

The observation panels histogram the one-week-ahead forecast made from the frozen fit: a cumulative and a new-count panel for each still-reported count stream the forecast carries. The 90% predictive interval is shaded, and the count observed by the current cut-off is a dashed black rule. Where a stream has its own individual (single-stream) fit, that fit's forecast from the same frozen cut-off is overlaid as a dotted step outline on the joint's own histogram bins.

Forecast-versus-observed plot
julia
validation_fig = plot_forecast_vs_truth(
    validation_forecast;
    observed = keep_streams(validation_observed, reporting_cum_cols),
    baseline = keep_streams(validation_baseline, reporting_cum_cols),
    breaks = validation_breaks,
    individual = keep_streams(validation_individual, reporting_cum_cols)
);

The bed panel scores last week's projected occupancy against the beds occupied now (the dashed rule), with the individual (treatment-only) fit's own projection overlaid as a dotted step outline on the joint's own histogram bins.

Bed forecast-versus-observed plot
julia
validation_beds_fig = plot_forecast_beds_vs_truth(
    validation_forecast;
    isolation = _obs_beds, individual = validation_individual_isolation
);

The latent quantities are not observed, so they are scored distribution against distribution: what the frozen fit forecast for the past week's new infections, onsets and deaths against what the current fit now estimates for the same window.

Forecast-versus-now latent plot
julia
# Current fit's draws of the new latent counts over the past week, the last
# seven days of each cumulative-trajectory deterministic.
function _now_new(chn, key)
    mat = chn[key]
    trajs = [collect(v) for v in vec(collect(mat))]
    return Float64[t[end] - t[max(1, length(t) - 7)] for t in trajs]
end
now_latent = (;
    infections_new = _now_new(chn_joint, :cumulative_infections),
    onsets_new = _now_new(chn_joint, :cumulative_onsets),
    deaths_latent_new = _now_new(chn_joint, :cumulative_expected_deaths),
)

validation_latent_fig = plot_forecast_vs_truth_latent(
    validation_forecast; now = now_latent
);

Streams no longer reported ​

The situation reports have stopped updating some of the streams the model fits, listed with the date each was last reported below. The panels show what the frozen fit projected for those streams over the same week, without an observed rule, since the count they would be scored against has not moved since the stream stopped.

Forecast for the streams no longer reported
julia
# The last-reported date per stopped stream, and the frozen fit's own
# projection for them. `plot_forecast` draws a panel per new-count column
# the frame carries, so passing the stopped streams' columns alone gives the
# projection without the fabricated truth rule the validation figure would
# otherwise draw against a repeated total.
validation_stopped_streams = let s = stream_report_status(obs),
        ids = [stream_id(c) for c in stopped_cum_cols]

    keep = [r.stream in ids for r in eachrow(s)]
    DataFrame(
        "Stream" => s[keep, :label],
        "Last reported" => s[keep, :last_date]
    )
end
_stopped_new_cols = [
    c
        for c in new_cols(stopped_cum_cols)
        if c in propertynames(validation_forecast)
]
validation_stopped_fig = plot_forecast(
    validation_forecast[!, _stopped_new_cols]
);
StreamLast reported
Suspected cases2026-05-26
Suspected deaths2026-05-26

Forecast scoring across releases ​

Every release's saved one- to four-week-ahead forecast is scored against the data observed since, against a persistence baseline and, where one exists, the stream's own individual fit as well as the joint. The tables in this section are the joint model's, one row per stream. See forecast scoring against a persistence baseline for how the scores, the relative skill and the baseline are built. Recovered has no individual fit of its own, so its comparison is the baseline against the joint only. Reported cases and suspected deaths stopped being updated by the situation reports partway through the outbreak, and exports' confirmed-detection series is anchored to an earlier cut-off. Exports therefore contributes no scored forecast, and reported cases and suspected deaths each rest on exactly one matched forecast, a single window rather than a settled sample.

Only a minority of the daily releases examined contribute a row to the table below, each a reconstruction of an earlier model version rather than the current fit. Only the newest few releases carry the current model's own individual-stream forecasts, and the backfilled reconstructions carry none at all. Every row also rests on one to a handful of matched forecasts, shown as its own count rather than rounded away.

Four things are excluded from the scores here and in the frozen section below, each for a stated reason rather than for scoring badly, and the second of them applies to the frozen section alone.

  • One whole reconstruction (results-v1.6.0): its chain forecasts a near-zero median at every horizon and stream, with the upper predictive tail occasionally reaching five- and six-digit values, which is the signature of a chain that failed to sample rather than a forecast.

  • Frozen section only: the confirmed-death rows of the fourteen frozen reconstructions cut between 16 July and 15 August 2026, whose forecaster could not project that stream from its own trajectory and floored it at zero, so each carries a one-week median of exactly zero against an observed 250 to 370. Reconstructions cut after that window project the stream normally.

  • An onset window containing a vintage whose reread total falls, since its increment is not what the situation reports added.

  • A stream that carries no persistence baseline, which is what makes a window scoreable at all.

Nothing is dropped from the archive itself; data/forecast_scores*.csv and data/forecast_overlay*.csv record everything that was scored.

The symptom-onset stream is scored on the new reported count each vintage adds rather than on its level, because every vintage rereads the whole figure. Its printed total therefore moves with the scan error as well as with late reporting. On fourteen vintages the reread total falls, which a cumulative onset curve cannot do, and on many others it repeats unchanged. The scored truth cannot absorb that, since it is the increment between the vintages at the two ends of a window. A window containing a falling vintage is therefore left unscored, the rule the province scores already apply to a window holding a harmonisation-break day. It bites hardest at the longer horizons, a four-week window being more likely to contain a reread than a one-week one: the frozen onset row keeps three of its twenty-nine windows, all at one week, and the cross-release row six of thirty-eight. Read the onset row's skill against the baseline rather than its coverage, and read it as resting on a handful of windows.

Load and summarise the cross-release forecast scores
julia
forecast_scores_df = _release_data(
    "forecast_scores.csv",
    (;
        release = String, made_date = Date, stream = String, horizon = Int,
        target_date = Date, fit = String, crps = Float64,
        log_crps = Float64, dispersion = Float64, overprediction = Float64,
        underprediction = Float64, coverage_50 = Float64,
        coverage_90 = Float64,
        bias = Float64, n_samples = Int,
        log_rel_to_baseline = Float64,
    )
)
forecast_overlay_df = _release_data(
    "forecast_overlay.csv",
    (;
        release = String, made_date = Date, stream = String, horizon = Int,
        target_date = Date, fit = String, observed = Float64,
        median = Float64, lo30 = Float64, hi30 = Float64, lo60 = Float64,
        hi60 = Float64, lo90 = Float64, hi90 = Float64,
    )
)
# The digitised onset triangle's own per-vintage total, as calendar dates,
# and the windows it makes unscoreable. A vintage that rereads the figure
# lower gives a window an increment the reports did not add, so the window
# is dropped from the scores and the overlay alike (see
# `drop_rescanned_onset_windows`). The current triangle is used to judge
# every release, since the scored truth is read off this one series.
_onset_vintage_dates = grid_date.(obs.onset_report_history.days)
_onset_vintage_totals = obs.onset_report_history.counts
_drop_rescanned(tbl) = drop_rescanned_onset_windows(
    tbl; vintage_dates = _onset_vintage_dates,
    vintage_totals = _onset_vintage_totals
)
forecast_scores_df = _drop_rescanned(forecast_scores_df)
forecast_overlay_df = _drop_rescanned(forecast_overlay_df)

# One row per (stream, fit) pooled over every horizon and release. The
# by-horizon and by-release detail tables carry the same columns at a finer
# grain (see src/scoring.jl). Every fit is kept here, since the
# relative-skill figure below compares the roles against each other. The
# tables rendered in this section select the joint role, and the individual
# fits are tabulated in their own section.
forecast_score_overview_table = forecast_score_overview(forecast_scores_df)
forecast_score_by_horizon_table = forecast_score_by_horizon(forecast_scores_df)
forecast_score_by_release_table = forecast_score_by_release(forecast_scores_df)

joint_score_overview_table = select_fit_role(
    forecast_score_overview_table, "joint"
)
joint_score_by_horizon_table = select_fit_role(
    forecast_score_by_horizon_table, "joint"
)
# The trailing `;` on this last assignment matters: without it, this whole
# setup chunk's last statement (the DataFrame it assigns) is Literate's
# implicitly displayed "result" for the chunk, on top of the deliberate
# display further down -- and a bare DataFrame is html-showable, so it
# goes out as a second, undisplayed-in-source `@raw html` block that (for
# a table this size) can itself hit the PCRE limit described above.
joint_score_by_release_table = select_fit_role(
    forecast_score_by_release_table, "joint"
);

The headline pools every horizon and release into one row per stream for the joint model: the mean CRPS and its decomposition, coverage, bias, and the relative skill against the persistence baseline, on both the natural and the log scale. Each row also carries relative skill against the stream's own individual fit where one exists.

streamfitncrpsrel_to_baselinelog_crpslog_rel_to_baselinerel_to_individuallog_rel_to_individualdispersionoverpredictionunderpredictioncoverage_50coverage_90bias
confirmed casesjoint100892.216.590.2191.970.770.7813.2478.530.440.710.940.3
confirmed deathsjoint98890.289.270.46530.810.99786.6616.0187.610.370.92-0.17
isolation bedsjoint9580.811.430.11.330.840.6751.7323.395.690.660.950.19
onset reportsjoint24131.550.940.1960.90.650.6178.1411.941.520.710.96-0.1
recoveredjoint14111.511.670.3061.196.16510.3611-0.14

The same relative skill against the baseline, by horizon: one panel per stream, one series per fit role, on a log-scaled skill axis with the reference line at one.

julia
forecast_relative_skill_fig = plot_forecast_relative_skill(
    forecast_score_by_horizon_table
);

What that error is made of, by horizon: the mean CRPS split into its width, its overprediction and its underprediction, one stacked bar per horizon and fit role.

julia
forecast_crps_by_horizon_fig = plot_forecast_crps_by_horizon(
    forecast_score_by_horizon_table
);

Scores by horizon
streamhorizonfitncrpsrel_to_baselinelog_crpslog_rel_to_baselinerel_to_individuallog_rel_to_individualdispersionoverpredictionunderpredictioncoverage_50coverage_90bias
confirmed cases7joint31115.742.740.242.770.660.6669.0746.050.620.650.840.37
confirmed cases14joint27217.372.860.2112.660.680.67155.0361.850.490.740.960.27
confirmed cases21joint23408.782.430.1931.540.740.71320.588.010.280.7810.27
confirmed cases28joint193703.2511.170.2261.260.880.833559.26143.720.270.6810.28
confirmed deaths7joint3174.772.930.484.391.060.9627.6211.8435.310.390.81-0.08
confirmed deaths14joint26165.942.780.4393.480.940.8969.0718.2678.60.310.92-0.13
confirmed deaths21joint22310.692.40.4532.520.820.99174.0715.42121.190.271-0.23
confirmed deaths28joint193883.1717.460.4892.030.691.193716.3820.41146.390.531-0.28
isolation beds7joint2973.931.780.0891.630.730.5635.5732.256.110.480.930.29
isolation beds14joint2578.481.580.0941.430.830.6549.2524.414.830.640.960.19
isolation beds21joint2283.351.270.1021.180.90.7360.5516.356.450.820.950.12
isolation beds28joint1991.421.160.121.150.960.869.4316.75.290.790.950.12
onset reports7joint1085.450.640.2470.750.750.6255.053.5526.840.90.9-0.09
onset reports14joint8141.941.370.1651.370.620.6278.115.6548.190.621-0.06
onset reports21joint6194.560.970.150.940.580.58116.6720.8257.080.51-0.15
recovered7joint618.111.50.1851.514.790.732.5911-0.09
recovered14joint389.71.950.3931.4668.015.8115.8711-0.23
recovered21joint3297.163.260.4221.16269.916.0511.211-0.09
recovered28joint2145.990.650.3670.57121.86024.1311-0.26

The same relative skill against the baseline, release by release, so a run of releases that lost to the baseline reads as a run rather than as an average.

julia
forecast_skill_by_cutoff_fig = plot_forecast_skill_by_cutoff(
    forecast_score_by_release_table;
    title = "Relative skill against the baseline, by release"
);

Scores by release
made_datestreamfitncrpsrel_to_baselinelog_crpslog_rel_to_baselinerel_to_individuallog_rel_to_individualdispersionoverpredictionunderpredictioncoverage_50coverage_90bias
2026-06-07confirmed casesjoint2363.9112.530.91310.43341.0622.860110.17
2026-06-10confirmed casesjoint3534.876.730.7915.08435.6799.20110.27
2026-07-01confirmed casesjoint415862.7874.230.5832.8215377.31485.470010.66
2026-07-06confirmed casesjoint4421.281.320.2690.96311.85109.4300.510.49
2026-07-08confirmed casesjoint4390.981.220.2530.96239.5151.4700.2510.52
2026-07-23confirmed casesjoint4335.570.880.1690.671.020.82299.2735.680.62110.13
2026-07-25confirmed casesjoint4222.50.980.1340.980.640.63218.044.460110.07
2026-07-26confirmed casesjoint4235.691.120.1411.120.790.67229.655.680.36110.06
2026-07-27confirmed casesjoint4247.971.280.1421.120.930.83240.087.880110.11
2026-07-31confirmed casesjoint4259.642.130.1491.930.640.68235.7323.760.15110.13
2026-08-01confirmed casesjoint4231.262.020.1392.030.690.66226.844.310.11110.05
2026-08-02confirmed casesjoint4278.452.020.1451.680.940.85263.7714.680110.14
2026-08-03confirmed casesjoint4247.581.280.140.940.850.8237.889.70110.08
2026-08-04confirmed casesjoint4221.562.430.1332.40.630.69220.70.860110.03
2026-08-07confirmed casesjoint4213.542.70.142.20.690.73209.7603.7811-0.09
2026-08-11confirmed casesjoint4197.494.430.1353.920.590.71191.5305.9711-0.13
2026-08-15confirmed casesjoint4387.776.540.1933.81.240.91294.0293.750110.37
2026-08-17confirmed casesjoint4458.716.140.2223.541.530.97290.38168.3300.510.46
2026-08-22confirmed casesjoint4336.095.030.1783.5510.87228.94107.1500.7510.45
2026-08-24confirmed casesjoint4246.114.040.1443.420.780.77118.34127.770010.62
2026-08-25confirmed casesjoint4198.573.210.122.790.660.65108.4890.10010.56
2026-08-30confirmed casesjoint397.181.930.0731.370.380.4173.8223.360110.31
2026-08-31confirmed casesjoint394.411.80.0721.490.460.4575.3519.060110.22
2026-09-01confirmed casesjoint377.231.260.0611.030.320.3568.48.840110.13
2026-09-06confirmed casesjoint296.613.080.0972.630.610.5557.339.3200.510.5
2026-09-10confirmed casesjoint2160.441.590.1731.490.740.7851.24109.20010.78
2026-09-12confirmed casesjoint2332.382.210.361.910.460.5264.18268.210000.95
2026-09-13confirmed casesjoint1183.193.020.3112.70.450.5136.62146.570000.92
2026-09-15confirmed casesjoint1227.822.30.4162.020.560.6243.47184.360000.97
2026-09-16confirmed casesjoint1217.342.780.4012.420.580.6442.14175.20000.99
2026-09-19confirmed casesjoint1300.576.040.5474.861.011.0146.48254.090000.99
2026-06-07confirmed deathsjoint141.491.730.9032.3936.9404.5511-0.2
2026-06-10confirmed deathsjoint264.932.280.7353.0662.70.991.2411-0.03
2026-07-01confirmed deathsjoint41719679.780.5491.1917103.1192.890110.28
2026-07-06confirmed deathsjoint4197.20.910.8712.37110.35086.850.251-0.49
2026-07-08confirmed deathsjoint4177.590.80.7131.8984.29093.30.251-0.61
2026-07-23confirmed deathsjoint4271.491.750.4842.60.391.08172.61098.8811-0.45
2026-07-25confirmed deathsjoint4241.832.020.5043.451.022.15106.460135.3701-0.57
2026-07-26confirmed deathsjoint4268.462.40.5544.391.052.17101.590166.8701-0.64
2026-07-27confirmed deathsjoint4249.472.320.5344.070.82.12109.30140.1701-0.59
2026-07-31confirmed deathsjoint4274.552.370.564.160.881.94108.790165.760.251-0.58
2026-08-01confirmed deathsjoint4313.482.680.6834.581.122.794.480219.0101-0.68
2026-08-02confirmed deathsjoint4258.172.750.5074.371.011.83128.190129.980.251-0.53
2026-08-03confirmed deathsjoint4268.891.940.582.891.22.37123.560145.330.251-0.55
2026-08-04confirmed deathsjoint4298.064.270.65681.22.3293.540204.5201-0.67
2026-08-07confirmed deathsjoint4366.295.110.888.341.582.7176.060290.2300.75-0.79
2026-08-11confirmed deathsjoint4301.1713.150.79824.011.232.3796.550204.6101-0.7
2026-08-15confirmed deathsjoint4205.935.140.325.320.840.94180.35025.5811-0.26
2026-08-17confirmed deathsjoint4202.494.110.3044.030.811.02187.02015.4711-0.21
2026-08-22confirmed deathsjoint4178.9820.3052.460.740.63156.16022.8211-0.26
2026-08-24confirmed deathsjoint492.941.350.1051.240.420.1945.4247.5100.2510.57
2026-08-25confirmed deathsjoint4741.080.0841.050.30.1145.0828.9200.7510.41
2026-08-30confirmed deathsjoint347.61.060.0751.170.220.0827.2920.3100.6710.42
2026-08-31confirmed deathsjoint360.230.920.0990.990.30.1329.930.330010.57
2026-09-01confirmed deathsjoint353.541.060.0851.110.260.0929.1324.4100.3310.54
2026-09-06confirmed deathsjoint252.442.890.1052.410.280.0919.6332.8200.510.57
2026-09-10confirmed deathsjoint296.571.290.2221.190.780.320.6575.92000.50.9
2026-09-12confirmed deathsjoint2122.21.620.2891.412.252.1522.2999.920000.94
2026-09-13confirmed deathsjoint163.971.660.2411.571.961.8913.3950.580000.91
2026-09-15confirmed deathsjoint176.481.880.3091.772.892.7513.5862.90001
2026-09-16confirmed deathsjoint179.121.730.3271.622.772.6114.964.220000.98
2026-09-19confirmed deathsjoint182.475.280.3294.645.334.6812.8469.620000.95
2026-07-01isolation bedsjoint490.191.450.1381.4813.48076.7100-0.99
2026-07-06isolation bedsjoint442.870.660.0660.6721.07021.801-0.64
2026-07-08isolation bedsjoint454.751.510.0731.5111.6743.080010.8
2026-07-23isolation bedsjoint480.062.750.1012.661.441.1429.3950.670010.58
2026-07-25isolation bedsjoint480.842.10.1052.061.561.2832.6548.1900.510.53
2026-07-26isolation bedsjoint473.831.990.0971.931.20.9636.9636.8700.510.46
2026-07-27isolation bedsjoint471.582.470.0932.371.20.9539.5232.0600.510.44
2026-07-31isolation bedsjoint471.051.670.0881.620.860.6751.6919.3600.7510.34
2026-08-01isolation bedsjoint4721.130.0941.10.810.6164.517.480.02110.18
2026-08-02isolation bedsjoint483.691.650.1021.50.990.7252.6731.0200.7510.39
2026-08-03isolation bedsjoint478.831.740.0991.610.990.7460.4218.410110.3
2026-08-04isolation bedsjoint4100.741.480.1371.421.160.9863.4737.2800.7510.38
2026-08-07isolation bedsjoint483.811.390.1221.550.80.7169.03014.7811-0.24
2026-08-11isolation bedsjoint474.710.440.1090.440.730.6472.4802.2311-0.08
2026-08-15isolation bedsjoint480.960.940.0920.850.630.4571.729.240110.17
2026-08-17isolation bedsjoint495.081.390.1061.210.920.6667.3527.740110.33
2026-08-22isolation bedsjoint493.151.50.0981.30.640.4578.9814.180110.23
2026-08-24isolation bedsjoint490.981.810.0961.540.720.5366.324.6700.7510.34
2026-08-25isolation bedsjoint475.541.10.0820.960.570.4371.773.740.02110.08
2026-08-30isolation bedsjoint371.791.760.0791.580.580.4754.6717.1200.6710.31
2026-08-31isolation bedsjoint360.951.330.0691.260.520.4354.186.230.54110.15
2026-09-01isolation bedsjoint370.871.850.0871.970.610.5757.32013.5511-0.22
2026-09-06isolation bedsjoint253.071.090.0631.080.470.4344.0409.0311-0.21
2026-09-10isolation bedsjoint253.641.330.0611.290.40.3544.40.278.9611-0.18
2026-09-12isolation bedsjoint2144.422.340.1512.131.090.8462.0182.4100.510.58
2026-09-13isolation bedsjoint1131.392.730.1392.531.24152.8978.510010.68
2026-09-15isolation bedsjoint192.682.730.0942.550.830.6651.5241.160010.57
2026-09-16isolation bedsjoint194.953.970.0953.570.840.6656.5138.440010.53
2026-09-19isolation bedsjoint1266.1340.2823.542.632.1153.06213.080000.91
2026-08-02onset reportsjoint380.680.590.1010.60.760.5977.9102.7711-0.13
2026-08-03onset reportsjoint387.11.310.1190.970.880.7676.4910.610110.22
2026-08-04onset reportsjoint383.710.680.130.630.610.574.122.177.42110
2026-08-07onset reportsjoint3279.150.850.3880.840.670.59690210.1500.67-0.83
2026-08-11onset reportsjoint3192.721.20.2851.240.80.6783.450109.260.331-0.59
2026-08-15onset reportsjoint3173.321.950.1591.340.470.57107.0566.2600.3310.44
2026-08-17onset reportsjoint2102.221.630.1331.260.420.5378.8323.390110.27
2026-08-22onset reportsjoint268.050.580.1340.740.570.6567.4400.6111-0.05
2026-08-24onset reportsjoint163.90.890.3031.80.760.7857.506.411-0.2
2026-08-25onset reportsjoint162.880.280.3120.560.670.6261.191.690110.08
2026-07-01recoveredjoint3312.696.630.4461.94289.5523.140110.27
2026-07-06recoveredjoint485.760.80.4070.8868.39017.3711-0.33
2026-07-08recoveredjoint463.080.760.292144.51018.5711-0.38
2026-09-15recoveredjoint18.921.050.0490.968.810.110110.09
2026-09-16recoveredjoint18.660.80.0490.78.250.410110.15
2026-09-19recoveredjoint110.180.660.0560.638.8901.2911-0.23

Forecasts made at each release against the value observed since, one panel per stream and horizon, the observed value in black. The median and 90% interval are coloured by fit role: the persistence baseline, the stream's individual fit and the joint. The x-axis is the date each forecast was made, so an incident stream's observed window pairs unambiguously with the forecast that made it. Each panel's axis is cropped to a small multiple of what that stream actually reached, so one very wide interval cannot squash every other series flat. An interval or median too wide for the panel is clamped at the top and marked with an open triangle rather than silently cut off.

Forecasts-versus-now overlay
julia
forecast_overlay_fig = plot_forecast_overlay(
    scored_overlay(forecast_overlay_df)
);

Frozen-fit forecast evaluation ​

The current model, frozen at earlier data cut-offs (see Forecast-versus-frozen evaluation), is scored the same way as the cross-release forecasts above, against the same persistence baseline. The tables in this section are the frozen joint model's, one row per stream. Each stream's own frozen fit is also scored where one exists, at the one-week-back cut-off and for still-reported streams only, and is carried by the skill figures rather than by the tables. Every other cut-off carries the joint alone. The May cut-offs predate the first reported bed occupancy and the first reported recoveries, so those windows are left unscored rather than scored against a series that had not started. The baseline carries a weaker data-vintage guarantee than the cross-release one, since its snapshot was taken weeks after the frozen cut-off and can hold later revisions to earlier days (see forecast scoring against a persistence baseline).

Load and summarise the frozen-fit forecast scores
julia
frozen_scores_df = _release_data(
    "forecast_scores_frozen.csv",
    (;
        release = String, made_date = Date, stream = String, horizon = Int,
        target_date = Date, fit = String, crps = Float64,
        log_crps = Float64, dispersion = Float64, overprediction = Float64,
        underprediction = Float64, coverage_50 = Float64,
        coverage_90 = Float64,
        bias = Float64, n_samples = Int,
        log_rel_to_baseline = Float64,
    )
)
frozen_overlay_df = _release_data(
    "forecast_overlay_frozen.csv",
    (;
        release = String, made_date = Date, stream = String, horizon = Int,
        target_date = Date, fit = String, observed = Float64,
        median = Float64, lo30 = Float64, hi30 = Float64, lo60 = Float64,
        hi60 = Float64, lo90 = Float64, hi90 = Float64,
    )
)
# Rows a superseded forecaster produced are dropped before anything is
# summarised or drawn, from the scores and the overlay alike, so the
# tables and the figures rest on one set of rows. See
# `drop_superseded_forecasts` for the one exclusion in force and why.
frozen_scores_df = _drop_rescanned(
    drop_superseded_forecasts(frozen_scores_df)
)
frozen_overlay_df = _drop_rescanned(
    drop_superseded_forecasts(frozen_overlay_df)
)

# The frozen joint carries `FROZEN_FIT`, so it is named as the joint role
# here and compared against each stream's own frozen fit. A release
# published before the archive had a `fit` column carries joint rows only.
frozen_score_overview_table = forecast_score_overview(
    frozen_scores_df; joint_fit = FROZEN_FIT
)
frozen_score_by_horizon_table = forecast_score_by_horizon(
    frozen_scores_df; joint_fit = FROZEN_FIT
)
frozen_score_by_release_table = forecast_score_by_release(
    frozen_scores_df; joint_fit = FROZEN_FIT
)

# The tables show the frozen joint alone, as the cross-release tables show
# the joint alone, so a row reads as one model at one cut-off rather than a
# stream interleaving two fits. The archive's single-stream frozen fits stay
# in the scored data and in the figures, which compare the roles against
# each other. Selecting the joint role leaves `fit` single-valued, so it is
# dropped and the model named in the prose instead.
_frozen_joint_only(tbl) = drop_degenerate_fit_column(
    select_fit_role(tbl, "joint")
)
frozen_score_overview_display = _frozen_joint_only(
    frozen_score_overview_table
)
frozen_score_by_horizon_display = _frozen_joint_only(
    frozen_score_by_horizon_table
)
frozen_score_by_release_display = _frozen_joint_only(
    frozen_score_by_release_table
)

# One row per release for the cut-offs more than one release forecast.
# See the comment above `joint_score_by_release_table`'s assignment for why
# this setup chunk's last statement needs a trailing `;`.
frozen_score_by_vintage_table = forecast_score_by_vintage(
    frozen_scores_df; joint_fit = FROZEN_FIT
)
frozen_score_by_vintage_display = _frozen_joint_only(
    frozen_score_by_vintage_table
);
streamncrpsrel_to_baselinelog_crpslog_rel_to_baselinerel_to_individuallog_rel_to_individualdispersionoverpredictionunderpredictioncoverage_50coverage_90bias
confirmed cases247292.732.420.3641.150.470.55151.03124.916.810.410.790.27
confirmed deaths19146.850.880.3690.582.262.1927.813.995.050.540.87-0.05
isolation beds21789.040.780.140.580.760.645.9424.9118.180.50.980.1
onset reports28145.821.020.2320.8673.4221.2951.10.570.93-0.11
recovered97296.035.180.5453.35152.09140.853.090.580.690.38

The same relative skill against the baseline, by horizon, for the frozen cut-offs.

julia
frozen_relative_skill_fig = plot_forecast_relative_skill(
    frozen_score_by_horizon_table
);

What that error is made of, by horizon, as in the cross-release section above.

julia
frozen_crps_by_horizon_fig = plot_forecast_crps_by_horizon(
    frozen_score_by_horizon_table;
    title = "CRPS decomposition by horizon, frozen cut-offs"
);

Scores by horizon
streamhorizonncrpsrel_to_baselinelog_crpslog_rel_to_baselinerel_to_individuallog_rel_to_individualdispersionoverpredictionunderpredictioncoverage_50coverage_90bias
confirmed cases712093.741.110.3960.780.490.5634.8124.3434.590.40.71-0.06
confirmed cases1456263.283.810.2853.360.460.52140.12123.1500.460.890.57
confirmed cases2152610.683.920.4222.66289.25321.4400.270.770.65
confirmed cases2819766.131.880.2391.14538.92227.2100.6810.41
confirmed deaths710619.840.80.510.622.312.247.123.649.090.470.78-0.36
confirmed deaths144247.961.120.1730.72.222.131.6416.270.050.790.980.3
confirmed deaths213898.540.720.2210.3471.427.1400.5510.34
confirmed deaths285217.351.830.1681.63102.93114.420010.63
isolation beds76058.321.330.0940.940.550.4529.4826.032.810.620.920.31
isolation beds145679.771.540.1291.091.240.9743.7832.923.070.6810.32
isolation beds215295.440.570.1510.4351.2719.2824.890.351-0.12
isolation beds2849130.450.60.1970.4562.9120.3847.150.311-0.2
onset reports71498.90.680.2720.752.9314.8731.110.570.93-0.09
onset reports1410146.141.340.1731.2680.817.4447.90.70.9-0.07
onset reports214309.221.390.2381.34126.7153.42129.090.251-0.27
recovered730117.55.740.5353.8542.8173.031.660.470.80.45
recovered1426243.968.620.5485.3396.3145.182.480.540.580.43
recovered2122363.064.870.5462.95176.98182.423.660.640.640.34
recovered2819571.554.260.5542.17372.16193.895.50.740.740.26

The same relative skill against the baseline, cut-off by cut-off, pooled over the horizons each cut-off forecast.

julia
frozen_skill_by_cutoff_fig = plot_forecast_skill_by_cutoff(
    frozen_score_by_release_table;
    xlabel = "Frozen cut-off",
    title = "Relative skill against the baseline, by frozen cut-off"
);

Scores by frozen cut-off
made_datestreamncrpsrel_to_baselinelog_crpslog_rel_to_baselinerel_to_individuallog_rel_to_individualdispersionoverpredictionunderpredictioncoverage_50coverage_90bias
2026-05-23confirmed cases3028.110.340.1810.319.7308.370.71-0.32
2026-05-27confirmed cases30141.720.790.9450.7612.130129.5900.1-0.94
2026-06-08confirmed cases90367.577.440.4024.21157.96209.6200.260.820.68
2026-07-16confirmed cases4294.680.470.1680.29283.4610.980.24110.07
2026-07-18confirmed cases4361.260.780.1850.52333.8527.410110.15
2026-07-19confirmed cases4439.621.090.2090.72351.8387.790110.3
2026-07-20confirmed cases4385.620.870.1850.56332.653.020110.22
2026-07-24confirmed cases4422.921.340.1920.95343.779.220110.32
2026-07-25confirmed cases4460.722.030.2091.54356.82103.910110.34
2026-07-26confirmed cases4315.561.480.1641.27284.5930.970110.24
2026-07-27confirmed cases4346.391.760.1761.33306.9739.420110.27
2026-07-28confirmed cases4396.781.340.1560.84361.5235.260110.22
2026-07-31confirmed cases4498.384.080.2132.74369.58128.800.7510.38
2026-08-04confirmed cases4281.572.970.1562.7253.9927.590110.18
2026-08-08confirmed cases4253.513.220.152.27246.676.120.72110.03
2026-08-10confirmed cases4244.513.740.152.782411.71.8111-0.01
2026-08-15confirmed cases4623.411.740.2816.2384.1239.300.2510.52
2026-08-17confirmed cases4426.555.490.2343.51166.54260.010010.75
2026-08-18confirmed cases4468.253.910.2762.8155.86312.4000.750.85
2026-08-23confirmed cases4562.747.990.3146.34140.78421.96000.750.88
2026-08-24confirmed cases4437.046.820.2465.48160.51276.53000.750.82
2026-08-25confirmed cases4288.324.340.1764.06113.18175.140010.74
2026-08-30confirmed cases3126.372.50.0941.8279.5246.8600.6710.44
2026-09-03confirmed cases3199.663.570.1413.0873.77125.890010.68
2026-09-05confirmed cases386.861.330.071.2971.9914.410.46110.12
2026-09-06confirmed cases2171.325.60.164.4371.07100.250010.65
2026-09-08confirmed cases2177.683.460.1722.960.30.3373.15104.530010.68
2026-09-09confirmed cases2253.513.20.2462.750.420.4379.73173.78000.50.81
2026-09-12confirmed cases2381.772.590.412.20.530.5965.93315.840000.96
2026-09-14confirmed cases1223.91.760.4071.60.540.6231.54192.360000.98
2026-09-15confirmed cases1255.022.340.4632.070.630.6933.78221.240001
2026-09-18confirmed cases1227.712.720.4612.390.580.6627.64200.070000.99
2026-09-19confirmed cases1183.324.120.3653.580.620.6833.5149.820000.97
2026-05-23confirmed deaths308.150.330.3550.253.4704.680.41-0.51
2026-05-27confirmed deaths3027.840.861.1840.992.11025.7200.4-0.92
2026-06-08confirmed deaths9048.080.70.2070.4639.617.940.540.8810.11
2026-08-17confirmed deaths491.871.810.0971.2759.332.5700.510.44
2026-08-18confirmed deaths4115.91.320.1341.0255.9159.990010.63
2026-08-23confirmed deaths4145.191.720.1661.5151.7193.480010.72
2026-08-24confirmed deaths4110.911.550.121.3755.255.7100.2510.53
2026-08-25confirmed deaths487.421.250.11.2244.6542.7700.2510.53
2026-08-30confirmed deaths347.791.030.0771.1429.4118.3800.6710.47
2026-09-03confirmed deaths366.071.50.0961.329.0637.0100.3310.48
2026-09-05confirmed deaths337.510.920.0660.8628.767.491.26110.1
2026-09-06confirmed deaths229.361.40.0711.4523.425.320.62110.14
2026-09-08confirmed deaths263.061.870.1291.711.761.5927.0136.0500.510.62
2026-09-09confirmed deaths285.221.760.1791.672.392.0728.3256.890010.76
2026-09-12confirmed deaths2114.031.540.2651.331.961.8123.4490.590000.94
2026-09-14confirmed deaths160.691.440.2411.392.152.111.7148.980000.95
2026-09-15confirmed deaths175.861.660.3091.582.622.511.2964.570000.99
2026-09-18confirmed deaths190.942.090.4061.922.692.511.279.750001
2026-09-19confirmed deaths170.775.180.294.635.274.6710.7160.060000.97
2026-06-08isolation beds12083.010.490.1550.444.928.4229.680.521-0.13
2026-07-16isolation beds473.222.350.0962.2420.0153.220010.6
2026-07-18isolation beds477.782.450.1032.3318.7159.070010.61
2026-07-19isolation beds4100.013.480.1323.3122.3877.630010.64
2026-07-20isolation beds482.232.790.1062.6419.8962.3300.2510.61
2026-07-24isolation beds476.612.60.0962.4410.1466.470010.73
2026-07-25isolation beds4106.432.830.1362.7313.8492.59000.750.73
2026-07-26isolation beds494.882.430.1222.2922.6172.2700.250.750.63
2026-07-27isolation beds485.562.770.1132.6838.9246.6500.510.48
2026-07-28isolation beds4153.082.90.2012.641.15111.9400.50.750.67
2026-07-31isolation beds4119.063.040.1392.7841.3677.700.50.750.57
2026-08-04isolation beds4110.511.450.1531.3963.2647.2400.750.750.42
2026-08-08isolation beds459.170.930.0780.9551.547.150.480.7510.16
2026-08-10isolation beds455.991.130.0791.1855.830.020.15110
2026-08-15isolation beds4109.361.220.1151.0174.0735.30110.34
2026-08-17isolation beds4103.62.030.1111.7372.6230.9800.7510.43
2026-08-18isolation beds493.51.230.11.0371.3122.1900.7510.34
2026-08-23isolation beds4145.693.680.1493.1560.6785.020010.66
2026-08-24isolation beds4108.372.490.1132.1168.4739.900.510.46
2026-08-25isolation beds471.11.040.0871.0260.24010.8611-0.27
2026-08-30isolation beds361.341.50.0771.5550.26011.0811-0.2
2026-09-03isolation beds368.390.850.0850.8343.71024.680.671-0.42
2026-09-05isolation beds375.381.730.0941.8454.64020.740.671-0.34
2026-09-06isolation beds2237.055.490.53310.43154.34082.720.51-0.47
2026-09-08isolation beds272.91.390.0761.240.50.3864.616.771.52110.05
2026-09-09isolation beds290.521.790.0891.480.660.4769.7720.750110.3
2026-09-12isolation beds2172.613.080.182.781.381.152.96119.650010.72
2026-09-14isolation beds193.691.590.1011.540.840.736.9456.750010.75
2026-09-15isolation beds166.120.0691.910.550.4637.7228.380010.52
2026-09-18isolation beds155.421.080.0621.060.460.4328.7526.670010.57
2026-09-19isolation beds177.621.170.091.140.730.6838.4939.130010.57
2026-07-26onset reports267.722.220.1192.0265.222.50110.12
2026-07-27onset reports268.011.040.120.8556.661.3510.0111-0.15
2026-07-28onset reports278.720.490.1290.3861.396.6610.6711-0.07
2026-07-31onset reports2117.020.690.2390.675.7241.300.510.4
2026-08-04onset reports393.150.750.1360.6467.91.3323.9211-0.13
2026-08-08onset reports3339.70.730.4760.6471.380268.3300.33-0.88
2026-08-10onset reports3199.841.860.311.8366.10133.7401-0.7
2026-08-15onset reports3267.223.060.2211.9135.96131.250010.66
2026-08-17onset reports285.091.290.1811.5971.82013.2611-0.26
2026-08-18onset reports2113.521.370.2381.6878.01035.510.51-0.32
2026-08-23onset reports165.010.970.2231.3858.2306.7811-0.2
2026-08-24onset reports169.420.870.3191.7462.2507.1711-0.19
2026-08-25onset reports1117.720.550.3350.6435.2882.440010.8
2026-09-09onset reports170.840.530.130.2458.4612.380110.26
2026-07-16recovered4139.341.070.4691.06135.503.8411-0.1
2026-07-18recovered4149.612.250.4552.19146.942.670110.1
2026-07-19recovered4181.472.290.4822.16179.352.120110.07
2026-07-20recovered4171.812.040.4171.84169.632.180110.06
2026-07-24recovered4179.372.380.4092.01175.364.010110.13
2026-07-25recovered4177.72.080.3791.7173.524.180110.12
2026-07-26recovered4139.991.560.3811.66138.990111-0.05
2026-07-27recovered4219.872.180.4831.85218.3801.4911-0.06
2026-07-28recovered4544.073.960.3690.9543.250.50.33110.02
2026-07-31recovered4249.162.450.4021.51242.916.250110.12
2026-08-04recovered4157.982.250.4572.57146.41011.5711-0.19
2026-08-08recovered4125.591.620.4172.14109.72015.8711-0.23
2026-08-10recovered4157.11.850.5372.38116.31040.7911-0.32
2026-08-15recovered4384.4511.890.4425.52335.6248.830110.29
2026-08-17recovered4548.5822.70.7799.99117.93430.640000.94
2026-08-18recovered4596.2724.450.83411.4199.02497.250000.95
2026-08-23recovered4553.2928.690.73914.14114.01439.28000.250.92
2026-08-24recovered4518.5828.30.72614.89112.37406.220000.94
2026-08-25recovered4500.2432.490.71616.4890.53409.710000.96
2026-08-30recovered3355.3423.60.64112.5877.58277.76000.330.92
2026-09-03recovered3399.7131.330.70617.6272.05327.66000.330.9
2026-09-05recovered3306.8221.730.59412.769.91236.9000.330.9
2026-09-06recovered2130.9210.430.3496.5463.3867.540010.65
2026-09-08recovered2307.3528.320.71815.8261.07246.29000.50.88
2026-09-09recovered2325.7817.680.81410.9460.31265.480000.95
2026-09-12recovered2291.7327.950.67515.1764.47227.26000.50.9
2026-09-14recovered1152.3113.740.5588.4931.66120.650000.9
2026-09-15recovered1156.4317.20.5810.5238.33118.10010.86
2026-09-18recovered1166.4519.950.63212.7828.76137.690010.89
2026-09-19recovered1164.6810.930.576.5833.81130.860010.86

Frozen skill by release ​

Skill at each cut-off more than one release forecast, one point per release rather than pooled across releases. Releases run in the order they were cut, evenly spaced rather than to calendar scale.

Frozen skill per release
julia
frozen_skill_by_vintage_fig = plot_forecast_skill_by_vintage(
    frozen_score_by_vintage_table
);

Frozen scores by release
streamreleaserelease_datencrpsrel_to_baselinelog_crpslog_rel_to_baselinerel_to_individuallog_rel_to_individualdispersionoverpredictionunderpredictioncoverage_50coverage_90bias
confirmed casesresults-13592026-07-165242.130.4541.08121.4495.4625.20.610.07
confirmed casesresults-13912026-07-185242.12.940.4541.04121.4495.4625.20.610.07
confirmed casesresults-13942026-07-195242.130.4541.07121.4495.4625.20.610.07
confirmed casesresults-14412026-07-205245.993.050.4661.08113.62106.9225.450.20.80.16
confirmed casesresults-14462026-07-245245.992.940.4661.07113.62106.9225.450.20.80.16
confirmed casesresults-v1.12.02026-07-255245.992.970.4661.08113.62106.9225.450.20.80.16
confirmed casesresults-14622026-07-265247.613.030.4581.06122.2799.4625.890.60.80.13
confirmed casesresults-14692026-07-275247.612.960.4581.07122.2799.4625.890.60.80.13
confirmed casesresults-14792026-07-285247.612.970.4581.04122.2799.4625.890.60.80.13
confirmed casesresults-14892026-07-315247.613.040.4581.06122.2799.4625.890.60.80.13
confirmed casesresults-v1.13.22026-08-045247.613.060.4581.08122.2799.4625.890.60.80.13
confirmed casesresults-15052026-08-085247.612.990.4581.06122.2799.4625.890.60.80.13
confirmed casesresults-15272026-08-105247.613.170.4581.08122.2799.4625.890.60.80.13
confirmed casesresults-15732026-08-155256.513.10.4711.08131.5899.0425.890.40.80.13
confirmed casesresults-15912026-08-175304.063.670.5091.1985.78190.6927.590.20.60.3
confirmed casesresults-v1.15.02026-08-185304.063.740.5091.1885.78190.6927.590.20.60.3
confirmed casesresults-v1.16.02026-08-235318.573.640.5131.1895.65195.7327.190.20.40.3
confirmed casesresults-16252026-08-245318.423.90.5131.2295.56195.7327.130.20.40.3
confirmed casesresults-v1.17.02026-08-255278.343.410.4711.182.69170.1325.510.20.40.28
confirmed casesresults-v1.18.02026-08-305278.343.430.4711.1182.69170.1325.510.20.40.28
confirmed casesresults-17392026-09-035288.73.50.4751.192.73169.1126.860.20.60.22
confirmed casesresults-18282026-09-055285.093.460.5051.19102.17151.5831.3500.60.15
confirmed casesresults-18582026-09-065290.993.570.5091.1888.02171.6231.3500.60.14
confirmed casesresults-19702026-09-085295.583.450.5181.1796.92166.931.7700.60.15
confirmed casesresults-v2.1.02026-09-095295.583.70.5181.2796.92166.931.7700.60.15
confirmed casesresults-21772026-09-125311.3840.5011.2106.74175.6928.9500.60.19
confirmed casesresults-23602026-09-145174.792.140.4220.9966.8474.4433.5100.80.07
confirmed casesresults-25062026-09-155140.971.730.3620.8658.1853.1829.610.20.80.02
confirmed casesresults-25362026-09-185135.091.60.3660.8448.9255.0831.090.20.80.03
confirmed casesresults-27202026-09-195161.241.980.3990.9656.2273.1231.8900.80.08
confirmed deathsresults-13592026-07-16536.610.690.5530.6928.040.078.490.60.8-0.39
confirmed deathsresults-13912026-07-18536.610.70.5530.7128.040.078.490.60.8-0.39
confirmed deathsresults-13942026-07-19536.610.70.5530.6928.040.078.490.60.8-0.39
confirmed deathsresults-14412026-07-20537.950.740.5790.7428.760.448.760.60.8-0.35
confirmed deathsresults-14462026-07-24537.950.740.5790.7428.760.448.760.60.8-0.35
confirmed deathsresults-v1.12.02026-07-25537.950.710.5790.7328.760.448.760.60.8-0.35
confirmed deathsresults-14622026-07-26537.320.70.6350.7927.4809.840.60.8-0.43
confirmed deathsresults-14692026-07-27537.320.730.6350.8227.4809.840.60.8-0.43
confirmed deathsresults-14792026-07-28537.320.710.6350.827.4809.840.60.8-0.43
confirmed deathsresults-14892026-07-31537.320.70.6350.7927.4809.840.60.8-0.43
confirmed deathsresults-v1.13.22026-08-04537.320.690.6350.7827.4809.840.60.8-0.43
confirmed deathsresults-15052026-08-08537.320.70.6350.827.4809.840.60.8-0.43
confirmed deathsresults-15272026-08-10537.320.70.6350.8127.4809.840.60.8-0.43
confirmed deathsresults-15732026-08-15537.790.710.6310.7827.840.149.820.60.8-0.4
confirmed deathsresults-15912026-08-17536.940.710.2880.3720.912.143.90.610
confirmed deathsresults-v1.15.02026-08-18536.940.720.2880.3720.912.143.90.610
confirmed deathsresults-v1.16.02026-08-23538.990.750.270.3424.1711.593.220.610.04
confirmed deathsresults-16252026-08-24538.960.780.270.3524.1511.593.220.610.05
confirmed deathsresults-v1.17.02026-08-25535.050.670.2560.3221.9610.142.940.61-0.03
confirmed deathsresults-v1.18.02026-08-30535.050.670.2560.3221.9610.142.940.61-0.03
confirmed deathsresults-17392026-09-03534.710.660.2740.3422.918.353.450.61-0.02
confirmed deathsresults-18282026-09-05528.720.550.2950.3820.393.265.070.60.8-0.21
confirmed deathsresults-18582026-09-06535.330.680.3140.421.978.265.10.60.8-0.17
confirmed deathsresults-19702026-09-08531.570.590.2960.3722.64.244.730.60.8-0.19
confirmed deathsresults-v2.1.02026-09-09531.570.60.2960.3722.64.244.730.60.8-0.19
confirmed deathsresults-21772026-09-12539.440.770.2930.3825.679.913.860.61-0.05
confirmed deathsresults-23602026-09-14541.870.80.2940.3823.5414.653.680.410.01
confirmed deathsresults-25062026-09-15528.50.550.2520.3220.714.243.550.81-0.1
confirmed deathsresults-25362026-09-18531.720.620.2750.3620.547.353.830.81-0.08
confirmed deathsresults-27202026-09-19533.370.640.2730.3520.938.923.520.61-0.01
isolation bedsresults-13592026-07-16477.620.470.1580.4141.551.5434.540.51-0.22
isolation bedsresults-13912026-07-18477.620.460.1580.4141.551.5434.540.51-0.22
isolation bedsresults-13942026-07-19477.620.450.1580.441.551.5434.540.51-0.22
isolation bedsresults-14412026-07-20477.320.460.1580.4138.491.0237.810.51-0.22
isolation bedsresults-14462026-07-24477.320.450.1580.3938.491.0237.810.51-0.22
isolation bedsresults-v1.12.02026-07-25477.320.460.1580.438.491.0237.810.51-0.22
isolation bedsresults-14622026-07-26471.120.420.1440.3738.043.5329.560.51-0.2
isolation bedsresults-14692026-07-27471.120.420.1440.3738.043.5329.560.51-0.2
isolation bedsresults-14792026-07-28471.120.410.1440.3638.043.5329.560.51-0.2
isolation bedsresults-14892026-07-31471.120.410.1440.3638.043.5329.560.51-0.2
isolation bedsresults-v1.13.22026-08-04471.120.430.1440.3838.043.5329.560.51-0.2
isolation bedsresults-15052026-08-08471.120.440.1440.3938.043.5329.560.51-0.2
isolation bedsresults-15272026-08-10471.120.420.1440.3738.043.5329.560.51-0.2
isolation bedsresults-15732026-08-15476.50.470.1540.4135.782.1538.570.51-0.22
isolation bedsresults-15912026-08-17473.460.430.1470.3736.12.1335.230.51-0.2
isolation bedsresults-v1.15.02026-08-18473.460.450.1470.3936.12.1335.230.51-0.2
isolation bedsresults-v1.16.02026-08-23473.460.440.1470.3935.252.0536.160.51-0.2
isolation bedsresults-16252026-08-24473.460.440.1470.3835.252.0536.160.51-0.2
isolation bedsresults-v1.17.02026-08-25474.840.450.1510.3939.271.4434.130.51-0.23
isolation bedsresults-v1.18.02026-08-30474.840.440.1510.3939.271.4434.130.51-0.23
isolation bedsresults-17392026-09-03475.350.440.1550.3940.750.9833.630.51-0.24
isolation bedsresults-18282026-09-05473.290.440.1490.3935.511.8135.980.51-0.24
isolation bedsresults-18582026-09-06467.650.410.1360.3636.961.3129.390.751-0.2
isolation bedsresults-19702026-09-08476.590.460.1560.4137.540.8838.170.51-0.28
isolation bedsresults-v2.1.02026-09-09476.590.460.1560.4137.540.8838.170.51-0.28
isolation bedsresults-21772026-09-12476.80.460.1560.4134.071.341.430.51-0.24
isolation bedsresults-23602026-09-144205.61.190.2410.61126.5279.0800.2510.58
isolation bedsresults-25062026-09-154131.190.770.1790.4685.3645.8300.7510.46
isolation bedsresults-25362026-09-184101.70.610.1470.3972.2929.4100.7510.4
isolation bedsresults-27202026-09-194122.90.760.1670.4577.5545.3400.510.45

The frozen forecasts made at each cut-off against the value observed since, one panel per stream and horizon, the observed value in black. Each panel carries the frozen forecast and the persistence baseline, coloured as in the cross-release overlay above, and the x-axis is the cut-off each forecast was made from.

Frozen-fit forecasts-versus-now overlay
julia
# The frozen joint and the persistence baseline only. A single-stream
# frozen fit exists at the one-week-back cut-off alone, so its series lands
# on one made date of a panel spanning every cut-off, overplotting the
# joint point it sits beside rather than reading as a second series. The
# single-stream frozen fits are compared against the joint in the skill
# figures and in the validation plot at that cut-off.
frozen_overlay_fig = plot_forecast_overlay(
    scored_overlay(
        vcat(
            select_fit_role(frozen_overlay_df, "joint"),
            select_fit_role(frozen_overlay_df, "baseline")
        )
    )
);

The frozen re-fits below freeze the renewal data to an earlier cut-off and re-fit, so that a change driven by newer data can be distinguished from one driven by a change of method. Each uses the full headline settings: 1000 draws across two chains.

Freeze the renewal data to a cut-off and re-fit
julia
# Frozen re-fits and released_df are prepared in the setup block above.

Individual fits against the baseline ​

This section carries the same cross-release forecast scoring as Forecast scoring across releases above, for each stream's own individual fit rather than the joint, against the same persistence baseline.

Individual-fit rows of the cross-release scores
julia
# The relative skill against a stream's individual fit is only ever
# computed on the joint model's row, so on these rows it is missing by
# construction and the column is dropped rather than shown empty.
individual_score_overview_table = drop_individual_fit_columns(
    select_fit_role(forecast_score_overview_table, "individual")
)
individual_score_by_horizon_table = drop_individual_fit_columns(
    select_fit_role(forecast_score_by_horizon_table, "individual")
)
# See the comment above `joint_score_by_release_table`'s assignment for why
# this setup chunk's last statement needs a trailing `;`.
individual_score_by_release_table = drop_individual_fit_columns(
    select_fit_role(forecast_score_by_release_table, "individual")
);
streamfitncrpsrel_to_baselinelog_crpslog_rel_to_baselinedispersionoverpredictionunderpredictioncoverage_50coverage_90bias
confirmed casesconfirmed83316.92.680.2272.52261.0354.471.40.920.930.16
confirmed deathsconfirmed_deaths83249.823.090.4233.68198.7724.0127.040.821-0.05
isolation bedstreatment8399.351.740.152.0387.340.1911.820.961-0.18
onset reportsonsets24208.11.480.3211.48110.9430.0667.110.621-0.12

The same relative skill against the baseline, by horizon, one panel per stream (dataset), for each stream's own individual fit.

julia
individual_relative_skill_fig = plot_forecast_relative_skill(
    individual_score_by_horizon_table;
    empty_message = "Empty: no release old enough for its targets to " *
        "have been observed carries an individual-stream " *
        "forecast. Not a missing forecast."
);

Scores by horizon
streamhorizonfitncrpsrel_to_baselinelog_crpslog_rel_to_baselinedispersionoverpredictionunderpredictioncoverage_50coverage_90bias
confirmed cases7confirmed26151.943.40.2613.1380.7268.882.350.810.810.19
confirmed cases14confirmed22253.13.340.2123.04193.2957.722.090.910.950.13
confirmed cases21confirmed19374.842.720.1992.32342.4231.980.43110.13
confirmed cases28confirmed16603.872.210.2241.69550.5253.260.09110.17
confirmed deaths7confirmed_deaths2672.42.650.3973.9949.023.3620.020.691-0.1
confirmed deaths14confirmed_deaths22175.783.270.4584.93130.059.3236.40.771-0.12
confirmed deaths21confirmed_deaths19316.52.990.4423.58256.1224.2236.160.951-0.05
confirmed deaths28confirmed_deaths16560.753.190.3932.46468.4977.5114.750.9410.13
isolation beds7treatment26101.642.60.1563.0788.680.1912.7711-0.16
isolation beds14treatment2298.411.990.1482.2992.150.066.211-0.12
isolation beds21treatment1997.291.430.1441.6284.620.2812.390.951-0.19
isolation beds28treatment1699.391.20.1511.481.790.2617.340.881-0.3
onset reports7onsets10114.480.860.41.22796.5128.970.91-0.06
onset reports14onsets8230.692.220.2672.21116.2844.1570.260.51-0.08
onset reports21onsets6334.031.670.261.62157.0350.52126.480.331-0.28
Scores by release
made_datestreamfitncrpsrel_to_baselinelog_crpslog_rel_to_baselinedispersionoverpredictionunderpredictioncoverage_50coverage_90bias
2026-07-23confirmed casesconfirmed4327.810.860.2060.82319.434.843.5511-0.03
2026-07-25confirmed casesconfirmed4346.241.520.2131.55343.882.30.06110.02
2026-07-26confirmed casesconfirmed4299.071.420.2111.67291.390.167.5111-0.12
2026-07-27confirmed casesconfirmed4267.621.380.1711.35262.035.380.21110.04
2026-07-31confirmed casesconfirmed4402.873.310.2192.82383.8219.050110.09
2026-08-01confirmed casesconfirmed4336.312.930.2093.06322.9813.330110.08
2026-08-02confirmed casesconfirmed4296.982.160.1711.97286.156.724.1111-0.03
2026-08-03confirmed casesconfirmed4291.81.50.1751.18281.4210.080.3110.05
2026-08-04confirmed casesconfirmed4350.843.850.1913.45323.4127.40.04110.11
2026-08-07confirmed casesconfirmed4308.43.90.1923.01294.779.514.1211-0.01
2026-08-11confirmed casesconfirmed4335.757.530.1915.54307.327.620.84110.1
2026-08-15confirmed casesconfirmed4312.225.270.2114.16308.061.822.3411-0.03
2026-08-17confirmed casesconfirmed4299.384.010.2283.64293.3306.0611-0.09
2026-08-22confirmed casesconfirmed4335.335.020.2064.11293.0242.310110.15
2026-08-24confirmed casesconfirmed4315.975.190.1854.41253.0562.920110.26
2026-08-25confirmed casesconfirmed4302.154.880.1844.27236.9365.230110.27
2026-08-30confirmed casesconfirmed3257.625.120.1773.34185.6571.970110.34
2026-08-31confirmed casesconfirmed3203.033.880.1583.28165.7737.260110.22
2026-09-01confirmed casesconfirmed3242.923.960.1732.91189.5153.410110.24
2026-09-06confirmed casesconfirmed2159.635.080.1764.79136.1223.510110.23
2026-09-10confirmed casesconfirmed2216.462.140.2221.92135.5980.8700.510.43
2026-09-12confirmed casesconfirmed2715.774.760.6953.6864.25651.520001
2026-09-13confirmed casesconfirmed1409.596.760.6095.2839.18370.410000.99
2026-09-15confirmed casesconfirmed1410.164.150.6693.2442.163680001
2026-09-16confirmed casesconfirmed1375.694.80.6293.7943.88331.810001
2026-09-19confirmed casesconfirmed1297.615.980.5424.8141.56256.050001
2026-07-23confirmed deathsconfirmed_deaths4701.544.530.4482.4383.29318.2500.2510.56
2026-07-25confirmed deathsconfirmed_deaths4237.781.990.2341.6214.2823.50110.18
2026-07-26confirmed deathsconfirmed_deaths4256.462.30.2552.02231.6524.810110.13
2026-07-27confirmed deathsconfirmed_deaths4310.272.880.2521.92280.8429.430110.21
2026-07-31confirmed deathsconfirmed_deaths4312.362.70.2892.15292.2120.080.07110.08
2026-08-01confirmed deathsconfirmed_deaths4280.922.40.2531.7261.4619.460110.1
2026-08-02confirmed deathsconfirmed_deaths4256.052.730.2762.38238.2316.880.94110.03
2026-08-03confirmed deathsconfirmed_deaths4224.51.620.2451.22212.210.831.46110.02
2026-08-04confirmed deathsconfirmed_deaths4248.773.560.2833.44242.124.292.3511-0.02
2026-08-07confirmed deathsconfirmed_deaths4231.53.230.3253.08223.681.026.7911-0.11
2026-08-11confirmed deathsconfirmed_deaths4244.610.680.33710.13233.591.959.0611-0.13
2026-08-15confirmed deathsconfirmed_deaths4244.836.120.345.64239.910.044.8911-0.1
2026-08-17confirmed deathsconfirmed_deaths4251.325.10.2973.95246.093.931.311-0.02
2026-08-22confirmed deathsconfirmed_deaths42422.710.4823.89219.73022.2711-0.18
2026-08-24confirmed deathsconfirmed_deaths4220.023.20.5436.42144.48075.5411-0.4
2026-08-25confirmed deathsconfirmed_deaths4247.193.620.7969.93151.31095.880.751-0.45
2026-08-30confirmed deathsconfirmed_deaths3215.224.780.91614.3298.90116.320.331-0.53
2026-08-31confirmed deathsconfirmed_deaths3199.793.070.7827.79109.7909011-0.42
2026-09-01confirmed deathsconfirmed_deaths3203.764.020.93212.0989.670114.090.331-0.52
2026-09-06confirmed deathsconfirmed_deaths2189.4610.431.18927.1945.040144.4201-0.69
2026-09-10confirmed deathsconfirmed_deaths2123.891.660.7474.0367.86056.040.51-0.46
2026-09-12confirmed deathsconfirmed_deaths254.20.720.1350.6627.6926.520010.56
2026-09-13confirmed deathsconfirmed_deaths132.70.850.1270.8314.5418.160010.63
2026-09-15confirmed deathsconfirmed_deaths126.420.650.1120.6416.2210.20110.49
2026-09-16confirmed deathsconfirmed_deaths128.530.620.1250.6215.9112.620010.56
2026-09-19confirmed deathsconfirmed_deaths115.480.990.070.9914.590.890110.13
2026-07-23isolation bedstreatment455.731.910.0892.3247.7907.9411-0.21
2026-07-25isolation bedstreatment451.681.340.0821.647.140.014.5211-0.14
2026-07-26isolation bedstreatment461.671.660.1254.820.236.6311-0.12
2026-07-27isolation bedstreatment459.472.050.0982.554.6704.811-0.18
2026-07-31isolation bedstreatment482.761.940.1322.4168.17014.60.751-0.28
2026-08-01isolation bedstreatment488.621.40.1531.7973.94014.670.751-0.25
2026-08-02isolation bedstreatment484.371.660.1412.0769.7014.670.751-0.28
2026-08-03isolation bedstreatment479.931.770.1342.1871.9108.0111-0.21
2026-08-04isolation bedstreatment487.061.280.1411.4582.731.223.1111-0.03
2026-08-07isolation bedstreatment4104.981.740.1712.1785.76019.2111-0.29
2026-08-11isolation bedstreatment4102.610.610.1710.790.93011.6811-0.21
2026-08-15isolation bedstreatment4127.511.480.2041.8898.57028.9411-0.33
2026-08-17isolation bedstreatment4103.761.520.1611.8493.32010.4411-0.19
2026-08-22isolation bedstreatment4144.982.330.2182.89120.25024.7311-0.22
2026-08-24isolation bedstreatment4127.242.530.1812.9115.131.0511.0611-0.11
2026-08-25isolation bedstreatment4131.921.910.1912.22111.86020.0611-0.2
2026-08-30isolation bedstreatment3123.93.030.1693.38114.881.727.311-0.07
2026-08-31isolation bedstreatment3117.932.570.1612.94108.450.059.4311-0.14
2026-09-01isolation bedstreatment3117.143.050.1533.47112.3304.811-0.09
2026-09-06isolation bedstreatment2113.982.350.1492.55110.8103.1611-0.09
2026-09-10isolation bedstreatment2133.863.310.1743.66115.73018.1211-0.14
2026-09-12isolation bedstreatment21322.140.1812.55113.46018.5411-0.16
2026-09-13isolation bedstreatment1106.222.210.1392.52105.3500.8711-0.06
2026-09-15isolation bedstreatment1112.093.30.1433.87109.3602.7311-0.08
2026-09-16isolation bedstreatment1113.014.720.1435.499.88013.1311-0.2
2026-09-19isolation bedstreatment1101.371.520.1341.68101.120.250110.03
2026-08-02onset reportsonsets3123.070.90.1881.1396.84026.2311-0.28
2026-08-03onset reportsonsets3105.781.590.1661.36100.321.054.4111-0.03
2026-08-04onset reportsonsets3140.891.150.2651.2888.340.0252.520.671-0.26
2026-08-07onset reportsonsets3379.321.150.6341.3782.32029701-0.8
2026-08-11onset reportsonsets3241.791.510.4271.8685.910155.870.331-0.64
2026-08-15onset reportsonsets3371.884.190.2812.37205.04166.8300.3310.57
2026-08-17onset reportsonsets2244.443.90.2522.39153.6190.8300.510.46
2026-08-22onset reportsonsets2119.291.020.2071.15104.1215.180110.21
2026-08-24onset reportsonsets184.591.180.3912.3282.0902.511-0.11
2026-08-25onset reportsonsets194.280.420.5050.9188.575.710110.17

Saving forecast outputs ​

The one-week-back validation forecast, in the same archive format as the release forecast, so the frozen "last week versus now" forecast is recorded as a release asset alongside the forecast it is scored against.

Write forecast outputs
julia
output_dir = get(
    ENV, "BVD_OUTPUT_DIR",
    joinpath(pkgdir(BVDOutbreakSize), "output")
)
mkpath(output_dir)
CSV.write(
    joinpath(output_dir, "forecast_validation.csv"),
    forecast_archive(
        [(7, validation_forecast)];
        made_date = frozen_lastweek.o.cutoff, thin = 5
    )
)
"/home/runner/work/BVDOutbreakSize/BVDOutbreakSize/output/forecast_validation.csv"
Write the summary bullets
julia
# The bullets under the summary heading at the top of the page. They read
# tables built further down, so they are written here and read back when
# the site is assembled.
evaluation_forecast_national_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    scored(tbl) = filter(r -> !ismissing(r.rel_to_baseline), tbl)
    function beat(label, tbl)
        rows = scored(tbl)
        size(rows, 1) == 0 &&
            return string("- **", label, ":** no scored forecasts yet.")
        return string(
            "- **", label, ":** beat the baseline on ",
            count(<(1), rows.rel_to_baseline), " of ", size(rows, 1),
            " streams."
        )
    end
    function block(lead, tbl)
        rows = scored(tbl)
        size(rows, 1) == 0 && return nothing
        return join(
            vcat(
                [string("**", lead, "**"), ""],
                [
                    string(
                        "- ", r.stream, ": relative skill ",
                        fmt(r.rel_to_baseline), ", 90% coverage ",
                        fmt(r.coverage_90), ", bias ", fmt(r.bias), " over ",
                        r.n, " forecasts."
                    )
                        for r in eachrow(rows)
                ]
            ), "\n"
        )
    end
    frozen_joint = select_fit_role(frozen_score_overview_table, "joint")
    overall = [
        beat("Joint model across releases", joint_score_overview_table),
        beat("Frozen joint model", frozen_joint),
    ]
    blocks = filter(
        !isnothing,
        [
            block("Across releases", joint_score_overview_table),
            block("Frozen fits", frozen_joint),
        ]
    )
    join(vcat([join(overall, "\n")], blocks), "\n\n")
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_forecast_national.md"),
    evaluation_forecast_national_summary
);