Province forecast evaluation

How the province forecasts on the province forecasts page have scored against what each province went on to report. Scoring follows the national forecast evaluation, against the same persistence baseline. The in-sample province checks are on the province in-sample checks page.

Load packages, data and fitted chains

julia

# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))

validation_forecast_from (generic function with 1 method)

julia

# The fits this page reads, loaded from the cache here.
frozen_lastweek = load_fit("frozen_validation");
chn_joint = load_fit("joint");

Summary

The overall bullets come first, then a short block per province, from the scores below.

Provinces: 0 of 3 provinces with scored forecasts beat the baseline on every stream scored for them.
Province streams: 1 of 8 beat the baseline.

Ituri

confirmed cases: relative skill 1.2, 90% coverage 1.0 over 22 forecasts.
confirmed deaths: relative skill 0.72, 90% coverage 1.0 over 22 forecasts.

Nord-Kivu

confirmed cases: relative skill 1.33, 90% coverage 0.95 over 22 forecasts.
confirmed deaths: relative skill 1.43, 90% coverage 0.95 over 22 forecasts.

Haut-Uele

confirmed cases: relative skill 2.05, 90% coverage 1.0 over 22 forecasts.
confirmed deaths: relative skill 1.32, 90% coverage 1.0 over 22 forecasts.

Sud-Kivu

No scored forecast yet.

Tshopo

No scored forecast yet.

Bas-Uele

No scored forecast yet.

Sud-Ubangi

No scored forecast yet.

Nord-Ubangi

No scored forecast yet.

Forecast by province

The forecast is defined in the province forecast Methods section. Every release's archived province forecast is scored against what has since been observed in Forecast by province across releases.

Province forecast against observed

julia

# The frozen fit's one-week-ahead national forecast, the same one the
# forecast evaluation page validates. `validation_forecast_from` is defined
# in the shared setup.
validation_forecast = validation_forecast_from("frozen_validation");

# Per-province cumulative confirmed cases and deaths at the frozen cut-off
# and at the current one, so the truth for the week is their difference.
# Read off the same increment matrices the compositions are scored on, so
# the clamped revision is treated identically on both sides.
province_truth = let
    cur_c = province_increment_matrix(
        obs.province_confirmed_history,
        PROVINCE_NAMES, N_PATCHES
    )
    cur_d = province_increment_matrix(
        obs.province_death_history,
        PROVINCE_NAMES, N_PATCHES
    )
    froz_c = province_increment_matrix(
        frozen_lastweek.o.province_confirmed_history,
        PROVINCE_NAMES, N_PATCHES
    )
    froz_d = province_increment_matrix(
        frozen_lastweek.o.province_death_history, PROVINCE_NAMES, N_PATCHES
    )
    (;
        observed = vec(sum(cur_c.increments; dims = 2)),
        baseline = vec(sum(froz_c.increments; dims = 2)),
        death_observed = vec(sum(cur_d.increments; dims = 2)),
        death_baseline = vec(sum(froz_d.increments; dims = 2)),
    )
end

province_validation_table = province_forecast_vs_truth(
    fit_forecast("frozen_validation"), validation_forecast;
    observed = province_truth.observed,
    baseline = province_truth.baseline,
    death_observed = province_truth.death_observed,
    death_baseline = province_truth.death_baseline,
    n_patches = N_PATCHES
);

Province	Stream	Lower 90%	Upper 90%	Observed	Within 90% PI
Ituri	Confirmed cases	181	481	265	true
Ituri	Confirmed deaths	68	195	140	true
Nord-Kivu	Confirmed cases	139	393	183	true
Nord-Kivu	Confirmed deaths	67	190	82	true
Haut-Uele	Confirmed cases	8	75	28	true
Haut-Uele	Confirmed deaths	1	33	18	true
Sud-Kivu	Confirmed cases	0	1	0	true
Sud-Kivu	Confirmed deaths	0	1	0	true
Tshopo	Confirmed cases	2	31	13	true
Tshopo	Confirmed deaths	0	12	5	true
Bas-Uele	Confirmed cases	0	8	0	true
Bas-Uele	Confirmed deaths	0	4	0	true
Sud-Ubangi	Confirmed cases	0	2	0	true
Sud-Ubangi	Confirmed deaths	0	1	0	true
Nord-Ubangi	Confirmed cases	0	0	3	false
Nord-Ubangi	Confirmed deaths	0	0	1	false

The frozen fit's one-week-ahead forecast of each province's confirmed cases and deaths, against what each province went on to report.

Build the forecast-versus-observed figures

julia

frozen_province_draws = fit_forecast("frozen_validation");
frozen_province_projection = forecast_provinces(
    frozen_province_draws; horizon = 7, n_patches = N_PATCHES
);
# The patients in isolation each province printed on the target day, the
# current cut-off, or `NaN` where it printed nothing that day.
province_isolation_now = let
    rows = province_care_observations(
        obs.province_isolation_history, PROVINCE_NAMES
    )
    v = fill(NaN, N_PATCHES)
    for (d, p, c) in zip(rows.days, rows.patches, rows.counts)
        d == obs.n && p <= N_PATCHES && (v[p] = c)
    end
    v
end
province_rt_now = province_map_summary(
    chn_joint, :R_T_patch, N_PATCHES
).values
province_observed = (;
    confirmed_new = province_truth.observed .- province_truth.baseline,
    confirmed_deaths_new = province_truth.death_observed .-
        province_truth.death_baseline,
    isolation_level = province_isolation_now,
    rt_forecast = province_rt_now,
)
province_vs_observed_fig = plot_province_forecast(
    frozen_province_draws, frozen_province_projection;
    n_patches = N_PATCHES, observed = province_observed,
    title = "Last week's forecast against what was observed"
);
province_vs_observed_detail(p) = plot_province_forecast_detail(
    frozen_province_draws, frozen_province_projection;
    province = p, n_patches = N_PATCHES,
    observed = NamedTuple(
        k => v[p] for (k, v) in pairs(province_observed) if isfinite(v[p])
    )
);

The same forecast against what each province went on to report, one panel per target. A cross marks the observed count over the week for cases and deaths, the patients in isolation on the target day where the province printed them, and the current fit's reproduction number at the cut-off.

Each province's forecast, with the dashed rule at what it went on to report.

Ituri

Nord-Kivu

Haut-Uele

Sud-Kivu

Tshopo

Bas-Uele

Sud-Ubangi

Nord-Ubangi

Forecast by province across releases

The archived province forecast of each release is scored against what each province went on to report. A window holding a harmonisation-break day is left unscored, because that day's backfill is published for the country and not by province. The scores fill in as releases carrying them become old enough for their targets to have been observed. The joint patch model is the only model that forecasts the provinces, so every table here is the joint model's, one row per stream and province.

Load and summarise the province forecast scores

julia

province_scores_df = _release_data(
    "province_forecast_scores.csv",
    (;
        release = String, made_date = Date, stream = String, horizon = Int,
        target_date = Date, fit = String, crps = Float64,
        log_crps = Float64, dispersion = Float64, overprediction = Float64,
        underprediction = Float64, coverage_50 = Float64,
        coverage_90 = Float64,
        bias = Float64, n_samples = Int,
        log_rel_to_baseline = Float64,
    )
)
# There is no individual single-stream fit to compare against, and `fit` is
# single-valued by construction once the baseline is set aside. Both are
# dropped rather than rendered as columns that cannot vary. The figures
# keep the full tables, since they compare the joint against the baseline.
province_score_by_horizon_table = forecast_score_by_horizon(
    province_scores_df
)
province_score_by_release_table = forecast_score_by_release(
    province_scores_df
)
_province_display(tbl) = drop_degenerate_fit_column(
    drop_individual_fit_columns(tbl)
)
province_score_overview_display = _province_display(
    forecast_score_overview(province_scores_df)
)
province_score_by_horizon_display = _province_display(
    province_score_by_horizon_table
)
# See the comment on `joint_score_by_release_table` on the forecast
# evaluation page for why this setup chunk's last statement needs a
# trailing `;`.
province_score_by_release_display = _province_display(
    province_score_by_release_table
);

_province_empty = "No scored province forecasts yet. This fills in as " *
    "releases carrying the per-province projection accumulate.";
# A plain statement above the tables when nothing is scored yet, and
# nothing otherwise.
province_scores_note = size(province_scores_df, 1) == 0 ?
    Markdown.parse(
        "No province forecast has been scored yet. This section fills in " *
        "as releases carrying the per-province projection accumulate."
    ) : nothing;

stream	n	crps	rel_to_baseline	log_crps	log_rel_to_baseline	dispersion	overprediction	underprediction	coverage_50	coverage_90	bias
confirmed cases [haut_uele]	22	11.32	2.05	0.175	1.63	8.89	2.36	0.06	1	1	0.27
confirmed cases [ituri]	22	49.88	1.2	0.091	1.19	37.77	12.02	0.09	0.77	1	0.3
confirmed cases [nord_kivu]	22	44.38	1.33	0.143	1.15	28.45	7.69	8.24	0.64	0.95	0.01
confirmed cases [other]	22	9.97	2.34	0.333	1.28	5.48	4.27	0.21	0.5	1	0.36
confirmed deaths [haut_uele]	22	3.99	1.32	0.171	1.2	3.48	0.2	0.32	1	1	-0.01
confirmed deaths [ituri]	22	16.61	0.72	0.071	0.85	14.69	0.68	1.23	0.95	1	-0.06
confirmed deaths [nord_kivu]	22	27.08	1.43	0.164	1.22	13.29	13.7	0.08	0.59	0.95	0.45
confirmed deaths [other]	22	4.21	2.11	0.478	1.1	2.05	2.06	0.09	0.55	1	0.46

julia

province_relative_skill_fig = plot_forecast_relative_skill(
    province_score_by_horizon_table; empty_message = _province_empty
);

Relative skill against the baseline by horizon, one panel per stream and province, on a log scale with the reference line at one.

julia

province_crps_by_horizon_fig = plot_forecast_crps_by_horizon(
    province_score_by_horizon_table;
    title = "CRPS decomposition by horizon, by province",
    empty_message = _province_empty
);

The mean CRPS by horizon split into its width, overprediction and underprediction.

Province scores by horizon

stream	horizon	n	crps	rel_to_baseline	log_crps	log_rel_to_baseline	dispersion	overprediction	underprediction	coverage_50	coverage_90	bias
confirmed cases [haut_uele]	7	8	5.32	1.69	0.183	1.4	4.11	1.08	0.12	1	1	0.21
confirmed cases [haut_uele]	14	7	10.3	1.91	0.17	1.67	7.58	2.65	0.07	1	1	0.32
confirmed cases [haut_uele]	21	5	15.87	2.7	0.162	2.27	12.94	2.92	0	1	1	0.29
confirmed cases [haut_uele]	28	2	27.51	1.89	0.195	1.6	22.44	5.07	0	1	1	0.28
confirmed cases [ituri]	7	8	29.55	1.33	0.106	1.15	19.18	10.31	0.06	0.62	1	0.34
confirmed cases [ituri]	14	7	48.87	1.59	0.086	1.45	34.76	14.12	0	0.71	1	0.34
confirmed cases [ituri]	21	5	64.59	1.22	0.078	1.22	53.63	10.64	0.33	1	1	0.19
confirmed cases [ituri]	28	2	97.94	0.77	0.086	0.78	83.02	14.92	0	1	1	0.25
confirmed cases [nord_kivu]	7	8	32.5	1.23	0.185	1.12	15.14	14.31	3.04	0.38	0.88	0.27
confirmed cases [nord_kivu]	14	7	44.4	0.88	0.126	0.79	26.99	7.79	9.61	0.71	1	-0.08
confirmed cases [nord_kivu]	21	5	55.85	2.21	0.12	2.52	38.18	0.01	17.65	0.8	1	-0.26
confirmed cases [nord_kivu]	28	2	63.24	2.98	0.094	3.11	62.53	0	0.72	1	1	-0.06
confirmed cases [other]	7	8	3.76	1.02	0.327	0.91	2.21	1.26	0.3	0.38	1	0.27
confirmed cases [other]	14	7	7.9	1.24	0.326	1.01	4.27	3.3	0.33	0.71	1	0.3
confirmed cases [other]	21	5	13.58	6.81	0.3	4.46	7.85	5.73	0	0.6	1	0.46
confirmed cases [other]	28	2	32.97	6.92	0.469	3.58	16.9	16.07	0	0	1	0.69
confirmed deaths [haut_uele]	7	8	2.67	1.38	0.221	1.27	2.17	0.15	0.35	1	1	-0.08
confirmed deaths [haut_uele]	14	7	4.09	1.36	0.163	1.25	3.31	0.19	0.59	1	1	-0.01
confirmed deaths [haut_uele]	21	5	4.8	1.29	0.122	1.05	4.57	0.23	0	1	1	0.07
confirmed deaths [haut_uele]	28	2	6.94	1.19	0.126	0.97	6.56	0.29	0.08	1	1	0.03
confirmed deaths [ituri]	7	8	10.91	0.92	0.088	0.91	9.14	1.43	0.34	0.88	1	0.01
confirmed deaths [ituri]	14	7	15.87	0.95	0.064	1	13.83	0.36	1.67	1	1	-0.09
confirmed deaths [ituri]	21	5	21.24	0.64	0.058	0.72	19.36	0.19	1.7	1	1	-0.1
confirmed deaths [ituri]	28	2	30.41	0.48	0.061	0.56	28.28	0	2.12	1	1	-0.15
confirmed deaths [nord_kivu]	7	8	18.88	1.82	0.223	1.48	8.28	10.6	0	0.62	0.88	0.47
confirmed deaths [nord_kivu]	14	7	24.66	0.95	0.135	0.77	12.63	12.02	0.02	0.71	1	0.45
confirmed deaths [nord_kivu]	21	5	31.72	1.64	0.116	1.52	17.16	14.23	0.33	0.6	1	0.35
confirmed deaths [nord_kivu]	28	2	56.72	2.04	0.15	1.89	26.01	30.71	0	0	1	0.62
confirmed deaths [other]	7	8	1.87	0.79	0.554	0.76	1	0.7	0.16	0.62	1	0.39
confirmed deaths [other]	14	7	3.2	1.14	0.408	0.89	1.68	1.41	0.11	0.71	1	0.32
confirmed deaths [other]	21	5	5.87	8.07	0.411	5.2	2.87	3	0	0.4	1	0.62
confirmed deaths [other]	28	2	12.97	14.53	0.584	7.57	5.54	7.43	0	0	1	0.78

julia

province_skill_by_cutoff_fig = plot_forecast_skill_by_cutoff(
    province_score_by_release_table;
    title = "Relative skill against the baseline by release, by province",
    empty_message = _province_empty
);

Relative skill against the baseline release by release, so a run of releases that lost to the baseline reads as a run rather than as an average.

Province scores by release

made_date	stream	n	crps	rel_to_baseline	log_crps	log_rel_to_baseline	dispersion	overprediction	underprediction	coverage_50	coverage_90	bias
2026-09-21	confirmed cases [haut_uele]	4	18.91	2.15	0.214	1.78	13.68	5.23	0	1	1	0.42
2026-09-22	confirmed cases [haut_uele]	4	13.66	1.97	0.166	1.65	11.54	2.12	0	1	1	0.26
2026-09-25	confirmed cases [haut_uele]	3	10.74	2.3	0.174	1.78	8.62	2.12	0	1	1	0.33
2026-09-26	confirmed cases [haut_uele]	3	10.13	2.48	0.179	2.01	7.43	2.7	0	1	1	0.39
2026-09-27	confirmed cases [haut_uele]	3	8.51	2.15	0.146	1.8	6.88	1.63	0	1	1	0.32
2026-10-02	confirmed cases [haut_uele]	2	7.46	2.17	0.183	1.8	5.86	1.6	0	1	1	0.33
2026-10-06	confirmed cases [haut_uele]	2	5.67	1.02	0.154	0.82	5.11	0	0.56	1	1	-0.21
2026-10-10	confirmed cases [haut_uele]	1	4.24	1.72	0.16	1.73	3.96	0	0.29	1	1	-0.17
2026-09-21	confirmed cases [ituri]	4	54.93	0.81	0.082	0.91	48.58	6.35	0	1	1	0.25
2026-09-22	confirmed cases [ituri]	4	75.9	1.26	0.109	1.33	49.65	26.25	0	0.5	1	0.48
2026-09-25	confirmed cases [ituri]	3	33.57	0.98	0.062	0.74	31.53	2.04	0	1	1	0.15
2026-09-26	confirmed cases [ituri]	3	60.77	1.79	0.128	1.75	36.73	24.04	0	0.33	1	0.52
2026-09-27	confirmed cases [ituri]	3	32.33	1.12	0.065	0.97	31.64	0	0.69	1	1	-0.08
2026-10-02	confirmed cases [ituri]	2	37.69	1.45	0.094	1.32	25.86	11.83	0	0.5	1	0.45
2026-10-06	confirmed cases [ituri]	2	46.89	2.27	0.105	1.83	31.83	15.06	0	1	1	0.4
2026-10-10	confirmed cases [ituri]	1	24.82	1.35	0.084	1.21	22.89	1.92	0	1	1	0.15
2026-09-21	confirmed cases [nord_kivu]	4	41.63	1.5	0.122	1.58	36.16	5.13	0.34	0.75	1	0.19
2026-09-22	confirmed cases [nord_kivu]	4	39.68	2.32	0.111	2.5	35.57	4.05	0.06	0.75	1	0.16
2026-09-25	confirmed cases [nord_kivu]	3	29.89	1.04	0.094	0.83	22.76	0.02	7.11	1	1	-0.22
2026-09-26	confirmed cases [nord_kivu]	3	36.24	1.04	0.116	0.96	27.49	0.46	8.3	1	1	-0.17
2026-09-27	confirmed cases [nord_kivu]	3	65.34	1.19	0.216	0.96	22.17	0	43.17	0	1	-0.68
2026-10-02	confirmed cases [nord_kivu]	2	22.5	0.41	0.086	0.4	20.5	0	2	1	1	-0.14
2026-10-06	confirmed cases [nord_kivu]	2	68.9	3.7	0.204	2.64	29.95	38.94	0	0	1	0.67
2026-10-10	confirmed cases [nord_kivu]	1	74	1.38	0.363	1.24	20.85	53.15	0	0	0	0.91
2026-09-21	confirmed cases [other]	4	18.65	3.26	0.484	1.48	9.1	9.56	0	0	1	0.7
2026-09-22	confirmed cases [other]	4	18.97	3.74	0.526	1.65	8.35	10.62	0	0	1	0.75
2026-09-25	confirmed cases [other]	3	6.69	3.18	0.272	2.32	4.45	2.24	0	0.67	1	0.49
2026-09-26	confirmed cases [other]	3	6.35	3.54	0.289	2.35	4.53	1.82	0	0.67	1	0.45
2026-09-27	confirmed cases [other]	3	4.08	2.63	0.179	2.23	3.73	0.36	0	1	1	0.25
2026-10-02	confirmed cases [other]	2	3.13	0.46	0.196	0.43	2.65	0	0.47	1	1	-0.24
2026-10-06	confirmed cases [other]	2	4.32	0.44	0.263	0.42	2.47	0	1.85	0.5	1	-0.47
2026-10-10	confirmed cases [other]	1	2.46	3.71	0.163	3.7	2.46	0	0	1	1	0
2026-09-21	confirmed deaths [haut_uele]	4	4.97	1.62	0.149	1.25	4.48	0.48	0.01	1	1	0.13
2026-09-22	confirmed deaths [haut_uele]	4	4.12	1.12	0.14	1.11	4	0.01	0.11	1	1	-0.06
2026-09-25	confirmed deaths [haut_uele]	3	3.41	1.11	0.141	0.87	3.38	0.02	0	1	1	0.03
2026-09-26	confirmed deaths [haut_uele]	3	4.04	1.27	0.194	1.1	3.35	0.69	0	1	1	0.26
2026-09-27	confirmed deaths [haut_uele]	3	3.48	1.54	0.146	1.44	3.42	0.06	0	1	1	0.08
2026-10-02	confirmed deaths [haut_uele]	2	3.03	1.09	0.171	1.1	2.52	0	0.52	1	1	-0.21
2026-10-06	confirmed deaths [haut_uele]	2	4.36	1.37	0.262	1.46	2.51	0	1.85	1	1	-0.39
2026-10-10	confirmed deaths [haut_uele]	1	3.94	1.63	0.305	1.83	2.04	0	1.9	1	1	-0.5
2026-09-21	confirmed deaths [ituri]	4	18.31	0.55	0.061	0.68	17.39	0.01	0.91	1	1	-0.07
2026-09-22	confirmed deaths [ituri]	4	19.34	0.68	0.062	0.79	18.92	0.11	0.31	1	1	0
2026-09-25	confirmed deaths [ituri]	3	16.44	0.86	0.069	0.76	13.61	0	2.83	1	1	-0.27
2026-09-26	confirmed deaths [ituri]	3	19.4	0.7	0.098	0.85	14.65	4.74	0	0.67	1	0.33
2026-09-27	confirmed deaths [ituri]	3	15.42	0.8	0.067	0.97	13.29	0.08	2.05	1	1	-0.14
2026-10-02	confirmed deaths [ituri]	2	11.91	1.05	0.07	1.12	11.17	0	0.74	1	1	-0.17
2026-10-06	confirmed deaths [ituri]	2	14.05	1.23	0.077	1.16	11.02	0	3.03	1	1	-0.2
2026-10-10	confirmed deaths [ituri]	1	9.1	0.73	0.075	0.83	9.02	0	0.08	1	1	-0.05
2026-09-21	confirmed deaths [nord_kivu]	4	54.99	2.28	0.312	2.47	16.78	38.21	0	0	0.75	0.83
2026-09-22	confirmed deaths [nord_kivu]	4	40.65	2.64	0.233	2.99	16.51	24.14	0	0	1	0.7
2026-09-25	confirmed deaths [nord_kivu]	3	15.66	1.03	0.094	0.75	11.81	3.85	0	1	1	0.38
2026-09-26	confirmed deaths [nord_kivu]	3	16.69	0.94	0.106	0.71	12.59	4.1	0	1	1	0.35
2026-09-27	confirmed deaths [nord_kivu]	3	12.49	0.4	0.076	0.29	11.91	0	0.58	1	1	-0.08
2026-10-02	confirmed deaths [nord_kivu]	2	12.97	0.87	0.095	0.8	9.95	3.02	0	1	1	0.34
2026-10-06	confirmed deaths [nord_kivu]	2	15.25	1.09	0.096	0.93	10.77	4.48	0	1	1	0.25
2026-10-10	confirmed deaths [nord_kivu]	1	22.14	2.67	0.218	2.39	8.93	13.21	0	0	1	0.66
2026-09-21	confirmed deaths [other]	4	8.6	3.33	0.74	1.38	3.33	5.27	0	0	1	0.83
2026-09-22	confirmed deaths [other]	4	6.43	3.08	0.562	1.37	2.84	3.59	0	0.25	1	0.71
2026-09-25	confirmed deaths [other]	3	3.28	2.88	0.535	1.42	1.78	1.51	0	0.33	1	0.64
2026-09-26	confirmed deaths [other]	3	2.82	3.27	0.49	1.79	1.74	1.08	0	0.67	1	0.59
2026-09-27	confirmed deaths [other]	3	2.22	3.2	0.3	2.21	1.52	0.7	0	1	1	0.46
2026-10-02	confirmed deaths [other]	2	1.85	0.4	0.319	0.32	1.1	0	0.76	1	1	-0.4
2026-10-06	confirmed deaths [other]	2	1.31	0.45	0.228	0.43	1.03	0	0.28	1	1	-0.28
2026-10-10	confirmed deaths [other]	1	1.25	0.6	0.231	0.6	1.14	0.11	0	1	1	0.23

Load the archived province forecasts and their outcomes

julia

# Written by `scripts/score_releases.jl` from each release's
# `province_forecast.csv`, projection rows only, in the national overlay's
# schema. A missing file reads as an empty table.
evaluation_province_overlay_df = _release_data(
    joinpath("province", "forecast_overlay.csv"),
    (;
        release = String, made_date = Date, stream = String, horizon = Int,
        target_date = Date, fit = String, observed = Float64,
        median = Float64, lo30 = Float64, hi30 = Float64, lo60 = Float64,
        hi60 = Float64, lo90 = Float64, hi90 = Float64,
    )
)
evaluation_province_overlay_fig = plot_forecast_overlay(
    scored_overlay(evaluation_province_overlay_df);
    empty_message = _province_empty
);

Each release's province forecasts against what each province went on to report, one panel per stream and horizon, by the cut-off each was made from. Each forecast shows its median and 90% predictive interval beside the persistence baseline and the observed count.

Saving province forecast outputs

Write the summary bullets

julia

# The bullets under the summary heading at the top of the page. They read
# tables built further down, so they are written here and read back when
# the site is assembled.
evaluation_forecast_province_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    overview = forecast_score_overview(province_scores_df)
    # The archive labels each province stream `<stream> [<province name>]`.
    function scored(p)
        tag = string(" [", PROVINCE_NAMES[p], "]")
        rows = filter(
            r -> endswith(r.stream, tag) && !ismissing(r.rel_to_baseline),
            overview
        )
        return (; tag, rows)
    end
    with_scores = [p for p in 1:N_PATCHES if size(scored(p).rows, 1) > 0]
    all_rows = filter(r -> !ismissing(r.rel_to_baseline), overview)
    n_beat = count(
        p -> all(<(1), scored(p).rows.rel_to_baseline), with_scores
    )
    overall = if isempty(with_scores)
        [
            "- **Forecasts:** no province forecast has been scored yet. " *
                "This fills in as releases carrying the per-province " *
                "projection accumulate.",
        ]
    else
        [
            string(
                "- **Provinces:** ", n_beat, " of ", length(with_scores),
                " provinces with scored forecasts beat the baseline on ",
                "every stream scored for them."
            ),
            string(
                "- **Province streams:** ",
                count(<(1), all_rows.rel_to_baseline), " of ",
                size(all_rows, 1), " beat the baseline."
            ),
        ]
    end
    function detail(p)
        sc = scored(p)
        bullets = [
            string(
                "- ", replace(r.stream, sc.tag => ""), ": relative skill ",
                fmt(r.rel_to_baseline), ", 90% coverage ",
                fmt(r.coverage_90), " over ", r.n, " forecasts."
            )
                for r in eachrow(sc.rows)
        ]
        isempty(bullets) && (bullets = ["- No scored forecast yet."])
        return join(
            vcat([string("**", PROVINCE_LABELS[p], "**"), ""], bullets), "\n"
        )
    end
    join(
        vcat([join(overall, "\n")], [detail(p) for p in 1:N_PATCHES]),
        "\n\n"
    )
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_forecast_province.md"),
    evaluation_forecast_province_summary
);

Province forecast evaluation ​

Summary ​

Forecast by province ​

Forecast by province across releases ​

Saving province forecast outputs ​

Province forecast evaluation

Summary

Forecast by province

Forecast by province across releases

Saving province forecast outputs