Skip to content

Health-zone in-sample checks ​

This page checks whether the health-zone model reproduces the per-zone data it was fitted to. The national checks are on the national in-sample checks page and the provincial ones on the province in-sample checks page. The zone data enter the model as compositions of their patch's totals, so the checks are on each zone's share of its patch. A later section sets each zone's reproduction number from the current fit against the same fit a week earlier, and the last follows each zone's published estimates across releases.

Load packages, data and fitted chains
julia
# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))
validation_forecast_from (generic function with 1 method)
julia
# The fits this page reads, loaded from the cache here. The headline joint
# carries the patch trajectories the zone stage conditions on, and the
# frozen zone fit is the same fit a week earlier.
chn_joint = load_fit("joint");
chn_local = load_fit("local");
frozen_local = load_fit("local_frozen_validation");

Summary ​

The overall bullets come first, then a table of each province's bias and coverage, from the zone composition calibration below. A cell is one zone at one vintage, and a composition that was not scored reads n/a. Bias and coverage are defined as on the national in-sample checks page, with a nominal coverage of 0.9.

  • Least well reproduced: Haut-Uele (cases), Tshopo (cases) and Bas-Uele (cases) have the largest absolute bias.

  • Coverage: 16 of 16 composition rows have 90% coverage of at least 0.8; the lowest is Haut-Uele (cases) at 0.96.

  • Province levels: over the last two weeks the zone stage's infections are within 36.8% of the joint model's in every province, the furthest being Sud-Ubangi.

ProvinceStreamBias90% coverageCells
IturiConfirmed cases00.974060
IturiConfirmed deaths00.983976
Nord-KivuConfirmed cases00.982380
Nord-KivuConfirmed deaths00.982278
Haut-UeleConfirmed cases0.030.96832
Haut-UeleConfirmed deaths0.010.98704
Sud-KivuConfirmed cases011
Sud-KivuConfirmed deaths011
TshopoConfirmed cases0.020.97364
TshopoConfirmed deaths00.99203
Bas-UeleConfirmed cases0.01130
Bas-UeleConfirmed deaths0112
Sud-UbangiConfirmed cases012
Sud-UbangiConfirmed deaths011
Nord-UbangiConfirmed cases0112
Nord-UbangiConfirmed deaths012

Zone compositions ​

The zone compositions sit one level below the province compositions. Cases and deaths are separate compositions in the zone observations, so the two are checked separately.

Load the zone fit's inputs and prior
julia
# `zone_stage_inputs` and `zone_prior_draws` are defined in the shared setup,
# so the zone estimates and forecast pages read the same fixed inputs.
zone_inputs = zone_stage_inputs();
zone_patch = zone_inputs.patch_of_zone;
prior_chn_zone = zone_prior_draws(zone_inputs);

Confirmed cases ​

Zone case composition predictive checks
julia
# Per-draw expected, predictive and observed zone shares at every vintage,
# from the posterior and from the zone prior, per vintage and cumulative.
zone_ppc = zone_composition_draws(chn_local, zone_inputs);
zone_ppc_prior = zone_composition_draws(prior_chn_zone, zone_inputs);
zone_ppc_cum = zone_composition_draws(
    chn_local, zone_inputs;
    cumulative = true
);
zone_ppc_prior_cum = zone_composition_draws(
    prior_chn_zone, zone_inputs;
    cumulative = true
);
zone_ppc_fig = plot_zone_shares(
    zone_ppc.expected, zone_ppc.observed,
    zone_inputs.dates, zone_inputs.zone_labels;
    zone_patch, patch_labels = zone_inputs.patch_labels,
    pred_draws = zone_ppc.predictive_share,
    prior_draws = zone_ppc_prior.predictive_share, top = 15, ncols = 5
);
zone_ppc_cum_fig = plot_zone_shares(
    zone_ppc_cum.expected,
    zone_ppc_cum.observed, zone_inputs.dates, zone_inputs.zone_labels;
    zone_patch, patch_labels = zone_inputs.patch_labels,
    pred_draws = zone_ppc_cum.predictive_share,
    prior_draws = zone_ppc_prior_cum.predictive_share, top = 15, ncols = 5,
    title = "Zone share of the patch's cumulative confirmed cases, " *
        "modelled against observed"
);

The modelled share of the patch's confirmed cases at each vintage against the observed share, for the zones with the largest observed shares. A zone with a handful of cases per vintage has an observed share that jumps between zero and one, and its band spans the same range.

The same check on each zone's share of its patch's cumulative allocated cases at every vintage.

Confirmed deaths ​

Zone death composition predictive check
julia
zone_death_ppc_cum = zone_composition_draws(
    chn_local, zone_inputs;
    stream = :deaths, cumulative = true
);
zone_death_ppc_prior_cum = zone_composition_draws(
    prior_chn_zone, zone_inputs;
    stream = :deaths, cumulative = true
);
zone_death_ppc_cum_fig = plot_zone_shares(
    zone_death_ppc_cum.expected, zone_death_ppc_cum.observed,
    zone_inputs.dates, zone_inputs.zone_labels;
    zone_patch, patch_labels = zone_inputs.patch_labels,
    pred_draws = zone_death_ppc_cum.predictive_share,
    prior_draws = zone_death_ppc_prior_cum.predictive_share,
    top = 15, ncols = 5,
    title = "Zone share of the patch's cumulative confirmed deaths, " *
        "modelled against observed"
);

Each zone's share of its patch's cumulative allocated deaths at every vintage, for the zones with the largest observed death shares. Most per-vintage death cells are empty, so the check uses the cumulative allocation.

Zone composition calibration ​

Both compositions are scored the way the stream calibration scores the national streams, over every zone and vintage of each patch.

Zone composition calibration
julia
# The two compositions in one table, each patch row named by the
# composition it belongs to so the calibration plot reads them apart.
zone_calibration_table = let
    function tagged(stream, what)
        tbl = zone_composition_calibration(
            chn_local, zone_inputs; stream = stream
        )
        tbl[!, "Stream"] = string.(tbl[!, "Stream"], " ($what)")
        return tbl
    end
    vcat(tagged(:cases, "cases"), tagged(:deaths, "deaths"))
end;
zone_calibration_fig = plot_stream_calibration(zone_calibration_table);

StreamVintagesBias50% coverage90% coverage
Ituri (cases)406000.840.97
Nord-Kivu (cases)238000.860.98
Haut-Uele (cases)8320.030.830.96
Sud-Kivu (cases)1011
Tshopo (cases)3640.020.880.97
Bas-Uele (cases)300.010.91
Sud-Ubangi (cases)2011
Nord-Ubangi (cases)1200.831
All zones (cases)768100.850.97
Ituri (deaths)397600.90.98
Nord-Kivu (deaths)227800.880.98
Haut-Uele (deaths)7040.010.920.98
Sud-Kivu (deaths)1011
Tshopo (deaths)20300.920.99
Bas-Uele (deaths)1200.921
Sud-Ubangi (deaths)1011
Nord-Ubangi (deaths)2011
All zones (deaths)717700.890.98
The mean bias of the predictive allocation and the fractions of observed zone counts inside the central 50% and 90% predictive intervals. A cell with no case or death is covered by any interval that includes zero, so the coverage is read with the share of empty cells in mind.

Province levels ​

The zone model scores compositions within each province, so it should produce province infection estimates equivalent to the joint model. The table sets the zone infections summed over each province against the joint model's infections there, as the posterior mean over each of the last four weeks to the cut-off.

Zone stage against joint infections by province
julia
# The zone stage's patch infections against the joint's draws they were
# melded from, per week and province.
zone_level_table = let tbl = zone_patch_level_check(
        chn_local, zone_inputs, chn_joint
    )
    tbl[!, "Zone stage"] = round.(tbl[!, "Zone stage"]; digits = 0)
    tbl[!, "Joint"] = round.(tbl[!, "Joint"]; digits = 0)
    tbl[!, "Ratio"] = round.(tbl[!, "Ratio"]; digits = 3)
    tbl
end;
ProvinceWeek endingZone stageJointRatio
Ituri2026-10-264324320.999
Nord-Kivu2026-10-263973990.994
Haut-Uele2026-10-2661601.006
Sud-Kivu2026-10-26000.758
Tshopo2026-10-2615150.987
Bas-Uele2026-10-26110.947
Sud-Ubangi2026-10-26000.725
Nord-Ubangi2026-10-26331.013
All provinces2026-10-269089110.997
Ituri2026-11-024224221
Nord-Kivu2026-11-023903930.992
Haut-Uele2026-11-0267670.997
Sud-Kivu2026-11-02000.742
Tshopo2026-11-0215150.983
Bas-Uele2026-11-02110.914
Sud-Ubangi2026-11-02000.699
Nord-Ubangi2026-11-02331.023
All provinces2026-11-028999020.996
Ituri2026-11-094164151.004
Nord-Kivu2026-11-093923960.988
Haut-Uele2026-11-0974740.996
Sud-Kivu2026-11-09000.719
Tshopo2026-11-0916160.98
Bas-Uele2026-11-09110.871
Sud-Ubangi2026-11-09000.669
Nord-Ubangi2026-11-09441.031
All provinces2026-11-099039070.996
Ituri2026-11-164144150.998
Nord-Kivu2026-11-163974070.977
Haut-Uele2026-11-1682840.982
Sud-Kivu2026-11-16000.687
Tshopo2026-11-1616170.968
Bas-Uele2026-11-16110.818
Sud-Ubangi2026-11-16000.632
Nord-Ubangi2026-11-16551.005
All provinces2026-11-169159280.986
Posterior mean infections in each week ending on the date shown, from the zone stage summed over the province's zones and from the joint model, and their ratio.

Change over the past week ​

The comparison reads each zone's reproduction number on the frozen cut-off day from the frozen and live fits, alongside the live estimate at the current cut-off. The two fits condition on different weeks of data and on different parent fits. A zone carries its own walk only once it has reported 30 confirmed cases, so the comparison is restricted to zones walking in both fits.

Zone cut-off summaries shared by the comparisons
julia
# The frozen fit's fixed inputs, from the shared setup.
frozen_zone_inputs = frozen_zone_stage_inputs();

# One row per zone in `zs` with the median and the 50% and 90% intervals
# of its draws (one vector per zone), in the schema the zone dot plots
# read, plus the zone's key to match variants by and its confirmed cases
# to rank by. A zone with no finite draw is left out.
function _zone_cutoff_summary(draws, inputs, zs = eachindex(draws))
    keep = [i for (i, v) in enumerate(draws) if any(isfinite, v)]
    z = zs[keep]
    tbl = zone_summary_table(
        [filter(isfinite, draws[i]) for i in keep],
        inputs.zone_labels[z], inputs.patch_of_zone[z]
    )
    tbl.key = inputs.zone_keys[z]
    tbl.cases = inputs.cumulative[z]
    return tbl
end

# The `top` zones by confirmed cases in the first variant, one column per
# variant for the reproduction number. Each cell is a median with its 90%
# interval. Variants are matched by zone key.
function _zone_comparison_table(variants; top::Integer = 10)
    function cell(t, key, d, scale)
        i = findfirst(==(key), t.key)
        i === nothing && return ""
        f(x) = string(round(scale * x; digits = d))
        return string(f(t.median[i]), " (", f(t.lo90[i]), "–", f(t.hi90[i]), ")")
    end
    base = first(last(first(variants)))
    order = sortperm(base.cases; rev = true)[1:min(top, size(base, 1))]
    df = DataFrame(
        "Zone" => base.label[order],
        "Province" => [PROVINCE_LABELS[p] for p in base.patch[order]],
        "Cases" => base.cases[order]
    )
    for (q, name, d, scale) in ((:R, "R", 2, 1),),
            (label, s) in variants

        haskey(s, q) || continue
        df[!, "$name ($label)"] = [
            cell(s[q], k, d, scale)
                for k in base.key[order]
        ]
    end
    return df
end
_zone_comparison_table (generic function with 1 method)
Zone reproduction numbers from the frozen and live fits
julia
# Daily reproduction numbers rebuilt from both fits, and the zones walking
# in both as live index => frozen index, matched by key. The summaries
# take their labels and cases from the live inputs.
zone_rt_live = reconstruct_zone_rt(chn_local, zone_inputs)
zone_rt_frozen = reconstruct_zone_rt(frozen_local.chn, frozen_zone_inputs)
zone_week_pairs = [
    z => j
        for (z, k) in enumerate(zone_inputs.zone_keys)
        for j in (findfirst(==(k), frozen_zone_inputs.zone_keys),)
        if j !== nothing && zone_inputs.walking[z] &&
        frozen_zone_inputs.walking[j]
]
zone_week_variants = let zs = first.(zone_week_pairs), js = last.(zone_week_pairs),
        n_f = frozen_zone_inputs.n

    [
        "frozen fit at its cut-off" => (;
            R = _zone_cutoff_summary(
                [zone_rt_frozen[j][:, n_f] for j in js],
                zone_inputs, zs
            ),
        ),
        "live fit on the same day" => (;
            R = _zone_cutoff_summary(
                [zone_rt_live[z][:, n_f] for z in zs],
                zone_inputs, zs
            ),
        ),
        "live fit at its cut-off" => (;
            R = _zone_cutoff_summary(
                [zone_rt_live[z][:, end] for z in zs],
                zone_inputs, zs
            ),
        ),
    ]
end
zone_week_table = _zone_comparison_table(zone_week_variants);
zone_week_fig = plot_zone_comparison(
    [l => s.R for (l, s) in zone_week_variants];
    xlabel = "Reproduction number", reference_line = 1.0,
    title = "Zone reproduction number from the frozen and live fits"
);

# The frozen trajectories padded onto the live grid, undefined past the
# frozen cut-off, behind the live ones over the zone grid. The plot reads
# an undefined day as `missing`, where the reconstruction writes `NaN`.
zone_week_rt_fig = let grid = zone_inputs.t0:obs.n, n_f = frozen_zone_inputs.n
    asmissing(m) = replace(m, NaN => missing)
    frozen = map(zone_week_pairs) do (z, j)
        m = fill(NaN, size(zone_rt_frozen[j], 1), obs.n)
        m[:, 1:n_f] .= zone_rt_frozen[j]
        asmissing(m[:, grid])
    end
    zs = first.(zone_week_pairs)
    plot_rt_zones(
        [asmissing(zone_rt_live[z][:, grid]) for z in zs],
        zone_inputs.zone_labels[zs], zone_inputs.patch_of_zone[zs];
        patch_labels = zone_inputs.patch_labels,
        dates = grid_date.(grid), as_of_date = obs.cutoff,
        ranking = zone_inputs.cumulative[zs],
        ranking_label = "confirmed cases to date", top = 12,
        reference_rt = frozen, reference_label = "Frozen fit",
        title = "Zone reproduction number from the live fit, " *
            "with the frozen fit behind"
    )
end

ZoneProvinceCasesR (frozen fit at its cut-off)R (live fit on the same day)R (live fit at its cut-off)
BuniaIturi21050.67 (0.31–1.15)0.89 (0.53–1.35)0.85 (0.48–1.32)
RwamparaIturi12350.68 (0.31–1.22)0.86 (0.5–1.36)0.84 (0.45–1.41)
BeniNord-Kivu9050.89 (0.66–1.14)0.92 (0.68–1.22)0.92 (0.67–1.16)
NiziIturi8830.68 (0.27–1.39)0.98 (0.54–1.61)0.9 (0.47–1.52)
MongbwaluIturi7400.9 (0.35–2.04)0.97 (0.42–2.01)0.88 (0.35–2.04)
KatwaNord-Kivu6980.8 (0.46–1.26)0.85 (0.56–1.26)0.88 (0.57–1.31)
MandimaIturi6390.83 (0.36–1.38)0.71 (0.3–1.35)0.67 (0.23–1.38)
MangalaIturi4600.85 (0.36–1.8)1.4 (0.72–2.46)1.16 (0.56–2.01)
ButemboNord-Kivu3560.82 (0.48–1.29)0.86 (0.56–1.28)0.89 (0.57–1.33)
LitaIturi3520.68 (0.27–1.4)0.91 (0.48–1.56)0.85 (0.42–1.51)
The reproduction number from the frozen and live fits for the ten zones with most confirmed cases.

The fifteen zones the frozen fit ranks highest.

Reproduction number trajectories for the twelve zones with most confirmed cases, with the frozen fit drawn behind the live one.

Estimate evolution across releases ​

How the reproduction number and confirmed cases of the twelve zones with most confirmed cases to date have moved as releases accrued, read from the estimates each release published at its own cut-off. A zone that a release did not estimate has no mark for that release.

Load the per-zone estimates of each release
julia
# Written by `scripts/score_releases.jl` from each release's
# `zone_estimates.csv`, one row per release, zone and quantity, keyed as
# `<province>.<zone>` like the zone model. A missing file reads as an empty
# table, which the figures report as nothing collected.
zone_release_df = _release_data(
    joinpath("zone", "released_estimates.csv"),
    (;
        release = String, date = Date, area = String, label = String,
        province = String, patch = String, quantity = String,
        median = Float64, lo90 = Float64, hi90 = Float64,
    )
)
# The twelve zones with most confirmed cases in the current data, among
# those any release estimated.
_release_zones = let keys = Set(zone_release_df.area),
        order = sortperm(zone_inputs.cumulative; rev = true)

    first(
        [zone_inputs.zone_keys[z] for z in order if zone_inputs.zone_keys[z] in keys],
        12
    )
end
_release_zone_titles = Dict(zip(zone_inputs.zone_keys, zone_inputs.zone_labels))
_zone_release_groups(q) = area_evolution_groups(
    zone_release_df, q; areas = _release_zones, labels = _release_zone_titles
)
_no_zone_release_note = "No releases scored yet. This fills in as releases accumulate."
zone_rt_release_fig = plot_evolution_by_group(
    _zone_release_groups("R_T");
    ylabel = "Reproduction number",
    title = "Zone reproduction number by release",
    released_label = "Released estimate", marks_label = "median, 90% bar",
    refline = 1.0, ymax = 3.0, ncols = 4,
    empty_note = _no_zone_release_note
);
zone_cases_release_fig = plot_evolution_by_group(
    _zone_release_groups("cases");
    ylabel = "Confirmed cases",
    title = "Zone confirmed cases by release",
    released_label = "Reported by the release cut-off", marks_label = "",
    shared_yrange = false, ncols = 4, empty_note = _no_zone_release_note
);

Reproduction number by release ​

Each zone's reproduction number at the cut-off of each release, as a median with its 90% interval bar. The axis is fixed at three, and an interval past it is clamped and marked with an open triangle.

Confirmed cases by release ​

The confirmed cases each zone had reported by the cut-off of each release, each panel on its own scale.

Saving zone in-sample outputs ​

Write the summary bullets
julia
# The bullets under the summary heading at the top of the page. They read
# the calibration table built further down, so they are written here and
# read back when the site is assembled.
evaluation_insample_zone_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    # The per-patch rows only: the table's "All zones" rows pool them and
    # would be counted twice.
    fitted = filter(
        r -> isfinite(r["Bias"]) && !startswith(r["Stream"], "All zones"),
        zone_calibration_table
    )
    worst = first(
        sort(fitted, "Bias"; by = abs, rev = true), min(3, size(fitted, 1))
    )[!, "Stream"]
    low = first(sort(fitted, "90% coverage"), min(1, size(fitted, 1)))
    overall = [
        isempty(worst) ? nothing :
            string(
                "- **Least well reproduced:** ", join_names(worst),
                length(worst) == 1 ? " has" : " have",
                " the largest absolute bias."
            ),
        string(
            "- **Coverage:** ", count(>=(0.8), fitted[!, "90% coverage"]),
            " of ", size(fitted, 1),
            " composition rows have 90% coverage of at least 0.8",
            size(low, 1) == 0 ? "." :
                string(
                    "; the lowest is ", low[1, "Stream"], " at ",
                    fmt(low[1, "90% coverage"]), "."
                )
        ),
    ]
    # The province whose zone-stage level sits furthest from the joint's
    # over the last fortnight.
    recent = filter(
        r -> r["Province"] != "All provinces" &&
            r["Week ending"] >= maximum(zone_level_table[!, "Week ending"]) -
            Day(7),
        zone_level_table
    )
    if size(recent, 1) > 0
        far = recent[argmax(abs.(recent[!, "Ratio"] .- 1)), :]
        push!(
            overall, string(
                "- **Province levels:** over the last two weeks the zone ",
                "stage's infections are within ",
                fmt(100 * abs(far["Ratio"] - 1)),
                "% of the joint model's in every province, the furthest ",
                "being ", far["Province"], "."
            )
        )
    end
    # One row per province and composition.
    rows = Dict(r["Stream"] => r for r in eachrow(zone_calibration_table))
    cells = [
        (PROVINCE_LABELS[p], what, label) for p in 1:N_PATCHES
            for (what, label) in
            (("cases", "Confirmed cases"), ("deaths", "Confirmed deaths"))
    ]
    value(label, what, col) = let r = get(
            rows, string(label, " (", what, ")"), nothing
        )
        r === nothing ? missing : r[col]
    end
    table = DataFrame(
        "Province" => [c[1] for c in cells],
        "Stream" => [c[3] for c in cells],
        "Bias" => [finite_round(value(c[1], c[2], "Bias")) for c in cells],
        "90% coverage" => [
            finite_round(value(c[1], c[2], "90% coverage")) for c in cells
        ],
        "Cells" => [value(c[1], c[2], "Vintages") for c in cells],
    )
    join(
        [
            join(filter(!isnothing, overall), "\n"),
            markdown_table(table; missing_text = "n/a"),
        ],
        "\n\n"
    )
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_insample_zone.md"),
    evaluation_insample_zone_summary
);