Skip to content

Province in-sample checks ​

This page checks whether the joint model reproduces the per-province data it was fitted to, follows each province's published estimates across releases, and refits it as a single population to check the spatial structure. The national checks are on the national in-sample checks page.

Load packages, data and fitted chains
julia
# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))
validation_forecast_from (generic function with 1 method)
julia
# The fits this page reads, loaded from the cache here.
chn_joint = load_fit("joint")
# `sens_no_patches` is the headline with `n_patches = 1`, the check on the
# spatial structure.
chn_no_patches = load_fit("sens_no_patches")
posterior_C_joint = vec(Array(chn_joint[:C_T]))
posterior_C_no_patches = vec(Array(chn_no_patches[:C_T]));

Summary ​

Whether each province's cases and deaths are reproduced, from the checks below. Bias and coverage are defined as on the national in-sample checks page, with coverage of the 90% predictive interval. The split columns score each province's share at the observed national total, and the count columns score its counts once the national total is predicted as well. A check that was not scored reads n/a.

  • Least well reproduced: Analysed specimens, Nord-Kivu; Analysed specimens, Haut-Uele and Analysed specimens, Ituri have the largest absolute bias in the province split.

  • Coverage: 24 of 24 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Nord-Kivu at 0.92.

  • Count-scale coverage: 16 of 16 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Ituri at 0.92.

ProvinceStreamBias (split)90% coverage (split)Bias (counts)90% coverage (counts)
IturiConfirmed cases-0.010.950.050.95
IturiConfirmed deaths0.010.94-0.030.92
Nord-KivuConfirmed cases0.010.930.040.96
Nord-KivuConfirmed deaths-0.030.92-0.060.94
Haut-UeleConfirmed cases0.050.930.060.96
Haut-UeleConfirmed deaths0.010.97-0.010.98
Sud-KivuConfirmed cases0.010.990.010.99
Sud-KivuConfirmed deaths00.9901
TshopoConfirmed cases0.040.950.040.97
TshopoConfirmed deaths0.010.980.010.98
Bas-UeleConfirmed cases00.9800.98
Bas-UeleConfirmed deaths0.010.970.010.97
Sud-UbangiConfirmed cases00.9800.98
Sud-UbangiConfirmed deaths00.9900.99
Nord-UbangiConfirmed cases0.010.980.010.98
Nord-UbangiConfirmed deaths0.010.990.010.99

Province prior predictive check ​

Whether the province model's prior, before any data are fitted, brackets each province's observed share of the confirmed cases and deaths.

Draw from the province model's prior
julia
prior_patch_chn = patch_prior_draws(obs);

prior_province_table = patch_overview_table(prior_patch_chn, N_PATCHES);
Show prior province summary table
ProvinceReproduction numberCumulative infectionsShare of infections (%)Relative ascertainment
Ituri0.77 (0.07–1.67)1110252 (1504–3940792)74.4 (14.2–99.1)1.01 (0.59–1.74)
Nord-Kivu1.16 (0.08–2.21)69277 (13–7401325)7.5 (0.2–34.2)1.01 (0.63–1.76)
Haut-Uele1.12 (0.07–2.18)21031 (4–2001289)2.9 (0.1–12.7)1.0 (0.57–1.63)
Sud-Kivu1.19 (0.09–2.3)35901 (6–6392968)4.3 (0.1–26.0)0.99 (0.55–1.69)
Tshopo1.19 (0.09–2.27)11549 (2–2515810)1.5 (0.0–11.1)1.0 (0.6–1.67)
Bas-Uele1.21 (0.09–2.27)5413 (1–1219884)0.7 (0.0–5.3)1.01 (0.6–1.69)
Sud-Ubangi1.23 (0.09–2.44)6986 (1–2686280)0.8 (0.0–10.1)1.01 (0.59–1.65)
Nord-Ubangi1.22 (0.09–2.35)3543 (1–1389117)0.5 (0.0–5.5)1.0 (0.56–1.65)
Province composition prior predictive checks
julia
prior_case_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province, prior"
);

prior_death_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province, prior"
);

Spatial hyperparameters against their prior ​

Whether the data move the province model's spatial hyperparameters away from their prior.

Spatial hyperparameter pair plot
julia
spatial_pair_fig = plot_pair(
    chn_joint, spatial_params;
    prior = prior_patch_chn, labels = spatial_labels
);

Joint posterior of the spatial hyperparameters, with the prior overlaid. The province reproduction numbers and case totals are derived from these and are in the [province estimates](@ref "Province estimates").

Province compositions ​

Whether each province's modelled share of the national confirmed cases and deaths reproduces the observed share at every spatial vintage (see the province compositions Methods section).

Province composition posterior predictive checks
julia
province_case_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province"
);

province_death_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province"
);

The four province terms for the isolation and laboratory data are checked the same way. A gap in a panel is a day on which that province printed nothing.

Province laboratory, occupancy, bed and admission split posterior predictive checks
julia
province_lab_bin_days = [
    maximum(province_lab.days[province_lab.bins .== b])
        for b in 1:maximum(province_lab.bins)
];

province_lab_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    days = province_lab_bin_days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Analysed specimen share by province, by week"
);

province_occupancy_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_occupancy_share, rows = province_isolation,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation occupancy share by province"
);

province_beds_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_capacity_share, rows = province_capacity,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation bed share by province"
);

province_admissions_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_admissions_share, rows = province_admissions,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Admission share by province"
);

The split of each calendar week's analysed specimens among the provinces, which identifies the background split.

The split of the patients in isolation among the provinces printed that day, on the weekly days the fit scores, against the per-patch bed demand.

The split of the beds among the provinces printed that day, on the days a count changed, against each patch's modelled share of the national capacity.

The split of each day's 24h admissions among the provinces printed that day, against each patch's modelled admissions.

Province stream calibration ​

Whether each province's share predictions are calibrated at the observed national total, scored as in the national stream calibration. The analysed-specimen rows score the weekly laboratory split the same way, at each week's observed national analysed total.

Build the province calibration panels
julia
province_case_panels = province_composition_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    stream = "Confirmed cases", n_patches = N_PATCHES
);
province_death_panels = province_composition_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_lab_panels = province_composition_panels(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    stream = "Analysed specimens", n_patches = N_PATCHES
);
province_panels = vcat(
    province_case_panels, province_death_panels, province_lab_panels
);
province_calibration_table = stream_calibration(province_panels);
province_calibration_fig = plot_stream_calibration(
    province_calibration_table
);

Province calibration table
StreamVintagesBias50% coverage90% coverage
Confirmed cases, Ituri133-0.010.610.95
Confirmed cases, Nord-Kivu1330.010.560.93
Confirmed cases, Haut-Uele1330.050.730.93
Confirmed cases, Sud-Kivu1330.010.990.99
Confirmed cases, Tshopo1330.040.830.95
Confirmed cases, Bas-Uele13300.920.98
Confirmed cases, Sud-Ubangi13300.980.98
Confirmed cases, Nord-Ubangi1330.010.980.98
Confirmed deaths, Ituri1320.010.640.94
Confirmed deaths, Nord-Kivu132-0.030.620.92
Confirmed deaths, Haut-Uele1320.010.790.97
Confirmed deaths, Sud-Kivu13200.990.99
Confirmed deaths, Tshopo1320.010.890.98
Confirmed deaths, Bas-Uele1320.010.970.97
Confirmed deaths, Sud-Ubangi13200.990.99
Confirmed deaths, Nord-Ubangi1320.010.990.99
Analysed specimens, Ituri210.080.951
Analysed specimens, Nord-Kivu21-0.170.91
Analysed specimens, Haut-Uele21-0.160.481
Analysed specimens, Sud-Kivu210.060.711
Analysed specimens, Tshopo210.020.91
Analysed specimens, Bas-Uele210.050.711
Analysed specimens, Sud-Ubangi210.080.951
Analysed specimens, Nord-Ubangi210.060.951

Province counts ​

Whether each province's cases and deaths are reproduced as counts once the national total is predicted as well.

Build the province count panels
julia
# Each province's confirmed cases and deaths split from the national
# posterior predictive by the fitted shares, shared with the dashboard's
# province series (see `joint_count_replicates` in `docs/pages/_setup.jl`).
province_count_replicates = joint_count_replicates();
province_case_count_panels = province_count_replicates.cases.panels;
province_death_count_panels = province_count_replicates.deaths.panels;
province_count_calibration_table = stream_calibration(
    vcat(province_case_count_panels, province_death_count_panels)
);
province_count_calibration_fig = plot_stream_calibration(
    province_count_calibration_table
);
Province count calibration plot

Province count calibration table
StreamVintagesBias50% coverage90% coverage
Confirmed cases, Ituri1330.050.710.95
Confirmed cases, Nord-Kivu1330.040.690.96
Confirmed cases, Haut-Uele1330.060.710.96
Confirmed cases, Sud-Kivu1330.010.990.99
Confirmed cases, Tshopo1330.040.850.97
Confirmed cases, Bas-Uele13300.920.98
Confirmed cases, Sud-Ubangi13300.980.98
Confirmed cases, Nord-Ubangi1330.010.980.98
Confirmed deaths, Ituri132-0.030.630.92
Confirmed deaths, Nord-Kivu132-0.060.60.94
Confirmed deaths, Haut-Uele132-0.010.740.98
Confirmed deaths, Sud-Kivu13200.991
Confirmed deaths, Tshopo1320.010.890.98
Confirmed deaths, Bas-Uele1320.010.970.97
Confirmed deaths, Sud-Ubangi13200.990.99
Confirmed deaths, Nord-Ubangi1320.010.990.99

Province correlations and totals ​

Which province quantities trade off against each other, and whether each province's case and death totals agree with the observed ones.

Province posterior correlation heatmap
julia
# Per-province entries of a vector deterministic, one draw vector each,
# keyed `<name>_<patch>` so they can sit side by side in one table.
function _patch_columns(chn, key::Symbol, name::AbstractString; f = identity)
    draws = [collect(v) for v in vec(collect(chn[key]))]
    return [
        Symbol(name, "_", p) => Float64[f(d[p]) for d in draws]
            for p in 1:N_PATCHES
    ]
end

# LaTeX label for one province's entry of a quantity. Spaces in a province
# name need escaping inside `\mathrm`.
_province_tex(sym, p) = string(
    sym, raw"\ (\mathrm{", replace(PROVINCE_LABELS[p], " " => raw"\ "), "})"
)
province_correlation_draws = NamedTuple(
    vcat(
        [:C_T => vec(Array(chn_joint[:C_T]))],
        _patch_columns(chn_joint, :R_T_patch, "R_T"),
        _patch_columns(chn_joint, :province_ascertainment, "asc"),
        _patch_columns(chn_joint, :CFR_patch, "CFR")
    )
);
province_correlation_labels = merge(
    Dict(:C_T => raw"C_T"),
    Dict(
        Symbol(name, "_", p) => _province_tex(tex, p)
            for p in 1:N_PATCHES
            for (name, tex) in (
                ("R_T", raw"R_T"), ("asc", raw"p_\mathrm{asc}"),
                ("CFR", raw"\mathrm{CFR}"),
            )
    )
);
province_correlation_fig = plot_correlation_heatmap(
    province_correlation_draws; labels = province_correlation_labels
);

Province totals against observed
julia
# Keyed by panel title, which names the stream and the province. A total
# that is the same in every draw (a patch with no deaths at any vintage) has
# no spread to plot and gives the corner plot a zero-width axis, so it is
# left out.
_varying_panels = filter(
    p -> length(unique(sum(r) for r in p.replicates)) > 1, province_panels
);
province_totals = NamedTuple(
    Symbol(p.title) => [sum(Float64.(r)) for r in p.replicates]
        for p in _varying_panels
);
province_observed = NamedTuple(
    Symbol(p.title) => Float64(sum(p.observed)) for p in _varying_panels
);
province_pairs_fig = plot_stream_pairs(province_totals, province_observed);

Estimate evolution across releases ​

How each province's estimates have moved as releases accrued, read from the estimates each release published at its own cut-off. Each panel starts at the first release that published per-province estimates.

Load the per-province estimates of each release
julia
# Written by `scripts/score_releases.jl` from each release's
# `province_estimates.csv`, one row per release, province and quantity. A
# missing file reads as an empty table, which the figures report as nothing
# collected.
province_release_df = _release_data(
    joinpath("province", "released_estimates.csv"),
    (;
        release = String, date = Date, area = String, patch = String,
        pooled = Int, quantity = String, median = Float64, lo90 = Float64,
        hi90 = Float64,
    )
)
# One panel per province, the patches first in their usual order. A
# province pooled into a shared patch is marked, since it carries its
# patch's reproduction number and case-fatality ratio.
_release_provinces = unique(vcat(PROVINCE_NAMES, province_release_df.area))
_release_pooled = Set(
    r.area for r in eachrow(province_release_df)
        if coalesce(r.pooled, 0) == 1
)
_release_titles = Dict(
    a => string(
        get(Dict(zip(PROVINCE_NAMES, PROVINCE_LABELS)), a, a),
        a in _release_pooled ? " (pooled)" : ""
    ) for a in _release_provinces
)
_province_release_groups(q) = area_evolution_groups(
    province_release_df, q;
    areas = _release_provinces, labels = _release_titles
)
_no_release_note = "No releases scored yet. This fills in as releases accumulate."
province_rt_release_fig = plot_evolution_by_group(
    _province_release_groups("R_T");
    ylabel = "Reproduction number",
    title = "Province reproduction number by release",
    released_label = "Released estimate", marks_label = "median, 90% bar",
    refline = 1.0, ymax = 3.0, ncols = 4,
    empty_note = _no_release_note
);
province_cases_release_fig = plot_evolution_by_group(
    _province_release_groups("cases");
    ylabel = "Confirmed cases",
    title = "Province confirmed cases by release",
    released_label = "Reported by the release cut-off", marks_label = "",
    shared_yrange = false, ncols = 4, empty_note = _no_release_note
);
province_cfr_release_fig = plot_evolution_by_group(
    _province_release_groups("CFR");
    ylabel = "Case-fatality ratio",
    title = "Province case-fatality ratio by release",
    released_label = "Released estimate", marks_label = "median, 90% bar",
    ymax = 1.0, ncols = 4,
    empty_note = _no_release_note
);

Reproduction number by release ​

Each province's reproduction number at the cut-off of each release, as a median with its 90% interval bar. The axis is fixed at three, and an interval past it is clamped and marked with an open triangle.

Confirmed cases by release ​

The confirmed cases each province had reported by the cut-off of each release, each panel on its own scale.

Case-fatality ratio by release ​

Each province's case-fatality ratio at each release, as a median with its 90% interval bar.

Province sensitivity to assumptions ​

Spatial structure sensitivity ​

We refit the joint model with its eight patches reduced to one well-mixed population. Splitting the country into provinces adds no national data, so the national outbreak size should not move far either way. The two still differ, because the country grows at the force-weighted mean of the provinces' reproduction numbers, which sits above the central trend they pool toward. That gap is first order in the deviation scale, so a large gap between the two posteriors would point at the deviation priors.

julia
spatial_sensitivity_table = streams_table(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population (n_patches = 1)" => posterior_C_no_patches
);
spatial_sensitivity_table
2×7 DataFrame
RowStreamLower 90%Lower 60%Lower 30%Upper 30%Upper 60%Upper 90%
StringFloat64Float64Float64Float64Float64Float64
1Meta-population (headline)15569.018036.020118.024013.026620.032374.0
2Single population (n_patches = 1)15598.018658.020715.025069.027865.033332.0
Spatial-structure density overlay
julia
spatial_sensitivity_fig = plot_density_overlay(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population" => posterior_C_no_patches;
    xlabel = "Cumulative infections"
);

Posterior densities of the cumulative infections under the headline province model and the single-population fit.

The spatial structure could also move other national quantities that both fits estimate, so their posteriors should sit on top of each other.

National quantities under both structures
julia
spatial_rt_fig = plot_rt_streams(
    [
        (;
            label = "Single population (n_patches = 1)",
            chn = chn_no_patches, rt_start = _rt_start_plot,
            rt_walk_start = clamp(
                _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
                obs.n
            ), colour = :steelblue,
        ),
    ];
    joint = (;
        chn = chn_joint, rt_start = _rt_start_plot,
        rt_walk_start = clamp(
            _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
            obs.n
        ),
    ),
    n = obs.n, breakpoint = _BREAKPOINT,
    as_of_date = string(obs.cutoff), seeding = obs.seeding,
    display_start = _rt_start_plot, ncols = 1,
    title = "National reproduction number under both structures",
    reference_label = "the meta-population headline",
    panel_label = "the single-population fit"
);

spatial_cfr_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:CFR])),
    "Single population" => vec(Array(chn_no_patches[:CFR]));
    xlabel = "Case-fatality ratio"
);

spatial_rt_density_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:R_T])),
    "Single population" => vec(Array(chn_no_patches[:R_T]));
    xlabel = "Reproduction number at the cut-off"
);

The national reproduction number from the single-population fit (coloured) against the central trend of the headline province model (grey), which is hidden where the two agree.

Posterior densities of the case-fatality ratio under both structures.

Posterior densities of the reproduction number at the cut-off under both structures.
National quantities under both structures, as a table
julia
# One row per national quantity, one column per structure, each cell a
# median with a 90% credible interval. Built here rather than by stacking
# two `summary_table` calls, so the two structures sit side by side and the
# reader compares along a row.
spatial_quantities_table = let
    # A count rounded to zero decimals still prints a trailing ".0", so
    # whole-number quantities go through `Int`.
    fmt(x, d) = d <= 0 ? string(round(Int, x)) : string(round(x; digits = d))
    cell(v, d) = string(
        fmt(quantile(v, 0.5), d), " (",
        fmt(quantile(v, 0.05), d), "–", fmt(quantile(v, 0.95), d), ")"
    )
    rows = [
        ("Cumulative infections", :C_T, 0),
        ("Reproduction number at the cut-off", :R_T, 2),
        ("Case-fatality ratio", :CFR, 2),
        ("Outbreak age (days)", :T, 0),
        ("Latest growth rate (per day)", :r, 3),
    ]
    DataFrame(
        "Quantity" => [r[1] for r in rows],
        "Meta-population (headline)" => [
            cell(
                vec(Array(chn_joint[r[2]])), r[3]
            )
                for r in rows
        ],
        "Single population" => [
            cell(vec(Array(chn_no_patches[r[2]])), r[3])
                for r in rows
        ]
    )
end;
5×3 DataFrame
RowQuantityMeta-population (headline)Single population
StringStringString
1Cumulative infections22083 (15569–32374)22612 (15598–33332)
2Reproduction number at the cut-off1.03 (0.76–1.4)0.97 (0.69–1.26)
3Case-fatality ratio0.52 (0.4–0.63)0.52 (0.4–0.64)
4Outbreak age (days)228 (219–244)230 (220–245)
5Latest growth rate (per day)0.002 (-0.019–0.025)-0.002 (-0.024–0.016)
The cumulative infections, reproduction number at the cut-off, case-fatality ratio, outbreak age and latest growth rate under both structures, as credible intervals. The agreement is read from this table.

Province parameter recovery ​

Whether the model recovers each province's values when fitted to data it simulated itself, from the same runs as the national parameter recovery.

Recovery figure
julia
province_recovery_results = recovery_results()
_recovery_short = [
    "C_T_patch" => "C_T", "R_T_patch" => "R_T", "CFR_patch" => "CFR",
    "province_ascertainment" => "ascertainment",
    "region_drift_sd" => "drift SD",
]
province_recovery_quantities = [
    "$(k)[$(p)]" for (k, _) in _recovery_short for p in PROVINCE_LABELS
]
province_recovery_fig = isempty(province_recovery_results.params) ? nothing :
    plot_recovery(
        province_recovery_results.params, province_recovery_results.draws,
        province_recovery_results.prior;
        quantities = province_recovery_quantities,
        labels = Dict(
            "$(k)[$(p)]" => "$(v), $(p)" for (k, v) in _recovery_short
            for p in PROVINCE_LABELS
        ),
        panel_labels = Dict(
            "$(k)[$(p)]" => p for (k, _) in _recovery_short for p in PROVINCE_LABELS
        ),
        row_labels = last.(_recovery_short),
        log_x = ["C_T_patch[$(p)]" for p in PROVINCE_LABELS]
    );

Each seed's posterior median with its 50% and 90% intervals divided by its true value (top). Below, each quantity by province is on its own scale, with the prior in grey, each seed's posterior in its colour and its true value dashed.
Summary table across seeds
julia
province_recovery_summary = isempty(province_recovery_results.params) ?
    DataFrame() :
    recovery_summary_table(province_recovery_results.params; province = true);

province_recovery_summary_display = isempty(province_recovery_summary) ?
    Markdown.parse("No parameter-recovery run is available for this build.") :
    MarkdownTable(province_recovery_summary);
QuantityTruth, lowestTruth, highestRelative error, medianRelative error, rangez, medianTruth in 90%Outside 99%
CFR_patch[Bas-Uele]0.2480.593-0.08-0.16 to 0.07-0.413/30
CFR_patch[Haut-Uele]0.2020.508-0.01-0.04 to 0.31-0.063/30
CFR_patch[Ituri]0.1690.5610-0.11 to 0.570.032/30
CFR_patch[Nord-Kivu]0.2020.5410-0.07 to 0.320.063/30
CFR_patch[Nord-Ubangi]0.1960.537-0.08-0.15 to 0.35-0.533/30
CFR_patch[Sud-Kivu]0.2120.4860.05-0.1 to 0.250.323/30
CFR_patch[Sud-Ubangi]0.2170.5650.03-0.13 to 0.210.223/30
CFR_patch[Tshopo]0.2050.5340.09-0.07 to 0.30.493/30
C_T_patch[Bas-Uele]22360.30.08 to 2.390.872/31
C_T_patch[Haut-Uele]42.71680.07-0.01 to 0.240.443/30
C_T_patch[Ituri]219029700-0.02-0.14 to 0.03-0.053/30
C_T_patch[Nord-Kivu]44.23810.130.04 to 0.170.623/30
C_T_patch[Nord-Ubangi]8.72212-0.18-0.3 to -0.01-0.573/30
C_T_patch[Sud-Kivu]113157-0.09-0.1 to -0.02-0.233/30
C_T_patch[Sud-Ubangi]11.648.90.320.15 to 0.491.113/30
C_T_patch[Tshopo]10.261.5-0.11-0.2 to 0.34-0.133/30
R_T_patch[Bas-Uele]1.181.72-0.03-0.22 to 0.13-0.112/30
R_T_patch[Haut-Uele]0.9741.510.04-0.14 to 0.430.352/30
R_T_patch[Ituri]1.031.460.07-0.16 to 0.250.582/30
R_T_patch[Nord-Kivu]0.7681.430.1-0.13 to 0.250.693/30
R_T_patch[Nord-Ubangi]1.272.23-0.09-0.22 to 0.07-0.643/30
R_T_patch[Sud-Kivu]1.291.84-0.09-0.29 to 0.04-0.652/30
R_T_patch[Sud-Ubangi]1.181.66-0.03-0.03 to 0.06-0.023/30
R_T_patch[Tshopo]0.8111.470.07-0.12 to 0.390.53/30
province_ascertainment[Bas-Uele]1.171.27-0.19-0.26 to -0.05-1.651/30
province_ascertainment[Haut-Uele]0.7031.43-0.03-0.23 to 0.26-0.063/30
province_ascertainment[Ituri]0.8971.12-0.03-0.07 to 0.13-0.283/30
province_ascertainment[Nord-Kivu]0.9731.13-0.07-0.12 to 0.04-0.253/30
province_ascertainment[Nord-Ubangi]0.4981.360.39-0.19 to 0.741.861/30
province_ascertainment[Sud-Kivu]0.7910.8710.160.1 to 0.231.022/30
province_ascertainment[Sud-Ubangi]1.181.47-0.2-0.3 to -0.06-1.32/30
province_ascertainment[Tshopo]0.7841.040.16-0.08 to 0.280.793/30
province_suspected_ascertainment[Bas-Uele]0.6060.8220.07-0.11 to 0.140.023/30
province_suspected_ascertainment[Haut-Uele]0.5660.8130.1-0.11 to 0.190.533/30
province_suspected_ascertainment[Ituri]0.5370.9030.01-0.05 to 0.07-0.063/30
province_suspected_ascertainment[Nord-Kivu]0.7240.819-0.01-0.37 to 0.24-0.162/30
province_suspected_ascertainment[Nord-Ubangi]0.4890.750.03-0.11 to 0.780.043/30
province_suspected_ascertainment[Sud-Kivu]0.4360.710.090.08 to 0.30.383/30
province_suspected_ascertainment[Sud-Ubangi]0.6040.7730.1-0.08 to 0.160.493/30
province_suspected_ascertainment[Tshopo]0.5310.7770.110.02 to 0.150.423/30
region_drift_sd[Bas-Uele]0.0160.119-0.19-0.47 to -0.16-0.132/30
region_drift_sd[Haut-Uele]0.01210.0935-0.2-0.34 to 0.03-0.213/30
region_drift_sd[Ituri]0.009980.115-0.03-0.41 to 0.240.142/30
region_drift_sd[Nord-Kivu]0.009480.129-0.2-0.36 to 0.33-0.173/30
region_drift_sd[Nord-Ubangi]0.005310.131-0.37-0.45 to 1.28-0.612/30
region_drift_sd[Sud-Kivu]0.00830.132-0.31-0.52 to 0.47-0.522/30
region_drift_sd[Sud-Ubangi]0.006160.0753-0.08-0.1 to 1.070.023/30
region_drift_sd[Tshopo]0.01340.142-0.09-0.54 to -0.080.022/30
The error of each seed's posterior median relative to the truth, and the z-score of the truth, summarised across seeds.
Each seed's recovered province values
julia
province_recovery = let r = province_recovery_results.params
    isempty(r) ? DataFrame() :
        r[
            occursin.("[", r.quantity), [
                :seed, :quantity, :truth, :median, :lower_90, :upper_90,
                :covered_90,
            ],
        ]
end;

province_recovery_display = isempty(province_recovery) ?
    Markdown.parse("No parameter-recovery run is available for this build.") : MarkdownTable(province_recovery);
seedquantitytruthmedianlower_90upper_90covered_90
1CFR_patch[Bas-Uele]0.2740.2520.1810.336true
1CFR_patch[Haut-Uele]0.2620.2510.1830.336true
1CFR_patch[Ituri]0.2420.2430.1740.325true
1CFR_patch[Nord-Kivu]0.2490.2480.1810.333true
1CFR_patch[Nord-Ubangi]0.3060.2590.1870.352true
1CFR_patch[Sud-Kivu]0.2740.2480.1780.332true
1CFR_patch[Sud-Ubangi]0.2430.2510.1830.337true
1CFR_patch[Tshopo]0.2270.2470.180.33true
1C_T_patch[Bas-Uele]2.0056.8033.49412.524false
1C_T_patch[Haut-Uele]88.949110.19983.809153.448true
1C_T_patch[Ituri]11471.59411248.9359631.8613686.914true
1C_T_patch[Nord-Kivu]380.947396.593291.664548.025true
1C_T_patch[Nord-Ubangi]37.6826.38916.5842.746true
1C_T_patch[Sud-Kivu]113.196102.33373.7150.677true
1C_T_patch[Sud-Ubangi]42.21955.88138.66379.78true
1C_T_patch[Tshopo]61.52948.9433.98171.413true
1R_T_patch[Bas-Uele]1.3751.5611.2371.999true
1R_T_patch[Haut-Uele]1.5111.5731.2572.022true
1R_T_patch[Ituri]1.4631.5711.2662.009true
1R_T_patch[Nord-Kivu]1.4291.5711.2562.004true
1R_T_patch[Nord-Ubangi]1.4671.5691.2582.007true
1R_T_patch[Sud-Kivu]1.5091.5651.2562.005true
1R_T_patch[Sud-Ubangi]1.4781.5651.262.028true
1R_T_patch[Tshopo]1.4741.5761.2642.006true
1province_ascertainment[Bas-Uele]1.2690.9370.5281.173false
1province_ascertainment[Haut-Uele]1.4311.1070.9241.558true
1province_ascertainment[Ituri]1.1231.0480.8681.372true
1province_ascertainment[Nord-Kivu]1.131.0560.8971.378true
1province_ascertainment[Nord-Ubangi]0.4980.8680.4981.059false
1province_ascertainment[Sud-Kivu]0.8711.0090.8061.282true
1province_ascertainment[Sud-Ubangi]1.2771.0270.841.361true
1province_ascertainment[Tshopo]0.7840.9990.7681.277true
1province_suspected_ascertainment[Bas-Uele]0.8220.8770.4610.977true
1province_suspected_ascertainment[Haut-Uele]0.8130.8940.6660.977true
1province_suspected_ascertainment[Ituri]0.9030.9160.7670.981true
1province_suspected_ascertainment[Nord-Kivu]0.7240.8980.6810.977true
1province_suspected_ascertainment[Nord-Ubangi]0.4890.8680.4360.975true
1province_suspected_ascertainment[Sud-Kivu]0.6870.8930.670.978true
1province_suspected_ascertainment[Sud-Ubangi]0.7730.8930.6730.976true
1province_suspected_ascertainment[Tshopo]0.7770.8910.6670.979true
1region_drift_sd[Bas-Uele]0.0160.0130.001040.0497true
1region_drift_sd[Haut-Uele]0.01210.01240.001010.0492true
1region_drift_sd[Ituri]0.009980.01240.001050.0491true
1region_drift_sd[Nord-Kivu]0.009480.01260.001130.045true
1region_drift_sd[Nord-Ubangi]0.005310.01210.001060.0472true
1region_drift_sd[Sud-Kivu]0.00830.01220.001020.0487true
1region_drift_sd[Sud-Ubangi]0.006160.01270.001040.0498true
1region_drift_sd[Tshopo]0.01340.01230.0009780.0481true
2CFR_patch[Bas-Uele]0.5930.4990.3670.624true
2CFR_patch[Haut-Uele]0.5080.5040.3810.625true
2CFR_patch[Ituri]0.5610.4990.3690.62true
2CFR_patch[Nord-Kivu]0.5410.5010.3750.624true
2CFR_patch[Nord-Ubangi]0.5370.4960.3710.623true
2CFR_patch[Sud-Kivu]0.4860.5110.3840.635true
2CFR_patch[Sud-Ubangi]0.5650.4940.3650.619true
2CFR_patch[Tshopo]0.5340.4950.3650.622true
2C_T_patch[Bas-Uele]235.585254.041172.87387.453true
2C_T_patch[Haut-Uele]42.66742.28226.07970.967true
2C_T_patch[Ituri]2190.9542262.7871613.8713226.471true
2C_T_patch[Nord-Kivu]44.24251.64833.25481.344true
2C_T_patch[Nord-Ubangi]211.842173.179108.441281.429true
2C_T_patch[Sud-Kivu]127.22124.90577.754198.671true
2C_T_patch[Sud-Ubangi]48.86256.42533.14596.042true
2C_T_patch[Tshopo]10.2059.1144.6916.693true
2R_T_patch[Bas-Uele]1.7151.3411.0731.666false
2R_T_patch[Haut-Uele]0.9741.3921.0611.796false
2R_T_patch[Ituri]1.0271.2821.0631.601false
2R_T_patch[Nord-Kivu]0.7680.9610.6731.327true
2R_T_patch[Nord-Ubangi]2.2291.7341.3572.298true
2R_T_patch[Sud-Kivu]1.8441.3141.021.676false
2R_T_patch[Sud-Ubangi]1.6631.6161.2472.207true
2R_T_patch[Tshopo]0.8111.1310.7771.541true
2province_ascertainment[Bas-Uele]1.171.1160.9431.393true
2province_ascertainment[Haut-Uele]0.7030.8880.6051.119true
2province_ascertainment[Ituri]0.9740.9410.7621.123true
2province_ascertainment[Nord-Kivu]1.0540.9270.6751.16true
2province_ascertainment[Nord-Ubangi]1.3561.0980.8941.456true
2province_ascertainment[Sud-Kivu]0.810.8950.6721.116true
2province_ascertainment[Sud-Ubangi]1.1811.1130.8861.549true
2province_ascertainment[Tshopo]0.9121.0620.7851.566true
2province_suspected_ascertainment[Bas-Uele]0.6060.5420.3560.791true
2province_suspected_ascertainment[Haut-Uele]0.5660.5020.3040.774true
2province_suspected_ascertainment[Ituri]0.5370.5130.3570.735true
2province_suspected_ascertainment[Nord-Kivu]0.8190.5170.3240.769false
2province_suspected_ascertainment[Nord-Ubangi]0.6210.550.3560.805true
2province_suspected_ascertainment[Sud-Kivu]0.4360.4730.2870.722true
2province_suspected_ascertainment[Sud-Ubangi]0.6040.5560.3590.811true
2province_suspected_ascertainment[Tshopo]0.5310.5440.3380.808true
2region_drift_sd[Bas-Uele]0.1190.06250.03380.117false
2region_drift_sd[Haut-Uele]0.09350.06210.03080.112true
2region_drift_sd[Ituri]0.1150.06830.03880.111false
2region_drift_sd[Nord-Kivu]0.1290.08260.04790.135true
2region_drift_sd[Nord-Ubangi]0.1310.0720.03950.124false
2region_drift_sd[Sud-Kivu]0.1320.06290.03090.111false
2region_drift_sd[Sud-Ubangi]0.07530.06960.03750.121true
2region_drift_sd[Tshopo]0.1420.06510.03350.122false
3CFR_patch[Bas-Uele]0.2480.2640.1880.348true
3CFR_patch[Haut-Uele]0.2020.2640.1920.347true
3CFR_patch[Ituri]0.1690.2650.1920.344false
3CFR_patch[Nord-Kivu]0.2020.2670.1930.349true
3CFR_patch[Nord-Ubangi]0.1960.2650.1910.348true
3CFR_patch[Sud-Kivu]0.2120.2660.1940.352true
3CFR_patch[Sud-Ubangi]0.2170.2630.1860.34true
3CFR_patch[Tshopo]0.2050.2670.1920.349true
3C_T_patch[Bas-Uele]9.1811.9597.39719.79true
3C_T_patch[Haut-Uele]168.277179.476133.423260.212true
3C_T_patch[Ituri]29685.57125430.26321679.67735085.561true
3C_T_patch[Nord-Kivu]164.657186.445137.531266.96true
3C_T_patch[Nord-Ubangi]8.7258.6584.96814.981true
3C_T_patch[Sud-Kivu]157.284143.694105.111214.14true
3C_T_patch[Sud-Ubangi]11.62317.36411.06627.543true
3C_T_patch[Tshopo]12.64316.94410.5127.619true
3R_T_patch[Bas-Uele]1.1791.1490.9021.442true
3R_T_patch[Haut-Uele]1.3651.1790.9421.449true
3R_T_patch[Ituri]1.3881.1690.9531.41true
3R_T_patch[Nord-Kivu]1.3031.1380.8761.421true
3R_T_patch[Nord-Ubangi]1.2741.1570.91.442true
3R_T_patch[Sud-Kivu]1.2941.1740.9321.471true
3R_T_patch[Sud-Ubangi]1.1771.1410.91.44true
3R_T_patch[Tshopo]1.2921.1380.8691.414true
3province_ascertainment[Bas-Uele]1.2280.9970.761.227false
3province_ascertainment[Haut-Uele]1.0921.0580.9191.319true
3province_ascertainment[Ituri]0.8971.0160.861.245true
3province_ascertainment[Nord-Kivu]0.9731.0090.8541.235true
3province_ascertainment[Nord-Ubangi]0.7090.9860.7331.206false
3province_ascertainment[Sud-Kivu]0.7910.9690.7911.121false
3province_ascertainment[Sud-Ubangi]1.4721.0360.8581.38false
3province_ascertainment[Tshopo]1.0350.9550.671.133true
3province_suspected_ascertainment[Bas-Uele]0.680.7780.5310.928true
3province_suspected_ascertainment[Haut-Uele]0.6610.7870.5520.932true
3province_suspected_ascertainment[Ituri]0.7410.790.5660.93true
3province_suspected_ascertainment[Nord-Kivu]0.7840.7790.5470.922true
3province_suspected_ascertainment[Nord-Ubangi]0.750.7730.5210.923true
3province_suspected_ascertainment[Sud-Kivu]0.710.7730.5260.928true
3province_suspected_ascertainment[Sud-Ubangi]0.7150.7860.5590.932true
3province_suspected_ascertainment[Tshopo]0.6890.7640.4980.921true
3region_drift_sd[Bas-Uele]0.03350.02810.004780.0698true
3region_drift_sd[Haut-Uele]0.03520.0280.005140.0669true
3region_drift_sd[Ituri]0.02740.02650.004470.0659true
3region_drift_sd[Nord-Kivu]0.03540.02840.004580.0691true
3region_drift_sd[Nord-Ubangi]0.04340.02720.004420.0686true
3region_drift_sd[Sud-Kivu]0.0430.02950.00460.069true
3region_drift_sd[Sud-Ubangi]0.02970.02680.004570.0667true
3region_drift_sd[Tshopo]0.03060.02790.004640.0676true

Saving province in-sample outputs ​

Write the summary bullets
julia
# The bullets under the summary heading at the top of the page. They read
# tables built further down, so they are written here and read back when
# the site is assembled.
evaluation_insample_province_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    fitted = filter(r -> isfinite(r["Bias"]), province_calibration_table)
    worst = first(
        sort(fitted, "Bias"; by = abs, rev = true), min(3, size(fitted, 1))
    )[!, "Stream"]
    function coverage_bullet(heading, tbl, scale)
        low = first(sort(tbl, "90% coverage"), min(1, size(tbl, 1)))
        return string(
            "- **", heading, ":** ", count(>=(0.8), tbl[!, "90% coverage"]),
            " of ", size(tbl, 1), " province streams have 90% coverage of ",
            "at least 0.8", scale,
            size(low, 1) == 0 ? "." :
                string(
                    "; the lowest is ", low[1, "Stream"], " at ",
                    fmt(low[1, "90% coverage"]), "."
                )
        )
    end
    overall = [
        isempty(worst) ? nothing :
            string(
                "- **Least well reproduced:** ", join_names(worst),
                length(worst) == 1 ? " has" : " have",
                " the largest absolute bias in the province split."
            ),
        coverage_bullet("Coverage", fitted, ""),
        coverage_bullet(
            "Count-scale coverage",
            filter(
                r -> isfinite(r["Bias"]), province_count_calibration_table
            ),
            ""
        ),
    ]
    # One row per province and stream, the split and the count checks
    # side by side. A check that was not scored reads n/a.
    rows(tbl) = Dict(r["Stream"] => r for r in eachrow(tbl))
    cal = rows(province_calibration_table)
    count_cal = rows(province_count_calibration_table)
    cells = [
        (PROVINCE_LABELS[p], kind) for p in 1:N_PATCHES
            for kind in ("Confirmed cases", "Confirmed deaths")
    ]
    value(tbl, label, kind, col) = let r = get(
            tbl, string(kind, ", ", label), nothing
        )
        r === nothing ? missing : finite_round(r[col])
    end
    column(tbl, col) = Union{Missing, Float64}[
        value(tbl, label, kind, col) for (label, kind) in cells
    ]
    table = DataFrame(
        "Province" => first.(cells),
        "Stream" => last.(cells),
        "Bias (split)" => column(cal, "Bias"),
        "90% coverage (split)" => column(cal, "90% coverage"),
        "Bias (counts)" => column(count_cal, "Bias"),
        "90% coverage (counts)" => column(count_cal, "90% coverage"),
    )
    join(
        [
            join(filter(!isnothing, overall), "\n"),
            markdown_table(table; missing_text = "n/a"),
        ],
        "\n\n"
    )
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_insample_province.md"),
    evaluation_insample_province_summary
);