Skip to content

Province in-sample checks ​

This page checks whether the joint model reproduces the per-province data it was fitted to, follows each province's published estimates across releases, and refits it as a single population to check the spatial structure. The national checks are on the national in-sample checks page.

Load packages, data and fitted chains
julia
# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))
validation_forecast_from (generic function with 1 method)
julia
# The fits this page reads, loaded from the cache here.
chn_joint = load_fit("joint")
# `sens_no_patches` is the headline with `n_patches = 1`, the check on the
# spatial structure.
chn_no_patches = load_fit("sens_no_patches")
posterior_C_joint = vec(Array(chn_joint[:C_T]))
posterior_C_no_patches = vec(Array(chn_no_patches[:C_T]));

Summary ​

Whether each province's cases and deaths are reproduced, from the checks below. Bias and coverage are defined as on the national in-sample checks page, with coverage of the 90% predictive interval. The split columns score each province's share at the observed national total, and the count columns score its counts once the national total is predicted as well. A check that was not scored reads n/a.

  • Least well reproduced: Analysed specimens, Haut-Uele; Analysed specimens, Nord-Kivu and Analysed specimens, Nord-Ubangi have the largest absolute bias in the province split.

  • Coverage: 24 of 24 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Nord-Kivu at 0.92.

  • Count-scale coverage: 16 of 16 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Ituri at 0.92.

ProvinceStreamBias (split)90% coverage (split)Bias (counts)90% coverage (counts)
IturiConfirmed cases-0.010.950.050.97
IturiConfirmed deaths0.010.94-0.030.92
Nord-KivuConfirmed cases0.010.930.040.97
Nord-KivuConfirmed deaths-0.030.92-0.050.94
Haut-UeleConfirmed cases0.050.950.060.96
Haut-UeleConfirmed deaths0.010.96-0.010.97
Sud-KivuConfirmed cases0.010.990.010.99
Sud-KivuConfirmed deaths00.9900.99
TshopoConfirmed cases0.040.950.040.97
TshopoConfirmed deaths0.010.980.010.98
Bas-UeleConfirmed cases00.9900.99
Bas-UeleConfirmed deaths0.010.9700.98
Sud-UbangiConfirmed cases00.9900.99
Sud-UbangiConfirmed deaths00.9900.99
Nord-UbangiConfirmed cases0.010.990.010.98
Nord-UbangiConfirmed deaths0.010.990.010.99

Province prior predictive check ​

Whether the province model's prior, before any data are fitted, brackets each province's observed share of the confirmed cases and deaths.

Draw from the province model's prior
julia
prior_patch_chn = patch_prior_draws(obs);

prior_province_table = patch_overview_table(prior_patch_chn, N_PATCHES);
Show prior province summary table
ProvinceReproduction numberCumulative infectionsShare of infections (%)Relative ascertainment
Ituri0.74 (0.09–1.6)1294956 (2099–3907131)68.1 (14.1–98.9)1.01 (0.55–1.66)
Nord-Kivu1.12 (0.08–2.27)80285 (10–7435028)9.0 (0.1–36.8)1.01 (0.57–1.64)
Haut-Uele1.11 (0.07–2.28)30001 (4–2011760)3.3 (0.0–12.8)1.01 (0.58–1.79)
Sud-Kivu1.15 (0.08–2.31)36283 (6–6428616)4.7 (0.1–26.6)1.0 (0.62–1.72)
Tshopo1.15 (0.07–2.36)16737 (2–2521223)1.7 (0.0–11.4)1.01 (0.58–1.74)
Bas-Uele1.15 (0.08–2.35)6220 (1–1225929)0.8 (0.0–5.7)1.0 (0.59–1.68)
Sud-Ubangi1.22 (0.08–2.4)8436 (1–2689943)0.9 (0.0–10.2)1.01 (0.56–1.67)
Nord-Ubangi1.19 (0.08–2.4)5235 (1–1395426)0.6 (0.0–5.5)1.0 (0.56–1.74)
Province composition prior predictive checks
julia
prior_case_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province, prior"
);

prior_death_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province, prior"
);

Spatial hyperparameters against their prior ​

Whether the data move the province model's spatial hyperparameters away from their prior.

Spatial hyperparameter pair plot
julia
spatial_pair_fig = plot_pair(
    chn_joint, spatial_params;
    prior = prior_patch_chn, labels = spatial_labels
);

Joint posterior of the spatial hyperparameters, with the prior overlaid. The province reproduction numbers and case totals are derived from these and are in the [province estimates](@ref "Province estimates").

Province compositions ​

Whether each province's modelled share of the national confirmed cases and deaths reproduces the observed share at every spatial vintage (see the province compositions Methods section).

Province composition posterior predictive checks
julia
province_case_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province"
);

province_death_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province"
);

The four province terms for the isolation and laboratory data are checked the same way. A gap in a panel is a day on which that province printed nothing.

Province laboratory, occupancy, bed and admission split posterior predictive checks
julia
province_lab_bin_days = [
    maximum(province_lab.days[province_lab.bins .== b])
        for b in 1:maximum(province_lab.bins)
];

province_lab_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    days = province_lab_bin_days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Analysed specimen share by province, by week"
);

province_occupancy_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_occupancy_share, rows = province_isolation,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation occupancy share by province"
);

province_beds_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_capacity_share, rows = province_capacity,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation bed share by province"
);

province_admissions_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_admissions_share, rows = province_admissions,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Admission share by province"
);

The split of each calendar week's analysed specimens among the provinces, which identifies the background split.

The split of the patients in isolation among the provinces printed that day, on the weekly days the fit scores, against the per-patch bed demand.

The split of the beds among the provinces printed that day, on the days a count changed, against each patch's modelled share of the national capacity.

The split of each day's 24h admissions among the provinces printed that day, against each patch's modelled admissions.

Province stream calibration ​

Whether each province's share predictions are calibrated at the observed national total, scored as in the national stream calibration. The analysed-specimen rows score the weekly laboratory split the same way, at each week's observed national analysed total.

Build the province calibration panels
julia
province_case_panels = province_composition_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    stream = "Confirmed cases", n_patches = N_PATCHES
);
province_death_panels = province_composition_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_lab_panels = province_composition_panels(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    stream = "Analysed specimens", n_patches = N_PATCHES
);
province_panels = vcat(
    province_case_panels, province_death_panels, province_lab_panels
);
province_calibration_table = stream_calibration(province_panels);
province_calibration_fig = plot_stream_calibration(
    province_calibration_table
);

Province calibration table
StreamVintagesBias50% coverage90% coverage
Confirmed cases, Ituri147-0.010.630.95
Confirmed cases, Nord-Kivu1470.010.540.93
Confirmed cases, Haut-Uele1470.050.730.95
Confirmed cases, Sud-Kivu1470.010.990.99
Confirmed cases, Tshopo1470.040.860.95
Confirmed cases, Bas-Uele14700.930.99
Confirmed cases, Sud-Ubangi14700.990.99
Confirmed cases, Nord-Ubangi1470.010.960.99
Confirmed deaths, Ituri1450.010.630.94
Confirmed deaths, Nord-Kivu145-0.030.610.92
Confirmed deaths, Haut-Uele1450.010.780.96
Confirmed deaths, Sud-Kivu14500.990.99
Confirmed deaths, Tshopo1450.010.890.98
Confirmed deaths, Bas-Uele1450.010.970.97
Confirmed deaths, Sud-Ubangi14500.990.99
Confirmed deaths, Nord-Ubangi1450.010.990.99
Analysed specimens, Ituri230.060.961
Analysed specimens, Nord-Kivu23-0.160.911
Analysed specimens, Haut-Uele23-0.170.521
Analysed specimens, Sud-Kivu230.050.740.96
Analysed specimens, Tshopo230.010.871
Analysed specimens, Bas-Uele230.040.651
Analysed specimens, Sud-Ubangi230.020.911
Analysed specimens, Nord-Ubangi230.080.910.96

Province counts ​

Whether each province's cases and deaths are reproduced as counts once the national total is predicted as well.

Build the province count panels
julia
# Each province's confirmed cases and deaths split from the national
# posterior predictive by the fitted shares, shared with the dashboard's
# province series (see `joint_count_replicates` in `docs/pages/_setup.jl`).
province_count_replicates = joint_count_replicates();
province_case_count_panels = province_count_replicates.cases.panels;
province_death_count_panels = province_count_replicates.deaths.panels;
province_count_calibration_table = stream_calibration(
    vcat(province_case_count_panels, province_death_count_panels)
);
province_count_calibration_fig = plot_stream_calibration(
    province_count_calibration_table
);
Province count calibration plot

Province count calibration table
StreamVintagesBias50% coverage90% coverage
Confirmed cases, Ituri1470.050.730.97
Confirmed cases, Nord-Kivu1470.040.690.97
Confirmed cases, Haut-Uele1470.060.720.96
Confirmed cases, Sud-Kivu1470.010.990.99
Confirmed cases, Tshopo1470.040.860.97
Confirmed cases, Bas-Uele14700.930.99
Confirmed cases, Sud-Ubangi14700.990.99
Confirmed cases, Nord-Ubangi1470.010.960.98
Confirmed deaths, Ituri145-0.030.620.92
Confirmed deaths, Nord-Kivu145-0.050.590.94
Confirmed deaths, Haut-Uele145-0.010.760.97
Confirmed deaths, Sud-Kivu14500.990.99
Confirmed deaths, Tshopo1450.010.90.98
Confirmed deaths, Bas-Uele14500.970.98
Confirmed deaths, Sud-Ubangi14500.990.99
Confirmed deaths, Nord-Ubangi1450.010.990.99

Province correlations and totals ​

Which province quantities trade off against each other, and whether each province's case and death totals agree with the observed ones.

Province posterior correlation heatmap
julia
# Per-province entries of a vector deterministic, one draw vector each,
# keyed `<name>_<patch>` so they can sit side by side in one table.
function _patch_columns(chn, key::Symbol, name::AbstractString; f = identity)
    draws = [collect(v) for v in vec(collect(chn[key]))]
    return [
        Symbol(name, "_", p) => Float64[f(d[p]) for d in draws]
            for p in 1:N_PATCHES
    ]
end

# LaTeX label for one province's entry of a quantity. Spaces in a province
# name need escaping inside `\mathrm`.
_province_tex(sym, p) = string(
    sym, raw"\ (\mathrm{", replace(PROVINCE_LABELS[p], " " => raw"\ "), "})"
)
province_correlation_draws = NamedTuple(
    vcat(
        [:C_T => vec(Array(chn_joint[:C_T]))],
        _patch_columns(chn_joint, :R_T_patch, "R_T"),
        _patch_columns(chn_joint, :province_ascertainment, "asc"),
        _patch_columns(chn_joint, :CFR_patch, "CFR")
    )
);
province_correlation_labels = merge(
    Dict(:C_T => raw"C_T"),
    Dict(
        Symbol(name, "_", p) => _province_tex(tex, p)
            for p in 1:N_PATCHES
            for (name, tex) in (
                ("R_T", raw"R_T"), ("asc", raw"p_\mathrm{asc}"),
                ("CFR", raw"\mathrm{CFR}"),
            )
    )
);
province_correlation_fig = plot_correlation_heatmap(
    province_correlation_draws; labels = province_correlation_labels
);

Province totals against observed
julia
# Keyed by panel title, which names the stream and the province. A total
# that is the same in every draw (a patch with no deaths at any vintage) has
# no spread to plot and gives the corner plot a zero-width axis, so it is
# left out.
_varying_panels = filter(
    p -> length(unique(sum(r) for r in p.replicates)) > 1, province_panels
);
province_totals = NamedTuple(
    Symbol(p.title) => [sum(Float64.(r)) for r in p.replicates]
        for p in _varying_panels
);
province_observed = NamedTuple(
    Symbol(p.title) => Float64(sum(p.observed)) for p in _varying_panels
);
province_pairs_fig = plot_stream_pairs(province_totals, province_observed);

Estimate evolution across releases ​

How each province's estimates have moved as releases accrued, read from the estimates each release published at its own cut-off. Each panel starts at the first release that published per-province estimates.

Load the per-province estimates of each release
julia
# Written by `scripts/score_releases.jl` from each release's
# `province_estimates.csv`, one row per release, province and quantity. A
# missing file reads as an empty table, which the figures report as nothing
# collected.
province_release_df = _release_data(
    joinpath("province", "released_estimates.csv"),
    (;
        release = String, date = Date, area = String, patch = String,
        pooled = Int, quantity = String, median = Float64, lo90 = Float64,
        hi90 = Float64,
    )
)
# One panel per province, the patches first in their usual order. A
# province pooled into a shared patch is marked, since it carries its
# patch's reproduction number and case-fatality ratio.
_release_provinces = unique(vcat(PROVINCE_NAMES, province_release_df.area))
_release_pooled = Set(
    r.area for r in eachrow(province_release_df)
        if coalesce(r.pooled, 0) == 1
)
_release_titles = Dict(
    a => string(
        get(Dict(zip(PROVINCE_NAMES, PROVINCE_LABELS)), a, a),
        a in _release_pooled ? " (pooled)" : ""
    ) for a in _release_provinces
)
_province_release_groups(q) = area_evolution_groups(
    province_release_df, q;
    areas = _release_provinces, labels = _release_titles
)
_no_release_note = "No releases scored yet. This fills in as releases accumulate."
province_rt_release_fig = plot_evolution_by_group(
    _province_release_groups("R_T");
    ylabel = "Reproduction number",
    title = "Province reproduction number by release",
    released_label = "Released estimate", marks_label = "median, 90% bar",
    refline = 1.0, ymax = 3.0, ncols = 4,
    empty_note = _no_release_note
);
province_cases_release_fig = plot_evolution_by_group(
    _province_release_groups("cases");
    ylabel = "Confirmed cases",
    title = "Province confirmed cases by release",
    released_label = "Reported by the release cut-off", marks_label = "",
    shared_yrange = false, ncols = 4, empty_note = _no_release_note
);
province_cfr_release_fig = plot_evolution_by_group(
    _province_release_groups("CFR");
    ylabel = "Case-fatality ratio",
    title = "Province case-fatality ratio by release",
    released_label = "Released estimate", marks_label = "median, 90% bar",
    ymax = 1.0, ncols = 4,
    empty_note = _no_release_note
);

Reproduction number by release ​

Each province's reproduction number at the cut-off of each release, as a median with its 90% interval bar. The axis is fixed at three, and an interval past it is clamped and marked with an open triangle.

Confirmed cases by release ​

The confirmed cases each province had reported by the cut-off of each release, each panel on its own scale.

Case-fatality ratio by release ​

Each province's case-fatality ratio at each release, as a median with its 90% interval bar.

Province sensitivity to assumptions ​

Spatial structure sensitivity ​

We refit the joint model with its eight patches reduced to one well-mixed population. Splitting the country into provinces adds no national data, so the national outbreak size should not move far either way. The two still differ, because the country grows at the force-weighted mean of the provinces' reproduction numbers, which sits above the central trend they pool toward. That gap is first order in the deviation scale, so a large gap between the two posteriors would point at the deviation priors.

julia
spatial_sensitivity_table = streams_table(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population (n_patches = 1)" => posterior_C_no_patches
);
spatial_sensitivity_table
2×7 DataFrame
RowStreamLower 90%Lower 60%Lower 30%Upper 30%Upper 60%Upper 90%
StringFloat64Float64Float64Float64Float64Float64
1Meta-population (headline)16685.019577.021896.025894.028423.034217.0
2Single population (n_patches = 1)16880.020006.022175.026679.029850.036188.0
Spatial-structure density overlay
julia
spatial_sensitivity_fig = plot_density_overlay(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population" => posterior_C_no_patches;
    xlabel = "Cumulative infections"
);

Posterior densities of the cumulative infections under the headline province model and the single-population fit.

The spatial structure could also move other national quantities that both fits estimate, so their posteriors should sit on top of each other.

National quantities under both structures
julia
spatial_rt_fig = plot_rt_streams(
    [
        (;
            label = "Single population (n_patches = 1)",
            chn = chn_no_patches, rt_start = _rt_start_plot,
            rt_walk_start = clamp(
                _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
                obs.n
            ), colour = :steelblue,
        ),
    ];
    joint = (;
        chn = chn_joint, rt_start = _rt_start_plot,
        rt_walk_start = clamp(
            _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
            obs.n
        ),
    ),
    n = obs.n, breakpoint = _BREAKPOINT,
    as_of_date = string(obs.cutoff), seeding = obs.seeding,
    display_start = _rt_start_plot, ncols = 1,
    title = "National reproduction number under both structures",
    reference_label = "the meta-population headline",
    panel_label = "the single-population fit"
);

spatial_cfr_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:CFR])),
    "Single population" => vec(Array(chn_no_patches[:CFR]));
    xlabel = "Case-fatality ratio"
);

spatial_rt_density_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:R_T])),
    "Single population" => vec(Array(chn_no_patches[:R_T]));
    xlabel = "Reproduction number at the cut-off"
);

The national reproduction number from the single-population fit (coloured) against the central trend of the headline province model (grey), which is hidden where the two agree.

Posterior densities of the case-fatality ratio under both structures.

Posterior densities of the reproduction number at the cut-off under both structures.
National quantities under both structures, as a table
julia
# One row per national quantity, one column per structure, each cell a
# median with a 90% credible interval. Built here rather than by stacking
# two `summary_table` calls, so the two structures sit side by side and the
# reader compares along a row.
spatial_quantities_table = let
    # A count rounded to zero decimals still prints a trailing ".0", so
    # whole-number quantities go through `Int`.
    fmt(x, d) = d <= 0 ? string(round(Int, x)) : string(round(x; digits = d))
    cell(v, d) = string(
        fmt(quantile(v, 0.5), d), " (",
        fmt(quantile(v, 0.05), d), "–", fmt(quantile(v, 0.95), d), ")"
    )
    rows = [
        ("Cumulative infections", :C_T, 0),
        ("Reproduction number at the cut-off", :R_T, 2),
        ("Case-fatality ratio", :CFR, 2),
        ("Outbreak age (days)", :T, 0),
        ("Latest growth rate (per day)", :r, 3),
    ]
    DataFrame(
        "Quantity" => [r[1] for r in rows],
        "Meta-population (headline)" => [
            cell(
                vec(Array(chn_joint[r[2]])), r[3]
            )
                for r in rows
        ],
        "Single population" => [
            cell(vec(Array(chn_no_patches[r[2]])), r[3])
                for r in rows
        ]
    )
end;
5×3 DataFrame
RowQuantityMeta-population (headline)Single population
StringStringString
1Cumulative infections23750 (16685–34217)24304 (16880–36188)
2Reproduction number at the cut-off1.0 (0.73–1.38)0.96 (0.69–1.32)
3Case-fatality ratio0.52 (0.41–0.63)0.52 (0.4–0.63)
4Outbreak age (days)243 (234–258)244 (234–260)
5Latest growth rate (per day)-0.0 (-0.022–0.025)-0.003 (-0.024–0.02)
The cumulative infections, reproduction number at the cut-off, case-fatality ratio, outbreak age and latest growth rate under both structures, as credible intervals. The agreement is read from this table.

Province parameter recovery ​

Whether the model recovers each province's values when fitted to data it simulated itself, from the same runs as the national parameter recovery.

Recovery figure
julia
province_recovery_results = recovery_results()
_recovery_short = [
    "C_T_patch" => "C_T", "R_T_patch" => "R_T", "CFR_patch" => "CFR",
    "province_ascertainment" => "ascertainment",
    "region_drift_sd" => "drift SD",
]
province_recovery_quantities = [
    "$(k)[$(p)]" for (k, _) in _recovery_short for p in PROVINCE_LABELS
]
province_recovery_fig = isempty(province_recovery_results.params) ? nothing :
    plot_recovery(
        province_recovery_results.params, province_recovery_results.draws,
        province_recovery_results.prior;
        quantities = province_recovery_quantities,
        labels = Dict(
            "$(k)[$(p)]" => "$(v), $(p)" for (k, v) in _recovery_short
            for p in PROVINCE_LABELS
        ),
        panel_labels = Dict(
            "$(k)[$(p)]" => p for (k, _) in _recovery_short for p in PROVINCE_LABELS
        ),
        row_labels = last.(_recovery_short),
        log_x = ["C_T_patch[$(p)]" for p in PROVINCE_LABELS]
    );

Each seed's posterior median with its 50% and 90% intervals divided by its true value (top). Below, each quantity by province is on its own scale, with the prior in grey, each seed's posterior in its colour and its true value dashed.
Summary table across seeds
julia
province_recovery_summary = isempty(province_recovery_results.params) ?
    DataFrame() :
    recovery_summary_table(province_recovery_results.params; province = true);

province_recovery_summary_display = isempty(province_recovery_summary) ?
    Markdown.parse("No parameter-recovery run is available for this build.") :
    MarkdownTable(province_recovery_summary);
QuantityTruth, lowestTruth, highestRelative error, medianRelative error, rangez, medianTruth in 90%Outside 99%
CFR_patch[Bas-Uele]0.1470.4120.02-0.07 to 0.510.132/30
CFR_patch[Haut-Uele]0.1350.376-0.01-0.03 to 0.6102/30
CFR_patch[Ituri]0.1690.385-0.01-0.09 to 0.27-0.023/30
CFR_patch[Nord-Kivu]0.1750.380.08-0.06 to 0.310.463/30
CFR_patch[Nord-Ubangi]0.1340.3720.040.03 to 0.690.282/30
CFR_patch[Sud-Kivu]0.1680.3670.040.01 to 0.370.223/30
CFR_patch[Sud-Ubangi]0.1920.423-0.02-0.14 to 0.1-0.13/30
CFR_patch[Tshopo]0.1580.4220.07-0.14 to 0.390.393/30
C_T_patch[Bas-Uele]2.55283-0.17-0.58 to -0.09-0.723/30
C_T_patch[Haut-Uele]22.3607-0.16-0.58 to -0.11-0.472/31
C_T_patch[Ituri]186034600-0.21-0.47 to -0.02-1.082/31
C_T_patch[Nord-Kivu]1432230-0.19-0.36 to -0.05-0.682/30
C_T_patch[Nord-Ubangi]1.58102-0.05-0.52 to 1.640.031/30
C_T_patch[Sud-Kivu]43.8986-0.23-0.37 to -0.08-0.883/30
C_T_patch[Sud-Ubangi]3.416.330.23-0.34 to 0.320.553/30
C_T_patch[Tshopo]4.99145-0.06-0.19 to 0.07-0.143/30
R_T_patch[Bas-Uele]0.6210.946-0.05-0.57 to 0.25-0.483/30
R_T_patch[Haut-Uele]0.6370.8090.09-0.52 to 0.210.653/30
R_T_patch[Ituri]0.6340.933-0.02-0.54 to 0.23-0.143/30
R_T_patch[Nord-Kivu]0.6360.979-0.09-0.54 to 0.23-1.053/30
R_T_patch[Nord-Ubangi]0.6670.780.18-0.52 to 0.310.582/30
R_T_patch[Sud-Kivu]0.6531-0.12-0.54 to 0.19-1.323/30
R_T_patch[Sud-Ubangi]0.6070.8410.07-0.55 to 0.270.63/30
R_T_patch[Tshopo]0.6070.9110.12-0.59 to 0.270.793/30
province_ascertainment[Bas-Uele]0.6210.740.18-0.02 to 0.180.463/30
province_ascertainment[Haut-Uele]0.61.320.61-0.05 to 0.892.031/31
province_ascertainment[Ituri]0.9141.250.02-0.02 to 0.060.213/30
province_ascertainment[Nord-Kivu]0.9541.18-0.06-0.1 to 0.09-0.233/30
province_ascertainment[Nord-Ubangi]0.9391.26-0.13-0.23 to 0.09-0.723/30
province_ascertainment[Sud-Kivu]0.5581.020-0.16 to 0.350.172/30
province_ascertainment[Sud-Ubangi]1.12.68-0.23-0.35 to -0.15-0.983/30
province_ascertainment[Tshopo]0.8451.16-0.09-0.2 to 0.15-0.513/30
province_suspected_ascertainment[Bas-Uele]0.4880.5310.5-0.1 to 0.561.463/30
province_suspected_ascertainment[Haut-Uele]0.410.6350.570.09 to 0.912.31/30
province_suspected_ascertainment[Ituri]0.540.8630.19-0.03 to 0.560.852/30
province_suspected_ascertainment[Nord-Kivu]0.5330.8250.16-0.02 to 0.560.642/30
province_suspected_ascertainment[Nord-Ubangi]0.5360.889-0.12-0.14 to 0.54-0.473/30
province_suspected_ascertainment[Sud-Kivu]0.5410.7230.080.05 to 0.50.433/30
province_suspected_ascertainment[Sud-Ubangi]0.5430.795-0.02-0.18 to 0.56-0.272/30
province_suspected_ascertainment[Tshopo]0.5330.7820.02-0.0 to 0.560.013/30
region_drift_sd[Bas-Uele]0.01620.052-0.02-0.63 to 0.640.143/30
region_drift_sd[Haut-Uele]0.01950.04110.01-0.56 to 0.450.183/30
region_drift_sd[Ituri]0.02220.0338-0.15-0.38 to 0.13-0.273/30
region_drift_sd[Nord-Kivu]0.0180.02770.04-0.38 to 0.460.213/30
region_drift_sd[Nord-Ubangi]0.0210.0494-0.31-0.65 to 0.25-0.623/30
region_drift_sd[Sud-Kivu]0.01770.0516-0.18-0.64 to 0.46-0.243/30
region_drift_sd[Sud-Ubangi]0.01710.0427-0.29-0.57 to 0.55-0.533/30
region_drift_sd[Tshopo]0.02080.0411-0.22-0.51 to 0.23-0.383/30
The error of each seed's posterior median relative to the truth, and the z-score of the truth, summarised across seeds.
Each seed's recovered province values
julia
province_recovery = let r = province_recovery_results.params
    isempty(r) ? DataFrame() :
        r[
            occursin.("[", r.quantity), [
                :seed, :quantity, :truth, :median, :lower_90, :upper_90,
                :covered_90,
            ],
        ]
end;

province_recovery_display = isempty(province_recovery) ?
    Markdown.parse("No parameter-recovery run is available for this build.") : MarkdownTable(province_recovery);
seedquantitytruthmedianlower_90upper_90covered_90
1CFR_patch[Bas-Uele]0.1470.2220.1490.313false
1CFR_patch[Haut-Uele]0.1350.2170.1440.312false
1CFR_patch[Ituri]0.1690.2150.1440.307true
1CFR_patch[Nord-Kivu]0.1750.230.1540.323true
1CFR_patch[Nord-Ubangi]0.1340.2260.1540.318false
1CFR_patch[Sud-Kivu]0.1680.230.1530.338true
1CFR_patch[Sud-Ubangi]0.1920.2120.1380.306true
1CFR_patch[Tshopo]0.1580.2210.1450.317true
1C_T_patch[Bas-Uele]2.5491.0640.2962.763true
1C_T_patch[Haut-Uele]180.07275.36149.309120.056false
1C_T_patch[Ituri]1863.085986.185752.661407.137false
1C_T_patch[Nord-Kivu]402.815258.324184.2399.23false
1C_T_patch[Nord-Ubangi]101.83648.50730.63680.791false
1C_T_patch[Sud-Kivu]43.76527.7116.32947.079true
1C_T_patch[Sud-Ubangi]3.414.2011.9478.303true
1C_T_patch[Tshopo]4.9855.3452.66610.309true
1R_T_patch[Bas-Uele]0.6210.7760.4271.161true
1R_T_patch[Haut-Uele]0.6370.7710.4271.135true
1R_T_patch[Ituri]0.6340.7770.4251.148true
1R_T_patch[Nord-Kivu]0.6360.7840.4391.156true
1R_T_patch[Nord-Ubangi]0.6670.7890.4391.166true
1R_T_patch[Sud-Kivu]0.6530.7790.4341.145true
1R_T_patch[Sud-Ubangi]0.6070.7710.4221.145true
1R_T_patch[Tshopo]0.6070.770.4211.142true
1province_ascertainment[Bas-Uele]0.6210.7330.3061.416true
1province_ascertainment[Haut-Uele]1.0962.0731.4642.887false
1province_ascertainment[Ituri]0.9140.9690.731.285true
1province_ascertainment[Nord-Kivu]0.9860.8840.6681.182true
1province_ascertainment[Nord-Ubangi]0.9391.0280.711.459true
1province_ascertainment[Sud-Kivu]0.5580.470.2870.747true
1province_ascertainment[Sud-Ubangi]2.6751.7391.0143.653true
1province_ascertainment[Tshopo]1.1630.9320.5341.614true
1province_suspected_ascertainment[Bas-Uele]0.5310.8270.460.966true
1province_suspected_ascertainment[Haut-Uele]0.5390.8470.5820.969false
1province_suspected_ascertainment[Ituri]0.540.8430.6030.966false
1province_suspected_ascertainment[Nord-Kivu]0.5340.8310.5450.966false
1province_suspected_ascertainment[Nord-Ubangi]0.5360.8270.5210.964true
1province_suspected_ascertainment[Sud-Kivu]0.5410.8090.3240.963true
1province_suspected_ascertainment[Sud-Ubangi]0.5430.8440.5910.969false
1province_suspected_ascertainment[Tshopo]0.5330.8320.5070.965true
1region_drift_sd[Bas-Uele]0.01620.02650.003990.072true
1region_drift_sd[Haut-Uele]0.01950.02830.00440.0712true
1region_drift_sd[Ituri]0.02220.02510.004260.066true
1region_drift_sd[Nord-Kivu]0.0180.02630.003780.068true
1region_drift_sd[Nord-Ubangi]0.0210.02630.004040.0687true
1region_drift_sd[Sud-Kivu]0.01770.02590.004020.0676true
1region_drift_sd[Sud-Ubangi]0.01710.02650.004060.0667true
1region_drift_sd[Tshopo]0.02080.02550.004050.0692true
2CFR_patch[Bas-Uele]0.4120.3820.2840.498true
2CFR_patch[Haut-Uele]0.3760.3640.2670.465true
2CFR_patch[Ituri]0.3850.350.2580.443true
2CFR_patch[Nord-Kivu]0.380.3550.2620.449true
2CFR_patch[Nord-Ubangi]0.3540.3640.2710.466true
2CFR_patch[Sud-Kivu]0.3670.3710.2790.468true
2CFR_patch[Sud-Ubangi]0.4230.3650.2710.463true
2CFR_patch[Tshopo]0.4220.3620.2710.455true
2C_T_patch[Bas-Uele]283.118235.441166.039347.969true
2C_T_patch[Haut-Uele]22.3218.85611.9730.365true
2C_T_patch[Ituri]2044.952005.1081665.0012662.539true
2C_T_patch[Nord-Kivu]142.673135.902103.614187.773true
2C_T_patch[Nord-Ubangi]1.5824.1791.9388.128false
2C_T_patch[Sud-Kivu]985.578907.283726.0021257.18true
2C_T_patch[Sud-Ubangi]6.3344.1542.0247.951true
2C_T_patch[Tshopo]144.798136.816103.439193.187true
2R_T_patch[Bas-Uele]0.9040.3890.1420.984true
2R_T_patch[Haut-Uele]0.7790.3770.1360.924true
2R_T_patch[Ituri]0.8130.3750.1340.928true
2R_T_patch[Nord-Kivu]0.8340.380.1360.938true
2R_T_patch[Nord-Ubangi]0.780.3740.1330.951true
2R_T_patch[Sud-Kivu]0.830.380.1370.954true
2R_T_patch[Sud-Ubangi]0.8410.3750.1360.937true
2R_T_patch[Tshopo]0.9110.3760.1370.946true
2province_ascertainment[Bas-Uele]0.7010.8250.641.03true
2province_ascertainment[Haut-Uele]0.60.9640.6721.259false
2province_ascertainment[Ituri]1.2321.2131.0061.545true
2province_ascertainment[Nord-Kivu]1.1831.1130.9271.455true
2province_ascertainment[Nord-Ubangi]1.2640.9770.6421.289true
2province_ascertainment[Sud-Kivu]1.0221.0270.8751.268true
2province_ascertainment[Sud-Ubangi]1.1420.9670.6191.257true
2province_ascertainment[Tshopo]1.1071.0050.8141.26true
2province_suspected_ascertainment[Bas-Uele]0.4920.7370.4350.942true
2province_suspected_ascertainment[Haut-Uele]0.410.7820.4820.95false
2province_suspected_ascertainment[Ituri]0.8630.8390.6320.958true
2province_suspected_ascertainment[Nord-Kivu]0.8250.8090.5960.955true
2province_suspected_ascertainment[Nord-Ubangi]0.8890.7830.4840.947true
2province_suspected_ascertainment[Sud-Kivu]0.7230.7840.5630.948true
2province_suspected_ascertainment[Sud-Ubangi]0.7950.7810.4460.95true
2province_suspected_ascertainment[Tshopo]0.7820.8010.5550.952true
2region_drift_sd[Bas-Uele]0.0520.01910.001640.0644true
2region_drift_sd[Haut-Uele]0.04110.01790.00150.0614true
2region_drift_sd[Ituri]0.03210.01980.00170.0616true
2region_drift_sd[Nord-Kivu]0.02770.0170.001470.0572true
2region_drift_sd[Nord-Ubangi]0.04940.01730.001440.0604true
2region_drift_sd[Sud-Kivu]0.05160.01850.001390.059true
2region_drift_sd[Sud-Ubangi]0.04090.01780.001540.0605true
2region_drift_sd[Tshopo]0.03640.0180.001430.0609true
3CFR_patch[Bas-Uele]0.3930.4010.2890.537true
3CFR_patch[Haut-Uele]0.370.3670.2640.485true
3CFR_patch[Ituri]0.3660.3630.2540.479true
3CFR_patch[Nord-Kivu]0.3570.3870.2840.501true
3CFR_patch[Nord-Ubangi]0.3720.3890.2850.512true
3CFR_patch[Sud-Kivu]0.3590.3740.2640.489true
3CFR_patch[Sud-Ubangi]0.3920.3840.2760.501true
3CFR_patch[Tshopo]0.3570.3830.2780.496true
3C_T_patch[Bas-Uele]76.8569.92539.551121.385true
3C_T_patch[Haut-Uele]607.293540.298399.969834.979true
3C_T_patch[Ituri]34557.93927168.68320642.49939189.037true
3C_T_patch[Nord-Kivu]2229.9671795.4141282.6712829.063true
3C_T_patch[Nord-Ubangi]36.21634.54520.92358.693true
3C_T_patch[Sud-Kivu]140.317107.51775.356168.319true
3C_T_patch[Sud-Ubangi]4.2475.6232.75811.484true
3C_T_patch[Tshopo]137.809111.57774.156180.522true
3R_T_patch[Bas-Uele]0.9460.90.7781.05true
3R_T_patch[Haut-Uele]0.8090.8860.7531.02true
3R_T_patch[Ituri]0.9330.9180.8221.039true
3R_T_patch[Nord-Kivu]0.9790.8920.7881.032true
3R_T_patch[Nord-Ubangi]0.6760.8850.751.041false
3R_T_patch[Sud-Kivu]1.00.8820.7411.024true
3R_T_patch[Sud-Ubangi]0.8140.8730.7221.022true
3R_T_patch[Tshopo]0.7650.8550.7021.006true
3province_ascertainment[Bas-Uele]0.740.7270.51.012true
3province_ascertainment[Haut-Uele]1.3181.2560.9981.645true
3province_ascertainment[Ituri]1.2541.2751.0111.658true
3province_ascertainment[Nord-Kivu]0.9541.0410.8761.304true
3province_ascertainment[Nord-Ubangi]1.0380.9050.6291.213true
3province_ascertainment[Sud-Kivu]0.8921.2030.9381.624false
3province_ascertainment[Sud-Ubangi]1.0960.8410.4081.172true
3province_ascertainment[Tshopo]0.8450.9760.7511.256true
3province_suspected_ascertainment[Bas-Uele]0.4880.4370.2180.824true
3province_suspected_ascertainment[Haut-Uele]0.6350.6940.4480.907true
3province_suspected_ascertainment[Ituri]0.5720.6830.4750.893true
3province_suspected_ascertainment[Nord-Kivu]0.5330.6190.3890.878true
3province_suspected_ascertainment[Nord-Ubangi]0.6370.5470.2810.861true
3province_suspected_ascertainment[Sud-Kivu]0.6290.6610.4080.896true
3province_suspected_ascertainment[Sud-Ubangi]0.6420.5230.190.852true
3province_suspected_ascertainment[Tshopo]0.6030.6010.340.88true
3region_drift_sd[Bas-Uele]0.02820.02770.009130.0606true
3region_drift_sd[Haut-Uele]0.03190.03240.01110.0682true
3region_drift_sd[Ituri]0.03380.02860.0110.054true
3region_drift_sd[Nord-Kivu]0.02570.02680.00910.0566true
3region_drift_sd[Nord-Ubangi]0.0410.02830.009810.0618true
3region_drift_sd[Sud-Kivu]0.0350.02880.00940.0599true
3region_drift_sd[Sud-Ubangi]0.04270.03040.01040.066true
3region_drift_sd[Tshopo]0.04110.03190.01080.0679true

Saving province in-sample outputs ​

Write the summary bullets
julia
# The bullets under the summary heading at the top of the page. They read
# tables built further down, so they are written here and read back when
# the site is assembled.
evaluation_insample_province_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    fitted = filter(r -> isfinite(r["Bias"]), province_calibration_table)
    worst = first(
        sort(fitted, "Bias"; by = abs, rev = true), min(3, size(fitted, 1))
    )[!, "Stream"]
    function coverage_bullet(heading, tbl, scale)
        low = first(sort(tbl, "90% coverage"), min(1, size(tbl, 1)))
        return string(
            "- **", heading, ":** ", count(>=(0.8), tbl[!, "90% coverage"]),
            " of ", size(tbl, 1), " province streams have 90% coverage of ",
            "at least 0.8", scale,
            size(low, 1) == 0 ? "." :
                string(
                    "; the lowest is ", low[1, "Stream"], " at ",
                    fmt(low[1, "90% coverage"]), "."
                )
        )
    end
    overall = [
        isempty(worst) ? nothing :
            string(
                "- **Least well reproduced:** ", join_names(worst),
                length(worst) == 1 ? " has" : " have",
                " the largest absolute bias in the province split."
            ),
        coverage_bullet("Coverage", fitted, ""),
        coverage_bullet(
            "Count-scale coverage",
            filter(
                r -> isfinite(r["Bias"]), province_count_calibration_table
            ),
            ""
        ),
    ]
    # One row per province and stream, the split and the count checks
    # side by side. A check that was not scored reads n/a.
    rows(tbl) = Dict(r["Stream"] => r for r in eachrow(tbl))
    cal = rows(province_calibration_table)
    count_cal = rows(province_count_calibration_table)
    cells = [
        (PROVINCE_LABELS[p], kind) for p in 1:N_PATCHES
            for kind in ("Confirmed cases", "Confirmed deaths")
    ]
    value(tbl, label, kind, col) = let r = get(
            tbl, string(kind, ", ", label), nothing
        )
        r === nothing ? missing : finite_round(r[col])
    end
    column(tbl, col) = Union{Missing, Float64}[
        value(tbl, label, kind, col) for (label, kind) in cells
    ]
    table = DataFrame(
        "Province" => first.(cells),
        "Stream" => last.(cells),
        "Bias (split)" => column(cal, "Bias"),
        "90% coverage (split)" => column(cal, "90% coverage"),
        "Bias (counts)" => column(count_cal, "Bias"),
        "90% coverage (counts)" => column(count_cal, "90% coverage"),
    )
    join(
        [
            join(filter(!isnothing, overall), "\n"),
            markdown_table(table; missing_text = "n/a"),
        ],
        "\n\n"
    )
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_insample_province.md"),
    evaluation_insample_province_summary
);