BVDOutbreakSize

Appearance

Province in-sample checks ​

This page checks whether the joint model reproduces the per-province data it was fitted to, and refits it as a single population to check the spatial structure. The national checks are on the national in-sample checks page.

Load packages, data and fitted chains
julia
# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))
validation_forecast_from (generic function with 1 method)
julia
# The fits this page reads, loaded from the cache here.
chn_joint = load_fit("joint")
# `sens_no_patches` is the headline with `n_patches = 1`, the check on the
# spatial structure.
chn_no_patches = load_fit("sens_no_patches")
posterior_C_joint = vec(Array(chn_joint[:C_T]))
posterior_C_no_patches = vec(Array(chn_no_patches[:C_T]));

Summary ​

Whether each province's cases and deaths are reproduced, from the checks below. Bias and coverage are defined as on the national in-sample checks page, with coverage of the 90% predictive interval.

  • Least well reproduced: Analysed specimens, Sud-Kivu (bias 0.31, 90% coverage 1.0); Analysed specimens, Nord-Kivu (bias -0.27, 90% coverage 1.0); Analysed specimens, Sud-Ubangi (bias 0.11, 90% coverage 1.0).

  • Coverage: 24 of 24 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Nord-Kivu at 0.91.

  • Count-scale coverage: 16 of 16 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Ituri at 0.93.

Ituri

  • Confirmed cases: bias -0.02 and 90% coverage 0.95 over 121 vintages.

  • Confirmed deaths: bias 0.02 and 90% coverage 0.93 over 121 vintages.

  • Confirmed cases as counts: bias 0.05 and 90% coverage 0.96 over 121 vintages.

  • Confirmed deaths as counts: bias -0.04 and 90% coverage 0.93 over 121 vintages.

Nord-Kivu

  • Confirmed cases: bias 0.02 and 90% coverage 0.96 over 121 vintages.

  • Confirmed deaths: bias -0.04 and 90% coverage 0.91 over 121 vintages.

  • Confirmed cases as counts: bias 0.05 and 90% coverage 0.97 over 121 vintages.

  • Confirmed deaths as counts: bias -0.07 and 90% coverage 0.93 over 121 vintages.

Haut-Uele

  • Confirmed cases: bias 0.06 and 90% coverage 0.96 over 121 vintages.

  • Confirmed deaths: bias 0.01 and 90% coverage 0.96 over 121 vintages.

  • Confirmed cases as counts: bias 0.07 and 90% coverage 0.96 over 121 vintages.

  • Confirmed deaths as counts: bias -0.02 and 90% coverage 0.98 over 121 vintages.

Sud-Kivu

  • Confirmed cases: bias 0.01 and 90% coverage 0.99 over 121 vintages.

  • Confirmed deaths: bias 0.0 and 90% coverage 0.99 over 121 vintages.

  • Confirmed cases as counts: bias 0.02 and 90% coverage 0.99 over 121 vintages.

  • Confirmed deaths as counts: bias 0.0 and 90% coverage 0.99 over 121 vintages.

Tshopo

  • Confirmed cases: bias 0.04 and 90% coverage 0.96 over 121 vintages.

  • Confirmed deaths: bias -0.0 and 90% coverage 0.98 over 121 vintages.

  • Confirmed cases as counts: bias 0.04 and 90% coverage 0.97 over 121 vintages.

  • Confirmed deaths as counts: bias -0.0 and 90% coverage 0.98 over 121 vintages.

Bas-Uele

  • Confirmed cases: bias 0.0 and 90% coverage 0.98 over 121 vintages.

  • Confirmed deaths: bias 0.01 and 90% coverage 0.98 over 121 vintages.

  • Confirmed cases as counts: bias 0.01 and 90% coverage 0.98 over 121 vintages.

  • Confirmed deaths as counts: bias 0.01 and 90% coverage 0.98 over 121 vintages.

Sud-Ubangi

  • Confirmed cases: bias 0.0 and 90% coverage 0.98 over 121 vintages.

  • Confirmed deaths: bias 0.0 and 90% coverage 0.99 over 121 vintages.

  • Confirmed cases as counts: bias 0.0 and 90% coverage 0.98 over 121 vintages.

  • Confirmed deaths as counts: bias 0.0 and 90% coverage 0.99 over 121 vintages.

Nord-Ubangi

  • Confirmed cases: bias 0.01 and 90% coverage 0.99 over 121 vintages.

  • Confirmed deaths: bias 0.0 and 90% coverage 0.99 over 121 vintages.

  • Confirmed cases as counts: bias 0.01 and 90% coverage 0.99 over 121 vintages.

  • Confirmed deaths as counts: bias 0.0 and 90% coverage 0.99 over 121 vintages.

Province prior predictive check ​

Whether the province model's prior, before any data are fitted, brackets each province's observed share of the confirmed cases and deaths.

Draw from the province model's prior
julia
prior_patch_chn = patch_prior_draws(obs);

prior_province_table = patch_overview_table(prior_patch_chn, N_PATCHES);
Show prior province summary table
ProvinceReproduction numberCumulative infectionsShare of infections (%)Relative ascertainment
Ituri0.78 (0.08–1.66)1139933 (2130–3923354)75.9 (14.2–99.0)1.0 (0.67–1.6)
Nord-Kivu1.2 (0.09–2.26)44266 (10–7403236)6.8 (0.1–33.1)1.0 (0.63–1.49)
Haut-Uele1.16 (0.07–2.22)16341 (7–2005187)2.8 (0.1–12.2)1.0 (0.65–1.56)
Sud-Kivu1.21 (0.12–2.31)23577 (7–6368819)3.7 (0.1–26.0)1.0 (0.63–1.54)
Tshopo1.25 (0.1–2.3)8757 (3–2512212)1.4 (0.0–10.8)1.0 (0.64–1.54)
Bas-Uele1.25 (0.1–2.36)3874 (1–1218820)0.7 (0.0–5.3)1.0 (0.65–1.55)
Sud-Ubangi1.3 (0.12–2.4)4937 (1–2668819)0.8 (0.0–10.3)1.0 (0.61–1.57)
Nord-Ubangi1.24 (0.13–2.32)2290 (1–1379666)0.4 (0.0–5.3)1.0 (0.6–1.56)
Province composition prior predictive checks
julia
prior_case_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province, prior"
);

prior_death_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province, prior"
);

Province prior and posterior pair plot
julia
# Per-province entries of a vector deterministic, one draw vector each,
# keyed `<name>_<patch>` so they can sit side by side in one table.
function _patch_columns(chn, key::Symbol, name::AbstractString; f = identity)
    draws = [collect(v) for v in vec(collect(chn[key]))]
    return [
        Symbol(name, "_", p) => Float64[f(d[p]) for d in draws]
            for p in 1:N_PATCHES
    ]
end
_province_pair_draws(chn) = NamedTuple(
    vcat(
        _patch_columns(chn, :R_T_patch, "R_T"),
        _patch_columns(chn, :C_T_patch, "log10_C_T"; f = x -> log10(x + 1))
    )
)
_province_pair_labels = Dict(
    Symbol(name, "_", p) => string(label, ", ", PROVINCE_LABELS[p])
        for p in 1:N_PATCHES
        for (name, label) in (("R_T", "R_T"), ("log10_C_T", "log10 C_T"))
)
province_pair_fig = plot_pair(
    _province_pair_draws(chn_joint);
    prior = _province_pair_draws(prior_patch_chn),
    labels = _province_pair_labels
);

Province compositions ​

Whether each province's modelled share of the national confirmed cases and deaths reproduces the observed share at every spatial vintage (see the province compositions Methods section).

Province composition posterior predictive checks
julia
province_case_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province"
);

province_death_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province"
);

The three province terms for the isolation and laboratory data are checked the same way. A gap in a panel is a day on which that province printed nothing.

Province laboratory, occupancy and bed split posterior predictive checks
julia
province_lab_bin_days = [
    maximum(province_lab.days[province_lab.bins .== b])
        for b in 1:maximum(province_lab.bins)
];

province_lab_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    days = province_lab_bin_days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Analysed specimen share by province, by week"
);

province_occupancy_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_occupancy_share, rows = province_isolation,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation occupancy share by province"
);

province_beds_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_capacity_share, rows = province_capacity,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation bed share by province"
);

The split of each calendar week's analysed specimens among the provinces, which identifies the background split.

The split of the patients in isolation among the provinces printed that day, on the weekly days the fit scores, against the per-patch bed demand.

The split of the beds among the provinces printed that day, on the days a count changed, against each patch's modelled share of the national capacity.

Province stream calibration ​

Whether each province's share predictions are calibrated at the observed national total, scored as in the national stream calibration. The analysed-specimen rows score the weekly laboratory split the same way, at each week's observed national analysed total.

Build the province calibration panels
julia
province_case_panels = province_composition_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    stream = "Confirmed cases", n_patches = N_PATCHES
);
province_death_panels = province_composition_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_lab_panels = province_composition_panels(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    stream = "Analysed specimens", n_patches = N_PATCHES
);
province_panels = vcat(
    province_case_panels, province_death_panels, province_lab_panels
);
province_calibration_table = stream_calibration(province_panels);
province_calibration_fig = plot_stream_calibration(
    province_calibration_table
);

Province calibration table
StreamVintagesBias50% coverage90% coverage
Confirmed cases, Ituri121-0.020.630.95
Confirmed cases, Nord-Kivu1210.020.580.96
Confirmed cases, Haut-Uele1210.060.730.96
Confirmed cases, Sud-Kivu1210.010.990.99
Confirmed cases, Tshopo1210.040.830.96
Confirmed cases, Bas-Uele12100.910.98
Confirmed cases, Sud-Ubangi12100.980.98
Confirmed cases, Nord-Ubangi1210.010.980.99
Confirmed deaths, Ituri1210.020.640.93
Confirmed deaths, Nord-Kivu121-0.040.60.91
Confirmed deaths, Haut-Uele1210.010.790.96
Confirmed deaths, Sud-Kivu12100.990.99
Confirmed deaths, Tshopo12100.890.98
Confirmed deaths, Bas-Uele1210.010.970.98
Confirmed deaths, Sud-Ubangi12100.990.99
Confirmed deaths, Nord-Ubangi12100.990.99
Analysed specimens, Ituri190.0611
Analysed specimens, Nord-Kivu19-0.270.951
Analysed specimens, Haut-Uele190.030.681
Analysed specimens, Sud-Kivu190.310.581
Analysed specimens, Tshopo19-0.090.631
Analysed specimens, Bas-Uele1900.741
Analysed specimens, Sud-Ubangi190.110.951
Analysed specimens, Nord-Ubangi190.070.951

Province counts ​

Whether each province's cases and deaths are reproduced as counts once the national total is predicted as well.

Build the province count panels
julia
# The joint posterior predictive, shared with the national page (see
# `joint_posterior_predictive` in `docs/pages/_setup.jl`). Its draws pair
# one to one with `chn_joint`, whose shares split them.
pp_joint = joint_posterior_predictive();
_pp_draws(vn) = collect(pp_joint[BVDOutbreakSize.FlexiChains.Prefixed(vn)]);

# The national confirmed cases are replicated per laboratory window, oldest
# first, as on the national page. The first confirmed vintage is an
# unreplicated baseline.
_conf_windows = BVDOutbreakSize.confirmed_positivity_windows(
    obs.confirmed_history, obs.lab_history, obs.lab_daily_history
);
_conf_obs = collect(
    first(
        pp_joint[k] for k in keys(pp_joint)
            if occursin(
                "confirmed_state.confirmed_positives.positives", string(k)
            )
    )
);
_conf_replicates = [
    vcat(collect(e), collect(p), collect(l))
        for (e, p, l) in zip(
            vec(_pp_draws(@varname(early_increments.increments))),
            vec(_conf_obs),
            vec(_pp_draws(@varname(late_increments.increments)))
        )
];
province_case_count_panels = province_count_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    province_days = province_cases.days,
    national_days = vcat(
        _conf_windows.early_days, _conf_windows.obs_days,
        _conf_windows.late_days
    ),
    national_replicates = _conf_replicates,
    baseline = Int(first(obs.confirmed_history.counts)),
    stream = "Confirmed cases", n_patches = N_PATCHES
);
# The national confirmed deaths are replicated from zero at every vintage.
province_death_count_panels = province_count_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    province_days = province_deaths.days,
    national_days = obs.confirmed_deaths_history.days,
    national_replicates = _pp_draws(@varname(cdeath_increments.increments)),
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_count_calibration_table = stream_calibration(
    vcat(province_case_count_panels, province_death_count_panels)
);
province_count_calibration_fig = plot_stream_calibration(
    province_count_calibration_table
);
Province count calibration plot

Province count calibration table
StreamVintagesBias50% coverage90% coverage
Confirmed cases, Ituri1210.050.740.96
Confirmed cases, Nord-Kivu1210.050.740.97
Confirmed cases, Haut-Uele1210.070.730.96
Confirmed cases, Sud-Kivu1210.020.990.99
Confirmed cases, Tshopo1210.040.860.97
Confirmed cases, Bas-Uele1210.010.910.98
Confirmed cases, Sud-Ubangi12100.980.98
Confirmed cases, Nord-Ubangi1210.010.980.99
Confirmed deaths, Ituri121-0.040.680.93
Confirmed deaths, Nord-Kivu121-0.070.580.93
Confirmed deaths, Haut-Uele121-0.020.750.98
Confirmed deaths, Sud-Kivu12100.990.99
Confirmed deaths, Tshopo12100.890.98
Confirmed deaths, Bas-Uele1210.010.970.98
Confirmed deaths, Sud-Ubangi12100.990.99
Confirmed deaths, Nord-Ubangi12100.990.99

Province correlations and totals ​

Which province quantities trade off against each other, and whether each province's case and death totals agree with the observed ones.

Province posterior correlation heatmap
julia
# LaTeX label for one province's entry of a quantity. Spaces in a province
# name need escaping inside `\mathrm`.
_province_tex(sym, p) = string(
    sym, raw"\ (\mathrm{", replace(PROVINCE_LABELS[p], " " => raw"\ "), "})"
)
province_correlation_draws = NamedTuple(
    vcat(
        [:C_T => vec(Array(chn_joint[:C_T]))],
        _patch_columns(chn_joint, :R_T_patch, "R_T"),
        _patch_columns(chn_joint, :province_ascertainment, "asc"),
        _patch_columns(chn_joint, :CFR_patch, "CFR")
    )
);
province_correlation_labels = merge(
    Dict(:C_T => raw"C_T"),
    Dict(
        Symbol(name, "_", p) => _province_tex(tex, p)
            for p in 1:N_PATCHES
            for (name, tex) in (
                ("R_T", raw"R_T"), ("asc", raw"p_\mathrm{asc}"),
                ("CFR", raw"\mathrm{CFR}"),
            )
    )
);
province_correlation_fig = plot_correlation_heatmap(
    province_correlation_draws; labels = province_correlation_labels
);

Province totals against observed
julia
# Keyed by panel title, which names the stream and the province. A total
# that is the same in every draw (a patch with no deaths at any vintage) has
# no spread to plot and gives the corner plot a zero-width axis, so it is
# left out.
_varying_panels = filter(
    p -> length(unique(sum(r) for r in p.replicates)) > 1, province_panels
);
province_totals = NamedTuple(
    Symbol(p.title) => [sum(Float64.(r)) for r in p.replicates]
        for p in _varying_panels
);
province_observed = NamedTuple(
    Symbol(p.title) => Float64(sum(p.observed)) for p in _varying_panels
);
province_pairs_fig = plot_stream_pairs(province_totals, province_observed);

Province sensitivity to assumptions ​

Spatial structure sensitivity ​

We refit the joint model with its eight patches reduced to one well-mixed population. Splitting the country into provinces adds no national data, so the national outbreak size should not move far either way. The two still differ, because the country grows at the force-weighted mean of the provinces' reproduction numbers, which sits above the central trend they pool toward. That gap is first order in the deviation scale, so a large gap between the two posteriors would point at the deviation priors rather than at the data.

julia
spatial_sensitivity_table = streams_table(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population (n_patches = 1)" => posterior_C_no_patches
);
spatial_sensitivity_table
2×7 DataFrame
RowStreamLower 90%Lower 60%Lower 30%Upper 30%Upper 60%Upper 90%
StringFloat64Float64Float64Float64Float64Float64
1Meta-population (headline)14896.017752.019852.023821.026643.032793.0
2Single population (n_patches = 1)15015.017940.019782.023602.026617.032038.0
Spatial-structure density overlay
julia
spatial_sensitivity_fig = plot_density_overlay(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population" => posterior_C_no_patches;
    xlabel = "Cumulative infections"
);

Posterior densities of the cumulative infections under the headline province model and the single-population fit.

The spatial structure could also move other national quantities that both fits estimate, so their posteriors should sit on top of each other.

National quantities under both structures
julia
spatial_rt_fig = plot_rt_streams(
    [
        (;
            label = "Single population (n_patches = 1)",
            chn = chn_no_patches, rt_start = _rt_start_plot,
            rt_walk_start = clamp(
                _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
                obs.n
            ), colour = :steelblue,
        ),
    ];
    joint = (;
        chn = chn_joint, rt_start = _rt_start_plot,
        rt_walk_start = clamp(
            _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
            obs.n
        ),
    ),
    n = obs.n, breakpoint = _BREAKPOINT,
    as_of_date = string(obs.cutoff), seeding = obs.seeding,
    display_start = _rt_start_plot, ncols = 1,
    title = "National reproduction number under both structures",
    reference_label = "the meta-population headline",
    panel_label = "the single-population fit"
);

spatial_cfr_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:CFR])),
    "Single population" => vec(Array(chn_no_patches[:CFR]));
    xlabel = "Case-fatality ratio"
);

spatial_rt_density_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:R_T])),
    "Single population" => vec(Array(chn_no_patches[:R_T]));
    xlabel = "Reproduction number at the cut-off"
);

The national reproduction number from the single-population fit (coloured) against the headline province model (grey), which is hidden where the two agree.

Posterior densities of the case-fatality ratio under both structures.

Posterior densities of the reproduction number at the cut-off under both structures.
National quantities under both structures, as a table
julia
# One row per national quantity, one column per structure, each cell a
# median with a 90% credible interval. Built here rather than by stacking
# two `summary_table` calls, so the two structures sit side by side and the
# reader compares along a row.
spatial_quantities_table = let
    # A count rounded to zero decimals still prints a trailing ".0", so
    # whole-number quantities go through `Int`.
    fmt(x, d) = d <= 0 ? string(round(Int, x)) : string(round(x; digits = d))
    cell(v, d) = string(
        fmt(quantile(v, 0.5), d), " (",
        fmt(quantile(v, 0.05), d), "–", fmt(quantile(v, 0.95), d), ")"
    )
    rows = [
        ("Cumulative infections", :C_T, 0),
        ("Reproduction number at the cut-off", :R_T, 2),
        ("Case-fatality ratio", :CFR, 2),
        ("Outbreak age (days)", :T, 0),
        ("Latest growth rate (per day)", :r, 3),
    ]
    DataFrame(
        "Quantity" => [r[1] for r in rows],
        "Meta-population (headline)" => [
            cell(
                vec(Array(chn_joint[r[2]])), r[3]
            )
                for r in rows
        ],
        "Single population" => [
            cell(vec(Array(chn_no_patches[r[2]])), r[3])
                for r in rows
        ]
    )
end;
5×3 DataFrame
RowQuantityMeta-population (headline)Single population
StringStringString
1Cumulative infections21650 (14896–32793)21604 (15015–32038)
2Reproduction number at the cut-off1.08 (0.78–1.49)1.06 (0.78–1.51)
3Case-fatality ratio0.49 (0.38–0.6)0.5 (0.4–0.62)
4Outbreak age (days)217 (206–234)218 (207–234)
5Latest growth rate (per day)0.006 (-0.017–0.031)0.004 (-0.016–0.031)
The same three quantities as credible intervals under both structures, with the outbreak start date each fit implies. This is where the agreement is read rather than from the figures.

Province parameter recovery ​

Whether the model recovers each province's values when fitted to data it simulated itself, from the same runs as the national parameter recovery.

Recovery figure
julia
province_recovery_results = recovery_results()
_recovery_short = [
    "C_T_patch" => "C_T", "R_T_patch" => "R_T", "CFR_patch" => "CFR",
    "province_ascertainment" => "ascertainment",
    "region_drift_sd" => "drift SD",
]
province_recovery_quantities = [
    "$(k)[$(p)]" for (k, _) in _recovery_short for p in PROVINCE_LABELS
]
province_recovery_fig = isempty(province_recovery_results.params) ? nothing :
    plot_recovery(
        province_recovery_results.params, province_recovery_results.draws,
        province_recovery_results.prior;
        quantities = province_recovery_quantities,
        labels = Dict(
            "$(k)[$(p)]" => "$(v), $(p)" for (k, v) in _recovery_short
            for p in PROVINCE_LABELS
        ),
        panel_labels = Dict(
            "$(k)[$(p)]" => p for (k, _) in _recovery_short for p in PROVINCE_LABELS
        ),
        row_labels = last.(_recovery_short),
        log_x = ["C_T_patch[$(p)]" for p in PROVINCE_LABELS]
    );

Each seed's posterior median with its 50% and 90% intervals divided by its true value (top). Below, each quantity by province is on its own scale, with the prior in grey, each seed's posterior in its colour and its true value dashed.
Summary table across seeds
julia
province_recovery_summary = isempty(province_recovery_results.params) ?
    DataFrame() :
    recovery_summary_table(province_recovery_results.params; province = true);

province_recovery_summary_display = isempty(province_recovery_summary) ?
    Markdown.parse("No parameter-recovery run is available for this build.") :
    MarkdownTable(province_recovery_summary);
QuantityTruth, lowestTruth, highestRelative error, medianRelative error, rangez, medianTruth in 90%Outside 99%
CFR_patch[Bas-Uele]0.2420.4690.07-0.05 to 0.120.413/30
CFR_patch[Haut-Uele]0.3330.505-0.09-0.21 to 0.04-0.423/30
CFR_patch[Ituri]0.2580.506-0.03-0.08 to 0.01-0.093/30
CFR_patch[Nord-Kivu]0.1930.549-0.08-0.08 to 0.22-0.413/30
CFR_patch[Nord-Ubangi]0.3310.48-0.06-0.22 to 0.08-0.253/30
CFR_patch[Sud-Kivu]0.3310.552-0.08-0.22 to -0.03-0.613/30
CFR_patch[Sud-Ubangi]0.3350.548-0.08-0.3 to -0.07-0.563/30
CFR_patch[Tshopo]0.2550.628-0.06-0.18 to 0.07-0.232/30
C_T_patch[Bas-Uele]14.3171-0.01-0.03 to 0.350.143/30
C_T_patch[Haut-Uele]33.426000.280.19 to 0.831.053/30
C_T_patch[Ituri]3060190000.310.11 to 0.421.12/30
C_T_patch[Nord-Kivu]35.55270.470.28 to 0.481.012/30
C_T_patch[Nord-Ubangi]2.21160.20.08 to 1.450.782/30
C_T_patch[Sud-Kivu]75.41960.50.31 to 0.911.272/30
C_T_patch[Sud-Ubangi]4.9566.20.490.12 to 1.711.322/30
C_T_patch[Tshopo]12.62910.350.22 to 0.410.823/30
R_T_patch[Bas-Uele]0.8251.20.170.0 to 0.291.52/30
R_T_patch[Haut-Uele]0.941.210.110.05 to 0.160.73/30
R_T_patch[Ituri]1.041.130.05-0.01 to 0.230.462/30
R_T_patch[Nord-Kivu]1.011.190.070.05 to 0.180.683/30
R_T_patch[Nord-Ubangi]11.230.07-0.06 to 0.150.73/30
R_T_patch[Sud-Kivu]1.021.230.08-0.02 to 0.140.593/30
R_T_patch[Sud-Ubangi]1.011.170.070.05 to 0.20.652/30
R_T_patch[Tshopo]1.031.260.04-0.13 to 0.120.443/30
province_ascertainment[Bas-Uele]0.8921.310.1-0.01 to 0.10.383/30
province_ascertainment[Haut-Uele]0.8461.05-0.05-0.15 to 0.04-0.193/30
province_ascertainment[Ituri]0.8281.120.050.02 to 0.110.463/30
province_ascertainment[Nord-Kivu]1.012.12-0.11-0.36 to -0.0-0.73/30
province_ascertainment[Nord-Ubangi]0.5640.9980.16-0.0 to 0.760.812/30
province_ascertainment[Sud-Kivu]0.85810-0.05 to 0.070.013/30
province_ascertainment[Sud-Ubangi]0.9961.490.01-0.37 to 0.030.312/30
province_ascertainment[Tshopo]0.6191-0.01-0.1 to 0.36-0.153/30
region_drift_sd[Bas-Uele]0.009290.06320.22-0.55 to 0.320.433/30
region_drift_sd[Haut-Uele]0.00770.0347-0.07-0.12 to 1.150.213/30
region_drift_sd[Ituri]0.01550.0412-0.39-0.41 to 0.05-0.273/30
region_drift_sd[Nord-Kivu]0.008750.04240.2-0.35 to 0.920.433/30
region_drift_sd[Nord-Ubangi]0.008460.0518-0.16-0.46 to 0.90.123/30
region_drift_sd[Sud-Kivu]0.009850.0561-0.23-0.45 to 0.660.053/30
region_drift_sd[Sud-Ubangi]0.007020.03010-0.29 to 1.220.163/30
region_drift_sd[Tshopo]0.005090.0576-0.38-0.49 to 2.18-0.253/30
The error of each seed's posterior median relative to the truth, and the z-score of the truth, summarised across seeds.
Each seed's recovered province values
julia
province_recovery = let r = province_recovery_results.params
    isempty(r) ? DataFrame() :
        r[
            occursin.("[", r.quantity), [
                :seed, :quantity, :truth, :median, :lower_90, :upper_90,
                :covered_90,
            ],
        ]
end;

province_recovery_display = isempty(province_recovery) ?
    Markdown.parse("No parameter-recovery run is available for this build.") : MarkdownTable(province_recovery);
seedquantitytruthmedianlower_90upper_90covered_90
1CFR_patch[Bas-Uele]0.4690.5020.3680.622true
1CFR_patch[Haut-Uele]0.5050.5230.4020.645true
1CFR_patch[Ituri]0.5060.5140.3950.632true
1CFR_patch[Nord-Kivu]0.5490.5040.3740.622true
1CFR_patch[Nord-Ubangi]0.480.5160.390.639true
1CFR_patch[Sud-Kivu]0.5520.5060.380.626true
1CFR_patch[Sud-Ubangi]0.5480.5080.3870.628true
1CFR_patch[Tshopo]0.6280.5180.3970.627false
1C_T_patch[Bas-Uele]35.6935.45222.03358.438true
1C_T_patch[Haut-Uele]121.383144.344100.798217.793true
1C_T_patch[Ituri]16872.76618787.28215014.08124874.598true
1C_T_patch[Nord-Kivu]155.808228.493159.572341.58false
1C_T_patch[Nord-Ubangi]25.44427.56617.84944.141true
1C_T_patch[Sud-Kivu]101.378132.57191.271198.429true
1C_T_patch[Sud-Ubangi]40.99645.72929.38371.854true
1C_T_patch[Tshopo]36.91445.19731.03868.561true
1R_T_patch[Bas-Uele]0.8251.0680.841.373false
1R_T_patch[Haut-Uele]0.941.040.8031.333true
1R_T_patch[Ituri]1.0411.0950.8871.372true
1R_T_patch[Nord-Kivu]1.0141.0660.8381.371true
1R_T_patch[Nord-Ubangi]1.1641.0950.8681.421true
1R_T_patch[Sud-Kivu]1.0231.1090.8751.424true
1R_T_patch[Sud-Ubangi]1.0431.0930.8621.413true
1R_T_patch[Tshopo]1.2011.0460.7961.335true
1province_ascertainment[Bas-Uele]1.3111.4421.0052.127true
1province_ascertainment[Haut-Uele]0.8460.7190.5210.995true
1province_ascertainment[Ituri]0.8280.9190.7171.154true
1province_ascertainment[Nord-Kivu]1.2011.0740.8631.369true
1province_ascertainment[Nord-Ubangi]0.9061.0510.7811.521true
1province_ascertainment[Sud-Kivu]0.9680.9650.7491.226true
1province_ascertainment[Sud-Ubangi]1.1861.2280.9261.761true
1province_ascertainment[Tshopo]0.8720.7830.5161.054true
1region_drift_sd[Bas-Uele]0.06320.02860.00560.0688true
1region_drift_sd[Haut-Uele]0.03470.03050.006040.0709true
1region_drift_sd[Ituri]0.04120.02530.006190.0567true
1region_drift_sd[Nord-Kivu]0.04240.02740.006370.063true
1region_drift_sd[Nord-Ubangi]0.05180.02810.006060.0713true
1region_drift_sd[Sud-Kivu]0.05610.03090.006060.0717true
1region_drift_sd[Sud-Ubangi]0.03010.03020.006260.0717true
1region_drift_sd[Tshopo]0.05760.02920.006110.0694true
2CFR_patch[Bas-Uele]0.3350.3180.2220.43true
2CFR_patch[Haut-Uele]0.3380.3090.2120.418true
2CFR_patch[Ituri]0.3330.3080.2130.415true
2CFR_patch[Nord-Kivu]0.3380.3090.2150.417true
2CFR_patch[Nord-Ubangi]0.3340.3130.220.427true
2CFR_patch[Sud-Kivu]0.3310.320.2250.437true
2CFR_patch[Sud-Ubangi]0.3350.3080.2130.418true
2CFR_patch[Tshopo]0.3340.3150.2220.435true
2C_T_patch[Bas-Uele]171.059230.431161.677365.618true
2C_T_patch[Haut-Uele]2598.9933333.8862418.9025137.283true
2C_T_patch[Ituri]19040.12425006.82519256.82536802.588false
2C_T_patch[Nord-Kivu]526.568675.594486.0431067.088true
2C_T_patch[Nord-Ubangi]115.59139.0195.909218.767true
2C_T_patch[Sud-Kivu]195.517292.547200.05456.881false
2C_T_patch[Sud-Ubangi]66.20698.71965.787153.917true
2C_T_patch[Tshopo]290.917391.716278.796606.723true
2R_T_patch[Bas-Uele]1.1961.4031.1611.603true
2R_T_patch[Haut-Uele]1.2141.4081.1611.608true
2R_T_patch[Ituri]1.1351.3911.1621.575false
2R_T_patch[Nord-Kivu]1.1941.4071.1731.604true
2R_T_patch[Nord-Ubangi]1.2261.411.1781.625true
2R_T_patch[Sud-Kivu]1.2251.41.1491.599true
2R_T_patch[Sud-Ubangi]1.1731.411.1781.619false
2R_T_patch[Tshopo]1.261.4111.1681.605true
2province_ascertainment[Bas-Uele]1.0020.9870.8191.087true
2province_ascertainment[Haut-Uele]0.9991.0360.9511.255true
2province_ascertainment[Ituri]0.9941.0140.9141.215true
2province_ascertainment[Nord-Kivu]1.0071.0050.91.191true
2province_ascertainment[Nord-Ubangi]0.9980.9970.8441.147true
2province_ascertainment[Sud-Kivu]1.0040.9590.7351.05true
2province_ascertainment[Sud-Ubangi]0.9961.010.9031.245true
2province_ascertainment[Tshopo]1.00.9940.8581.116true
2region_drift_sd[Bas-Uele]0.009290.01130.001240.043true
2region_drift_sd[Haut-Uele]0.01210.01130.001340.0415true
2region_drift_sd[Ituri]0.01860.0110.001180.0424true
2region_drift_sd[Nord-Kivu]0.009050.01090.001220.0405true
2region_drift_sd[Nord-Ubangi]0.01320.01110.001210.0411true
2region_drift_sd[Sud-Kivu]0.01450.01120.001310.0431true
2region_drift_sd[Sud-Ubangi]0.01540.01090.001360.0422true
2region_drift_sd[Tshopo]0.01820.01130.001180.0422true
3CFR_patch[Bas-Uele]0.2420.270.170.389true
3CFR_patch[Haut-Uele]0.3330.2630.1680.382true
3CFR_patch[Ituri]0.2580.2490.1530.366true
3CFR_patch[Nord-Kivu]0.1930.2350.1230.354true
3CFR_patch[Nord-Ubangi]0.3310.2590.1630.379true
3CFR_patch[Sud-Kivu]0.3560.2780.1820.406true
3CFR_patch[Sud-Ubangi]0.3830.2680.170.394true
3CFR_patch[Tshopo]0.2550.2730.1760.403true
3C_T_patch[Bas-Uele]14.313.8086.49828.813true
3C_T_patch[Haut-Uele]33.36561.02332.081113.866true
3C_T_patch[Ituri]3056.0344345.7062723.1537230.576true
3C_T_patch[Nord-Kivu]35.50152.57526.369106.874true
3C_T_patch[Nord-Ubangi]2.2025.3982.34912.539false
3C_T_patch[Sud-Kivu]75.382144.03474.426273.494true
3C_T_patch[Sud-Ubangi]4.95113.4396.40828.529false
3C_T_patch[Tshopo]12.55117.7418.48837.413true
3R_T_patch[Bas-Uele]1.0741.0780.8991.325true
3R_T_patch[Haut-Uele]1.0261.0790.9111.312true
3R_T_patch[Ituri]1.081.0730.9131.288true
3R_T_patch[Nord-Kivu]1.0061.0790.9091.321true
3R_T_patch[Nord-Ubangi]1.0031.0750.9021.305true
3R_T_patch[Sud-Kivu]1.0961.0710.8941.278true
3R_T_patch[Sud-Ubangi]1.0121.0810.9091.309true
3R_T_patch[Tshopo]1.0291.0710.91.283true
3province_ascertainment[Bas-Uele]0.8920.9780.6231.394true
3province_ascertainment[Haut-Uele]1.0541.00.7041.393true
3province_ascertainment[Ituri]1.1221.1720.9111.758true
3province_ascertainment[Nord-Kivu]2.1221.3530.9592.461true
3province_ascertainment[Nord-Ubangi]0.5640.9910.6121.526false
3province_ascertainment[Sud-Kivu]0.8580.9160.651.173true
3province_ascertainment[Sud-Ubangi]1.490.940.5731.315false
3province_ascertainment[Tshopo]0.6190.840.4361.142true
3region_drift_sd[Bas-Uele]0.01260.01660.001390.0566true
3region_drift_sd[Haut-Uele]0.00770.01650.001360.0544true
3region_drift_sd[Ituri]0.01550.01620.001390.0543true
3region_drift_sd[Nord-Kivu]0.008750.01680.001370.0585true
3region_drift_sd[Nord-Ubangi]0.008460.0160.001430.0563true
3region_drift_sd[Sud-Kivu]0.009850.01640.001420.0561true
3region_drift_sd[Sud-Ubangi]0.007020.01560.001410.0553true
3region_drift_sd[Tshopo]0.005090.01620.001420.0568true

Saving province in-sample outputs ​

Write the summary bullets
julia
# The bullets under the summary heading at the top of the page. They read
# tables built further down, so they are written here and read back when
# the site is assembled.
evaluation_insample_province_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    fitted = filter(r -> isfinite(r["Bias"]), province_calibration_table)
    worst = first(
        sort(fitted, "Bias"; by = abs, rev = true), min(3, size(fitted, 1))
    )
    function coverage_bullet(heading, tbl, scale)
        low = first(sort(tbl, "90% coverage"), min(1, size(tbl, 1)))
        return string(
            "- **", heading, ":** ", count(>=(0.8), tbl[!, "90% coverage"]),
            " of ", size(tbl, 1), " province streams have 90% coverage of ",
            "at least 0.8", scale,
            size(low, 1) == 0 ? "." :
                string(
                    "; the lowest is ", low[1, "Stream"], " at ",
                    fmt(low[1, "90% coverage"]), "."
                )
        )
    end
    overall = [
        string(
            "- **Least well reproduced:** ",
            join(
                [
                    string(
                        r["Stream"], " (bias ", fmt(r["Bias"]),
                        ", 90% coverage ", fmt(r["90% coverage"]), ")"
                    )
                        for r in eachrow(worst)
                ], "; "
            ), "."
        ),
        coverage_bullet("Coverage", fitted, ""),
        coverage_bullet(
            "Count-scale coverage",
            filter(
                r -> isfinite(r["Bias"]), province_count_calibration_table
            ),
            ""
        ),
    ]
    rows(tbl) = Dict(r["Stream"] => r for r in eachrow(tbl))
    cal = rows(province_calibration_table)
    count_cal = rows(province_count_calibration_table)
    function calibration(tbl, kind, p, label)
        r = get(tbl, string(kind, ", ", PROVINCE_LABELS[p]), nothing)
        r === nothing && return string("- ", label, ": not scored.")
        return string(
            "- ", label, ": bias ", fmt(r["Bias"]), " and 90% coverage ",
            fmt(r["90% coverage"]), " over ", r["Vintages"], " vintages."
        )
    end
    detail = [
        join(
            [
                string("**", PROVINCE_LABELS[p], "**"), "",
                calibration(cal, "Confirmed cases", p, "Confirmed cases"),
                calibration(cal, "Confirmed deaths", p, "Confirmed deaths"),
                calibration(
                    count_cal, "Confirmed cases", p,
                    "Confirmed cases as counts"
                ),
                calibration(
                    count_cal, "Confirmed deaths", p,
                    "Confirmed deaths as counts"
                ),
            ], "\n"
        )
            for p in 1:N_PATCHES
    ]
    join(vcat([join(overall, "\n")], detail), "\n\n")
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_insample_province.md"),
    evaluation_insample_province_summary
);