Province in-sample checks

Whether the fitted joint model reproduces the per-province data it was fitted to. It also re-fits the joint as a single population to check the spatial structure. The national checks are on the national in-sample checks page.

Load packages, data and fitted chains

julia

# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))

validation_forecast_from (generic function with 1 method)

julia

# The fits this page reads, loaded from the cache here.
chn_joint = load_fit("joint")
# `sens_no_patches` is the headline with `n_patches = 1`, the check on the
# spatial structure.
chn_no_patches = load_fit("sens_no_patches")
posterior_C_joint = vec(Array(chn_joint[:C_T]))
posterior_C_no_patches = vec(Array(chn_no_patches[:C_T]));

Summary

Whether each province's cases and deaths are reproduced, from the checks further down this page. Bias runs from −1 to 1 and is negative when the model under-predicts, and coverage is the fraction of vintages inside the 90% predictive interval.

Least well reproduced: Analysed specimens, Nord-Kivu (bias -0.3, 90% coverage 1.0); Analysed specimens, Ituri (bias 0.22, 90% coverage 1.0); Confirmed cases, Haut-Uele (bias 0.11, 90% coverage 0.94).
Coverage: 12 of 12 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Ituri at 0.93.
Count-scale coverage: 8 of 8 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Ituri at 0.92.

Ituri

Confirmed cases: bias -0.04 and 90% coverage 0.95 over 99 vintages.
Confirmed deaths: bias 0.01 and 90% coverage 0.93 over 99 vintages.
Confirmed cases as counts: bias 0.06 and 90% coverage 0.96 over 99 vintages.
Confirmed deaths as counts: bias -0.06 and 90% coverage 0.92 over 99 vintages.

Nord-Kivu

Confirmed cases: bias 0.02 and 90% coverage 0.95 over 99 vintages.
Confirmed deaths: bias -0.03 and 90% coverage 0.94 over 99 vintages.
Confirmed cases as counts: bias 0.07 and 90% coverage 0.98 over 99 vintages.
Confirmed deaths as counts: bias -0.08 and 90% coverage 0.97 over 99 vintages.

Haut-Uele

Confirmed cases: bias 0.11 and 90% coverage 0.94 over 99 vintages.
Confirmed deaths: bias 0.03 and 90% coverage 0.97 over 99 vintages.
Confirmed cases as counts: bias 0.13 and 90% coverage 0.95 over 99 vintages.
Confirmed deaths as counts: bias -0.01 and 90% coverage 0.97 over 99 vintages.

Other provinces

Confirmed cases: bias 0.03 and 90% coverage 0.98 over 99 vintages.
Confirmed deaths: bias -0.01 and 90% coverage 0.98 over 99 vintages.
Confirmed cases as counts: bias 0.05 and 90% coverage 0.98 over 99 vintages.
Confirmed deaths as counts: bias -0.01 and 90% coverage 0.98 over 99 vintages.

Province prior predictive check

Whether the four-patch prior, before any data are fitted, brackets each province's observed share of the confirmed cases and deaths.

Draw from the four-patch prior

julia

prior_patch_chn = patch_prior_draws(obs);

prior_province_table = patch_overview_table(prior_patch_chn, N_PATCHES);

Show prior province summary table

Province	Reproduction number	Cumulative infections	Share of infections (%)	Relative ascertainment
Ituri	1.01 (0.11–1.84)	324436 (1630–3882486)	88.1 (15.0–99.4)	1.0 (0.65–1.51)
Nord-Kivu	1.35 (0.17–2.44)	13100 (15–7118721)	4.2 (0.2–29.9)	1.0 (0.65–1.49)
Haut-Uele	1.34 (0.15–2.39)	5261 (6–1931361)	1.8 (0.1–10.1)	1.0 (0.67–1.48)
Other provinces	1.39 (0.19–2.47)	14067 (13–12309122)	4.7 (0.2–49.1)	1.0 (0.65–1.51)

Province composition prior predictive checks

julia

prior_case_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province, prior"
);

prior_death_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province, prior"
);

Province prior and posterior pair plot

julia

# Per-province entries of a vector deterministic, one draw vector each,
# keyed `<name>_<patch>` so they can sit side by side in one table.
function _patch_columns(chn, key::Symbol, name::AbstractString; f = identity)
    draws = [collect(v) for v in vec(collect(chn[key]))]
    return [
        Symbol(name, "_", p) => Float64[f(d[p]) for d in draws]
            for p in 1:N_PATCHES
    ]
end
_province_pair_draws(chn) = NamedTuple(
    vcat(
        _patch_columns(chn, :R_T_patch, "R_T"),
        _patch_columns(chn, :C_T_patch, "log10_C_T"; f = x -> log10(x + 1))
    )
)
_province_pair_labels = Dict(
    Symbol(name, "_", p) => string(label, ", ", PROVINCE_LABELS[p])
        for p in 1:N_PATCHES
        for (name, label) in (("R_T", "R_T"), ("log10_C_T", "log10 C_T"))
)
province_pair_fig = plot_pair(
    _province_pair_draws(chn_joint);
    prior = _province_pair_draws(prior_patch_chn),
    labels = _province_pair_labels
);

Province compositions

Whether each province's modelled share of the national confirmed cases and deaths reproduces the observed share at every spatial vintage (see the province compositions Methods section).

Province composition posterior predictive checks

julia

province_case_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province"
);

province_death_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province"
);

The three province terms for the isolation and laboratory data are checked the same way. The laboratory panel is the split of each calendar week's analysed specimens, which identifies the background split. The occupancy panel is the split of the patients in isolation among the provinces printed that day, on the weekly days the fit scores, over the per-patch bed demand. The bed panel is the split of the beds among the provinces printed that day, on the days a count changed, over each patch's modelled share of the national capacity. A gap in a panel is a day on which that province printed nothing.

Province laboratory, occupancy and bed split posterior predictive checks

julia

province_lab_bin_days = [
    maximum(province_lab.days[province_lab.bins .== b])
        for b in 1:maximum(province_lab.bins)
];

province_lab_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    days = province_lab_bin_days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Analysed specimen share by province, by week"
);

province_occupancy_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_occupancy_share, rows = province_isolation,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation occupancy share by province"
);

province_beds_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_capacity_share, rows = province_capacity,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation bed share by province"
);

Province stream calibration

Whether each province's share predictions are calibrated at the observed national total, scored as in the national stream calibration. The analysed-specimen rows score the weekly laboratory split the same way, at each week's observed national analysed total.

Build the province calibration panels

julia

province_case_panels = province_composition_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    stream = "Confirmed cases", n_patches = N_PATCHES
);
province_death_panels = province_composition_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_lab_panels = province_composition_panels(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    stream = "Analysed specimens", n_patches = N_PATCHES
);
province_panels = vcat(
    province_case_panels, province_death_panels, province_lab_panels
);
province_calibration_table = stream_calibration(province_panels);
province_calibration_fig = plot_stream_calibration(
    province_calibration_table
);

Province calibration table

Stream	Vintages	Bias	50% coverage	90% coverage
Confirmed cases, Ituri	99	-0.04	0.62	0.95
Confirmed cases, Nord-Kivu	99	0.02	0.57	0.95
Confirmed cases, Haut-Uele	99	0.11	0.74	0.94
Confirmed cases, Other provinces	99	0.03	0.9	0.98
Confirmed deaths, Ituri	99	0.01	0.67	0.93
Confirmed deaths, Nord-Kivu	99	-0.03	0.63	0.94
Confirmed deaths, Haut-Uele	99	0.03	0.79	0.97
Confirmed deaths, Other provinces	99	-0.01	0.91	0.98
Analysed specimens, Ituri	16	0.22	0.88	1
Analysed specimens, Nord-Kivu	16	-0.3	0.75	1
Analysed specimens, Haut-Uele	16	0.02	0.31	1
Analysed specimens, Other provinces	16	0.1	0.81	1

Province counts

Whether each province's cases and deaths are reproduced as counts once the national total is predicted as well.

Build the province count panels

julia

# The joint posterior predictive, shared with the national page (see
# `joint_posterior_predictive` in `docs/pages/_setup.jl`). Its draws pair
# one to one with `chn_joint`, whose shares split them.
pp_joint = joint_posterior_predictive();
_pp_draws(vn) = collect(pp_joint[BVDOutbreakSize.FlexiChains.Prefixed(vn)]);

# The national confirmed cases are replicated per laboratory window, oldest
# first, as on the national page. The first confirmed vintage is an
# unreplicated baseline.
_conf_windows = BVDOutbreakSize.confirmed_positivity_windows(
    obs.confirmed_history, obs.lab_history, obs.lab_daily_history
);
_conf_obs = collect(
    first(
        pp_joint[k] for k in keys(pp_joint)
            if occursin(
                "confirmed_state.confirmed_positives.positives", string(k)
            )
    )
);
_conf_replicates = [
    vcat(collect(e), collect(p), collect(l))
        for (e, p, l) in zip(
            vec(_pp_draws(@varname(early_increments.increments))),
            vec(_conf_obs),
            vec(_pp_draws(@varname(late_increments.increments)))
        )
];
province_case_count_panels = province_count_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    province_days = province_cases.days,
    national_days = vcat(
        _conf_windows.early_days, _conf_windows.obs_days,
        _conf_windows.late_days
    ),
    national_replicates = _conf_replicates,
    baseline = Int(first(obs.confirmed_history.counts)),
    stream = "Confirmed cases", n_patches = N_PATCHES
);
# The national confirmed deaths are replicated from zero at every vintage.
province_death_count_panels = province_count_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    province_days = province_deaths.days,
    national_days = obs.confirmed_deaths_history.days,
    national_replicates = _pp_draws(@varname(cdeath_increments.increments)),
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_count_calibration_table = stream_calibration(
    vcat(province_case_count_panels, province_death_count_panels)
);
province_count_calibration_fig = plot_stream_calibration(
    province_count_calibration_table
);

Province count calibration plot

Province count calibration table

Stream	Vintages	Bias	50% coverage	90% coverage
Confirmed cases, Ituri	99	0.06	0.7	0.96
Confirmed cases, Nord-Kivu	99	0.07	0.77	0.98
Confirmed cases, Haut-Uele	99	0.13	0.78	0.95
Confirmed cases, Other provinces	99	0.05	0.87	0.98
Confirmed deaths, Ituri	99	-0.06	0.66	0.92
Confirmed deaths, Nord-Kivu	99	-0.08	0.63	0.97
Confirmed deaths, Haut-Uele	99	-0.01	0.74	0.97
Confirmed deaths, Other provinces	99	-0.01	0.92	0.98

Province correlations and totals

Which province quantities trade off against each other, and whether each province's case and death totals agree with the observed ones.

Province posterior correlation heatmap

julia

# LaTeX label for one province's entry of a quantity. Spaces in a province
# name need escaping inside `\mathrm`.
_province_tex(sym, p) = string(
    sym, raw"\ (\mathrm{", replace(PROVINCE_LABELS[p], " " => raw"\ "), "})"
)
province_correlation_draws = NamedTuple(
    vcat(
        [:C_T => vec(Array(chn_joint[:C_T]))],
        _patch_columns(chn_joint, :R_T_patch, "R_T"),
        _patch_columns(chn_joint, :province_ascertainment, "asc"),
        _patch_columns(chn_joint, :CFR_patch, "CFR")
    )
);
province_correlation_labels = merge(
    Dict(:C_T => raw"C_T"),
    Dict(
        Symbol(name, "_", p) => _province_tex(tex, p)
            for p in 1:N_PATCHES
            for (name, tex) in (
                ("R_T", raw"R_T"), ("asc", raw"p_\mathrm{asc}"),
                ("CFR", raw"\mathrm{CFR}"),
            )
    )
);
province_correlation_fig = plot_correlation_heatmap(
    province_correlation_draws; labels = province_correlation_labels
);

Province totals against observed

julia

# Keyed by panel title, which names the stream and the province. A total
# that is the same in every draw (a patch with no deaths at any vintage) has
# no spread to plot and gives the corner plot a zero-width axis, so it is
# left out.
_varying_panels = filter(
    p -> length(unique(sum(r) for r in p.replicates)) > 1, province_panels
);
province_totals = NamedTuple(
    Symbol(p.title) => [sum(Float64.(r)) for r in p.replicates]
        for p in _varying_panels
);
province_observed = NamedTuple(
    Symbol(p.title) => Float64(sum(p.observed)) for p in _varying_panels
);
province_pairs_fig = plot_stream_pairs(province_totals, province_observed);

Province sensitivity to assumptions

Spatial structure sensitivity

The headline runs the model over four patches, one each for Ituri, Nord-Kivu and Haut-Uele and a fourth pooling the other affected provinces. Reducing it to a single patch collapses it onto one well-mixed population, which is the model the earlier releases used. The model overview on the methods page describes what the provinces add.

Splitting the country into provinces adds no national data, so the national outbreak size should not move far either way. The two are not identical by construction: the provinces run free and the country grows at the force-weighted mean of their reproduction numbers, which sits above the central trend they pool toward. That gap is first order in the deviation scale. A large gap between the two posteriors below would therefore point at the deviation priors rather than at the data.

julia

spatial_sensitivity_table = streams_table(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population (n_patches = 1)" => posterior_C_no_patches
);
spatial_sensitivity_table

2×7 DataFrame

Row	Stream	Lower 90%	Lower 60%	Lower 30%	Upper 30%	Upper 60%	Upper 90%
	String	Float64	Float64	Float64	Float64	Float64	Float64
1	Meta-population (headline)	12196.0	14801.0	16297.0	19667.0	21712.0	25514.0
2	Single population (n_patches = 1)	12404.0	14490.0	16064.0	19173.0	21460.0	26598.0

Spatial-structure density overlay

julia

spatial_sensitivity_fig = plot_density_overlay(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population" => posterior_C_no_patches;
    xlabel = "Cumulative infections"
);

The size is the gate, but it is not the only quantity the spatial structure could move. The three figures below set the national reproduction number, the case-fatality ratio and the reproduction number at the cut-off from the two fits against each other. Each is a national quantity that both models estimate, so the two posteriors should sit on top of each other. In the trajectory figure they do, closely enough that the grey reference is hidden behind the coloured band for most of the window. The table after them gives the same quantities as numbers, which is where the agreement is read rather than eyeballed.

National quantities under both structures

julia

spatial_rt_fig = plot_rt_streams(
    [
        (;
            label = "Single population (n_patches = 1)",
            chn = chn_no_patches, rt_start = _rt_start_plot,
            rt_walk_start = clamp(
                _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
                obs.n
            ), colour = :steelblue,
        ),
    ];
    joint = (;
        chn = chn_joint, rt_start = _rt_start_plot,
        rt_walk_start = clamp(
            _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
            obs.n
        ),
    ),
    n = obs.n, breakpoint = _BREAKPOINT,
    as_of_date = string(obs.cutoff), seeding = obs.seeding,
    display_start = _rt_start_plot, ncols = 1,
    title = "National reproduction number under both structures",
    reference_label = "the meta-population headline",
    panel_label = "the single-population fit"
);

spatial_cfr_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:CFR])),
    "Single population" => vec(Array(chn_no_patches[:CFR]));
    xlabel = "Case-fatality ratio"
);

spatial_rt_density_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:R_T])),
    "Single population" => vec(Array(chn_no_patches[:R_T]));
    xlabel = "Reproduction number at the cut-off"
);

The table gathers the same three quantities as credible intervals, alongside the outbreak start date the two fits imply.

National quantities under both structures, as a table

julia

# One row per national quantity, one column per structure, each cell a
# median with a 90% credible interval. Built here rather than by stacking
# two `summary_table` calls, so the two structures sit side by side and the
# reader compares along a row.
spatial_quantities_table = let
    # A count rounded to zero decimals still prints a trailing ".0", so
    # whole-number quantities go through `Int`.
    fmt(x, d) = d <= 0 ? string(round(Int, x)) : string(round(x; digits = d))
    cell(v, d) = string(
        fmt(quantile(v, 0.5), d), " (",
        fmt(quantile(v, 0.05), d), "–", fmt(quantile(v, 0.95), d), ")"
    )
    rows = [
        ("Cumulative infections", :C_T, 0),
        ("Reproduction number at the cut-off", :R_T, 2),
        ("Case-fatality ratio", :CFR, 2),
        ("Outbreak age (days)", :T, 0),
        ("Latest growth rate (per day)", :r, 3),
    ]
    DataFrame(
        "Quantity" => [r[1] for r in rows],
        "Meta-population (headline)" => [
            cell(
                vec(Array(chn_joint[r[2]])), r[3]
            )
                for r in rows
        ],
        "Single population" => [
            cell(vec(Array(chn_no_patches[r[2]])), r[3])
                for r in rows
        ]
    )
end;

5×3 DataFrame

Row	Quantity	Meta-population (headline)	Single population
	String	String	String
1	Cumulative infections	17916 (12196–25514)	17481 (12404–26598)
2	Reproduction number at the cut-off	0.8 (0.56–1.05)	0.78 (0.55–1.02)
3	Case-fatality ratio	0.49 (0.37–0.6)	0.5 (0.38–0.6)
4	Outbreak age (days)	197 (186–213)	196 (186–213)
5	Latest growth rate (per day)	-0.016 (-0.04–0.004)	-0.017 (-0.038–0.001)

Province parameter recovery

Whether the model recovers each province's values when fitted to data it simulated itself, from the same runs as the national parameter recovery. The top panel shows each seed's posterior median with its 50% and 90% intervals divided by that seed's true value. Below, each quantity by province is on its own scale with the prior in grey, each seed's posterior in its colour and its true value as a dashed line.

Recovery figure

julia

province_recovery_results = recovery_results()
_recovery_short = [
    "C_T_patch" => "C_T", "R_T_patch" => "R_T", "CFR_patch" => "CFR",
    "province_ascertainment" => "ascertainment",
    "region_drift_sd" => "drift SD",
]
province_recovery_quantities = [
    "$(k)[$(p)]" for (k, _) in _recovery_short for p in PROVINCE_LABELS
]
province_recovery_fig = isempty(province_recovery_results.params) ? nothing :
    plot_recovery(
        province_recovery_results.params, province_recovery_results.draws,
        province_recovery_results.prior;
        quantities = province_recovery_quantities,
        labels = Dict(
            "$(k)[$(p)]" => "$(v), $(p)" for (k, v) in _recovery_short
            for p in PROVINCE_LABELS
        ),
        panel_labels = Dict(
            "$(k)[$(p)]" => p for (k, _) in _recovery_short for p in PROVINCE_LABELS
        ),
        row_labels = last.(_recovery_short),
        log_x = ["C_T_patch[$(p)]" for p in PROVINCE_LABELS]
    );

The error of each seed's posterior median relative to the truth, and the z-score of the truth, summarised across seeds.

Summary table across seeds

julia

province_recovery_summary = isempty(province_recovery_results.params) ?
    DataFrame() :
    recovery_summary_table(province_recovery_results.params; province = true);

province_recovery_summary_display = isempty(province_recovery_summary) ?
    Markdown.parse("No parameter-recovery run is available for this build.") :
    MarkdownTable(province_recovery_summary);

Quantity	Truth, lowest	Truth, highest	Relative error, median	Relative error, range	z, median	Truth in 90%
CFR_patch[Haut-Uele]	0.173	0.255	0.07	-0.18 to 0.23	0.27	3/3
CFR_patch[Ituri]	0.153	0.288	-0.06	-0.14 to 0.17	-0.28	3/3
CFR_patch[Nord-Kivu]	0.158	0.27	0	-0.16 to 0.17	-0.1	3/3
CFR_patch[Other provinces]	0.146	0.242	0.12	-0.16 to 0.24	0.52	3/3
C_T_patch[Haut-Uele]	0.458	46.4	0.12	-0.04 to 0.14	0.43	3/3
C_T_patch[Ituri]	3280	12200	0.08	-0.34 to 0.23	0.49	2/3
C_T_patch[Nord-Kivu]	0.907	117	-0.02	-0.11 to 0.02	0.09	3/3
C_T_patch[Other provinces]	0.848	119	-0.04	-0.1 to 0.05	0.25	3/3
R_T_patch[Haut-Uele]	0.696	1.38	0.03	-0.2 to 0.27	0.34	3/3
R_T_patch[Ituri]	0.765	1.42	0.02	-0.2 to 0.1	0.29	2/3
R_T_patch[Nord-Kivu]	0.768	1.43	0.03	-0.19 to 0.06	0.3	3/3
R_T_patch[Other provinces]	0.813	1.31	-0.01	-0.15 to 0.04	0.03	3/3
province_ascertainment[Haut-Uele]	0.844	1.05	-0.02	-0.2 to -0.02	-0.03	3/3
province_ascertainment[Ituri]	0.944	1.11	0.09	-0.1 to 0.24	0.47	3/3
province_ascertainment[Nord-Kivu]	0.855	1.04	-0.02	-0.1 to -0.0	0.06	3/3
province_ascertainment[Other provinces]	0.985	1.25	0	-0.03 to 0.35	0.17	3/3
region_drift_sd[Haut-Uele]	0.00819	0.0967	0.36	-0.44 to 0.47	0.51	3/3
region_drift_sd[Ituri]	0.00414	0.0619	0.49	0.19 to 1.56	0.59	3/3
region_drift_sd[Nord-Kivu]	0.00432	0.0429	0	-0.1 to 1.47	0.26	3/3
region_drift_sd[Other provinces]	0.00988	0.0236	0.16	0.12 to 1.7	0.38	3/3

Each seed's recovered province values

julia

province_recovery = let r = province_recovery_results.params
    isempty(r) ? DataFrame() :
        r[
            occursin.("[", r.quantity), [
                :seed, :quantity, :truth, :median, :lower_90, :upper_90,
                :covered_90,
            ],
        ]
end;

province_recovery_display = isempty(province_recovery) ?
    Markdown.parse("No parameter-recovery run is available for this build.") : MarkdownTable(province_recovery);

seed	quantity	truth	median	lower_90	upper_90	covered_90
1	CFR_patch[Haut-Uele]	0.255	0.272	0.188	0.352	true
1	CFR_patch[Ituri]	0.288	0.269	0.174	0.369	true
1	CFR_patch[Nord-Kivu]	0.27	0.271	0.165	0.348	true
1	CFR_patch[Other provinces]	0.242	0.271	0.18	0.35	true
1	C_T_patch[Haut-Uele]	0.458	0.438	0.0843	1.634	true
1	C_T_patch[Ituri]	3283.156	3538.278	2600.94	5462.002	true
1	C_T_patch[Nord-Kivu]	0.907	0.93	0.199	3.237	true
1	C_T_patch[Other provinces]	0.848	0.811	0.17	2.86	true
1	R_T_patch[Haut-Uele]	0.696	0.881	0.532	1.411	true
1	R_T_patch[Ituri]	0.765	0.843	0.578	1.098	true
1	R_T_patch[Nord-Kivu]	0.768	0.815	0.492	1.238	true
1	R_T_patch[Other provinces]	0.813	0.842	0.451	1.216	true
1	province_ascertainment[Haut-Uele]	1.047	1.028	0.77	1.743	true
1	province_ascertainment[Ituri]	0.944	1.029	0.795	2.095	true
1	province_ascertainment[Nord-Kivu]	1.027	1.002	0.698	1.504	true
1	province_ascertainment[Other provinces]	0.985	0.953	0.345	1.184	true
1	region_drift_sd[Haut-Uele]	0.0967	0.0544	0.0125	0.119	true
1	region_drift_sd[Ituri]	0.0619	0.0735	0.0141	0.146	true
1	region_drift_sd[Nord-Kivu]	0.0429	0.0429	0.0146	0.115	true
1	region_drift_sd[Other provinces]	0.0199	0.0537	0.0148	0.136	true
2	CFR_patch[Haut-Uele]	0.173	0.213	0.148	0.318	true
2	CFR_patch[Ituri]	0.153	0.179	0.122	0.242	true
2	CFR_patch[Nord-Kivu]	0.158	0.184	0.13	0.249	true
2	CFR_patch[Other provinces]	0.146	0.181	0.127	0.241	true
2	C_T_patch[Haut-Uele]	46.44	52.146	31.537	89.111	true
2	C_T_patch[Ituri]	7207.671	4792.522	4085.586	6359.346	false
2	C_T_patch[Nord-Kivu]	116.645	113.924	77.485	175.711	true
2	C_T_patch[Other provinces]	119.014	107.106	73.163	162.585	true
2	R_T_patch[Haut-Uele]	1.248	1.29	1.075	1.595	true
2	R_T_patch[Ituri]	1.254	1.283	1.088	1.547	true
2	R_T_patch[Nord-Kivu]	1.251	1.283	1.068	1.574	true
2	R_T_patch[Other provinces]	1.28	1.273	1.055	1.56	true
2	province_ascertainment[Haut-Uele]	0.88	0.704	0.457	0.99	true
2	province_ascertainment[Ituri]	1.048	1.299	0.981	1.812	true
2	province_ascertainment[Nord-Kivu]	1.045	1.041	0.861	1.299	true
2	province_ascertainment[Other provinces]	1.037	1.04	0.865	1.322	true
2	region_drift_sd[Haut-Uele]	0.00819	0.012	0.000993	0.045	true
2	region_drift_sd[Ituri]	0.00414	0.0106	0.000986	0.0389	true
2	region_drift_sd[Nord-Kivu]	0.00432	0.0107	0.00096	0.0416	true
2	region_drift_sd[Other provinces]	0.00988	0.0111	0.000894	0.0414	true
3	CFR_patch[Haut-Uele]	0.192	0.157	0.0965	0.24	true
3	CFR_patch[Ituri]	0.189	0.163	0.0973	0.248	true
3	CFR_patch[Nord-Kivu]	0.197	0.166	0.103	0.252	true
3	CFR_patch[Other provinces]	0.188	0.157	0.0975	0.24	true
3	C_T_patch[Haut-Uele]	11.878	13.569	6.097	29.445	true
3	C_T_patch[Ituri]	12213.649	15079.068	9646.976	25885.321	true
3	C_T_patch[Nord-Kivu]	48.34	42.912	21.411	87.432	true
3	C_T_patch[Other provinces]	38.746	40.533	20.294	80.54	true
3	R_T_patch[Haut-Uele]	1.383	1.11	0.778	1.397	true
3	R_T_patch[Ituri]	1.416	1.14	0.908	1.366	false
3	R_T_patch[Nord-Kivu]	1.433	1.156	0.879	1.461	true
3	R_T_patch[Other provinces]	1.313	1.12	0.818	1.381	true
3	province_ascertainment[Haut-Uele]	0.844	0.83	0.459	1.245	true
3	province_ascertainment[Ituri]	1.106	0.99	0.666	1.509	true
3	province_ascertainment[Nord-Kivu]	0.855	0.77	0.451	1.088	true
3	province_ascertainment[Other provinces]	1.253	1.688	1.019	2.804	true
3	region_drift_sd[Haut-Uele]	0.0218	0.0296	0.00221	0.0948	true
3	region_drift_sd[Ituri]	0.0192	0.0287	0.0023	0.081	true
3	region_drift_sd[Nord-Kivu]	0.0288	0.0261	0.00247	0.0839	true
3	region_drift_sd[Other provinces]	0.0236	0.0274	0.00245	0.0821	true

Saving province in-sample outputs

Write the summary bullets

julia

# The bullets under the summary heading at the top of the page. They read
# tables built further down, so they are written here and read back when
# the site is assembled.
evaluation_insample_province_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    fitted = filter(r -> isfinite(r["Bias"]), province_calibration_table)
    worst = first(
        sort(fitted, "Bias"; by = abs, rev = true), min(3, size(fitted, 1))
    )
    function coverage_bullet(heading, tbl, scale)
        low = first(sort(tbl, "90% coverage"), min(1, size(tbl, 1)))
        return string(
            "- **", heading, ":** ", count(>=(0.8), tbl[!, "90% coverage"]),
            " of ", size(tbl, 1), " province streams have 90% coverage of ",
            "at least 0.8", scale,
            size(low, 1) == 0 ? "." :
                string(
                    "; the lowest is ", low[1, "Stream"], " at ",
                    fmt(low[1, "90% coverage"]), "."
                )
        )
    end
    overall = [
        string(
            "- **Least well reproduced:** ",
            join(
                [
                    string(
                        r["Stream"], " (bias ", fmt(r["Bias"]),
                        ", 90% coverage ", fmt(r["90% coverage"]), ")"
                    )
                        for r in eachrow(worst)
                ], "; "
            ), "."
        ),
        coverage_bullet("Coverage", fitted, ""),
        coverage_bullet(
            "Count-scale coverage",
            filter(
                r -> isfinite(r["Bias"]), province_count_calibration_table
            ),
            ""
        ),
    ]
    rows(tbl) = Dict(r["Stream"] => r for r in eachrow(tbl))
    cal = rows(province_calibration_table)
    count_cal = rows(province_count_calibration_table)
    function calibration(tbl, kind, p, label)
        r = get(tbl, string(kind, ", ", PROVINCE_LABELS[p]), nothing)
        r === nothing && return string("- ", label, ": not scored.")
        return string(
            "- ", label, ": bias ", fmt(r["Bias"]), " and 90% coverage ",
            fmt(r["90% coverage"]), " over ", r["Vintages"], " vintages."
        )
    end
    detail = [
        join(
            [
                string("**", PROVINCE_LABELS[p], "**"), "",
                calibration(cal, "Confirmed cases", p, "Confirmed cases"),
                calibration(cal, "Confirmed deaths", p, "Confirmed deaths"),
                calibration(
                    count_cal, "Confirmed cases", p,
                    "Confirmed cases as counts"
                ),
                calibration(
                    count_cal, "Confirmed deaths", p,
                    "Confirmed deaths as counts"
                ),
            ], "\n"
        )
            for p in 1:N_PATCHES
    ]
    join(vcat([join(overall, "\n")], detail), "\n\n")
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_insample_province.md"),
    evaluation_insample_province_summary
);

Province in-sample checks ​

Summary ​

Province prior predictive check ​

Province compositions ​

Province stream calibration ​

Province counts ​

Province correlations and totals ​

Province sensitivity to assumptions ​

Spatial structure sensitivity ​

Province parameter recovery ​

Saving province in-sample outputs ​

Province in-sample checks

Summary

Province prior predictive check

Province compositions

Province stream calibration

Province counts

Province correlations and totals

Province sensitivity to assumptions

Spatial structure sensitivity

Province parameter recovery

Saving province in-sample outputs