Province in-sample checks

Whether the fitted joint model reproduces the per-province data it was fitted to. It also re-fits the joint as a single population to check the spatial structure. The national checks are on the national in-sample checks page.

Load packages, data and fitted chains

julia

# Shared setup: packages, observations and the fit registry. See
# `docs/pages/_setup.jl`.
using BVDOutbreakSize
include(joinpath(pkgdir(BVDOutbreakSize), "docs", "pages", "_setup.jl"))

validation_forecast_from (generic function with 1 method)

julia

# The fits this page reads, loaded from the cache here.
chn_joint = load_fit("joint")
# `sens_no_patches` is the headline with `n_patches = 1`, the check on the
# spatial structure.
chn_no_patches = load_fit("sens_no_patches")
posterior_C_joint = vec(Array(chn_joint[:C_T]))
posterior_C_no_patches = vec(Array(chn_no_patches[:C_T]));

Summary

Whether each province's cases and deaths are reproduced, from the checks further down this page. Bias runs from −1 to 1 and is negative when the model under-predicts, and coverage is the fraction of vintages inside the 90% predictive interval.

Least well reproduced: Analysed specimens, Nord-Kivu (bias -0.31, 90% coverage 1.0); Analysed specimens, Ituri (bias 0.23, 90% coverage 1.0); Confirmed cases, Haut-Uele (bias 0.11, 90% coverage 0.94).
Coverage: 12 of 12 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Ituri at 0.93.
Count-scale coverage: 8 of 8 province streams have 90% coverage of at least 0.8; the lowest is Confirmed deaths, Ituri at 0.92.

Ituri

Confirmed cases: bias -0.04 and 90% coverage 0.94 over 99 vintages.
Confirmed deaths: bias 0.01 and 90% coverage 0.93 over 99 vintages.
Confirmed cases as counts: bias 0.06 and 90% coverage 0.96 over 99 vintages.
Confirmed deaths as counts: bias -0.06 and 90% coverage 0.92 over 99 vintages.

Nord-Kivu

Confirmed cases: bias 0.03 and 90% coverage 0.95 over 99 vintages.
Confirmed deaths: bias -0.03 and 90% coverage 0.93 over 99 vintages.
Confirmed cases as counts: bias 0.07 and 90% coverage 0.99 over 99 vintages.
Confirmed deaths as counts: bias -0.08 and 90% coverage 0.97 over 99 vintages.

Haut-Uele

Confirmed cases: bias 0.11 and 90% coverage 0.94 over 99 vintages.
Confirmed deaths: bias 0.03 and 90% coverage 0.98 over 99 vintages.
Confirmed cases as counts: bias 0.13 and 90% coverage 0.95 over 99 vintages.
Confirmed deaths as counts: bias -0.01 and 90% coverage 0.97 over 99 vintages.

Other provinces

Confirmed cases: bias 0.03 and 90% coverage 0.98 over 99 vintages.
Confirmed deaths: bias -0.01 and 90% coverage 0.96 over 99 vintages.
Confirmed cases as counts: bias 0.05 and 90% coverage 0.98 over 99 vintages.
Confirmed deaths as counts: bias -0.01 and 90% coverage 0.98 over 99 vintages.

Province prior predictive check

Whether the four-patch prior, before any data are fitted, brackets each province's observed share of the confirmed cases and deaths.

Draw from the four-patch prior

julia

prior_patch_chn = patch_prior_draws(obs);

prior_province_table = patch_overview_table(prior_patch_chn, N_PATCHES);

Show prior province summary table

Province	Reproduction number	Cumulative infections	Share of infections (%)	Relative ascertainment
Ituri	1.01 (0.11–1.84)	324436 (1630–3882486)	88.1 (15.0–99.4)	1.0 (0.65–1.51)
Nord-Kivu	1.35 (0.17–2.44)	13100 (15–7118721)	4.2 (0.2–29.9)	1.0 (0.65–1.49)
Haut-Uele	1.34 (0.15–2.39)	5261 (6–1931361)	1.8 (0.1–10.1)	1.0 (0.67–1.48)
Other provinces	1.39 (0.19–2.47)	14067 (13–12309122)	4.7 (0.2–49.1)	1.0 (0.65–1.51)

Province composition prior predictive checks

julia

prior_case_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province, prior"
);

prior_death_ppc_fig = plot_province_composition_ppc(
    prior_patch_chn;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province, prior"
);

Province prior and posterior pair plot

julia

# Per-province entries of a vector deterministic, one draw vector each,
# keyed `<name>_<patch>` so they can sit side by side in one table.
function _patch_columns(chn, key::Symbol, name::AbstractString; f = identity)
    draws = [collect(v) for v in vec(collect(chn[key]))]
    return [
        Symbol(name, "_", p) => Float64[f(d[p]) for d in draws]
            for p in 1:N_PATCHES
    ]
end
_province_pair_draws(chn) = NamedTuple(
    vcat(
        _patch_columns(chn, :R_T_patch, "R_T"),
        _patch_columns(chn, :C_T_patch, "log10_C_T"; f = x -> log10(x + 1))
    )
)
_province_pair_labels = Dict(
    Symbol(name, "_", p) => string(label, ", ", PROVINCE_LABELS[p])
        for p in 1:N_PATCHES
        for (name, label) in (("R_T", "R_T"), ("log10_C_T", "log10 C_T"))
)
province_pair_fig = plot_pair(
    _province_pair_draws(chn_joint);
    prior = _province_pair_draws(prior_patch_chn),
    labels = _province_pair_labels
);

Province compositions

Whether each province's modelled share of the national confirmed cases and deaths reproduces the observed share at every spatial vintage (see the province compositions Methods section).

Province composition posterior predictive checks

julia

province_case_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    days = province_cases.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed case share by province"
);

province_death_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    days = province_deaths.days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Confirmed death share by province"
);

The three province terms for the isolation and laboratory data are checked the same way. The laboratory panel is the split of each calendar week's analysed specimens, which identifies the background split. The occupancy panel is the split of the patients in isolation among the provinces printed that day, on the weekly days the fit scores, over the per-patch bed demand. The bed panel is the split of the beds among the provinces printed that day, on the days a count changed, over each patch's modelled share of the national capacity. A gap in a panel is a day on which that province printed nothing.

Province laboratory, occupancy and bed split posterior predictive checks

julia

province_lab_bin_days = [
    maximum(province_lab.days[province_lab.bins .== b])
        for b in 1:maximum(province_lab.bins)
];

province_lab_ppc_fig = plot_province_composition_ppc(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    days = province_lab_bin_days, seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Analysed specimen share by province, by week"
);

province_occupancy_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_occupancy_share, rows = province_isolation,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation occupancy share by province"
);

province_beds_ppc_fig = plot_province_split_ppc(
    chn_joint;
    share_key = :province_capacity_share, rows = province_capacity,
    seeding = obs.seeding, n_patches = N_PATCHES,
    title = "Isolation bed share by province"
);

Province stream calibration

Whether each province's share predictions are calibrated at the observed national total, scored as in the national stream calibration. The analysed-specimen rows score the weekly laboratory split the same way, at each week's observed national analysed total.

Build the province calibration panels

julia

province_case_panels = province_composition_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    stream = "Confirmed cases", n_patches = N_PATCHES
);
province_death_panels = province_composition_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_lab_panels = province_composition_panels(
    chn_joint;
    share_key = :province_lab_shares,
    obs_increments = province_lab.increments,
    stream = "Analysed specimens", n_patches = N_PATCHES
);
province_panels = vcat(
    province_case_panels, province_death_panels, province_lab_panels
);
province_calibration_table = stream_calibration(province_panels);
province_calibration_fig = plot_stream_calibration(
    province_calibration_table
);

Province calibration table

Stream	Vintages	Bias	50% coverage	90% coverage
Confirmed cases, Ituri	99	-0.04	0.61	0.94
Confirmed cases, Nord-Kivu	99	0.03	0.56	0.95
Confirmed cases, Haut-Uele	99	0.11	0.75	0.94
Confirmed cases, Other provinces	99	0.03	0.9	0.98
Confirmed deaths, Ituri	99	0.01	0.67	0.93
Confirmed deaths, Nord-Kivu	99	-0.03	0.62	0.93
Confirmed deaths, Haut-Uele	99	0.03	0.8	0.98
Confirmed deaths, Other provinces	99	-0.01	0.91	0.96
Analysed specimens, Ituri	16	0.23	0.88	1
Analysed specimens, Nord-Kivu	16	-0.31	0.75	1
Analysed specimens, Haut-Uele	16	0.01	0.25	1
Analysed specimens, Other provinces	16	0.08	0.75	1

Province counts

Whether each province's cases and deaths are reproduced as counts once the national total is predicted as well.

Build the province count panels

julia

# The joint posterior predictive, shared with the national page (see
# `joint_posterior_predictive` in `docs/pages/_setup.jl`). Its draws pair
# one to one with `chn_joint`, whose shares split them.
pp_joint = joint_posterior_predictive();
_pp_draws(vn) = collect(pp_joint[BVDOutbreakSize.FlexiChains.Prefixed(vn)]);

# The national confirmed cases are replicated per laboratory window, oldest
# first, as on the national page. The first confirmed vintage is an
# unreplicated baseline.
_conf_windows = BVDOutbreakSize.confirmed_positivity_windows(
    obs.confirmed_history, obs.lab_history, obs.lab_daily_history
);
_conf_obs = collect(
    first(
        pp_joint[k] for k in keys(pp_joint)
            if occursin(
                "confirmed_state.confirmed_positives.positives", string(k)
            )
    )
);
_conf_replicates = [
    vcat(collect(e), collect(p), collect(l))
        for (e, p, l) in zip(
            vec(_pp_draws(@varname(early_increments.increments))),
            vec(_conf_obs),
            vec(_pp_draws(@varname(late_increments.increments)))
        )
];
province_case_count_panels = province_count_panels(
    chn_joint;
    share_key = :province_shares,
    obs_increments = province_cases.increments,
    province_days = province_cases.days,
    national_days = vcat(
        _conf_windows.early_days, _conf_windows.obs_days,
        _conf_windows.late_days
    ),
    national_replicates = _conf_replicates,
    baseline = Int(first(obs.confirmed_history.counts)),
    stream = "Confirmed cases", n_patches = N_PATCHES
);
# The national confirmed deaths are replicated from zero at every vintage.
province_death_count_panels = province_count_panels(
    chn_joint;
    share_key = :province_death_shares,
    obs_increments = province_deaths.increments,
    province_days = province_deaths.days,
    national_days = obs.confirmed_deaths_history.days,
    national_replicates = _pp_draws(@varname(cdeath_increments.increments)),
    stream = "Confirmed deaths", n_patches = N_PATCHES
);
province_count_calibration_table = stream_calibration(
    vcat(province_case_count_panels, province_death_count_panels)
);
province_count_calibration_fig = plot_stream_calibration(
    province_count_calibration_table
);

Province count calibration plot

Province count calibration table

Stream	Vintages	Bias	50% coverage	90% coverage
Confirmed cases, Ituri	99	0.06	0.7	0.96
Confirmed cases, Nord-Kivu	99	0.07	0.76	0.99
Confirmed cases, Haut-Uele	99	0.13	0.76	0.95
Confirmed cases, Other provinces	99	0.05	0.87	0.98
Confirmed deaths, Ituri	99	-0.06	0.68	0.92
Confirmed deaths, Nord-Kivu	99	-0.08	0.61	0.97
Confirmed deaths, Haut-Uele	99	-0.01	0.76	0.97
Confirmed deaths, Other provinces	99	-0.01	0.92	0.98

Province correlations and totals

Which province quantities trade off against each other, and whether each province's case and death totals agree with the observed ones.

Province posterior correlation heatmap

julia

# LaTeX label for one province's entry of a quantity. Spaces in a province
# name need escaping inside `\mathrm`.
_province_tex(sym, p) = string(
    sym, raw"\ (\mathrm{", replace(PROVINCE_LABELS[p], " " => raw"\ "), "})"
)
province_correlation_draws = NamedTuple(
    vcat(
        [:C_T => vec(Array(chn_joint[:C_T]))],
        _patch_columns(chn_joint, :R_T_patch, "R_T"),
        _patch_columns(chn_joint, :province_ascertainment, "asc"),
        _patch_columns(chn_joint, :CFR_patch, "CFR")
    )
);
province_correlation_labels = merge(
    Dict(:C_T => raw"C_T"),
    Dict(
        Symbol(name, "_", p) => _province_tex(tex, p)
            for p in 1:N_PATCHES
            for (name, tex) in (
                ("R_T", raw"R_T"), ("asc", raw"p_\mathrm{asc}"),
                ("CFR", raw"\mathrm{CFR}"),
            )
    )
);
province_correlation_fig = plot_correlation_heatmap(
    province_correlation_draws; labels = province_correlation_labels
);

Province totals against observed

julia

# Keyed by panel title, which names the stream and the province. A total
# that is the same in every draw (a patch with no deaths at any vintage) has
# no spread to plot and gives the corner plot a zero-width axis, so it is
# left out.
_varying_panels = filter(
    p -> length(unique(sum(r) for r in p.replicates)) > 1, province_panels
);
province_totals = NamedTuple(
    Symbol(p.title) => [sum(Float64.(r)) for r in p.replicates]
        for p in _varying_panels
);
province_observed = NamedTuple(
    Symbol(p.title) => Float64(sum(p.observed)) for p in _varying_panels
);
province_pairs_fig = plot_stream_pairs(province_totals, province_observed);

Province sensitivity to assumptions

Spatial structure sensitivity

The headline runs the model over four patches, one each for Ituri, Nord-Kivu and Haut-Uele and a fourth pooling the other affected provinces. Reducing it to a single patch collapses it onto one well-mixed population, which is the model the earlier releases used. The model overview on the methods page describes what the provinces add.

Splitting the country into provinces adds no national data, so the national outbreak size should not move far either way. The two are not identical by construction: the provinces run free and the country grows at the force-weighted mean of their reproduction numbers, which sits above the central trend they pool toward. That gap is first order in the deviation scale. A large gap between the two posteriors below would therefore point at the deviation priors rather than at the data.

julia

spatial_sensitivity_table = streams_table(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population (n_patches = 1)" => posterior_C_no_patches
);
spatial_sensitivity_table

2×7 DataFrame

Row	Stream	Lower 90%	Lower 60%	Lower 30%	Upper 30%	Upper 60%	Upper 90%
	String	Float64	Float64	Float64	Float64	Float64	Float64
1	Meta-population (headline)	12371.0	14885.0	16574.0	19815.0	21957.0	26103.0
2	Single population (n_patches = 1)	11973.0	14504.0	16190.0	19412.0	21419.0	25866.0

Spatial-structure density overlay

julia

spatial_sensitivity_fig = plot_density_overlay(
    "Meta-population (headline)" => posterior_C_joint,
    "Single population" => posterior_C_no_patches;
    xlabel = "Cumulative infections"
);

The size is the gate, but it is not the only quantity the spatial structure could move. The three figures below set the national reproduction number, the case-fatality ratio and the reproduction number at the cut-off from the two fits against each other. Each is a national quantity that both models estimate, so the two posteriors should sit on top of each other. In the trajectory figure they do, closely enough that the grey reference is hidden behind the coloured band for most of the window. The table after them gives the same quantities as numbers, which is where the agreement is read rather than eyeballed.

National quantities under both structures

julia

spatial_rt_fig = plot_rt_streams(
    [
        (;
            label = "Single population (n_patches = 1)",
            chn = chn_no_patches, rt_start = _rt_start_plot,
            rt_walk_start = clamp(
                _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
                obs.n
            ), colour = :steelblue,
        ),
    ];
    joint = (;
        chn = chn_joint, rt_start = _rt_start_plot,
        rt_walk_start = clamp(
            _BREAKPOINT - RT_WALK_LEAD, _rt_start_plot,
            obs.n
        ),
    ),
    n = obs.n, breakpoint = _BREAKPOINT,
    as_of_date = string(obs.cutoff), seeding = obs.seeding,
    display_start = _rt_start_plot, ncols = 1,
    title = "National reproduction number under both structures",
    reference_label = "the meta-population headline",
    panel_label = "the single-population fit"
);

spatial_cfr_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:CFR])),
    "Single population" => vec(Array(chn_no_patches[:CFR]));
    xlabel = "Case-fatality ratio"
);

spatial_rt_density_fig = plot_density_overlay(
    "Meta-population (headline)" => vec(Array(chn_joint[:R_T])),
    "Single population" => vec(Array(chn_no_patches[:R_T]));
    xlabel = "Reproduction number at the cut-off"
);

The table gathers the same three quantities as credible intervals, alongside the outbreak start date the two fits imply.

National quantities under both structures, as a table

julia

# One row per national quantity, one column per structure, each cell a
# median with a 90% credible interval. Built here rather than by stacking
# two `summary_table` calls, so the two structures sit side by side and the
# reader compares along a row.
spatial_quantities_table = let
    # A count rounded to zero decimals still prints a trailing ".0", so
    # whole-number quantities go through `Int`.
    fmt(x, d) = d <= 0 ? string(round(Int, x)) : string(round(x; digits = d))
    cell(v, d) = string(
        fmt(quantile(v, 0.5), d), " (",
        fmt(quantile(v, 0.05), d), "–", fmt(quantile(v, 0.95), d), ")"
    )
    rows = [
        ("Cumulative infections", :C_T, 0),
        ("Reproduction number at the cut-off", :R_T, 2),
        ("Case-fatality ratio", :CFR, 2),
        ("Outbreak age (days)", :T, 0),
        ("Latest growth rate (per day)", :r, 3),
    ]
    DataFrame(
        "Quantity" => [r[1] for r in rows],
        "Meta-population (headline)" => [
            cell(
                vec(Array(chn_joint[r[2]])), r[3]
            )
                for r in rows
        ],
        "Single population" => [
            cell(vec(Array(chn_no_patches[r[2]])), r[3])
                for r in rows
        ]
    )
end;

5×3 DataFrame

Row	Quantity	Meta-population (headline)	Single population
	String	String	String
1	Cumulative infections	18127 (12371–26103)	17719 (11973–25866)
2	Reproduction number at the cut-off	0.8 (0.55–1.06)	0.78 (0.56–1.02)
3	Case-fatality ratio	0.48 (0.37–0.59)	0.5 (0.39–0.62)
4	Outbreak age (days)	195 (184–211)	197 (186–213)
5	Latest growth rate (per day)	-0.016 (-0.039–0.004)	-0.017 (-0.037–0.001)

Province parameter recovery

Whether the model recovers each province's values when fitted to data it simulated itself, from the same runs as the national parameter recovery. The top panel shows each seed's posterior median with its 50% and 90% intervals divided by that seed's true value. Below, each quantity by province is on its own scale with the prior in grey, each seed's posterior in its colour and its true value as a dashed line.

Recovery figure

julia

province_recovery_results = recovery_results()
_recovery_short = [
    "C_T_patch" => "C_T", "R_T_patch" => "R_T", "CFR_patch" => "CFR",
    "province_ascertainment" => "ascertainment",
    "region_drift_sd" => "drift SD",
]
province_recovery_quantities = [
    "$(k)[$(p)]" for (k, _) in _recovery_short for p in PROVINCE_LABELS
]
province_recovery_fig = isempty(province_recovery_results.params) ? nothing :
    plot_recovery(
        province_recovery_results.params, province_recovery_results.draws,
        province_recovery_results.prior;
        quantities = province_recovery_quantities,
        labels = Dict(
            "$(k)[$(p)]" => "$(v), $(p)" for (k, v) in _recovery_short
            for p in PROVINCE_LABELS
        ),
        panel_labels = Dict(
            "$(k)[$(p)]" => p for (k, _) in _recovery_short for p in PROVINCE_LABELS
        ),
        row_labels = last.(_recovery_short),
        log_x = ["C_T_patch[$(p)]" for p in PROVINCE_LABELS]
    );

The error of each seed's posterior median relative to the truth, and the z-score of the truth, summarised across seeds.

Summary table across seeds

julia

province_recovery_summary = isempty(province_recovery_results.params) ?
    DataFrame() :
    recovery_summary_table(province_recovery_results.params; province = true);

province_recovery_summary_display = isempty(province_recovery_summary) ?
    Markdown.parse("No parameter-recovery run is available for this build.") :
    MarkdownTable(province_recovery_summary);

Quantity	Truth, lowest	Truth, highest	Relative error, median	Relative error, range	z, median	Truth in 90%
CFR_patch[Haut-Uele]	0.173	0.212	0.23	-0.16 to 0.34	0.88	3/3
CFR_patch[Ituri]	0.153	0.276	-0.01	-0.12 to 0.15	-0.02	3/3
CFR_patch[Nord-Kivu]	0.158	0.288	-0.01	-0.14 to 0.17	-0.02	3/3
CFR_patch[Other provinces]	0.146	0.285	-0.02	-0.15 to 0.23	0.01	3/3
C_T_patch[Haut-Uele]	0.458	46.4	0.08	-0.2 to 0.12	0.38	3/3
C_T_patch[Ituri]	3280	12200	0.03	-0.34 to 0.2	0.27	2/3
C_T_patch[Nord-Kivu]	0.907	117	-0.03	-0.14 to 0.12	0.01	3/3
C_T_patch[Other provinces]	0.848	119	0.02	-0.11 to 0.14	0.25	3/3
R_T_patch[Haut-Uele]	0.696	1.38	0.03	-0.19 to 0.18	0.33	3/3
R_T_patch[Ituri]	0.765	1.42	0.02	-0.19 to 0.12	0.29	2/3
R_T_patch[Nord-Kivu]	0.768	1.43	0.02	-0.19 to 0.1	0.29	3/3
R_T_patch[Other provinces]	0.813	1.31	-0.01	-0.15 to 0.04	0.05	3/3
province_ascertainment[Haut-Uele]	0.844	1.05	-0.02	-0.17 to 0.0	-0.06	3/3
province_ascertainment[Ituri]	0.944	1.11	0.06	-0.1 to 0.19	0.31	3/3
province_ascertainment[Nord-Kivu]	0.855	1.04	-0.05	-0.11 to -0.01	-0.43	3/3
province_ascertainment[Other provinces]	0.985	1.25	0.01	-0.01 to 0.34	0.1	3/3
region_drift_sd[Haut-Uele]	0.00819	0.0967	0.35	-0.67 to 0.4	0.49	2/3
region_drift_sd[Ituri]	0.00414	0.0619	0.6	-0.48 to 1.55	0.61	3/3
region_drift_sd[Nord-Kivu]	0.00432	0.0429	-0.07	-0.29 to 1.33	0.17	3/3
region_drift_sd[Other provinces]	0.00988	0.0236	0.18	0.07 to 0.69	0.4	3/3

Each seed's recovered province values

julia

province_recovery = let r = province_recovery_results.params
    isempty(r) ? DataFrame() :
        r[
            occursin.("[", r.quantity), [
                :seed, :quantity, :truth, :median, :lower_90, :upper_90,
                :covered_90,
            ],
        ]
end;

province_recovery_display = isempty(province_recovery) ?
    Markdown.parse("No parameter-recovery run is available for this build.") : MarkdownTable(province_recovery);

seed	quantity	truth	median	lower_90	upper_90	covered_90
1	CFR_patch[Haut-Uele]	0.212	0.283	0.204	0.361	true
1	CFR_patch[Ituri]	0.276	0.274	0.204	0.359	true
1	CFR_patch[Nord-Kivu]	0.288	0.284	0.214	0.373	true
1	CFR_patch[Other provinces]	0.285	0.28	0.204	0.383	true
1	C_T_patch[Haut-Uele]	0.458	0.367	0.0927	1.294	true
1	C_T_patch[Ituri]	3283.156	3374.994	2515.863	4688.509	true
1	C_T_patch[Nord-Kivu]	0.907	1.014	0.272	2.635	true
1	C_T_patch[Other provinces]	0.848	0.97	0.228	3.333	true
1	R_T_patch[Haut-Uele]	0.696	0.82	0.576	1.029	true
1	R_T_patch[Ituri]	0.765	0.855	0.649	1.038	true
1	R_T_patch[Nord-Kivu]	0.768	0.847	0.609	1.057	true
1	R_T_patch[Other provinces]	0.813	0.849	0.587	1.075	true
1	province_ascertainment[Haut-Uele]	1.047	1.048	0.908	1.744	true
1	province_ascertainment[Ituri]	0.944	0.999	0.714	1.536	true
1	province_ascertainment[Nord-Kivu]	1.027	0.974	0.585	1.121	true
1	province_ascertainment[Other provinces]	0.985	0.99	0.697	1.284	true
1	region_drift_sd[Haut-Uele]	0.0967	0.0315	0.00748	0.0697	false
1	region_drift_sd[Ituri]	0.0619	0.0324	0.0085	0.0661	true
1	region_drift_sd[Nord-Kivu]	0.0429	0.0305	0.00652	0.0708	true
1	region_drift_sd[Other provinces]	0.0199	0.0336	0.00641	0.0786	true
2	CFR_patch[Haut-Uele]	0.173	0.213	0.148	0.323	true
2	CFR_patch[Ituri]	0.153	0.176	0.119	0.239	true
2	CFR_patch[Nord-Kivu]	0.158	0.184	0.13	0.247	true
2	CFR_patch[Other provinces]	0.146	0.18	0.123	0.241	true
2	C_T_patch[Haut-Uele]	46.44	50.35	30.062	88.576	true
2	C_T_patch[Ituri]	7207.671	4788.46	4127.482	6326.404	false
2	C_T_patch[Nord-Kivu]	116.645	112.786	74.589	172.218	true
2	C_T_patch[Other provinces]	119.014	105.342	71.234	166.288	true
2	R_T_patch[Haut-Uele]	1.248	1.281	1.074	1.583	true
2	R_T_patch[Ituri]	1.254	1.28	1.087	1.549	true
2	R_T_patch[Nord-Kivu]	1.251	1.28	1.074	1.572	true
2	R_T_patch[Other provinces]	1.28	1.271	1.063	1.561	true
2	province_ascertainment[Haut-Uele]	0.88	0.731	0.47	1.004	true
2	province_ascertainment[Ituri]	1.048	1.25	0.958	1.817	true
2	province_ascertainment[Nord-Kivu]	1.045	1.035	0.856	1.293	true
2	province_ascertainment[Other provinces]	1.037	1.032	0.841	1.316	true
2	region_drift_sd[Haut-Uele]	0.00819	0.0111	0.000997	0.0448	true
2	region_drift_sd[Ituri]	0.00414	0.0106	0.000907	0.0399	true
2	region_drift_sd[Nord-Kivu]	0.00432	0.0101	0.000888	0.0424	true
2	region_drift_sd[Other provinces]	0.00988	0.0106	0.00104	0.0429	true
3	CFR_patch[Haut-Uele]	0.192	0.162	0.0982	0.247	true
3	CFR_patch[Ituri]	0.189	0.166	0.102	0.257	true
3	CFR_patch[Nord-Kivu]	0.197	0.169	0.102	0.259	true
3	CFR_patch[Other provinces]	0.188	0.16	0.0972	0.245	true
3	C_T_patch[Haut-Uele]	11.878	13.331	6.203	28.892	true
3	C_T_patch[Ituri]	12213.649	14663.326	9624.377	25333.56	true
3	C_T_patch[Nord-Kivu]	48.34	41.528	21.844	87.393	true
3	C_T_patch[Other provinces]	38.746	39.405	19.96	83.386	true
3	R_T_patch[Haut-Uele]	1.383	1.122	0.795	1.418	true
3	R_T_patch[Ituri]	1.416	1.147	0.926	1.371	false
3	R_T_patch[Nord-Kivu]	1.433	1.16	0.888	1.475	true
3	R_T_patch[Other provinces]	1.313	1.12	0.813	1.379	true
3	province_ascertainment[Haut-Uele]	0.844	0.83	0.451	1.216	true
3	province_ascertainment[Ituri]	1.106	0.994	0.665	1.531	true
3	province_ascertainment[Nord-Kivu]	0.855	0.763	0.448	1.095	true
3	province_ascertainment[Other provinces]	1.253	1.675	1.043	2.763	true
3	region_drift_sd[Haut-Uele]	0.0218	0.0306	0.00357	0.09	true
3	region_drift_sd[Ituri]	0.0192	0.0308	0.00292	0.0845	true
3	region_drift_sd[Nord-Kivu]	0.0288	0.0267	0.00289	0.084	true
3	region_drift_sd[Other provinces]	0.0236	0.0278	0.00268	0.0893	true

Saving province in-sample outputs

Write the summary bullets

julia

# The bullets under the summary heading at the top of the page. They read
# tables built further down, so they are written here and read back when
# the site is assembled.
evaluation_insample_province_summary = let
    fmt(x) = ismissing(x) || !isfinite(x) ? "n/a" :
        string(round(x; digits = 2))
    fitted = filter(r -> isfinite(r["Bias"]), province_calibration_table)
    worst = first(
        sort(fitted, "Bias"; by = abs, rev = true), min(3, size(fitted, 1))
    )
    function coverage_bullet(heading, tbl, scale)
        low = first(sort(tbl, "90% coverage"), min(1, size(tbl, 1)))
        return string(
            "- **", heading, ":** ", count(>=(0.8), tbl[!, "90% coverage"]),
            " of ", size(tbl, 1), " province streams have 90% coverage of ",
            "at least 0.8", scale,
            size(low, 1) == 0 ? "." :
                string(
                    "; the lowest is ", low[1, "Stream"], " at ",
                    fmt(low[1, "90% coverage"]), "."
                )
        )
    end
    overall = [
        string(
            "- **Least well reproduced:** ",
            join(
                [
                    string(
                        r["Stream"], " (bias ", fmt(r["Bias"]),
                        ", 90% coverage ", fmt(r["90% coverage"]), ")"
                    )
                        for r in eachrow(worst)
                ], "; "
            ), "."
        ),
        coverage_bullet("Coverage", fitted, ""),
        coverage_bullet(
            "Count-scale coverage",
            filter(
                r -> isfinite(r["Bias"]), province_count_calibration_table
            ),
            ""
        ),
    ]
    rows(tbl) = Dict(r["Stream"] => r for r in eachrow(tbl))
    cal = rows(province_calibration_table)
    count_cal = rows(province_count_calibration_table)
    function calibration(tbl, kind, p, label)
        r = get(tbl, string(kind, ", ", PROVINCE_LABELS[p]), nothing)
        r === nothing && return string("- ", label, ": not scored.")
        return string(
            "- ", label, ": bias ", fmt(r["Bias"]), " and 90% coverage ",
            fmt(r["90% coverage"]), " over ", r["Vintages"], " vintages."
        )
    end
    detail = [
        join(
            [
                string("**", PROVINCE_LABELS[p], "**"), "",
                calibration(cal, "Confirmed cases", p, "Confirmed cases"),
                calibration(cal, "Confirmed deaths", p, "Confirmed deaths"),
                calibration(
                    count_cal, "Confirmed cases", p,
                    "Confirmed cases as counts"
                ),
                calibration(
                    count_cal, "Confirmed deaths", p,
                    "Confirmed deaths as counts"
                ),
            ], "\n"
        )
            for p in 1:N_PATCHES
    ]
    join(vcat([join(overall, "\n")], detail), "\n\n")
end
dashboard_dir = joinpath(
    pkgdir(BVDOutbreakSize), "docs", "src", "summary_assets"
)
mkpath(dashboard_dir)
write(
    joinpath(dashboard_dir, "evaluation_insample_province.md"),
    evaluation_insample_province_summary
);

Province in-sample checks ​

Summary ​

Province prior predictive check ​

Province compositions ​

Province stream calibration ​

Province counts ​

Province correlations and totals ​

Province sensitivity to assumptions ​

Spatial structure sensitivity ​

Province parameter recovery ​

Saving province in-sample outputs ​

Province in-sample checks

Summary

Province prior predictive check

Province compositions

Province stream calibration

Province counts

Province correlations and totals

Province sensitivity to assumptions

Spatial structure sensitivity

Province parameter recovery

Saving province in-sample outputs