Sensitivity analyses

Data processing

Population normalisation

We scored forecasts and observations normalised by country population per 100,000. We also scored against the raw total incident weekly count per target. This had near-zero impact on model fitting or results on the log-transformed scores. Difference in median WIS (across all targets):

Table 1: Difference in median WIS between scoring on the raw count and per-100,000 population scales, by epidemiological outcome and scale.

Target	Scale	Raw count	Per 100k
Cases	log	0.292	0.278
Cases	natural	2426	34.4
Deaths	log	0.239	0.093
Deaths	natural	14.7	0.199

Natural scale WIS

We present results from scoring forecast error on the natural scale (difference between observation and prediction in count of case or death incidence), from which the WIS is then calculated and the analysis repeated.

Table 2: Participation and forecasts by model structure and epidemiological outcome, scored on the natural scale.

	Cases	Deaths	Both outcomes
	Models (%)	Models (%)	Models (%)	Participation (%)
Overall	43 (100%)	39 (100%)	48 (100%)	3%
Method
Semi-mechanistic	9 (20.9%)	10 (25.6%)	12 (25%)	4%
Statistical	12 (27.9%)	8 (20.5%)	13 (27.1%)	7%
Mechanistic	16 (37.2%)	16 (41%)	17 (35.4%)	3%
Agent-based	3 (7%)	2 (5.1%)	3 (6.2%)	3%
Judgement	3 (7%)	3 (7.7%)	3 (6.2%)	3%

Model specification

We tried several alternative model specifications.

Covariate selection

Table 3: Covariates considered across model versions, the version in which each was introduced, and their impact on model fit and estimates.

Domain	Variable	Model version	Impact
Forecast target
	EpiTarget	V2*	None
	VariantPhase	V2*	Some
	Trend	All	Untested
	Horizon	All	Untested
	Incidence	All	Untested
	Location	All	Untested
Forecaster strategy
	CountryTargets	All	Untested
	Model	All	Untested
Outcome	Method	All

* Covariates included in V2 were added based on reviewer feedback.

Error family and link

We first used a Gaussian distribution with a log link. Because LWIS is strongly right-skewed, this left high residual skewness (5.8) and kurtosis (77.6). We avoid using additional data transformations (e.g. an additional log transform) on the outcome (LWIS) as this would violate propriety of the score. We instead addressed the skew through the error family, comparing Gaussian, Gamma and Tweedie families as reported under Model fitting above, and adopting the Tweedie family for the reported analysis.

The fitted relationships and partial effects remained stable across parameterisations.

Structure by outcome under a Gaussian family

The structure-by-outcome effects are sensitive to the error family, in magnitude throughout and in direction for two of the five structures. The table below refits the same specification with a Gaussian family and log link, alongside the primary Tweedie fit. The case-versus-death contrast keeps its sign under the Gaussian family for judgement, mechanistic and statistical models. It does not for agent-based models, where the separation vanishes, or for semi-mechanistic models, where it reverses; in both the Gaussian contrast is close to zero. The Gaussian fit is dominated by the largest scores, which fall disproportionately among case forecasts, so it weights precisely the observations the Tweedie family is chosen to accommodate. Including or excluding the Hub baseline model makes no material difference to these estimates. We report the Tweedie fit throughout; the size of any single structure-by-outcome contrast should be read with this sensitivity in mind.

Table 4: Performance ratio by model structure and epidemiological outcome, under the primary Tweedie family and under a Gaussian family with a log link. A ratio below 1 indicates better-than-average performance.

Model structure	Outcome	Tweedie	Gaussian
Agent-based	Cases	1.09	0.99
	Deaths	0.87	1.00
Judgement	Cases	0.95	0.97
	Deaths	0.98	1.03
Mechanistic	Cases	0.95	0.98
	Deaths	1.04	1.00
Semi-mechanistic	Cases	1.06	1.06
	Deaths	1.10	0.97
Statistical	Cases	0.96	1.00
	Deaths	1.02	1.01