This is our evaluation of forecasting models and individual forecasters who have contributed forecasts of Covid-19 case and death numbers in Germany and Poland. These forecasts were submitted to the German and Polish Forecast Hub each week. Currently, we submit an unweighted mean ensemble to the Forecast Hub, but this will likely be changed in the future. We will transition to submitting forecasts to the European Forecast Hub in the future. The evaluations are our own and not authorised by the German Forecast Hub team. We cannot rule out mistakes and the analyses are subject to change.

Note that forecasts have now shifted to the European Forecast Hub. You can find our evaluation for these forecasts here.

If you have questions or want to give feedback, please create an issue on our github repository

You can register to become a forecaster here.



Forecaster ranking

Here is an overall ranking of all forecasters. The ranking is made according to relative skill. Relative skill is calculated by looking at all pairwise comparisons between forecasters in terms of the weighted interval score (WIS). See below for a more detailed explanation of the scoring metrics used. ‘Overall’ shows the complete ranking, ‘latest’ only spans the last 5-6 weeks of data. ‘Detailed’ represents the full data set that you can download for your own analysis.

overall

latest

Cases

TRUE

Deaths

TRUE

Germany

TRUE

Poland

TRUE

overall by horizon

TRUE

Detailed

TRUE

The following metrics are used:


Forecast visualisation

This is a visualisation of all forecasts made so far.

2021-04-05

Germany

case

death

Poland

case

death

2021-03-29

Germany

case

death

Poland

case

death

2021-03-22

Germany

case

death

Poland

case

death

2021-03-15

Germany

case

death

Poland

case

death

2021-03-08

Germany

case

death

Poland

case

death

2021-03-01

Germany

case

death

Poland

case

death

2021-02-22

Germany

case

death

Poland

case

death

2021-02-15

Germany

case

death

Poland

case

death

2021-02-08

Germany

case

death

Poland

case

death

2021-02-01

Germany

case

death

Poland

case

death

2021-01-25

Germany

case

death

Poland

case

death

2021-01-18

Germany

case

death

Poland

case

death

2021-01-11

Germany

case

death

Poland

case

death

2021-01-04

Germany

case

death

Poland

case

death

2020-12-28

Germany

case

death

Poland

case

death

2020-12-21

Germany

case

death

Poland

case

death

2020-12-14

Germany

case

death

Poland

case

death

2020-12-07

Germany

case

death

Poland

case

death

2020-11-30

Germany

case

death

Poland

case

death

2020-11-23

Germany

case

death

Poland

case

death

2020-11-16

Germany

case

death

Poland

case

death

2020-11-09

Germany

case

death

Poland

case

death

2020-11-02

Germany

case

death

Poland

case

death

2020-10-26

Germany

case

death

Poland

case

death

2020-10-19

Germany

case

death

Poland

case

death

2020-10-12

Germany

case

death

Poland

case

death


Performance and observations over time

Here you can see a visualisation of forecaster scores together next to the true observed values:


Ranks over time

This table shows you either your rank among all forecasters or the standardised rank. The standardised rank is computed as (100 - the forecaster percentile rank) among all forecasters for a given target and forecast date. What happens is basically this: Every forecaster gets assigned a rank (1 is the best and the worst equals the number of available forecasts for that date). This rank is then transformed to a scale from 1 to 100 such that 100 is best and 0 is worst. Ranks are determined based on the weighted interval scores.

model rank

standardised model rank


Weighted Interval Score Decomposition

The weighted interval score can be decomposed into three parts: sharpness (the amount of uncertainty around the forecast), overprediction and underprediction. This visualisation gives an impression of the distribution between these three forms of penalties for the different forecasters.


Models and available forecasts

The following graphic gives an overview of the forecasters and models analysed and the number of forecasts they contributed.

Most of the ‘models’ are human forecasters, but some are not: