Evaluating the model

Having built a model to forecast the future progression of coronavirus in different geographical locations, and calibrated assumptions to match the UK experience, I regularly check the forecasts against reported figures.

In the UK there are several different sources for coronavirus related data. The headline figures (those with greatest media focus) are published alongside the government’s daily press conference. Slightly unhelpfully, the government changed their definition of deaths on 29 April 2020 – moving from UK hospital deaths (although, peculiarly, the definition also included non-hospital deaths outside England) to “all-settings” deaths across the UK. The change to “all-settings” does make sense because it encompasses more COVID-19 deaths.

However, there is a further set of UK deaths figures, from the Office of National Statistics, these provide an even more complete picture of COVID-19 deaths. Unlike the Public Heath England deaths, used in the government’s press briefings, which only include positive tests, the ONS figures include all deaths where COVID-19 is mentioned on the death certificate (as described in this well written explanation). The ONS figures are published weekly, and there is a time lag, but they are the probably the best data source.

The government have been including the ONS figures, weekly, labelled as “all settings” deaths in their press briefings. But, confusingly, they now call the PHE data “all settings” deaths – even though the definitions are different.

A further issue with deaths reported at the UK press briefings is that they are shown for the day that the death is reported. This can be helpful for quickly seeing the trend in deaths – but isn’t ideal for analysis, because a reported death may have occurred weeks beforehand. Date of death data is preferable for analysis – but deaths from the most recent days are unlikely to be fully reported. NHS England provide date of death data.

I’ve compared my model against the different data sources to establish its accuracy as follows:

  1. Comparison against government reported figures 6 May 2020

daily reported forecast

The first graph shows my model’s daily forecast against the government’s reported “all settings” figures. The reported figures are volatile from day to day, so it’s not easy to see the full trend.

7 day average forecast

Using a 7 day rolling average better illustrates the trend

total reported forecast

The graph above shows the cumulative reported death figures compared to the forecast. The model is tracking above the reported figures. However, I’ve also included the weekly ONS data on this graph (the black dots), which has a wider definition of COVID-19 deaths, the last of which (17 April 2020) was above both my model and the reported all-settings data.

2. NHS data of death data

NHS England publish date of death data for hospital deaths in England. I’ve adapted my model for England (rather than UK) by adjusting the starting population to 56 million, and reducing the number of imported cases – to produce the following comparison:

NHS daily deathsNHS total deaths

These show hospital only deaths, without the reporting lag, from my model to compare with date-of-death figures. Deaths from most recent days wont yet be reported, which is one reason why the model produces higher figures for most recent dates.

I set out the actual numbers for these graphs, and future forecasts, below. These all assume the current UK full lockdown continues:

Date Total deaths model (all settings) Total reported deaths (PHE all settings) Total reported deaths ONS (all settings) Daily deaths model (all deaths) Daily reported (PHE all settings)
08/03/2020 19 2 4 0
09/03/2020 25 3 6 1
10/03/2020 32 7 7 4
11/03/2020 41 7 9 0
12/03/2020 52 9 11 2
13/03/2020 65 10 13 1
14/03/2020 82 28 17 18
15/03/2020 102 43 20 15
16/03/2020 127 65 25 22
17/03/2020 158 81 31 16
18/03/2020 196 115 38 34
19/03/2020 242 158 46 43
20/03/2020 298 194 56 36
21/03/2020 367 250 69 56
22/03/2020 451 285 84 35
23/03/2020 554 359 103 74
24/03/2020 679 508 125 149
25/03/2020 832 694 153 186
26/03/2020 1018 877 186 183
27/03/2020 1245 1161 226 284
28/03/2020 1518 1455 274 294
29/03/2020 1847 1669 329 214
30/03/2020 2240 2043 393 374
31/03/2020 2705 2425 464 382
01/04/2020 3246 3095 542 670
02/04/2020 3869 3747 623 652
03/04/2020 4573 4461 4526 704 714
04/04/2020 5356 5221 783 760
05/04/2020 6210 5865 854 644
06/04/2020 7125 6433 916 568
07/04/2020 8090 7471 965 1038
08/04/2020 9090 8505 1000 1034
09/04/2020 10112 9608 1022 1103
10/04/2020 11144 10760 11413 1032 1152
11/04/2020 12175 11599 1031 839
12/04/2020 13197 12285 1022 686
13/04/2020 14203 13029 1006 744
14/04/2020 15190 14073 987 1044
15/04/2020 16156 14915 965 842
16/04/2020 17099 15944 943 1029
17/04/2020 18018 16879 20909 920 935
18/04/2020 18916 17994 897 1115
19/04/2020 19790 18492 875 498
20/04/2020 20643 19051 853 559
21/04/2020 21474 20223 831 1172
22/04/2020 22284 21060 810 837
23/04/2020 23073 21787 789 727
24/04/2020 23841 22792 768 1005
25/04/2020 24589 23635 748 843
26/04/2020 25317 24055 728 420
27/04/2020 26026 24393 708 338
28/04/2020 26714 25302 689 909
29/04/2020 27384 26097 670 795
30/04/2020 28034 26771 651 674
01/05/2020 28667 27510 632 739
02/05/2020 29281 28131 614 621
03/05/2020 29877 28446 596 315
04/05/2020 30456 28734 579 288
05/05/2020 31018 562
06/05/2020 31563 545
07/05/2020 32092 529
08/05/2020 32605 513
09/05/2020 33103 497
10/05/2020 33585 482
11/05/2020 34052 467
12/05/2020 34505 453
13/05/2020 34944 439
14/05/2020 35369 425
15/05/2020 35781 412
16/05/2020 36180 399
17/05/2020 36566 386
18/05/2020 36939 374
19/05/2020 37301 362
20/05/2020 37651 350
21/05/2020 37989 339
22/05/2020 38317 327
23/05/2020 38634 317
24/05/2020 38940 306
25/05/2020 39236 296
26/05/2020 39523 286
27/05/2020 39800 277
28/05/2020 40067 268
29/05/2020 40326 259
30/05/2020 40576 250
31/05/2020 40817 242
01/06/2020 41051 233
02/06/2020 41276 226
03/06/2020 41494 218
04/06/2020 41705 210
05/06/2020 41908 203
06/06/2020 42104 196
07/06/2020 42294 190
08/06/2020 42477 183
09/06/2020 42653 177
10/06/2020 42824 171
11/06/2020 42989 165
12/06/2020 43148 159
13/06/2020 43301 153
14/06/2020 43449 148
15/06/2020 43592 143
16/06/2020 43730 138
17/06/2020 43864 133
18/06/2020 43992 129
19/06/2020 44116 124
20/06/2020 44236 120
21/06/2020 44351 115
22/06/2020 44463 111
23/06/2020 44570 108
24/06/2020 44674 104
25/06/2020 44774 100
26/06/2020 44871 97
27/06/2020 44964 93
28/06/2020 45054 90
29/06/2020 45140 87
30/06/2020 45224 84
01/07/2020 45304 81
02/07/2020 45382 78
03/07/2020 45457 75
04/07/2020 45530 72
05/07/2020 45600 70
06/07/2020 45667 67
07/07/2020 45732 65
08/07/2020 45794 63
09/07/2020 45855 60
10/07/2020 45913 58
11/07/2020 45969 56
12/07/2020 46024 54
13/07/2020 46076 52
14/07/2020 46126 50
15/07/2020 46175 49
16/07/2020 46222 47
17/07/2020 46267 45
18/07/2020 46311 44
19/07/2020 46353 42
20/07/2020 46394 41
21/07/2020 46433 39
22/07/2020 46471 38
23/07/2020 46507 36
24/07/2020 46542 35
25/07/2020 46576 34
26/07/2020 46609 33
27/07/2020 46641 32
28/07/2020 46671 30
29/07/2020 46701 29
30/07/2020 46729 28
31/07/2020 46756 27
01/08/2020 46783 26
02/08/2020 46808 26
03/08/2020 46833 25
04/08/2020 46857 24
05/08/2020 46880 23
06/08/2020 46902 22
07/08/2020 46923 21
08/08/2020 46944 21
09/08/2020 46964 20
10/08/2020 46983 19
11/08/2020 47002 19
12/08/2020 47019 18
13/08/2020 47037 17
14/08/2020 47054 17
15/08/2020 47070 16
16/08/2020 47085 16
17/08/2020 47100 15
18/08/2020 47115 15
19/08/2020 47129 14
20/08/2020 47143 14
21/08/2020 47156 13
22/08/2020 47168 13
23/08/2020 47181 12
24/08/2020 47192 12
25/08/2020 47204 11
26/08/2020 47215 11
27/08/2020 47226 11
28/08/2020 47236 10
29/08/2020 47246 10
30/08/2020 47256 10
31/08/2020 47265 9
01/09/2020 47274 9
02/09/2020 47283 9
03/09/2020 47292 9
04/09/2020 47300 8
05/09/2020 47308 8
06/09/2020 47316 8
07/09/2020 47323 7
08/09/2020 47330 7
09/09/2020 47337 7
10/09/2020 47344 7
11/09/2020 47351 7
12/09/2020 47357 6
13/09/2020 47363 6
14/09/2020 47369 6
15/09/2020 47375 6
16/09/2020 47381 6
17/09/2020 47386 5
18/09/2020 47392 5
19/09/2020 47397 5
20/09/2020 47402 5
21/09/2020 47407 5
22/09/2020 47412 5
23/09/2020 47416 5
24/09/2020 47421 4
25/09/2020 47425 4
26/09/2020 47429 4
27/09/2020 47433 4
28/09/2020 47437 4
29/09/2020 47441 4
30/09/2020 47445 4
01/10/2020 47449 4
02/10/2020 47453 4
03/10/2020 47456 4
04/10/2020 47459 3
05/10/2020 47463 3
06/10/2020 47466 3
07/10/2020 47469 3
08/10/2020 47472 3
09/10/2020 47475 3
10/10/2020 47478 3
11/10/2020 47481 3

 

Leave a comment