Dear Ben,

Thank you for taking the time to meet with the Office for Statistics Regulation (OSR), the regulatory arm of the UK Statistics Authority. We provide independent regulation of all official statistics produced in the UK, and investigate concerns raised with the Authority.

As you know, we were contacted about the presentation of modelling in the Executive Summary of the Module 2 report of the UK Covid-19 Inquiry, specifically the section which says that “Had a mandatory lockdown been imposed on or immediately after 16 March 2020, modelling has established that the number of deaths in England in the first wave up until 1 July 2020 would have been reduced by 48% – equating to approximately 23,000 fewer deaths”. Evidence given to the inquiry by Professor Neil Ferguson from Imperial College London and from an academic paper (Knock et al) provided the figure of a 48% reduction in deaths. The 23,000 figure was calculated by the Inquiry by subtracting the estimated number of deaths from the actual number of deaths.

While the inquiry report does not fall within our statutory remit, we considered the issue on an advisory basis, focusing on the communication of these figures in relation to the Standards for the Public Use of Statistics, Data and Wider Analysis, within the Code of Practice for Statistics.

I want to emphasise that we are not questioning the inquiry’s conclusions, including those on the timing of advisory and mandated lockdowns, nor would it be appropriate for us to do so. Our focus is the presentation of statistical evidence, and the risk that the wording chosen for the Executive Summary is a misleading representation of the underlying analysis. We thought it would be helpful to share our findings with you because in future modules of the Inquiry report, you may wish to present modelling or other analytical evidence and it will be important that this is communicated clearly and accurately.

Firstly, while steps have been taken to communicate uncertainty in the main body of the report, we consider that the Inquiry’s Executive Summary of the modelling does not sufficiently communicate the level of uncertainty associated with the analysis. Uncertainty is inherent in statistical models, some of which is quantifiable as an interval around an estimate, but there is also broader uncertainty due to unavoidable assumptions made in the modelling process. The use of the words “established” and “would have been reduced” overstates the confidence in the 48% figure and has the potential to mislead the public about the certainty of the finding.

The second aspect concerns what the ‘23,000’ figure represents. The Inquiry’s Executive Summary for Module 2 says the ‘23,000’ is an estimate of the effect of bringing the mandatory lockdown forward a week. However, in investigating this case, we have confirmed with Professor Ferguson that the ‘counter-factual’ (the hypothetical scenario which the model aimed to describe) was that both the mandatory lockdown and the voluntary measures announced on 16 March were each brought a week earlier. While we acknowledge that the Knock et al paper did not explicitly set this out as the counterfactual, there is a risk that the Executive Summary is misleading about what the ‘23,000’ actually refers to.

Given the weight put on the ‘23,000’ figure in the Executive Summary and in the subsequent media coverage, we consider it is important to have clarified this point about the counterfactual.

We invite you to set out the steps you are taking in response to our findings, to support the future communication of statistics and analysis by the UK Covid-19 Inquiry.

Yours sincerely

Ed Humpherson
Director General for OSR