Dear Iain

Communicating models and methods to users of the Coronavirus (COVID-19) Infection Survey

Thank you to you and your team for engaging with us as we investigated concerns about communication of the methods used to produce the estimates of the incidence rate of COVID-19 from the Coronavirus (COVID-19) Infection Survey. These have centred around the availability of clear information about methods and quality that meets the needs of all types of users, and developing an ongoing positive dialogue with users.

Estimates of COVID-19 positivity and incidence rates are widely used and play a vital role in informing the government response to the pandemic. These statistics are unusual in their complexity, and their wide user base, so it is important that the modelling and data are presented in a clear and accessible way to this broad range of users to meet their needs. The challenge is that the needs of general users must be met, by making the explanations accessible, and at the same time, the needs of technical users must be met, by providing adequate detail about the methods used.

The additional information published in the bulletin on 11 December regarding the calculation of incidence rates demonstrates ONS’s commitment to improving its communication of the underlying methods. However, it is clear that the presentation of the statistics and descriptions of the methods needs to be much clearer still, both for general and technical users. For example, the distinction between the ‘official reported estimate’ and ‘modelled estimate’ of the incidence rate remains a source of confusion for users, particularly as the estimates are presented alongside each other.

Due to the complexity of the statistical models, it is also important that detailed information is accessible to technical users to allow them to evaluate the methods used and to fully understand the data and the way that estimates change due to the modelling. The pre-print academic paper (which has yet to be peer-reviewed), submitted by your academic partners, goes into considerable detail on the methods used. However, it would be helpful to reflect upon recent concerns raised by users and work with users to produce clear and accessible summaries of the rationale for the methodology choices made. This should include information about the assumptions made and how the performance of the models is evaluated and improved.

The COVID-19 Infection Survey continues to deliver valuable insight about the pandemic, at speed and to a wide range of users, and we support its ongoing development. We appreciate the resource constraints and pressure that you are under, but it would benefit ONS to demonstrate more clearly that it is open to engagement with users. This would be achieved by sharing information about the user engagement you are already undertaking and engaging more openly with a wide range of users, including those with new requests or questions.

We will follow up the points raised in this letter through our in-depth review of the survey.

Yours sincerely

 

Ed Humpherson

Director General for Regulation